Herding Packets

This year (and especially in the past few months) there have been a lot of new solutions announced in the network virtualization and network overlay platform arenas. These solutions hold great potential, but in my opinion the vendors of these solutions need to get on board with a team approach to IT and avoid marketing to server engineers by throwing the networking team under the bus.

While I’m glad to see that in detailed technical articles/blogs and in conversation, there’s an acknowledged need for a robust and reliable underlay network, I’m concerned at the broader marketing message that several of these vendors seem to be pushing. To paraphrase and add a little snark, it goes something like this:

“Hey all you smart, handsome server engineers! Stop waiting on those slow, inefficient network guys! They take two months to create a VLAN and get in the way of your application deployment! Just use SpiffyCo Network Overlay Technology and do it yourself! Say ‘sayonara’ to waiting on the network! Tell your CIO you want SpiffyCo Overlay Networks because you care much more about getting work done than those lazy, weird-smelling network goons! Order today and receive a free vial of geniune unicorn tears (just pay separate handling and processing)!”

The target audience of these messages is pretty obvious. So, why is the network team perceived to be so slow? It’s not all FUD. Sometimes the network team is slow. But it’s not because it takes two months to type “vlan 1440” a dozen times or mount and connect a new switch. Here are a few likely reasons:

• The network guys usually have to do design, engineering, capacity planning, and install activity. A new VLAN might require a change to multiple switches that may require one or more maintenance windows (depending on the organization’s tolerance for change). Or adding that new application may demand larger links from the access layer up to the core to handle load from the new application servers talking to database or storage systems. While turning up a server is pretty unlikely to impact other servers around it, modifying the network on which everything rides usually makes management folks a bit nervous and that means waiting for some maintenance window to create the VLAN, add it to trunks, etc.

• The lack of automation and tools force very manual workflows compared to virtual server technologies that basically instantiate a new server in seconds. This is a legitimate gripe considering the impact of server virtualization in terms of new application provisioning. It’s absolutely fair to say that it may take somewhat longer to get a new VLAN, routed interface, firewall zone, and load-balancer context created end-to-end than it does to start booting the new virtual servers that will go in it.

• While server engineers are usually focused in the data center, the network engineer often wears many hats. The person or team doing data center networking is often also expanding the LAN for the newly built floor of the building, migrating the WAN to reduce costs, generating reports for management on employee Internet usage, and troubleshooting the application issues that are all blamed on “the network”. While some very large or very well organized enterprises differentiate between data center and enterprise networking teams, many of the medium-size enterprises I work with really do not.

• The network team is often simply understaffed. Some light research was unable to find much in terms of recommended guidelines for network ports or devices per admin (it’s highly variable as you can imagine) but in my experience it’s not uncommon to see a network “team” or one or two engineers doing the vast majority of networking for enterprises that may consist of hundreds of network devices and thousands or even tens of thousands of ports. With very small teams of network admins/engineers handling infrastructures of that size, with all the duties mentioned above, it’s no wonder requests sit in a queue for days, weeks, or months.

Network virtualization has the potential to improve things a lot. It can improve the responsiveness of data center network admins to project and M/A/C requests by separating the design and scaling of the underlay from the complexity and constant change of the overlay. It can help compartmentalize, and thereby reduce the risk of, that change. In theory the data center underlay fabric should remain very stable and require little operational change, and making necessary change to portions of the overlay (even adding a new overlay) should be treated as lower-risk than modifying the foundational underlay network. The automation capabilities of network virtualization platforms will also offer a means to speed up deployment such as a new VLAN or VRF definition, possibly along with firewall and application delivery policy, in a centralized control-plane which is immediately propagated to all relevant forwarding plane elements in the network.

On the other hand, unless organizations begin to make a more explicit delineation between data center and enterprise networking roles, that same network admin still has to chase down the network loop on the 16th floor, troubleshoot the CEO’s remote access VPN, prep temporary network access for the trade show booth, and prove that it’s “not the network.”

My biggest hope is that as the network virtualization market solidifies, the message will shift from enticing server engineers to end-run around the network team, to a message of improving real teamwork across functions in IT and leveraging various skill-sets collaboratively to create visible business results faster and less disruptively.