On November 6, I was fortunate to attend the Cisco Application Centric Infrastructure launch event in New York City as part of the Tech Field Day blogger delegation. This event was the much-anticipated unveiling (and acquisition announcement) of Insieme Networks, Cisco’s “SDN Spin-in” which maintained a pretty impressive amount of secrecy over its relatively short existence. The main keynote/announcement event consisted of a lot of flashy marketing videos and various tech executives praising each others’ companies. The tech press has been atwitter with coverage on Cisco’s ACI strategy and various components. I’m not going to try to recap the entire announcement, as others have done a much better job of that than I could, but I’m going to provide my take on each of what I considered to be four related, but somewhat distinct announcements that day. In this post, I cover the Nexus 9000 line of switches.
Nexus 9000
The keystone product that Insieme Networks was working on as part of their strategy that developed into ACI was the Nexus 9000 line. Nexus 9500 is a chassis switch, intended primarily for large data center spine nodes (with the ability to scale up to 288 line rate 40G ports per chassis), and the 9300 line are mostly-fixed-configuration units intended primarily as data center leaf nodes with 10G access and 40G uplinks — although even Cisco acknowledges that they could make a good collapsed aggregation/access layer switch in a more traditional design, especially with Nexus 2000-series FEX hung off of them.
What I Think is Interesting
The Nexus 9500 line has substantial 40G density (288 ports per chassis) and truly impressive 10G density (1152 ports per chassis) in a 13RU package. So with three 9508s stacked up in one rack you could provide nearly 3,500 line-rate 10G ports. Admittedly that would require using quad-SFP breakout cables from each of the 288 40G QSFP+ ports which would be a royal mess.
Still, these things are built for large-scale, high-performance data centers. Big switching fabrics and FEX support could allow us to see (and this is 100% pure speculation!) a 48-port 10G FEX that could use, say, 6 or 8 40G uplinks to get down to 2:1 or even 1.5:1 oversubscription which would scale well in flatter designs. Of course, by doing a leaf-spine architecture, this setup could scale absolutely enormously. Cisco also included the “Insieme ASIC” in the Nexus 9000 (or *will* include it, things were a little unclear at the announcement) and that has the additional benefit of enabling the ACI behavior. According to the Nexus 9500 datasheet, the initial 36-port 40G line card will not be upgradeable to use ACI, a curious decision that I assume will be rectified with a “V2” card — this is probably where the confusion over whether the Insieme ASIC was included from FCS stemmed from.
By basing the basic L2/L3 function of the switch on the Broadcom Trident 2 ASIC, additional functionality is “automatically” there including VXLAN VTEP functionality which assists in compatibility with VXLAN-based overlay networks (including VMWare NSX). Use of this merchant silicon will also hopefully result in attractive pricing. With vendors such as Big Switch and Cumulous pushing the whitebox movement, which revolves around Broadcom Trident- (and Trident 2-) based switches, and other vendors like Arista building products around the same chipset, Cisco needs to respond to the increased pricing pressure by making its own switches based (in part) on the same silicon competitively priced.
While no pricing has been announced yet to my knowledge, I found an interesting quote in this Lippis report on Cisco’s website about the Nexus 9508’s power efficiency. It states:
The three-year cost to power the Cisco
Nexus® 9508 is estimated at $16,208
when fully populated with 1,152 10GbE
ports. The three-year energy cost is
some 3.52% of list price.
Simple algebra tells us then, that $16,208 = (0.0352)ListPrice and thus ListPrice = $460,454. That’s not “cheap” by any definition I’ve ever seen, but on a per-port basis that’s a bargain. A list price of approximately $460,500 divided by the 288 40G ports that would be required to provide the 1152 10G ports Mr. Lippis mentions works out to just about $1600 per 40G port (and of course, no one really pays list, now do they?). That’s only twice the per-port price of 10G ports on the Nexus 5548, or to look at it another way, the per-port 10G pricing is half that of the Nexus 5548. Assuming the quoted number is accurate, the big question then is how much of that $460K list price might be the chassis/fabrics/supervisors vs. the individual line cards.
While I will write about Cisco ACI and the also-announced “optimized NX-OS” in an upcoming post, I think it was a very wise decision on Cisco’s part to make the Nexus 9000 series hardware capable of running as “normal” switches as well as participating in the farther-reaching ACI strategy. This will allow some grass-roots penetration of the platform into networks through new deployments and technology refresh that may ease an organization’s introduction into ACI.
What I Don’t Like
In a word: Fragmentation. The Nexus line is starting to turn into “a few switches for every problem statement.” We have the Nexus 3000 for ultra-low-latency switching, but the 9000 boasts 1-2 microsecond switch latency (and personally I think those willing to pay a premium to get a few hundred nanoseconds of switching latency vs. 1 microsecond switching latency should email me about a bridge I have for sale…). We have the Nexus 6000 line for dense 40G aggregation and 10G access with 40G uplinks, but that’s exactly what the 9508 and 9300 series can provide now. Then there’s the 5548 for when we need some relatively cost-effective 10G ports, but rumor is that the 9K line will be competitively priced against the other Trident 2 switches hitting the market. We have the 7000 series which will maintain some of the advanced feature set that has been stripped from the NX-OS on the 9000 (and which I will address in greater detail soon), but why have two different lines of full-size chassis-based Nexus? As a consultant who does my fair share of pre-sales engineering, I’m trying to digest what is turning into a very complex set of criteria to determine the answer to “which Nexus should I buy?” As the product lines move forward, I expect (and hope) we’ll see some reconvergence; as this point there are at least as many active lines of switches in the Nexus portfolio as in the Catalyst portfolio!
Closing Thoughts
Although the Cisco ACI launch event was decidedly not about hardware, I came away impressed with the Nexus 9000 line. They promise to provide highly-scalable platforms for cloud-scale data centers, but also (hopefully) be competitively priced to battle the developing merchant silicon-based commodity Top of Rack switch market. This will also (hopefully) make them somewhat attainable for smaller and mid-sized customers which is where I primarily focus my time. Cisco has clearly recognized the need to price aggressively and be ready for future growth (according to the datasheet the backplace of the 9508 has “100% headroom” for future line card advances).
In future posts I’ll discuss my thoughts on the ACI software end of things, the “optimized NX-OS” that will become available, and the other exciting hardware announcement that came with the N9K, the 40G BiDi optics.
More Information About the Nexus 9000 Line
I attended the Cisco ACI launch event as a Tech Field Day delegate. Cisco Systems organized the ACI launch event, and thus indirectly paid some portion of my travel expenses to the event. At no time did they ask for, nor where they promised any kind of consideration in the writing of this post. The opinions and analysis provided within are my own and any errors or omissions are mine and mine alone.
Tagged aci, nexus9k, openflow, sdn 
Herding Packets 
In some ways,the 9000 comes in a similar predicament as the 7000 did: Strictly a DC connectivity switch with little of the more advanced features you might need to talk to the rest of your enterprise or elsewhere. I remember seeing the 7000 as having too few nerd knobs to completely replace my 6500, and the same can be said of the 9000 vs 7000 right now.
I think, the 7000 will continue to have its place in the family for a while because of OTV, MPLS and a few other things. Not sure I am seeing the long term value in the 5500 and the 6000 to be honest. Unless they drop the price on those significantly.
[…] Cisco Application Centric Infrastructure: Nexus 9000 […]
[…] initial message was that the Nexus 9000 was out to usurp the role of the Nexus 7000 family. Although this is not what Cisco messaged, it was what came through social channels. I do not […]
[…] was a bit higher than numbers being slung about for some other recent dense-40G solutions from some other vendors but the port density of the Z9500 is higher than just about anything else I’ve heard of so far. […]
[…] been a lot of comments around Cisco ACI/Nexus 9K pricing, and in spite of what Cisco’s said about price per port being market-leading, my gut feel […]