Herding Packets

The release of the Cisco CSR1000V is definitely garnering some buzz around the Internets. The platform is intended for deployment in a VMWare ESXi environment, but it seems to work just fine using VMWare’s desktop virtualization products as well. Join me as I boot up a couple of cloudy routers on my Mac Mini! The CSR platform was announced at Cisco Live 2012 in San Diego. It’s finally been released and already tutorials and information are popping up on the ‘net. If you have a Cisco.com account, you can download the software here. You can also check out some tutorials such as this blog post from INE on Installing the Cloud Services Router in VMWare ESXi, or this work by Daniel Dib on integrating the CSR1000V with Dynamips. As the CSR is distributed in both an OVA format for import into ESXi, and also in an ISO format, I figured I’d see if the ISO would install under VMWare Player on my work laptop. It did, and seemed to run fine. So this evening, I’m going to try it on VMWare Fusion 5 running on my Mac Mini (quad-core i7 and 16GB RAM, so we should have plenty of resources to run a couple of them).

An Easier Install than Turning Rack Screws

After downloading the image from Cisco’s site, we need to create a VM. I go through the normal VMWare Fusion new machine setup, with the exception of customizing a few settings at the end. Cisco dictates the VM for the CSR must have 4GB of RAM and 4 cores. I will oblige. First, select the ISO image for installation: Next, we select the OS type. It’s well known that the CSR (and all IOS XE) runs on a Linux base. I’m assuming it’s a 2.6 kernel. It seems to work. After clicking “Continue” above and seeing the machine summary, I edit the virtual machine settings to set the CPU and memory per Cisco’s specs: By the way, when they say 4GB of RAM, they mean it. Three of these things ate up a solid 10GB of RAM on my test machine. The VMX processes were using about 2.9 GB of RAM at idle. Next, we set up the NICs. The Gig0 interface on the CSR is automatically put in to a management VRF called “Mgmt-intf” and as is typical for management interfaces on things like the ASR it seems impossible to remove it from that VRF. So that Gi0 NIC will always be an OOB management interface. For all of my example devices, I bridged this onto my Ethernet adapter and just let it pick up DHCP. I then hit the “Add Device…” button and added another network adapter. This one, I attached to a “host-only” network that I created in Fusion called vmnet3. This will act as one of the links to another CSR in my lab. By the way, a graphical means for creating new VMnet segments is finally available in Fusion 5 via the application preferences. 

Fire It Up!

Once the VM is configured, hit the “Power On” button. We see that the CSR posts and then loads GRUB:

Booting these things seems to take a while. Particularly the first boot, as it’s actually installing from the CD image. After a while it will spit out some console output:

Following the install process, the VM automatically ejects the disc image, reboots, and then chugs a bit more. After some patience, we are rewarded with a familiar screen:

You can see above that the Gi0 management interface actually got DHCP from my server already. Lo and behold, that most basic of tests seems to work:

So there we go, a virtual IOS-XE router running in VMWare Fusion, sending pings to the universal ping target.

Rinse, and Repeat

One router is fun to poke around in, but it takes more than one router to make a network. At this point, I shut my router off, and in Finder simply made a few copies by holding Option while dragging the icon to make a new one. I added -1, -2, and -3 identifiers to them. Note that the complete VM is about 650MB.

I went back into VMWare Fusion, and went to File > Open… to re-import the new VMs. I went into each and added the -1, -2, and -3 to the names. I also reassigned some NICs so my topology went like this:

I booted all of the routers. You can see in this screenshot that the UDI of router 2 was changed. The UDI is basically the identifying element that licenses are based on (much like a serial number), so if you run off copies of a router they must be licensed individually. More on licensing at the end of this post.

Remember what I was saying about memory? Here is my machine with all 3 CSRs booted:

Ouch. Anyway, once all the routers were up, I put some very basic config on each (like an enable and line password), then used a normal terminal to telnet into each on their Gi0 management interfaces. Success!

Back to Text Mode

Now that you’ve seen all the routers get built and booted, and we’re into telnet on each of them, I will stop with the screenshots and just show normal terminal output. By the way, if you don’t want to use Gi0 for management, you can add a serial port to the VM and run the command “platform console serial,” and the console output will go to the serial port instead of the VMWare virtual console. I didn’t mess with that in my lab. Apparently in the fancy version of ESXi vSphere you can easily provide console server functionality in which case this might be a good feature. Once I got the routers up, I noticed that CDP is disabled by default. I’m assuming maybe they disabled it to minimize control-plane activity that would chew up virtual CPU, especially since the CSR is kind of intended for an environment that it may not really have a lot of neighbors to be sharing link info with. Once I enabled CDP globally and on each interface (yes, both were disabled by default), we got CDP info:

CSR-2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
CSR-2(config)#cdp run
CSR-2(config)#^Z
CSR-2#show cdp int
CSR-2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
CSR-2(config)#int gi1
CSR-2(config-if)#cdp en
CSR-2(config-if)#int gi2
CSR-2(config-if)#cdp en
CSR-2(config-if)#do sh cdp int
GigabitEthernet1 is up, line protocol is up
  Encapsulation ARPA
  Sending CDP packets every 60 seconds
  Holdtime is 180 seconds
GigabitEthernet2 is up, line protocol is up
  Encapsulation ARPA
  Sending CDP packets every 60 seconds
  Holdtime is 180 seconds
CSR-2(config-if)#^Z
CSR-2#
CSR-2#
CSR-2#sh cdp ne
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
                  D - Remote, C - CVTA, M - Two-port Mac Relay
 
Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
CSR-3.mccouch.private
                 Gig 2             168              R I   CSR1000V  Gig 1
CSR-1.mccouch.private
                 Gig 1             159              R I   CSR1000V  Gig 1
CSR-2#

Next I did a simple EIGRP config and checked to see that I had adjacencies and routes for connected interfaces and an X.X.X.X loopback that I’d already added to each router:

CSR-1#show run | s router ei
router eigrp 123
 network 0.0.0.0
 
 
CSR-1#show ip ei ne
EIGRP-IPv4 Neighbors for AS(123)
H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq
                                                   (sec)         (ms)       Cnt Num
0   10.1.12.2               Gi1                      11 00:00:26    2   100  0  7

CSR-1#show ip ro ei | b ^Gate

Gateway of last resort is not set
      2.0.0.0/24 is subnetted, 1 subnets
D        2.2.2.0 [90/130816] via 10.1.12.2, 00:00:31, GigabitEthernet1
      3.0.0.0/24 is subnetted, 1 subnets
D        3.3.3.0 [90/131072] via 10.1.12.2, 00:00:31, GigabitEthernet1
      10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
D        10.1.23.0/24 [90/3072] via 10.1.12.2, 00:00:31, GigabitEthernet1
CSR-1# 

Cool! We have routes! Do we have reachability? A ping script is a great way to find out:

CSR-1#tclsh
CSR-1(tcl)#foreach ip {
+>1.1.1.1
+>2.2.2.2
+>3.3.3.3
+>10.1.12.1
+>10.1.12.2
+>10.1.23.2
+>10.1.23.3
+>} { ping $ip repeat 2 timeout 1 }
Type escape sequence to abort.
Sending 2, 100-byte ICMP Echos to 1.1.1.1, timeout is 1 seconds:
!!
Success rate is 100 percent (2/2), round-trip min/avg/max = 1/1/1 ms
Type escape sequence to abort.
Sending 2, 100-byte ICMP Echos to 2.2.2.2, timeout is 1 seconds:
!!
Success rate is 100 percent (2/2), round-trip min/avg/max = 1/2/3 ms
Type escape sequence to abort.
Sending 2, 100-byte ICMP Echos to 3.3.3.3, timeout is 1 seconds:
!!
Success rate is 100 percent (2/2), round-trip min/avg/max = 2/2/2 ms
Type escape sequence to abort.
Sending 2, 100-byte ICMP Echos to 10.1.12.1, timeout is 1 seconds:
!!
Success rate is 100 percent (2/2), round-trip min/avg/max = 1/1/1 ms
Type escape sequence to abort.
Sending 2, 100-byte ICMP Echos to 10.1.12.2, timeout is 1 seconds:
!!
Success rate is 100 percent (2/2), round-trip min/avg/max = 1/1/1 ms
Type escape sequence to abort.
Sending 2, 100-byte ICMP Echos to 10.1.23.2, timeout is 1 seconds:
!!
Success rate is 100 percent (2/2), round-trip min/avg/max = 1/1/1 ms
Type escape sequence to abort.
Sending 2, 100-byte ICMP Echos to 10.1.23.3, timeout is 1 seconds:
!!
Success rate is 100 percent (2/2), round-trip min/avg/max = 2/2/2 ms
CSR-1(tcl)#exit
 

Closing Thoughts (and maybe a bit more)

I’ll be perfectly honest: I have a relatively hard time wrapping my brain around the supposedly myriad uses for the CSR platform. I can understand a use case for a public cloud type thing where you have an instance acting as an MPLS CE node for WAN connectivity or a VPN connection back to a main data center. Maybe that’s pretty much what it’s for in most cases. Extending MPLS switching into the cloud data center and LISP and stuff like that is a little outside of my grasp to see how helpful it would be.

I have a couple of customers looking into public cloud “stuff” where perhaps the CSR could offer a more attractive option for connecting the cloud servers back to the customer’s network than either a provider-based solution (like the Amazon EC2 VPN) or some other Linux-based firewall solution like PFSense being used to generate a VPN tunnel (both of which I’ve recently come across in my day-job).

Honestly, I’m much more interested in the CSR as a flexible lab platform. For years now, network engineers have had only a few options for Cisco labbing:

• A physical lab – Real world behavior, but expensive to build, power hungry, noisy.. Still has potential issues with Cisco licensing, and with the new ISR G2 license model, real-gear labbing with advanced features could become quite difficult.
• GNS3/Dynamips – Actually can work very well and the non-Cisco extensions for running other types of hosts using QEMU and VirtualBox are an excellent edition to GNS3. Unfortunately, Dynamips is reaching the end of the road with IOS 15.1, as it only emulates MIPS platforms like the 2600, 3600, 3700, and earlier 7200 NPEs.
• IOU/IOL – I’m not going to go into great detail about Cisco’s internal builds of IOS for Linux and Unix platforms (Google can help you if you’d like more info) but the difficulty with these is obtaining them, since they’re internal Cisco tools not made available for public access, and in addition they tend to be buggy on certain features. The only benefit of IOU is that there is a mostly-working layer 2 implementation that can act like a switch. IOU is what is used in the CCIE R&S troubleshooting section (and rumored to be used in the config section of the upcoming version 5), but it’s not a very good option for every day lab work.

None of these options combines convenience, flexibility, reasonable cost, and the ability to use them without being on the wrong side of Cisco’s licensing or even copyright terms.

The CSR potentially does offer this. According to the release notes, the basic install is enabled for 2.5 Mb/s of throughput, apparently with a 60-day full feature trial license. It’s not entirely clear to me what happens after 60-days, whether it reverts to a base license that forgoes some of the advanced features like VPN and QOS, or if it stops functioning entirely until a license subscription is purchased, or if it keeps working full featured, but at the limited throughput level. In any case, having to reinstall a new instance after 60 days to reset the demo timeline is just fine with me from a lab perspective.

I sincerely hope that people won’t abuse this option by trying to run these things in production perpetually using only the demo license, as doing so will quickly force Cisco to drop this option. The 2.5 Mb/s throughput limit would probably also be a big hindrance in real-world usage as most enterprises that would be actually deploying public-cloud infrastructure would probably want more than a couple T1’s worth of throughput, but for lab use that limitation is also really not a problem.

Once an Amazon AMI version is officially released (or if someone were to convert the VMWare image to AMI), you could run a 9 or 10-router lab (using EC2 Medium nodes) for about $1/hr which is much less expensive than most rack rental options. A home lab would probably need to be an ESX box with 32GB of RAM — not a super-thrifty option, but still probably less expensive than building a physical rack of 10 routers. Especially current-generation, IOS XE routers!

I’m looking forward to playing more with the CSR1000V and reading through the configuration guide in greater detail. There are a number of new commands present to handle the virtual nature of this platform and I’d like to understand those. This will also intersect well with my desire to start getting more into VMWare by building a decent home ESXi lab.