Herding Packets

I have an interesting project going on with two of my customers. One of them is buying a web-hosting business from the other, and we have to migrate all of the resources (including IP space) from one organization to the other in the same colo. Of course there is a lot involved with this, but one of the things we had to consider was the routing. For the short term (the next few months), the company buying the hosting business (let’s call them BUYER, Inc), will not have its own Internet connection from the colo, but will instead transit through the selling company (we’ll call them SELLER Corp.).

My first thought on safely controlling this routing was to do BGP. BGP allows us to safely move the IP space presently used by SELLER over to BUYER either in whole or piecemeal and we can provide a number of controls to make sure neither of these companies causes any problems (accidental or otherwise) to the other during the transition. Also, since SELLER’s existing design utilizes two BGP-capable Internet routers, and the new design I spec’d for BUYER also has redundant BGP routers, this made it easy to provide a backup path in the event of a router or link failure. Here is a simple diagram of what we have:

Note the interesting thing here: SELLER Corp. has a private AS number. This is really how the topology is, as SELLER Corp has two connections to the same ISP, and only that ISP. And because BUYER Inc. will be doing the same thing, neither company is using a public ASN at this time. This will be important in a moment.

The basic idea of the design (both from ISP to SELLER and SELLER to BUYER) is that by using BGP attributes like Local Preference, the entity can ensure that only one link is in use at a time. ISP provides a certain bandwidth guarantee to SELLER, and SELLER is providing a bandwidth guarantee to BUYER. It is much easier to enforce such contracts with a policer if only one link is active at a time, but the second link is waiting in the wings to take over if the primary link has a problem.

The Problem

As BUYER hadn’t yet begun any sort of coordination with the ISP yet, we didn’t know what BGP AS might be assigned to them for a private peering. Hence, we had to come up with our own private BGP AS for the peering between SELLER (AS 64512) and BUYER. I chose AS 65001 for BUYER, for no particular reason.

The issue here is that the ISP shouldn’t be accepting any advertisements that include any BGP AS other than the one they assigned to SELLER (64512). So how do we hide the downstream network? Luckily, Cisco’s BGP implementation has a tool for that, the neighbor remove-private-as command. It’s meant for exactly this purpose, to “hide” downstream networks that have private AS numbers and instead substitute the AS (assumed to be a public ASN) of the network sending it on to another eBGP neighbor. The ISP likely uses this same technique at their border with other ISP peers to hide SELLER’s private ASN.

Here was my concern: SELLER also uses a private ASN. So what happens? Does the command fail? Does it remove all of the AS_PATH info completely? The docs seem to cover this with the following statements:

The neighbor remove-private-as command removes private AS numbers even if the AS path contains only private AS numbers. There is no likelihood of a 0-length AS path because this command can be applied to eBGP peers only, in which case the AS number of the local router is appended to the AS path.

This seems to cover my exact case, and it also makes logical sense that the “remove” function happens on the advertisements prior to sending the update to the eBGP peer (and the moment of sending the update is when the local ASN is appended). But if there is one thing I’ve learned, it’s that when dealing with corner cases it’s best to verify before you assume it works as documented or worse yet, assume how you think it might work. So it’s lab time.

The Lab

To replicate this, I came up with a simple lab topology in GNS3. I didn’t need to make the entire, redundant design end-to-end, as I was really just testing the behavior of the “remove-private-as” command. Here was my topology:

Links between routers use the “XY” format, so the link between R1 and R2 is 12.12.12.0/24, and the link from R3 to R4 is 34.34.34.0/24. The routers had stub networks hanging off of them in the X.X.X.0/24 format, so R4’s stub network (representing the “hidden” private network) was 4.4.4.0/24.

The basic BGP setup is extremely straight-forward, so I will leave it as an exercise for you, dear reader, to work it out yourself if you want to replicate it. Once all my IBGP and EBGP peerings were up, I looked at the BGP table of R1 (representing our ISP). Be sure to scroll to the far right of the output to see the important bits:

R1#show ip bgp
BGP table version is 5, local router ID is 12.12.12.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i - IGP, e - EGP, ? - incomplete

Network          Next Hop            Metric LocPrf Weight Path
*> 1.1.1.0/24       0.0.0.0                  0         32768 i
*> 2.2.2.0/24    12.12.12.2                  0             0 64512 i
*> 3.3.3.0/24    12.12.12.2                  0             0 64512 i
*> 4.4.4.0/24    12.12.12.2                                0 64512 65001 i

R1#ping 4.4.4.1 so 1.1.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.1, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/81/116 ms

From a connectivity standpoint this all looks good. But R1 is seeing BUYER’s ASN, 65001, in the path. In the real life scenario, this won’t fly because the ISP sure ought to be filtering out any BGP advertisements from SELLER that have anything other than SELLER’s private ASN of 64512.

So next we apply the “remove-private-as” command to SELLER’s EBGP peering to ISP:

R2(config)#router bgp 64512
R2(config-router)#neighbor 12.12.12.1 remove-private-as
R2(config-router)#do sh run | s router bgp
router bgp 64512
bgp log-neighbor-changes
network 2.2.2.0 mask 255.255.255.0
network 3.3.3.0 mask 255.255.255.0
neighbor 12.12.12.1 remote-as 2001
neighbor 12.12.12.1 remove-private-as
neighbor 23.23.23.3 remote-as 64512
neighbor 23.23.23.3 next-hop-self
R2(config-router)#

After giving BGP a moment to update, we take another look at R1, and success!

R1#show ip bgp
BGP table version is 6, local router ID is 12.12.12.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 1.1.1.0/24       0.0.0.0                  0         32768 i
*> 2.2.2.0/24       12.12.12.2               0             0 64512 i
*> 3.3.3.0/24       12.12.12.2               0             0 64512 i
*> 4.4.4.0/24       12.12.12.2                             0 64512 i

R1#ping 4.4.4.1 so 1.1.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.1, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 60/76/96 ms

R1 (the ISP) no longer sees any evidence of the downstream AS 65001, and yet the networks it is advertising up to SELLER in AS 64512 are reachable from the ISP and anything upstream of it.

Conclusion

In the end, this worked out pretty much the way you’d expect. But this was a feature I’d never used in this way before and with a tight turn-up window available and neither customer being too tolerant of experimentation on their production systems it certainly seemed prudent to spend a few minutes verifying the behavior. The end result was that the turn-up went according to plan and I looked like I had a clue as to what I was doing 🙂