Removing Private BGP ASNs and the Importance of Lab Validation

I have an interesting project going on with two of my customers. One of them is buying a web-hosting business from the other, and we have to migrate all of the resources (including IP space) from one organization to the other in the same colo. Of course there is a lot involved with this, but one of the things we had to consider was the routing. For the short term (the next few months), the company buying the hosting business (let’s call them BUYER, Inc), will not have its own Internet connection from the colo, but will instead transit through the selling company (we’ll call them SELLER Corp.).

My first thought on safely controlling this routing was to do BGP. BGP allows us to safely move the IP space presently used by SELLER over to BUYER either in whole or piecemeal and we can provide a number of controls to make sure neither of these companies causes any problems (accidental or otherwise) to the other during the transition. Also, since SELLER’s existing design utilizes two BGP-capable Internet routers, and the new design I spec’d for BUYER also has redundant BGP routers, this made it easy to provide a backup path in the event of a router or link failure. Here is a simple diagram of what we have:

Photo Aug 17, 11 06 58 AM

Note the interesting thing here: SELLER Corp. has a private AS number. This is really how the topology is, as SELLER Corp has two connections to the same ISP, and only that ISP. And because BUYER Inc. will be doing the same thing, neither company is using a public ASN at this time. This will be important in a moment.

The basic idea of the design (both from ISP to SELLER and SELLER to BUYER) is that by using BGP attributes like Local Preference, the entity can ensure that only one link is in use at a time. ISP provides a certain bandwidth guarantee to SELLER, and SELLER is providing a bandwidth guarantee to BUYER. It is much easier to enforce such contracts with a policer if only one link is active at a time, but the second link is waiting in the wings to take over if the primary link has a problem.

The Problem

As BUYER hadn’t yet begun any sort of coordination with the ISP yet, we didn’t know what BGP AS might be assigned to them for a private peering. Hence, we had to come up with our own private BGP AS for the peering between SELLER (AS 64512) and BUYER. I chose AS 65001 for BUYER, for no particular reason.

The issue here is that the ISP shouldn’t be accepting any advertisements that include any BGP AS other than the one they assigned to SELLER (64512). So how do we hide the downstream network? Luckily, Cisco’s BGP implementation has a tool for that, the neighbor remove-private-as command. It’s meant for exactly this purpose, to “hide” downstream networks that have private AS numbers and instead substitute the AS (assumed to be a public ASN) of the network sending it on to another eBGP neighbor. The ISP likely uses this same technique at their border with other ISP peers to hide SELLER’s private ASN.

Here was my concern: SELLER also uses a private ASN. So what happens? Does the command fail? Does it remove all of the AS_PATH info completely? The docs seem to cover this with the following statements:

The neighbor remove-private-as command removes private AS numbers even if the AS path contains only private AS numbers. There is no likelihood of a 0-length AS path because this command can be applied to eBGP peers only, in which case the AS number of the local router is appended to the AS path.

This seems to cover my exact case, and it also makes logical sense that the “remove” function happens on the advertisements prior to sending the update to the eBGP peer (and the moment of sending the update is when the local ASN is appended). But if there is one thing I’ve learned, it’s that when dealing with corner cases it’s best to verify before you assume it works as documented or worse yet, assume how you think it might work. So it’s lab time.

The Lab

To replicate this, I came up with a simple lab topology in GNS3. I didn’t need to make the entire, redundant design end-to-end, as I was really just testing the behavior of the “remove-private-as” command. Here was my topology:

Photo Aug 17, 10 28 03 AM

Links between routers use the “XY” format, so the link between R1 and R2 is 12.12.12.0/24, and the link from R3 to R4 is 34.34.34.0/24. The routers had stub networks hanging off of them in the X.X.X.0/24 format, so R4’s stub network (representing the “hidden” private network) was 4.4.4.0/24.

The basic BGP setup is extremely straight-forward, so I will leave it as an exercise for you, dear reader, to work it out yourself if you want to replicate it. Once all my IBGP and EBGP peerings were up, I looked at the BGP table of R1 (representing our ISP). Be sure to scroll to the far right of the output to see the important bits:

R1#show ip bgp
BGP table version is 5, local router ID is 12.12.12.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i - IGP, e - EGP, ? - incomplete

Network          Next Hop            Metric LocPrf Weight Path
*> 1.1.1.0/24       0.0.0.0                  0         32768 i
*> 2.2.2.0/24    12.12.12.2                  0             0 64512 i
*> 3.3.3.0/24    12.12.12.2                  0             0 64512 i
*> 4.4.4.0/24    12.12.12.2                                0 64512 65001 i

R1#ping 4.4.4.1 so 1.1.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.1, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/81/116 ms

From a connectivity standpoint this all looks good. But R1 is seeing BUYER’s ASN, 65001, in the path. In the real life scenario, this won’t fly because the ISP sure ought to be filtering out any BGP advertisements from SELLER that have anything other than SELLER’s private ASN of 64512.

So next we apply the “remove-private-as” command to SELLER’s EBGP peering to ISP:

R2(config)#router bgp 64512
R2(config-router)#neighbor 12.12.12.1 remove-private-as
R2(config-router)#do sh run | s router bgp
router bgp 64512
bgp log-neighbor-changes
network 2.2.2.0 mask 255.255.255.0
network 3.3.3.0 mask 255.255.255.0
neighbor 12.12.12.1 remote-as 2001
neighbor 12.12.12.1 remove-private-as
neighbor 23.23.23.3 remote-as 64512
neighbor 23.23.23.3 next-hop-self
R2(config-router)#

After giving BGP a moment to update, we take another look at R1, and success!

R1#show ip bgp
BGP table version is 6, local router ID is 12.12.12.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 1.1.1.0/24       0.0.0.0                  0         32768 i
*> 2.2.2.0/24       12.12.12.2               0             0 64512 i
*> 3.3.3.0/24       12.12.12.2               0             0 64512 i
*> 4.4.4.0/24       12.12.12.2                             0 64512 i

R1#ping 4.4.4.1 so 1.1.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.1, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 60/76/96 ms

R1 (the ISP) no longer sees any evidence of the downstream AS 65001, and yet the networks it is advertising up to SELLER in AS 64512 are reachable from the ISP and anything upstream of it.

Conclusion

In the end, this worked out pretty much the way you’d expect. But this was a feature I’d never used in this way before and with a tight turn-up window available and neither customer being too tolerant of experimentation on their production systems it certainly seemed prudent to spend a few minutes verifying the behavior. The end result was that the turn-up went according to plan and I looked like I had a clue as to what I was doing 🙂

Advertisements
Tagged ,

5 thoughts on “Removing Private BGP ASNs and the Importance of Lab Validation

  1. Read and reread your vendor docs carefully. The behavior of this knob can vary between versions – as in the Cisco doc linked – and between vendors. Especially in mixed AS paths. Sometimes only the left-most private AS is removed. In other cases, no AS is removed.

    • bobmccouch says:

      Thanks for that caveat! As you point out, the behavior does even differ between IOS versions. All the more reason to do some quick lab validation when using something that appears to be an unusual case.

      Thanks for reading!

  2. Jerome Tissieres says:

    Hi Bob.
    Nice post.
    Did you consider the possibility to use confederations for that?
    The result would be the Seller_Corp AS as confederation main AS and the Buyer_Corp AS a confed private AS, so “invisible” to the external peers.

  3. […] Recently, I posted about some lab verification I did during a customer setup where I had to strip private BGP AS numbers (64512-65535) before sending BGP-learned prefixes upstream to an ISP. […]

  4. […] Recently, I posted about some lab verification I did during a customer setup where I had to strip private BGP AS numbers (64512-65535) before sending BGP-learned prefixes upstream to an ISP. […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

@greatwhitetec

Virtualization, Storage, and other techy stuff

The Stupid Engineer

I ask those questions you're too clever to.

Sunay Tripathi's Blog

Pluribus Networks Founder's Blog on OS, Networking, Virtualization, Cloud Computing, Solaris Architecture, etc

Ed Koehler's Blog

Just another WordPress.com weblog

JGS.io

Data networking, stray thoughts, nerdy fun...

Network Heresy

Tales of the network reformation

The Borg Queen

Jottings on the intersection of tech and humanness

Networking From The Trenches

Ramblings about my thoughts, experiences, and ideas.

In Search of Tech

Looking for the next big thing.

Packet Maniac

A day in the life of a maniac packet

Fryguy's Blog

A Network Blog by a Network Engineer

Networking 40,000

Attaining my CCIE with the help of Warhammer 40k

stubby router

just another networking blog

Ronnie Angello

Network Architect . CCIE 17846 . CCDE 2012::1

The Peering Introvert

The sundry interests of Ethan Banks including books, cars, hiking in New Hampshire, religion, music, home theater, technology, geek culture, and social media. And maybe cats.

%d bloggers like this: