Herding Packets

Although I look forward to network fabric management seeing broad deployment, the fact is that many networks (and especially enterprise LAN/WAN) will be managed with traditional methods for some time yet. Inconsistencies in device configurations can present a barrier to some types of automation. In this article, we’ll explore that very challenge and a resolution I came up with to handle it.

Not long ago, I was trying to automate an ACL line insertion task with a popular network configuration push tool that basically does CLI interaction with something like Expect. I needed to push a similar change to about 20 devices with minimal effort. Unfortunately, when looking at the ACL on several sample devices out of the target device pool, I saw things like this:

R1(config)#do sh access-list NAT
Extended IP access list NAT
14 deny ip 10.10.1.48 0.0.0.7 10.0.0.0 0.255.255.255
20 deny ip 10.11.1.48 0.0.0.7 10.0.0.0 0.255.255.255
25 permit ip 10.10.1.48 0.0.0.7 192.168.0.0 0.0.255.255
30 permit ip 10.10.1.48 0.0.0.7 any
43 permit ip 10.11.1.48 0.0.0.7 any

You can see from the line numbers that various line item inserts have been done over time. This was the case with several of the devices, making me less than confident that simply inserting a specific line number ACE into the ACL would work reliably. For example, given the ACL above, trying to insert a line with entry number 25 (which worked on several of the devices) would result in this:

R1(config)#ip access-list ext NAT
R1(config-ext-nacl)# 25 deny ip 10.10.1.48 0.0.0.7 172.16.0.0 0.15.255.255
% Duplicate sequence number

Unlike the Cisco ASA, where inserting a duplicate line number has a true “insert” effect pushing the rest of the ACL down, on IOS the insert fails and I’m left to either go look at each unit’s ACL and pick a device-specific number I can use, or try the config push over and over guessing numbers.

Luckily, IOS has a feature to relabel the ACEs in an ACL using whatever starting index and increment value you wish. By using this “resequence” command at the start of my script, I was able to normalize each ACL to a predictable set of sequence numbers and do my insert in the right spot using a sequence number that I now knew couldn’t conflict. I then had my script run the resequence again to normalize the ACL following the insert, so that next time around I’d be less likely to his this issue.

!--- This command renumbers the ACEs in Access List "NAT", starting at entry number 10 and incrementing each line number by 10.
R1(config)#ip access-list resequence NAT 10 10
R1(config)#!
R1(config)#do sh access-list NAT
Extended IP access list NAT
10 deny ip 10.10.1.48 0.0.0.7 10.0.0.0 0.255.255.255
20 deny ip 10.11.1.48 0.0.0.7 10.0.0.0 0.255.255.255
30 permit ip 10.10.1.48 0.0.0.7 192.168.0.0 0.0.255.255
40 permit ip 10.10.1.48 0.0.0.7 any
50 permit ip 10.11.1.48 0.0.0.7 any

You can see from the above output that the resequence command normalized my ACE numbering. Now, I can make my tweaks using line numbers that won’t conflict with the existing entries:

R1(config)#ip access-list ext NAT
R1(config-ext-nacl)# 25 deny ip 10.10.1.48 0.0.0.7 172.16.0.0 0.15.255.255
R1(config-ext-nacl)#!
R1(config-ext-nacl)#ip access-list resequence NAT 10 10
R1(config)#end
R1#
!--- Now we check the ACL again to make sure our new ACE landed in the right spot:
R1#show access-list NAT
Extended IP access list NAT
10 deny ip 10.10.1.48 0.0.0.7 10.0.0.0 0.255.255.255
20 deny ip 10.11.1.48 0.0.0.7 10.0.0.0 0.255.255.255
30 deny ip 10.10.1.48 0.0.0.7 172.16.0.0 0.15.255.255
40 permit ip 10.10.1.48 0.0.0.7 192.168.0.0 0.0.255.255
50 permit ip 10.10.1.48 0.0.0.7 any
60 permit ip 10.11.1.48 0.0.0.7 any

This worked perfectly, and it occurred to me that it’s probably not a bad idea at all to add this normalization process to just about any automated ACL tweak, just to ensure I don’t hit an unexpected sequence number conflict. This experience is a good reminder that in all forms of network automation, we make assumptions about the consistency of all our device configurations. It’s important to validate these assumptions and correct inconsistencies if we want network automation to work. Even in a future world where we’re a less bound by CLI behaviors, what if one of the devices had a different ACL name, or the NAT was configured with a route-map instead of a simple ACL? Inconsistencies are a real enemy for automation, so we should always be on the lookout for them, testing scripts in a non-invasive manner to validate what they do, correcting inconsistencies in our environments before they pose automation problems, and monitoring/auditing configurations to identify when inconsistencies creep in.