Herding Packets

I’ve been working on Cisco’s ASA firewall platform for years, and I continue to work on a variety of environments with multiple generations of the ASA for clients at H.A. Storage. One of my favorite features of the ASA platform has been the quality of the high-availability failover mechanism, which is generally very reliable, fast, and seamless.

The ASA operates in an Active/Standby high-availability model (don’t believe that the ASA is *truly* Active/Active — that’s a marketing feature). However, one sore spot that has frustrated me as long as I’ve been working on them is the fact that the filesystem has no synchronization between failover mates and requires manual efforts to keep files in sync. Other configuration aspects of the ASAs including some XML customization files that are not stored in the running config all get automatically sync’d to the standby unit, but for actual files that show up on the flash filesystem, this does not happen.

This has certainly caused me some frustration and occasional embarrassment over the years, but one thing I’ve learned along the way is that when doing file operations either from the CLI or the ASDM, it’s important to follow one simple rule:

The reason for this rule is fairly obvious when you see it in action. First, “Delete from the active” is based on the fact that the “delete” CLI command actually gets replicated from the active ASA to the standby ASA. For example:

FW1/act# dir

Directory of disk0:/

108 -rwx 25251840 06:25:26 Aug 15 2012 asa844-5-k8.bin
110 -rwx 25214976 07:28:06 Jul 13 2012 asa844-1-k8.bin
117 -rwx 18927088 07:28:38 Jul 13 2012 asdm-649.bin

260296704 bytes total (112099328 bytes free)

If we look a the contents of the standby’s flash, we see the same files (I’ve removed some non-pertinent file entries just to keep things obvious):

FW1/stby# dir

Directory of disk0:/

108 -rwx 25251840 06:31:26 Aug 15 2012 asa844-5-k8.bin
109 -rwx 25214976 05:08:20 Jul 10 2012 asa844-1-k8.bin
116 -rwx 18927088 05:08:54 Jul 10 2012 asdm-649.bin

260296704 bytes total (112128000 bytes free)

Now, the file is deleted from the active unit:

FW1/act# del /noconfirm disk0:/asa844-1-k8.bin
FW1/act# dir

Directory of disk0:/

108 -rwx 25251840 06:25:26 Aug 15 2012 asa844-5-k8.bin
117 -rwx 18927088 07:28:38 Jul 13 2012 asdm-649.bin

260296704 bytes total (137314304 bytes free)

And it’s also gone on the standby:

FW1/stby# dir

Directory of disk0:/

108 -rwx 25251840 06:31:26 Aug 15 2012 asa844-5-k8.bin
116 -rwx 18927088 05:08:54 Jul 10 2012 asdm-649.bin

260296704 bytes total (137342976 bytes free)

So you can save yourself a step of deleting identical files on the two units by doing the deletes from the active firewall.

The other half of the rule “upload to the standby first” is necessary because if you upload a file only to the active unit, and then reference that filename in a piece of CLI configuration, the command will be sent over to the standby unit referencing the nonexistent file, where it will silently fail to apply. The command simply won’t be in the standby ASA’s configuration. Even if you access the standby after that and upload the file, the command that referenced it will still be missing unless you reboot the standby or execute a “write standby” to force a full re-sync of the standby’s configuration.

The example below shows an all-too-common oversight that I’ve committed more times than I can count, resulting from this behavior.

FW1/act# show run web
webvpn
enable outside
enable inside
anyconnect image disk0:/anyconnect-win-3.1.00495-k9.pkg 1
anyconnect image disk0:/anyconnect-macosx-i386-3.1.00495-k9.pkg 2
anyconnect image disk0:/anyconnect-linux-3.1.00495-k9.pkg 3
anyconnect image disk0:/anyconnect-linux-64-3.1.00495-k9.pkg 4
anyconnect profiles WebVPN disk0:/webvpn.xml
anyconnect enable

FW1/act# dir
Directory of disk0:/

108 -rwx 25251840 06:25:26 Aug 15 2012 asa844-5-k8.bin
114 -rwx 29806775 06:26:08 Aug 15 2012 anyconnect-win-3.1.00495-k9.pkg
115 -rwx 11191648 06:26:24 Aug 15 2012 anyconnect-macosx-i386-3.1.00495-k9.pkg
116 -rwx 9872957 06:26:36 Aug 15 2012 anyconnect-linux-64-3.1.00495-k9.pkg
117 -rwx 18927088 07:28:38 Jul 13 2012 asdm-649.bin
118 -rwx 10978512 06:26:46 Aug 15 2012 anyconnect-linux-3.1.00495-k9.pkg
119 -rwx 2337 16:59:18 Sep 05 2012 webvpn.xml

Note that the WebVPN configuration stanza calls out the AnyConnect XML profile stored in disk0:/webvpn.xml. When we run the same commands on the standby ASA, we notice a difference. The file is missing from the flash storage “dir” output, but the reference to it is also missing from the configuration stanza:

FW1/stby# show run web
webvpn
enable outside
enable inside
anyconnect image disk0:/anyconnect-win-3.1.00495-k9.pkg 1
anyconnect image disk0:/anyconnect-macosx-i386-3.1.00495-k9.pkg 2
anyconnect image disk0:/anyconnect-linux-3.1.00495-k9.pkg 3
anyconnect image disk0:/anyconnect-linux-64-3.1.00495-k9.pkg 4
anyconnect enable

FW1/stby# dir
Directory of disk0:/

108 -rwx 25251840 06:31:26 Aug 15 2012 asa844-5-k8.bin
113 -rwx 29806775 06:32:00 Aug 15 2012 anyconnect-win-3.1.00495-k9.pkg
114 -rwx 11191648 06:32:16 Aug 15 2012 anyconnect-macosx-i386-3.1.00495-k9.pkg
115 -rwx 9872957 06:32:30 Aug 15 2012 anyconnect-linux-64-3.1.00495-k9.pkg
116 -rwx 18927088 05:08:54 Jul 10 2012 asdm-649.bin
117 -rwx 10978512 06:32:40 Aug 15 2012 anyconnect-linux-3.1.00495-k9.pkg

260296704 bytes total (137342976 bytes free)
FW1/stby#

Why did this happen? Simple. I goofed, and forgot to manually copy the webvpn.xml file down from the active unit where it was created and up to the standby ASA. Because of this, the AnyConnect profile would not have applied if the standby unit had taken over the active role. This is a very easy mistake to make, so always be sure to check the file system on both units when dealing with ASDM images, AnyConnect images, CSD/HostScan images, or XML files.

If you do find yourself in this pickle, it’s easy to resolve. Just upload the missing files to the standby unit, and execute (FROM THE ACTIVE UNIT!) a “write standby” which will resync the standby, like this:

FW1/act# write standby
Building configuration...
[OK]
FW1/act# Beginning configuration replication: Sending to mate.
End Configuration Replication to mate

However, it’s better to check things out and discover this before a failover occurs, rather than after your primary firewall dies and you find out that your seamless firewall failover wasn’t so seamless because AnyConnect wasn’t actually enabled on the secondary unit.

I really wish Cisco would provide an option to automatically sync the file systems of failover mates, or at least offer a simple process to copy files from the active to the standby unit. It would also be nice if a periodic full re-sync was performed to ensure the standby was always up to date. The incremental-update-only approach used now can let some configuration elements slip through the cracks if the filesystem operations are done in the wrong order.