Recently, I built myself a nice VMWare vSphere server. I’ll cover the server setup itself in more detail sometime, but once I got it kicking I decided I should find some novel (for me, at least) ways to use it. As I looked around my home network trying to find improvements to make with the vast powers of virtualization, my gaze settled on my little Cisco ASA5505 home firewall.
I’ve been working with the Cisco ASA pretty much since it came out to replace the PIX line (for which I hold no lingering love). Rather than turn this post into a thesis on the ASA’s strengths and weaknesses, I will simply state that while I think the ASA does some things very well, I think it does others very poorly. For many of my enterprise consulting clients, the ASA is a natural fit and an excellent firewall platform. For my home environment, I’ve been getting increasingly frustrated by some of its limitations and behaviors. I decided it was time for a change, and an adventure into network function virtualization!
Virtually a Firewall
I began searching for a virtualizable, software-based firewall platform, and arrived at pfSense. What is pfSense? Their website explains it much better than I can, but essentially it’s a comprehensive firewall (and other network services) platform built using the BSD “pf” packet filter firewall software, with a completely functional, fairly elegant (and not Java-based, Cisco should note!) web-based interface, all delivered in a live CD ISO based on the rock-solid FreeBSD operating system that can run on minimal RAM and CPU resources. And of course it’s open-source and free. In the course of my day-to-day work, I’ve built VPNs from customers’ edge firewalls to pfSense instances at cloud providers, seen pfSense used as a toss-it-up firewall to protect temporary environments, and things of that nature. I had relatively little exposure to the platform (and while I can get around in Linux, I know next-to-nothing about BSD), and that seemed as good a reason as any to give it a try! A pfSense virtual firewall would give me additional utilization from my vSphere server, and an excellent opportunity to branch out from Cisco products and learn a new platform.
It may sound odd for a CCIE to want to branch out from Cisco products, but as the networking landscape has been changing over the past couple of years, and my career progresses, I’ve been increasingly finding myself looking for opportunities to gain experience with other vendors and products as I focus more on solving networking problems at an architecture level and focusing less on the exact how-to syntax on a given platform. If I know what I’m trying to accomplish, discovering the configuration bits to do it becomes the relatively easy part. It’s good to get familiar with your options.
After doing some reading on pfSense, I put together a simple list of what my home network environment would gain and lose by a switch from ASA to pfSense:
Things gained:
- IPv6 tunnels (6to4, protocol 41)
- DNS forwarder, auto-populated with DHCP lease info, ability to do split DNS
- DynDNS updater
- Multiple WAN links with robust liveliness checks/thresholds
- Policy-based routing
- BGP (OK, I guess I don’t need this for my home broadband connection…)
Things lost:
- AnyConnect/Clientless SSLVPN
- CLI management
- More Cisco experience
- Deep layer 7 inspection rules
- Power efficiency – ASA5505 draws just a few watts, VMWare server needs 180+. This impacts UPS runtime, etc.
Of the “things lost” list, the CLI wasn’t much of a concern. I’ve come to accept (in some cases embrace!) a GUI world. I’ve been less than thrilled with the AnyConnect Clientless experience and have been using client-based VPN for home access so the loss of clientless was not a concern. I get plenty of Cisco experience every day so running a rarely-changing home firewall (that is still too “production” to use as a lab piece) was also no loss. Deep layer 7 inspection is something that (a) the ASA doesn’t do all that well compared to true next-gen firewalls, and (b) pfSense can also do, I think, if I cared to dig in and figure out how. Also, to keep some perspective on things, we’re talking about my home network here, not a data center full of corporate trade secrets or electronic health records. Finally, the power efficiency is something I’ll miss. My ASA would run for 2 hours on a small UPS, whereas the server will drain my largest UPS in about 15 minutes. Obviously, on-going cost to run the server is also higher, but I was going to have the server running anyway so I’m just trying to get additional use out of it.
The Setup
This post is not a tutorial on installing pfSense. There are many out there. In fact, there is a good doc on the pfSense website about running a pfSense virtual machine on vSphere 5. When configuring the VM, I did stick with E1000 NICs and while I’ve heard some people mention either performance issues or bugginess when using that vNIC, I’ve not had a problem yet. There are three schools of thought with regard to connecting your vSphere server to your unfiltered Internet feed. The first is “don’t do it.” If this was a high-security environment, I’d be much more leery. For my home network, I am willing to accept some risk to enable the virtual firewall idea. Also, as we move everything into cloud environments, this concept separating trusted from untrusted using virtualization has started to gain commonality (consider even VLANs and VRFs on traditional networking gear), so you have to decide for yourself how comfortable you are. Next, some people like dedicating a NIC in a PCI pass-through format, with the intention of even keeping the NIC off of vSphere’s radar which should limit the possibility of an attacker compromising the vSwitch or host itself. This, however, requires that you pass the entire NIC in to the VM, and my server configuration is such that I don’t have a dedicated, single-port NIC that I can pass in at this time (though I’m thinking about adding one). So I simply created a dedicated vSwitch for untrusted, external connectivity, and gave that vSwitch a single, untagged port-group and a single physical vmnic.
My pfSense VM’s “WAN” interface has a vNIC that connects to this external vSwitch, and two other vNICs representing the internal LAN and semi-trusted lab networks which pass into another vSwitch on VLAN-tagged port-groups that connect up to my internal LAN switching, like so:
Additionally, within vSphere I created a resource pool for firewall functions, which both reserved a reasonable amount of CPU for this important edge connectivity function, as well as limited the maximum CPU that could be used so that if the firewall ever came under some sort of attack that tried to eat all the CPU, it wouldn’t take the entire VM infrastructure down with it.
You can also see the OpenVPN Access Server VM in the same resource pool, which was the solution I’m using for VPN access. Yes, I know you can enable OpenVPN right on pfSense. I kind of liked the idea of separating those functions, and the free, 2-connection flavor of the OpenVPN AS vAppliance is working out nicely for me so far.
Finally, I did install the OpenVMTools package that is in the pfSense package manager. This lets vSphere interact with the VM better for graceful shutdown requests, learning the machine IPs, etc.
The Migration
By setting the pfSense VM up ahead of time on the LAN at a spare IP and giving it a default route out through the existing ASA firewall, I was able to get it all configured, migrate my meager rule base, pre-configure my handful of inbound NATs, configure the DNS host entries, DHCP pools (deactivated until the cutover), DynDNS updater, and all other necessary configuration. Due to the fact that pfSense can do some of these things like actually act as a DNS forwarder/resolver and perform DynDNS updates, I am able to simplify and concentrate services (and thus administration points) within the home network. For a geek like me, that means more than you might think… I have fair amount of stuff on my home network!
Once I coordinated an outage window with the user population, the cutover was just a matter of removing the LAN default gateway that pointed to the old firewall, changing the IP of the pfSense VM’s LAN interface, and activating the DHCP service to start handing out addresses. I had to reboot my cable-modem to learn the new VM’s MAC address and do a little physical recabling as well, but all went very smoothly.
The Conclusion… and the Future
After several weeks running on a pfSense virtual firewall (and a virtual appliance VPN concentrator), I’ve been quite pleased. Performance is fine — I’m able to max out my cable connection on speedtests and the vSphere host’s CPUs barely even notices.
The ease of deployment, flexibility of a VM, and relative ease of backups, snapshot/rollback, and all the other VM goodies are making me think that perhaps the age of virtual network appliances really is here. It seems The Cloud already knew this, I’ve just been playing catch-up.
Herding Packets 
Nice post. ocserv has experimental AnyConnect support. http://www.infradead.org/ocserv/
Very interesting, I will be keeping an eye on this one! Thanks for the tip, and thanks for reading.
Nice write-up. I’m glad you brought up the power efficiency point. This has been been keeping me from trying to run virtual appliances “in production” for home use. I don’t like the idea of running a server like that 24×7 just for the cost and I’m kind of an efficiency freak.
I suppose the fact that I have an Juniper SRX100 / Palo Alto PA-200 might be preventing me as well 🙂
Hows it going now, a few months down the line?
Really well! The pfSense VM has been rock solid, only being rebooted for a couple of software updates that came out (and were painless to apply). I’ve not had reason to dig into the deep technical details of pfSense very much so far, which is good from an operational standpoint but I’d like to learn some more about the inner workings at some point.
Next on my list is experimenting more with backup and recovery. I’d like to set up an automatic backup pull onto a utility machine and then see how it goes restoring that config backup onto another VM or maybe even a physical machine. I think understanding the capabilities and limitations there could be useful for home “DR” events or if I start to use pfSense more in a professional setting.
Thanks for stopping by!
To the things lost and AnyConnect in particular: I think it should be possible to run ASA behind (DMZ-like fashion?) and do PAT to it from pfSense’s outside interfaces for anyconnect’s ports. Couple it with OSPF and some redistribution…
Other options are opened that way too, like running deep inspections on guest wifi.
Yes, I’ve run an ASA behind another PAT before and AnyConnect actually works just fine through it (although you should remember to PAT UDP 443 as well for DTLS service for the best performance). It’s definitely an option, though I’ve found my OpenVPN VM to also do exactly what I need for my simple home setup (Heartbleed patching not withstanding…).
Thanks for reading!
Thanks for the post. This is really helpful. What version of vsphere did you install?
I’m running vSphere 5.5 at home. Not sure the update level, I actually haven’t updated since the initial install about a year ago (yeah, it’s on the to-do list…). Thanks for reading!
Thank you. Are you using the free version?
No, I purchased the vSphere Essentials kit for my home environment so I could use the vSphere web client without having to constantly reinstall the eval.
Thank you. I was leaning the same way. Then, I could also setup a web server, DHCP and DNS server for home. This allows me to keep WiFi and Private LAN separate. Would you still do the same thing today? I am going to start the project tonight.
In fact, I am still doing the same thing today! I’m still running the pfSense VM as my home firewall and I’ve been very pleased with it. I’ve moved all those services — DHCP, DNS, etc. — over to the pfSense and it has done an excellent job. I use the firewall to separate my home and lab networks from one another and treat the lab like a DMZ. Anything in my lab I want to access from outside is accessible in the Lab network, and the internal Home network is able to reach the Lab network without restriction, but the Lab network can initiate only a few select sessions inbound to the internal network.
I’ve found it to be an excellent solution for a home setup that is planning to do other virtualization. The vSphere Essentials kit does not include some advanced features like vMotion, HA, FT, etc., but I only have a single VM server with local storage for the moment, so none of that really impacts me.
Best of luck in your project! I’m sure you’ll find it as rewarding as I did.
would be interested on how you can enable failover on the pfsense firewall with another esxi host.
usually company without budget will choose pfsense as their firewall, and the good thing is you can configure ipv6 with your pfesense
I’m surprised more companies with some budget don’t select pfSense with a support subscription. It’s a solid package for most needs.
Hey,
I know this is an old post but was just looking for some ideas for my home esxi server (just waiting on ramm, slow postal) pfsense was my choice for firewall and ocserv (openconnect) for a vpn server. But thought I might mention to check out owncloud and an esx appliance for Nas/San functions.
Thanks for reading, and for the suggestion!