Option 1
Build the VPNs off the Internet routers themselves. Route AWS traffic in to the corporate network through the firewall. In an ideal world, you’d probably dedicate some routers for this purpose, but I’ve never had anyone do that. We’re talking about a LAN-to-LAN VPN, here; one doesn’t commonly deploy totally dedicated infrastructure for each new VPN. When I built my first VPN to AWS, this was the only supported method.
AWS to IOS VPN with BGP
- Terminate a pair of VPNs on each Internet router for equipment redundancy
- BGP routing across the VPN determines which VPN/router to use. Could even be used for load sharing, depending on VPC design.
- Requires no additional routing magic internally. Firewall just follows default.
- Fast(ish) Failover, in as long as it takes BGP to converge with tuned-down timers (so on the order of 30 seconds).
- Since traffic comes out of VPN on the router, ACLs applied on the firewall can control inbound traffic from AWS.
- PAT may still be possible to further protect corporate network from AWS (depends what, if any, traffic needs to originate from AWS)
Cons:
- Requires Security feature set on the routers.
- Crypto performance on a router relative to the ASA is somewhat lower. Your mileage may vary based on router model and Internet speed.
- Some private-IP’d traffic will leak out of the firewall and get to the routers (which then put it in the VPNs).
- May require Internet routers having visbility of some private IP space.
- If Internet routers do BGP for Internet routing, care must be taken not to advertise Internet routes to AWS or AWS routes to Internet.
- Have to remember that we’re doing VPNs on the Internet routers and not on the firewalls. It’s not very obvious when troubleshooting.
- Hairpin traffic for remote access users to reach AWS (if needed) is a bit more complex.
- Double the cost to get equipment redundancy on the customer end. The AWS VPN connection is charged at $0.05/hr, so two VPN configurations means ~$73/mo just for the VPN.
Option 2
AWS to ASA VPN
Pros:
- Simple and obvious configuration. A site-to-site like any other.
- With ASA failover pair, equipment redundancy on the customer side is handled with no additional AWS VPN configurations.
- No leakage of internal IPs or non-encrypted traffic of any kind outside the firewall.
- Crypto throughput of the ASA is relatively high and would not be challenged given the Internet link speeds in this case.
- PAT may still be possible to further protect corporate network from AWS (depends what, if any, traffic needs to originate from AWS)
Cons:
- Dependent on ISAKMP keepalives and backup peer for failover — failover on the order of several minutes.
- If internal network is complex or has multiple exits, a dynamic decision has to be made via IGP on which path to use (like a redistributed static from the cores, etc), so potentially more complex internal routing.
- Applying firewall policy to inbound traffic from AWS requires use of VPN Filter ACLs which are somewhat complex and finicky, particularly for port-level filtering.
- Dynamic routing across the tunnel is all but impossible.
Conclusion
When I started putting this list together for my client, I was leaning toward recommending the BGP-on-IOS model. I like having all the tunnels up and having the (more) rapid failover of the BGP timeout. I like not having to deal with ASA’s finicky VPN Filter behavior. I like the idea of having the ability to steer traffic over multiple paths. And frankly, my love affair with the ASA as a platform is fading.
As I evaluated the pro’s and con’s, though, I started to realize that the BGP/IOS model at least as I’ve done it in the past (using the Internet routers themselves) adds a lot of complexity and non-obvious behavior to the network. It would require particular attention during future changes to avoid breaking it or mixing the AWS and Internet BGP networks. And while “virtualizing” the routers with VRFs would alleviate some of those concerns, that idea adds its own complexity.
In the end, I determined that the tradeoffs by terminating the VPNs on the redundant ASA cluster are probably worth it for an obvious design that is pretty much treated like any other L2L VPN and still meets the client’s needs with its slower failover. In the end that’s the path I recommended.
Just one of many decisions to make on any given day, and certainly a minor one in the grand scheme of things. But the exercise seemed to me to be a good reminder that we should take a couple minutes to really consider and evaluate even small decisions because sometimes we’ll surprise ourselves with the outcome.
Herding Packets 
Thanks for that article. We were recently approached to do something similar, but included using our F5s to load balance to pool members in AWS. We recommended against it, but this goes to show what is achievable. Good stuff!
Thanks for reading! Of my clients using AWS, several use it for Dev/experimental stuff and one is running a full-time DR environment and even some production services in it.
I’ve seen a lot of argument that it’s not a great value if you don’t really need the crazy elasticity, but in some cases it’s helped my clients provide experimental infrastructure to devs and avoid the “Shadow IT” problem.
Hi, nice article. Are these drawings handmade or is it a program? They look great. Good job!
Thanks! I do these diagrams in 53 Paper on iPad.
Did you think about using the Brocade vRouter? It’s available in the Amazon AWS Marketplace and since the vRouter has firewall and VPN built-in, it could eliminate a lot of the security cons you outlined.
https://aws.amazon.com/marketplace/pp/B009I5TLOE
*Note: I work for Brocade.
Hi Matt,
Thanks for reading! Could you expand on how running the AWS end of the VPN into an AWS instance eliminates the security concerns that I mentioned which primarily effect the non-AWS end of the link? I agree that if you didn’t like terminating the far end of the VPN on an Amazon-controlled endpoint and would prefer to control both ends of the tunnel you could do this, but now you’re paying for an EC2 instance to terminate your VPN whether it’s a Brocade Vyatta instance (looks like about $262/mo for a small), a pfSense instance (about $224/mo), or a Cisco CSR1000V ($87/mo plus Cisco’s license fee). Those are all drastically more than the $0.05/hr or about $37/mo that Amazon charges for the VPN.
One could choose to terminate the enterprise-side of the VPN not on an edge device, but on a virtual firewall/VPN appliance deeper in the enterprise network but that presents its own challenges including having the VPN endpoint behind a NAT, or choosing to expose a VM host to the untrusted public side of your network.
Thanks!
Bob
[…] Weighing AWS VPN Options | Herding Packets – Bob McCouch writes about his options: […]
I recently had a similar experience although in my case the goal was to connect to multiple VPCs. This comes with a requirement of a unique client-side IP per VPN connection and presented difficulties considering our public addresses are a /28 block and multiple interfaces can’t bind addresses in the same subnet. In the end we took a router/vrf/supernet approach that worked, albeit at the cost of addresses. Minutes after wrapping up a PoC I got a notification from the AWS management page linking to the following forum post https://forums.aws.amazon.com/ann.jspa?annID=2397# that details the introduction of VPC peering. Granted, it’s limited to in-region VPCs but this would’ve served my needs and removed the requirement for such an exotic configuration. Seems like every time I tool around an AWS limitation it becomes a solution shortly thereafter.
Joe B – Id like to hear more of the multiple VPN from same region configuration. I am going through this right now. Have a router with a single public ip address with one or two more addition public ips available in that range. Was thinking of creating a secondary interface and using that as the second same region endpoint. Although its in the same subnet as the first ip.
[…] McCouch has a nice write-up on options for VPNs to AWS. If you’re needing to build out such a solution, you might want to read his post for some […]
[…] McCouch has a nice write-up on options for VPNs to AWS. If you’re needing to build out such a solution, you might want to read his post for some […]
ASA 9.2 now supports BGP on the ASA firewall. So using this as a solution might give you the best of both worlds.
That’s a good point. One thing I’m not sure of is: Can the ASA do BGP over its own VPN tunnel? That seems like just the kind of case that the ASA wouldn’t support….
I’ll have to try it sometime.
Hi Bob, thanks for a great article. I got an ASA working well with a tunnel to AWS in “static routing” mode and I was just looking into exactly what you mention above – BGP over a tunnel terminating on an ASA running 9.x. My idea was to use ASAs at 2 customer sites as resilient VPN gateways into AWS, so other customer sites don’t lose their link to AWS if the power at the one site with a VPN gateway goes out.
Cisco have an example of the ASA doing BGP over its own tunnel http://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/118835-config-asa-00.html
However, a further problem seems to be that AWS in “dynamic routing (BGP)” mode expect the tunnel to have inside addresses in a /30, and they want to peer with the inside tunnel address on the customer side.
I’m not an expert on this – but doesn’t that suggest tunnel interfaces which the ASA doesn’t have?
I found this on how to do route-based VPN on an ASA in the absence of tunnel interfaces, but it seems a little convoluted. http://packetsneverlie.blogspot.co.uk/2012/06/route-based-ipsec-vpn-on-asa.html
I wonder if I am missing an easier way of achieving the original objective re: resilience from multi-site gateways. If all else fails I may try terminating the VPNs on routers instead as you suggest. It does seem a bit of a shame given the extra hardware needed (the main site routers are managed by a service provider so I’d have to provide additional routers) and the ASA’s better crypto performance. Do you have any thoughts? Thanks again for your great stuff.
Unfortunately I know of no clean way to do a route-based or interface-based VPN on ASA. I see what you mean about the problem with BGP over the VPN to AWS, since AWS is assigning a private (or, APIPA really) /30 to the tunnel. Even in the example you cite for a route-based VPN it looks like the VPN interface of the ASA still needs a public IP. If there is a way to make it work, it’s going to live somewhere in the “convoluted to unsupported” range, I’m afraid. Best of luck!
Great Post!
I am wondering if I could use either of these techniques to address my question here: http://stackoverflow.com/questions/28824041/limiting-access-to-aws-network-when-using-hardware-vpn-connection ?
I’m not sure just what facilities within AWS might be able to control incoming VPN traffic. One option might be deploying your own VPN endpoint (be is Cisco ASAv, Cisco CSR1000V, or something like pfSense) within your VPC so that you have a control point with flexibility beyond what AWS provides. I’ve not looked too deeply into the AWS VPN configuration on the AWS side, but I wouldn’t be too surprised if their model is that anything with a VPN into the VPC is “implicitly trusted” and they assume that anything semi- or non-trusted would be coming across public infrastructure and not through a VPN. That’s just a guess, but certainly seems plausible. Good luck in your search!
Great goods from you, man. I’ve understand your stuff previous to and you
are just too excellent. I actually like what you’ve acquired
here, really like what you’re saying and the way in which you say it.
You make it enjoyable and you still care
for to keep it smart. I can’t wait to read much more from you.
This is really a great website.
What should I do if I have both a Direct Connection and a VPN connection to my client’s VPC where I want:
1) All primary traffic to route via the Direct Connection
2) In an event that my Direct Connection is lost, all traffic will automatically re-route via VPN
In other words, how can I make the VPN Connection a less priority than the Direct Connect?
Hi Hunt, I’ve never worked directly with AWS Direct Connect, but according to this page (https://aws.amazon.com/directconnect/faqs/), the answer is that failover and failback will happen automatically:
“Q. Can I use AWS Direct Connect and a VPN Connection to the same VPC simultaneously?
Yes. However, only in fail-over scenario’s. The Direct Connect path will always be preferred, when established, regardless of AS path prepending.”