Weighing AWS VPN Options

Earlier this week, a client asked for some assistance in building a VPN from their corporate office to Amazon Web Services for a project they were doing. I’ve done this a few times before, a few different ways, so I proceeded to give my client some pros and cons of the two most common methods I’ve used. After putting that analysis together, I realized it could be helpful for others so here it is (with the addition of a few snazzy diagrams!).
This post is not meant to be a treatise on AWS connectivity, just a quick analysis with some (maybe) little-considered effects of a given design choice. Amazon documents several other recipes which are, of course, valid in various circumstances. Note that I don’t have any examples of configuration. The AWS documentation pages have very thorough configuration examples for each design.

 Option 1

Build the VPNs off the Internet routers themselves. Route AWS traffic in to the corporate network through the firewall. In an ideal world, you’d probably dedicate some routers for this purpose, but I’ve never had anyone do that. We’re talking about a LAN-to-LAN VPN, here; one doesn’t commonly deploy totally dedicated infrastructure for each new VPN. When I built my first VPN to AWS, this was the only supported method.

 The design calls for a pair of VPNs from an IOS router to the AWS Virtual Private Gateway (VPG) which is the connection point to the Virtual Private Cloud (VPC). IPSec VTI tunnel interfaces are constructed to AWS, and BGP routing is done over the tunnels. BGP tuning can be used to prefer inbound and outbound paths via multiple routers.
AWS to IOS VPN with BGP

AWS to IOS VPN with BGP

Pros:

  • Terminate a pair of VPNs on each Internet router for equipment redundancy
  • BGP routing across the VPN determines which VPN/router to use. Could even be used for load sharing, depending on VPC design.
  • Requires no additional routing magic internally. Firewall just follows default.
  • Fast(ish) Failover, in as long as it takes BGP to converge with tuned-down timers (so on the order of 30 seconds).
  • Since traffic comes out of VPN on the router, ACLs applied on the firewall can control inbound traffic from AWS.
  • PAT may still be possible to further protect corporate network from AWS (depends what, if any, traffic needs to originate from AWS)

Cons:

  • Requires Security feature set on the routers.
  • Crypto performance on a router relative to the ASA is somewhat lower. Your mileage may vary based on router model and Internet speed.
  • Some private-IP’d traffic will leak out of the firewall and get to the routers (which then put it in the VPNs).
  • May require Internet routers having visbility of some private IP space.
  • If Internet routers do BGP for Internet routing, care must be taken not to advertise Internet routes to AWS or AWS routes to Internet.
  • Have to remember that we’re doing VPNs on the Internet routers and not on the firewalls. It’s not very obvious when troubleshooting.
  • Hairpin traffic for remote access users to reach AWS (if needed) is a bit more complex.
  • Double the cost to get equipment redundancy on the customer end. The AWS VPN connection is charged at $0.05/hr, so two VPN configurations means ~$73/mo just for the VPN.
Concerns about leaking private IPs outside the firewall (even if it is only to the routers in the same rack) could probably be mitigated by putting the tunnel interfaces into a VRF, and peeling off a separate interface from the Internet router (also in that VRF) that heads into an extranet DMZ on the firewall.

Option 2

Terminate the VPNs on the Cisco ASA firewall cluster. This method relies on the ASA having two peers defined for the VPN. Failover in this case is dependent on ISAKMP keepalive timeouts and the ASA switching to the backup peer. This is not always a rapid process, and one of the AWS VPN connections will always show as “down” since the ASA will only maintain the tunnel to the single active peer.
AWS to ASA VPN

AWS to ASA VPN

Pros:

  • Simple and obvious configuration. A site-to-site like any other.
  • With ASA failover pair, equipment redundancy on the customer side is handled with no additional AWS VPN configurations.
  • No leakage of internal IPs or non-encrypted traffic of any kind outside the firewall.
  • Crypto throughput of the ASA is relatively high and would not be challenged given the Internet link speeds in this case.
  • PAT may still be possible to further protect corporate network from AWS (depends what, if any, traffic needs to originate from AWS)

Cons:

  • Dependent on ISAKMP keepalives and backup peer for failover — failover on the order of several minutes.
  • If internal network is complex or has multiple exits, a dynamic decision has to be made via IGP on which path to use (like a redistributed static from the cores, etc), so potentially more complex internal routing.
  • Applying firewall policy to inbound traffic from AWS requires use of VPN Filter ACLs which are somewhat complex and finicky, particularly for port-level filtering.
  • Dynamic routing across the tunnel is all but impossible.

Conclusion

When I started putting this list together for my client, I was leaning toward recommending the BGP-on-IOS model. I like having all the tunnels up and having the (more) rapid failover of the BGP timeout. I like not having to deal with ASA’s finicky VPN Filter behavior. I like the idea of having the ability to steer traffic over multiple paths. And frankly, my love affair with the ASA as a platform is fading.

As I evaluated the pro’s and con’s, though, I started to realize that the BGP/IOS model at least as I’ve done it in the past (using the Internet routers themselves) adds a lot of complexity and non-obvious behavior to the network. It would require particular attention during future changes to avoid breaking it or mixing the AWS and Internet BGP networks. And while “virtualizing” the routers with VRFs would alleviate some of those concerns, that idea adds its own complexity.

In the end, I determined that the tradeoffs by terminating the VPNs on the redundant ASA cluster are probably worth it for an obvious design that is pretty much treated like any other L2L VPN and still meets the client’s needs with its slower failover. In the end that’s the path I recommended.

Just one of many decisions to make on any given day, and certainly a minor one in the grand scheme of things. But the exercise seemed to me to be a good reminder that we should take a couple minutes to really consider and evaluate even small decisions because sometimes we’ll surprise ourselves with the outcome.

Advertisements
Tagged , , ,

20 thoughts on “Weighing AWS VPN Options

  1. Thanks for that article. We were recently approached to do something similar, but included using our F5s to load balance to pool members in AWS. We recommended against it, but this goes to show what is achievable. Good stuff!

    • bobmccouch says:

      Thanks for reading! Of my clients using AWS, several use it for Dev/experimental stuff and one is running a full-time DR environment and even some production services in it.

      I’ve seen a lot of argument that it’s not a great value if you don’t really need the crazy elasticity, but in some cases it’s helped my clients provide experimental infrastructure to devs and avoid the “Shadow IT” problem.

  2. Leo says:

    Hi, nice article. Are these drawings handmade or is it a program? They look great. Good job!

  3. Matt Wolpin says:

    Did you think about using the Brocade vRouter? It’s available in the Amazon AWS Marketplace and since the vRouter has firewall and VPN built-in, it could eliminate a lot of the security cons you outlined.

    https://aws.amazon.com/marketplace/pp/B009I5TLOE

    *Note: I work for Brocade.

    • bobmccouch says:

      Hi Matt,
      Thanks for reading! Could you expand on how running the AWS end of the VPN into an AWS instance eliminates the security concerns that I mentioned which primarily effect the non-AWS end of the link? I agree that if you didn’t like terminating the far end of the VPN on an Amazon-controlled endpoint and would prefer to control both ends of the tunnel you could do this, but now you’re paying for an EC2 instance to terminate your VPN whether it’s a Brocade Vyatta instance (looks like about $262/mo for a small), a pfSense instance (about $224/mo), or a Cisco CSR1000V ($87/mo plus Cisco’s license fee). Those are all drastically more than the $0.05/hr or about $37/mo that Amazon charges for the VPN.

      One could choose to terminate the enterprise-side of the VPN not on an edge device, but on a virtual firewall/VPN appliance deeper in the enterprise network but that presents its own challenges including having the VPN endpoint behind a NAT, or choosing to expose a VM host to the untrusted public side of your network.

      Thanks!
      Bob

  4. […] Weighing AWS VPN Options | Herding Packets – Bob McCouch writes about his options: […]

  5. Joe B says:

    I recently had a similar experience although in my case the goal was to connect to multiple VPCs. This comes with a requirement of a unique client-side IP per VPN connection and presented difficulties considering our public addresses are a /28 block and multiple interfaces can’t bind addresses in the same subnet. In the end we took a router/vrf/supernet approach that worked, albeit at the cost of addresses. Minutes after wrapping up a PoC I got a notification from the AWS management page linking to the following forum post https://forums.aws.amazon.com/ann.jspa?annID=2397# that details the introduction of VPC peering. Granted, it’s limited to in-region VPCs but this would’ve served my needs and removed the requirement for such an exotic configuration. Seems like every time I tool around an AWS limitation it becomes a solution shortly thereafter.

    • RobH says:

      Joe B – Id like to hear more of the multiple VPN from same region configuration. I am going through this right now. Have a router with a single public ip address with one or two more addition public ips available in that range. Was thinking of creating a secondary interface and using that as the second same region endpoint. Although its in the same subnet as the first ip.

  6. […] McCouch has a nice write-up on options for VPNs to AWS. If you’re needing to build out such a solution, you might want to read his post for some […]

  7. […] McCouch has a nice write-up on options for VPNs to AWS. If you’re needing to build out such a solution, you might want to read his post for some […]

  8. Kerry says:

    ASA 9.2 now supports BGP on the ASA firewall. So using this as a solution might give you the best of both worlds.

    • bobmccouch says:

      That’s a good point. One thing I’m not sure of is: Can the ASA do BGP over its own VPN tunnel? That seems like just the kind of case that the ASA wouldn’t support…. :-/ I’ll have to try it sometime.

      • znorg48 says:

        Hi Bob, thanks for a great article. I got an ASA working well with a tunnel to AWS in “static routing” mode and I was just looking into exactly what you mention above – BGP over a tunnel terminating on an ASA running 9.x. My idea was to use ASAs at 2 customer sites as resilient VPN gateways into AWS, so other customer sites don’t lose their link to AWS if the power at the one site with a VPN gateway goes out.

        Cisco have an example of the ASA doing BGP over its own tunnel http://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/118835-config-asa-00.html

        However, a further problem seems to be that AWS in “dynamic routing (BGP)” mode expect the tunnel to have inside addresses in a /30, and they want to peer with the inside tunnel address on the customer side.

        I’m not an expert on this – but doesn’t that suggest tunnel interfaces which the ASA doesn’t have?

        I found this on how to do route-based VPN on an ASA in the absence of tunnel interfaces, but it seems a little convoluted. http://packetsneverlie.blogspot.co.uk/2012/06/route-based-ipsec-vpn-on-asa.html

        I wonder if I am missing an easier way of achieving the original objective re: resilience from multi-site gateways. If all else fails I may try terminating the VPNs on routers instead as you suggest. It does seem a bit of a shame given the extra hardware needed (the main site routers are managed by a service provider so I’d have to provide additional routers) and the ASA’s better crypto performance. Do you have any thoughts? Thanks again for your great stuff.

      • bobmccouch says:

        Unfortunately I know of no clean way to do a route-based or interface-based VPN on ASA. I see what you mean about the problem with BGP over the VPN to AWS, since AWS is assigning a private (or, APIPA really) /30 to the tunnel. Even in the example you cite for a route-based VPN it looks like the VPN interface of the ASA still needs a public IP. If there is a way to make it work, it’s going to live somewhere in the “convoluted to unsupported” range, I’m afraid. Best of luck!

  9. Alex Rothberg says:

    Great Post!

    I am wondering if I could use either of these techniques to address my question here: http://stackoverflow.com/questions/28824041/limiting-access-to-aws-network-when-using-hardware-vpn-connection ?

    • bobmccouch says:

      I’m not sure just what facilities within AWS might be able to control incoming VPN traffic. One option might be deploying your own VPN endpoint (be is Cisco ASAv, Cisco CSR1000V, or something like pfSense) within your VPC so that you have a control point with flexibility beyond what AWS provides. I’ve not looked too deeply into the AWS VPN configuration on the AWS side, but I wouldn’t be too surprised if their model is that anything with a VPN into the VPC is “implicitly trusted” and they assume that anything semi- or non-trusted would be coming across public infrastructure and not through a VPN. That’s just a guess, but certainly seems plausible. Good luck in your search!

  10. android 4.0 says:

    Great goods from you, man. I’ve understand your stuff previous to and you
    are just too excellent. I actually like what you’ve acquired
    here, really like what you’re saying and the way in which you say it.
    You make it enjoyable and you still care
    for to keep it smart. I can’t wait to read much more from you.
    This is really a great website.

  11. Hunt says:

    What should I do if I have both a Direct Connection and a VPN connection to my client’s VPC where I want:

    1) All primary traffic to route via the Direct Connection
    2) In an event that my Direct Connection is lost, all traffic will automatically re-route via VPN

    In other words, how can I make the VPN Connection a less priority than the Direct Connect?

    • bobmccouch says:

      Hi Hunt, I’ve never worked directly with AWS Direct Connect, but according to this page (https://aws.amazon.com/directconnect/faqs/), the answer is that failover and failback will happen automatically:

      “Q. Can I use AWS Direct Connect and a VPN Connection to the same VPC simultaneously?
      Yes. However, only in fail-over scenario’s. The Direct Connect path will always be preferred, when established, regardless of AS path prepending.”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

@greatwhitetec

Virtualization, Storage, and other techy stuff

The Stupid Engineer

I ask those questions you're too clever to.

Sunay Tripathi's Blog

Pluribus Networks Founder's Blog on OS, Networking, Virtualization, Cloud Computing, Solaris Architecture, etc

Ed Koehler's Blog

Just another WordPress.com weblog

JGS.io

Data networking, stray thoughts, nerdy fun...

Network Heresy

Tales of the network reformation

The Borg Queen

Jottings on the intersection of tech and humanness

Networking From The Trenches

Ramblings about my thoughts, experiences, and ideas.

In Search of Tech

Looking for the next big thing.

Packet Maniac

A day in the life of a maniac packet

Fryguy's Blog

A Network Blog by a Network Engineer

Networking 40,000

Attaining my CCIE with the help of Warhammer 40k

stubby router

just another networking blog

Ronnie Angello

Network Architect . CCIE 17846 . CCDE 2012::1

The Peering Introvert

The sundry interests of Ethan Banks including books, cars, hiking in New Hampshire, religion, music, home theater, technology, geek culture, and social media. And maybe cats.

%d bloggers like this: