Alternately you can inventory your instances in AWS Systems Manager and start an SSH session directly from the AWS CLI itself [1] -- no Bastion servers or SSH key management necessary, pure IAM access only.
Downside is that it doubles access options when you have a mixed environment, i.e. multiple clouds and/or non-cloud servers. Other than that it's a great solution if IAM is your default user source.
Always glad to see one of our products mentioned :)
Huge shameless plug for the coolness that is Session Manager, I can only think of a few scenarios where you wouldn't just prefer it over the alternatives. If you haven't played with it (or other SSM stuff), you totally should! It's cool and useful and easy and more or less free.
You can! The agent can be installed on almost any linux flavor, and is open source on GitHub, so it can be compiled and installed on any EC2 machines (or on-prem machines) you want. (This might not be easy, I haven't done it myself)
If you can make a good case for it, open a ticket or make a stink on the forums and there's a chance an engineer can be tasked to add it to their list of Officially Supported Platforms. This is how eg raspbian got added, though I don't know if the program is still running for adding platforms as it's been a few years.
It was a total no-brainer once we discovered it. Really nice work.
If you're interested in more feedback, I'd be happy to share. As with many things AWS, there were some undocumented rougher edges that took some trial and error to figure. Although the end result was well worth it. Totally understand if you're not on HN to do that sort of thing, though.
Hey, I'm all ears. Just an engineer, but customer feedback is taken super seriously (albeit with a tragically long turnaround time - it's a surprisingly large organization with a lot of irons on a lot of fires). I'll totally pass on anything you have to say though.
e: or put an email in your profile and I'll even reach out!
Is there a way to use SSM without using the access keys? I feel using access keys is incredibly not secure because rotating keys is a hassle and people might not do them all the time.
Also can u restrict access to ssh through ssm to certain ips?
Maybe I might have missed these, so any help would be appreciated.
> Is there a way to use SSM without using the access keys? I feel using access keys is incredibly not secure because rotating keys is a hassle and people might not do them all the time.
Technically there is, you can use federated login. Might not be very convenient, depending on your identity provider.
A solution I use, while not technically "not using access kys" is storing them in the system credential store with aws-vault [0]. Works on Windows, Linux and Mac. And you can combine this with multi factor auth.
> Also can u restrict access to ssh through ssm to certain ips?
Yes, with an IAM policy. The policy below requires connecting with an MFA and from a specific IP range. It only allows connecting to a specific instance.
Or you can run ssm agent on a fargate task for double the fun (just need to register as hybrid and remember to clean up (scheduled lambda works a treat for this)
While it's possible to do port forwarding with SSM [0] I would encourage people to test it well.
I've tried using it for development, and it was a terrible experience. The connections would hang all the time for no reason. I ended up starting an SSH session (via SSM) and done the port forwarding via that...
You can do this with a VPN as well, no bastion required.
Seems to me that the bastion approach is a bit weird at this point; no node should be directly attached to the public internet (unless you enjoy babysitting one). You usually have an LB in front of it, then perhaps a CDN+WAF (like CloudFlare) in front of that. The same can be done for SSH and even RDP.
The concept that your bastion is 'more secure' than your other systems needs to go away, make all of the systems that secure to begin with if you want to allow human access and shells.
Agreed on this. AWS Client VPN is a decent (though admittedly a little more pricey) option: https://aws.amazon.com/vpn/pricing/. But then again you're paying for the managed service.
Yeah, the managed part does add to the cost (understandably) but if the half-managed approach of appliances works for you I think OpenVPN-AS is an option as well.
Some people are using wireguard containers on Fargate to do the same thing.
Do you happen to have a solid link / reference / GitHub repo for Wireguard on Fargate? That's exactly what I'm looking for!
OpenVPN is a hot mess, but is currently the most supported mechanism across platforms.
Background: I want to give my (100% global remote) development team network access to our AWS dev environment Aurora Postgresql, EFS / NFS, redis, microservices, etc. We already have a local env with docker-compose but need to debug and test in the shared cloud dev environment.
I looked into this some time ago and found that wireguard cannot run on fargate due to requiring kernel modifications and the userspace implementation[0] has similar blockers.
Sadly, no. A lot of the available 'nice' containers require host access that isn't available on Fargate. The ones people tend to use on Fargate are pure user-space versions that don't work as well as others. We have evaluated a bunch of finds from blogs and public repositories but they all lack one way or another.
Most of our setups use OpenVPN via OpnSense on AWS, second most popular option is OpenVPN-AS with a paid license, third is AWS OpenVPN, because of the price tag.
A few test setups rely on a small EC2 instance per group (t3.small for example) with a single container and it's a bit quirky to automate, especially on large user groups.
This is our main issue with WG in production so far: while the technology seems totally fine, it's not at a point where you can smoothly roll out a 'service' and get going, there are too many hoops to jump through and too many duct-tape constructions to make it integrate. (somewhat ironic, considering OpenVPN)
I manage a lot of ec2 instances on dozens of AWS accounts... I can't see a reason to use this particular method... Even if you want to spin up bastion on demand like this, why Fargate, why terraform?
In the simplest form, I'd go with AWS CloudFormation of an EC2 instance. Remove the additional step of an ECS cluster if you don't already have one. An EC2 instance can access cluster VMs, or other VPC objects just as easily. Can it's wrap this in AWS SAM or CloudFormation + a controller like GitLab or Jenkins...
The benefit here is that when you're done, you have an AWS CloudFormation stack to kill... No need for anyone to have teraform installed to find and remove all associated resources.
Of course, this is all if you can't use Systems manager or the CLI to achieve your goals
Bastions are a bad pattern. They are a way to paper over bad systems of authentication and access control. There are better ways to achieve the same thing, especially with cloud providers.
I don’t currently understand the additional security from bastion hosts. Yes you can have them short lived. But if someone cracks the bastion it’s not very good for security.
31 comments
[ 3.3 ms ] story [ 74.1 ms ] thread[1] https://docs.aws.amazon.com/systems-manager/latest/userguide...
Huge shameless plug for the coolness that is Session Manager, I can only think of a few scenarios where you wouldn't just prefer it over the alternatives. If you haven't played with it (or other SSM stuff), you totally should! It's cool and useful and easy and more or less free.
If you can make a good case for it, open a ticket or make a stink on the forums and there's a chance an engineer can be tasked to add it to their list of Officially Supported Platforms. This is how eg raspbian got added, though I don't know if the program is still running for adding platforms as it's been a few years.
If you're interested in more feedback, I'd be happy to share. As with many things AWS, there were some undocumented rougher edges that took some trial and error to figure. Although the end result was well worth it. Totally understand if you're not on HN to do that sort of thing, though.
e: or put an email in your profile and I'll even reach out!
Also can u restrict access to ssh through ssm to certain ips?
Maybe I might have missed these, so any help would be appreciated.
Technically there is, you can use federated login. Might not be very convenient, depending on your identity provider.
A solution I use, while not technically "not using access kys" is storing them in the system credential store with aws-vault [0]. Works on Windows, Linux and Mac. And you can combine this with multi factor auth.
> Also can u restrict access to ssh through ssm to certain ips?
Yes, with an IAM policy. The policy below requires connecting with an MFA and from a specific IP range. It only allows connecting to a specific instance.
[0] AWS Vault: https://github.com/99designs/aws-vault/https://github.com/elpy1/ssh-over-ssm
I've tried using it for development, and it was a terrible experience. The connections would hang all the time for no reason. I ended up starting an SSH session (via SSM) and done the port forwarding via that...
[0] https://docs.aws.amazon.com/systems-manager/latest/userguide...
Seems to me that the bastion approach is a bit weird at this point; no node should be directly attached to the public internet (unless you enjoy babysitting one). You usually have an LB in front of it, then perhaps a CDN+WAF (like CloudFlare) in front of that. The same can be done for SSH and even RDP.
The concept that your bastion is 'more secure' than your other systems needs to go away, make all of the systems that secure to begin with if you want to allow human access and shells.
Some people are using wireguard containers on Fargate to do the same thing.
OpenVPN is a hot mess, but is currently the most supported mechanism across platforms.
Background: I want to give my (100% global remote) development team network access to our AWS dev environment Aurora Postgresql, EFS / NFS, redis, microservices, etc. We already have a local env with docker-compose but need to debug and test in the shared cloud dev environment.
[0]: https://github.com/aws/containers-roadmap/issues/239
Most of our setups use OpenVPN via OpnSense on AWS, second most popular option is OpenVPN-AS with a paid license, third is AWS OpenVPN, because of the price tag.
A few test setups rely on a small EC2 instance per group (t3.small for example) with a single container and it's a bit quirky to automate, especially on large user groups.
This is our main issue with WG in production so far: while the technology seems totally fine, it's not at a point where you can smoothly roll out a 'service' and get going, there are too many hoops to jump through and too many duct-tape constructions to make it integrate. (somewhat ironic, considering OpenVPN)
In the simplest form, I'd go with AWS CloudFormation of an EC2 instance. Remove the additional step of an ECS cluster if you don't already have one. An EC2 instance can access cluster VMs, or other VPC objects just as easily. Can it's wrap this in AWS SAM or CloudFormation + a controller like GitLab or Jenkins...
The benefit here is that when you're done, you have an AWS CloudFormation stack to kill... No need for anyone to have teraform installed to find and remove all associated resources.
Of course, this is all if you can't use Systems manager or the CLI to achieve your goals
Because it's easy to throw containers at the problem. Coupled with savings plans this can be pretty cheap.
> why terraform
Because it's easier for most than CloudFormation, and has all the benefits of decomposable stacks just like CF.
I think it’s better with VPN plus SSM.