I'd love to learn how you're using Session Manager or what other features/integrations you'd like to see us explore. Also if the terraform module packaging is useful. There are additional Session Manager features like port forwarding that I plan to write about soon.
Our organization recently looked into AWS Session Manager for tunneling but couldn't find documentation on how to make it work for our usecase. We were trying to tunnel into our VPC in order to be able to connect to an Amazon DocumentDB cluster. We don't have any EC2 instances which seems to be the only thing Session Manager has support for. Despite the callouts that Session Manager replaces bastion servers, that didn't seem to be the case for us. Did we miss something in our research?
Last I checked the "tunneling" only works to redirect traffic to a different port on the same SSM managed instance. The tunnel cannot be established with another box in the same VPC. So I don't think you can call it tunneling until they add that feature. Here's the GitHub issue where they discuss the limitation and a workaround: https://github.com/aws/amazon-ssm-agent/issues/208
Can you write one about port forwarding? Specifically, I would like to understand how various web interfaces on EMR cluster can be accessed through Sessions Manager. (Ganglia, Spark history server, etc.)
You're right in a sense, but there's no aws-managed bastion. Session manager communicates with your instance via an outbound-created websocket connection. Inputs and outputs are piped through it.
Would be interesting to lock down the session manager agent (if possible) so that the only way to privileged access is through sudo-like priv esc that uses 2fa.
As far as I know, SSH over SSM doesn't do anything regarding user management. It just establishes an SSH connection. Management of users on the host, authorized SSH keys, etc. is totally out of scope for SSM.
So if you already have access control setup on your host, then SSM doesn't do anything to undermine it. If you don't have it, you'll still need to add it.
It's great for managing active SSH sessions, but not so much for the other purpose for bastions: fine-grained network access control+routing. It would be cool if they made a more specific version of this just for network traffic without the SSH component.
FWIW, the project is open source, so you could build a modified agent for your purposes and inject it via cloud-init or your favorite config management tool: https://github.com/aws/amazon-ssm-agent
I never see mention of Windows with Session Manager. I have a mixed infrastructure with a number of Windows (IIS) app servers running various things.
We currently connect via SSH to a bastionhost, then tunnel from there to various systems, which allows connecting to SSH (linux instances), RDP (Windows), or basically any other network services like Redis or a database. I ended up writing some scripts to automate all this, so as long as you have the right certificates and IAM permissions, you can connect with a single command -- for Windows instances, it even retrieves the randomized password from the EC2 API. The end result is for any EC2 instances you're instantly popped into a shell/RDP session without having to enter credentials.
I'd love to replace this with something better (eg Session Manager), but I've not seen how to do this for RDP, and haven't had the time to go experimenting on my own to see if it's even possible. If I can't 100% replace the bastionhosts, having two entirely different connection methods doesn't solve anything (and in fact makes it worse, because it's harder to use).
Be careful with SSM in general. The documentation suggests adding the AmazonEC2RoleforSSM policy to the role of the EC2 instances you want to access via Session Manager. This role grants read/write to all S3 buckets in your account (amongst other things). See this article for better steps and unavoidable risky things: https://cloudonaut.io/aws-ssm-is-a-trojan-horse-fix-it-now/
Good call to watch out for this stuff. The examples in the repo we set up use the AmazonSSMManagedInstanceCore
managed policy, which does not grant any S3 permissions, just various ssm, ssmmessages, and ec2messages permissions.
> The documentation suggests adding the AmazonEC2RoleforSSM policy to the role of the EC2 instances
Which documentation do you mean? The article mentions the policy AmazonSSMManagedInstanceCore, which is the same as what's mentioned in the SSM setup guide:
Thanks for clarifying, I didn’t recheck since we rolled out SSM in mid-2019 and then scrambled when we realised we’d granted account wide S3 permissions. The article I linked to also has a recommended minimal IAM policy for Run Command and SSM. I’ll update my comment to mention this.
AmazonSSMManagedInstanceCore is still too much access, it has unscoped ssm:GetParameter! I hope you weren't trying to protect any secrets in ParameterStore!
There are so many AWS managed policies that provide access far beyond what one might suspect given the policy name and description.
Implementing least privilege with IAM can be really difficult in any moderately complex environment.
And close to impossible in highly complex environments. We're looking to adopt using AWS accounts to achieve a few objectives but one being reducing the blast radius of potential breaches and outages.
Does anyone know how this works with other utils that use SSH protocol, like rsync? What about tunneling other services to or from a local host? I'd love to have fewer hosts to maintain and a smaller network/attack surface, but we use SSH for more than just gaining commandline access to our instances.
>"No more bastion hosts required! Session Manager uses AWS APIs to communicate with your instances, so you can remove the administrative burden of maintaining bastion hosts."
Does this presume the EC2 instances have a public IP or is there a way this would also work with EC2 instances on private subnets?
48 comments
[ 25.6 ms ] story [ 1885 ms ] threadwe have some regulatory requirements that require us to use hardware tokens for 2FA access to servers.
2. what about SSH tunnels ?
If you're doing a federated login with Okta or another provider, you need to set up the hardware MFA there.
There is SSH tunneling support as well, will add an update on that soon.
"Tunnel created using SSM only allows single connection to destination port" - https://forums.aws.amazon.com/thread.jspa?threadID=314882&ts...
This has been sitting open in the support forums unanswered for over two months :/
1) Is logging for access from CLI finally supported?
2) Can I setup which shell is used?
3) Are logs readable when I switch to something else than sh?
4) Is U2F supported (awscli question)
Once all of these are fixed, then it can be possible to claim that SSM solves these issues. Otherwise it’s nothing more than for adhoc usage.
[1] https://cloud.google.com/compute/docs/instances/connecting-a...
But it seems no one like Azure these days, but they have some nifty features.
Would be interesting to lock down the session manager agent (if possible) so that the only way to privileged access is through sudo-like priv esc that uses 2fa.
For federation you need to rely on the config in whatever your identity provider is, like Okta.
So if you already have access control setup on your host, then SSM doesn't do anything to undermine it. If you don't have it, you'll still need to add it.
We currently connect via SSH to a bastionhost, then tunnel from there to various systems, which allows connecting to SSH (linux instances), RDP (Windows), or basically any other network services like Redis or a database. I ended up writing some scripts to automate all this, so as long as you have the right certificates and IAM permissions, you can connect with a single command -- for Windows instances, it even retrieves the randomized password from the EC2 API. The end result is for any EC2 instances you're instantly popped into a shell/RDP session without having to enter credentials.
I'd love to replace this with something better (eg Session Manager), but I've not seen how to do this for RDP, and haven't had the time to go experimenting on my own to see if it's even possible. If I can't 100% replace the bastionhosts, having two entirely different connection methods doesn't solve anything (and in fact makes it worse, because it's harder to use).
Which documentation do you mean? The article mentions the policy AmazonSSMManagedInstanceCore, which is the same as what's mentioned in the SSM setup guide:
https://docs.aws.amazon.com/systems-manager/latest/userguide...
Though I always suggest to read what managed profiles are doing. Many of them are very permissive
https://aws.amazon.com/about-aws/whats-new/2019/07/session-m...
>"No more bastion hosts required! Session Manager uses AWS APIs to communicate with your instances, so you can remove the administrative burden of maintaining bastion hosts."
Does this presume the EC2 instances have a public IP or is there a way this would also work with EC2 instances on private subnets?
Currently I have an EKS cluster accessible only on private subnets. It would be wonderful to to be able to access this without OpenVPN in the mix.
Connectivity from a private subnet to the AWS API could be (a) NAT gateway (b) HTTP proxy (c) PrivateLink VPC endpoint.