Hey segment, are you aware that you can have several env (dev, stag, prod, ...) on the same AWS account? :p
You can secure each environement with different credentials (IAM) so no need to create several AWS accounts!
I was also alarmed to see that dynamodb:DescribeStream is limited to 10 QPS per account, making low-latency stream consumers into noisy neighbors of each other. Then there's all the "open a support ticket" per-account limits on table provisioning and running EC2 instances…
It is recommended to use multiple accounts for isolation. AWS has recently launched Organizations to make creation and management of multiple accounts simple. See https://aws.amazon.com/organizations
Tools like aws-vault and now aws-okta make securing credentials a piece of cake.
In practice, cramming all this into the same account doesn't work. Segment is following best practice here.
For example, IAM doesn't provide the granularity in resources and conditions that you'd want to effectively isolate the blast radius of developer keys. ec2:TerminateInstances didn't (doesn't?) support VPC level conditions, so being able to terminate one instance meant you could terminate all instances.
Similarly, you might want your engineering team to iam:PutUserPolicy in development, but have a much more restricted group in production which isn't possible with IAM today.
The other bit would be blast radius. What if someone does get access to your single account? How confident are you that your policies were airtight? By using many accounts, you create clear isolation boundaries that require opt-in sharing.
>>> By using many accounts, you create clear isolation boundaries that require opt-in sharing.
In theory yes. In practice, you will achieve the opposite of that.
Developers and ops will have to juggle between 10 keys and accounts to get anything. The keys will end up saved and written all over the systems. It will be impossible to have audit between all the accounts and access.
ec2:TerminateInstances still doesn’t support the vpc as a condition but it does allow you to use tags. You can also limit a role to tagging only the instances they create (as long as they include the tags when they call ec2:RunInstances). You can even require that specific tags are present. Combine all of that with some kind of “owner” tag and it’s a pretty decent solution to the problem. Add automatic tagging on the backend and it’s even better.
That said, it’s not perfect and there’s probably plenty of resources it wouldn’t work for. It’s also comparatively fragile.
This just in: directories are pointless because you can put all files in the same one!
---
There are advantages in multiple accounts, for organization and security. Random example: adding tags to ec2 instances is an account-wide permission, for all ec2 instances. Multiple various things by 100 and accounts make sense.
OP here. You _can_ do this but IAM Policy wise it is a mess and not very maintainable.
You end up needing to constantly grant and revoke access to individual resources if you went this route. Instead, it's nice to give engineers access to everything that's in their account, and not worry about IAM policies.
Really the problem is that IAM is not a capability system. If you can write a policy at all, there are no limits on what you can put in it, so we can only let trusted admins touch policies. An engineer should be able to grant some of their own privileges to the things they run.
I honestly don't know of the feature parity between the two, but we are definitely Okta power-users having added close to 500 apps in the last year. I don't know if we could drive all of the same processes we have built (especially on the HR side) using GSuite.
GSuite can be an IdP for SAML but it’s sometimes limited. AFAIK you can’t use it to connect to AWS using SAML because AWS makes some assumptions about attributes in the identity. I may be wrong here, but I tried to do this a few months ago (with Duo and Yubikeys added in) and was not able to get it working. I ended up using Okta and it was simple.
About a year ago, I migrated my team's AWS access from manually-provisioned AWS keys to GSuite authentication, using a combination of SAML and OpenID Connect (OIDC):
- Detailed instructions for Google-federated login to the AWS Management Console through SAML are available [1], and worked more or less as described.
- CLI/API access is a bit trickier and less thoroughly documented, but is possible using the 'Web Identity Federation' feature [2]. Basically, you generate/refresh temporary AWS credentials by passing a Google OIDC token to the AssumeRoleWithWebIdentity API. The tricky part is keeping both your Google OIDC and AWS STS tokens conveniently refreshed. I wrote some open-source glue code for this that hooks into the AWS Ruby SDK and CLI [3]. It's still a little rough around the edges and not yet extracted into a standalone project, but it's been working well enough for my team over the last year.
Do you have some sort of failsafe in place for developers to be able to access AWS in the event of an Okta outage? Or are the Okta user sessions long-lived enough that you'd be able to mitigate some other way if Okta was down?
Glad Segment is making good use of aws-vault and it's components! Did you consider whether adding okta support in aws-vault might have made sense? I've been thinking about doing that for a while, interested what made y'all decide to create a new one.
P.S. Any chance of a link back in the mention of aws-vault in the article? (aws-vault author here)
Hey. I'll make sure to update the blog to include a shout-out to aws-vault! We were heavy users for the last few years.
We are sometimes wary about contributing stuff like this upstream because it's not always accepted, and then we are in a weird spot maintaining a fork (it happened with the bitly oauth proxy when we added okta support). I'll see what type of work it would take to make it fit in to aws-vault nicely!
Hi there. I'm confused by your statement that "The integration guide recommends connecting the identity provider to each AWS account ..."
This used to be the way Okta recommended integrating to AWS, but in the guide you linked[1] they are using pretty much exactly the same approach you did - authentication to an identity account (ops account in your terminology) then assume-role into the target account.
I can't see any major differences between the standard Okta approach and the way you outlined in your blog. Could let me know if I missed something?
How would this compare to an LDAP, an LDAP with org wide access control can pretty much enable all of this(keeping in mind GDPR & other security concerns) from just 1 AWS account ?
I can't answer for Segment, but our AWS account strategy is very similar, so we've found that there is still a need for a centralized KMS solution (system-to-system auth). Our teams are working on a project to wrap more tooling around Vault and make it a centralized KMS. T-Mobile has done some work on this... https://opensource.t-mobile.com/blog/posts/introducing-tvaul...
If you use aws-vault --server or something like it that mocks a metadata server, all the sdks handle rotation properly. Just need to fork to connect it to whatever generates tokens
It's good to see an article talking about this. A lot of organizations could benefit from using more than one account to enforce security (IAM is hard!) and separation of concerns.
It doesn't really explain it but to do this, the root account has to be enrolled for AWS Organization[1]. This is what is being used to handle all the accounts and consolidate the billing. It also allows to create rules span all the accounts. Recently terraform gained support for the Organization API[2] so it's possible to control the account list in a declarative manner.
The biggest issue is that now that there are a lot of accounts, the developers need a way to switch between them. Using the IAM assume-role mechanism is a good way to avoid needing a lot of AWS keys per developer.
I don't know if I agree with using Okta as it adds another party that now has access to AWS. I don't see the difference between having a AWS secret or and Okta secret in the keychain security-wise. Okta might provide audit logging facilities but so does AWS.
In either case you will need to generate a `~/.aws/config` per developer. There is also a Chrome plugin[3] that can read this file format and populate the AWS Console switch role. I don't know if the extension publisher is reputable yet as it gives a lot of access to the extension.
It should be noted that Orgs should be setup when new AWS accounts are created, as it can be tricky to perform once your have systems in production. I would even go so far as to say if you want to commit to Orgs, create a new AWS account hierarchy and move your applications and infrastructure into it.
"Warning: We strongly recommend that you do not attach SCPs to the root of your organization without thoroughly testing the impact that the policy has on accounts. Instead, create an OU that you can move your accounts into one at a time, or at least in small numbers, to ensure that you don't inadvertently lock users out of key services."
If you already have an internal directory/SSO (e.g. Active Directory) the advantage to using something like Okta (or another decent SAML app/provider) is the user experience. It makes SSO really simple (you could install the agent on a desktop), has simple directory sync and allows you to create network source rules (e.g. trusted networks) and integrates with several 2FA/MFA solutions out of the box. If you don’t already have an internal system like Active Directory it might not be worth the extra single point of failure/compromise.
I can see Amazon displacing these services over time (they have a lot of the components already) but last time I looked they weren’t there yet.
I don’t want it to seem like this is a sales pitch either. Okta has plenty of warts, odd decisions, glaringly obvious missing features and has done some bizarre 180s in terms of organizational direction.
It was a long time ago but one bug I won’t forget that was shocking: a combination of special characters in a password resulted in a sync failure for a couple of accounts. That’s bad on its own because it raises all kinds of questions about how they’re handling and sanitizing strings. But the whopper was that the errors presented to admins had the passwords in clear text. You would assume masking would be a key thing in their code and logging systems and it made me question if they even had masking at all if a user password made it all the way to the dashboard.
My organization evaluated Okta and I also came away worried about introducing another third party that had access to all of our services, both from a security perspective and as another point of failure.
In particular, in Okta's SWA apps they have a setting for those apps to use the same password as the user's Okta password. If there's a way they do this without storing all of their user passwords unhashed, I couldn't figure it out and the Okta reps weren't able to provide a clear answer.
Edit: I _can_ think of one potential solution: Okta initially sets user's password to a random string, when you authenticate with Okta / change your Okta password, they use the unhashed version to make admin calls to update the apps' password and never persist it. But they didn't give any indication that this was the case.
Out of curiosity, how do people handle communication between services that are in different accounts? VPC peering to connect all accounts? Public Internet and OAuth? Another method?
You need to define “between services”. You mentioned oauth and VPC peering. One is authentication, the other is networking accessibility. There is no one solution.
Generally you have: VPC peering between accounts, Network Access Control List (NACL) for VPC port control, security groups between instances (and some AWS services which uses SG to limit port access), IAM roles to authenticate and authorized certain AWS services to do things (e.gz Lambda to read S3 bucket) - but IAM and policies (bucket policies, SQS policies) govern authentication and authorization. Finally there is also organization service which allows you to control what AWS services are allowrd in a group of AWS accounts.
Sorry, on mobile so I can’t make a prettier list. I am generally disappointed at the complexity of authentication and authorization mechanisms exists for AWS services to be really honest.
Oh, sorry, absolutely. I'm actually talking about services that talk to each other using HTTP.
One option I've though of is using the public internet to connect them and use OAuth to make sure that they are actually allowed to communicate, solving the security issues on the application level.
Another option would be to allow only access from the internal network, but then you have to connect them somehow.
I'd recommend against the latter in any medium or large organization (and small as well, but sometimes one needs to make trade offs).
That's putting too much trust in the network. You don't want one phished internal user to expose the entire set of services that developers all think of as "on a trusted network".
This is a good question. I will simplify this to two rules:
* internal traffic goes through VPC, AWS backone, and Direct Connect / VPN (company and AWS accessing each other). Worth noting that, all S3 requests used to go through the Internet, but now we can enable S3 endpoint so requests originated from VPC is now made within AWS backbones).
* incoming public traffic comes through AWS public infrastructure (e.g. load balancer) before handing off to some EC2 instances (the instance could either be in "public" subnet, or "private subnet")
There is a whole lot of architecture approaches highly dependent on the requirements, and I don't think we can discuss them here.
For one microservice to talk to the other one, if all within the same VPC, you just use security group (Network ACL "defends" subnet, but you are better off just using route table). If not within the same VPC, you can peer. If not within the same account, you can peer. I believe now you can peer region too. In some cases, you have to route traffic from VPC1 to VPC2 through your corporate routing...
PrivateLink endpoints between accounts and VPCs if the service must remain private [1], public TLS endpoints either with or without TLS fingerprinting for services that must be public.
"You can create your own application in your VPC and configure it as an AWS PrivateLink-powered service (referred to as an endpoint service). Other AWS principals can create a connection from their VPC to your endpoint service using an interface VPC endpoint. You are the service provider, and the AWS principals that create connections to your service are service consumers."
Excellent article thanks for sharing. It's awesome to see someone documenting their experiences with multi-account strategy.
You said you manage account with Terraform, are there any other resource types that you don't give the dev teams access to such as networking? Also, how are you deploying Terraform at scale across all of your accounts?
What are best practices for sharing build artifacts (e.g. images in ECR, files in S3) among AWS accounts? Cross-account IAM policies, explicit promotion process to move artifacts between accounts, or something else?
We use ECR and have all of our images in our hub account, and make them readable from the prod/stage/dev accounts using a cross account policy.
For S3, it's more complicated because of object permissions and all the insanity that comes with cross-account writing etc.
Edit for more info: No explicit promotion process for containers. Engineers build images in our CI pipeline when their pull request is approved and merged to master, which is pushed to the hub account.
Users authenticate to an internal website with ADFS (including MFA) and are then presented with a list of roles where they can either click through to the website assuming a role in that account for an hour, or click an option to access temporary credentials.
The AWS roles are deployed from our CI/CD pipeline to all of our AWS accounts, so we don't have to have user accounts anywhere and can still deploy features from our Pipeline without logging in.
We are also in the process of setting up automated account provisioning from our HR system, Workday. Based on the users team and job title, they'll be added into an Active Directory group which would then give them access to the resources required for their role.
Once complete, this will save the support team a lot of time!
This is really similar to what Amazon uses in-house to manage aws roles. Not sure if there’s a commercial or open-source alternative but this could be a fruitful project for someone!
I am interested in the thought processes behind setting up separate AWS accounts "for GDPR compliance" as opposed to having one AWS account and running multi regions (via VPC etc.) under that account?
Is it because teams in different locations (US, EU etc.) can run semi-independently? How do you manage shared resources (e.g. S3 or RDS SQL servers) that need to be accessed from multiple regions, yet still maintain GDPR compliance?
56 comments
[ 2.9 ms ] story [ 121 ms ] threadIt sucks not being able to fix the production A record because somebody did a lot of changes on the staging zone.
Tools like aws-vault and now aws-okta make securing credentials a piece of cake.
For example, IAM doesn't provide the granularity in resources and conditions that you'd want to effectively isolate the blast radius of developer keys. ec2:TerminateInstances didn't (doesn't?) support VPC level conditions, so being able to terminate one instance meant you could terminate all instances.
Similarly, you might want your engineering team to iam:PutUserPolicy in development, but have a much more restricted group in production which isn't possible with IAM today.
I've taken this pretty far in the past to attempt segmenting within one account, but always run into limits: https://github.com/witoff/self-service-iam
In theory yes. In practice, you will achieve the opposite of that.
Developers and ops will have to juggle between 10 keys and accounts to get anything. The keys will end up saved and written all over the systems. It will be impossible to have audit between all the accounts and access.
Per-account isolation is great for security and especially reliability, if you run in to constant ratelimit issues like we do.
That said, it’s not perfect and there’s probably plenty of resources it wouldn’t work for. It’s also comparatively fragile.
---
There are advantages in multiple accounts, for organization and security. Random example: adding tags to ec2 instances is an account-wide permission, for all ec2 instances. Multiple various things by 100 and accounts make sense.
You end up needing to constantly grant and revoke access to individual resources if you went this route. Instead, it's nice to give engineers access to everything that's in their account, and not worry about IAM policies.
As with all of AWS (and software, and life) there is no one size fits all answer.
https://aws.amazon.com/blogs/security/how-to-set-up-federate...
- Detailed instructions for Google-federated login to the AWS Management Console through SAML are available [1], and worked more or less as described.
- CLI/API access is a bit trickier and less thoroughly documented, but is possible using the 'Web Identity Federation' feature [2]. Basically, you generate/refresh temporary AWS credentials by passing a Google OIDC token to the AssumeRoleWithWebIdentity API. The tricky part is keeping both your Google OIDC and AWS STS tokens conveniently refreshed. I wrote some open-source glue code for this that hooks into the AWS Ruby SDK and CLI [3]. It's still a little rough around the edges and not yet extracted into a standalone project, but it's been working well enough for my team over the last year.
[1] https://aws.amazon.com/blogs/security/how-to-set-up-federate...
[2] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_pr...
[3] https://github.com/code-dot-org/code-dot-org/blob/staging/li...
If there was an emergency we have our root accounts to recover access, but it's a good point that we should probably have a second IDP as a backup.
P.S. Any chance of a link back in the mention of aws-vault in the article? (aws-vault author here)
We are sometimes wary about contributing stuff like this upstream because it's not always accepted, and then we are in a weird spot maintaining a fork (it happened with the bitly oauth proxy when we added okta support). I'll see what type of work it would take to make it fit in to aws-vault nicely!
This used to be the way Okta recommended integrating to AWS, but in the guide you linked[1] they are using pretty much exactly the same approach you did - authentication to an identity account (ops account in your terminology) then assume-role into the target account.
I can't see any major differences between the standard Okta approach and the way you outlined in your blog. Could let me know if I missed something?
[1] https://support.okta.com/help/servlet/fileField?retURL=/help...
For example, if you want to run the Athena JDBC driver locally or read something from S3 in a longish (>1h) running Python script.
It doesn't really explain it but to do this, the root account has to be enrolled for AWS Organization[1]. This is what is being used to handle all the accounts and consolidate the billing. It also allows to create rules span all the accounts. Recently terraform gained support for the Organization API[2] so it's possible to control the account list in a declarative manner.
The biggest issue is that now that there are a lot of accounts, the developers need a way to switch between them. Using the IAM assume-role mechanism is a good way to avoid needing a lot of AWS keys per developer.
I don't know if I agree with using Okta as it adds another party that now has access to AWS. I don't see the difference between having a AWS secret or and Okta secret in the keychain security-wise. Okta might provide audit logging facilities but so does AWS.
In either case you will need to generate a `~/.aws/config` per developer. There is also a Chrome plugin[3] that can read this file format and populate the AWS Console switch role. I don't know if the extension publisher is reputable yet as it gives a lot of access to the extension.
[1]: https://aws.amazon.com/organizations/
[2]: https://github.com/terraform-providers/terraform-provider-aw...
[3]: https://chrome.google.com/webstore/detail/aws-extend-switch-...
EDIT: https://docs.aws.amazon.com/organizations/latest/userguide/o...
"Warning: We strongly recommend that you do not attach SCPs to the root of your organization without thoroughly testing the impact that the policy has on accounts. Instead, create an OU that you can move your accounts into one at a time, or at least in small numbers, to ensure that you don't inadvertently lock users out of key services."
* create a new AWS account
* setup IAM and AWS Org
* add 2nd factor on the root account, put the Gemalto in a safe
Then add the "legacy" account to the org and slowly port all the resources to fresh new accounts.
I can see Amazon displacing these services over time (they have a lot of the components already) but last time I looked they weren’t there yet.
I don’t want it to seem like this is a sales pitch either. Okta has plenty of warts, odd decisions, glaringly obvious missing features and has done some bizarre 180s in terms of organizational direction.
It was a long time ago but one bug I won’t forget that was shocking: a combination of special characters in a password resulted in a sync failure for a couple of accounts. That’s bad on its own because it raises all kinds of questions about how they’re handling and sanitizing strings. But the whopper was that the errors presented to admins had the passwords in clear text. You would assume masking would be a key thing in their code and logging systems and it made me question if they even had masking at all if a user password made it all the way to the dashboard.
In particular, in Okta's SWA apps they have a setting for those apps to use the same password as the user's Okta password. If there's a way they do this without storing all of their user passwords unhashed, I couldn't figure it out and the Okta reps weren't able to provide a clear answer.
Edit: I _can_ think of one potential solution: Okta initially sets user's password to a random string, when you authenticate with Okta / change your Okta password, they use the unhashed version to make admin calls to update the apps' password and never persist it. But they didn't give any indication that this was the case.
Generally you have: VPC peering between accounts, Network Access Control List (NACL) for VPC port control, security groups between instances (and some AWS services which uses SG to limit port access), IAM roles to authenticate and authorized certain AWS services to do things (e.gz Lambda to read S3 bucket) - but IAM and policies (bucket policies, SQS policies) govern authentication and authorization. Finally there is also organization service which allows you to control what AWS services are allowrd in a group of AWS accounts.
Sorry, on mobile so I can’t make a prettier list. I am generally disappointed at the complexity of authentication and authorization mechanisms exists for AWS services to be really honest.
Another option would be to allow only access from the internal network, but then you have to connect them somehow.
That's putting too much trust in the network. You don't want one phished internal user to expose the entire set of services that developers all think of as "on a trusted network".
* internal traffic goes through VPC, AWS backone, and Direct Connect / VPN (company and AWS accessing each other). Worth noting that, all S3 requests used to go through the Internet, but now we can enable S3 endpoint so requests originated from VPC is now made within AWS backbones).
* incoming public traffic comes through AWS public infrastructure (e.g. load balancer) before handing off to some EC2 instances (the instance could either be in "public" subnet, or "private subnet")
There is a whole lot of architecture approaches highly dependent on the requirements, and I don't think we can discuss them here.
For one microservice to talk to the other one, if all within the same VPC, you just use security group (Network ACL "defends" subnet, but you are better off just using route table). If not within the same VPC, you can peer. If not within the same account, you can peer. I believe now you can peer region too. In some cases, you have to route traffic from VPC1 to VPC2 through your corporate routing...
"You can create your own application in your VPC and configure it as an AWS PrivateLink-powered service (referred to as an endpoint service). Other AWS principals can create a connection from their VPC to your endpoint service using an interface VPC endpoint. You are the service provider, and the AWS principals that create connections to your service are service consumers."
[1] https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/endpo...
For S3, it's more complicated because of object permissions and all the insanity that comes with cross-account writing etc.
Edit for more info: No explicit promotion process for containers. Engineers build images in our CI pipeline when their pull request is approved and merged to master, which is pushed to the hub account.
https://developer.atlassian.com/blog/2017/12/introducing-clo...
Users authenticate to an internal website with ADFS (including MFA) and are then presented with a list of roles where they can either click through to the website assuming a role in that account for an hour, or click an option to access temporary credentials.
The AWS roles are deployed from our CI/CD pipeline to all of our AWS accounts, so we don't have to have user accounts anywhere and can still deploy features from our Pipeline without logging in.
We are also in the process of setting up automated account provisioning from our HR system, Workday. Based on the users team and job title, they'll be added into an Active Directory group which would then give them access to the resources required for their role.
Once complete, this will save the support team a lot of time!
I'll bring it up again. Personally I don't see why not!
Is it because teams in different locations (US, EU etc.) can run semi-independently? How do you manage shared resources (e.g. S3 or RDS SQL servers) that need to be accessed from multiple regions, yet still maintain GDPR compliance?
The GDPR line is about improving our least privilege access story. There's a lot more to GDPR than that, though