It's great to see Kubernetes being integrated more tightly with the AWS ecosystem. If now all cloud providers open sourced their MySQL/PostgreSQL forks...
Are there any good docker compose or helm files out there right now that would work for a nice production ready MySQL or PostgreSQL dB?
It seems like k8s has everything you would need to have the redundant data sources, failover, and point in time recovery options that cloudsql or auroradb have.
The open source database vitess is pretty much designed to be a cloud native production mysql and there are helm charts for it (https://github.com/vitessio/vitess/tree/master/helm) an operator based on the the metacontrollers (https://github.com/vitessio/vitess-operator) and a company called planetscale thats making an operator that has even more features then those two (I work for planetscale and have been building our operator )
The cloud fight of 2019-2020: AWS vs GCP in the Kubernetes arena. Curious to see who's going to win, of if it's going to be a tie.
Jokes apart: GCP got a head start in containers thanks to Kubernetes; AWS realized it and tried to catch up. Dominating the space will have huge consequences down the road.
My humble view is that whoever starts a RedHat-like service (with support, and SLAs, and enterprise services) on top of Kubernetes, might get the upper hand. Having built Kubernetes might not be enough for GCP to maintain the lead.
For serverless technologies to "win" they have to solve the "cold start" problem. AWS likes to pitch Lambdas as an easy mobile backed, but if you need to talk to a DB (which most mobile backends do) then you'll want to put your Lambda in a VPC, which makes cold starts on the order of 5-10 seconds, which is a deal breaker for most synchronous APIs.
I don't understand why AWS or GCP haven't added "pre-warming" requests to their cloud functions, similar to App Engine.
It is somewhat arbitrary though, isn't it? If AWS adds the ability to use security groups without a VPC, a lot of these issues go away wrt vpc or siloed infrastructure limits (cold starts).
There are still reasons to be in a private network - Being "one typo away" from exposing your services/db to the world is scary. But that seems like a solveable problem as well...
> The point is pretty moot though, because you can schedule Cloudwatch Events every 4 minutes to keep a lambda warm, if necessary.
I encourage you to read this article, https://theburningmonk.com/2018/01/im-afraid-youre-thinking-... , because if you're running a web API with Lambdas, keeping one instance warm with the "cloudwatch event every 4 minutes" trick will most definitely not solve your cold start issues.
> I think many still repeat the "conventional wisdom" about the cold start, and never get past that point.
First comment on your article nails it. At the end of the day lambdas scheduling is a black box. People have deduced certain behavior, but AWS is explicit about not relying on undocumented behavior.
I would be loath to recommend lambda for any application where business performance is sensitive to the services latencies.
It kind of makes sense, but mostly from a marketing and planning standpoint. If you need pre-warming, that probably means you know enough about lambda (and AWS) to know that you need pre-warming. Setting it up via a Cloudwatch event is easy as pie.
Think about the implications if they added a button to the lambda console to "pre-warm". There are two options: (1) set up the cloudwatch event for you (which is a similar pattern we've seen AWS use for things like DynamoDB table autoscaling), or (2) have some other internal system which can keep them warm.
Its easy to say "just do (1), it'd be so easy", but the issue is that it introduces a very weird cost pattern to lambda. Lambda isn't just billed per invocation, its billed essentially with time live. So if they auto-configure a cloudwatch event, lets say it sends an empty `{}` argument, they have no idea how long your function is designed to run given that input. Moreover, they don't even know that your function won't error with that input. So they've got this new feature and even they can't predict what it will do to your bill or system stability, given the fact that we're dealing with arbitrary code blobs.
The only option is (2). Now think about allocating engineering effort to this problem: as a manager, would you rather allocate a team to work on an extra complex scheduling parameter, or continue to improve the fundamental warm-up time for any function? Maybe both. But now you've got this extra parameter there which increases customer expectations and makes future scheduling work much more difficult.
The keep-alive behaviour of Lambda is essentially a subsidy. I suspect that AWS have paid a lot of attention to ways to cut it down, or are at least hoping the they can drive down the idling cost without anyone noticing too much of a performance hit.
> For serverless technologies to "win" they have to solve the "cold start" problem.
I've said here and elsewhere before that autoscaling is easy to say and hard to do.
We keep looking to autoscalers to divine our economic preferences, which they cannot do for us. What's been missing is the ability to explicitly trade off latency for expense.
The best you can do is to a) attack startup time any how, any way possible, b) react sanely to unexpected traffic changes, c) make reasonable forecasts and d) explicitly tune cost of idleness vs cost of delay vs probability of delay. These help, but the problem will never fully go away.
(Unless you've discovered an escape hatch from either of causality or integral calculus. If you have, please share it with the class.)
Have a low latency container based API with min replicas and auto-scale, almost like an atomic CRUD API. Move as much to async serverless which is triggered on events.
> My humble view is that whoever starts a RedHat-like service (with support, and SLAs, and enterprise services) on top of Kubernetes, might get the upper hand.
Does Red Hat count as Red Hat-like? Because they've had OpenShift Origin for several years now.
>we need to set up a way to manage AWS IAM credentials to Kubernetes pods... In a production system, this should be done using a tool such as kube2iam or kiam...
I am curious if AWS has any plans to build an IAM integration for K8s that provides IAM credentials/roles directly to pods. An integration through EKS or K8s directly would make interacting with AWS resources very easy.
AWS always launches services half-baked - I actually kinda love them for it. It's not always polished, but it's usually good enough to help me get stuff done.
I'm sure we'll see much tighter integration over time.
https://github.com/jtblin/kube2iam is probably what you're looking for, it uses iptables to allow/disallow pods requests to the ec2 metadata service based on kubernetes annotations
edit: haven't fully read the article yet but if the operator supports managing IAM roles thru a CRD you could potentially create the role and attach it via annotation in one go.
double edit: looks like IAM roles aren't directly supported yet, the following is what appears to be supported:
I understand that both kube2iam and kiam exist and they're both fine. I am interested in Amazon/EKS directly supporting something. Hopefully we will see something soon via CRD or otherwise.
If I still need to create an IAM role externally for the pod granting it access to the different resources.. Might as well create the resources themselves as well?
Correct, they are on the roadmap, I've been waffling on the implementation because this could open security issues. I'm happy to say we'll at least be able to use k8s RBAC to gate who can get, list, create, update and delete the Roles but your security posture from the node perspective still will need to gate what the pods could assume. https://github.com/awslabs/aws-service-operator/issues/58https://github.com/awslabs/aws-service-operator/issues/59 are the issues if you'd like to add any extra notes or check out the potential implementation.
I too am excited to hear more. The necessity for kube2iam and kaim are one of the major blockers that have prevented us from considering EKS for a client, the major sticking point being that they move the implementation of shipping security material around from the AWS to the customer side of the shared responsibility model.
We've been using kube2iam for this for a couple of years.
You just create a role give it an assume role policy that allows the node to assume it. Then annotate your pod w/ the role arn. When they make a call to get their instance profile you get the role instead.
It's a little annoying in that your pod code thinks its making a metadata call (which is super super fast), but what is actually happening is kube2iam intercepted that and will make a sts:assumerole call... which takes forever. So people just need to set their timeout a little higher than normal.
I bet you could register the containers as on-prem instances with the Systems Manager agent, though it's kind of a pain. It gives you a shim into IAM from non-AWS machines.
(I work at AWS SSM, but not directly on the on-prem featureset.)
Amazon has had that for a while too, but I just learned about it recently, see [1]. I wonder if this is doing the same thing under the hood, or if it is a competing project within Amazon. Regardless, something like this is sorely needed for making infrastructure changes deployable along with application changes.
The difference between "meet customers at a Kubernetes CRD" and "meet customers at an OSBAPI broker" is small enough that it's not worth paying strategy tax to crimp the latter, especially since these customer groups will be overlapping but not equivalent sets (this is how enterprise software grows so vast).
This is absolutely awesome. I've thought about doing this a couple of times, as a abstraction layer on top of different clouds, but this is really cool... It's also the first time that I think we have really seen AWS really contribute something to the K8s ecosystem they do lots of good work at the CNCF that is interesting and innovative. (EKS is not as capable as GKS or AKS, and even things like HPA only recently are enabled).
Great question, it's a little more complicated than one might think at first. In trying to build a "batteries included" experience I'd need to have per-view into your cluster and what VPC, Subnet and AZ you are running in I don't want to make this a configuration option so I need to build out a way to collect this information dynamically so that I can make sure we create DB subnets for the RDS to provision into. Then I need to configure depending on the engine (pg, mysql etc) the port and security group configuration. All in all the CFT and is more complicated and with the way the resources are code generated it requires heavy customization. All that being said it is well up on the top of my list to implement. Also always interested in letting other folks come and contribute if they feel inclined. :)
55 comments
[ 3.6 ms ] story [ 101 ms ] threadIt seems like k8s has everything you would need to have the redundant data sources, failover, and point in time recovery options that cloudsql or auroradb have.
https://www.crunchydata.com/products/crunchy-postgresql-for-...
Jokes apart: GCP got a head start in containers thanks to Kubernetes; AWS realized it and tried to catch up. Dominating the space will have huge consequences down the road.
My humble view is that whoever starts a RedHat-like service (with support, and SLAs, and enterprise services) on top of Kubernetes, might get the upper hand. Having built Kubernetes might not be enough for GCP to maintain the lead.
https://www.openshift.com
I don't understand why AWS or GCP haven't added "pre-warming" requests to their cloud functions, similar to App Engine.
For many years now, essentially all AWS services are tied to a VPC.
Each account gets 5 VPCs per region, by default.
Whether you use RDS or EC2 to setup a database server, it will be tied to a VPC for networking isolation purposes.
As such you then would need the Lambda in the VPC, or to allow public internet access to the database.
The point is pretty moot though, because you can schedule Cloudwatch Events every 4 minutes to keep a lambda warm, if necessary.
Frameworks like Zappa even do this for you automatically.
There are still reasons to be in a private network - Being "one typo away" from exposing your services/db to the world is scary. But that seems like a solveable problem as well...
I encourage you to read this article, https://theburningmonk.com/2018/01/im-afraid-youre-thinking-... , because if you're running a web API with Lambdas, keeping one instance warm with the "cloudwatch event every 4 minutes" trick will most definitely not solve your cold start issues.
First comment on your article nails it. At the end of the day lambdas scheduling is a black box. People have deduced certain behavior, but AWS is explicit about not relying on undocumented behavior.
I would be loath to recommend lambda for any application where business performance is sensitive to the services latencies.
Think about the implications if they added a button to the lambda console to "pre-warm". There are two options: (1) set up the cloudwatch event for you (which is a similar pattern we've seen AWS use for things like DynamoDB table autoscaling), or (2) have some other internal system which can keep them warm.
Its easy to say "just do (1), it'd be so easy", but the issue is that it introduces a very weird cost pattern to lambda. Lambda isn't just billed per invocation, its billed essentially with time live. So if they auto-configure a cloudwatch event, lets say it sends an empty `{}` argument, they have no idea how long your function is designed to run given that input. Moreover, they don't even know that your function won't error with that input. So they've got this new feature and even they can't predict what it will do to your bill or system stability, given the fact that we're dealing with arbitrary code blobs.
The only option is (2). Now think about allocating engineering effort to this problem: as a manager, would you rather allocate a team to work on an extra complex scheduling parameter, or continue to improve the fundamental warm-up time for any function? Maybe both. But now you've got this extra parameter there which increases customer expectations and makes future scheduling work much more difficult.
I've said here and elsewhere before that autoscaling is easy to say and hard to do.
We keep looking to autoscalers to divine our economic preferences, which they cannot do for us. What's been missing is the ability to explicitly trade off latency for expense.
The best you can do is to a) attack startup time any how, any way possible, b) react sanely to unexpected traffic changes, c) make reasonable forecasts and d) explicitly tune cost of idleness vs cost of delay vs probability of delay. These help, but the problem will never fully go away.
(Unless you've discovered an escape hatch from either of causality or integral calculus. If you have, please share it with the class.)
Have a low latency container based API with min replicas and auto-scale, almost like an atomic CRUD API. Move as much to async serverless which is triggered on events.
Does Red Hat count as Red Hat-like? Because they've had OpenShift Origin for several years now.
I am curious if AWS has any plans to build an IAM integration for K8s that provides IAM credentials/roles directly to pods. An integration through EKS or K8s directly would make interacting with AWS resources very easy.
Being able to authenticate to the K8s cluster using https://github.com/kubernetes-sigs/aws-iam-authenticator is nice, but it doesn't help give pods IAM roles.
I'm sure we'll see much tighter integration over time.
in fact, if you check out the source (located here: https://github.com/awslabs/aws-service-operator) it's recommended to use kube2iam
edit: haven't fully read the article yet but if the operator supports managing IAM roles thru a CRD you could potentially create the role and attach it via annotation in one go.
double edit: looks like IAM roles aren't directly supported yet, the following is what appears to be supported:
- cloudformation templates
- dynamodb
- s3
- sns subscriptions and topics
- sqs queues
- ecr repos
Full disclosure: I work on EKS at AWS
You just create a role give it an assume role policy that allows the node to assume it. Then annotate your pod w/ the role arn. When they make a call to get their instance profile you get the role instead.
It's a little annoying in that your pod code thinks its making a metadata call (which is super super fast), but what is actually happening is kube2iam intercepted that and will make a sts:assumerole call... which takes forever. So people just need to set their timeout a little higher than normal.
https://github.com/jtblin/kube2iam
(I work at AWS SSM, but not directly on the on-prem featureset.)
Service catalog is based on the open service broker spec.
[1] https://cloud.google.com/kubernetes-engine/docs/how-to/add-o...
1. https://aws.amazon.com/blogs/opensource/provision-aws-servic...
Rephrasing: AWS are smart to have a bob each way.