> dependencies like databases being provisioned through static analysis in real-time.
I'm probably extremely biased (I'm a sysadmin), but this feels like a terrifying way to make your infrastructure entirely implicit and remove anybody's ability to coherently reason about the stuff underneath your application.
I'm not entirely averse to magic infrastructure beans but having an intermediate step where things are explicitly spelled out would make me far less nervous. Using shuttle to spit out a TF file – cool, using shuttle to automagically spin stuff up – terrifying.
It seems like it'd be all too easy to accidentally create databases and then forget about them when you correct an annotation typo or something. I'm 100% sure that I would do that.
Or, if they try to solve that, you always have the much more exciting possibility of just deleting databases when you accidentally stopped referencing them for one commit:)
There are, of course, a range of other possibilities, such as sending an email when the last reference to a database is removed so that a human can intervene appropriately or having a web app that provides access to the current list of unreferenced databases.
> but this feels like a terrifying way to make your infrastructure entirely implicit
This exact problem is my main beef with an unfortunately common and popular style of functional programming : stuff so condensed, so abstracted and so focused on describing the problem all the while hiding the actual execution / implementation / solution layers it's quasi-impossible to reason about and understand what actually happens when the program runs.
You can add tons of structured logs, log runtime configuration... as opposed to trying to "observe" what is going by collecting unrelated pieces of informal logs (the state of the art). People code at high level, tools write instrumented low-level code.
Hey! I'm one of the maintainers of the project so I thought I would address some of the comments here :)
> this feels like a terrifying way to make your infrastructure entirely implicit and remove anybody's ability to coherently reason about the stuff underneath your application.
There's two parts to this. The first is understanding what is happening under the hood since we're automatically provisioning so much for you (subdomain, LBs, DBs, etc.). Right now this is a little opaque, but a dashboard is in the roadmap to give users visibility on what's happening 'under the hood' - showing your provisioned infrastructure as well as how it's all wired up.
The second point is about potentially destructive actions - here we're going to be following a 'terraformesque' philosophy where infrastructure diffs are presented to the user and need to be accepted explicitly when deploying (or via a `--auto-approve` flag).
I'm glad that it's in your sights. And yeah, a terraform style model where you generate an explicit plan and then separately apply it helps with both problems.
This is a pedantic side comment, but I believe the very first sentence of this article is incorrect! AFAIK, Facemash was hosted on Zuckerberg's personal computer in his dorm room; ditto for the first version of Facebook shortly thereafter. After that, it was moved to a rented server.
source: I didn't attend Harvard, but I worked for their IT department beginning in 2005. Several of my coworkers had been directly involved in the disciplinary action against Zuckerberg and were very familiar with the details. That said, it is entirely possible I am misremembering. But other accounts seem to agree, e.g. https://www.quora.com/How-did-Mark-Zuckerberg-initially-host...
I think the future of hackathon/startup coding will involve tools to make infrastructure that's far more cookie cutter and easy to setup than today. This is a cool approach - starting with Rust is a ballsy move, though. I feel like the niche this serves for rapid prototyping might be more suited for python/typescript. Not sure if static analysis is easier or harder with those though.
Maybe I'm conservative, but Infrastructure as code seems like a great idea.
Infrastructure from code seems convenient for side-projects, but absolutely awful for something big where your infrastructure is now implicit in your code??
As long as the reporter can attach table schemas (presumably also created in-code) and data (which can be loaded by the code... presumably) then sure yeah. It's a lot better than having to ask for versions and reproduction cases that don't often come automatically.
> Plus, having to deploy code from the ticket just to be able to reproduce a bug seems like a huge PITA. Give me the local execution!
Ideally the code could be deployed automatically -- perhaps only from trusted/internal parties depending on the product -- so you don't have to actually do that much work. Less than you would if you had to create a database with the specific flags and environment exactly matching those of the reporter.
I dunno, I dream of this sort of reproducible bug report. I don't like the "what version" "what queries" "what errors" back and forth that always seems to happen, even with internal users.
Is spinning up a cloud-managed database really infrastructure as code? Seems more like SaaS from code. To me, infrastructure from code would at bare minimum mean spinning up a Kubernetes cluster capable of deploying database and similar containers so that you're not paying a premium to have some blackbox cloud service supposedly managing your services.
Infrastructure has multiple layers. Why does your bare minimum infrastructure start with Kubernetes and not with the OS underneath? Why does it start with the OS and not the firmware configuration? Why does it start with the firmware and not the hardware configuration you order from the vendor?
I mean we have to be realistic in that people will use cloud infrastructure, so it will start with the base infrastructure components they have access to.
> the code that developers write implicitly defines infrastructure. What your code actually needs in terms of infrastructure should be inferred as you build your application, instead of you having to think upfront about what infrastructure piece is needed and how to wire it up.
Your code runs somewhere. On something. In something. At a physical place, with physical cords running around to physical boxes of stuff. There is a finite amount of space at this physical place, meaning there are actual real-world limits that your code operates within. There is no unlimited scaling. There is no unlimited bandwidth, unlimited memory, unlimited CPU, unlimited connections. Your code operates within limits. On a server. In a datacenter. Often these limits are comically smaller than you might assume.
Because of this unfortunate air-quotes Real World air-quotes that your code is forced to grapple with, if you want your code to not crash, you will have to actually consider the implications of where, on what, and how it is running. Now, I know you may not care about the running of your application. You might give zero shits about anything outside of your IDE. But the business paying your six-figure paycheck to write this code, does care. The user who uses your code, does cares. So on behalf of these groups, I implore you in the strongest terms, to please start giving a crap about the infrastructure, operating system, container orchestrator, monitoring, etc.
Now. Am I saying you need to become an AWS expert? A Linux kernel expert? A Kubernetes expert? A tape robot, network switch, SAN, lights-off-management, power-distribution-unit expert?
Heck. No.
There is a middle ground. A place where you don't need to be an expert on everything your code depends on. But you can know something about it. Just enough to have a remote idea that your application will actually support more than 10 users, and work somewhere outside your laptop. For example:
- How much CPU and memory does your application need per request, or per user? You should find out. And you should make sure that whatever "thing" is running your code has enough CPU and memory to serve the maximum number of requests, or number of users, you want using your code. (You don't know what maximum number of requests/users your application is supposed to support? You should find that out too.) You might also want to find out what happens when these limits are exceeded, and whether your application can "scale up" in that event. And then - what happens when your application can't scale up.
- What maximum number of database connections do you want to be able to make? If the answer is "unlimited", that's... not an answer. If the answer is "2000", well, then you better check that the database can actually handle that many connections, because often it can't. And you better also check how many connections your application will actually use per # requests/users/number of instances of your app running.
- Do you know when your application stops working? And when it does, who is going to fix it? How will they do it, and how quickly? These things are directly related to that whole annoying "people depending on your application to keep running" thing I mentioned before. It's not a question of "if" your app stops running, but "when". So you should figure out how to monitor your application, so that you 1) know when it's "down" (can a user use all its functionality as expected?), 2) know approximately how to "make it work again", and 3) write up instructions for whomever has to "make it work again". You might do well to brainstorm about what things outside of your app might cause the app to stop working (disks, cpu, memory, network).
There is no way to infer all these things from just code. You need performance testing, and you need to know what scale this is su...
I believe the direction they are going is that you have it implicitely define a default infrastructure, and then if you want to customize it you can. This seems reasonable to me. For prototyping or MVP, just go with the defaults. As your product gains maturity you can hire domain experts to fine tune the infrastructure, while keeping it as separate from the business logic as possible
There is no such thing as default infrastructure if you care if your application works. Hiring "experts" later is really just hiring someone to do what the developer should have done, and will probably have to ask the dev to do more work anyway. To an Ops person it's literally a black box. To "fine-tune" the infrastructure, they have to interrogate the developer, and learn only the bare minimum of whatever app stack the developer decided to use. The end result is literally black box reverse engineering, which is wasteful and leads to bad outcomes.
Imagine being a fashion designer making clothes without ever seeing a human being put them on. There's fabric that covers skin, and there's arm holes and leg holes and buttons, so they are "feature-complete" clothes. But just because you pinned up a bunch of fabric on a dummy doesn't mean you know what a person looks like when they put it on and move around in it, or whether a person will want to wear them, or if they'll fall apart after a week. You have to know about different body shapes, how bodies move, evaluate their fit, adapt them to the wear and tear of the real world, and be interested in the outcome. You don't have to be a medical doctor, and you don't have to be an engineer, but you do need to know the absolute minimum of how those different domains interact with clothing.
The fact that people use Heroku, Firebase, Vercel, and other infrastructure providers means that there are plenty of developers out there that don't write their code around infrastructure at the beginning.
GAE was far worse from a UX perspective. As an outsider from Google, the handcuffs imposed by AppEngine we're severely restrictive compared to Lambda-like services.
I'm skeptical. Eventually you need to write things for which Rust is the wrong language. How do you get anything to play nice with each other, if you've turned your infrastructure into a magic box? What about preventing things from playing with each other (security)? DNS, WAF, caching, etc.? Are these all just Rust annotations?
> To reap the benefits of serverless, you have to build your application as a multitude of stateless request or event handlers, often requiring a bottoms-up redesign of your system. For some use-cases the serverless paradigm works, but in many cases breaking things into discrete, decoupled functions may not be optimal or even feasible. The next question is, can we have our cake and eat it too? Can we maintain the paradigm of stateful processes and abstract away the underlying infrastructure and orchestration?
There's a key distinction here that Shuttle seems to pave over. In microservices design, ideally each microservice entirely owns its own datastores; to interact with the datastore, outside actors must interact with the service and not directly with the datastore (enforced either by convention or firewalls). In serverless land, the use of single-purpose functions intentionally has many different architectural components (even if they are only the different functions themselves) interact with the underlying datastore. Therefore it is important for the underlying datastore to be distinct and managed separately.
Additionally, this passage has a rather important non-sequitur: if the problem with serverless is that it "[requires] a bottoms-up redesign of your system", then how exactly is Shuttle solving this problem when it puts the database into the magic box of things developers no longer need to worry about, rather than enabling developers to work with existing databases (because the entire paradigm is about starting from brown-field and not green-field)?
Right idea, wrong language. At least for me. Not really into Rust a lot and trying to figure it out gives me a headache. I've tried. Also, I think the severe risk here is that anyone who knows Rust well enough to like it for this would probably pull up their nose for devops jobs and find something more interesting to do with their skills.
But, I like the idea of using a statically compiled language rather than some yaml (terraform), json/xml cloudformation, etc.
I'm kind of liking Kotlin and Kotlin script right now (KTS). I might be biased because I write a lot of Kotlin. But it seems to have some unique features that make it suitable for developing your own DSLs.
BTW. I think Gradle's KTS support is actually a bad example of KTS because it inherits a lot of its complexity from when it was still exclusively a Groovy thing (which is of course a scripting language). The Gradle APIs tend to give me a headache. It's never obvious what you need to do. And it keeps on changing as well over time. Which makes Googling the solution really tedious because 95% of what you find is usually a combination of obsolete, wrong, and outdated. Automcomplete via Google is broken. And because Groovy is so dynamic, it also complicates autocomplete in Kotlin Script (which is a major selling point otherwise).
And autocomplete via Google is exactly what makes many solutions in the devops/PAAS/IAAS/configuration management etc. sphere so tedious to deal with. There's just no way around wading through lots of misinformation, product/marketing pages that try to tell you wonderful the shit you are trying to get working is, blog articles, poorly written/structured documentation, etc.
One of our people actually hacked together a little KTS script that generates our github action yaml files. If you've ever dealt with GH actions, they are nice but the yaml sure is fiddly. Having a KTS script that generates those is wonderful:
- A lot less errors that you discover after you commit because if it compiles, it's probably fine. Which means I can tinker with it, commit my changes, and have it work correctly the first time. Amazing.
- No more copy paste 'reuse'. We can have functions and reuse them across different workflows and we can actually reuse bits and pieces that we figured out. I'm always amazed at how much copy paste and adapt is still the leading paradigm in devops automation.
- No more dealing with yaml syntax, it's quirks, and limitations. Kotlin has types, DSL support, multi line strings, string templates, etc. In other words, it's very suitable for this kind of stuff. And when it doesn't have a feature you need, you can just add it because it's a programming language. Not that hard.
This should be a thing in more devops stuff. Terraform would probably be easier with KTS, for example. And I'm pretty sure people are working on that. Though of course the support in Intellij for Terraform is not bad.
I love this idea, i've been thinking about this for a few years under the pseudonym 'compile to cloud'. Would be really cool if I could write some code that uses a dictionary & threads which automatically get transposed to a dynamodb instance connected to a bunch of lambda nodes. I'll keep an eye on this project.
Infrastructure from code is great, when your CI system generates the production diff and adds it to the code review. Then it's clear exactly what will get deleted. The deployment process also needs to be restartable and steps skippable. So for example, when a migration fails, you can perform it manually, re-deploy and skip the broken step. So far, no infrastructure-as-code tool works like this.
I know of only two tools that work effectively: Terraform & Pulumi. Terraform's config language has major flaws and Hashicorp is not interested in fixing them. For example, one cannot use a single Terraform config to create a database instance and a database inside it, or a VM and a server inside it, etc. So every application requires multi-stage configs which are full of footguns. Pulumi supports proper languages like Python & Golang and does not have this problem.
I would like to see a Rust version of Pulumi that uses static typing to eliminate run-time errors in infrastructure config. Maybe Shuttle will become that? It looks like they're integrating the web server with the infrastructure code. I think they should go in the opposite direction and make everything modular (usable separately) and composable. I want to write `deploy.rs` and run `cargo deploy`.
> Pulumi supports proper languages like Python & Golang and does not have this problem.
Having battled with Chef I can say with quite a bit of vim and vigor: noooooo. Everybody loves the idea of having a Turing complete language at their disposal until they manage to blow both their feet off one toe at a time.
I was going to say that Hashicorp made the deliberate design choice to avoid putting a bunch of conditional logic into HCL. However:
43 comments
[ 3.3 ms ] story [ 95.2 ms ] threadActually we do[0]! You can deploy a shuttle app with an annotation and a single cargo command -> `cargo shuttle deploy` :)
[0] https://www.shuttle.rs/
I'm probably extremely biased (I'm a sysadmin), but this feels like a terrifying way to make your infrastructure entirely implicit and remove anybody's ability to coherently reason about the stuff underneath your application.
I'm not entirely averse to magic infrastructure beans but having an intermediate step where things are explicitly spelled out would make me far less nervous. Using shuttle to spit out a TF file – cool, using shuttle to automagically spin stuff up – terrifying.
This exact problem is my main beef with an unfortunately common and popular style of functional programming : stuff so condensed, so abstracted and so focused on describing the problem all the while hiding the actual execution / implementation / solution layers it's quasi-impossible to reason about and understand what actually happens when the program runs.
> this feels like a terrifying way to make your infrastructure entirely implicit and remove anybody's ability to coherently reason about the stuff underneath your application.
There's two parts to this. The first is understanding what is happening under the hood since we're automatically provisioning so much for you (subdomain, LBs, DBs, etc.). Right now this is a little opaque, but a dashboard is in the roadmap to give users visibility on what's happening 'under the hood' - showing your provisioned infrastructure as well as how it's all wired up.
The second point is about potentially destructive actions - here we're going to be following a 'terraformesque' philosophy where infrastructure diffs are presented to the user and need to be accepted explicitly when deploying (or via a `--auto-approve` flag).
source: I didn't attend Harvard, but I worked for their IT department beginning in 2005. Several of my coworkers had been directly involved in the disciplinary action against Zuckerberg and were very familiar with the details. That said, it is entirely possible I am misremembering. But other accounts seem to agree, e.g. https://www.quora.com/How-did-Mark-Zuckerberg-initially-host...
Thanks for this! I found a different source[0] but I'll make the update shortly!
[0] https://qr.ae/pvAGDH
Infrastructure from code seems convenient for side-projects, but absolutely awful for something big where your infrastructure is now implicit in your code??
That DB still needs tables created and test data loaded.
Plus, having to deploy code from the ticket just to be able to reproduce a bug seems like a huge PITA. Give me the local execution!
> Plus, having to deploy code from the ticket just to be able to reproduce a bug seems like a huge PITA. Give me the local execution!
Ideally the code could be deployed automatically -- perhaps only from trusted/internal parties depending on the product -- so you don't have to actually do that much work. Less than you would if you had to create a database with the specific flags and environment exactly matching those of the reporter.
I dunno, I dream of this sort of reproducible bug report. I don't like the "what version" "what queries" "what errors" back and forth that always seems to happen, even with internal users.
Your code runs somewhere. On something. In something. At a physical place, with physical cords running around to physical boxes of stuff. There is a finite amount of space at this physical place, meaning there are actual real-world limits that your code operates within. There is no unlimited scaling. There is no unlimited bandwidth, unlimited memory, unlimited CPU, unlimited connections. Your code operates within limits. On a server. In a datacenter. Often these limits are comically smaller than you might assume.
Because of this unfortunate air-quotes Real World air-quotes that your code is forced to grapple with, if you want your code to not crash, you will have to actually consider the implications of where, on what, and how it is running. Now, I know you may not care about the running of your application. You might give zero shits about anything outside of your IDE. But the business paying your six-figure paycheck to write this code, does care. The user who uses your code, does cares. So on behalf of these groups, I implore you in the strongest terms, to please start giving a crap about the infrastructure, operating system, container orchestrator, monitoring, etc.
Now. Am I saying you need to become an AWS expert? A Linux kernel expert? A Kubernetes expert? A tape robot, network switch, SAN, lights-off-management, power-distribution-unit expert?
Heck. No.
There is a middle ground. A place where you don't need to be an expert on everything your code depends on. But you can know something about it. Just enough to have a remote idea that your application will actually support more than 10 users, and work somewhere outside your laptop. For example:
- How much CPU and memory does your application need per request, or per user? You should find out. And you should make sure that whatever "thing" is running your code has enough CPU and memory to serve the maximum number of requests, or number of users, you want using your code. (You don't know what maximum number of requests/users your application is supposed to support? You should find that out too.) You might also want to find out what happens when these limits are exceeded, and whether your application can "scale up" in that event. And then - what happens when your application can't scale up.
- What maximum number of database connections do you want to be able to make? If the answer is "unlimited", that's... not an answer. If the answer is "2000", well, then you better check that the database can actually handle that many connections, because often it can't. And you better also check how many connections your application will actually use per # requests/users/number of instances of your app running.
- Do you know when your application stops working? And when it does, who is going to fix it? How will they do it, and how quickly? These things are directly related to that whole annoying "people depending on your application to keep running" thing I mentioned before. It's not a question of "if" your app stops running, but "when". So you should figure out how to monitor your application, so that you 1) know when it's "down" (can a user use all its functionality as expected?), 2) know approximately how to "make it work again", and 3) write up instructions for whomever has to "make it work again". You might do well to brainstorm about what things outside of your app might cause the app to stop working (disks, cpu, memory, network).
There is no way to infer all these things from just code. You need performance testing, and you need to know what scale this is su...
Imagine being a fashion designer making clothes without ever seeing a human being put them on. There's fabric that covers skin, and there's arm holes and leg holes and buttons, so they are "feature-complete" clothes. But just because you pinned up a bunch of fabric on a dummy doesn't mean you know what a person looks like when they put it on and move around in it, or whether a person will want to wear them, or if they'll fall apart after a week. You have to know about different body shapes, how bodies move, evaluate their fit, adapt them to the wear and tear of the real world, and be interested in the outcome. You don't have to be a medical doctor, and you don't have to be an engineer, but you do need to know the absolute minimum of how those different domains interact with clothing.
This is older than author realizes. Google App Engine was doing this in 2008, 6 years before Lambda.
> To reap the benefits of serverless, you have to build your application as a multitude of stateless request or event handlers, often requiring a bottoms-up redesign of your system. For some use-cases the serverless paradigm works, but in many cases breaking things into discrete, decoupled functions may not be optimal or even feasible. The next question is, can we have our cake and eat it too? Can we maintain the paradigm of stateful processes and abstract away the underlying infrastructure and orchestration?
There's a key distinction here that Shuttle seems to pave over. In microservices design, ideally each microservice entirely owns its own datastores; to interact with the datastore, outside actors must interact with the service and not directly with the datastore (enforced either by convention or firewalls). In serverless land, the use of single-purpose functions intentionally has many different architectural components (even if they are only the different functions themselves) interact with the underlying datastore. Therefore it is important for the underlying datastore to be distinct and managed separately.
Additionally, this passage has a rather important non-sequitur: if the problem with serverless is that it "[requires] a bottoms-up redesign of your system", then how exactly is Shuttle solving this problem when it puts the database into the magic box of things developers no longer need to worry about, rather than enabling developers to work with existing databases (because the entire paradigm is about starting from brown-field and not green-field)?
But, I like the idea of using a statically compiled language rather than some yaml (terraform), json/xml cloudformation, etc.
I'm kind of liking Kotlin and Kotlin script right now (KTS). I might be biased because I write a lot of Kotlin. But it seems to have some unique features that make it suitable for developing your own DSLs.
BTW. I think Gradle's KTS support is actually a bad example of KTS because it inherits a lot of its complexity from when it was still exclusively a Groovy thing (which is of course a scripting language). The Gradle APIs tend to give me a headache. It's never obvious what you need to do. And it keeps on changing as well over time. Which makes Googling the solution really tedious because 95% of what you find is usually a combination of obsolete, wrong, and outdated. Automcomplete via Google is broken. And because Groovy is so dynamic, it also complicates autocomplete in Kotlin Script (which is a major selling point otherwise).
And autocomplete via Google is exactly what makes many solutions in the devops/PAAS/IAAS/configuration management etc. sphere so tedious to deal with. There's just no way around wading through lots of misinformation, product/marketing pages that try to tell you wonderful the shit you are trying to get working is, blog articles, poorly written/structured documentation, etc.
One of our people actually hacked together a little KTS script that generates our github action yaml files. If you've ever dealt with GH actions, they are nice but the yaml sure is fiddly. Having a KTS script that generates those is wonderful:
- A lot less errors that you discover after you commit because if it compiles, it's probably fine. Which means I can tinker with it, commit my changes, and have it work correctly the first time. Amazing.
- No more copy paste 'reuse'. We can have functions and reuse them across different workflows and we can actually reuse bits and pieces that we figured out. I'm always amazed at how much copy paste and adapt is still the leading paradigm in devops automation.
- No more dealing with yaml syntax, it's quirks, and limitations. Kotlin has types, DSL support, multi line strings, string templates, etc. In other words, it's very suitable for this kind of stuff. And when it doesn't have a feature you need, you can just add it because it's a programming language. Not that hard.
This should be a thing in more devops stuff. Terraform would probably be easier with KTS, for example. And I'm pretty sure people are working on that. Though of course the support in Intellij for Terraform is not bad.
You can’t readily build infrastructure from Code in a way that allows for solid uptime, redundancy, resiliency, or performance.
It sounds like a system like this would be fine for testing and possibly QA but definitely not for production.
I know of only two tools that work effectively: Terraform & Pulumi. Terraform's config language has major flaws and Hashicorp is not interested in fixing them. For example, one cannot use a single Terraform config to create a database instance and a database inside it, or a VM and a server inside it, etc. So every application requires multi-stage configs which are full of footguns. Pulumi supports proper languages like Python & Golang and does not have this problem.
I would like to see a Rust version of Pulumi that uses static typing to eliminate run-time errors in infrastructure config. Maybe Shuttle will become that? It looks like they're integrating the web server with the infrastructure code. I think they should go in the opposite direction and make everything modular (usable separately) and composable. I want to write `deploy.rs` and run `cargo deploy`.
Having battled with Chef I can say with quite a bit of vim and vigor: noooooo. Everybody loves the idea of having a Turing complete language at their disposal until they manage to blow both their feet off one toe at a time.
I was going to say that Hashicorp made the deliberate design choice to avoid putting a bunch of conditional logic into HCL. However:
https://www.terraform.io/cdktf