Launch HN: Depot (YC W23) – Fast Docker Builds in the Cloud
Building Docker images in CI today is slow. CI runners are ephemeral, so they must save and load the cache for every build. They have constrained resources, with limited CPUs, memory, and disk space. And they do not support native Arm or multi-platform container builds, and instead require emulation.
Over 4 years of working together, we spent countless hours optimizing and reoptimizing Dockerfiles, managing layer caching in CI, and maintaining custom runners for multi-platform images. We were working around the limitation of multi-platform builds inside of GitHub Actions via QEMU emulation when we thought "wouldn't it be nice if someone just offered both an Intel and Arm builder for Docker images without having to run all that infrastructure ourselves". Around January of 2022 we started working on Depot, designed as the service we wished we could use ourselves.
Depot provides managed VMs running BuildKit, the backing build engine for Docker. Each VM includes 16 CPUs, 32GB of memory, and a persistent 50GB SSD cache disk that is automatically available across builds—no saving or loading of layer cache over the network. We launch both native Intel and native Arm machines inside of AWS. This combination of native CPUs, fast networks, and persistent disks significantly lowers build time — we’ve seen speedups ranging from 2x all the way to 20x. We have customers with builds that took three hours before that now take less than ten minutes.
We believe that today we are the fastest hosted build service for Docker images, and the only hosted build service offering the ability to natively build multi-platform Docker images without emulation.
We did a Show HN last September: https://news.ycombinator.com/item?id=33011072. Since then, we have added the ability to use Depot in your own AWS account; added support for Buildx bake; increased supported build parallelism; launched an eu-central-1 region; switched to a new mTLS backend for better build performance; simplified pricing and added a free tier; and got accepted into YC W23!
Depot is a drop-in replacement for `docker buildx build`, so anywhere you are running `docker build` today, you replace it with `depot build` and get faster builds. Our CLI is wrapping the Buildx library, so any parameters you pass to your Docker builds today are fully compatible with Depot. We also have a number of integrations that match Docker integrations inside of CI providers like GitHub Actions.
We’re soon launching a public API to programmatically build Docker images for companies that need to securely build Docker images on behalf of their customers.
You can sign up at https://depot.dev/sign-up, and we have a free tier of 60 build minutes per month. We would love your feedback and look forward to your comments!
96 comments
[ 4.1 ms ] story [ 149 ms ] threadIn the end I created a hacky build process where I use a single container to build both the x64 and ARM versions serially, and then create multi-arch containers in a separate step. It was very painful to get the right native libraries installed, and it's not terribly easy to build these two platforms in parallel.
In short, having access to real ARM builders would be great, and persistent disks would probably boost my build performance quite a bit.
The dockerfile that I had to use: https://github.com/mmastrac/progscrape/blob/master/Dockerfil...
Example build run (~20 mins): https://github.com/mmastrac/progscrape/actions/runs/42285298...
Couldn't agree more with the pain points you mention, it was the biggest things that led us to start Depot and we're really excited about what we can do next.
If you ever want to do way with your hacky build process and try out Depot, we have a free tier now.
We've been using Depot with Plane (https://plane.dev/). Prior to depot, I had to disable arm64 builds because they slowed the build down so much (30m+) on GitHub's machines. With Depot, we get arm64 and amd64 images in ~2m.
Sorry to ask this silly question, but since your team is an expert in the "fast Docker images" area, could somebody avoid the traditional `docker build` with say NixOS or Bazel and achieve the same results as Depot (aka, fast building with the output being an OCI/Docker image)? Is that what Depot is doing at a high level? Was this considered?
> Our CLI is wrapping the Buildx library
I'm surprised you're able to build Docker images faster than Docker using their code/libraries?
It's not a code/library problem. Knowing what Buildkit options to use is the easy part. It's almost entirely a storage infrastructure and networking problem as it has huge implications on whether or not you'll be able to easily cache build layers.
On a single machine, would NixOS/Bazel handle this better than Dockerfile/Docker/BuildKit?
I've also seen people use Nix or Bazel inside their Dockerfile, like ultimately the build has to execute somewhere, be that inside or outside a Dockerfile.
[1] https://github.com/nlewo/nix2container
Yes! You can think of an OCI image as a special kind of tarball, so things like NixOS and Bazel are able to construct that same tarball, potentially fairly quick if it just has to copy prebuilt artifacts from the store.
Today we're running BuildKit, so we support all the typical Docker things as well as other systems that use BuildKit, e.g. Dagger, and I believe there are nix frontends for BuildKit. In that sense, we can be an accelerated compute provider for anything compatible with BuildKit.
> build Docker images faster than Docker
Today the trick is in the hosting and orchestration. We're using fast machines, launching Graviton instances for Arm builds (no emulation) or multiple machines for multi-platform build requests, orchestrating persistent volumes, etc. It's more advanced than what hosted CI providers give you today, and closer to something you'd need to script together yourself with your own runners. There's also some Docker build features (e.g. cache mounts) that _only_ work with a persistent disk.
But there is a few steps to get there in that setup. I believe you have to have three CodeBuild projects, one for each architecture & then the manifest merge. So it works but is a bit of config to stitch together.
With Depot, you would just install our `depot` CLI in your config and run `depot build --platform linux/amd64,linux/arm64` instead of `docker build`. We handle building the image on both architectures in parallel and can push the merged result to your registry. We can even run the builders in your own AWS account so you maintain total control of the underlying infra building your image.
We are working on other features for Depot that would go beyond the speed & multi-platform capabilities. We want to surface insights and telemetry about image builds that could help them be smaller, faster, and smarter. We are also thinking about things in the container security space such as container signing, sboms, etc. Happy to answer more questions about any of this!
What happens when GitHub adds native arm support though? Seems like a big value add of your service is immediately displaced and additionally can use self-hosted runners with GitHub to solve caching.
That will make it much faster to build arm images on GHA natively - in that scenario, Depot should still be several times faster like we are on Intel today, primarily due to how we're managing layer cache to avoid needing to save it or load it between builds (cache-to / cache-from), as well as just having larger runners and more sophisticated orchestration. We can take advantage of BuildKit's ability to share cache and deduplicate steps between concurrent builds for instance.
We're also expanding Depot in a few different directions, including along the security path with container signing and SBOM support, as well as some upcoming build insights and analytics features. The goal is that it's always super easy to `depot build` from wherever you `docker build` today, and that Depot provides the best possible experience for container builds.
But the pricing hurts for people that need more than 60 minutes of build time (which is pretty easy to go through in one month) but would be using it for personal projects like myself.
I could certainly see myself paying X/min over 60, but $49+5c/min for personal stuff is a hard no.
For now, if you want to sign up for our free tier we can flip your org to this type of structure you are talking about, 1 project, 1 user, first 60 build minutes free & 5 cents/minute after that.
They're incredibly knowledgeable about the subject and are making amazing strides for build speeds. I'd encourage anyone who doesn't believe these results and benchmarks to just try it out. They're completely real and it's delightful.
What's the issue with https://docs.docker.com/build/building/multi-platform/? I only just learned about this today but I've already got it building cross-platform images correctly in Github Actions.
The difference between Depot and doing it directly in GitHub Actions is the "native CPU" part - on GitHub Actions, the Arm builds will run with emulation using QEMU. It works, but it's often at least 10x slower than it could be if the build was running on native Arm CPUs. Builds that should take a few minutes can take hour(s).
For multi-platform, if you want to keep things fast, you need a simultaneous connection to both an Intel and an Arm machine, and the two work in concert to build the multi-platform image.
There are workarounds with emulation, or pushing individual images separately and merging the result after they're in a registry. But if you just want to `build --platform linux/amd64,linux64` and it be fast, we handle all the orchestration for that.
If you're already building images (because that's what we're talking about) what difference does it make at which base image you start?
>...persistent disks significantly lowers build time
Does this mean your solution places specific caches, like bazel, node_modules, .yarn, and other intermediary artifacts onto a shared volume and reuses them among jobs?
Now as an end user you still have to add this to your Dockerfile, but if subsequent builds are able to continually use this cache, build times will drastically improve.
- [0] https://docs.docker.com/build/cache/
One of the reasons we created Depot is that cache mounts aren't supported in GitHub Actions, since each CI run is entirely ephemeral, so the files saved in a cache mount aren't saved across runs. BuildKit doesn't export those cache types via cache-to. There are some manual workarounds creating tarballs of the BuildKit context directory, but we wanted something that just works without needing to save/load tarballs, which can be quite slow.
1. If you need an uncached individual build, you can pass the `--no-cache` flag to build everything from scratch
2. We have a "clear all cache" button in our UI and CLI command, to wipe the entire project's cache disk.
Today we're using vertical scaling, so we run BuildKit on larger EC2 instances, tune `max-parallelism`, and let BuildKit handle processing multiple builds / deduplicating steps across builds / etc. What our website calls a "project" basically equates to a new EC2 instance (so two different projects are fully isolated from each other).
We'd like to expand into horizontal scaling, probably with some kind of tiered caching, so the builders would be able to use local cache on local SSDs if available, but fall back to copying cache from S3 if not available locally.
We have plans to expand to more horizontal scaling with tiered caching, so we can keep the speedups we see today but further increase potential parallelism.
You say that this is meant to solve problems with CI being ephemeral. Maybe I'm old fashioned, but my own CI cluster uses dedicated hardware and nothing is ephemeral. It could also use dedicated VMs and the same thing would apply. We run TeamCity and the agents are persistent, builds run in dedicated workspaces so caching is natural and easy. This doesn't cost very much.
When you add more features then I can see there's some value there (SBOM etc) but then again, surely such features are more easily done by standalone tools rather than having to rent infrastructure from you.
We do charge $0.05 per minute of build time used, but in theory that $49/mo plan gives you access to up to 20 build machines, if you're building 10 projects at once.
That said, if you already have your own dedicated build cluster / CI setup, you may prefer to just use that! Depot is effectively doing that kind of thing for you if you don't already have your own hosted CI system or would prefer not to orchestrate Docker layer cache.
We will be expanding to more things like SBOMs, container signing, insights and analytics about what's happening inside the builds, but hopefully in more integrated ways, since we control the execution environment itself.
Clearly from other responses in this thread there are people who feel this is a good deal, so best of luck to you. But I'm kinda reminded here of 37signals saying they can save $7M over 5 years by leaving the cloud. It seems the goal here is to dig people out of performance problems they get by using one type of cloud service, by selling them another type of cloud service!
Rounding error for most companies.
>The extra cost is a bit of Linux sys admin which can be fully automated
You are overweighting hard dollar costs and underweighting the value of engineering time. Maybe you're the worlds greatest devops/platform engineer/sysadmin and once you wire up everything in under 5 minutes it will never need maintenance ever again but for most everyone else speeding up image builds by using a service that someone else thinks about and does maintenance on is absolutely worth it for $170/mo.
On the other hand, I've experienced first hand how cloud costs can explode uncontrollably in absurd ways. One company I worked at had a cloud cost crisis and they weren't even serving online services, just shovelling money into Azure for generic dev services like VMs for load tests, DBs for testing, super-slow CI agents, etc. They never managed to properly fix this because of the mentality you express here: a few hundred bucks a month here, a few hundred there, everyone gets access to spin up resources and it's all worth it because we're all soooo valuable. Then one day you realize you're inexplicably burning millions on subscription services and cloud spend, yet nobody can identify quite why or on what, or how to push costs down. Death by a thousand cuts, it was quite the revelation. Free cloud credits are murder, because they embed a culture of profligacy and "my time is too valuable to optimize this". By the time the startup credits run out it's too late.
Having network issues due to slow async calls? Just increase instance size until the machine is so fast it completes everything before it becomes a problem. Now you are paying 10x more for something that’s a few hours of dev time.
At a bigger company you'd just ask a junior to set it up or maybe a sysadmin. You can probably contract to get part timers too. I guess I spent half a day or so setting up the CI cluster at the start and have barely touched it since. That wasn't much cost even if I was earning a big salary. Our builds can use caches to speed them up (not docker, other types of cache) and when we turned that on it was like a 3x speed win, so having persistent disks is definitely worth it for many types of program. Especially if you're brave and trust your build system to cache unit test results between builds!
Just paying extra for all the above to magically give you faster builds is easy, fast, predictable, guaranteed benefits. Totally worth it.
If you need a lot of milk every year, you could buy a cow, or you could just buy milk at the store. Most people agree the extra cost is worth it.
The alternative here is having CI workers that use regular disks that aren't wiped between builds, and constraining the jobs that benefit from caching to run on them. If you have CI already set up it shouldn't be that hard? People are acting like you need to pay a L7 Staff SWE $20k to set this stuff up, is that really a cost problem many companies are facing? It'd have been considered junior level stuff not so long ago.
As for most people feeling it's worth it, I dunno man. Yes if I read the comments people seem keen, but my post is voted to the top of this thread. It feels like a lot of people have doubled down on functional-programming like approach to server management and now have really slow infrastructure that spends lots of time doing redundant work and then throwing it away. Yes, you can do that and it even has some advantages, like FP sometimes does, but you can also just accept that computers are under the hood stateful mutable machines and set up things to lean into that. That's what this startup is basically doing, right? Just seems like a 21st century problem somehow.
Just the act of running a single VM - to do it right - requires technical expertise (just because you find it easy doesn't mean it is, or that most people would do it right), in addition to maintenance tasks, operational overhead, etc. Deal with the extra infra costs in your corp cloud budget, write the extra software to handle advanced caching on auto scaling instances on multiple platforms, understand how Docker works under the hood (far fewer people know that than you assume)... It is extra work someone has to do that has no bearing on what someone actually wants to be doing, which is just running a Docker build faster.
I would have to assign two engineers to build and maintain this for a medium sized company, at $120K per employee, plus infra cost, plus maintenance, plus the lead time to build it, etc. And they'd probably do a crap job.
So, pay $50 a month for a working solution right now? To increase velocity of sw development, with no other changes? Sign my ass up. It's a tragedy that people don't understand the value here.
Yeah that's why I'm asking. It's genuine curiosity so thanks for your answers.
Yes if you wanted to make the same product as these guys you'd have to spend the same amount or more, so sure, that'd be a poor use of money. No disagreement there. Productization is a lot of work.
But you wouldn't need a full product to solve this for your own use case! I guess what I'm struggling with is the apparent reluctance to fix this problem by just running ordinary computers. We're hackers, we're software developers, this is our bread and butter right? How can we as an industry apparently be forgetting how to set up and run computers? That's the message that seems to be coming through here - it's too hard, too much work, the people who can do it are too expensive. That'd be like chemists forgetting how to use a bunsen burner and needing to outsource it!? Computers are cheap, they're fast, they can basically maintain themselves if told to! To make your Docker builds faster you can just run them on a plain vanilla Linux machine that just sits around running builds in the normal way, the same way a laptop would run them, with permanently connected disk and cache directory and stuff.
I totally get it that maybe a new generation has learned programming with NodeJS on a Mac and AWS, maybe they haven't ever installed Linux before, in the way we all seemed to learn how to run servers a couple of decades ago. Times change, sure, I get that. Still, the results are kind of mind boggling.
Especially if you want/need multi-arch. That basically requires buildx which doesn’t cache locally by default. There’s a half dozen types of caching to figure out. Then buildx is very buggy and needs qemu setup even when building natively otherwise you run into decade old bugs doing things like running sudo in a dockerfile.
It took a couple of weeks of on and off tinkering to get a stable arm builder running on a Mac m1. To get the GitHub action server to run stably and not time out was a PITA. It required IT tuning cpu limits and page caching. Not fun.
We run native machines but I would’ve much preferred a cloud solution so I could do my actual job.
For servers I'd have thought you'd make a decision up front about whether to use classical x64 machines or ARM, based on load tests or cost/benefit analysis. Then you'd build one or the other. It sounds like a lot of people are putting a lot of effort into the optionality of having both, and then they are using languages and tools that can't cross-compile or JIT compile. Are you using Rust or Go or something? Hmmm.
We've been using Depot for the past two months, and without changing anything the builds became faster (compared to our CI).
Good luck Depot team and keep up the good work!