Show HN: Managed GitHub Actions Runners for AWS
Here's a video demo: https://www.youtube.com/watch?v=VX5Z-k1mGc8, and here’s our blog post: https://depot.dev/blog/depot-github-actions-runners.
While GitHub Actions is one of the most prevalent CI providers, Actions is slow, for a few reasons: GitHub uses underpowered CPUs, network throughput for cache and the internet at large is capped at 1 Gbps, and total cache storage is limited to 10GB per repo. It is also rather expensive for runners with more than 2 CPUs, and larger runners frequently take a long time to start running jobs.
Depot-managed runners solve this! Rather than your CI jobs running on GitHub's slow compute, Depot routes those same jobs to fast EC2 instances. And not only is this faster, it’s also 1/2 the cost of GitHub Actions!
We do this by launching a dedicated instance for each job, registering that instance as a self-hosted Actions runner in your GitHub organization, then terminating the instance when the job is finished. Using AWS as the compute provider has a few advantages:
- CPUs are typically 30%+ more performant than alternatives (the m7a instance type).
- Each instance has high-throughput networking of up to 12.5 Gbps, hosted in us-east-1, so interacting with artifacts, cache, container registries, or the internet at large is quick.
- Each instance has a public IPv4 address, so it does not share rate limits with anyone else.
We integrated the runners with the distributed cache system (backed by S3 and Ceph) that we use for Docker build cache, so jobs automatically save / restore cache from this cache system, with speeds of up to 1 GB/s, and without the default 10 GB per repo limit.
Building this was a fun challenge; some matrix workflows start 40+ jobs at once, then requiring 40 EC2 instances to launch at once.
We’ve effectively gotten very good at starting EC2 instances with a "warm pool" system which allows us to prepare many EC2 instances to run a job, stop them, then resize and start them when an actual job request arrives, to keep job queue times around 5 seconds. We're using a homegrown orchestration system, as alternatives like autoscaling groups or Kubernetes weren't fast or secure enough.
There are three alternatives to our managed runners currently:
1. GitHub offers larger runners: these have more CPUs, but still have slow network and cache. Depot runners are also 1/2 the cost per minute of GitHub's runners.
2. You can self-host the Actions runner on your own compute: this requires ongoing maintenance, and it can be difficult to ensure that the runner image or container matches GitHub's.
3. There are other companies offering hosted GitHub Actions runners, though they frequently use cheaper compute hosting providers that are bottlenecked on network throughput or geography.
Any feedback is very welcome! You can sign up at https://depot.dev/sign-up for a free trial if you'd like to try it out on your own workflows. We aren't able to offer a trial without a signup gate, both because using it requires installing a GitHub app, and we're offering build compute, so we need some way to keep out the cryptominers :)
58 comments
[ 4.1 ms ] story [ 118 ms ] threadWe have some docs on this for our container builder product - still need to write the docs for Actions runners too, though they use the same underlying system: https://depot.dev/docs/self-hosted/overview.
To be clear, this is a genuine question, as compute (even when efficiently orchestrated and arbitraged) is a commodity. Your cache strategy is good (will be interested in testing to tease out where is S3 and where is Ceph), but not a moat and somewhat straightforward to replicate.
(again, questions from a place of curiosity, nothing more)
GitHub's incentives and design constraints are different than ours. GitHub needs to offer something that covers a very large user-base, to cover the widest possible number of workflows, and they've done this by offering basic ephemeral VMs on-demand. CI and builds are also not GitHub's primary focus as an org.
We're trying to be the absolute fastest place to build software, with a deep focus on achieving maximum performance and reducing build time as much as possible (even to 0 with caching). Software builds today are often wildly inefficient, and I personally believe there's an opportunity to do for build compute what has been done for application compute over the last 10 years.
GitHub Actions workflows are more of an "input" for us then (similar to how container image builds have been), with the goal of adding more input types over time and applying the same core tech to all of them.
Wishing you much success.
They might simply charge for everything working out of the box convenience. Or even for not being aware there are other options.
It is very hard for a single player to get favourable treatment from Azure / AWS / GCP to handle many thousands of jobs every day / hour.
I wish Depot all the luck, I think they've done good work wrt caching.
- Depot runners are hosted in AWS us-east-1, which has implications for network speed, cache speed, access to internet services, etc. (BuildJet is hosted in Europe - maybe Hetzner?)
- Also thanks to AWS: each runner has a dedicated public IP address, so you're not sharing any third-party rate limits (e.g. Docker Hub) with other users
- We have an option to deploy the runners in your own AWS account or VPC-peer with your VPC
- We're integrating Actions runners with the acceleration tech we've built for container builds, starting with distributed caching
I couldn't find it anywhere on the page, but do you support Graviton3 (i.e. m7g instances) for GHA Runners? If the answer is no, are there any plans to support it in the future?
> start them when an actual job request arrives, to keep job queue times around 5 seconds
Did you have to fine-tune Ubuntu kernel/systemd boot to reach such fast startup times?
This 5 seconds might be the warm start, not cold. I.e. they likely have a pool of autoscaled, multi tenant workers
The challenge with Arm is actually just that GitHub doesn't have a runner image defined for Arm. For the Intel runners, we build our image directly from GitHub's source[0], and we're doing the same for the Arm runners by patching those same Packer scripts for arm64. It also looks like some popular actions, like `actions/setup-*`, don't always have arm support either.
So the disclaimers for launching Depot `-arm` instances at the moment is basically just (1) we have no idea if our image is compatible with your workflows, and (2) those instances take a bit longer to start.
On achieving fast startup times, it's a challenge. :) The main slowdown that prevents a <5s kernel boot is actually EBS lazy-loading the AMI from S3 on launch.
To address that at the moment, we do keep a pool of instances that boot once, load their volume contents, then shutdown until they're needed for a job. That works, at the cost of extra complexity and extra money - we're experimenting some with more exotic solutions now though like netbooting the AMI. That'll be a nice blog post someday I think.
[0] https://github.com/actions/runner-images/tree/main/images/ub...
https://github.com/philips-labs/terraform-aws-github-runner
With Depot, we're moving towards deeper performance optimizations and observability than vanilla GitHub runners - we've integrated the runners with a cache storage cluster for instance, and we're working on deeper integration with the compute platform that we built for distributed container image builds - as well as expanding the types of builds we can process beyond Actions and Docker, for instance.
But different options will be better for different folks, and the `philips-labs` project is good at what it does.
Definitely linking OP to my team now.
1. Launch, prepare basic software, shut down
2. A GitHub job request arrives at Depot
3. The job is assigned to the stopped VM, which is then started
4. The job runs on the VM completes
5. The VM is terminated
So the pool exists to speed up the EC2 instance launch time, but the VMs themselves are both single-tenant and single-use.
do you actually get the promised 12.5 Gbps? I've been doing some experiments and it's really hard to get over 2.5Gbit/s upstream from AWS EC2, even when using large 64 vCPU machines. Intra-AWS (e.g. VPC) traffic is another thing and that seems to be ok.
- For instances with >= 32 vCPUs, traffic to an internet gateway can use 50% of the throughput
- For instances with < 32 vCPUs, traffic to an internet gateway can use 5 Gbps
- Traffic inside the VPC can use the full throughput
So for us, that means traffic outbound to the public internet can use up to 5 Gbps, but for things like our distributed cache or pulling Docker images from our container builders, we can get the full 12.5 Gbps.
[0] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-inst...
with that pull quote, I thought you were going to point out their use of us-fail-1. I struggle to think of a service that I care so little about its availability that I'd host it there, but CI/CD for sure wouldn't be one
Are you building your base image from the GitHub runner-images repo?
Do you have any appetite for building self hosted EC2 agents for Azure DevOps and GitHub?
I'm happy to help if you are, I'm working on something similar myself for my employer.
Overall it’s pretty simple terraform setup plus a couple dockerfiles. And we get to run in the same region as the rest of our infra that’s close to most of our devs (us-west-2).
ECS might sound more complicated than “just use ec2” but we don’t have to screw around with lambdas and the terraform is pretty simple, much simpler then the Philips-labs one. It’s about 1400 lines of Terraform across 2 files since ECS has so much stuff built in and integrates with auto scale groups well.
Also curious how much y'all isolate it from your other infra. I've thought about this but I've been torn on whether I'd set up a separate vpc for it.
A managed runner means not only entrusting a third party with your code but also typically providing it with enough data/network connectivity to make testing/validation feasible as a part of the build process. While this is doable per se, it introduces multiple major failure modes outside of data owners' control.
Failure scenario (hypothetical): you hydrate your test DB using live data; you store it in a dedicated secure S3 bucket, which you make accessible for the build process. Now the managed runner organization gets hacked because making resilient infra is hard, and the attackers intercept the S3 credentials used by your build process. Boom! Your live data is now at the mercy of the attackers.
But besides just compute, I think the bigger long-term unlock for build performance is a new distributed compute engine, to free build workloads from single machines. We've started building this for our container build product, and plan to integrate Actions jobs as an input as well, starting with the cache integration we have today.
The problem is actually really that GitHub’s caching offering is very limited for anything except the most basic of use cases and also they don’t offer a way to colo your own cache with them so that you aren’t paying cloud fees back and forth. You have to use their machines, their storage and their protocol which is only really viable if your definition of caching is literally just “upload files here” and “check if the uploaded built file already exists”.
Yes, I’m aware that buildkit offers “experimental” GHA caching support. But given how fat image layers are it’s basically unusable for anything beyond a toy project that builds a couple layers on top of an alpine image (as of the time of writing this post GHA limits cache size to 10gb per repo. Fine if you’re building npm or pypi packages or whatever, but hilariously inadequate for buildkit layer caching)
Though the site (depot.dev) focuses on that aspect, this post doesn't.
@jacobwg - Do the runners in AWS get the same docker caching performance as depot.dev hosted runners?
I'd like to have a more automatic integration at some point - the challenge is that a lot of BuildKit's architecture performs best when many different build requests all arrive at a single build host, it is then able to efficiently deduplicate and cache work across all those build requests. So you really want the many different Actions jobs all communicating with the same BuildKit host.
We have some ideas for reducing the amount of change to Actions workflows to adopt ^ - longer term we're also working on our own build engine, to free those workloads from being confined to single hosts (be that single CI runners or single container builders).
https://runs-on.com/reference/caching/ https://runs-on.com/features/s3-cache-for-github-actions/
Edit: I see. This solution you linked doesn’t use GHA hosted runners at all - it’s intended to be a turnkey self hosted runner solution. In other words, a direct competitor to the service linked in OP. That wasn’t super clear from your comment but after reading your links it is more clear. I do really like the pricing here, if it actually works as advertised it’s a pretty great value prop for a lot of orgs.
It does work as advertised, try it :) And yes RunsOn is a direct competitor to the 5 YCombinator-funded companies operating in this space (Ubicloud, Warpbuild, Buildjet, Blacksmith, Depot).
You've hit on all the main points regarding Docker image cache in GHA. Persisting the massive layer cache over networks is incredibly slow and has weird limits (like 10GB per repo). We persist the layer cache to ceph volumes and orchestrate your cache so it's immediately available across builds with our first service, accelerated container image builds. Our GHA runners run right next to that same infra, so you don't have ingress/egress. All that can be hosted in your own AWS account (we're also open to running that in any general compute environment for folks who need it).
Our original version of that system used vanilla BuildKit + EBS volumes + orchestration, nowadays we've replaced EBS with a distributed ceph storage cluster for significantly faster IOPS and throughput and have modified BuildKit to be better suited for high-performance distributed builds.
Both the container build service and the Actions runners are in the same AWS VPC, so they get good network performance between the two and don't need to egress over the internet.
- create a virtual machine with everything you need in gcloud (would work for aws as well). Pick something nice and fast. Suspend it.
- in your github action, resume the vm, ssh into it to run your build script, and suspend it afterwards.
Super easy to implement and easy to script using gcloud commands. It adds about 30 seconds of time to the build for starting the vm. On the machine, we simply pull from git and checkout the relevant branch. Doesn't work for concurrent builds but it's a nice low tech solution. And you only pay for the time the machine is up and running, which is a few minutes per day. So, you can get away with using vms that have lots of CPU and memory.