I come from the development background and the number one use case of containers on macOS is development enviroments, as on Windows too. For this use case, file system IO has always been bottleneck, not CPU. I do not know if there is some silver bullet in the horizon that could make this faster.
I've been really impressed with what I can accomplish with lima-vm, beyond just "replace docker desktop". The latest pre-release supports Virtualization.framework, including virtiofs.
Last I checked (which was not that long ago), Lima was stuck using 9p because QEMU for Darwin didn’t yet support virtiofs. Has this changed? Or maybe I’m just misremembering the details…
Not if you’re running x86 emulation on an ARM host. It’s terribly slow.
EDIT: This is from experience. When I started using a M1 mac, our docker builds on x86 images took up to 5x or 10x longer vs running on an arm64 image. Had slightly faster, but similar results running my own docker host on Lima, or podman, or building without docker inside a UTM host which also uses qemu.
Well, it can be. You can still emulate additional hardware with QEMU, and QEMU still has other backends (which it will use if you run a x86-64 OS on ARM macs).
It's not abstraction, QEMU uses Hypervisor framework (which is equivalent to KVM). Virtualization framework is Apple own hypervisor (which is equivalent to QEMU).
so arm macbooks are still problematic for developers who use containers and x86 servers...
it's still surprising to me that they did this as i always got the sense that developer word of mouth helped drive a lot of macbook adoption in the 2000s.
I use ARM containers locally to dev and test locally, check in the code and the build server creates an x86 image for deployment to the server.
Worst case you have to build the ARM container yourself locally and tag it with the "remote" name and docker will just use it instead of pulling the remote version.
you have to add support for your infrastructure to support building of everything for two architectures. sometimes problems are architecture dependent. if you have to use x86 containers for some reason, they will be dog slow because you'll be running the kernel under emulation and in userland rosetta won't help.
Rosetta absolutely does help.
You register rosetta (which can be mounted into the VM using macOS's virtualization framework) with binfmt_misc and your x86 binaries inside the VM will run through Rosetta.
Not dog slow but slower than they could. My experience is: a Rails test suite takes 75 seconds on my 2014 Intel ZBook laptop with a SATA bus and a SSD. It takes 50 seconds on a M1 MacBook. I'm sure I could have found some expensive Intel laptop as fast as that back in 2014. Docker on the M1 runs with something called Mutagen and we have two different docker-compose.yml, but one Dockerfile.
Aside from the basis of your comment being not really true (as mrunkel commented), every thread about Apple Silicon Macs is full of people praising the performance and battery life. This is more positive word-of-mouth than Apple's had in years.
I don't understand why one needs to emulate a virtual machine to run a container. Why do you need a kernel? Why emulate physical devices, interrupts and everything else? Why cannot one run a container as a protected mode application?
Adding to other answers, not only containers of the docker and LXC/D types are Linux things that need a Linux kernel but often the software inside the container must be deployed and run on a Linux host. Often the build is done on a Linux server in a CI system and not on developers machines but developers do have to run that software too, kind of by definition.
It’s difficult to emulate a kernel. You can try and there have been multiple attempts to do so with varying levels of success but most have pitfalls that people eventually run into.
Not right now. Can be an order of magnitude slower compared to native. Depends a lot on the program and current task. Also, the QEMU defaults are not optimal.
Depends. QEMU can use Apple's Hypervisor virtualization framework to run ARM64 operating systems on Apple Silicon without emulation.
It seems to be possible to then use Rosetta 2 to run x64-code on those operating systems.
Rosetta 2 works great for AoT stuff but for JIT you'll always end up with at least a 2x slowdown since it needs to always JIT the JVMs JIT output. That said, it working at all is pretty amazing.
The CPU architecture of the VM and Host have to match, otherwise you get software emulation which is very very slow. It's good enough for occasional usage, but nowhere near good enough for daily development purposes.
I experimented with LXD containers on Linux recently, but I found the technology it builds on (cgroups) too hard to wrap my head around, and tutorials leave me in the dark.
Your initial comment mentions experimentation with lxd - did you mean experiment with lxc? The level of abstraction is very different (this is the raison-d'être for lxd, after all!).
That's documentation for RHEL 6 which is now 12 years old. It pertains to cgroups v1 which was horribly complex. It was replaced by cgroups v2 which is much simpler; there's now a single hierarchy of cgroups.
Well, I started looking into it because there were all these mounted filesystems that I couldn't unmount, even as root. I guess I will have to look into namespaces also.
Yeah, it's pretty much an implementation detail - the thing running your containers likely exposes knobs like 'memory limit' and 'cpu limit' and under the hood it takes care of manipulating the files within the mounted cgroup filesystem.
You'll be pleased that at least with cgroups v2 there's only a single /sys/fs/cgroup instead of half a dozen filesystems mounted underneath that path. :)
"""
The goal of this project is to enable MacOS users to:
Easily spin up and manage lightweight Alpine Linux environments.
Use tiny VMs to take advantage of containerisation technologies, including LXD and Docker.
Build and test software on x86_64 and aarch64 systems
"""
I ended up just getting a System 76 Meerkat computer for doing anything locally with LXC/Docker. Linux emulation in Windows or macOS is unbearably bad if you really need to do serious work.
Using a VM for Docker gives you the worst of both worlds.
You get all the downsides of a VM (persistent wasted RAM/disk space, awful disk IO performance, etc) AND all the downsides of containers (difficult to introspect when your minimal image has no shell, janky networking, etc).
On recent M1/M2 Macs you take the additional pain (and thus performance hit) of translation to x86 (because that's almost certainly what your containers are in.
> On recent M1/M2 Macs you take the additional pain (and thus performance hit) of translation to x86 (because that's almost certainly what your containers are in.
What on Earth are you talking about? arm64 Docker images predate the emergence of M1, and their number has proliferated in the last two years like there was no tomorrow. Here is an excerpt from my own tiny, non-representative collection of Docker images I have had to deal with lately:
$ (for i in `docker images | tail -n +2 | awk '{ printf "%s:%s\n", $1, $2 }'`; do docker inspect $i | jq '.[].Architecture'; done) | grep -c amd64
7
$ (for i in `docker images | tail -n +2 | awk '{ printf "%s:%s\n", $1, $2 }'`; do docker inspect $i | jq '.[].Architecture'; done) | grep -c arm64
23
Most of the 23 arm64 Docker images is pretty obscure stuff, with one or two manually rebuilt for the arm64 architecture. Yet, I have downloaded all of my obscure images directly from the Docker Hub – not traded them for a pinch of spice from a shady Docker image dealer at a drench inducing corner of the dark web. All of the arm64 images run at the native speed, locally and in the cloud.
The only reason I store x86 (i.e. «amd64») images locally is my laptop being a transient transit point between their source and the destination. I would have rebuilt them as well but the build process is either broken for each of them or is too convoluted to justify spending a time trying to fix it.
But then it's not the same container, right? Either you'd need to switch your whole cloud infrastructure to run on ARM (which I see is possible now, and I'd be curious to hear if people have done that successfully) or we're back to "works on my machine ^W architecture".
> Either you'd need to switch your whole cloud infrastructure to run on ARM …
Nope, the whole cloud infrastructure does not need to switch.
Since containers are fully self-contained, self-sufficient and isolated from each other, they can coexist peacefully and run alongside each other in a mixed setup in AWS ECS/Fargate clusters.
And they do. AWS Lambda also supports aarch64, and the degree of isolation for the lambdas is even higher and they can also run out of a x86 or an aarch64 Docker image (if required), and 30-40% cheaper with performance loss incurred for the workloads I am interested in.
I was exaggerating slightly - you will get some images in ARM.
But it's hard to not see this as evidence for my point when you were only able to rebuild "one or two" of the 8-9 amd64 images you're running without significant time investment.
By your own stats roughly one third of the images you use don't support ARM. And if you want to run one of those on ARM - you're likely screwed!
Docker ARM support is improving, but acting like it's ready for prime time is laughable. Try doing your day job with no amd64 containers for any reason for a month and come back to me.
You had made a response to the following statement – the one I was responding to:
> On recent M1/M2 Macs you take the additional pain (and thus performance hit) of translation to x86 (because that's almost certainly what your containers are in.
The statement is false. x86 containers do not use the translation, hence they can't run on M1. People have been successfully running aarch64 (ARM) containers, including on AWS Graviton (re:Invent 2018) and on Graviton2/3 (more recently), for a few years now. Oracle cloud have been offering Ampere cloud compute resources for… a couple of years now?
But you have conveniently dodged it, for it did not suit your agenda and presented it as «exaggerating slightly». It is not slight and it is not exaggeration, it is a blantantly distorted and also a likely ignorant statement.
> But it's hard to not see this as evidence for my point when you were only able to rebuild "one or two" of the 8-9 amd64 images you're running without significant time investment.
> By your own stats roughly one third of the images you use don't support ARM. And if you want to run one of those on ARM - you're likely screwed!
Whether one is screwed or not is a matter of perspective (I am certainly not the one), just as 23 aarch64 images are hardly «were only able to rebuild "one or two"». I have revised my 9 locally stored x86 images, thanks to you, and purged one stray one, with the remaining ones being, in fact, an images itself, and an Docker image tag slapped on it. Therefore, I only have 4x remaining x86 Docker images, but of course it is not going to convince you and you will resort to a distorted narrative again.
> Docker ARM support is improving, but acting like it's ready for prime time is laughable.
«Docker for ARM» means «Docker running on Linux/ARM». Docker has supported Linux, and Linux has support aarch64 for a long time now. Solutions I design for my clients have been running as/in aarch64 containers for a few years now, and my clients indeed now find it laughable how they could not switch to ARM sooner due to 30-40% lesser cloud compute bills. Most solutions do not require neither containers nor servers, though, it is only the platform related stuff.
> Try doing your day job with no amd64 containers for any reason for a month and come back to me.
If you don't understand the difference between the aarch64, and that the Apple instantiation of the aarch64 architecture is not the only one, then you it is unlikely that you understand what Docker is and what Docker is not. Therefore, it will be difficult for you to self-assess and quantify the magnitude of your own uproarious laughter. We are at a point now when the CPU has become a configuration setting at the infrastructure provisioning time, and is not an insurmountable barrier.
All of the containers I need to run for my current needs, run, and they do it perfectly well in mix and match Fargate container clusters. Most containers are Graviton2 containers, with a few select ones (the ones that are cumbersome to rebuild), run in AWS x86 containers, and both types coexist peacefully. Last but not least, I do not even run most containers locally, I occasionally help developers with build related problems or to cobble a POC together to verify an idea, evolve it into a solution or a product, and scrap the container setup. Everything else, including production workloads, runs in the cloud, and it just works 24x7.
> The statement is false. x86 containers do not use the translation, hence they can't run on M1.
Sorry, let me admit I actually don't own an M1/M2 Mac and have only used one by proxy helping my coworkers. To be completely honest, I just assumed it was possible to run amd64 containers on them through some form of translation (QEMU/Rosetta).
That actually makes my whole point stronger though - if the container you want doesn't support amd64, you either fix it yourself (often a time consuming and difficult process) or you just can't use it at all.
The rest of your post is just chasing ghosts. We're talking about M1 Macs, remember, not cloud servers? Thanks for letting me know about Graviton, but I am literally employed by AWS and build aarch64 containers as part of my day job, I'm well aware.
I'm a follower on the MS SQL Server Docker github repo... it's definitely a bigger issue for some than others. Depending on what the container/service does, the amd64 translation isn't too bad, it's the disk i/o that really sucks, which is being worked on/improved. Even then, more people need to understand in Mac/Windows how/when to use Volume containers for storage, not local/native/host disk directly, which helps a lot.
In the end, it'll get better... I'm happier with WSL than Mac, even if I personally prefer straight Linux.
At this point I’ve only got one non-arm64 container, and it’s solely optional support for testing some obscure cases that our CI infrastructure handles more robustly anyways.
> persistent wasted RAM/disk space, awful disk IO performance
I'm getting cranky about arguments against VMs.
1) I have had more RAM than I ever use for about 25 years. I upgrade about every 4 to 6 years. Is everyone else writing code on Amigas?
2) I keep getting told by everyone (IT, devops, DBA, my boss, his boss) that disk space is cheap.
3) VirtualBox has been consistently ok for about 10 years and I almost always can outrun scripts in a Linux VM on my Windows machine compared to our production AWS instance running K8s. Of course my machine would never keep up with a high user count but this is about disk performance, and locally I can prove both my monolith and my VM are faster.
I don't see how arguments like these are good anymore with the modern hardware of a local dev system. That makes containers an over complication. The only reason to do them is to save devops the headache of having to create a docker file. I mean the guys read books and watch YouTube while I'm writing code, so...they can write the damn docker file.
Even then, access is still significantly better on a local machine in a VM than it is on a shared server... You are effectively the only real user, generally speaking. The Gen4 NVME drive and 64gb+ ram on my local system is still better than shared server resources are generally going to be, short of dedicated (ie: expensive) server environments to do the same.
I'm not a huge fan of VM development, generally speaking... that said, my current work setup is Windows + WSL + Rancher Desktop + VS Code (remoting extensions) for nearly everything. My windows terminal default is set to WSL, so it's pretty close to in the box... I can launch windows apps from the wsl environment... the integration is pretty seamless. In terms of performance, if you're stuck on an older Windows version (I am), then you can limit the max memory that the WSL environment gets, similar to a VM... I usually reserve 8-12gb for my host environment and give the rest to my dev environment.
My personal desktop is Linux (Ubuntu-Budgie). My previous job the laptop was an M1 Max, which had faster ram/disk than the windows laptop for the current job, or my personal desktop (which is impressive to me).
In the end, for most things, it's not significant enough to worry about... The aarch macs and i/o have been more problematic than wsl and other x86 environments... I think as rosetta support improves with qemu and docker, it will change and get significantly better.
That said, with VS Code's remoting extensions, I can be working on a remote system nearly as effectively as local anyway... which is how I typically use my laptop. Wireguard to home, vs code w/ ssh to my personal desktop.
At the end of the day, are they all just wrappers around QEMU?
The aarch64 -> x86_64 virtualization performance is pretty rough.
Like, to the point where compiling a small hobbyist Rust project that needs to be deployed on Linux x86_64 in the cloud (cheap VPS)... you might as well just build it in the cloud and not locally on Mac host because it takes 10 minutes instead of 10 seconds
Keep in mind lxd can manage two types of “containers” these days - the traditional cgroup-based kind that runs as its own set of processes on top of the hosts kernel with isolation, and traditional qemu-backed virtual machines. The user experience is homogeneous but the backing engines are different as noted here.
this is so .. exciting! .. however please recall that you the user are now using hardware that is remotely run in most cases by the OS vendor (and who-knows-what-else), with opaque code executing at multiple layers.
I like the progress that is being made for running containerized workloads on macOS. In my case I like some of the benefits of running the workload on a remote machine; such as no fan noise, less heat, less power consumption (especially on laptops). However the downsides can be also quite annoying, such as file sync times or IDE input lag.
My current setup is to have both data and workload run on a remote machine and I connect to it via SSH. I can either run neovim inside or use the remote development plugin from VSCode. But as mentioned, the input lag can be very annoying. I’m wondering if there’s another setup where I can still retain some of the upsides of running the workloads remotely and still having a decent user experience (reduced lag)
I've been using this for years, editing locally with Sublime, while running everything on a remote Linux vm. Because it's two way sync, anything created/modified on the server is also sync'd back to your local filesystem. I end up syncing most of the remote $HOME so I can easily fire up a new VM, sync, and go. It really is fantastic.
I've never noticed any input lag with VSCode remote SSH. I was under the impression the editor cached file contents (pulled entire files at a time over SSH) to resolve this issue.
One useful tip for reducing the latency with ssh commands (like docker, with DOCKER_HOST pointing to ssh server - for example enabling tab/autocomplete for subcommands (which does ssh to lookup list on tab)) - is to use PersisControl with ssh:
This is cool and a worthwhile thing, but how is this different than the many (b/x)hyve clones and others based on QEMU that use MacOS’s virtualization framework to run a minimal Linux for containers? What’s the differentiator that makes this better (hopefully?) from what’s come before?
It's appealing to me because I can use LXD on both Mac and Linux now. One less discrepancy between those systems.
That said, there now seems to be a huge number of options for running various combinations of VMs and containers on Mac and Lenix now. I just use whatever seems popular and is easy to set up, but I have no sense of the pros and cons among them. I would love a coherent document on this topic.
At first, I thought this was based on a syscall compatibility layer like Solaris' Linux Zones or WSL1 (RIP), or the Linux support in FreeBSD and NetBSD.
If you've ever tried to spin up a whole bunch of Docker containers in WSL2 and watched `vmmem` memory and CPU usage explode, you know that 'near-native speed' in VMs comes with lots of asterisks.
Does macOS have usable native macOS containers yet?
So is there some canonical guide to running a docker compose style app on Mac m1 machines that has good filesystem performance? It seems like there’s many ways to approach the topic now so it’s hard to tell which one is “winning”.
I’d love to containerize all of my local development efforts (scripts and rails apps) but the slow ass filesystem always ruined it in the past.
Have you tried enabling Virtiofs in Docker Desktop? On my M1 Mac that sorted the performance for me for large PHP apps, and I haven't experienced the permissions problems that some people have (there is a test build that appears to fix those out now).
I haven’t tried it for ruby, though I have used vanilla nix shell for ruby projects before and it worked quite well after I over-rode GEM_PATH and GEM_HOME to the correct values.
I've been using Colima lately and it's been a great experience. The only thing that didn't quite work was pushing an image to AWS ECR, but pulling images, building images (with BuildKit), and running containers all worked fine, and faster than Docker.
I've tried Colima too but it has always felt sluggish compared to local dev. I don't think any form of cross-host filesystem sharing could ever be fast enough.
I hadn't heard of Colima before seeing your comment, decided to give it a whirl and it's a breath of fresh air. Much faster and uses less resources. Thanks!
Nix builds native software for your platform reproducibly. There is no OCI image, no Docker, just a way to reliably install software, which is a great foundation for building per-project development environments.
(Aside: you can build oci images using nix, but that’s probably what you would use for building deployable artifacts for production, not for development).
I would not say it is comparable with a Dockerfile - a Dockerfile is closer to a shell script, with caching between each step. Credit to Dockerfile syntax for being very easy to grok, but it is inherently not reproducible.
The perceived reliability of docker for development probably comes from the fact that you can share a prebuilt image, but rebuilding might not always work.
I've recently switched to using nix-shell and overmind with a Procfile to officially retired docker-compose from all of my projects. Such a breath of fresh air.
Like Mitchell Hashimoto (Hashicorp), I no longer do any development on macOS directly; I have a Linux VM where all my projects live and have their environments managed with a shell.nix file. Highly highly recommend it.
Would you elaborate on the setup? I recall seeing a comment or article where Mitchell mentioned this setup previously. I’ve been using Make + Docker for development on macOS for a long time now and perhaps it’s time for a change. I’ve looked at nix once or twice over the years but it never really clicked.
Then I can run `nix-shell` to load all the dependencies and `overmind start` to bring up Postgres + Meilisearch with ignored data dirs relative to the project folder. If any of these packages need to be pinned to a specific version you can do that with this tool[1].
That's pretty much it - with all the background services running you can load your monolith or microservices with cargo/npm/etc. It's all very nice and clean!
[1]: https://lazamar.co.uk/nix-versions/ - I actually just used this in a new shell.nix for my personal website to pin a version of Hugo from 2019 (when I last touched it) so that it would build again
this is great, and I never heard of Overmind until now (looks cool!) but you could also flake this config out and get a couple more perks, like possibly not having to use the nix-versions thing
It's less than perfect and being worked on afaik... Main thing to avoid are any volume mounts to the host environment. Declare/use volume containers in your docker-compose.yaml, will help a lot, but still more sluggish than host/native.
If you use VS Code the remoting extensions pack includes SSH, for my personal laptop, I'm usually using code to my linux desktop, and editing remotely... it's pretty slick. Code will run a "code server" in the remote environment, and is effectively just displaying the text view to you, the actual work is happening in the remote envornment. You can do similar with containers, or in windows+wsl.
I'm thinking about giving this a try but using an iPad connected to a monitor setup for this. I rarely code on the go for personal use, so this could work pretty well...
There are codium forks that will host a web server environment for you... easy enough to VPN home, or another trusted environment, and use a code(ium) server on your host.
Too bad almost all containers rely on bridged networking and different ports. Why not just bind to 127.0.x.y, where x is the project number, and y is the machine number. That way, you can just use default ports
Because the loopback address doesn't work like that. 127.0.0.0/8 is defined as all loopback, which means the OS shouldn't differentiate between them, so any VMs binding on 127.* are going to interfere with the OS binding there.
133 comments
[ 3.1 ms ] story [ 207 ms ] threadI come from the development background and the number one use case of containers on macOS is development enviroments, as on Windows too. For this use case, file system IO has always been bottleneck, not CPU. I do not know if there is some silver bullet in the horizon that could make this faster.
EDIT: Looks like this was indeed just added in the last couple of weeks https://github.com/lima-vm/lima/commit/c18ae239b69a47db77436...
I am personally slowly moving back to Docker after having used limactl for a long time. It's just a pain in the ass.
A Linux laptop is better for all intents and purposes, except the shitty battery life...
So in other words, qemu runs at near-native speeds?
EDIT: This is from experience. When I started using a M1 mac, our docker builds on x86 images took up to 5x or 10x longer vs running on an arm64 image. Had slightly faster, but similar results running my own docker host on Lima, or podman, or building without docker inside a UTM host which also uses qemu.
https://developer.apple.com/documentation/virtualization/run...
From there you can register Rosetta as a runtime and use it to run x86-64 Linux apps on ARM64 if you want. (https://developer.apple.com/documentation/virtualization/run...)
The point of running stuff in qemu is to virtualize / emulate the hardware. You still need a “normal” OS inside qemu.
it's still surprising to me that they did this as i always got the sense that developer word of mouth helped drive a lot of macbook adoption in the 2000s.
I use ARM containers locally to dev and test locally, check in the code and the build server creates an x86 image for deployment to the server.
Worst case you have to build the ARM container yourself locally and tag it with the "remote" name and docker will just use it instead of pulling the remote version.
E.g. here is page 2 of one tutorial: https://access.redhat.com/documentation/en-us/red_hat_enterp...
All these rules made no sense to me, and while I suppose they become clear at some point, I like my tutorials to be clear from the start.
3 minutes search reveals an easy to follow tutorial.
https://ubuntu.com/server/docs/containers-lxd
I'm guessing bocker would be more useful for that: https://news.ycombinator.com/item?id=33218094
Lxc is somewhat to lxd as jails are to ezjails...
https://access.redhat.com/documentation/en-us/red_hat_enterp...
That said I don't remember having to learn all that much about cgroups when I last looked at lxc...
Well, I started looking into it because there were all these mounted filesystems that I couldn't unmount, even as root. I guess I will have to look into namespaces also.
You'll be pleased that at least with cgroups v2 there's only a single /sys/fs/cgroup instead of half a dozen filesystems mounted underneath that path. :)
""" The goal of this project is to enable MacOS users to:
Easily spin up and manage lightweight Alpine Linux environments. Use tiny VMs to take advantage of containerisation technologies, including LXD and Docker. Build and test software on x86_64 and aarch64 systems """
You get all the downsides of a VM (persistent wasted RAM/disk space, awful disk IO performance, etc) AND all the downsides of containers (difficult to introspect when your minimal image has no shell, janky networking, etc).
On recent M1/M2 Macs you take the additional pain (and thus performance hit) of translation to x86 (because that's almost certainly what your containers are in.
What on Earth are you talking about? arm64 Docker images predate the emergence of M1, and their number has proliferated in the last two years like there was no tomorrow. Here is an excerpt from my own tiny, non-representative collection of Docker images I have had to deal with lately:
7 23Most of the 23 arm64 Docker images is pretty obscure stuff, with one or two manually rebuilt for the arm64 architecture. Yet, I have downloaded all of my obscure images directly from the Docker Hub – not traded them for a pinch of spice from a shady Docker image dealer at a drench inducing corner of the dark web. All of the arm64 images run at the native speed, locally and in the cloud.
The only reason I store x86 (i.e. «amd64») images locally is my laptop being a transient transit point between their source and the destination. I would have rebuilt them as well but the build process is either broken for each of them or is too convoluted to justify spending a time trying to fix it.
Nope, the whole cloud infrastructure does not need to switch.
Since containers are fully self-contained, self-sufficient and isolated from each other, they can coexist peacefully and run alongside each other in a mixed setup in AWS ECS/Fargate clusters.
And they do. AWS Lambda also supports aarch64, and the degree of isolation for the lambdas is even higher and they can also run out of a x86 or an aarch64 Docker image (if required), and 30-40% cheaper with performance loss incurred for the workloads I am interested in.
Should read: «… with no performance loss incurred for the workloads I am interested in». A typo crept in.
But it's hard to not see this as evidence for my point when you were only able to rebuild "one or two" of the 8-9 amd64 images you're running without significant time investment.
By your own stats roughly one third of the images you use don't support ARM. And if you want to run one of those on ARM - you're likely screwed!
Even some _official_ images don't support ARM, random example: https://hub.docker.com/_/percona
Docker ARM support is improving, but acting like it's ready for prime time is laughable. Try doing your day job with no amd64 containers for any reason for a month and come back to me.
> On recent M1/M2 Macs you take the additional pain (and thus performance hit) of translation to x86 (because that's almost certainly what your containers are in.
The statement is false. x86 containers do not use the translation, hence they can't run on M1. People have been successfully running aarch64 (ARM) containers, including on AWS Graviton (re:Invent 2018) and on Graviton2/3 (more recently), for a few years now. Oracle cloud have been offering Ampere cloud compute resources for… a couple of years now?
But you have conveniently dodged it, for it did not suit your agenda and presented it as «exaggerating slightly». It is not slight and it is not exaggeration, it is a blantantly distorted and also a likely ignorant statement.
> But it's hard to not see this as evidence for my point when you were only able to rebuild "one or two" of the 8-9 amd64 images you're running without significant time investment.
> By your own stats roughly one third of the images you use don't support ARM. And if you want to run one of those on ARM - you're likely screwed!
Whether one is screwed or not is a matter of perspective (I am certainly not the one), just as 23 aarch64 images are hardly «were only able to rebuild "one or two"». I have revised my 9 locally stored x86 images, thanks to you, and purged one stray one, with the remaining ones being, in fact, an images itself, and an Docker image tag slapped on it. Therefore, I only have 4x remaining x86 Docker images, but of course it is not going to convince you and you will resort to a distorted narrative again.
> Docker ARM support is improving, but acting like it's ready for prime time is laughable.
«Docker for ARM» means «Docker running on Linux/ARM». Docker has supported Linux, and Linux has support aarch64 for a long time now. Solutions I design for my clients have been running as/in aarch64 containers for a few years now, and my clients indeed now find it laughable how they could not switch to ARM sooner due to 30-40% lesser cloud compute bills. Most solutions do not require neither containers nor servers, though, it is only the platform related stuff.
> Try doing your day job with no amd64 containers for any reason for a month and come back to me.
If you don't understand the difference between the aarch64, and that the Apple instantiation of the aarch64 architecture is not the only one, then you it is unlikely that you understand what Docker is and what Docker is not. Therefore, it will be difficult for you to self-assess and quantify the magnitude of your own uproarious laughter. We are at a point now when the CPU has become a configuration setting at the infrastructure provisioning time, and is not an insurmountable barrier.
All of the containers I need to run for my current needs, run, and they do it perfectly well in mix and match Fargate container clusters. Most containers are Graviton2 containers, with a few select ones (the ones that are cumbersome to rebuild), run in AWS x86 containers, and both types coexist peacefully. Last but not least, I do not even run most containers locally, I occasionally help developers with build related problems or to cobble a POC together to verify an idea, evolve it into a solution or a product, and scrap the container setup. Everything else, including production workloads, runs in the cloud, and it just works 24x7.
Sorry, let me admit I actually don't own an M1/M2 Mac and have only used one by proxy helping my coworkers. To be completely honest, I just assumed it was possible to run amd64 containers on them through some form of translation (QEMU/Rosetta).
That actually makes my whole point stronger though - if the container you want doesn't support amd64, you either fix it yourself (often a time consuming and difficult process) or you just can't use it at all.
The rest of your post is just chasing ghosts. We're talking about M1 Macs, remember, not cloud servers? Thanks for letting me know about Graviton, but I am literally employed by AWS and build aarch64 containers as part of my day job, I'm well aware.
In the end, it'll get better... I'm happier with WSL than Mac, even if I personally prefer straight Linux.
Your information is pretty out of date.
I'm getting cranky about arguments against VMs.
1) I have had more RAM than I ever use for about 25 years. I upgrade about every 4 to 6 years. Is everyone else writing code on Amigas?
2) I keep getting told by everyone (IT, devops, DBA, my boss, his boss) that disk space is cheap.
3) VirtualBox has been consistently ok for about 10 years and I almost always can outrun scripts in a Linux VM on my Windows machine compared to our production AWS instance running K8s. Of course my machine would never keep up with a high user count but this is about disk performance, and locally I can prove both my monolith and my VM are faster.
I don't see how arguments like these are good anymore with the modern hardware of a local dev system. That makes containers an over complication. The only reason to do them is to save devops the headache of having to create a docker file. I mean the guys read books and watch YouTube while I'm writing code, so...they can write the damn docker file.
I'm not a huge fan of VM development, generally speaking... that said, my current work setup is Windows + WSL + Rancher Desktop + VS Code (remoting extensions) for nearly everything. My windows terminal default is set to WSL, so it's pretty close to in the box... I can launch windows apps from the wsl environment... the integration is pretty seamless. In terms of performance, if you're stuck on an older Windows version (I am), then you can limit the max memory that the WSL environment gets, similar to a VM... I usually reserve 8-12gb for my host environment and give the rest to my dev environment.
My personal desktop is Linux (Ubuntu-Budgie). My previous job the laptop was an M1 Max, which had faster ram/disk than the windows laptop for the current job, or my personal desktop (which is impressive to me).
In the end, for most things, it's not significant enough to worry about... The aarch macs and i/o have been more problematic than wsl and other x86 environments... I think as rosetta support improves with qemu and docker, it will change and get significantly better.
That said, with VS Code's remoting extensions, I can be working on a remote system nearly as effectively as local anyway... which is how I typically use my laptop. Wireguard to home, vs code w/ ssh to my personal desktop.
https://github.com/LoganDark/initbuntu
Yes you are.
https://github.com/moby/hyperkit
(Or maybe you mean a non-Docker For Mac VM)
The aarch64 -> x86_64 virtualization performance is pretty rough.
Like, to the point where compiling a small hobbyist Rust project that needs to be deployed on Linux x86_64 in the cloud (cheap VPS)... you might as well just build it in the cloud and not locally on Mac host because it takes 10 minutes instead of 10 seconds
My current setup is to have both data and workload run on a remote machine and I connect to it via SSH. I can either run neovim inside or use the remote development plugin from VSCode. But as mentioned, the input lag can be very annoying. I’m wondering if there’s another setup where I can still retain some of the upsides of running the workloads remotely and still having a decent user experience (reduced lag)
[0] https://www.telepresence.io
https://www.devspace.sh/
- Local and remote copy of files - Two way sync
I've been using this for years, editing locally with Sublime, while running everything on a remote Linux vm. Because it's two way sync, anything created/modified on the server is also sync'd back to your local filesystem. I end up syncing most of the remote $HOME so I can easily fire up a new VM, sync, and go. It really is fantastic.
https://mutagen.io/documentation/synchronization
Or are you just talking about when using neovim?
https://docs.rackspace.com/blog/speeding-up-ssh-session-crea...
For reducing the latency of an interactive ssh session, there's mosh: https://mosh.org/
That said, there now seems to be a huge number of options for running various combinations of VMs and containers on Mac and Lenix now. I just use whatever seems popular and is easy to set up, but I have no sense of the pros and cons among them. I would love a coherent document on this topic.
If you've ever tried to spin up a whole bunch of Docker containers in WSL2 and watched `vmmem` memory and CPU usage explode, you know that 'near-native speed' in VMs comes with lots of asterisks.
Does macOS have usable native macOS containers yet?
I’d love to containerize all of my local development efforts (scripts and rails apps) but the slow ass filesystem always ruined it in the past.
I think the new virtualization framework just came out of beta though so now it’s just VirtuoFS that needs to be enabled.
They all use the underlying OS hypervisor so I’m not sure you can get perceivable performance out of one solution over the other.
What do you mean? Multipass only runs Ubuntu server cloud images.
I believe multipass uses SSHFS (https://discourse.ubuntu.com/t/how-to-improve-mounts-perform...) to mount filesystems between the host and the VM. Performance has been excellent.
Lately I’ve been trying https://devenv.sh/ and it works great!
I haven’t tried it for ruby, though I have used vanilla nix shell for ruby projects before and it worked quite well after I over-rode GEM_PATH and GEM_HOME to the correct values.
nix file helps you build OCI image and then you run it not locally/somewhere else?
can that be compared apples to apples to docker where you can build with Dockerfile syntax, and then run locally the image?
(Aside: you can build oci images using nix, but that’s probably what you would use for building deployable artifacts for production, not for development).
I would not say it is comparable with a Dockerfile - a Dockerfile is closer to a shell script, with caching between each step. Credit to Dockerfile syntax for being very easy to grok, but it is inherently not reproducible.
The perceived reliability of docker for development probably comes from the fact that you can share a prebuilt image, but rebuilding might not always work.
Can devenv environment put itself in a chroot / sandbox where it cannot access my files, and is firewalled by default?
Like Mitchell Hashimoto (Hashicorp), I no longer do any development on macOS directly; I have a Linux VM where all my projects live and have their environments managed with a shell.nix file. Highly highly recommend it.
shell.nix - this brings the packages I need to develop into the $PATH when I run `nix-shell` in the repo
Procfile.dev - this is used to start the background services I need, the format is "name: command" Then I can run `nix-shell` to load all the dependencies and `overmind start` to bring up Postgres + Meilisearch with ignored data dirs relative to the project folder. If any of these packages need to be pinned to a specific version you can do that with this tool[1].That's pretty much it - with all the background services running you can load your monolith or microservices with cargo/npm/etc. It's all very nice and clean!
[1]: https://lazamar.co.uk/nix-versions/ - I actually just used this in a new shell.nix for my personal website to pin a version of Hugo from 2019 (when I last touched it) so that it would build again
[1]: https://m.youtube.com/watch?v=ubDMLoWz76U
Make sure that you're running a recent version of Lima, and use a 9p mount.
Edit: by editing one of these yamls: https://github.com/lima-vm/lima/tree/master/examples
If you use VS Code the remoting extensions pack includes SSH, for my personal laptop, I'm usually using code to my linux desktop, and editing remotely... it's pretty slick. Code will run a "code server" in the remote environment, and is effectively just displaying the text view to you, the actual work is happening in the remote envornment. You can do similar with containers, or in windows+wsl.
Macpine: https://github.com/beringresearch/macpine/blob/71788e9c3c09c...
colima: https://github.com/abiosoft/colima/blob/7ebcf14a69158afa43b2...
So it seems that it has same performance as colima project as well.
As for IO performance, see this colima issue https://github.com/abiosoft/colima/issues/146#issuecomment-1...
Wow, I'm dying to see how it works!
> brew install qemu
:|