Ask HN: Do you run apps bare metal?
- build binary
- copy binary, config files and static assets to the production server
- do blue green deployment (with nginx) to get zero-downtime deployment
- profit
(This is automated of course! I use Ansible, and I can easily rollback if needed. I can also deploy the same app to multiple machines if needed).
On my local machine I use Docker to test the Go code, but I don't really see the benefit of deploying my Go app in a container. My colleage told me "it's easier to deploy Docker containers. You just pull the image and voila!". I don't see how my approach could be "more complicated". Also, isn't my approach better in terms of performance? If my golang app runs "bare metal" instead of via a container, then sure the performance should be better, right?
73 comments
[ 3.5 ms ] story [ 179 ms ] threadTry to deploy code without Docker and you have one problem.
Try to deploy code with Docker and you have two problems.
You might have a 10 second build process that does 1 MB of disk IO and it becomes a 100 second build process that does 10 GB of disk IO because you ran it in Docker.
I see people who "know how Docker works" put no care at all in the order of their image builds and end up doing 10-100x more IO than they need to.
Docker is fast compared to the time scale most people deploy servers at but is slow compared to the scale of a healthy software development process particularly if you are not running on a cloud server or are working on a gigabit connection.
If you took the time you spent trying to eliminate entropy from Docker, Kubernetes and such and actually tracked down the entropy that comes in from your environment (like the vandal who set your platform charset to not be "utf-8") you might find you don't have time to check your phone or send your co-workers stupid GIFs on slack waiting for builds.
I don't want to go into no-true-scotsman territory here, but this really is a literal contradiction.
How and why? Unless you use a wrong image ( like Alpine for Python) or your Dockerfile is poorly made and cache doesn't work, nothing should change besides immutability and proper and clean dependencies ( your code won't work because you have some lib installed on your local machine by chance, and fail in prod because the lib doesn't exist or is ot a wrong version).
1. this is implementation detail.
2. it shouldn't be so easy to write a bad Dockerfile
3. if it works, it works, and Docker encourages this approach.
4. and this damn cache: Docker is only tolerated in any way at all because of this. alternatively, it might be that baking an apple pie from scratch by first creating the universe is not the sensible thing to do.
i think given the mass proliferation of Docker, i'm going to face some disagreement with this opinion: i don't think we're in a particularly good place where we have a billion deployments musl libc, busybox, nginx, and what else, all of varying versions. on top of the security implications of this, think of all the time and energy wasted in transferring all of this repeated data, on every deployment.
i think it's madness.
Assuming it worked, I could wait a few hours to bring down an image for the the first time. Often I could get the job done in the time it takes to suck down the first image if "doing the job" could be done without Docker.
I never managed to use it enough that the images were in the cache when I needed them!
To make matters worse, unlike many systems that do big downloads (esp. BitTorrent) Docker is not reliable at doing large image downloads.
What people don't get is that the internet infrastructure is more expensive than they think because it is paid for in bits and pieces everywhere -- for instance it is an environment disaster that console games now come in 100 GB + downloads:
https://gizmodo.com/downloaded-games-have-a-larger-carbon-fo...
You are introducing another layer of complexity.
Unless you built it statically, that's another thing
The solution shouldn't be don't install security updates. It should be having a workflow that tests and validates security updates before deploying them to production.
And you should have this with or without docker.
I built a Go web app side project which runs smoothly on a VPS with nginx as reverse proxy, letsencrypt SSL and dead simple supervisor config to run. Boom. Forget containers. Forget docker.
Having worked with and without Docker for various web apps, it removes some dependency management and server setup at the cost of another layer (or two...) of complexity. It's not always worth it to go to containers.
Docker seems to make the most sense in cloud environments that scale horizontally.
What benefits would you get exactly?
But with just a Go binary, I think you're doing it right.
[Edit] Thanks for the comments, seems I'm just plain old.
I once had a small kernel patch for my ultrasparc that kept 14 of the system's 16 CPUs from being scheduled for anything but my application, just before they came up with cpu pinning and other, better interfaces for that... I didn't consider that "bare metal" at the time, but it was definitely thinly wrapped.
No. When you run with an OS, the OS doesn't intercept every single hardware instruction. It only interjects occasionally (or upon requests) - and the rest of the time, your code (or that of other processes) runs directly on the hardware.
... to be honest, that is also partially true in Virtual Machines, but there, the I/O devices and other facilities you see are mostly virtualized, so that when you interact with them you're actually triggering the VM host to run its code rather than triggering physical hardware to do something.
The definition may be changing, but at least Wikipedia still has it this way.
Many times while looking through a project I find on GitHub, if they have a docker image available to try it out, I can try it by running a single command. But if they have more than 3 installation steps, I usually pass.
I usually do the opposite, even if there's a docker image available, to try it out I follow the build and manual setup. I need to know _how_ it works even if at the end I use the image. If something breaks, I own this and need to be able to debug it. Depending on the program and its architecture, I also need to know how to scale it.
It's fine to serve as a reference, and for reproducible builds, and for appliances like a mail server. But I never opt to run an app that way even just to try it out, and I especially can not stand when an app developer only offers that as the only way they distribute an app (there are some, and not open source so you really have no choice). This prioritizes the developers convenience over the users. If it's their right to do so, then it must be exactly equally my right to say I don't like that and try to avoid using their apps.
Then it should really make no difference in terms of performance. A process in a docker container is also a process in the host OS, but with some network and user permission stuff on top.
When I read the title I thought you meant bare metal as with no OS layer, just a go app running in the processor, which I think would be pretty cool to see.
A container _is_ an ordinary process.
https://iximiuz.com/en/posts/not-every-container-has-an-oper...
(we just discussed another article by the same author a few days ago on slimmer containers)
"The container process is isolated (namespaces) from the rest of the system and restricted from both the resource consumption (cgroups) and security (capabilities, AppArmor, Seccomp) standpoints. But in the end, this is still a regular process, same as any other process on the host system."
Note that configurations modifying syscall behavior is not unique to containers. There are other ways, only rarely encountered in the wild, to configure your Linux system to inadvertently break the same required syscall behavior.
Linux makes no guarantees about the behaviors of syscalls. Nonetheless, high-performance software is necessarily designed under the assumption of specific behaviors that are almost always true on bare metal and modern virtual machines when configured sensibly. AFAIK there is no way to configure a container to ensure the same behavior, which is the problem. VMs in the early days had many of the same issues. It may be sorted eventually (like with VMs) but for now it causes severe regressions in many codes.
Containers have no overhead, you use the system kernel to do I/O for you, just like any other process.
No process serious about I/O performance uses the system kernel for I/O. Doing it in user space is integer factor higher throughput, hence why Linux has APIs to make this possible for software that needs it. These work perfectly on bare metal and virtual machines with negligible loss of performance in the case of the latter.
That aside, containers are currently (inexplicably) missing other capabilities that have been assumed and idiomatic for high-performance software for decades, such as strict hardware resource control and discovery. You can sort of work around this with gross hacks but that doesn’t scale well. Basically, containers made things that were easy for decades stop working or become complicated, and these things were highly valuable for some types of software so discarding them is not an option.
How do you handle multiple applications not interacting or affecting the data of another if there's an exploit?
Our contract is that a unit of deployment is a .tar.gz file, and inside that there is an executable file bin/run which starts the app. We build apps in a few languages, but they all have a start script like that.
You can use static or dynamic linking, compiled or interpreted languages, anything you like, as long as the start script sets it up correctly. For Go apps the script can be trivial.
Tarballs must be self-contained - binaries, libraries, config files, static assets. We make an exception for interpreters, because those are huge and change slowly. Those are manually installed in /opt/something. I'd like it if we did something more disciplined and reproducible, but it's good enough for now.
We don't have many machines. Maybe 20? You can get a lot done with 20 full-size physical machines.
I don't see how something like Docker would help us here. It would just be the same, but less hackable.
Docker is fine but if you don't need it then fine (and managing it yourself is usually a pain). But do make sure your systems are up to date (basically your linux distro)
One downside that we find surprizing is that way too many Ops people are more familiar or comfortable in cloud or Kubernetes environments these days. We find it hard to find local talent who are willing to deal with bare metal hardware.
Because of that we plan to migrate to k8s at some point later this year. I'm neither support nor oppose it, to be honest. Introducing an extra Docker build step is annoying, but the idea of adding a few hundreds of Yaml lines to get monitoring, log aggregation, tracing, etc. sounds really nice, too.
You can even provide assets (like say a JS app and associated HTML, CSS) that are stored inside the app. You can then build a full app as a single binary with no extra dependencies or files. There are a few gotchas like when you need native library support in your app (if you are using a sqlite driver) and then you need to ship a .node file with your app. But, generally, it is a great solution once you get the hang of it.
You compile against a specific version of node, and that runtime is included in your binary. You don't need the client to install node or choose a version, you lock that down when you build your app.
I've experimented with a lot of different nodejs packaging tools, but pkg seems to work the best with the fewest surprises.
Container technology run your software just as it was a standard process.
There are no performance implications in running code inside or outside the container.
Here be careful, that on MacOS docker is implemented as a virtual machine. In such case there could be somehow serious performance implications. Especially around IO.
https://stackoverflow.com/questions/21889053/what-is-the-run...
Docker became popular because it fixed a lot of deployment issues for dynamic scripting languages. And really, Go static binaries and/or Java jars solve that same problem. So there is no need to complicate these deployments with Docker.
I do dockerize some Go binaries to run in ECS/Fargate, but otherwise I see no real need or reason to do so.
Bare metal usually refers to running without hardware or software virtualization (kernels within kernels), and if that were the case (docker on linux does not virtualize) it would be meaningfully faster, but that's not the case here.
The primary benefits you get from Docker may not be immediately apparent, so go for whatever works. Eventually, you'll probably find that it's substantially easier to scale out appropriately and automatically with some sort of orchestration tech, which is nearly all built on containers these days. Don't sweat it til you need it but at the same time, it's easier to learn sooner than when you're knee deep in tech debt for a dozen services.
"Bare metal" means that your application is running on the same OS that is also running the raw hardware (metal), aka no virtualization. Containers are not (generally) the same as virtualization.
The analog to bare metal is virtualized, where the hardware your program is seeing is not necessarily the hardware that is running on the actual host machine.
A docker container could ostensibly be considered running on bare metal. A container is really just isolation but the parent OS/kernel is in command. Here is a graphic that illustrates the differences: https://www.sdxcentral.com/wp-content/uploads/2019/05/Contai...
What you are really asking is do you need an abstraction layer or orchestration tool to manage doing this.
The short answer is that no you do not need it at all. If you can DIY this and are happy with it, that is sufficient. For example, a current deployment process for one of my clients (EC2 environment) involves stopping a custom systemd service, pulling the new binary/deps and then starting the systemd service. Really simple with a small instant of downtime, but within this environment that is not a problem.
Typical IT usage means something is running on an OS that’s installed on the hardware, as opposed to in a virtual machine.
https://en.wikipedia.org/wiki/Bare_machine
And you can definitely run a bare metal application virtualised, just like you can run an OS virtualised.
To my mind the main benefit of containerization would be that the
> binary, config files and static assets
would be bundled together inside the container, so there is less risk of you running a half-baked version where (e.g.) the binary gets updated but the other files don't. However since you're using Ansible to deploy, that risk already feels small. You could also consider using the embed feature in go1.16 to bundle all the assets inside the binary.
Arguably there might also some security benefit to running in a container, but I wouldn't want to try and make that argument without knowing a lot more about the specific details of your binary's behaviour.
If for now it's just you and you're not planning on spinning up in AWS ECS, CloudRun, any flavour of Kubernetes etc. then go with what works for you. All good.
Lots of people in here don't know what "Bare metal" means. There's no operating system - at all. Anything else is not bare metal.
To go further: Sometimes (if you're bringing up a new processor design) you don't even get cstdlib - you have to write it (or the processor's architecture won't even support C!). "Bare metal" means exactly that - it isn't the same as "no virtualization"... at all.
A process that runs inside an OS, is running inside an OS.
This is like when they redefined "literally" to literally mean figuratively. OK fine "language changes" but then now what word does the job that "literally" used to do? We had a word to express a concept, and now that word means the exact opposite of that concept, or worse, may still meam either one, making it meaningless and non-functional.
Although unconvinced, I gave up that fight and retreated to my usual protesting that a method and a methodology are two very different things, although it seems everything is a methodology these days. Longer is better?
Docker is helpful when you have a lot dependencies, or otherwise need to create a reproducible image (configuration, special directory structure, etc.) or when you want to control the process's resources (sandboxing).
Otherwise, because Go compiles into single binaries, if there's no other external dependencies/configuration to manage and you don't need process isolation/sandboxing, then Docker is just another moving part that adds complexity and could go wrong.