I don't see any reference to running Mastodon using Docker on the official installation manual. If there is, could you point me to the right direction?
Right, but is that up-to-date? The parent to my reply states that "more recent, and much more in-depth official installation manual" exists, but without Docker. Does it means Docker will no longer be officially supported?
At the risk of hijacking this discussion, why is installing server-side web apps so ludicrously complicated? It's madness.
I'd love to see more developers succeed with their self-hosted web apps. But it simply will not happen when the barrier to installation is so high. Too many developers vastly underestimate just how complicated installation is.
Or maybe they prefer the complications because it makes SaaS a much more attractive option to customers when there isn't a simple one-click install option?
I just find the situation exasperating. Would love to hear why people think this is a non-issue. And let me repeat: a simple, easy-to-install option for server-side software would unlock countless opportunities for developers.
Are you talking about these instructions in particular? What exactly is overcomplicated if you don't mind me asking?
This is not some trivial web application, it's a federated social network, and it's all laid out for you, basically up to and including steps for ensuring security and production readiness.
I am exasperated too by this stuff, and I am a professional that works with it every day (but these instructions are especially well laid out.)
I'm counting on Helm to save us. I've been admining a Jenkins server on Helm and Kubernetes and I just haven't seen the "miserable experience" I've heard of other Jenkins admins describing.
I've run several other projects using Helm and I'm convinced it's a pattern of admin-friendliness that is common to stable and well-maintained charts. They are just generally easier to work with, don't feel cobbled together, and you basically never get Configuration Drift.
But it still won't be universally easy until Kubernetes is ubiquitous. It seems easy to me now, because I've already grown accustomed to basic Kubernetes maintenance and it's mostly optional, thanks to the proliferation of managed Kubernetes providers on-the-cheap.
It's still a lot to reason about, and at my place of business the status quo is simply worse than that because of struggling with adoption.
Is this easier? It looks at least a bit more complicated at a glance to me. I haven't honestly tried either of them, but neither of these are simple to install. Add a config file to nginx, install erlang and use mix to grab the dependencies... these things are not totally foreign to me as a Ruby programmer, but I could see how a newbie would trip over them.
I still don't know which part people are tripping over, I'd like to hear from GP... The Pleroma doc might actually be simpler because it skips over all of the "Hardening your instance" stuff, but I'm only guessing about what people see as the hard part, since none of this looks particularly hard to me.
To be equally fair, it's really never as simple as "docker run someone/image:latest" – that's only the first step, so you can see it running. In Kubernetes terms, that's the equivalent of firing off a "run-once" pod. In other words, if that's all you did then it's not deployed yet.
It was a few months ago and I don't recall the details but I tried installing both, and Pleroma's intall process was noticeably easier. So it's just my experience.
Mastodon also has this reputation on its own platform.
Because most server-side web apps are complicated. The ones that aren't (too some extend) can just get [apt-get|dnf|yum|pacman]ed into your server and you're ready to go. WordPress and Roundcube are one command away from running on your server. If running apt-get, picking a database password and setting your domain name are too much to ask, it's likely that someone like that might not have the expertise to securely run a web-facing server-side application securely.
There's several systems set up to make self-hosting projects easy. It usually involves some behind-the-scenes docker containers with custom scripts and a custom web interface. If you're interested in something like this, try Sandstorm: https://sandstorm.io/ It has a free demo that shows exactly what it can do. It works quite nice, actually.
As you might have seen, there is only a limited amount of software packages available on Sandstorm. The trouble comes with dependencies and overhead. Not every project can talk to Postgres (WordPress mostly relies on MySQL), leading to multiple databases running at once unless you go and manually configure addons and config files to make software talk to each other. Software is designed to run well on a specific base system, be it Ubuntu Server with postgres, CentOS with MySQL or Arch with sqlite.
To make this easier, large software packages are usually packed into Docker containers. Running multiple Docker containers still requires some effort (you need to specify what persistent data needs to go where) but that's nothing compared to manual installation. It's also a requirement one cannot easily get around, as most people/companies have their own preferred storage paths and data storage requirements for important data (think encrypted databases, separate encrypted disk mounts, etc.). This is tiresome and requires reading up on the relevant documentation, but not difficult.
With most of the complexity removed (just ports and paths), you quickly end up running multiple versions of Nginx and Apache at once because every docker container comes with its own web server and its own specific, non-compatible add-ons. This wastes RAM and CPU which could be used to have your services run more smoothly. Most big applications therefore are designed to run best on their own separate hosts to take full advantage of the (virtual) hardware.
The article linked is a good example. It is not a manual about writing config stanzas and downloading packages; it's the bare minimum one needs to handle to configure something like Mastodon.
I’m skeptical there’s a large segment of people who can’t manage to type “docker run foo” who also are interested in running their own servers.
I’m curious what your actual objection or concern is — why do you think this is too high of a technical barrier? what does a better solution look like to you? etc.
Docker only solves the dependency problem for static file dependencies. For things like databases or other services, you still have to stand those up yourself or use another orchestration tool. Beyond that, Docker also puts the onus on the user to solve for networks and volumes and environment variables and secrets management and configuration management.
"Docker" is perhaps part of the solution, but only a very small part.
Yep. That’s where docker compose comes in. It takes care of the whole process of linking together different docker containers as well as configuration of each of them (such as env vars).
Compose is just a declarative syntax for your stack. Ironically it's not composable. In other words, it's not a package definition--you can't ship a compose file that will "just run" anywhere. For example, most web apps require a database, but you don't want to run one database per web app. How do you write a compose file that says, "use the system postgres if available, otherwise spin up a new one". It also totally punts on secret and config management.
Helm is the closest thing to a package manager for Docker, but it targets Kubernetes, not vanilla Docker (and k8s is overkill for a single server, at least considering the time/expertise required to stand up a productionized k8s installation).
I can honestly say developing in php feels great. Php removes the pain points. That's why it has a bad rap because it enables the novice to create and they make novice mistakes.
What languages have a lower pain point level in your opinion?
Have you tried Sandstorm? That's exactly what it offers - simple installation and management of server-side apps, sandboxed and using a common authentication and authentication infrastructure.
So it doesn't have a database (or it has an embedded database)?
With me, the gripes of installing these apps are not just the app itself but the dependencies, which is a necessary evil and impossible to avoid once things get sufficiently complicated.
>Would love to hear why people think this is a non-issue.
Most developers have never seen anything better. They start working within an artificially complicated ecosystem, get used to it and perpetuate all of the assumptions the ecosystem makes when they create their own products.
And when you point this out, most people just get angry, because for them it suggests they have been doing something wrong for as long as they've done engineering. This is why you always hear things like "buy my website is an application" or "but it has to be complicated to use, because it had complicated functionality".
Simplicity is not easy or intuitive.
I've only started to learn the principles of designing simple systems that do complex things around 10-12 years in my career, after accidentally discovering certain people's talks online (Alan Kay, Rich Hickey) and learning a bit about old-school AI systems. Most people aren't familiar with any of that stuff and don't really care.
One thing I personally cannot comprehend is why web applications are almost universally installed and managed through command line, rather than web interfaces. The only advantage of a command-line interface is pipe chaining, and it's never uses installation/configuration. There is nothing fundamental about command line. It's just a software simulation of a teletype terminal.
The command line has the advantage of presenting portable, terse and editable instructions that can be cobbled together into a recipe which can be abstracted and automated.
Web interfaces are used by half the population of the planet. Any command you run in a shell can be represented by a URL, an HTML tag or a form. URLs and HTML have editable text representations, and they also have semantics, standard composition rules and generic UI.
Any "recipe" you're getting is likely something you get from a web page to begin with. That can be a tool for composition and automation on it own.
So very few of those web interfaces used by half the population of the planet actually utilize semantics, standard composition rules, and generic UIs - even though they're available.
Not necessarily. Distro package maintainers use and construct recipes that are not web pages, in order to allow folks to install complex software with, in many cases, one click.
> cannot comprehend is why web applications are almost universally installed and managed through command line, rather than web interfaces. The only advantage of a command-line interface is pipe chaining
> Would love to hear why people think this is a non-issue
It is an issue, but like, you have to get over it.
Most of the complications come from dependencies (db, caching, etc). How can I, as a web service creator, make it easier to install and configure a postgres DB? I can't! Software is complicated, and sometimes the requirements for software are complicated.
Sure, I could release a version that uses sqlite but then people will whine that it's too slow and doesn't scale. If I release a version that supports both, now I've introduced an entire layer of complexity that I have to support (and tends to break endlessly in my experience).
Yeah, it sucks. but if you're really that worried about it, like you mentioned, use a SaaS service for the app itself or for its dependencies.
Why do we still need to manage domain names and DNS settings through web interfaces? Isn't there a command-line tool for this? I'm asking because every provider uses a different web interface, which is very inconvenient.
DNS provided by your hosting service is really just there for convenience IMO. Using services like DNS Made Easy or the like is worth the small annual fee to not have to worry about it anymore.
On a kubernetes cluster you can run external-dns, which always runs in the cluster and watches for new Ingress records routing traffic on a certain domain. And then external-dns will go manage Route53 DNS records (also works with the goog).
So I just `kubecctl create ingress` with a new domain and after a minute or two, the cluster has set it all the certs/dns set up for me (and in the case of CI, the pipeline will continue and run e2e tests against it).
It obv took investment to learn and set it all up, but on a team I could see how it could save people time who don't want to wrange dns records and such.
51 comments
[ 4.7 ms ] story [ 110 ms ] threadThe documentation also includes various guides on horizontal scaling, using pgBouncer etc.
For some reason Eugen didn't port everything to the new doc website.
I'd love to see more developers succeed with their self-hosted web apps. But it simply will not happen when the barrier to installation is so high. Too many developers vastly underestimate just how complicated installation is.
Or maybe they prefer the complications because it makes SaaS a much more attractive option to customers when there isn't a simple one-click install option?
I asked this question previously: https://news.ycombinator.com/item?id=18385497
I just find the situation exasperating. Would love to hear why people think this is a non-issue. And let me repeat: a simple, easy-to-install option for server-side software would unlock countless opportunities for developers.
This is not some trivial web application, it's a federated social network, and it's all laid out for you, basically up to and including steps for ensuring security and production readiness.
I am exasperated too by this stuff, and I am a professional that works with it every day (but these instructions are especially well laid out.)
I'm counting on Helm to save us. I've been admining a Jenkins server on Helm and Kubernetes and I just haven't seen the "miserable experience" I've heard of other Jenkins admins describing.
I've run several other projects using Helm and I'm convinced it's a pattern of admin-friendliness that is common to stable and well-maintained charts. They are just generally easier to work with, don't feel cobbled together, and you basically never get Configuration Drift.
But it still won't be universally easy until Kubernetes is ubiquitous. It seems easy to me now, because I've already grown accustomed to basic Kubernetes maintenance and it's mostly optional, thanks to the proliferation of managed Kubernetes providers on-the-cheap.
It's still a lot to reason about, and at my place of business the status quo is simply worse than that because of struggling with adoption.
To be fair, its cousin Pleroma is comparatively easy to install
https://i2p.rocks/blog/the-magical-world-of-pleroma-setting-...
Is this easier? It looks at least a bit more complicated at a glance to me. I haven't honestly tried either of them, but neither of these are simple to install. Add a config file to nginx, install erlang and use mix to grab the dependencies... these things are not totally foreign to me as a Ruby programmer, but I could see how a newbie would trip over them.
I still don't know which part people are tripping over, I'd like to hear from GP... The Pleroma doc might actually be simpler because it skips over all of the "Hardening your instance" stuff, but I'm only guessing about what people see as the hard part, since none of this looks particularly hard to me.
To be equally fair, it's really never as simple as "docker run someone/image:latest" – that's only the first step, so you can see it running. In Kubernetes terms, that's the equivalent of firing off a "run-once" pod. In other words, if that's all you did then it's not deployed yet.
Mastodon also has this reputation on its own platform.
There's several systems set up to make self-hosting projects easy. It usually involves some behind-the-scenes docker containers with custom scripts and a custom web interface. If you're interested in something like this, try Sandstorm: https://sandstorm.io/ It has a free demo that shows exactly what it can do. It works quite nice, actually.
As you might have seen, there is only a limited amount of software packages available on Sandstorm. The trouble comes with dependencies and overhead. Not every project can talk to Postgres (WordPress mostly relies on MySQL), leading to multiple databases running at once unless you go and manually configure addons and config files to make software talk to each other. Software is designed to run well on a specific base system, be it Ubuntu Server with postgres, CentOS with MySQL or Arch with sqlite.
To make this easier, large software packages are usually packed into Docker containers. Running multiple Docker containers still requires some effort (you need to specify what persistent data needs to go where) but that's nothing compared to manual installation. It's also a requirement one cannot easily get around, as most people/companies have their own preferred storage paths and data storage requirements for important data (think encrypted databases, separate encrypted disk mounts, etc.). This is tiresome and requires reading up on the relevant documentation, but not difficult.
With most of the complexity removed (just ports and paths), you quickly end up running multiple versions of Nginx and Apache at once because every docker container comes with its own web server and its own specific, non-compatible add-ons. This wastes RAM and CPU which could be used to have your services run more smoothly. Most big applications therefore are designed to run best on their own separate hosts to take full advantage of the (virtual) hardware.
The article linked is a good example. It is not a manual about writing config stanzas and downloading packages; it's the bare minimum one needs to handle to configure something like Mastodon.
I’m curious what your actual objection or concern is — why do you think this is too high of a technical barrier? what does a better solution look like to you? etc.
Docker. Its how any software can be packaged with its dependencies. Best for linux serverside/webapp software.
Install and run a docker with one console command, just provide a network-port and folder mapping.
Or use a webinterface, i like how Unraid does this. If you have a old server/PC to spare, definitely try Unraid.net by just booting it from USB stick.
"Docker" is perhaps part of the solution, but only a very small part.
Helm is the closest thing to a package manager for Docker, but it targets Kubernetes, not vanilla Docker (and k8s is overkill for a single server, at least considering the time/expertise required to stand up a productionized k8s installation).
What languages have a lower pain point level in your opinion?
If you are using something like digitalocean they have a one click lamp install. It couldn't be easier.
I am currently working on an end-to-end test suite, as required by cloudron.io platform, to add it to their appstore,
wget https://cloudron.io/cloudron-setup chmod +x ./cloudron-setup ./cloudron-setup --provider [digitalocean,ec2,generic,ovh,...]
https://cloudron.io/get.html And that's it, after that it's only a web UI.
Does anybody know other examples of software which provides command line install personnalized to the provider you use?
With me, the gripes of installing these apps are not just the app itself but the dependencies, which is a necessary evil and impossible to avoid once things get sufficiently complicated.
Most developers have never seen anything better. They start working within an artificially complicated ecosystem, get used to it and perpetuate all of the assumptions the ecosystem makes when they create their own products.
And when you point this out, most people just get angry, because for them it suggests they have been doing something wrong for as long as they've done engineering. This is why you always hear things like "buy my website is an application" or "but it has to be complicated to use, because it had complicated functionality".
Simplicity is not easy or intuitive.
I've only started to learn the principles of designing simple systems that do complex things around 10-12 years in my career, after accidentally discovering certain people's talks online (Alan Kay, Rich Hickey) and learning a bit about old-school AI systems. Most people aren't familiar with any of that stuff and don't really care.
One thing I personally cannot comprehend is why web applications are almost universally installed and managed through command line, rather than web interfaces. The only advantage of a command-line interface is pipe chaining, and it's never uses installation/configuration. There is nothing fundamental about command line. It's just a software simulation of a teletype terminal.
It's a very powerful interface.
Any "recipe" you're getting is likely something you get from a web page to begin with. That can be a tool for composition and automation on it own.
Not necessarily. Distro package maintainers use and construct recipes that are not web pages, in order to allow folks to install complex software with, in many cases, one click.
Repeatability, scriptability, debugability ... ?
It is an issue, but like, you have to get over it.
Most of the complications come from dependencies (db, caching, etc). How can I, as a web service creator, make it easier to install and configure a postgres DB? I can't! Software is complicated, and sometimes the requirements for software are complicated.
Sure, I could release a version that uses sqlite but then people will whine that it's too slow and doesn't scale. If I release a version that supports both, now I've introduced an entire layer of complexity that I have to support (and tends to break endlessly in my experience).
Yeah, it sucks. but if you're really that worried about it, like you mentioned, use a SaaS service for the app itself or for its dependencies.
https://stackexchange.github.io/dnscontrol/getting-started
I wish they had use the standard bind format to describe the zones, instead of JavaScript, but the tool looks great.
https://github.com/kubernetes-incubator/external-dns
A similar thing happens for ssl certs from LetsEncrypt https://github.com/jetstack/cert-manager
So I just `kubecctl create ingress` with a new domain and after a minute or two, the cluster has set it all the certs/dns set up for me (and in the case of CI, the pipeline will continue and run e2e tests against it).
It obv took investment to learn and set it all up, but on a team I could see how it could save people time who don't want to wrange dns records and such.
> Now let's add docker to the sudo group, that will allow us to run all the docker commands without sudo
This command is actually adding the current user to the docker group, not adding docker to the sudo group. Same end result though.
from TFA: `sudo apt-get install nginx emacs`