Is anyone aware of a supervisor system like otp supervisors that manage unix processes? I'm trying to throw together a collection of small services on a server in a reliable way, and there doesn't seem to be anything that does this on the OS level.
1) Use systemd; I've taken the time to learn it and the tooling it exposes for codifying process dependencies, how weak or strong they should be, and restart behaviors is thoroughly designed
2) Use Erlang/OTP - I went this route for an IoT product I worked on that used Erlang as the primary high-level "firmware", I codified the process tree and dependencies and custom supervisor restart behaviors for unix processes into Erlang/OTP's supervision tree; using Erlang process ports
The second approach has worked extremely well for me in the past but I would only recommend it if you're going to use a lot of Erlang to process output from the programs it spawns for you, which was the case for how I was using it at the time, the robust supervision of the processes it managed in a supervision tree was incidental to the primary use but extremely robust.
No it was not, it was a home-rolled solution. Nerves is pretty neat but at the time I wrote our solution it wasn't as mature as it is these days. I rolled our own buildroot + erlang release + "firmware" deployment system (including the cloud infrastructure and software). Very fun project.
Very cool! I think the platform has a pile of potential for small devices, I just haven't had a project yet where I feel like I could justify the learning curve. Getting BEAM running on a beaglebone via Nerves was pretty neat, but led to a bit of a "ok cool, now what?" situation.
Supervisord, upstart, systemd, monit and a bunch of other systems seem the obvious answers, so obvious that I probably misunderstood the question. Maybe you're looking for supervision trees or "start this after you started that"?
Do any of those tools to things like kill sibling processes to ensure a stable restart point? or allow deeply nested process trees with faults propagating upward if restarts aren't working?
There are plenty of things doing this at the operating system level, going back a quarter of a century.
One of the earliest was the System Resource Controller in IBM AIX 3.2 in 1992. Later that same decade, in the BSD and Linux worlds, came Daniel J. Bernstein's daemontools. A whole family of toolsets grew from that design, including Bruce Guenter's daemontools-encore, Gerrit Pape's runit, Laurent Bercot's s6, Wayne Marshall's perp, Adam Sampson's freedt, and nosh. There are also upstart, Linux systemd, MacOS launchd, and the Solaris Service Management Facility.
Daemon tools most certainly does not do what im talking about. It does blind restarts of single processes, not coordinated restarts and killing of collections of services.
You are voicing a long-standing fallacy. One builds the coördination on top of the base service management. The runit and perp way is to include the dependencies as calls to "sv check"/"perpok" in the run programs. s6-rc gives one "bundles". nosh has a mechanism of targets and "wants"/"conflicts" relationships.
Supervisors aren't just for restarting processes that crash, they're also responsible for cleaning up processes when the tree is being shutdown.
If you're using spawn to fire off a long-ish running process that is too small in scope to justify making a gen_<something>, you might be doing it wrong. You can easily affix the logic you need as a process on a supervisor which will (very importantly) help clean up those processes for you.
A common pattern is to use a high-level supervisor for starting up the primary gen_<whatever>'s with sub-supervisors responsible for short-lived and long-lived "small logic" children - you can also ask a supervisor to add and start a new child and to remove one, let the supervisors do the bookkeeping!
Supervisors make your process trees declarative and help you avoid process soup, even when the processes aren't that important. I highly recommend harnessing that as much as possible when designing non-trivial Erlang programs.
Interesting thoughts, thank you! I myself am guilty of spawning a few processes here and there that I consider not important enough to put into a supervisor.
I think the fact that Erlang provides an incentive for declaring what processes are managed upfront is a great benefit. I find myself planning a bit the structure of the supervision tree whereas in Go there is no benefit so it's much more ad hoc and messy.
Do check out the "blog post describing the design decisions" I mention in the README.md; you don't need to tell me every precise way in which it differs from Erlang and why Go can't be exactly like Erlang, I've already done that for you.
Like you, I have found it useful for structuring programs in many ways moreso than for reliability. That said, the reliability does sometimes end up being helpful.
By the way, am I the only one feeling that having to write the source code of the supervisor of a process is little more than boilerplate and the language should offer that by default as part? Obviously we want a supervisor to supervise the process and add it to our supervision tree.
It could be an argument of spawn, something like spawn :supervised, -> function end (maybe not the best syntax, it's only to show the idea). That would be a kind of anonymous supervisor. We should write our own supervisor code only to handle special cases.
you could, that is what Task.Supervisor do. But you then run into other problems. Who is supervising that supervisor? What are the strategy? What are the name? In which order to clean the process on shutdown ? When in the init should we spawn it?
In the end, it goes down to making a high level dynamic supervisor and putting everything under it. Which... is quite the opposite goal of a supervisor.
My idea is to provide sensible defaults and write custom code to handle everything else. What we do now is to write custom code every time. My point is that the custom code we write is almost always the same for every process in the application. It hurts productivity.
Let's use your list of questions as a checklist to validate the feasibility of this approach:
> Who is supervising that supervisor?
The supervisor of the process that spawned that process. Would that always work? Or the top supervisor of the application. Or your "high level dynamic supervisor" in the language runtime itself, which supervises only those anonymous supervisors and not every process.
> What are the strategy?
one-for-one should be a good default.
> What are the name?
Something automatically generated. A uuid. I'm not caring about that supervisor anyway. Would that work?
> In which order to clean the process on shutdown?
If I'm not caring about that process enough to write its supervisor explicitly maybe I don't care about its turn in the shutdown order. Anything easy to implement will do.
> When in the init should we spawn it?
Never, it's created only when the developer explicitly spawns the process. Well, we could reach the point when mix has a default task which spawns some processes with anonymous supervisors and there are no explicit supervisors left. This could change a little the nature of the language. I elaborate on this:
Erlang/Elixir processes are already somewhat equivalent to objects: they communicate with message passing, conceptually similar to Smalltalk and many object oriented languages (Ruby has send, JS has apply, etc). The main difference is that Erlang processes have their own CPU and are naturally concurrent, which is great. If supervision can be made so easy to disappear inside the language, perhaps processes become even more conceptually like objects, created when needed (a "spawn" is a "new") instead of carefully planned entities much like system administrators plan services in upstart/systemd. This could be good or bad, I don't know, but that every object has its CPU feels right.
BTW, google for Joe Armstrong Erlang is the only true object oriented language. It's not heresy :-)
Sounds like a great case for a supervisor macro at your shop. I doubt your defaults would be common enough to be part of the library though. . . Ordering and naming tend to matter.
I reply to myself days later. There was a language called Reia which did exactly that. Unfortunately the author dropped it in favor of Elixir even if in my opinion they are very different languages which would have had a different public. Reia could have been the language to move OO developers to functional, it's a much more difficult task for Elixir.
If you move this into the language itself, the language will grow. It is better to define the language as having the simplest possible API which will still work. Then build something like supervisors on top of that fabric.
Supervisors use the concept of a 'linked process' internally, which is the language-level construct which allows you to build supervisors in the first place. Another construct, the 'monitor' is also worth mentioning. It came later, and supercedes the link in some way.
Is it not cool to call these things worker threads anymore?
Nothing wrong with the article but I'm getting annoyed by the trend to rename constructs and common patterns in new languages. Wasting time learning about Y in language B only to realize that it's exactly the same as Z in language A which you already knew
These constructs were designed for erlang as far back at 1986. They also are not just worker threads, they do not have a similar concept in most languages.
Erlang processes are quite different from worker threads in other languages, since they're green threads (not OS threads). I'm not even sure if the expression "worker thread" was in common usage when the language was created.
I think you meant to reply that to me. But yeah, erlang is very old. It was originally designed from the ground up to run telephony switches and other telecom-related tasks.
32 comments
[ 1.6 ms ] story [ 51.2 ms ] threadhttps://en.wikipedia.org/wiki/IM_the_Supervisor
http://learnyousomeerlang.com/supervisors#from-bad-to-good
This is another terse (more reference-like) description of strategies:
http://erlang.org/doc/design_principles/sup_princ.html#strat...
1) Use systemd; I've taken the time to learn it and the tooling it exposes for codifying process dependencies, how weak or strong they should be, and restart behaviors is thoroughly designed
2) Use Erlang/OTP - I went this route for an IoT product I worked on that used Erlang as the primary high-level "firmware", I codified the process tree and dependencies and custom supervisor restart behaviors for unix processes into Erlang/OTP's supervision tree; using Erlang process ports
The second approach has worked extremely well for me in the past but I would only recommend it if you're going to use a lot of Erlang to process output from the programs it spawns for you, which was the case for how I was using it at the time, the robust supervision of the processes it managed in a supervision tree was incidental to the primary use but extremely robust.
* http://uwsgi-docs.readthedocs.io/en/latest/Systemd.html
One of the earliest was the System Resource Controller in IBM AIX 3.2 in 1992. Later that same decade, in the BSD and Linux worlds, came Daniel J. Bernstein's daemontools. A whole family of toolsets grew from that design, including Bruce Guenter's daemontools-encore, Gerrit Pape's runit, Laurent Bercot's s6, Wayne Marshall's perp, Adam Sampson's freedt, and nosh. There are also upstart, Linux systemd, MacOS launchd, and the Solaris Service Management Facility.
* http://jdebp.eu./FGA/daemontools-family.html
If you're using spawn to fire off a long-ish running process that is too small in scope to justify making a gen_<something>, you might be doing it wrong. You can easily affix the logic you need as a process on a supervisor which will (very importantly) help clean up those processes for you.
A common pattern is to use a high-level supervisor for starting up the primary gen_<whatever>'s with sub-supervisors responsible for short-lived and long-lived "small logic" children - you can also ask a supervisor to add and start a new child and to remove one, let the supervisors do the bookkeeping!
Supervisors make your process trees declarative and help you avoid process soup, even when the processes aren't that important. I highly recommend harnessing that as much as possible when designing non-trivial Erlang programs.
Do check out the "blog post describing the design decisions" I mention in the README.md; you don't need to tell me every precise way in which it differs from Erlang and why Go can't be exactly like Erlang, I've already done that for you.
Like you, I have found it useful for structuring programs in many ways moreso than for reliability. That said, the reliability does sometimes end up being helpful.
By the way, am I the only one feeling that having to write the source code of the supervisor of a process is little more than boilerplate and the language should offer that by default as part? Obviously we want a supervisor to supervise the process and add it to our supervision tree.
It could be an argument of spawn, something like spawn :supervised, -> function end (maybe not the best syntax, it's only to show the idea). That would be a kind of anonymous supervisor. We should write our own supervisor code only to handle special cases.
In the end, it goes down to making a high level dynamic supervisor and putting everything under it. Which... is quite the opposite goal of a supervisor.
Let's use your list of questions as a checklist to validate the feasibility of this approach:
> Who is supervising that supervisor?
The supervisor of the process that spawned that process. Would that always work? Or the top supervisor of the application. Or your "high level dynamic supervisor" in the language runtime itself, which supervises only those anonymous supervisors and not every process.
> What are the strategy?
one-for-one should be a good default.
> What are the name?
Something automatically generated. A uuid. I'm not caring about that supervisor anyway. Would that work?
> In which order to clean the process on shutdown?
If I'm not caring about that process enough to write its supervisor explicitly maybe I don't care about its turn in the shutdown order. Anything easy to implement will do.
> When in the init should we spawn it?
Never, it's created only when the developer explicitly spawns the process. Well, we could reach the point when mix has a default task which spawns some processes with anonymous supervisors and there are no explicit supervisors left. This could change a little the nature of the language. I elaborate on this:
Erlang/Elixir processes are already somewhat equivalent to objects: they communicate with message passing, conceptually similar to Smalltalk and many object oriented languages (Ruby has send, JS has apply, etc). The main difference is that Erlang processes have their own CPU and are naturally concurrent, which is great. If supervision can be made so easy to disappear inside the language, perhaps processes become even more conceptually like objects, created when needed (a "spawn" is a "new") instead of carefully planned entities much like system administrators plan services in upstart/systemd. This could be good or bad, I don't know, but that every object has its CPU feels right.
BTW, google for Joe Armstrong Erlang is the only true object oriented language. It's not heresy :-)
Supervisors use the concept of a 'linked process' internally, which is the language-level construct which allows you to build supervisors in the first place. Another construct, the 'monitor' is also worth mentioning. It came later, and supercedes the link in some way.
Nothing wrong with the article but I'm getting annoyed by the trend to rename constructs and common patterns in new languages. Wasting time learning about Y in language B only to realize that it's exactly the same as Z in language A which you already knew