Ask HN: Practical examples of runtime modified software

Not smalltalk, but my KaithemAutomation software is pure Python and heavily built around runtime modification(https://github.com/EternityForest/KaithemAutomation), and is designed to run for months, although it's not meant to be modified literally while running production(Not that that stops us from making last minute fixes...).

The big problem with it in Python is state. If you have a reference to something, and that something changes, and you have a reference to the old one, that should be a singleton, you are probably in an unhappy position.

The only way it can work in real life(In Python at least) is if you carefully design for it, and don't pass around objects and callbacks and stuff.

I have gone through several iterations of abstractions trying to make updatable objects and such to hide this from you, and none were very good.

I eventually settled on a message bus, and a data structure I'm calling a Tag Point adapted from the SCADA industry, which is like a variable, but you can subscribe to it, and spy on the value from the web UI, it stays around as long as there is a reference to it, and it's guaranteed that any tags with the same name are the same object, and a bunch of other stuff.

Files of code are essentially meant to be used like stateless microservices if you expect to update one, and if you want to access shared stuff, you make sure to not hold a reference to it.

A lot was(and some still is) based on weak references, and those can trouble and should probably not be relied on for correctness if possible.

They aren't the worst things though. I have code to run a few GC sweeps when a file is deleted, and it is extremely rare that anything stays around when it shouldn't.

Good enough for development, good enough for emergency fixes, not the best for regular updates to running systems, although it's 99.9% fine, and I can't say I really worry about it(But future versions should be more deterministic and more suited for live edits as a regular practice).

Erlang seems to like its functional-ness and I imagine that's a huge asset. Language level support is definitely a good thing.

Random unstructured code doesn't seem to work well with live updates.

If you are working in a general language like Python you really want to have your engine always know exactly what's going on, what subscriptions come from what module and which function is replacing what, etc.

You want to deterministically always be able to list any changes that a module made to anything else(Like subscribing to a function), and undo them.

But... weak refs work well enough.

The classic solution to tech problems is to reboot, live updates are kind of the opposite of that. Your new code has to perfectly pick up with what you keep from the old state, and if the old state is invalid somehow you have to deal with that too.

I've never heard of smalltalk as a live update language(In the Erlang telephone exchange sense), just that you can do interactive development in it, which is a lot easier.

I think if I intended to seriously to true live updates(Like phone exchanges not dropping active calls), I would really appreciate tools made for that.

Kaithem also does have a module with a visual script explicitly meant for changes in production. It's very limited and opinionated, essentially a state machine with event triggered actions attached to states, and variables at the state machine level.

That kind of transition rule system for simple tasks is very easy to live update. State is well separated from rules, and rules are simple and easy to parse programmatically, to do things like clean up after yourself if an event listener needs resources.

But just about any interpreted language works for interactive development.

A lot of people like FORTH for that, which I don't have any interest in learning but some love it.

In classical Smalltalk, there is no difference between development time and runtime. Everything, including system code modification, needs to be done in runtime. There is no difference between a running program and a development environment so writing a program in Smalltalk is a process of self-modification of the development environment to extend it to provide the functionality you want. So, for Smalltalk, it is essential to provide tools to allow that like objects identity swapping, migration of living instances to a new class definition, objects enumeration etc. The basic conceptual idea behind it is to look at Smalltalk as a living organism that is able to grow and evolve without the need for restarts/bootstrap.

You may try Pharo (https://pharo.org/features) to experience it yourself.

Perhaps you could say how you think it works in practice.

While I was at WhatsApp, we used hot loading for most server updates on our Erlang cluster(s). There's tricks and traps, but if you do things right, it basically works and you can fix a lot of bugs right away without dropping connections.

The way Erlang works is any module can have two versions loaded; you can call them current and old. A fully qualified function call (Module:Function) calls into the current module, but non-fully qualified calls (like function calls within the same module, and some function references) will stay with the version they were made from, so if you're in what was current code, and a new version is loaded, current becomes old and non-fully qualified calls go to old.

You can only have those two versions though, if you load a third, any processes that are in the old version will be killed before current becomes old and new becomes current. There's ways to check if there are any processes 'lingering' in old code, and design techniques to reduce the chances of that; the easiest is making sure your loop code calls ?MODULE:loop(...) instead of loop(...), and most of the OTP looping constructs (gen_X) do that, or even if they weren't reloadable, they're doing the looping and calling into your code with fully qualified calls anyway. Tail Call Optimization is the easiest way to get the older code off the bottom of your stack, although there are others.

Of course, you can do hot code loading in lots of other languages too. In C, you can do it intentionally with dlopen/dlsym, or if your OS uses mmaped libraries, by editing libraries while they're loaded (whoops! use install instead of cp to avoid doing that by accident).