I think it's safe to say there is just plain no way to invalidate it. Even if it were hacked together, it would be unusably dangerous for even trivial cases.
Module loading is something that can be added to a language later, but I suspect safe module unloading is something that has to go in from day one to ever work right. Even in languages that support it (Erlang) it's got a tricky list of caveats to keep track of that strike me as fairly essential to the problem, rather than accidental (in the Fred Brooks sense of the terms).
For those who wonder why: some of the things you have to make sure of are if you want to implement this correctly:
- no thread is running code in the plugin.
- no thread has a return address on the stack somewhere in the plugin.
- no variable stores a pointer pointing into the plugin (This includes objects whose destructor is implemented in the plugin)
- no signal handler points into the plugin.
- no kernel thread is reading the plug-ins memory.
In addition there are race conditions to take care of (e.g. thread T unloads a plugin that thread U is loading)
With a 64 bit address space, I guess you can get around most if not all problems by mapping the memory that was mapped to the plugin as "can't read, can't write, can't execute", so that you crash hard if somebody tries to access it, but if you do that, you can just as well keep the code around.
Go is garbage-collected, so it likely already has a lot of the logic to do this, but even then, combining the functionality to get a robust implementation of plugin unloading is a lot of work.
Curious what it would take (if at all possible?) to load such a .so from C. Because that would open the door for writing Go libs for all your favorite interpreted languages, web servers, etc.
It's a WIP and I haven't had the time to complete it, but here it is. You can load modules, handle redis commands and return responses, but you can't access redis itself yet.
I see an API fail coming. It seems plugin.Load takes the filename WITH extension where it shouldn't. Okay, right now it seems it's only supported on Linux but it ends up in Windows, you'd have to patch code to remove the .so. Mono handles it pretty well, the extension is optional.
I don't think so. I think it is better that the file naming is handled in the application. It is less restrictive, e.g. you can use custom extensions this way.
Consider another instance where file names are automatically mangled: GCC linking with libraries with the switch -lfoo. That has honestly caused me no end of problems. It would have been better if they required an explicit -llibfoo.so from the start.
Go already has several easy methods for OS-specific code if you want.
Not sure it should be be part of this package, but it's trivial to write a function that exposes something like "SharedLibraryFile(libname string) string". Depending on the OS it just formats the default shared object name.
For extra runtime speed you could build OS specific versions of that function using build flags, so for each target OS it has one implementation and you have no switch/case or anything in runtime.
Although this can be abused, the use case is not external dependencies IMO, but rather extending systems. Think of a web server etc. Each plugin is still a static binary that is self contained, but you can load capabilities into the server in runtime.
This is not like dynamic linking hell you can get yourself into when writing C.
But isn't that exactly how dynamic linking works? By loading a DLL at runtime? The only issues I've ever had with dynamic linking are at _runtime_, where XXX.dll can't be found.
yes, my above comment wasn't phrased that well. but do distinguish between a plugin system where you load so's or DLLs to extend an app, and an application that has dynamic linking against them, where the linker handles this.
So basically something like apache modules - either you can load the module or not - vs something like "cannot find symbol foo in foo.so.0.1."
Standard library packages are not the same thing as the language and are not mentioned in the Go specification. The standard library is just a collection of packages that implement common functionalities that might otherwise lead to multiple competing implementations (like hyper, tiny_http, rotor-http, and solicit in Rust). Including a plugin mechanism is not an endorsement of that style of architecture, but simply an acknowledgement that there are programs that will need plugins.
Standard library packages are not the same thing as the language and are not mentioned in the Go specification. The standard library is just a collection of packages that implement common functionalities that might otherwise lead to multiple competing implementations (like hyper, tiny_http, rotor-http, and solicit in Rust). Including a plugin mechanism is not an endorsement of that style of architecture, but simply an acknowledgement that there are programs that will need plugins.
I agree with this comment from Reddit four days ago:
> The inability to unload a plugin defeats the use I would have for this, unfortunately. A long running service with plugins may want to load new, updated versions of an existing plugin. Not being able to unload the old version, ends up creating stale references which will keep piling up for the lifetime of the process. As I understand it, this is essentially a memory leak. [...]
Amazing, so this is essentially a high performance alternative to Go scripting. One can already include the go build toolchain in a binary, right? Having JIT is only small steps away.
Disagree with the problem of unloading plugins. From the sizes of Go binaries and memory sizes, one could easily load 10k+ plugins before the process needed a restart. Would be nice to have GC, but having plugins is such an increased benefit. To me this sounds like, Git is crap because you can't unload stored objects.
32 comments
[ 2.9 ms ] story [ 73.6 ms ] threadWhy not?
Module loading is something that can be added to a language later, but I suspect safe module unloading is something that has to go in from day one to ever work right. Even in languages that support it (Erlang) it's got a tricky list of caveats to keep track of that strike me as fairly essential to the problem, rather than accidental (in the Fred Brooks sense of the terms).
Hence why Java and .NET restrict it to Class Loaders/AppDomains.
- no thread is running code in the plugin.
- no thread has a return address on the stack somewhere in the plugin.
- no variable stores a pointer pointing into the plugin (This includes objects whose destructor is implemented in the plugin)
- no signal handler points into the plugin.
- no kernel thread is reading the plug-ins memory.
In addition there are race conditions to take care of (e.g. thread T unloads a plugin that thread U is loading)
With a 64 bit address space, I guess you can get around most if not all problems by mapping the memory that was mapped to the plugin as "can't read, can't write, can't execute", so that you crash hard if somebody tries to access it, but if you do that, you can just as well keep the code around.
Go is garbage-collected, so it likely already has a lot of the logic to do this, but even then, combining the functionality to get a robust implementation of plugin unloading is a lot of work.
https://github.com/RedisLabs/rgm/blob/master/example/module....
" 5 // +build linux,cgo"
The Windows version isn't there yet - it could encapsulate that logic.
Consider another instance where file names are automatically mangled: GCC linking with libraries with the switch -lfoo. That has honestly caused me no end of problems. It would have been better if they required an explicit -llibfoo.so from the start.
Go already has several easy methods for OS-specific code if you want.
For extra runtime speed you could build OS specific versions of that function using build flags, so for each target OS it has one implementation and you have no switch/case or anything in runtime.
I would go so far as to say an absolute path should be required.
What's next? Generics?
This is not like dynamic linking hell you can get yourself into when writing C.
So basically something like apache modules - either you can load the module or not - vs something like "cannot find symbol foo in foo.so.0.1."
In my Go code, I've just used subprocesses for that. Stdin & stdout, plus a simple serialisation format, and I'm done.
Not suitable for high performance stuff, of course, but good enough for my needs.
No, but they will eventually need to handle plugin versions.
Because that doesn't sound very reasonable to me, even after ignoring that plugins aren't really dependencies anyway.
> The inability to unload a plugin defeats the use I would have for this, unfortunately. A long running service with plugins may want to load new, updated versions of an existing plugin. Not being able to unload the old version, ends up creating stale references which will keep piling up for the lifetime of the process. As I understand it, this is essentially a memory leak. [...]
> https://www.reddit.com/r/golang/comments/53adu8/a/d7rmmco
There is also this project by Hashicorp: https://github.com/hashicorp/go-plugin
Disagree with the problem of unloading plugins. From the sizes of Go binaries and memory sizes, one could easily load 10k+ plugins before the process needed a restart. Would be nice to have GC, but having plugins is such an increased benefit. To me this sounds like, Git is crap because you can't unload stored objects.