I'm missing the crucial design bits which would influence embeddability:
Do pointers move? How is the GC? How is the FFI? What are the FFI callback limitations?
With FFI callbacks, calling back into Lua functions and using Lua data you need either refcounting or a slow mark & sweep. Or just for FFI accessible data. Of you provide an API for this data to propagate moved pointers back to C.
There's a collection of c-exposed values called the api stack (not to be confused with the function stack). The stack keeps them from being collected, so there's no need for ref counting. OP writes about this.
OP doesn't write much about the GC itself, but I can tell you a little from my dayjob-using-lua experience. It's mark'n'sweep, but (i) it's narrowly tailored for lua tables, not general purpose allocations, and (ii) it's incremental, so you can cap individual pause times and amortise (imp for games).
> I'm missing the crucial design bits which would influence
> embeddability:
Your points are crucial but mundane. For people who have done extension and embedding with other languages like Perl, Python, Tcl, etc, what really matters for productivity and maintenance are the more subtle details.
For example, in Lua you can create _instances_ of a wrapped C function that has upvalues, which are analogous to closed-over variable slots. In other words, you can implement C callbacks as closures, a more functional-oriented approach which also generally more performant than many alternatives. In many languages to associate instance data with wrapped functions (as opposed to wrapped methods where you're always passed an instance object) you're stuck using some sort of global registry or other coarse mechanism. Lua supports many such alternatives, as well, including some novel alternatives. But the upvalue mechanism is particularly elegant and useful.
Another useful feature, which is discussed by that paper, is how Lua designed coroutines--the semantics, the implementation, and the public C API. Languages like JavaScript, Perl, and Python don't support stackful coroutines (i.e. being table to yield a coroutine from any arbitrary function, not simply the function initially invoked/resumed). They neither support them within the language nor outside the language for use by C, and with exception (a brilliantly complex Perl XS module comes to mind) you can't even begin to fake this. This is because implementations of those languages conflate the host C stack with the logical call stack of the language. And because you can neither grow the C stack nor easily yield and resume, these languages simply eschew stackful coroutines altogether. The implementation restricts the semantics they can implement at the language level.
Lua not only supports stackful coroutines, but it provides a mechanism and API for resuming and yielding a Lua/C function, all in strict ISO C. This was more limited in Lua 5.1, and made more powerful in Lua 5.2. In any event, this mechanism makes it easy to, among other things, write asynchronous I/O frameworks in Lua that behave like green threading. You don't need to pepper your code with async/await everywhere, nor do you need to create trampolines in your FFI binding code to get yield semantics from the C callback, which results in messy and complex modules.
Lua is awesome as both an extension and embedding language because it was built from the ground keeping those requirements and constraints in mind. The only serious concession most languages make for integration with C is either to avoid moving pointers, or providing a special type to wrap objects visible in C. All the rest occurs as an after thought, and while the results are often acceptable, they're just not remotely as convenient or seamless.
The biggest exception to my points above are Tcl. I find Lua more elegant and powerful, both the language and the C APIs, but Tcl deserves its due.
I really like Lua but the future seems uncertain to me at this point. There are more and more forks appearing, I don't know if that's a sign of health or entropy.
Pointers do not ever 'move'. You also should never store variables (to userdata, memory chunks, strings, etc.) by pointers or anything like that according to documentation because they might be collected at any time. To stuff like tables or values you never ever get a pointer and having a pointer to a "Lua value" would make no sense because it's a small struct with just the type number (in case of nil that's all that matters actually) and then either an inline value (light userdata, bool, number, etc.) or a pointer to the actual GC object (thread, string, table, function, internal data structures, etc.). Lack of exposing internal data structures (there is Buffer in luaL but it's the only one I can think of and it's so simple and just to collect few results together it doesn't matter) is probably the reasons why LuaJIT is ABI and API compatible with Lua 5.1 despite implementing stuff very differently internally (like values in NaN tagged doubles to keep them in a 64 bit boundary instead of just placing fields normally in structs and letting compiler figure it out like stock Lua does).
There is a feature in luaL part of the C API called 'refs' but that's basically storing stuff in a table array and addressing it by ints and it's done all using the C API itself, it's not a VM or language feature.
There is also a weak reference system (where table can be marked to have key, value or both be 'weak' and allow collection if there are no 'strong' references anywhere to these objects) and that is part of the VM itself (GC specifically, the table processing code) and has some caveats with stuff like strings (they are interned in Lua) and it doesn't remove intelligently form tables serving as arrays.
To call a function from C you push it and its arguments onto the stack (of the main thread or the coroutine) and execute it with one of API functions (depending on what you're trying to do)[1], to call it in Lua you 'just' call it or pass it to pcall or a when making a coroutine.
A C function exposed to Lua like that must implement its 'protocol'[2] which is getting and returning arguments by interacting with the lua state pointer so it's an extra effort to wrap them or you can use a third party library (there are some, especially C++ ones which use templates to automatically wrap stuff, but I don't use any myself).
C and Lua functions are indistinguishable in most cases, they are called the same way, etc. but sometimes they differ, i.e. Lua functions can be dumped with string.dump.
There is no FFI (as there is in C#, JNI, etc.) as such. LuaJIT has an FFI that parses C declarations and it's quite nice IMO. There are also third party FFI libs but I know nothing about them. Using that also loses you all the nice to haves that Lua has like multiple return values (it's customary to return false or nil and an error message string on errors from many functions and it's impossible to write an iterator without multiple return values), interacting with lua state in the function, arg type check calls with automatic errors, etc.
The GC is a mark and sweep one, in 5.0 it was a stop the world one - two color, from 5.1 onwards it's incremental tri color with forward and back write barriers(see [0] for an explanation of what that means and how it works). In 5.2 there was also (off by default) generational one. I have no idea how that one worked and it was removed in 5.3. LuaJIT also uses mark and sweep from 5.1 but with improvements and for 3.0 there was supposed to be a quad color one. You of course have full control over it (stop, start, adjust speed, force a full cycle or a single step, etc.) to complete right now) from Lua and C via the right function. It does nothing fancy like compaction or copying/moving.
You are also responsible for the memory allocation (in 64 bit LuaJIT it's taken over and attempting to do that is an error) and the default alloc is just a malloc wr...
I don't really follow the development of Lua in any way.
Lua 5.3 was very underwhelming to me, I didn't care for the ints and bit operators at all (and before 5.3 the one number type was a selling point) and it's annoying how there is such a clear strong split between 5.3 and LuaJIT because it effectively finalized the fork that 5.2 has sort of started. Some things (like Love2D, OpenResty and other performance conscious stuff) stuck with LuaJIT so basically a 5.1 on steroids with few 5.2 features. 5.1 also had the killer feature of incremental GC and came at a nice time and was the main Lua for almost 6 years so it stuck into gamedev a lot (which is where I picked Lua from, my hobbist gamedeving).
I also stick to 5.1 by default in my hobby gamedev to be able to bring in LuaJIT at any time and ignore 5.2 and 5.3 pretty much.
My thesis was about history of Lua a bit (rehashing what's on the main website plus few Roberto's articles pretty much) and then a lot about how stock Lua 5.1 is implemented (most of it is 1:1 transferable to other stock Luas) like how bytecode is executed, how does GC work, what does it mean Lua is register VM and not stack one, how are internal structures laid out, how functions and closures and threads work, etc. It's (official) English title was "Analysis of code of original implementation of Lua 5.1" but it is written in Polish ("Analiza kodu oryginalnej implementacji jezyka Lua 5.1"). I don't think it's available anywhere. That was my engineering (inżynier) thesis which I defended in February and I'm now doing a master thesis but it's purely mathematical CS stuff and not related to Lua at all.
As an aside, there’s stuff on the lua wiki about sandboxing lua, how would you go about sandboxing lua to limit heap, cpu cycles, and os level api calls? For lua 5.1? For luajit? An example use case would be letting users upload scripts to run in game?
I never did that actually and it seems involved, hard and very limiting for what such a script could do.
I dislike talking about touchy high risk stuff like that in general since someone might think what I say is a list of things to do to be 100% secure and get burned.
What it says on [0] is a good start I guess: only textual code input, no bytecode (5.1 has bytecode verification but it was removed in 5.2 because it had flaws and gave false sense of security, there were examples of breaking stuff up with malicious handcrafted bytecode), white listing.
To not allow something (like os or debug libs) to be used just don't include it in the environment for that used supplied function (it does restrict you quite a bit though I guess...).
For memory you could use your own allocator and enforce a limit there (and probably set your own atpanic function in 5.1 and longjmp out of it because you might kill the entire program with the default one when Lua won't be able to alloc the error message itself, in 5.2 and 5.3 it's preallocated in the VM so that won't happen).
To prevent runaway Lua loops you can use the count debug hook and throw a Lua error in it, pcall will get that (just remember to reset it between calls because it doesn't reset to 0 on its own). Preventing runaway C loops is up to making sure you don't expose something that can get stuck. You could also set count to something low or use line debug hook and check the time in it and error out if the code took too long already but it might slow the code down too badly (depends on what you do I guess).
That stuff (mem alloc, debug hooks, panic, environments, etc.) is in the docs.
Long strings can also DoS the hashing algorithm and that might totally kill performance with tables, it's a trivial change to fix that yourself or use one of the patches[1]. It came up on the mailing list too. I'm sure it happens on 5.1, not sure about latest 5.2 and 5.3 right now and I don't have time to look into it.
LuaJIT doesn't also let you set the allocator on 64 bit and has some other issues with out of memory errors[2] (I don't know what they are exactly about). I guess it's out of the question if you're really paranoid (stock Lua 5.1 is also way easier to read, modify and compile IMO). I'm not very familiar with LuaJIT.
I used Lua 5.2 (with 5.1 compatibility enabled) to embed in a C framework for building and executing data flow graphs. I implemented C .h header to JSON and back, and also an API wrapper generator, which really helped things along. The basic features include: co-operative FSM scheduler (using coroutines for sleep etc), structured FSM Engine, structured data pipes (stucture defined in C headers and piped in binary, but to Lua they look just like Lua tables syntactically), Event generation and propagation mechanism, ability to record and play back data pipe streams, fine grained MIDI parser and generators, and a bunch of other details. The embedding mechanism is generalised, and so there is alongside the Lua scripting a similarly complete FORTH drop in language, for those who might be interested. I think it's a pretty neat system, and works quite well.
The system had some turbulence during its inception in a now defunct very small company, but it all reverted to me. I would like to get it up on Github before I pass on (fairly soon, I fear), but need help fixing all the file headers, and some minor bugfixing. Volunteers? I have a minimal html description file if you'd like. But a build help file is needed.
12 comments
[ 3.6 ms ] story [ 38.0 ms ] threadDo pointers move? How is the GC? How is the FFI? What are the FFI callback limitations?
With FFI callbacks, calling back into Lua functions and using Lua data you need either refcounting or a slow mark & sweep. Or just for FFI accessible data. Of you provide an API for this data to propagate moved pointers back to C.
OP doesn't write much about the GC itself, but I can tell you a little from my dayjob-using-lua experience. It's mark'n'sweep, but (i) it's narrowly tailored for lua tables, not general purpose allocations, and (ii) it's incremental, so you can cap individual pause times and amortise (imp for games).
For example, in Lua you can create _instances_ of a wrapped C function that has upvalues, which are analogous to closed-over variable slots. In other words, you can implement C callbacks as closures, a more functional-oriented approach which also generally more performant than many alternatives. In many languages to associate instance data with wrapped functions (as opposed to wrapped methods where you're always passed an instance object) you're stuck using some sort of global registry or other coarse mechanism. Lua supports many such alternatives, as well, including some novel alternatives. But the upvalue mechanism is particularly elegant and useful.
Another useful feature, which is discussed by that paper, is how Lua designed coroutines--the semantics, the implementation, and the public C API. Languages like JavaScript, Perl, and Python don't support stackful coroutines (i.e. being table to yield a coroutine from any arbitrary function, not simply the function initially invoked/resumed). They neither support them within the language nor outside the language for use by C, and with exception (a brilliantly complex Perl XS module comes to mind) you can't even begin to fake this. This is because implementations of those languages conflate the host C stack with the logical call stack of the language. And because you can neither grow the C stack nor easily yield and resume, these languages simply eschew stackful coroutines altogether. The implementation restricts the semantics they can implement at the language level.
Lua not only supports stackful coroutines, but it provides a mechanism and API for resuming and yielding a Lua/C function, all in strict ISO C. This was more limited in Lua 5.1, and made more powerful in Lua 5.2. In any event, this mechanism makes it easy to, among other things, write asynchronous I/O frameworks in Lua that behave like green threading. You don't need to pepper your code with async/await everywhere, nor do you need to create trampolines in your FFI binding code to get yield semantics from the C callback, which results in messy and complex modules.
Lua is awesome as both an extension and embedding language because it was built from the ground keeping those requirements and constraints in mind. The only serious concession most languages make for integration with C is either to avoid moving pointers, or providing a special type to wrap objects visible in C. All the rest occurs as an after thought, and while the results are often acceptable, they're just not remotely as convenient or seamless.
The biggest exception to my points above are Tcl. I find Lua more elegant and powerful, both the language and the C APIs, but Tcl deserves its due.
There is a feature in luaL part of the C API called 'refs' but that's basically storing stuff in a table array and addressing it by ints and it's done all using the C API itself, it's not a VM or language feature.
There is also a weak reference system (where table can be marked to have key, value or both be 'weak' and allow collection if there are no 'strong' references anywhere to these objects) and that is part of the VM itself (GC specifically, the table processing code) and has some caveats with stuff like strings (they are interned in Lua) and it doesn't remove intelligently form tables serving as arrays.
To call a function from C you push it and its arguments onto the stack (of the main thread or the coroutine) and execute it with one of API functions (depending on what you're trying to do)[1], to call it in Lua you 'just' call it or pass it to pcall or a when making a coroutine.
A C function exposed to Lua like that must implement its 'protocol'[2] which is getting and returning arguments by interacting with the lua state pointer so it's an extra effort to wrap them or you can use a third party library (there are some, especially C++ ones which use templates to automatically wrap stuff, but I don't use any myself).
C and Lua functions are indistinguishable in most cases, they are called the same way, etc. but sometimes they differ, i.e. Lua functions can be dumped with string.dump.
There is no FFI (as there is in C#, JNI, etc.) as such. LuaJIT has an FFI that parses C declarations and it's quite nice IMO. There are also third party FFI libs but I know nothing about them. Using that also loses you all the nice to haves that Lua has like multiple return values (it's customary to return false or nil and an error message string on errors from many functions and it's impossible to write an iterator without multiple return values), interacting with lua state in the function, arg type check calls with automatic errors, etc.
The GC is a mark and sweep one, in 5.0 it was a stop the world one - two color, from 5.1 onwards it's incremental tri color with forward and back write barriers(see [0] for an explanation of what that means and how it works). In 5.2 there was also (off by default) generational one. I have no idea how that one worked and it was removed in 5.3. LuaJIT also uses mark and sweep from 5.1 but with improvements and for 3.0 there was supposed to be a quad color one. You of course have full control over it (stop, start, adjust speed, force a full cycle or a single step, etc.) to complete right now) from Lua and C via the right function. It does nothing fancy like compaction or copying/moving.
You are also responsible for the memory allocation (in 64 bit LuaJIT it's taken over and attempting to do that is an error) and the default alloc is just a malloc wr...
Also, what is your thesis?
Lua 5.3 was very underwhelming to me, I didn't care for the ints and bit operators at all (and before 5.3 the one number type was a selling point) and it's annoying how there is such a clear strong split between 5.3 and LuaJIT because it effectively finalized the fork that 5.2 has sort of started. Some things (like Love2D, OpenResty and other performance conscious stuff) stuck with LuaJIT so basically a 5.1 on steroids with few 5.2 features. 5.1 also had the killer feature of incremental GC and came at a nice time and was the main Lua for almost 6 years so it stuck into gamedev a lot (which is where I picked Lua from, my hobbist gamedeving).
I also stick to 5.1 by default in my hobby gamedev to be able to bring in LuaJIT at any time and ignore 5.2 and 5.3 pretty much.
My thesis was about history of Lua a bit (rehashing what's on the main website plus few Roberto's articles pretty much) and then a lot about how stock Lua 5.1 is implemented (most of it is 1:1 transferable to other stock Luas) like how bytecode is executed, how does GC work, what does it mean Lua is register VM and not stack one, how are internal structures laid out, how functions and closures and threads work, etc. It's (official) English title was "Analysis of code of original implementation of Lua 5.1" but it is written in Polish ("Analiza kodu oryginalnej implementacji jezyka Lua 5.1"). I don't think it's available anywhere. That was my engineering (inżynier) thesis which I defended in February and I'm now doing a master thesis but it's purely mathematical CS stuff and not related to Lua at all.
I dislike talking about touchy high risk stuff like that in general since someone might think what I say is a list of things to do to be 100% secure and get burned.
What it says on [0] is a good start I guess: only textual code input, no bytecode (5.1 has bytecode verification but it was removed in 5.2 because it had flaws and gave false sense of security, there were examples of breaking stuff up with malicious handcrafted bytecode), white listing.
To not allow something (like os or debug libs) to be used just don't include it in the environment for that used supplied function (it does restrict you quite a bit though I guess...).
For memory you could use your own allocator and enforce a limit there (and probably set your own atpanic function in 5.1 and longjmp out of it because you might kill the entire program with the default one when Lua won't be able to alloc the error message itself, in 5.2 and 5.3 it's preallocated in the VM so that won't happen).
To prevent runaway Lua loops you can use the count debug hook and throw a Lua error in it, pcall will get that (just remember to reset it between calls because it doesn't reset to 0 on its own). Preventing runaway C loops is up to making sure you don't expose something that can get stuck. You could also set count to something low or use line debug hook and check the time in it and error out if the code took too long already but it might slow the code down too badly (depends on what you do I guess).
That stuff (mem alloc, debug hooks, panic, environments, etc.) is in the docs.
Long strings can also DoS the hashing algorithm and that might totally kill performance with tables, it's a trivial change to fix that yourself or use one of the patches[1]. It came up on the mailing list too. I'm sure it happens on 5.1, not sure about latest 5.2 and 5.3 right now and I don't have time to look into it.
LuaJIT doesn't also let you set the allocator on 64 bit and has some other issues with out of memory errors[2] (I don't know what they are exactly about). I guess it's out of the question if you're really paranoid (stock Lua 5.1 is also way easier to read, modify and compile IMO). I'm not very familiar with LuaJIT.
[0] - http://lua-users.org/wiki/SandBoxes
[1] - http://lua-users.org/wiki/HashDos
[2] - http://luajit.org/status.html
The system had some turbulence during its inception in a now defunct very small company, but it all reverted to me. I would like to get it up on Github before I pass on (fairly soon, I fear), but need help fixing all the file headers, and some minor bugfixing. Volunteers? I have a minimal html description file if you'd like. But a build help file is needed.