impressive work! i suppose it implements a subset and not the full language? would be nice to see some benchmarks. everyone claims super fast nowadays. i have no doubt though, since it's cpp.
That's not all that's not supported. Through poking at the live demo, I've found that at least slots, kw-only args, some basic syntax elements, decorators, generators, generator comprehensions and async are additionally not.
Exceptions are marked as supported … but `raise Exception()` is a syntax error? (Re-raise and binding exceptions also fails to parse.)
I've been waiting a long time for MicroPython to become usable as an embedded script interpreter -- it can't be run asynchronously [1] so scripts can't do waits (aside from that, MicroPython is amazingly full-featured). CPython can't either, which is why the Stackless Python fork was created, which is used in games such as EVE Online. But that's far too heavy for my tastes.
So I was hoping that PocketPy could do that, but after looking around I can't find any suggestion it's possible, and the API looks very sparse. I'm very happy to see a new Python interpreter built for embedding, but I can't use it. Maybe it can be added easily enough. Keep it up!
If you want to see a scripting language designed from the ground up for embedding, Lua and Squirrel IMO have the most complete APIs (though not the most efficient or easy to use) [2][3], and LuaJIT has even better lua<->C bridging. I've looked at the APIs of many other small scripting languages, and haven't found any others that can do everything that these can.
This was of course available though inline assembly or code generators from the beginning. So C21 is more complicated for the extra looks-like-prepreprocessor-but-probably-isn't syntax with no increase in expressivity.
If it's implemented in the preprocessor - e.g. expanding to a comma delimited char array - then the performance justification for adding it to the compiler is lost.
However you might be using "preprocessor directive" in the language semantic phase sense, not in the "preprocessor does it" sense.
Have you ever looked at newLISP [0]? It's been on my radar for a while, but I've not yet had the chance to try it. Its FAQ's [1] item 18 lists a few projects it's been used as an embedded language in.
Thanks, I'll give it a look when I have time. There's certainly a lot of embeddable lisps due to their simplicity, but I've never really given any serious consideration.
What's the performance like compared to Lua? Since Lua is the standard here, this feels like nonstarter unless the performance is similar, even with the friendlier python syntax.
to my mind the differences in syntax between python and lua are like two white, long-sleeved, button-up, collared shirts, but they have different buttons.
The python shirt has a combo of buttons, zippers, Velcro, and drawstrings in different places and with a long history about why it's there and why it made sense at the time but no one would do it that way again.
If the author comes around, first of all congratulations on what seems like a really cool project. Second, I am curious as to whether the lack of 'star unpack' is due to any real blocker, or more of a 'not done yet'/'not prioritized'?
That is great information, but perhaps it would be even more interesting with a comparison focusing on supported features, standard library support, and so on.
MicroPython [1] is designed for "real" embedding, i.e. in the "embedded development" sense, on microcontrollers and so on and seems quite successful in that niche. Those are often a smaller target than a game running on either a phone/tablet or PC, which would seem to kind of invert things.
The other comparison would be memory and disk/ROM usage.
Also, micropython, in its hundreds of thousands of lines of code, supports quite a number of microcontrollers and boards; I suspect the core language is smaller, or counting lines used for a single build target would drop it considerably.
Micropython is made to run on dozens of boards, each with a nontrivial board support package (the platform-specific glue code to talk to hardware). A better comparison would be looking exclusively at micropython's unix/x86 platform specifically.
Hmm, wish this had been built as a PocketPy -> LLVM compiler. While the compiler might not have been lightweight, the final emitted binaries would have been superfast.
Option 1 is mandating mypy-style type annotations (with type inference where there’s no ambiguity).[1]
Option 2 is using C++-style vtables, and putting each value (including primitives) inside an object, with each object inheriting from a common object superclass that has stub methods like __add__, __sub__, __mul__, __div__, __str__, __repr__, etc, etc. You do all the dynamic checks at runtime.
With option 2, I’m sure vtable dispatch is still significantly faster than Python (while substantially slower than static typing).
One could also combine approaches 1 and 2 make the static typing optional.
[1] I realize implementation type inference isn’t trivial.
There are indeed lots of options. In this case, 1 is limiting and 2 is slow because of cases like integer operations. A very common way to make it "fast" is specialising at runtime based on usage, so it's kind of like option 1 but dynamic.
Re nr 2, Like stefncb's sibing comment mentions, numeric ops would be horrendeous due to excessive memory allocations (even if you did value like semantics driven you'd still be luggin an entire vtable-pointer along with your numbers and there is usually more value movement than operations in code).
Type-inference depends on the language. Languages with prepared semantics do it just fine(ML family), it's just that languages like Python and JS didn't consider it from day-1 so we're always stuck with explicit subsets (or extensions like TS, Cython/Mypy) to gain back some order. (I did my thesis work on JS type inference).
In practice, there is nothing that hinders the Python language to be as fast as JS (see PyPy or even faster with Cython annotations). The problem is that C-Python sees the simple FFI as an golden egg (for kind-of good reasons at this point) and this will always limit C-Pythons performance because it touches upon internals that are hard to change out without breaking the FFI contracts.
So, even if one would go about and bolt LLVM onto C-Python we'd still be way behind V8/JSC/Spidermonkey, because
1: The value and FFI model still would hold it back
2: LLVM while awesome at static code, would still have a ton of issues with the dynamic nature of Python. There has been a couple of articles on the new JIT of Erlang that is based on a custom "simple" compiler that is working better than many more "advanced" variations has worked because it's more tailored to the dynamic nature of Erlang rather than try to make a regular JIT make something sane out of Erlang code.
If you want to see improvements to Python there's 4 avenues..
1: Nuitka , this is one(?) guy who's doing a lot of manual engineering work together with some JIT ideas, it probably has a performance ceiling but he seems serious about doing it whilst retaining compatibility. (No idea how it'll fare with the regular series C-Python optimizations).
2: PyPy, Jython and GraalPython. These all depart from mainline C-Python and does deprive you of many C/C++-bound libraries whilst retaining the language semantics. The PyPy team has suggested a new better FFI that I do hope gets adopted by C-Python as the way forward for new modules (and ports of older ones) that would allow PyPy and C-Python to share modules as a transition way.
3: Cython, here you have the original source (iirc?) of the MyPy like annotations and an actual compiler that will specialize and should be mostly compatible with C-Python.
4: Shedskin, this is probably the fastest "Python", but it's a specialized subset that's mendable to full type-inference.
There is no "standard" terminology but I would say small line count is a definite factor here. Small LOC count implies low complexity of integration and low risk. Risk? Yes, the smaller the codebase, the easier it is to take your own fork and maintain it, even if the original project is no longer maintaining the codebase.
Of these two low complexity is probably the main selling point.
Small also implies generally "an elegant engineering solution to a problem that generally may involve layers of accidental complexity" and hence also incites a sense of beauty. Aesthetics can be important for people that are hand crafting their codebases.
Nice idea, as python is much easier to write code for than lua so should fit game scripting well.
But, the examples are not the kinds of things that I would care about before trying to put this into a game engine. It needs a different set of examples.
* How hard is it to call into a C function from a hook in the py code?
* How hard is it to wrap / unwrap values from their py representation to C?
If you want people to embed the interpreter into an engine then show that it is clean/easy to do the embedding, rather than showing that the interpreter can execute code. I would assume that it executes python, but need to know how well it embeds.
Is assembly harder to write than lua? How about lisp? I mean you must have some kind of rough ranking of language difficulty in your head. Sure, some of it will be cultural, in a world where middle schoolers are taught s expressions and reverse polish notation lisp might be a lot easier. There certainly is some subjectivity involved.
Still, I think it's reasonable to say that python is easier to write code for.
Scripting use is by definition a less demanding type of requirement than being able to write entire apps.
I wrote a fairly complete "C" subset as a scripting language for a programmable regression test tool (self-contained test-cases able to evaluate the test-case responses). To make it more friendly I made it dynamically typed (so no need to declare variables) and also added XMLPath syntax support as a type of built-in l-value since the requests/responses it was dealing with were XML/JSON, and this meant you could access any elements of the test request/response(s) as if they were just variables.
Anyhow, "C" may not be what comes to mind for a scripting language, but targeting a group of developers who knew C/C++, and with these minor language tweaks made it very simple to use. My point is that for this type of application I'm not sure that the choice of language makes a huge difference - choose something your target audience is comfortable with, and extend in DSL fashion to make it tightly integrated.
In my experience I've found it easier to write Python code. The set of basic types is a little larger, and the core builtins cover more of what I need to do. Lua is a bit more fiddly and verbose, which leads to a more difficult development process (in my opinion). Why would you argue that they have the same level of difficulty?
Lua scripting can be more commonly performed by people without an engineering background, depending on the workplace. Game designers and level artists probably don't have much C experience, so Python's relative proximity to C is a non-factor.
Wow, this is excellent! I love the fact that it is just in 5K lines. I have a few questions!
1. What were your motivations for building this?
2. Did you eval any similar projects before writing this one?
3. Can this be compiled to WASM and used in the browser? edit: I see that it already supports WASM!
4. I didn't see this on the README, but does it support IO (either disk or network)?
5. I noticed you linked to byterun. Does this also implement a VM similar to Python? Can I run the bytecode produced from CPython in the pocketpy VM? If so, are there any reasons for that? I thought writing an independent implementation would be faster/easier than a compatible one.
TinyPy is a bit larger at 10k LOC, seems not as easily embeddable since source split among numerous files, not really suitable for embedding since it's partially bootstrapped (parser and compiler written in tinypy; although pretty cool), it's dead and stuck at py2.
The issue for the first is you either have to download the shared lib precompiled or compile yourself and link as extra step. PocketPy is entirely in header file(s) that can be included in the project directly. That's easiness. (Of course header-only libs have their bads too.) It isn't really about the difference in the result but the process. Having part written in the language can contribute to previous but can also impact speed and memory use. That's suitability.
Still looking for a replacement for the excellent Jython project [1] on the JVM, which has been fantastic for adding scripting to Java/Scala based projects, but sadly stuck on Python 2.7. I wonder if this could be bolted onto or run inside a JVM somehow (perhaps the WebAssembly version?)
There's not only the question of why use this over Lua, but why use this over many compile-to-lua languages, providing a great range of syntax and typing features.
I can confirm that, for example, Fennel[1] is nice if you're looking for something lisp-y.
It's a bit unfortunate that it assumes that you are using Lua's standard library (so you can't use it in cases where you only want to embed Lua-the-language with your own restricted APIs), but other than that it's real nice.
So is this a clean reimplementation? Given how small this is it probably is.
Which means this is not Python but a language that is syntax-compatible ("Python-like"). To prove compatibility something like a (very) extensive set of compatibility tests would have to be written and mentioned in the README. Otherwise it's a minefield of subtle semantic differences.
I don't know if they plan to expand it, but it's a pretty small subset of python now. No bytes or bytearray types, for example. Only string escapes like "\t" are supported, so "\0" raises an error. Trying somestring[1:2:3] raises an error. It's missing list.reverse(), list.sort(), list(some_dictionary) raises an error, and so on. Most areas, if I dug in a little, were missing things.
Should be expected from the size of codebase but should also be clearly mentioned. And this isn't about just lacking Python's modules. Plain language features are also not there. For example reversing a string with "abcd"[-1::-1] doesn't work. Now there's a features list on README, but not sure whether this is meant to be the comprehensive list this subset has or just a TODO.
80 comments
[ 2.9 ms ] story [ 137 ms ] threadhttps://pocketpy.dev/performance/
Exceptions are marked as supported … but `raise Exception()` is a syntax error? (Re-raise and binding exceptions also fails to parse.)
So I was hoping that PocketPy could do that, but after looking around I can't find any suggestion it's possible, and the API looks very sparse. I'm very happy to see a new Python interpreter built for embedding, but I can't use it. Maybe it can be added easily enough. Keep it up!
If you want to see a scripting language designed from the ground up for embedding, Lua and Squirrel IMO have the most complete APIs (though not the most efficient or easy to use) [2][3], and LuaJIT has even better lua<->C bridging. I've looked at the APIs of many other small scripting languages, and haven't found any others that can do everything that these can.
[1] https://github.com/micropython/micropython/issues/3619
[2] In particular https://www.lua.org/manual/5.4/manual.html#4.5
[3] http://squirrel-lang.org/squirreldoc/reference/embedding_squ...
I think which allows you to write stuff like
Would be useful for embedded scripting. Especially for firmware.See [1] for more information (which, btw, also states that #embed is in C23, not C21 (which doesn't exist as far as I know)).
[1]: https://thephd.dev/finally-embed-in-c23
However you might be using "preprocessor directive" in the language semantic phase sense, not in the "preprocessor does it" sense.
[0] http://www.newlisp.org/ [1] http://www.newlisp.org/index.cgi?FAQ
> Because stack-based virtual machine is used, the performance is hard to reach Lua, where the latter is register-based.
> With the optimizer in place, I think the performance of PkPy can reach 1-2 times that of CPython in the future.
https://pocketpy.dev/performance/
It uses https://blueloveth.github.io/pocketpy/lib/pocketpy.wasm which is 766.66 kB - transferred compressed as 240.72 kB
Type `exit()` and press [Enter].
MicroPython [1] is designed for "real" embedding, i.e. in the "embedded development" sense, on microcontrollers and so on and seems quite successful in that niche. Those are often a smaller target than a game running on either a phone/tablet or PC, which would seem to kind of invert things.
[1]: https://micropython.org/
Also, micropython, in its hundreds of thousands of lines of code, supports quite a number of microcontrollers and boards; I suspect the core language is smaller, or counting lines used for a single build target would drop it considerably.
Note that to run correctly micropython also needs its python stdlib which is about 15k lines of Python.
It's _a lot_ of work to make a language like python "superfast", and most of it is really the dynamic typing part. LLVM doesn't do that for you.
Well, actually, I remember being passed on the autobahn by 2CV going well over 200 km/h. That was quite a shocker.
I'd love to see it though.
Option 2 is using C++-style vtables, and putting each value (including primitives) inside an object, with each object inheriting from a common object superclass that has stub methods like __add__, __sub__, __mul__, __div__, __str__, __repr__, etc, etc. You do all the dynamic checks at runtime.
With option 2, I’m sure vtable dispatch is still significantly faster than Python (while substantially slower than static typing).
One could also combine approaches 1 and 2 make the static typing optional.
[1] I realize implementation type inference isn’t trivial.
Type-inference depends on the language. Languages with prepared semantics do it just fine(ML family), it's just that languages like Python and JS didn't consider it from day-1 so we're always stuck with explicit subsets (or extensions like TS, Cython/Mypy) to gain back some order. (I did my thesis work on JS type inference).
In practice, there is nothing that hinders the Python language to be as fast as JS (see PyPy or even faster with Cython annotations). The problem is that C-Python sees the simple FFI as an golden egg (for kind-of good reasons at this point) and this will always limit C-Pythons performance because it touches upon internals that are hard to change out without breaking the FFI contracts.
So, even if one would go about and bolt LLVM onto C-Python we'd still be way behind V8/JSC/Spidermonkey, because
1: The value and FFI model still would hold it back
2: LLVM while awesome at static code, would still have a ton of issues with the dynamic nature of Python. There has been a couple of articles on the new JIT of Erlang that is based on a custom "simple" compiler that is working better than many more "advanced" variations has worked because it's more tailored to the dynamic nature of Erlang rather than try to make a regular JIT make something sane out of Erlang code.
If you want to see improvements to Python there's 4 avenues..
1: Nuitka , this is one(?) guy who's doing a lot of manual engineering work together with some JIT ideas, it probably has a performance ceiling but he seems serious about doing it whilst retaining compatibility. (No idea how it'll fare with the regular series C-Python optimizations).
2: PyPy, Jython and GraalPython. These all depart from mainline C-Python and does deprive you of many C/C++-bound libraries whilst retaining the language semantics. The PyPy team has suggested a new better FFI that I do hope gets adopted by C-Python as the way forward for new modules (and ports of older ones) that would allow PyPy and C-Python to share modules as a transition way.
3: Cython, here you have the original source (iirc?) of the MyPy like annotations and an actual compiler that will specialize and should be mostly compatible with C-Python.
4: Shedskin, this is probably the fastest "Python", but it's a specialized subset that's mendable to full type-inference.
Disclaimer: I worked on Cinder.
Was I wrong?
Of these two low complexity is probably the main selling point.
Small also implies generally "an elegant engineering solution to a problem that generally may involve layers of accidental complexity" and hence also incites a sense of beauty. Aesthetics can be important for people that are hand crafting their codebases.
But, the examples are not the kinds of things that I would care about before trying to put this into a game engine. It needs a different set of examples.
* How hard is it to call into a C function from a hook in the py code?
* How hard is it to wrap / unwrap values from their py representation to C?
If you want people to embed the interpreter into an engine then show that it is clean/easy to do the embedding, rather than showing that the interpreter can execute code. I would assume that it executes python, but need to know how well it embeds.
No, it isn't. And it isn't harder, too. They're just different.
Still, I think it's reasonable to say that python is easier to write code for.
I wrote a fairly complete "C" subset as a scripting language for a programmable regression test tool (self-contained test-cases able to evaluate the test-case responses). To make it more friendly I made it dynamically typed (so no need to declare variables) and also added XMLPath syntax support as a type of built-in l-value since the requests/responses it was dealing with were XML/JSON, and this meant you could access any elements of the test request/response(s) as if they were just variables.
Anyhow, "C" may not be what comes to mind for a scripting language, but targeting a group of developers who knew C/C++, and with these minor language tweaks made it very simple to use. My point is that for this type of application I'm not sure that the choice of language makes a huge difference - choose something your target audience is comfortable with, and extend in DSL fashion to make it tightly integrated.
I think it's easier to mentally map concepts between C and Python than Lua
1. What were your motivations for building this?
2. Did you eval any similar projects before writing this one?
3. Can this be compiled to WASM and used in the browser? edit: I see that it already supports WASM!
4. I didn't see this on the README, but does it support IO (either disk or network)?
5. I noticed you linked to byterun. Does this also implement a VM similar to Python? Can I run the bytecode produced from CPython in the pocketpy VM? If so, are there any reasons for that? I thought writing an independent implementation would be faster/easier than a compatible one.
6. Any plans on adding type annotations?
1. source split among numerous files, and
2. partial bootstrapping
affect embeddability? i'm asking in earnest. i thought to embed we (typically) just need a shared lib and to #include a header api.
Still looking for a replacement for the excellent Jython project [1] on the JVM, which has been fantastic for adding scripting to Java/Scala based projects, but sadly stuck on Python 2.7. I wonder if this could be bolted onto or run inside a JVM somehow (perhaps the WebAssembly version?)
1: https://www.jython.org/
It's a bit unfortunate that it assumes that you are using Lua's standard library (so you can't use it in cases where you only want to embed Lua-the-language with your own restricted APIs), but other than that it's real nice.
[1]: https://fennel-lang.org/
Which means this is not Python but a language that is syntax-compatible ("Python-like"). To prove compatibility something like a (very) extensive set of compatibility tests would have to be written and mentioned in the README. Otherwise it's a minefield of subtle semantic differences.
Still impressive for 5kLOC, of course.
It's amazing for a personal project but not good enough for a serious option.
Should be expected from the size of codebase but should also be clearly mentioned. And this isn't about just lacking Python's modules. Plain language features are also not there. For example reversing a string with "abcd"[-1::-1] doesn't work. Now there's a features list on README, but not sure whether this is meant to be the comprehensive list this subset has or just a TODO.
I was trying to build my own python interpreter to my project and this saved me a lot of work!
super nice-to-have ask: could you replace the is_prime test on the README from range(2,x) to range(2,int(math.sqrt(x)))?