A similar project to pythonnet (a CPython package that implements .NET) is IronPython (a C# implementation of Python 2.7) the latter of which isn't limited by the GIL since it's not using CPython.
Is there anything about doing it the other way around, i.e. embed python code in Go...? I wish we could get the easy-deployment story of Go around the easy-development story of Python.
I'm also curious to know how people who use Python for data science and combine it with some other platform for web facing endpoints architect their apps in the cloud.
Where do you put the web server? What's the purpose of it?
Or you mean to use Python as the client facing API server? I meant how to architect if you use some other platform for the publicly exposed API. Would you build an internal Python API at the lambdas that run the actual data science computation, and invoke that from the client facing API/app?
I haven't done this specifically but it seems like a perfect use case for a smaller separate service. You'd probably submit jobs through an internal API (REST, gRPC, pick your poison) or through a job queue like Celery, if they're slower operations.
Yes that's what I thought too. Building a bunch of data analytics microservices that host a Python API and serve the results internally to the actual outwards facing API server.
Python imports modules dynamically so even if we embedded python in Go, all the deployment woes of Python would still be around. Its kind of a fundamental problem with dynamically linked libraries and python being an interpreted language.
There isn't any fundamental problem, it's just about tooling. Interpreted languages typically do this by packing all the necessary modules into an executable binary together with an interpreter and preloading or faking import function to load them from memory. Dynamically linked libraries usually can be recompiled and relinked statically, sometimes you may need to fake dlopen() and add a namespace to functions as some people try to prevent static linking, and if recompiling is not a possibility, approaches like elf statifier [1] can be taken. It's too bad shipping a single self contained binary doesn't have a first class support in most languages as if deployment is somebody else's problem. But nothing actually prevents it.
This probably isn't what you're after, but interestingly there is a Google-developed Python dialect that can be embedded in a Go program, to facilitate runtime scriptability and programmable config files:
The gohttp library seems to do a lot less than what gunicorn/werkzeug/flask does. I would guess this is what makes the drastic performance difference, not the fact that the HTTP handler is written in go. I'm surprised that it was so close to go-net/http performance, though.
I haven't dealt with Go at all, but if you can use aiohttp on Python 3 (3.5+ IIRC), it is remarkably fast for handling asynchronous requests. In my testing, saw something like a 66% wall clock reduction over using multiprocessing to achieve parallelism. Sadly, cant use it in production, because aiohttp doesnt currently support Negotiate/SSPI auth.
I had not used python for like 5 year, since when I migrated all my work to go. Recently I went back to python and was shocked to see how hard (relatively) it is to setup a (moderately) high performing web server in python.
I mean in my case I had a “data science” type application and sometimes a request would block and take a second to finish, and this meant a handful of users would bring the server to it’s knees (due to extremely high mem usage I could not have a lot of indepentend worker processes running at the same time),
I wish I could call python code from within a go web server with some ease and safety.
This claim is specifically related to blocking webserver in python vs concurrent webservers in go.
I assumed this is common knowledge, that scaling python webservers is a complicated thing, while in go you get a concurrent web server in standard library,
How about comparing a blocking server in Go versus the same in Python, then an async server in Python vs the concurrent Go server? That would be a more apt comparison.
Efficiency is measured in a lot of different ways. Development speed? Team familiarity? Existing infrastructure (private pypi etc...)? Existing libraries?
If we go with what I would assume is your definition -- speed of execution versus resources used -- it is certainly possible to build fast Python applications that are efficient.
But use what makes you and your team happy, life is too short for anything else.
blocking webservers are a common problem in Python. Even if you have async, all it takes is a CPU bound task or some sync IO deep in some third party library and its lights out for your server, and no debug info for you either.
Wouldn't a job queue solve both issues? You could either make the data science stuff async (and spin up more servers as necessary). Or, with a job queue, you could use a go webserver to push requests into the queue for processing in python.
> I mean in my case I had a “data science” type application and sometimes a request would block and take a second to finish, and this meant a handful of users would bring the server to it’s knees
This sounds like an issue with the design of your program, not Python.
No, data science is typically cpu expensive. Python is fundamentally single threaded and slow at that, so you have to be very clever to work around the issue (e.g., running a separately scalable service for your data crunching work). Contrast that with Go where the runtime can use other cores.
No, but very often those programs are slower overall because of the pickling cost. Multiprocessing isn’t a magic bullet / there’s a reason threads exist.
Python is not fundamentally single threaded - it just has a lock that stops it from taking advantage of threads in cpu bound scenarios.
Python is used in data science because of the C bindings that make it not slow. Also, when in C, you can take advantage of threads since they live outside the GIL. e.g. Dask.
Yes it is. It is not designed to run fast on multi-core CPUs, because there were mostly single-core CPUs when Guido made the language. It has been always a problem since multi-core CPUs are more frequent and it's a front where Python is losing the battle (against Go for example because of way easier concurrency support).
> Python is not fundamentally single threaded - it just has a lock that stops it from taking advantage of threads in cpu bound scenarios.
Tomato tomahto
> Python is used in data science because of the C bindings that make it not slow. Also, when in C, you can take advantage of threads since they live outside the GIL. e.g. Dask.
Correct. Python is fast when you aren’t running Python. Of course using C (or anything else) only works in certain situations—there is a cost to crossing the language boundary and very often that cost is greater than what you save by using C. Never mind the added build/package complexity, the security issues, the maintainability issues, etc.
Python is a neat language, but it’s really expensive if your project ever might have tight performance requirements (where “tight” is laughably easy for most other languages). Python can often be made to meet them with enough shenanigans, it’s just costly to implement and maintain said shenanigans.
If the language makes you jump through unnecessary hoops to get passable performance, the issue is with the language, not the program. Otherwise we could generalize your perspective such that all languages are above criticism (no languages have problems, users just fail to find and implement the proper hacks).
We run python at work and I have to agree with this however with Python 3 and ASGI servers now existing this should be significantly less of an issue. Even just Python 3 WSGI using Gunicorn + gevent should be fairly performant for Python.
There is no way around the high memory usage, but a large number of the problems with Python concurrency is not loading nginx (or load balancer) in front and not switching to gevent from PreFork which uses a considerably higher amount of memory per “node” for higher concurrency. That said, gevent is only "performant" if what you’re doing is IO bound. Same thing with any AsyncIO based server.
"data science" sounds like DB or NoSQL heavy so should fit this case, but of course all of this is just general advice and depends on the app/code like others said.
First off, I totally agree. Not as easy as it should be to write an async web server in python. FastAPI is probably your best bet. I usually use Sanic. Easy to accidentally block though.
That said, it sounds like you’re serving a large model. No amount of async/await or goroutines can solve this problem. A non-blocking web server is a godsend for I/O-bound tasks, but a large model is just a deep call stack - lots of multiply, nonlinear function like RELu, then add, times a billion. This would still block, even if you had perfect async/await code.
I made some assumptions here, but if I’m right, the answer is “shrink your model” and/or “buy more compute”. Neither of which are easy. But if you’re trying to shrink a model, check out Distiller https://github.com/NervanaSystems/distiller
Edit: the restriction I talk about is for event-loop based servers using something like uvloop or asyncio under the hood. Maybe this restriction doesn’t hold for other concurrency modes.
Async doesn’t help with CPU-bound tasks, but yes, you can rig up systems that involve running multiple Python processes behind a load balancer. It’s just more work.
> No amount of async/await or goroutines can solve this problem. A non-blocking web server is a godsend for I/O-bound tasks
In the past, we were told that threads were cheap and to use them heavily, especially to achieve parallelism. Now with the advent of async models, we're being told that threads are expensive, and often that a single processor/thread async model is better than a multi-threaded blocking one.
I'm not a luddite, I do agree that async is often better. But I wonder how we got tricked into thinking more and more threads were the answer and how we avoid such trickery again.
A little off-topic, but I've built a small backend server in Node.js and Tensorflow.js to run a previously Pyhton-built model and was amazed by how performant and non blocking it runs.
The model can do around 10k predictions/s and does it with async, which allows Node to respond to web requests in the meanwhile.
I guess it's a matter of using the right tool for the task, whenever possible, Python for data science, Nodejs for a web backend.
He touches upon it in the "Runtime Overhead" section, but I think when calling Go from C like this you loose a lot of the goodness of Goroutines, and any code making heavy use of Goroutines could become seriously problematic.
They do not, the go runtime just locks the OS thread to the goroutine calling C until it comes back out. This can lead to fun deadlocks that only appear on single-core machines (because by default GOMAXPROCS is equal to the core count).
Similarly, Docker locks goroutines to the OS thread when using the unshare system call to spawn containers. These are of course later discarded. It has to be locked because any goroutine might be stopped at any (automatically inserted by the compiler) checkpoint and resumed on a different OS thread.
Unsharing the network interfaces from half your OS threads is a fun way to chaos test networking in Go.
Werkzeug/flask and gohttp in my view is not a fare comparison. Based on the details of the project werkzeug is designed to build WSGI apps easier and flask is just a convenient library on top of it providing easier way to manage request/response. It is designed for developer productivity and performance is left to specific infrastructure software like gunicorn, uwsgi and others. Flask/werkzeug or any WSGI/ASGI related servers were designed for interfacing with a http server like nginx, apache, gunicorn, uwsgi and many. A python LWAN http server can beat go http server [1] [2].
Also take a look at comparison of various frameworks in Python including go http server [3].
People are most productive in the language/framework they are most familiar and will defend it. Every language/framework has their own strength and weakness and I believe over a period of time ideas flow from one language to another.
Indeed today many people in Python community will be moving towards Rust or Go because they think it can solve all their problems they face with dynamic typing and performance, which might not be entirely true. It's upto an individual to decide if they want to go that path.
As an example werkzeug/flask framework developer Armin moved to Rust in spite of it being a complex language with very large syntax surface area and a steep learning curve. In my opinion Rust's complexity, difficulty to learn and understand, and probably a promise of type and memory safety (which is not 100% true given it needs to interface with C in unsafe mode and will only be as safe as the underlying C implementation), makes people adopt it to make them feel better programmers (Personally I would have chosen Haskell, if needed to do the same).
Now all the Python projects he worked on is mostly maintained by volunteers and David. But being a responsible open source developer and contributor before moving in that direction Armin did create pallets projecs [4]. So his decision to move is right for him given his preference and learning priorities.
I don’t think the EOL of CPython 2.7 should necessarily bother anyone using IronPython. It’s not like IronPython depends on CPython in any way, or is exposed to any CPython bugs or vulnerabilities. I can see it from a standardisation point if view of course.
True, IronPython3 isn't ready and with 2.7 EOL I would say that it may not be a great fit for someone currently weighing up their options between it and pythonnet to use in a current project.
Development on IP3 is, however, still very alive and active so I wouldn't consider the entire project dead:
59 comments
[ 3.2 ms ] story [ 126 ms ] threadIt makes calling C# as easy as:
And also works the other way around. In both cases you have to mind GIL though.https://github.com/IronLanguages/ironpython2
(although it looks like it's not close to supporting the needed functionality to import a typical python library)
There is an active fork here: https://github.com/grumpyhome/grumpy
Or you mean to use Python as the client facing API server? I meant how to architect if you use some other platform for the publicly exposed API. Would you build an internal Python API at the lambdas that run the actual data science computation, and invoke that from the client facing API/app?
[1] http://statifier.sourceforge.net/statifier/background.html
https://github.com/google/starlark-go
https://github.com/starlight-go/starlight
It seems to be used by the Delve debugger for example:
https://github.com/go-delve/delve/blob/master/Documentation/...
https://github.com/bazelbuild/starlark/blob/master/users.md
> Keep in mind that this is with 10 concurrent requests, so werkzeug-flask probably chokes more on the concurrency than the response time being slow.
I am not sure though. I’d imagine Go can beat Python performance enough to make up for the (clearly not very egregious) CGo penalties.
I wish I could call python code from within a go web server with some ease and safety.
Especially considering that there's quite a few sites which run entirely on Python.
If we go with what I would assume is your definition -- speed of execution versus resources used -- it is certainly possible to build fast Python applications that are efficient.
But use what makes you and your team happy, life is too short for anything else.
This sounds like an issue with the design of your program, not Python.
Python is not fundamentally single threaded - it just has a lock that stops it from taking advantage of threads in cpu bound scenarios.
Python is used in data science because of the C bindings that make it not slow. Also, when in C, you can take advantage of threads since they live outside the GIL. e.g. Dask.
Yes it is. It is not designed to run fast on multi-core CPUs, because there were mostly single-core CPUs when Guido made the language. It has been always a problem since multi-core CPUs are more frequent and it's a front where Python is losing the battle (against Go for example because of way easier concurrency support).
Tomato tomahto
> Python is used in data science because of the C bindings that make it not slow. Also, when in C, you can take advantage of threads since they live outside the GIL. e.g. Dask.
Correct. Python is fast when you aren’t running Python. Of course using C (or anything else) only works in certain situations—there is a cost to crossing the language boundary and very often that cost is greater than what you save by using C. Never mind the added build/package complexity, the security issues, the maintainability issues, etc.
Python is a neat language, but it’s really expensive if your project ever might have tight performance requirements (where “tight” is laughably easy for most other languages). Python can often be made to meet them with enough shenanigans, it’s just costly to implement and maintain said shenanigans.
There is no way around the high memory usage, but a large number of the problems with Python concurrency is not loading nginx (or load balancer) in front and not switching to gevent from PreFork which uses a considerably higher amount of memory per “node” for higher concurrency. That said, gevent is only "performant" if what you’re doing is IO bound. Same thing with any AsyncIO based server.
"data science" sounds like DB or NoSQL heavy so should fit this case, but of course all of this is just general advice and depends on the app/code like others said.
It ranks pretty high on performance in framework benchmarks
That said, it sounds like you’re serving a large model. No amount of async/await or goroutines can solve this problem. A non-blocking web server is a godsend for I/O-bound tasks, but a large model is just a deep call stack - lots of multiply, nonlinear function like RELu, then add, times a billion. This would still block, even if you had perfect async/await code.
I made some assumptions here, but if I’m right, the answer is “shrink your model” and/or “buy more compute”. Neither of which are easy. But if you’re trying to shrink a model, check out Distiller https://github.com/NervanaSystems/distiller
Edit: the restriction I talk about is for event-loop based servers using something like uvloop or asyncio under the hood. Maybe this restriction doesn’t hold for other concurrency modes.
Presumably he’s not serving the model but running it, which is cpu bound, in which case Goroutines would solve the problem.
In the past, we were told that threads were cheap and to use them heavily, especially to achieve parallelism. Now with the advent of async models, we're being told that threads are expensive, and often that a single processor/thread async model is better than a multi-threaded blocking one.
I'm not a luddite, I do agree that async is often better. But I wonder how we got tricked into thinking more and more threads were the answer and how we avoid such trickery again.
The model can do around 10k predictions/s and does it with async, which allows Node to respond to web requests in the meanwhile.
I guess it's a matter of using the right tool for the task, whenever possible, Python for data science, Nodejs for a web backend.
In native Go, Goroutines are very lightweight and cooperatively scheduled. In a CGo env I believe they each have an OS thread and a full stack. Source: https://www.cockroachlabs.com/blog/the-cost-and-complexity-o...
Similarly, Docker locks goroutines to the OS thread when using the unshare system call to spawn containers. These are of course later discarded. It has to be locked because any goroutine might be stopped at any (automatically inserted by the compiler) checkpoint and resumed on a different OS thread.
Unsharing the network interfaces from half your OS threads is a fun way to chaos test networking in Go.
https://golang.org/pkg/runtime/#LockOSThread
http://man7.org/linux/man-pages/man2/unshare.2.html
Also take a look at comparison of various frameworks in Python including go http server [3].
People are most productive in the language/framework they are most familiar and will defend it. Every language/framework has their own strength and weakness and I believe over a period of time ideas flow from one language to another.
Indeed today many people in Python community will be moving towards Rust or Go because they think it can solve all their problems they face with dynamic typing and performance, which might not be entirely true. It's upto an individual to decide if they want to go that path.
As an example werkzeug/flask framework developer Armin moved to Rust in spite of it being a complex language with very large syntax surface area and a steep learning curve. In my opinion Rust's complexity, difficulty to learn and understand, and probably a promise of type and memory safety (which is not 100% true given it needs to interface with C in unsafe mode and will only be as safe as the underlying C implementation), makes people adopt it to make them feel better programmers (Personally I would have chosen Haskell, if needed to do the same).
Now all the Python projects he worked on is mostly maintained by volunteers and David. But being a responsible open source developer and contributor before moving in that direction Armin did create pallets projecs [4]. So his decision to move is right for him given his preference and learning priorities.
[1] https://www.nexedi.com/NXD-Blog.Multicore.Python.HTTP.Server
[2] https://lwan.ws/
[3] https://www.freecodecamp.org/news/million-requests-per-secon...
[4] https://www.palletsprojects.com/
Development on IP3 is, however, still very alive and active so I wouldn't consider the entire project dead:
https://github.com/IronLanguages/ironpython3/commits/master