Thanks for taking the time to publish this, I've always been interested in tearing at CPython, but this pushed me over the edge. Looking forward to new and exciting ways to crash the runtime :)
I remember another article, but cannot find it. It had very different arguments what is slow about Python and would have disagreed with those two articles.
Python is slow because it is dynamically typed rather than statically typed? No. In most cases, type inference is good enough. See Javascript. Python is slow because it is interpreted rather than compiled? That is not really a limit in the long run. Insert a JIT compiler. Java was interpreted as well initially.
Python's object model can lead to inefficient memory access? No we are getting somewhere. Scripting languages in general tend to overuse hashmaps. That gives you a lot of flexibility and monkey patching. It is also a lot harder to optimize than dynamic dispatch.
Another problem of scripting languages is the inability to control your memory. Python forces you to make a lot of memory allocations and there is little control over the garbage collector. Even Java suffers from this, but the JVM engineers invested a lot to fix it. There are no value record types (struct) in Java.
Is there a reason why Python must be slower than Javascript? I.e. given economic and cultural incentives, couldn't there be something like V8, for python?
No, I don't know of anything in Python that could not be made as efficient as Javascript, using techniques that we already know about (and given the time and money that you've already acknowledged are also barriers).
I'm not considering Python's C extensions there. I don't know if you'd consider that part of the 'language' or 'ecosystem'. Pyston is trying to make Python fast, including the C extensions, which is not as easy.
JavaScript has fewer problematic features than Python which makes it possible to write efficient implementations. Also it has a standard which Python does not.
There isn't a document describing the semantics of Python is there?
That's what people mean by a 'standard'. Something at least trying to be somewhat semi-formal. If you mean that the implementation of CPython is a standard, well then I suppose that's arguable, but it's really not what anyone means.
I'm not sure why Armin is so categorical, but perhaps one way to rephrase what he said is that ES6 has had more opportunity to flesh out the spec/edge cases than Python, due to Javascript's popularity and healthy competition between implementations.
Because I spent months of my life a few years ago trying to work on modernizing the Python interpreter and it made me a bitter person. There is so much bizarre behavior in the interpreter that people depend on which is entirely undocumented and non standardized. Very few people know what even such a simple thing as "a + b" does in the language.
The question was about whether it is technically possible to create a Python as fast as JavaScript, ignoring the money needed.
So the number of features doesn't matter, only whether or not they are all possible to implement efficiently, and the lack of a standard doesn't matter, as with infinite money you could test comprehensively.
No need to disqualify the answer. The point of my question was of course to learn about aspects of Javascript-the-language that make it intrinsically easier to build a fast implementation for than building one for python.
If there are just more difficult-to-implement features that is interesting as well (although it would be nice to know which ones).
There are faster Python implementations, namely Pypy. I think one of the reasons Pypy isn't more popular is because it's C interoperability story is weaker than the reference Python implementation (CPython--not to be confused with Cython), or at least it's not compatible with many important C-based Python libraries (e.g., Pandas).
The C-compatbility story works both ways. I usually roll my eyes when people talk about making dynamic lanuages fast -- what's the point? In those cases where speed matters - use static language. As a result there is little incentive to use something like PyPy (although PyPy does have other nice features in addition to raw speed).
But on the web there is a point. Unless you can use HTML alone, you have to use JavaScript -- and you can't just plug in a module that you wrote in C.
It's about managing your resources. You can simply buy the fastest processor for cheap money. You can buy systems with many cores or even build a cluster for considerable money.
But you can not buy time.
Developing python applications in many cases is much quicker than developing comparable applications in C++, C or Assembler. Therefore it makes a lot of sense for many companies to choose tools with quick development time to get the product running and care for performance later.
Since there may be so much code be written in python, it cannot simply be converted into a faster language since most critical parts are already accelerated using the proper libraries provided by the language ecosystem. Porting a complex application to another language in many cases is a really hard problem and might even not be possible to be done in smaller steps.
Therefore we need every bit of performance we can squeeze out of python. The whole world will gain from that. If we manage great strides like JavaScript it would be glorious!
Also soon enough (if we aren't there already), fast hardware will be so cheap that the speed of execution at least for end user applications will rarely be a factor. Sure that database application you wrote in C is measurably faster, but can a human being detect the difference? More and more the response will be no.
In the last 5 years, clock speeds have stagnated. Hardware innovation has revolved around increasing cores and improving power efficiency. Unfortunately, Python and most other dynamic languages are poorly suited to take advantage of parallelism, so your C database can only get faster, but your Python database will not.
Honestly, the vast majority of languages are bad at parallelism. Dynamic vs. static doesn't have much to do with it. Task-parallel multicore (like goroutines and Web Workers and the like give you) is only a small fraction of what it means to "take advantage of parallelism", and if you're only using that (as opposed to SIMD, GPUs, concurrent data structures, work-stealing, etc.) then you are likely to see disappointing speedups in practice.
This is especially true in the last 5 years, which contrary to your post has seen a shift away from multicore task-parallel hardware (core counts are not increasing) and toward data-parallel hardware (wider SIMD lanes, better on-board GPUs, APU architectures, etc.)
You're missing the point. There's "suboptimal parallelism", and then there's "no parallelism". Python and many other dynamic languages fall into the latter category.
I'd call Python optimal parallelism. I prefer the scaling by instance method of Node and Python over the old ways- writing multithreaded Java apps. People can knock themselves out writing stuff like that if they want to.
But my purposes have been well-served scaling by spinning up instances per-core.
For me, explicit pthreading should never be done except with systems software. In which case you're going to be using a systems language like C, C++ or Rust anyway. Not the oddball middletier stuff that isn't high or low level like Go, Java and C#.
Go's parallelism model is fundamentally different than the other languages you mentioned. Even if you never write any concurrent Go yourself, you could write single threaded routes and let the webserver handle the parallelism (this is more efficient than spinning up an OS thread per request). With Python, you also have to configure uwsgi or some such to deploy your application to each thread; with Go, the runtime handles this for you completely transparently--just run the binary and you're all set. Go isn't perfect, but using Python the way you are is just leaving money on the table.
In the case of an I/O bound Web server, I don't think that Go is going to appreciably result in "leaving money on the table". Python releases the GIL on I/O, which is where your time is largely spent.
> this is more efficient than spinning up an OS thread per request
Linux is very fast at spawning threads. When you're doing I/O, it doesn't matter.
Go's concurrency is not "fundamentally different" from any of the other languages. It's just an implementation of thread-per-connection in userspace. This is an implementation detail, and for a typical Web server I think it doesn't matter much.
> In the case of an I/O bound Web server, I don't think that Go is going to appreciably result in "leaving money on the table". Python releases the GIL on I/O, which is where your time is largely spent.
GIL doesn't matter, the OP said he was writing single threaded application. This means he's doing async (unlikely) or blocking the entire application process (not just the current request) on I/O.
> Linux is very fast at spawning threads. When you're doing I/O, it doesn't matter.
Not as fast as goroutines, and spawn speed isn't the only (or even the most interesting) measure of efficiency--you can run several orders of magnitude more goroutines on a Linux box than you can threads. Of course, none of this is relevant to this conversation, since the OP is talking about running a single-threaded Python process per core.
> Go's concurrency is not "fundamentally different" from any of the other languages. It's just an implementation of thread-per-connection in userspace. This is an implementation detail, and for a typical Web server I think it doesn't matter much.
You're conflating Go's concurrency model with its webserver implementation. Go's concurrency model (movable M:N coroutines with implicit yielding and a synchronous abstraction over async IO) is fundamentally different than the other approaches (threads and vanilla coroutines with some blend of sync and unabstracted async I/O).
Since you mentioned thread-per-connection, Go's "threads" are much lighter than even Linux threads, so you can service many more simultaneous connections (by many orders of magnitude in extreme cases). This is especially true if you're not using async I/O, and blocking I/O is the default for most of these other languages.
>but using Python the way you are is just leaving money on the table.
Everything leaves some performance on the table. Everything also leaves some productivity on the table too.
Python is in my estimation 20-50% more productive as a language than Go. I run my web stuff on PyPy/gunicorn/Nginx which within reason, can't be leaving much to be wanted performance-wise.
I've given Go a good-go, but I haven't found anything that strikes that balance of ~25% more productive than Go, but still grant similar performance like PyPy does for me.
Assembler isn't statically typed, and C and C++ are particularly bad examples of productive static languages. Go is a much better example, but it disproves your thesis that dynamic languages are more productive than static languages (I have 6 years of Python experience and 3 years of Go experience, and Go is at least as easy as Python for small, single-threaded code bases and it scales much better with team size, code volume, and thread count).
Yeah, but Go is 20-50% less productive, depending on who you ask. From my experience I'd say it's at least 25% less productive than Python. Not horrible but I think 25% is a minimum value and a quarter of your time adds up. I don't think it's worth the trade off once you consider every language has its warts, Go has its own problems.
That's not been my experience, and I'm a Python developer by trade. Python simply doesn't scale well with code size, team size, performance requirements, etc.
Go's biggest problem is a lack of generics, which means that in some cases, you have to fall back to duck typing. So the worst case is no worse than Python's best case.
By that logic, why optimize anything? Just use the thing that's out there that's already faster. Clearly some people think there's a benefit to dynamic languages (I'm not one of them), and they'd like to preserve that benefit while minimizing the performance impact.
Python connecting to C is used mostly to attach to large C objects that do something complicated, like OpenSSL. Math inner loop intensive things like numpy could potentially be optimized up to C speed by PyPy. (At least if you could create an array of a fixed type in Python.)
"I think one of the reasons Pypy isn't more popular is because its C interoperability story is weaker than the reference Python implementation"
The CPython crowd apparently made some change between CPython 3.4 and CPython 3.5 which changed how Windows DLLs are called. This broke Py2exe, which no longer works for the current release of CPython. [1]
CPython's approach to C extensions usually requires having the same Microsoft C compiler version used to build Python. This is now a problem for Python 2.7, because the Microsoft compiler version used to build it is no longer available. Microsoft recently changed their API for the C runtime, and CPython has been changed accordingly.
CPython's "C interoperability story" isn't really that good on Windows.
It's good enough for practical purposes - sure, you have to build a separate version of your package for every different version of VC++ that is used by your supported versions of Python, but we're still talking about 3-4 binaries top.
When that is not enough, you can still recompile manually if need be, so long as source for the module is available (which it is in most cases).
On the other hand, native modules are such a big part of the Python ecosystem, that not supporting the API and ABI at all confines any alternative implementations to a very small niche.
The issue w/ MSVC is not that specific to CPython. It is generally quite hard to write libraries which use multiple C runtimes (Windows or not -- it just happens to be much more common on Windows than on Unix). I would be actually very surprised if that problem did not exist with any other language that distribute binaries.
This is true on other OSes as well. C extensions are assumed to be compiled with the same compiler as Python itself. If you use distutils it will also use the same flags.
This is less problematic on other OSes where GCC dominates and clang is compatible, but it can bite on HPC systems that use PGI compilers.
I am not a compiler/JIT expert by any stretch, but I have never heard of a strong technical reason why python could not be as fast as JS on V8 or compiled code produced by LISP compilers. But there are strong factors against it:
* the reference implementation is maintained by a team of contributors, very few people are paid full time to work on cpython. State of the art JIT is hard to come by when you can only afford working a few hours / week.
* contrary to JS, python had a huge set of packages available for a very long time, making "minor" language breaks unworkable.
* also contrary to JS, python had from the start a strong story to interoperate w/ the C ABI, which tightened a large subset of python libraries with the details of cpython implementation. Because of its "niche", JS never really had that problem. If you remove libraries w/ large C extensions, you remove a lot of what make python popular today (e.g. scientific usage is now the 2nd largest area of application of python after web development, before automation, testing, etc... which used to be where python shined).
It's possible, but Python doesn't need to be fast because it's used as glue for other (faster) languages like C. On the other hand, Javascript became the golden standard in its own niche (client-side web), there are no alternatives. Making it as fast as possible makes sense.
Not because I want to limit discussions here, but this has been posted before on HN and there are many great comments in these threads. So, this comment is more of a useful suggestion, that if you are interested, check these comments out too (both 2+ years old).
I have to wonder how much this is CPython specific and how much applies to all implementations of Python (including PyPy, which is noticeably faster than CPython).
I've found that PyPy is very fast in most cases. While I've seen hand-optimized C code beat PyPy by a mile, I've seen more cases where the gains were in the 10-20% range. And PyPy keeps getting faster.
Generally good but it makes it sound like what it calls the dynamic typing penalty is a matter of more instructions. If that was true Python would only be a factor of 2 slower or something like that. Instead the problem is that the cost of getting the value of an integer is first of all that after reading the type of the variable Python has to execute a fairly high entropy branch which will probably cost the branch delay penalty of 20 cycles or so. And then the extra memory lookup will cause a slowdown as well.
It is easy to miss the point with such articles. OP touched upon it in chapter "So Why Use Python?". Raw speed, while sexy, is not a deciding factor in most cases - not everybody is working on HFT. And when it is important to be really fast, you can always write an extension in C (or Rust I imagine?) just for that single tight loop and get raw speed without sacrificing maintainability. Seriously, most bottlenecks (in other people's software of course ;) I came across were either due to bad architecture or unreadable code, which lead to not noticing that too much work was being done. So yeah, Python might be slower than alternatives (though I have a hard time believing that it is slower than Java), but it is fast enough.
This might certainly be true for single threaded programs. For multithreaded programs, Python has a bottleneck known as the "Global Interpreter Lock" (or GIL), which is not so easily optimized away, unfortunately.
True, though this is not the kind of speed this article was talking about. You can get around that limitation somewhat by using multiprocessing module.
I'm not saying Python is the best language of all, but dissecting Python bytecode is kind of missing the point. Python's strong point is its ease of use.
GIL may and does get released in the interpreter (on I/O) and C extensions such as regex, lxml, numpy. Performance sensitive extensions may even use multiple CPU cores implicitly e.g., as numpy.dot() can do for large enough input without changing the code in any way.
The only reason people might think of Java as slow, is because of desktop applications like LibreOffice, IntelliJ, or Eclipse. Java on the server blows Python out of the water in terms of speed.
Oh, the crippling sense of irony that comes from reading this comment after reading the front page post about Azul and the JVM, with a lot of technical details about how HFT firms use Java.
What's so great with dynamic languages that justifies the performance hit? Most statically typed languages have type inferrence, which reduce the syntax to something very close to a dynamic language. And is it such a good idea to only catch mistyping at runtime, and require to write twice as much code with testing to compensate for the lack of typing?
With a dynamic or duck type system, though, you can avoid a lot of writing out interfaces or making class hierarchies. You end up with something less structurally uniform, but with a much shorter time to get results.
If you care about type mismatches, use a statically typed language. If you just want to get something working, use a dynamically typed one.
First, there aren't really any compiled languages where the syntax is similar in ease of use to Python (or any other popular dynamic language I'm familiar with).
All the compiled languages with which I'm familiar require a significant amount more work to accomplish the same thing; C apps are full of boilerplate to detect error cases, allocate memory, check buffer lengths, etc.; dynamic languages handle this for you so the total amount of code tends to be lower.
Writing a simple program to test an algorithm or proof of concept is generally significantly less code, the code/test/debug loop tends to be significantly shorter, and you can often get to a solution significantly quicker while avoiding the typical bugs that come with lower-level languages.
Python also has a pretty substantial standard library which is easy to learn and use and very consistent, meaning much less copypasta code or external dependencies.
On top of that, a great deal of use-cases for languages like Python aren't CPU-bound, but rather I/O bound (disk or network). In cases where your app is CPU-dependent, it can often be trivial to replace that inner loop with optimized native code written in C or other languages, while still keeping the rest of your application easy to read and maintain.
Lastly, for complex algorithmic work, there's a lot of packages which implement standard data structures and algorithms in native code and provide usable, idiomatic interfaces in Python, which means that for a lot of things you can work in native code with the ease of Python, at a minuscule performance hit but a dramatic increase in programming efficiency.
The thing is I can understand if we are comparing C and Python, but if you compare C# or modern VB.net to Python, I am not so sure the syntax is much heavier.
Now if you are not doing something that requires performance, I think there is still a good case for static typing: advanced tooling. Dynamic language limit a lot the ability of the IDE to help you finding bug, making the language and the API self discoverable, auto completing the code, providing advanced analysis (who calls what) or refactoring (rename a property). Some people do not need that, but to those who rely on that, dynamic languages must really be easier to write by an order of magnitude to end up being faster to prototype.
If I was suggesting a language to a beginner, I would push for a language with a strong IDE.
"Most statically typed languages have type inferrence, which reduce the syntax to something very close to a dynamic language."
To understand the appeal of dynamic languages, it's critical to consider the era they came from. Your statement is not true of the general programming landscape ten years ago. It's not really true of the general programming landscape today, for that matter, which is still by sheer mass C++ and Java. But 10 years ago, there weren't hardly any choices of statically-typed languages that weren't a bit of a bear to use. 1990s C++ is a monstrous nightmare. A lot of this monstrous nightmare was arguably the programming environment (Windows, Xwindows, UNIX, everything was a bear to program in back then), but when you took battle-scarred C++ Windows programmers and showed them
print "Hello world!"
they weren't in a mood to analyze exactly which differences between Python/Perl and C++ were essential and which were accidental; they were too busy drinking in the joy.
I think there's a general trend towards languages that either make the static typing easy enough to use that you don't feel like you're losing out on much, or make the static typing pay for itself better (Rust), and I think that will continue to be the story for the next ten years on the developing language front. I think it may be the case that if they weren't already entrenched, what I call the 1990s-style dynamic languages (because it's not just that they're "dynamic", Perl, Python, Javascript, and Ruby are almost the same language, compare with the "dynamic" Erlang to see a major difference) wouldn't succeed if someone wrote one for the first time this year. In fact they'd probably take a beating for the performance issues it would have. But they are entrenched and have over a decade of refined libraries, support documentation, tutorials, and of course, the big hitter, existing code. So they aren't going anywhere any time soon.
The other thing is tooling. There are lots of hardcore developers to whom even syntax highlighting is too much distraction, let alone error checking, auto completion, etc.
Well... To say the least I am not one of them. In fact to me, a large part of the simplicity of a language resides in the interaction with the IDE, the feedback it gives you on what's legal and what is not, where you may have a bug or an ambiguity, what can I do from there, etc. Dynamic languages defeat advanced and proactive tooling. And to me it makes them actually quite hard to work with, not self-discoverable, and all-in slower to develop. The time not spent writing a class or a return type ends up being spent on the reference page of the library (after you checked this is the correct version) checking what are the arguments of a function, etc.
What statically typed language with great type inference has the library ecosystem of python or javascript? I can't think of one, libraries matter a lot.
Any .net language (C#, VB.net, F#). In term of libraries they will be hard to beat. What they will not have is cross-platform though. Microsoft just kicked off a cross platform version but surely it will take at least 5 years before there is a large cross platform ecosystem.
Python is slow because the core dev team has long concerned with Python3. Priority is not on speed. CPython actually got slower with 3, and it still is. They say it's "a wash" but I haven't seen that in anything I've done. Since Microsoft got its claws into Python through the core dev team, we've seen a shift from the Linux (POSIX) text model in Python2 to the Windows text model in Python3.
It now matches how the .Net CLR handles unicode. Which is a shame since Linux is kind of increasingly popular.
There are some recent developments where Microsoft is investing through the core dev surrogates into Python3 and speed with Pyjion. That is promising but it's a framework where it requires CPython3 to use a JIT, which strikes me as a way to enforce control- a response to PyPy and alternatives taking the mantle when they abdicated the throne with Python3. But it remains to be seen how it works out, and PyPy already is likely to remain the leader on faster-Python.
79 comments
[ 3.4 ms ] story [ 148 ms ] threadPython is slow because it is dynamically typed rather than statically typed? No. In most cases, type inference is good enough. See Javascript. Python is slow because it is interpreted rather than compiled? That is not really a limit in the long run. Insert a JIT compiler. Java was interpreted as well initially.
Python's object model can lead to inefficient memory access? No we are getting somewhere. Scripting languages in general tend to overuse hashmaps. That gives you a lot of flexibility and monkey patching. It is also a lot harder to optimize than dynamic dispatch.
Another problem of scripting languages is the inability to control your memory. Python forces you to make a lot of memory allocations and there is little control over the garbage collector. Even Java suffers from this, but the JVM engineers invested a lot to fix it. There are no value record types (struct) in Java.
I'm not considering Python's C extensions there. I don't know if you'd consider that part of the 'language' or 'ecosystem'. Pyston is trying to make Python fast, including the C extensions, which is not as easy.
That's what people mean by a 'standard'. Something at least trying to be somewhat semi-formal. If you mean that the implementation of CPython is a standard, well then I suppose that's arguable, but it's really not what anyone means.
I'm not sure why Armin is so categorical, but perhaps one way to rephrase what he said is that ES6 has had more opportunity to flesh out the spec/edge cases than Python, due to Javascript's popularity and healthy competition between implementations.
So the number of features doesn't matter, only whether or not they are all possible to implement efficiently, and the lack of a standard doesn't matter, as with infinite money you could test comprehensively.
If there are just more difficult-to-implement features that is interesting as well (although it would be nice to know which ones).
But on the web there is a point. Unless you can use HTML alone, you have to use JavaScript -- and you can't just plug in a module that you wrote in C.
But you can not buy time.
Developing python applications in many cases is much quicker than developing comparable applications in C++, C or Assembler. Therefore it makes a lot of sense for many companies to choose tools with quick development time to get the product running and care for performance later.
Since there may be so much code be written in python, it cannot simply be converted into a faster language since most critical parts are already accelerated using the proper libraries provided by the language ecosystem. Porting a complex application to another language in many cases is a really hard problem and might even not be possible to be done in smaller steps.
Therefore we need every bit of performance we can squeeze out of python. The whole world will gain from that. If we manage great strides like JavaScript it would be glorious!
This is especially true in the last 5 years, which contrary to your post has seen a shift away from multicore task-parallel hardware (core counts are not increasing) and toward data-parallel hardware (wider SIMD lanes, better on-board GPUs, APU architectures, etc.)
For me, explicit pthreading should never be done except with systems software. In which case you're going to be using a systems language like C, C++ or Rust anyway. Not the oddball middletier stuff that isn't high or low level like Go, Java and C#.
> this is more efficient than spinning up an OS thread per request
Linux is very fast at spawning threads. When you're doing I/O, it doesn't matter.
Go's concurrency is not "fundamentally different" from any of the other languages. It's just an implementation of thread-per-connection in userspace. This is an implementation detail, and for a typical Web server I think it doesn't matter much.
> In the case of an I/O bound Web server, I don't think that Go is going to appreciably result in "leaving money on the table". Python releases the GIL on I/O, which is where your time is largely spent.
GIL doesn't matter, the OP said he was writing single threaded application. This means he's doing async (unlikely) or blocking the entire application process (not just the current request) on I/O.
> Linux is very fast at spawning threads. When you're doing I/O, it doesn't matter.
Not as fast as goroutines, and spawn speed isn't the only (or even the most interesting) measure of efficiency--you can run several orders of magnitude more goroutines on a Linux box than you can threads. Of course, none of this is relevant to this conversation, since the OP is talking about running a single-threaded Python process per core.
> Go's concurrency is not "fundamentally different" from any of the other languages. It's just an implementation of thread-per-connection in userspace. This is an implementation detail, and for a typical Web server I think it doesn't matter much.
You're conflating Go's concurrency model with its webserver implementation. Go's concurrency model (movable M:N coroutines with implicit yielding and a synchronous abstraction over async IO) is fundamentally different than the other approaches (threads and vanilla coroutines with some blend of sync and unabstracted async I/O).
Since you mentioned thread-per-connection, Go's "threads" are much lighter than even Linux threads, so you can service many more simultaneous connections (by many orders of magnitude in extreme cases). This is especially true if you're not using async I/O, and blocking I/O is the default for most of these other languages.
Everything leaves some performance on the table. Everything also leaves some productivity on the table too.
Python is in my estimation 20-50% more productive as a language than Go. I run my web stuff on PyPy/gunicorn/Nginx which within reason, can't be leaving much to be wanted performance-wise.
I've given Go a good-go, but I haven't found anything that strikes that balance of ~25% more productive than Go, but still grant similar performance like PyPy does for me.
Go's biggest problem is a lack of generics, which means that in some cases, you have to fall back to duck typing. So the worst case is no worse than Python's best case.
The next-step isn't Go. It's Elixir. Or Rust if you need a systems language.
The CPython crowd apparently made some change between CPython 3.4 and CPython 3.5 which changed how Windows DLLs are called. This broke Py2exe, which no longer works for the current release of CPython. [1]
CPython's approach to C extensions usually requires having the same Microsoft C compiler version used to build Python. This is now a problem for Python 2.7, because the Microsoft compiler version used to build it is no longer available. Microsoft recently changed their API for the C runtime, and CPython has been changed accordingly.
CPython's "C interoperability story" isn't really that good on Windows.
[1] http://stevedower.id.au/blog/building-for-python-3-5/
When that is not enough, you can still recompile manually if need be, so long as source for the module is available (which it is in most cases).
On the other hand, native modules are such a big part of the Python ecosystem, that not supporting the API and ABI at all confines any alternative implementations to a very small niche.
There is a CMake build system for 2.7 that works great with VS2013-2015:
https://github.com/python-cmake-buildsystem/python-cmake-bui...
Note that the problem you mentioned w/ python 2.7 is gone: MS has made a .msi available free of charge to build C extensions for 2.7, indefinitely (thanks to Steve Dower): https://www.microsoft.com/en-gb/download/details.aspx?id=442...
This is less problematic on other OSes where GCC dominates and clang is compatible, but it can bite on HPC systems that use PGI compilers.
https://github.com/Microsoft/Pyjion
[1] http://blog.kevmod.com/2016/07/why-is-python-slow/
* the reference implementation is maintained by a team of contributors, very few people are paid full time to work on cpython. State of the art JIT is hard to come by when you can only afford working a few hours / week.
* contrary to JS, python had a huge set of packages available for a very long time, making "minor" language breaks unworkable.
* also contrary to JS, python had from the start a strong story to interoperate w/ the C ABI, which tightened a large subset of python libraries with the details of cpython implementation. Because of its "niche", JS never really had that problem. If you remove libraries w/ large C extensions, you remove a lot of what make python popular today (e.g. scientific usage is now the 2nd largest area of application of python after web development, before automation, testing, etc... which used to be where python shined).
https://news.ycombinator.com/item?id=8625280
https://news.ycombinator.com/item?id=7721096
The following may also be of interest:
- Why Python is Slow: Looking Under the Hood [0]
- Fast Python, Slow Python by Alex Gaynor [1]
[0] https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow....
[1] https://www.youtube.com/watch?v=7eeEf_rAJds
I'm not saying Python is the best language of all, but dissecting Python bytecode is kind of missing the point. Python's strong point is its ease of use.
(I agree with your observation, though.)
With a dynamic or duck type system, though, you can avoid a lot of writing out interfaces or making class hierarchies. You end up with something less structurally uniform, but with a much shorter time to get results.
If you care about type mismatches, use a statically typed language. If you just want to get something working, use a dynamically typed one.
First, there aren't really any compiled languages where the syntax is similar in ease of use to Python (or any other popular dynamic language I'm familiar with).
All the compiled languages with which I'm familiar require a significant amount more work to accomplish the same thing; C apps are full of boilerplate to detect error cases, allocate memory, check buffer lengths, etc.; dynamic languages handle this for you so the total amount of code tends to be lower.
Writing a simple program to test an algorithm or proof of concept is generally significantly less code, the code/test/debug loop tends to be significantly shorter, and you can often get to a solution significantly quicker while avoiding the typical bugs that come with lower-level languages.
Python also has a pretty substantial standard library which is easy to learn and use and very consistent, meaning much less copypasta code or external dependencies.
On top of that, a great deal of use-cases for languages like Python aren't CPU-bound, but rather I/O bound (disk or network). In cases where your app is CPU-dependent, it can often be trivial to replace that inner loop with optimized native code written in C or other languages, while still keeping the rest of your application easy to read and maintain.
Lastly, for complex algorithmic work, there's a lot of packages which implement standard data structures and algorithms in native code and provide usable, idiomatic interfaces in Python, which means that for a lot of things you can work in native code with the ease of Python, at a minuscule performance hit but a dramatic increase in programming efficiency.
Now if you are not doing something that requires performance, I think there is still a good case for static typing: advanced tooling. Dynamic language limit a lot the ability of the IDE to help you finding bug, making the language and the API self discoverable, auto completing the code, providing advanced analysis (who calls what) or refactoring (rename a property). Some people do not need that, but to those who rely on that, dynamic languages must really be easier to write by an order of magnitude to end up being faster to prototype.
If I was suggesting a language to a beginner, I would push for a language with a strong IDE.
To understand the appeal of dynamic languages, it's critical to consider the era they came from. Your statement is not true of the general programming landscape ten years ago. It's not really true of the general programming landscape today, for that matter, which is still by sheer mass C++ and Java. But 10 years ago, there weren't hardly any choices of statically-typed languages that weren't a bit of a bear to use. 1990s C++ is a monstrous nightmare. A lot of this monstrous nightmare was arguably the programming environment (Windows, Xwindows, UNIX, everything was a bear to program in back then), but when you took battle-scarred C++ Windows programmers and showed them
they weren't in a mood to analyze exactly which differences between Python/Perl and C++ were essential and which were accidental; they were too busy drinking in the joy.I think there's a general trend towards languages that either make the static typing easy enough to use that you don't feel like you're losing out on much, or make the static typing pay for itself better (Rust), and I think that will continue to be the story for the next ten years on the developing language front. I think it may be the case that if they weren't already entrenched, what I call the 1990s-style dynamic languages (because it's not just that they're "dynamic", Perl, Python, Javascript, and Ruby are almost the same language, compare with the "dynamic" Erlang to see a major difference) wouldn't succeed if someone wrote one for the first time this year. In fact they'd probably take a beating for the performance issues it would have. But they are entrenched and have over a decade of refined libraries, support documentation, tutorials, and of course, the big hitter, existing code. So they aren't going anywhere any time soon.
The other thing is tooling. There are lots of hardcore developers to whom even syntax highlighting is too much distraction, let alone error checking, auto completion, etc.
Well... To say the least I am not one of them. In fact to me, a large part of the simplicity of a language resides in the interaction with the IDE, the feedback it gives you on what's legal and what is not, where you may have a bug or an ambiguity, what can I do from there, etc. Dynamic languages defeat advanced and proactive tooling. And to me it makes them actually quite hard to work with, not self-discoverable, and all-in slower to develop. The time not spent writing a class or a return type ends up being spent on the reference page of the library (after you checked this is the correct version) checking what are the arguments of a function, etc.
Swift looks promising in this regard.
It now matches how the .Net CLR handles unicode. Which is a shame since Linux is kind of increasingly popular.
There are some recent developments where Microsoft is investing through the core dev surrogates into Python3 and speed with Pyjion. That is promising but it's a framework where it requires CPython3 to use a JIT, which strikes me as a way to enforce control- a response to PyPy and alternatives taking the mantle when they abdicated the throne with Python3. But it remains to be seen how it works out, and PyPy already is likely to remain the leader on faster-Python.