Does anyone know if there are some milestones in sight regarding packages which use numpy? e.g. "working scipy or sklearn".
EDIT: BTW I forgot to add: tremendous thanks to PyPy team for your continued work on this project. I benefit from it and always look forward to the next release!
Pypy3 is a separate project. From previous comments by the PyPy team, their customers are almost all targeting Python 2, hence Python 3 doesn’t get nearly as much attention.
Pypy3’s last release was 2.4.0 in 2014. It targets Python 3.2, which is relatively unsupported compared to Python 3.3.
Could be they never received as many (or many) potential customers.
There have been a few services/apps I was looking to use but had to turn down after talking to them because they didn't support Linux (Xamarin, Codename One, etc.) They all said they didn't receive as many requests for Linux as they did for Mac or Windows. I guess this would be similar.
I think you're confusing cause and effect. In open source world, you can't charge for something that exists. It needs to be a need to do so. In order to push PyPy into putting significant effort from a core team - we would need exactly that. A big chunk of money dedicated to just that. We've spent all the money we got and we did get something out of it. But it needs to be more seriously supported in order to kickstart that. Obviously people are free on whatever they want in their free time, but unless there is enough to push the PyPy3 support to a significant postion, it's unlikely to move at a higher pace.
The vast majority of python codebases are still in python 2, even though many of the popular framework have been ported to 3, and there's no reason for most of them to bother with the amount of work it can be to port to py3 when they're happy with py2. Whether it be devs like Dropbox or open source software like NVDA, they will stay in the world of python 2 for the years to come. The amount of customers they could get by supporting python 3 is definitely not worth the effort unless they get paid explicitly for it.
In terms of libraries, python 3 is "mostly" ready (though there are still issues with highly popular ones like wxwidgets. Or amazing tools like Pyjamas, which replicated GWT with Python instead of Java. Or Google AppEngine, which still shows no sign of moving to 3.) but large codebases outside of highly worked on web frameworks haven't moved on and there's not enough gain to justify the effort it takes to port them. The Python 3 debacle caused a lot of devs to also look at other languages for new projects. Old python 2 codebases will stay python 2, but new projects might not necessarily be python at all. The anger with Python 3 is real.
"Looking at these graphs, we can see that in the past year Python 3.x has grown from roughly 2% of the total downloads from PyPI to roughly 5-6%"
I expect things like AppEngine to deprecate Python rather than port to 3, when/if python 2 reaches EOL and not enough people gather to maintain or even evolve that old branch.
Python 3 users are a vocal bunch, but the war they're fighting is a lost one.
If you want to use just Python 3, after the initial port there is no issues with having to maintain Python 2/3 compatibility.
Working in open source, there is going to be a time when I stop supporting Python 2.x, and already I and others are building libraries that are Python 3.x first, are developed on Python 3.x and Python 2.x is an after thought in testing.
PyPy will eventually have to catch up to 3.x, and no, the future is definitely not 2.7.
No matter how hard you're wishing it to be true, does not make it so. PyPy will "have" to catch up? in the name of what? PyPy is but one of the many projects that are still thriving in the 2.x ecosystem, there's even a competitor to PyPy that was launched 2 years ago that is also a 2.x exclusive: Pyston.
https://blog.pyston.org
Yes, a very new implementation project that happens to focus on 2.7.
Python 2 gets all the better runtimes. Long after the end of life of the official CPython 2.7, we will still be running 2.x programs, and doing it on much better runtimes than CPython.
We've been hearing the same tirades of Python 3's future over and over again by its partisans ever since its release in 2008. 8 years have passed since then. 8 years will pass and Python 3 will still have nothing but a minority of vocal open source users. Python will be treated as a legacy language the way COBOL and BASIC are treated before py3 starts picking up any steam. That's kinda what some companies are already doing, like Dropbox. Switching to Go here and there, while keeping the maintenance of their very large 2.7 python codebase with absolutely no intention whatsoever of moving that behemoth to 3.x. Why bother? they could write new software features, fix bugs and so on instead of waste time on porting. Why waste so much time on porting when it brings so little benefits? Programming is ultimately about problem solving. Porting to python 3 doesn't make my software better. It's certainly not running any more efficient either, that's PyPy and Pyston territory (Using PyPy can allow you, under some circumstances, to seriously cut down the amount of hardware you need). It doesn't magically add new features, or fix the bugs.
Python 3 will go down in history as the best example of what you should never do while growing a language. People like to complain about C++'s complexity, but unlike Python 3, C++ isn't threatened by its own past selves, and that's despite adding features that are arguably far more compelling to look into than the various differences between Py2 and 3.
It's also very hard to regain lost trust. The trust that was lost after what was done to Python with Python 3 can never be fully regained. I can write software in C++, Java, C#, Javascript, Lisp, Perl 5 (which is still being worked on) and so on and have an expectation not to have to go through something like Py3. To a lesser extent, even Ruby, although it breaks compatibility from time to time, it never went through anything as major in a one time fashion as Py3 did, which is why the ruby ecosystem hasn't self destructed.
With moore's law being utterly dead, the fact that all the interesting improvements to language implementation are happening only to 2.x rather than 3.x really contradicts your idea that the future lies in 3.x.
You should point to: https://trollius.readthedocs.org/ which states that it is discontinued and you should just move on to Python 3.
I think you should give asyncio a try first (especially on Python 3.5 with async/await keywords) before stating there's no compelling reasons. Another strong (for me) reason to go with Python 3 is that it has bunch of old warts removed. The code is just easier to maintain.
Wow. Just wow. Congrats to the PyPy team. It never stops.
I stopped calling myself a Python programmer sometime ago and started saying I'm a PyPy programmer because this is the future of Python. The underlying basis for PyPy has big implications for a number of dynamic languages though so we should all be cheering it on and contributing.
For those who want better Python3 support, there's only ONE answer so no need to keep begging. Donate more money to the project or donate your talents. Patches welcome!
After they get momentum by attracting people with their speed improvements, it will be a lot easier to attract talent and money to tackle Python 3. The reason Python 3 is on the back burner is that there simply isn't a large enough audience right now. That could change if they keep increasing their reach.
There are major bug fixes and compatibility and performance improvements in this release. They're also asking for a large amount of money to start a push to add Py 3 support. And NumPy support is still improving - they'll gain a lot of users as that becomes complete. None of this seems like "maturity phase", especially in terms of userbase or funding.
"20% consumption" is more a factor of how much memory you have installed and how much you intend to use than PyPy's maturity. I frequently use malloc()/mmap() to consume much more than 20% of system memory.
> After they get momentum by attracting people with their speed improvements, it will be a lot easier to attract talent and money to tackle Python 3
Well, this is a catch-22, because folks like me and my team won't touch it unless it's Python 3. We're not a Python shop, but have flirted with Python before. It wouldn't make sense for us to write new code in Python 2.
>The reason Python 3 is on the back burner is that there simply isn't a large enough audience right now.
Funnily enough the reason I haven't adopted using PyPy is down to their lack of Python3 support. I think you're underestimating the amount of people using Python3.
I can't use PyPy until it implements a real Python 3 version, i.e. one that supports asyncio, which is the real future. Until then, I will just be reading about PyPy on Hacker News and just sighing...
Edit: For everything I use, I try to find an asyncio version - Slack client [0], SSH [1], etc.
While this has historically been the case, Python 3.4/3.5 has really sped up adoption of Python 3, and from an open source maintainer, Python 3 is making large inroads.
Also, libraries are slowly going to drop Python 2.x support and go Python 3 only.
>While this has historically been the case, Python 3.4/3.5 has really sped up adoption of Python 3, and from an open source maintainer, Python 3 is making large inroads.
If by great inroads you mean "around 6-10% in PyPI after 8 years of trying", then yes.
But unless it gets over 30-40% it's not anywhere near significant in my charts...
Its a JIT compiler run-time. You put a program in the JIT-compiler's run-time. And it was JIT compiled. The program you gave it just so happens to have Turning Complete behavior. It doesn't change anything the JIT compiler is doing.
It's like a compiler boot-strapping itself. It seems really impressive, but really the program is just doing its job on an input. The input just happens to be itself.
It's not about novelty, it's about realization. PyPy exists and works as both an open source project and as a codebase, which is no easier to achieve than coming up with an idea. That said, PyPy has yet to demonstrate real impact outside the Python community, and projects with a somewhat similar aim to what's being discussed (the Parrot VM) have likewise not gotten to significant uptake. We'll see. PyPy is definitely already useful at other things, though.
The post I was replying to was misunderstanding "this is new!" as "this is the first time this exists!" when it's more about "this is the first time this is available!", where availability is complex and defined by factors like governance, distribution, quality, accessibility, infrastructure, communication ... these are the things that tend to be make or break long after novelty had its moment.
Whether "this is the first time this is available!" is true or not is open to debate (it depends in part on how you weigh the listed concerns), but parent's post didn't read on it.
So you just adding a user program to a REPL/Parser and cramming it into a JIT?
You can do this in Javascript, or LLVM-IR just as easily.
The main blessing is also its biggest curse. You are writing a language to move fast, and change often. Its a good learning tool, but a constantly changing language is next to impossible to build value upon.
Did you actually look at the blog post? The novelty is the idea that you generate a JIT compiler from the interpreter using a few hints. Truffle does that, but Truffle is newer than PyPy.
Are you familiar of the state of the art in JIT compilers at the time PyPy's JIT was developed? It was absolutely a novel idea at the time, and was published as a novel result in a peer-reviewed paper. I'm not sure how else to convince you of that.
> However, applying an unmodified tracing JIT to a program that is itself a bytecode interpreter results in very limited or no speedup. In this paper we show how to guide tracing JIT compilers to greatly improve the speed of bytecode interpreters. One crucial point is to unroll the bytecode dispatch loop, based on two kinds of hints provided by the implementer of the bytecode interpreter.
> Applying a trace-based optimizer to an interpreter and adding hints to help the tracer produce better results has been tried before in the context of the DynamoRIO project [27], which has been a great inspiration for our work. They achieve the same unrolling of the interpreter loop so that the unrolled version corresponds to the loops in the user pro- gram. However the approach is greatly hindered by the fact that they trace on the machine code level and thus have no high-level information available about the interpreter. This makes it necessary to add quite a large number of hints, be- cause at the assembler level it is not really visible anymore that e.g., a bytecode string is immutable. Also more ad- vanced optimizations like allocation removal would not be possible with that approach.
I don't want to be part of that "future". PyPy is good but "closed": It's very hard to tinker due to excessive compile times and unreachability of the C level (no, cffi is no substitute).
If you just use Python as a managed language, fine, but the magic is gone. I'd probably use Java if I wanted a "closed" language.
That's from almost 3 years ago. Have you ran the attached patch to check the benchmarks with a more current release? I don't know if that's been fixed, but it probably wouldn't take much time to test out.
Are there major technical issues that make it hard to do PyPy for Python 3.5, or is it just a matter of wanting to focus 100% on the fixed target of 2.7?
The main issue is that PyPy's commercial target audience (people who need fast Python, particularly for numerical calculations) doesn't care about Python 3. There's not nearly as much potential funding for Python 3 as for Python 2.
Technically, Python 3 is not any harder to implement than Python 2, it's just useless.
I think you're partially right about the reasons (the other half of the equation simply being the momentum of Python 2 efforts), but I think your last clause ("it's just useless") is incorrect.
I can tell you for 100% certain that, if a project of mine could run on a version of "Python" that was even twice as fast, I would switch immediately, because 6 application servers could become 3. If the gains are even greater, that's tremendous.
It's not just scientific computing that benefits, and in fact, a lot of the scientific / numerical analysis done in Python is really done in C / C ++ anyway (think Numpy, et al which delegate to compiled code for much of their expensive operations).
I think it comes down to the question how much funding would such folks be able to raise to support that effort. It is my understanding that PyPy is quite resource starved.
It's a matter of resources. I donated, and I invite everyone earning money with Python to do the same (http://pypy.org/py3donate.html).
More generally, the Python community really lacks money compared to other ones. GO, PHP and JS all have bigs players spending a lot of cash on it. While some companies does invest in Python, they don't spend nearly the same amount on it, and the PSF has a very tigh budget.
One of the reason is that Python is "good enough", and so people don't invest on it because they don't need more from it. While JS was so slow that Google spent millions to create the V8. It's sad, but being clean and robust and strongly community driven leads to a lack of funding for Python. I wish we had a Mark shuttlework for the language.
Because the future is whatever you're building software in today.
For the vast majority that's still Python2. For those starting new projects that might be Go, Elixir, Python2 or Python3.
Just because some entity like the PSF declares Python3 the future doesn't make it so. Even if some people buy into it and believe it. It has to take over from the bottom up and its been floundering since 3.0 landed in 2008.
It's not laziness, it's basic cost/benefit evaluation. Many feel that 2.7 isn't lacking in such a way that 3.x offers so great a benefit as to outweigh its costs (effort to convert, loss of some portion of the long tail of available packages.) Others disagree.
The success of 2.7 is an amazing thing. Most languages have one great major version, some have zero, Python has two.
Laziness? It's non-trivial to migrate to Python 3 if you've spent any time building up any number of bespoke libraries.
Look at it this way:
1) I can spend 2 days writing, testing, reviewing and pushing code that works perfectly fine in Python 2, or
2) I can spend a couple of weeks+ porting all the libraries I need for the tool to add python 3 support, maintaining backwards compatibility, fixing tests, reviewing etc. etc. etc., then spend 2 days to write the actual code I started out needing to do.
Which one do you think I'm going to be able to justify, given there is always a backlog of work? Multiple weeks of developer time is a non-trivial expense. There has to be a significant technical advantage to justify the effort and for a lot of things that Python is used for there just isn't enough of one.
I'm not anti-Python 3. I like it, in fact. Where I can use it, I do. It's just a lot of the time I can't because of all of the associated extra work involved.
This attitude is the exact reason why Python 3 hasn't been getting the uptake that was expected back when it was released. People who sit around and say "the future is today, and I already have Python 2 today" go on to start even totally new projects in Python 2, when they very well could and probably should be using Python 3. As the sibling comment points out, it's just laziness (and rationalization). There's no two ways around it.
Like it or not, the fact of the matter is: for Python programmers, Python 3 is the future. It's really only a matter of time, even if it's much more time than was expected.
This attitude is the exact reason why Python 3 hasn't been getting the uptake that was expected back when it was released.
For me Python's strength is the libraries not the language, Python 3 decided to not support all this libraries, so calling it laziness for users to not immediately move to python 3 is a really bad way to put it.
Who said anything about immediacy? Python 3 has been out for nearly 8 years. That's more than enough time to update libraries, especially given that the list of incompatibilities between Python 2 and Python 3 is tiny.
So, what have the library developers been doing for the past 8 years? Probably adding features to the Python 2 version of the library. Why? Because Python 3 has seen low adoption. Why? Because the libraries haven't been updated yet. Why? ...
It's an endless cycle. If people want Python 3 to be adopted more widely, they ought to adopt it themselves (and many programmers have, though clearly not enough).
You're confused. The unfortunate matter is that it has few to no technical merits. It defined technological churn with total absence of technical innovation. THAT is why it failed. Not because "everyone is lazy".
Python3 was lazy. Demanding that everyone including PyPy port, is lazy. Let Python3 live or die based on its technical merits like every other piece of technology.
Not to mention I wish the sword you're wielding cut both ways. You aren't entitled to demand someone else's labor any more than your buddies that litter every PyPy post demanding better Python3 support are. I hope you're harassing them for being lazy.
You made your own choice. You should have picked based on technical superiority rather than propaganda. To be honest with you, you should be ignored as you chose VERY poorly. Don't dare tell someone else what to use.
There are so many better choices to build new software in today than Python3. Python2 on PyPy is one of those. Those people can still port to Python3 if it ever does make sense. Even if that idea clearly threatens some of you, that's the smart play.
Yes, I have made my choice. And frankly, it isn't Python at all.
But the fact of the matter is that Python 3 is being actively developed, whereas Python 2 is in maintenance mode, and for how much longer? One of these days, Python 2 will become unsupported, and at that time you either finally move to Python 3, or you fork -- at which point you're no longer, strictly-speaking, programming in Python.
You're right that it's also lazy for people to simply demand Python 3 support from the PyPy project without being willing to invest in the effort. I have nothing to add to that, really.
I should also add that I'm probably one of the laziest programmers to ever exist, so I'm not trying to demonstrate superiority or anything like that. I'm simply pointing out that refusing to use Python 3 just because it hasn't seen much uptake only perpetuates the problem.
You'd be correct. The distinction is whose fault is that, the lazy Python3 core dev team or the lazy userbase? We'll just assume everyone is lazy in effort towards equality. Python3 was created without any backwards compatibility to make the core dev teams job easier, nothing more nothing less. LAZY?! Yes. That's their prerogative, but the rest of us don't have to use it either.
When I say lazy in this post, I mean simply acting on one's self-interests. Because that's what every party involved here is doing.
It's important to note the difference between Python the language, and CPython. Using Python2 doesn't mean you're using some deprecated language, there's virtually no difference. That's what made the break such a shame honestly. Moving to Python3 with new code is easy. Most people just have few great reasons to now with their existing projects.
People need to stop harassing the community and start harassing Guido van Rossum and the core dev team. Usually when people make a mistake, people fix it. GvR and his rogue band of developers have not corrected the Python3 mistake in any way. They've simply continued their mistake for so long, that people are becoming twisted thinking that it's EVERYONE ELSE who is at fault! Amazing really.
CPython2 maintenance mode is just a way for the core dev team to maintain their grip around this whole debacle. If they had the guts to dump it in 2015 they would've lost complete control of the situation. It wasn't out of compassion as they say. Because they simply removed the incentive for a 3rd party to swoop in and takeover CPython2.
Even that is backfiring now. People[0] are noticing how Python3 is turning into feature soup, which is never an improvement. Thus Python2's conservatism is starting to be seen as an asset.
As far as CPython3 being under active development, so is PyPy and it's superior to CPython. The language doesn't change too much at this point. Python is Python for the most part. When the time comes for someone on 2 to move to 3, it will be a non-issue. They're not "losing out" today in any way by acting in their best interests using 2.
Python3 has nothing that didn't already exist as a 3rd party library, nor does it have anything that doesn't exist in form of a backport.
Thus technical innovation contained within is zero.
Programmers aren't stupid. And I think the PSF folks think we are. Or just blatantly disregard the community that truly made Python what it is.
The existing Python2 community is overtly disrespected when called "lazy" and all this type of harassment that goes on. I know I'm in the majority opinion because upvotes don't lie.
I'm just a guy who likes "Python", the language. I can't blame you for not using it and honestly, what a mess Guido and his core dev team has created right?
Python is my preference but I'm not held hostage by the PSF or Python3. I use Python2, Python3, Boo, PyPy. I don't hate Python3 and don't know anyone who does. It's just no slam dunk, and that's the core development team's fault- and they may lose their battle against Python fans as a result. I do highly dislike the groupthink about Python3, with a portion of the Python community on this topic.
One more thing to add, is Python3 inevitable? No. It just fragmented the Python ecosystem. Nothing more. We all won't be on a single version again. GvR and his band of rogue developers should be ashamed of themselves. Again, change happens from the bottom up. Not top down. They failed the community and should abdicate their roles but I'm afraid their egos are too big.
wow, talk about ego. i disagree with pretty much your whole post, especially with conclusions you draw (abdicate? really?) and with most of the site you cite. (PEP501 is madness, never heard of it until now.)
i think you should abdicate from posting on HN about Python. you introduce conflict where there was only laziness. go to politics, leave engineers out of it.
Ok glad you voiced your opinion that you disagree. I hope you feel better now, since it's clear what I have to say is so true that you're tormented to post a response. Next time that would be better done with a downvote because you have zero to add.
Reading comprehension is key. I was defending against those creating conflict. There's nothing more political than Python3. And overthrow all BDFLs. That was the whole point, that went right over your head. You are not an engineer. Go back to Reddit.
The irony of lazy people calling other people lazy is pretty hilarious. Since you harass the PyPy team and 90% of the Python community to port to 3 while you remain a non-contributing zero? There's only one thing to say to you:
This comment breaks the HN guidelines, and unfortunately you've been doing so quite a bit in other comments too. We ban accounts that do this, so please stop doing it. Instead, please (re)-read the following and post civilly and substantively or not at all:
I read those links and as far as I see, I was confronting other folks who were breaking this rule-
'Please avoid introducing classic flamewar topics unless you have something genuinely new to say about them. '
While adding a lot of new substance to a topic that breaks the mold of "port PyPy to Python3" and how to achieve those goals. Perhaps my tone was a little strong and regrettable. I did break this one by accident but mostly out of frustration to a substanceless post. I regret the error.
'Please don't bait other users by inviting them to downvote you or announce that you expect to get downvoted. '
Glad to see you calling attention to HN's anti-flamewar rule. You're right that it is an important one, and people unfortunately break it all the time. Still, a "confronting" response just makes things worse, and one violation of the HN guidelines doesn't justify another.
It isn't easy to remain civil and substantive in response to comments that break the rules, especially when one is annoyed. I don't get it right all the time myself.
> The distinction is whose fault is that, the lazy Python3 core dev team or the lazy userbase?
I agree that Python core devs should have not made such breaking changes, And hindsight is 20/20, even for them. I also am a bit annoyed that they've been pretty stubborn about adding additional compatibility layers between the versions. (If six (maintained by a core dev) can help with compatibility, why can't the standard library have those shims too? :)
I also like Go's approach to unicode better than Python 3's approach.
"harassing" the core devs may have been helpful in back in 2008-2012, however, at this point it's a little late to start complaining. I doubt it will help at all.
> Python2's conservatism is starting to be seen as an asset
I totally agree. It's an unfortunate problem.
> is Python3 inevitable?
Yes. It's only a matter of time.
- Django is planning on removing Python 2 support from master in January.
- Nearly all popular libraries are compatible or have compatible forks.
I agree with a lot of what you're saying. On Django though (or any library that intends to drop 2.x support), I feel if they truly go through with it, they'll be in for a shock.
It will more than likely hurt them more than it hurts Python2 (I also don't understand the obsession with people wanting to kill or hurt Python2). People and companies may be more likely to start new projects in Flask or another alternative, maybe even fork Django which would be a disaster for them.
But as of today, Django's roadmap has the LTS release that supports 2.x as an unknown EOL. So it won't be anytime soon.
Overall I tend to agree that Python3 will eventually take over. That wasn't what I was arguing and I never said otherwise in my posts. My problem is with the harassment over Python2 folks to port and all the associated nonsense. That's the part that is more counterproductive than anything.
"A matter of time" is a really bad last leg to stand on as well, because time has an awfully broad range. :)
As far as the other points, I have doubts about the impact of Ubuntu or others defaulting to Python3. That won't stop me from building for PyPy, I can say that at least.
In general I feel that I'm on a similar page to your stance though. I do use Python3, and have since the 3.0 release in testing alone. For someone who has tried every release and found performance issues and bugs galore, it's a testament that I (and others) haven't given up completely. I think many have.
Working in open source on a variety of projects, many of us are starting to look forward to a Python 3 only world, and Python 2.x is not where we develop.
You are going to start seeing libraries come out that don't test on Python 2.6 and don't work there either.
When I went full-time Python in 2013 I directly started with Python 3. The new version of Python actually has a very healthy ecosystem, just about every good library supports it.
By now the lack of Python 3 is actually a very good filter when choosing a library: To me it says badly maintained. Many of the libraries which do not support Python 3 are also still hosted on SourceForge.
>Because the future is whatever you're building software in today.
This is a seriously unhealthy attitude for software engineering. This attitude will create legacy systems and legacy systems that will live on forever.
There's not much wrong with such legacy systems. Unfortunately, they are more like exceptions. Most likely, your legacy systems break and you have no idea how to fix them. Or, you have to hire one guy for life because he is the only one who can maintain and fix your systems. And you pray that he doesn't get sick or hit by the bus.
Startup time is ok. "Hello, World" runs in something like 0.08s on my computer. Not sure what this would be like for an app that actually has a lot of code. Startup time is a metric I care a lot about, because it speaks to whether it feels snappy to use for command-line applications.
But holy crap the binary is enormous. I'm getting 22MB of text, compared with 991kb for CPython.
Why does PyPy not get support from the big Python based giants - the Dropboxes or the Googles of the world. It seems to me that PyPy is struggling for funding.
PyPy has the potential to become the canonical Python - if the entire team had their finances taken care of (and not have to do things like http://baroquesoftware.com/ ).
PyPy is in the unfortunate position of being both uninteresting to many of those who rely on Python (numpy, etc and Spark work fine and are fast without PyPy) and irrelevant to application developers who switched off Python because it was too slow (to things like Go, Rust, etc).
(This isn't an insult to PyPy, btw--it's an amazing project that has accomplished a lot. I'm just explaining why it doesn't get the support one might expect.)
but assuming the vast majority of python users are using it for web application and so is Dropbox/Google, it seems to me that PyPy has a marketing+PR problem than anything else.
That's not a fair assumption: a huge number of Python users are using it for some level of data processing and data analysis. And that percentage is growing each day, because Python is a very hot technology in data science.
Had to happen. New PyPy release, new 2vs3 feud in comments. Can't we all just get along?
Anyway, my $0.02: I'm looking forward to test this with Hy (hylang.org) and seeing if some of the module referencing is fixed. Makes for a great JITed LISP already.
102 comments
[ 4.6 ms ] story [ 160 ms ] threadDoes anyone know if there are some milestones in sight regarding packages which use numpy? e.g. "working scipy or sklearn".
EDIT: BTW I forgot to add: tremendous thanks to PyPy team for your continued work on this project. I benefit from it and always look forward to the next release!
(They've largely used up the targeted crowdfunding; the unused money isn't enough to make a large amount of progress)
Pypy3’s last release was 2.4.0 in 2014. It targets Python 3.2, which is relatively unsupported compared to Python 3.3.
Here is a recent comment by a core member of the team on future Pypy3 work: https://news.ycombinator.com/item?id=11262150.
Odd statement. How can they have any customers who target Python 3 if they don't support it?
There have been a few services/apps I was looking to use but had to turn down after talking to them because they didn't support Linux (Xamarin, Codename One, etc.) They all said they didn't receive as many requests for Linux as they did for Mac or Windows. I guess this would be similar.
In terms of libraries, python 3 is "mostly" ready (though there are still issues with highly popular ones like wxwidgets. Or amazing tools like Pyjamas, which replicated GWT with Python instead of Java. Or Google AppEngine, which still shows no sign of moving to 3.) but large codebases outside of highly worked on web frameworks haven't moved on and there's not enough gain to justify the effort it takes to port them. The Python 3 debacle caused a lot of devs to also look at other languages for new projects. Old python 2 codebases will stay python 2, but new projects might not necessarily be python at all. The anger with Python 3 is real.
See also: https://caremad.io/2015/04/a-year-of-pypi-downloads/ https://www.reddit.com/r/Python/comments/45sm94/what_are_the...
"Looking at these graphs, we can see that in the past year Python 3.x has grown from roughly 2% of the total downloads from PyPI to roughly 5-6%"
I expect things like AppEngine to deprecate Python rather than port to 3, when/if python 2 reaches EOL and not enough people gather to maintain or even evolve that old branch.
Python 3 users are a vocal bunch, but the war they're fighting is a lost one.
But why?
http://python-future.org/compatible_idioms.html
Python3 is a different language than Python 2.7 and there is no compelling reason to move from 2.7 to 3.
Most of what would have been compelling reasons are being handled in PyPy. PyPy is the present and future of Python and that future is 2.7.
Working in open source, there is going to be a time when I stop supporting Python 2.x, and already I and others are building libraries that are Python 3.x first, are developed on Python 3.x and Python 2.x is an after thought in testing.
PyPy will eventually have to catch up to 3.x, and no, the future is definitely not 2.7.
Python 2 gets all the better runtimes. Long after the end of life of the official CPython 2.7, we will still be running 2.x programs, and doing it on much better runtimes than CPython.
We've been hearing the same tirades of Python 3's future over and over again by its partisans ever since its release in 2008. 8 years have passed since then. 8 years will pass and Python 3 will still have nothing but a minority of vocal open source users. Python will be treated as a legacy language the way COBOL and BASIC are treated before py3 starts picking up any steam. That's kinda what some companies are already doing, like Dropbox. Switching to Go here and there, while keeping the maintenance of their very large 2.7 python codebase with absolutely no intention whatsoever of moving that behemoth to 3.x. Why bother? they could write new software features, fix bugs and so on instead of waste time on porting. Why waste so much time on porting when it brings so little benefits? Programming is ultimately about problem solving. Porting to python 3 doesn't make my software better. It's certainly not running any more efficient either, that's PyPy and Pyston territory (Using PyPy can allow you, under some circumstances, to seriously cut down the amount of hardware you need). It doesn't magically add new features, or fix the bugs.
Python 3 will go down in history as the best example of what you should never do while growing a language. People like to complain about C++'s complexity, but unlike Python 3, C++ isn't threatened by its own past selves, and that's despite adding features that are arguably far more compelling to look into than the various differences between Py2 and 3. It's also very hard to regain lost trust. The trust that was lost after what was done to Python with Python 3 can never be fully regained. I can write software in C++, Java, C#, Javascript, Lisp, Perl 5 (which is still being worked on) and so on and have an expectation not to have to go through something like Py3. To a lesser extent, even Ruby, although it breaks compatibility from time to time, it never went through anything as major in a one time fashion as Py3 did, which is why the ruby ecosystem hasn't self destructed.
With moore's law being utterly dead, the fact that all the interesting improvements to language implementation are happening only to 2.x rather than 3.x really contradicts your idea that the future lies in 3.x.
That's just not true at all. Asyncio is a perfect example.
https://pypi.python.org/pypi/trollius
Or, better, gevent:
http://www.gevent.org
No compelling reason. That is why 8 years later 2.7 is still used by the majority.
You should point to: https://trollius.readthedocs.org/ which states that it is discontinued and you should just move on to Python 3.
I think you should give asyncio a try first (especially on Python 3.5 with async/await keywords) before stating there's no compelling reasons. Another strong (for me) reason to go with Python 3 is that it has bunch of old warts removed. The code is just easier to maintain.
> http://python-future.org/compatible_idioms.html
It's bit a BS, majority and I would say 90% of that you can simply use the Python 3 syntax and both Python 2 and 3 will accept it.
Also many people who program in Python 2.7 already use the new syntax without even realizing it.
I stopped calling myself a Python programmer sometime ago and started saying I'm a PyPy programmer because this is the future of Python. The underlying basis for PyPy has big implications for a number of dynamic languages though so we should all be cheering it on and contributing.
For those who want better Python3 support, there's only ONE answer so no need to keep begging. Donate more money to the project or donate your talents. Patches welcome!
EDIT: strike this, I misunderstood.
Well, this is a catch-22, because folks like me and my team won't touch it unless it's Python 3. We're not a Python shop, but have flirted with Python before. It wouldn't make sense for us to write new code in Python 2.
Funnily enough the reason I haven't adopted using PyPy is down to their lack of Python3 support. I think you're underestimating the amount of people using Python3.
I'm guessing that most people who use Python 3 are using recent versions of pip that have caching built in, so that may skew the results a bit.
https://caremad.io/2015/04/a-year-of-pypi-downloads/
>I'm guessing that most people who use Python 3 are using recent versions of pip that have caching built in, so that may skew the results a bit.
Since pip updates itself too, why wouldn't people using Python 2 also have newer versions?
Edit: For everything I use, I try to find an asyncio version - Slack client [0], SSH [1], etc.
[0]: https://github.com/gfreezy/slacker-asyncio/
[1]: https://github.com/ronf/asyncssh
Also, libraries are slowly going to drop Python 2.x support and go Python 3 only.
If by great inroads you mean "around 6-10% in PyPI after 8 years of trying", then yes.
But unless it gets over 30-40% it's not anywhere near significant in my charts...
All the developers needed was a kick in the butt. If only PSF did that sooner.
Isn't it just a tracing JIT? Aren't these relatively well understood and implemented dozens of different ways?
http://morepypy.blogspot.com/2011/04/tutorial-writing-interp...
See how PyPy might be a tinsy bit more relevant / meaningful to professional developers.
Its a JIT compiler run-time. You put a program in the JIT-compiler's run-time. And it was JIT compiled. The program you gave it just so happens to have Turning Complete behavior. It doesn't change anything the JIT compiler is doing.
It's like a compiler boot-strapping itself. It seems really impressive, but really the program is just doing its job on an input. The input just happens to be itself.
I have no idea what you are getting at here.
Whether "this is the first time this is available!" is true or not is open to debate (it depends in part on how you weigh the listed concerns), but parent's post didn't read on it.
You can do this in Javascript, or LLVM-IR just as easily.
The main blessing is also its biggest curse. You are writing a language to move fast, and change often. Its a good learning tool, but a constantly changing language is next to impossible to build value upon.
Are you familiar of the state of the art in JIT compilers at the time PyPy's JIT was developed? It was absolutely a novel idea at the time, and was published as a novel result in a peer-reviewed paper. I'm not sure how else to convince you of that.
> However, applying an unmodified tracing JIT to a program that is itself a bytecode interpreter results in very limited or no speedup. In this paper we show how to guide tracing JIT compilers to greatly improve the speed of bytecode interpreters. One crucial point is to unroll the bytecode dispatch loop, based on two kinds of hints provided by the implementer of the bytecode interpreter.
> Applying a trace-based optimizer to an interpreter and adding hints to help the tracer produce better results has been tried before in the context of the DynamoRIO project [27], which has been a great inspiration for our work. They achieve the same unrolling of the interpreter loop so that the unrolled version corresponds to the loops in the user pro- gram. However the approach is greatly hindered by the fact that they trace on the machine code level and thus have no high-level information available about the interpreter. This makes it necessary to add quite a large number of hints, be- cause at the assembler level it is not really visible anymore that e.g., a bytecode string is immutable. Also more ad- vanced optimizations like allocation removal would not be possible with that approach.
http://dl.acm.org/citation.cfm?id=1565827
If you just use Python as a managed language, fine, but the magic is gone. I'd probably use Java if I wanted a "closed" language.
https://bitbucket.org/pypy/pypy/issues/1538/
Technically, Python 3 is not any harder to implement than Python 2, it's just useless.
I can tell you for 100% certain that, if a project of mine could run on a version of "Python" that was even twice as fast, I would switch immediately, because 6 application servers could become 3. If the gains are even greater, that's tremendous.
It's not just scientific computing that benefits, and in fact, a lot of the scientific / numerical analysis done in Python is really done in C / C ++ anyway (think Numpy, et al which delegate to compiled code for much of their expensive operations).
More generally, the Python community really lacks money compared to other ones. GO, PHP and JS all have bigs players spending a lot of cash on it. While some companies does invest in Python, they don't spend nearly the same amount on it, and the PSF has a very tigh budget.
One of the reason is that Python is "good enough", and so people don't invest on it because they don't need more from it. While JS was so slow that Google spent millions to create the V8. It's sad, but being clean and robust and strongly community driven leads to a lack of funding for Python. I wish we had a Mark shuttlework for the language.
For the vast majority that's still Python2. For those starting new projects that might be Go, Elixir, Python2 or Python3.
Just because some entity like the PSF declares Python3 the future doesn't make it so. Even if some people buy into it and believe it. It has to take over from the bottom up and its been floundering since 3.0 landed in 2008.
The success of 2.7 is an amazing thing. Most languages have one great major version, some have zero, Python has two.
Look at it this way:
1) I can spend 2 days writing, testing, reviewing and pushing code that works perfectly fine in Python 2, or
2) I can spend a couple of weeks+ porting all the libraries I need for the tool to add python 3 support, maintaining backwards compatibility, fixing tests, reviewing etc. etc. etc., then spend 2 days to write the actual code I started out needing to do.
Which one do you think I'm going to be able to justify, given there is always a backlog of work? Multiple weeks of developer time is a non-trivial expense. There has to be a significant technical advantage to justify the effort and for a lot of things that Python is used for there just isn't enough of one.
I'm not anti-Python 3. I like it, in fact. Where I can use it, I do. It's just a lot of the time I can't because of all of the associated extra work involved.
Like it or not, the fact of the matter is: for Python programmers, Python 3 is the future. It's really only a matter of time, even if it's much more time than was expected.
For me Python's strength is the libraries not the language, Python 3 decided to not support all this libraries, so calling it laziness for users to not immediately move to python 3 is a really bad way to put it.
So, what have the library developers been doing for the past 8 years? Probably adding features to the Python 2 version of the library. Why? Because Python 3 has seen low adoption. Why? Because the libraries haven't been updated yet. Why? ...
It's an endless cycle. If people want Python 3 to be adopted more widely, they ought to adopt it themselves (and many programmers have, though clearly not enough).
https://python3wos.appspot.com/
Python3 was lazy. Demanding that everyone including PyPy port, is lazy. Let Python3 live or die based on its technical merits like every other piece of technology.
Not to mention I wish the sword you're wielding cut both ways. You aren't entitled to demand someone else's labor any more than your buddies that litter every PyPy post demanding better Python3 support are. I hope you're harassing them for being lazy.
You made your own choice. You should have picked based on technical superiority rather than propaganda. To be honest with you, you should be ignored as you chose VERY poorly. Don't dare tell someone else what to use.
There are so many better choices to build new software in today than Python3. Python2 on PyPy is one of those. Those people can still port to Python3 if it ever does make sense. Even if that idea clearly threatens some of you, that's the smart play.
But the fact of the matter is that Python 3 is being actively developed, whereas Python 2 is in maintenance mode, and for how much longer? One of these days, Python 2 will become unsupported, and at that time you either finally move to Python 3, or you fork -- at which point you're no longer, strictly-speaking, programming in Python.
You're right that it's also lazy for people to simply demand Python 3 support from the PyPy project without being willing to invest in the effort. I have nothing to add to that, really.
I should also add that I'm probably one of the laziest programmers to ever exist, so I'm not trying to demonstrate superiority or anything like that. I'm simply pointing out that refusing to use Python 3 just because it hasn't seen much uptake only perpetuates the problem.
When I say lazy in this post, I mean simply acting on one's self-interests. Because that's what every party involved here is doing.
It's important to note the difference between Python the language, and CPython. Using Python2 doesn't mean you're using some deprecated language, there's virtually no difference. That's what made the break such a shame honestly. Moving to Python3 with new code is easy. Most people just have few great reasons to now with their existing projects.
People need to stop harassing the community and start harassing Guido van Rossum and the core dev team. Usually when people make a mistake, people fix it. GvR and his rogue band of developers have not corrected the Python3 mistake in any way. They've simply continued their mistake for so long, that people are becoming twisted thinking that it's EVERYONE ELSE who is at fault! Amazing really.
CPython2 maintenance mode is just a way for the core dev team to maintain their grip around this whole debacle. If they had the guts to dump it in 2015 they would've lost complete control of the situation. It wasn't out of compassion as they say. Because they simply removed the incentive for a 3rd party to swoop in and takeover CPython2.
Even that is backfiring now. People[0] are noticing how Python3 is turning into feature soup, which is never an improvement. Thus Python2's conservatism is starting to be seen as an asset.
As far as CPython3 being under active development, so is PyPy and it's superior to CPython. The language doesn't change too much at this point. Python is Python for the most part. When the time comes for someone on 2 to move to 3, it will be a non-issue. They're not "losing out" today in any way by acting in their best interests using 2.
Python3 has nothing that didn't already exist as a 3rd party library, nor does it have anything that doesn't exist in form of a backport. Thus technical innovation contained within is zero.
Programmers aren't stupid. And I think the PSF folks think we are. Or just blatantly disregard the community that truly made Python what it is.
The existing Python2 community is overtly disrespected when called "lazy" and all this type of harassment that goes on. I know I'm in the majority opinion because upvotes don't lie.
I'm just a guy who likes "Python", the language. I can't blame you for not using it and honestly, what a mess Guido and his core dev team has created right?
Python is my preference but I'm not held hostage by the PSF or Python3. I use Python2, Python3, Boo, PyPy. I don't hate Python3 and don't know anyone who does. It's just no slam dunk, and that's the core development team's fault- and they may lose their battle against Python fans as a result. I do highly dislike the groupthink about Python3, with a portion of the Python community on this topic.
One more thing to add, is Python3 inevitable? No. It just fragmented the Python ecosystem. Nothing more. We all won't be on a single version again. GvR and his band of rogue developers should be ashamed of themselves. Again, change happens from the bottom up. Not top down. They failed the community and should abdicate their roles but I'm afraid their egos are too big.
[0]http://learning-python.com/books/python-changes-2014-plus.ht...
i think you should abdicate from posting on HN about Python. you introduce conflict where there was only laziness. go to politics, leave engineers out of it.
Reading comprehension is key. I was defending against those creating conflict. There's nothing more political than Python3. And overthrow all BDFLs. That was the whole point, that went right over your head. You are not an engineer. Go back to Reddit.
The irony of lazy people calling other people lazy is pretty hilarious. Since you harass the PyPy team and 90% of the Python community to port to 3 while you remain a non-contributing zero? There's only one thing to say to you:
Get to porting.
https://news.ycombinator.com/newsguidelines.html
https://news.ycombinator.com/newswelcome.html
'Please avoid introducing classic flamewar topics unless you have something genuinely new to say about them. '
While adding a lot of new substance to a topic that breaks the mold of "port PyPy to Python3" and how to achieve those goals. Perhaps my tone was a little strong and regrettable. I did break this one by accident but mostly out of frustration to a substanceless post. I regret the error.
'Please don't bait other users by inviting them to downvote you or announce that you expect to get downvoted. '
It isn't easy to remain civil and substantive in response to comments that break the rules, especially when one is annoyed. I don't get it right all the time myself.
I agree that Python core devs should have not made such breaking changes, And hindsight is 20/20, even for them. I also am a bit annoyed that they've been pretty stubborn about adding additional compatibility layers between the versions. (If six (maintained by a core dev) can help with compatibility, why can't the standard library have those shims too? :)
I also like Go's approach to unicode better than Python 3's approach.
"harassing" the core devs may have been helpful in back in 2008-2012, however, at this point it's a little late to start complaining. I doubt it will help at all.
> Python2's conservatism is starting to be seen as an asset
I totally agree. It's an unfortunate problem.
> is Python3 inevitable?
Yes. It's only a matter of time.
- Django is planning on removing Python 2 support from master in January.
- Nearly all popular libraries are compatible or have compatible forks.
- Large python projects, like OpenStack are putting in the work to be Python 3 compatible. http://blogs.rdoproject.org/7894/status-of-python-3-in-opens...
- Ubuntu and Fedora now ship without /usr/bin/python available by default. RHEL/CentOS 8 may very well do the same.
- Debian has plans of removing Python 2 completely from it's repository not long after PSF drops support.
It will more than likely hurt them more than it hurts Python2 (I also don't understand the obsession with people wanting to kill or hurt Python2). People and companies may be more likely to start new projects in Flask or another alternative, maybe even fork Django which would be a disaster for them.
But as of today, Django's roadmap has the LTS release that supports 2.x as an unknown EOL. So it won't be anytime soon.
Overall I tend to agree that Python3 will eventually take over. That wasn't what I was arguing and I never said otherwise in my posts. My problem is with the harassment over Python2 folks to port and all the associated nonsense. That's the part that is more counterproductive than anything.
"A matter of time" is a really bad last leg to stand on as well, because time has an awfully broad range. :)
As far as the other points, I have doubts about the impact of Ubuntu or others defaulting to Python3. That won't stop me from building for PyPy, I can say that at least.
In general I feel that I'm on a similar page to your stance though. I do use Python3, and have since the 3.0 release in testing alone. For someone who has tried every release and found performance issues and bugs galore, it's a testament that I (and others) haven't given up completely. I think many have.
You are going to start seeing libraries come out that don't test on Python 2.6 and don't work there either.
By now the lack of Python 3 is actually a very good filter when choosing a library: To me it says badly maintained. Many of the libraries which do not support Python 3 are also still hosted on SourceForge.
This is a seriously unhealthy attitude for software engineering. This attitude will create legacy systems and legacy systems that will live on forever.
[1]: http://morepypy.blogspot.de/2016/04/pypy-enterprise-edition....
But holy crap the binary is enormous. I'm getting 22MB of text, compared with 991kb for CPython.
The web server type stuff works. The scientific computing type stuff mostly doesn't work.
Is there a particular reason for this ?
(This isn't an insult to PyPy, btw--it's an amazing project that has accomplished a lot. I'm just explaining why it doesn't get the support one might expect.)
Anyway, my $0.02: I'm looking forward to test this with Hy (hylang.org) and seeing if some of the module referencing is fixed. Makes for a great JITed LISP already.