I'm not sure I buy this argument. The author seems to be arguing that it's a fatal design flaw for a language not to be implemented in itself. But as near as I can tell not that many languages are implemented in themselves. I'm pretty sure Java and C# (as examples) are implemented in C.
Also, his argument that the compatibility issues between JPython and CPython are because python isn't implemented in itself also don't hold water. One very simple counterexample would be Ruby and JRuby. Ruby isn't implemented in itself, and like JPython and CPython, JRuby and MRI are two different code bases written in two different base languages. Despite this, JRuby has kept very close syntax compatibility with the standard Ruby 1.8.x interpreter. The secret in their case isn't self hosting, but rather a very comprehensive unit testing suite for the language. (interestingly enough, developed initially by yet a third implementation of Ruby).
the compatibility issues between JPython and CPython are because python isn't implemented in itself
Didn't say that. Said that self-hosting is one solution (one Perl 6 uses), not the only one.
[Ruby's secret] isn't self hosting, but rather a very comprehensive unit testing suite for the language
No, Python also has a comprehensive test suite. The difference between Jython and Ruby is that Jython isn't as well maintained as the Ruby branches. Self-hosting the syntax is an architectural solution which requires zero additional effort to implement syntax upgrades.
I think the important property the author is talking about is not self-hosting, but rather extensible syntax. But your point still stands in that very few languages have this feature.
Ruby has a very flexible syntax, but there's no particular reason to call it extensible. It's also not a great example of forward progress considering that the latest language version (1.9) has been around for years and none of the alternate implementations fully support it, not even the one that is now on the verge of release (Rubinius).
"It's also not a great example of forward progress considering that the latest language version (1.9) has been around for years and none of the alternate implementations fully support it, not even the one that is now on the verge of release (Rubinius)."
Possible reasons for this: People working on alternate implementations of Ruby started on 1.8.x. 1.9 has been evolving over time (even after its release) , and is different enough from the 1.8 version that having a single interpreter that can handle both 1.8.x and 1.9 is non-trivial. The 1.8.x version is by far the most popular, so people working on other implementations focus on them, looking to fix bugs, make them faster, etc.
I think that were there a big demand for JRuby 1.9 or IronRuby 1.9 then the appropriate teams would have it done, but right now it isn't (I imagine) a big priority.
> I'm pretty sure Java and C# (as examples) are implemented in C.
Not to argue against your point (you don't need self-hosting to get cross-implementation compatibility), Jikes RVM < http://jikesrvm.org/ > is a JVM written in Java itself.
To down-voters: this is a valid question, and something I cut from the original post (trying to keep to meta-discussion).
I'm hitting one related issue right now at work: we want to switch to Jython to take advantage some JVM features, but need to weigh the risk that it might never move past Python 2.5 syntax [1]. That's not a deal-breaker, but I'm an enormous fan of where Python syntax has gone since then, so it pains me especially. With a self-hosting syntax interpreter, even a minimally maintained project like Jython could keep up with most changes.
Another less-practical issue is that of syntax evolution: like Apple, Python has taken the route of benevolent-overlord to make design decisions. Thanks to the good taste of Steve and GvR, I think that's worked out really well. But neither is perfect, and especially in the case of programming language design, I'd wager the end product could benefit from having a market system of minor syntax features competing in user-space, not just in Guido's head. Just as "from __future__ import print_function" changes the syntax of a file, so could "from __macros__ import anaphoric_if".
I think the vital quote from Frank's notice is "I fully expect that this new job will deepen my understanding of real world Python coding and make my Jython work more productive."
With this in mind, wouldn't his move away from Jython full time actually make it better (ie infuse new ideas) so we might get 2.6 comparability sometime down the line?
I don't understand python is dead or hurting because jython isn't keeping up with the latest branches?
Edit: Only in a language group like rubyists do you see quick adoption and change and for many applications this isn't always great I'll take stable libraries over ever changing applications any day.
Re-read the post. It seems like he _used_ to think Python was dead because he thought the Moratorium was implemented for different reasons than the actual rationale.
Now, I'm not sure if he still thinks Python is dead or not, but that's not what his post is about. He was just stating his opinion that Python has a "[...] fundamental design flaw: a syntax which requires human compilation."
I did re-read the post and it just seems like perl 6 has reassured him that python is dying but I have to disagree, maybe jython is dead for what he wants but python is not dead and from the looks of it jython is not dead.
Err, I guess this teaches me to not to even mention simplistic emotional responses like "Python is dead". Let me try to be as clear is possible:
Python is not dead. Python is hurting from a bad design decision that doesn't let multiple interpreter implementations share components that they could have. This makes it impossible for minimally maintained branches (like Jython) to keep up with the language, and so hard for better-maintained branches (like IronPython or pypy) that it necessitates a moratorium on the syntax as a whole.
I can understand where you are coming from and even though IronPython and other such things are interesting but from experience python is a much slower moving language, I still interact with systems using python 2.5 and I think the moratorium is a good thing for creating libraries and implementations and overall will make adoption quicker.
I still think python's focus is in a slower world and as a python user I am okay with that.
"Python is hurting from a bad design decision that doesn't let multiple interpreter implementations share components that they could have."
As many others have said, changing syntax is really one of the most trivial parts for alternative implementations to keep up with when moving to new versions. That "hurt" hasn't been fixed because it doesn't really hurt very much...
I don't understand python is [...] hurting because jython isn't keeping up with the latest branches?
Yes, Python has put a moratorium on syntax changes in part because IronPython, Jython, and pypy aren't keeping up with changes. As felideon points out, I regret saying it's dead - it's just that syntax innovation that's dead (and only for a little bit), and that's a shame.
In Smalltalk, this is a non-issue because the language is so small. A good programmer can hand-code a parser for most of Smalltalk in under a day. I know, because I've done it a couple of times. (2 hours is my fastest time.)
To relate this back to the blog post, the original sin is that Python has too much syntax. Every time somebody wants to add a "language" feature, it has to be implemented with some kind of syntactic change, and support for the semantics of the feature has to be added to the interpreter. That's a lot of work, and it means that alternate implementations like Jython can't keep up.
A good example of this is generators. Via PEP 255 they were introduced in Python 2.2, and required the addition of the "yield" statement to the syntax. Recently Squeak (an open-source Smalltalk implementation) added generators to the standard library, with no syntax changes required. It was a much smaller project, and the change can easily be ported to other Smalltalk implementations without participation from the maintainers of the core libraries or virtual machine.
So languages with very simple syntax, like Smalltalk and Lisp just don't suffer from the high level of language evolution friction that Python does. In practice, the diversity of dialects and the ease with which the language can change creates the kind of market system of language features that andreyf mentioned here on HN.
The neat thing about Perl6 is that it takes a different approach. Perl doesn't have a simple syntax like Smalltalk or Lisp. Instead it has a very powerful system for text manipulation that can handle the complexity of Perl's syntax. In theory, that's even more powerful than Lisp macros or Smalltalk's implemented-in-Smalltalk compiler. It'll be very interesting to see how it plays out in practice.
This is interesting, but I don't know that reality matches the author's theory. After all perl 6 is hardly a shining example of how a language should be upgraded. It's been in the works for at least a decade. Say what you will about python 3, but at least it exists and has been released.
> Say what you will about python 3, but at least it exists and has been released.
Yeah, and nobody is upgrading because it doesn't give you much incentive unless you enjoy breaking backwards compatibility. It has also been in development for at least 5 years.
I don't get why people get version numbers so literally. Perl6 is a new language inspired by Perl5. Python 3 is just a minor enhancement that happens to break backwards-compatibility. The two are not comparable.
Yea Perl 6 is a new language, but Perl 1-5 were the same language just like Python 1-3 are. What should have been Perl 6 or even 7 just got released as Perl 5.12. Perl 5 isn't going anywhere, I can imagine in 10 years we'll be seeing Perl 5.27. It doesn't really make sense, if they wanted to write a new language but they wanted to use the Perl moniker for marketing purposes they should have called it Perl++
Python does have a machine readable grammar. The reason Jython isn't at 2.6 is not because of syntax changes, those would probably take less than a day to do, it's all the semantic changes that take time.
You're right, I should have addressed that: there's more to syntax than grammar.
All of the semantic changes I can think of in Python could easily be implemented as syntax transformations in a way portable across implementations. Decorators, the with statement, print-as-a-function, etc... There might be some I'm missing that would be non-trivial as macros, but I'd benefit significantly just from having the ones that are plain syntax sugar without having to convince my team to use a py3-to-2 compiler.
I suggest you take a look at the 2.6 release notes (http://docs.python.org/whatsnew/2.6.html), the number of items on here that constitute simple syntax transformations is dwarfed by the number that require active work in the VM.
Absolutely, that would requre a ton of work to upgrade. Looking through that page is precisely what convinced me the moratorium is a practical thing for the language. But most changes there aren't semantic, either. I'm talking about changes to the language, not the implementation, changes like the "with" statement, like new packages, not changes like how site-packages works.
Maybe my point was a little on the subtle side: being able to implement changes to a language's syntax and semantics without doing it again for each and every implementation is a better solution to one problem the moratorium is solving.
Is implementation really the bottleneck when it comes to language syntax changes? My observation of Python's and also Java's progression makes me think that design issues, compatibility issues or political issues have been much more difficult than actually implementing what has been decided.
The benefit of self hosting is that all libraries are instantly available on every implementation. That's important because the libraries comprise hugely more code than the language itself I would think.
I think this would be a big deal in some languages that love to change. But python's whole philosophy is to resist change as much as possible and point you to extensions.
I wouldn't bet on huge changes in python anytime soon after they release 3000. So other implementations don't really have to hit a constantly moving target and it's not a huge deal. For instance IronPython only supports 2.6 right now. But the features in 3k don't really change that much so some of those are being back-ported to 2.6 and the others can be gotten around with extensions.
I'd love to hear what the guys behind other implementations think about how python handles bootstrapping the compiler and if it's a big deal to them.
No, he's probably arguing in favor of expressions (i.e. stuff that returns values). Smalltalk has expressions everywhere, with control blocks being just closures passed to methods on objects ...
a < b
ifTrue: [^'A is less than B']
ifFalse: [^'A is greater or equal to B']
> "Purely functional" Lisp has no statements, side-effects, etc, for instance.
Incorrect: ACL2, an industry-standard automated theorem-prover, is written in a side-effect-free variant of Common Lisp. It has both an implementation and users, including users outside of academia.
Any programming language that can't be cut-n-pasted from fifty different websites to make my SQL-injection-rich AJAX social awesome NoSQL site just isn't worth knowing. Theorem proving -- you ain't gonna need it!
What's next, you're going to tell me you write unit tests? Hahahahah.
There's no reason why there should be two kinds of things code can be made up of. (statements vs expressions) In a well-designed language, everything that is a statement in Python would just be implemented as an expression that returns nil.
"band-aids like the moratorium become necessary solutions, alleviating the symptoms of a more fundamental design flaw: syntax and semantics which require human implementation instead being portable from one implementation to another."
Would it be easier to for alternative implementations (non-CPython ala Jython, IronPython, etc) to keep up if the syntax was portable between them? Yes, of course. Is this a "Design Flaw"? Not in any way.
Portable syntax between implementations is not a primary goal of Python (the language) so I don't see any merits for criticizing it for not taking a feature never previously desired (most of the alternative runtimes are relatively young) into account in the underlying design.
If you need to alter/extend the syntax to python at some point I'd advising looking at the python magazine article on implementing DSL's w/pyparsing noted here:
http://pyparsing.wikispaces.com/Publications
If are in frequent need to alter/extend the syntax of the language you are using, and it is a non-trivial task, then you are using the wrong language.
I don't think it's very credible to hold Perl 6 up as the standard for how to encourage and enable complete implementations, considering that ten years in, it still has only partial implementations.
I like the fact that the language syntax evolves at a slower pace than the library space. I think that's the right proportion. If you want to add new functionality to the Python ecosystem you don't need to add new syntax, instead it is often very easy to just go write a new function, class or module and make it available to the rest of the world. Nothing is stopping anyone from doing that today. Also nothing is stopping you from using a mix of languages, with some parts of a system written in Python, some in C, some in Lisp, some in Java, etc. Mix and match and pick the best tool for the role.
54 comments
[ 3.9 ms ] story [ 109 ms ] threadAlso, his argument that the compatibility issues between JPython and CPython are because python isn't implemented in itself also don't hold water. One very simple counterexample would be Ruby and JRuby. Ruby isn't implemented in itself, and like JPython and CPython, JRuby and MRI are two different code bases written in two different base languages. Despite this, JRuby has kept very close syntax compatibility with the standard Ruby 1.8.x interpreter. The secret in their case isn't self hosting, but rather a very comprehensive unit testing suite for the language. (interestingly enough, developed initially by yet a third implementation of Ruby).
Didn't say that. Said that self-hosting is one solution (one Perl 6 uses), not the only one.
[Ruby's secret] isn't self hosting, but rather a very comprehensive unit testing suite for the language
No, Python also has a comprehensive test suite. The difference between Jython and Ruby is that Jython isn't as well maintained as the Ruby branches. Self-hosting the syntax is an architectural solution which requires zero additional effort to implement syntax upgrades.
Ruby has a very flexible syntax, but there's no particular reason to call it extensible. It's also not a great example of forward progress considering that the latest language version (1.9) has been around for years and none of the alternate implementations fully support it, not even the one that is now on the verge of release (Rubinius).
Possible reasons for this: People working on alternate implementations of Ruby started on 1.8.x. 1.9 has been evolving over time (even after its release) , and is different enough from the 1.8 version that having a single interpreter that can handle both 1.8.x and 1.9 is non-trivial. The 1.8.x version is by far the most popular, so people working on other implementations focus on them, looking to fix bugs, make them faster, etc.
I think that were there a big demand for JRuby 1.9 or IronRuby 1.9 then the appropriate teams would have it done, but right now it isn't (I imagine) a big priority.
Not to argue against your point (you don't need self-hosting to get cross-implementation compatibility), Jikes RVM < http://jikesrvm.org/ > is a JVM written in Java itself.
I'm hitting one related issue right now at work: we want to switch to Jython to take advantage some JVM features, but need to weigh the risk that it might never move past Python 2.5 syntax [1]. That's not a deal-breaker, but I'm an enormous fan of where Python syntax has gone since then, so it pains me especially. With a self-hosting syntax interpreter, even a minimally maintained project like Jython could keep up with most changes.
Another less-practical issue is that of syntax evolution: like Apple, Python has taken the route of benevolent-overlord to make design decisions. Thanks to the good taste of Steve and GvR, I think that's worked out really well. But neither is perfect, and especially in the case of programming language design, I'd wager the end product could benefit from having a market system of minor syntax features competing in user-space, not just in Guido's head. Just as "from __future__ import print_function" changes the syntax of a file, so could "from __macros__ import anaphoric_if".
1. The Jython project lead is no longer working on it full-time: http://fwierzbicki.blogspot.com/2010/02/my-new-job-at-sauce-...
With this in mind, wouldn't his move away from Jython full time actually make it better (ie infuse new ideas) so we might get 2.6 comparability sometime down the line?
Edit: Only in a language group like rubyists do you see quick adoption and change and for many applications this isn't always great I'll take stable libraries over ever changing applications any day.
Now, I'm not sure if he still thinks Python is dead or not, but that's not what his post is about. He was just stating his opinion that Python has a "[...] fundamental design flaw: a syntax which requires human compilation."
Python is not dead. Python is hurting from a bad design decision that doesn't let multiple interpreter implementations share components that they could have. This makes it impossible for minimally maintained branches (like Jython) to keep up with the language, and so hard for better-maintained branches (like IronPython or pypy) that it necessitates a moratorium on the syntax as a whole.
I still think python's focus is in a slower world and as a python user I am okay with that.
As many others have said, changing syntax is really one of the most trivial parts for alternative implementations to keep up with when moving to new versions. That "hurt" hasn't been fixed because it doesn't really hurt very much...
Yes, Python has put a moratorium on syntax changes in part because IronPython, Jython, and pypy aren't keeping up with changes. As felideon points out, I regret saying it's dead - it's just that syntax innovation that's dead (and only for a little bit), and that's a shame.
A good example of this is generators. Via PEP 255 they were introduced in Python 2.2, and required the addition of the "yield" statement to the syntax. Recently Squeak (an open-source Smalltalk implementation) added generators to the standard library, with no syntax changes required. It was a much smaller project, and the change can easily be ported to other Smalltalk implementations without participation from the maintainers of the core libraries or virtual machine.
So languages with very simple syntax, like Smalltalk and Lisp just don't suffer from the high level of language evolution friction that Python does. In practice, the diversity of dialects and the ease with which the language can change creates the kind of market system of language features that andreyf mentioned here on HN.
The neat thing about Perl6 is that it takes a different approach. Perl doesn't have a simple syntax like Smalltalk or Lisp. Instead it has a very powerful system for text manipulation that can handle the complexity of Perl's syntax. In theory, that's even more powerful than Lisp macros or Smalltalk's implemented-in-Smalltalk compiler. It'll be very interesting to see how it plays out in practice.
Yeah, and nobody is upgrading because it doesn't give you much incentive unless you enjoy breaking backwards compatibility. It has also been in development for at least 5 years.
I don't get why people get version numbers so literally. Perl6 is a new language inspired by Perl5. Python 3 is just a minor enhancement that happens to break backwards-compatibility. The two are not comparable.
Hrm, I think it is only even version numbers, maybe 5.28. :-)
(And check the Modern Perl movement and the backports of Perl 6 stuff like Moose. Arguably, Perl 5 should change name, too...)
The libraries becoming available is going to take a little while... work is being done to make pygtk available.
Numpy is a big library widely used - when this arrives then others will follow (I'd give it two years or so).
All of the semantic changes I can think of in Python could easily be implemented as syntax transformations in a way portable across implementations. Decorators, the with statement, print-as-a-function, etc... There might be some I'm missing that would be non-trivial as macros, but I'd benefit significantly just from having the ones that are plain syntax sugar without having to convince my team to use a py3-to-2 compiler.
Maybe my point was a little on the subtle side: being able to implement changes to a language's syntax and semantics without doing it again for each and every implementation is a better solution to one problem the moratorium is solving.
The benefit of self hosting is that all libraries are instantly available on every implementation. That's important because the libraries comprise hugely more code than the language itself I would think.
I wouldn't bet on huge changes in python anytime soon after they release 3000. So other implementations don't really have to hit a constantly moving target and it's not a huge deal. For instance IronPython only supports 2.6 right now. But the features in 3k don't really change that much so some of those are being back-ported to 2.6 and the others can be gotten around with extensions.
I'd love to hear what the guys behind other implementations think about how python handles bootstrapping the compiler and if it's a big deal to them.
Python's sin is that it has statements at all.
Such a Lisp also has no users.
http://en.wikipedia.org/wiki/ACL2
Hands up, who has heard of ACL2 before?
In fact, I am appalled so few people, apparently, heard of it.
What's next, you're going to tell me you write unit tests? Hahahahah.
I was under the impression that def x(a):... was only a pretty way to say x = function(a)...
I have to agree print is a weird thing, but what other statements bother you so much?
Would it be easier to for alternative implementations (non-CPython ala Jython, IronPython, etc) to keep up if the syntax was portable between them? Yes, of course. Is this a "Design Flaw"? Not in any way.
Portable syntax between implementations is not a primary goal of Python (the language) so I don't see any merits for criticizing it for not taking a feature never previously desired (most of the alternative runtimes are relatively young) into account in the underlying design.
If you need to alter/extend the syntax to python at some point I'd advising looking at the python magazine article on implementing DSL's w/pyparsing noted here: http://pyparsing.wikispaces.com/Publications
If are in frequent need to alter/extend the syntax of the language you are using, and it is a non-trivial task, then you are using the wrong language.
PyPy is taking precisely this approach, and GvR in an interview or two, has blessed it as the way forward.
So I think OP is a little late to the game.