Interesting idea. Given that you can use arbitrary Python expressions as functions, but that could pollute the functional purity, what stylistic restrictions to you expect users to self-impose?
About style, why perpetrate those double underscored __atrocities__? I'm sure there is a better design lurking around in other languages waiting to be copied. I also don't appreciate _variables.
But kudos for adding tail recursion to a pythonic language. At last.
If it's just for comment, that's some impressive work. The compiler is fairly short and is a good demo of how you can do this sort of thing surprisingly quickly if you know what you are doing.
If you intend this as something a lot of people will use, then I'd express very serious concerns about building this on top of Python. Python is one of the slowest languages there is [1]. Putting a mismatching paradigm on top of it is going to cost you some sort of multiplicative slowdown, which could very easily be in the 5x range and could even be in the 20x-50x range. Expect another penalty in memory consumption for what is again already a bit of a memory-using language. It's not long before you've got a language on your hands that is 200-500x slower than C, and perhaps more to the point, 100x slower than Haskell itself. This will make trying to convince anyone else to use it an uphill battle, because while performance isn't everything (Python is perfectly suitable for many uses, of course), insufficient performance is a critical blocker for using it for some project, and performance this bad will be "insufficient performance" for a lot of projects.
Let me be clear that I am NOT pointing this out as a "criticism" of the project. If you are doing this to learn or share some fun stuff, this is irrelevant to you, and let me re-emphasize that you're off to what looks like a good start! I mean this in the spirit of warning you about what may be your biggest problem, depending on your unspecified goals, so you can make good engineering decisions about what to look out for and what to tackle.
One possible solution you could consider is rather than targeting Python AST, which I think your code is doing?, you might consider targeting the bytecode directly. While you're still going to experience a certain amount of paradigm mismatch, you might find this cuts perhaps a 2x out of the multiplicative slowdown stack, gives you more options that Python directly does, and especially in the case of PyPy, may give you the option of examining what bytecode constructs PyPy is better able to optimize and give you some wiggle room in what you emit that you can exploit.
Even if this is just a learning project you may also find you learn useful things by doing this; "emitting bytecode" is a popular option nowadays, with things like LLVM. From emitting bytecode for Python you may even find it not to be too big a leap to emit native code with LLVM, too. A language that automatically works natively when code is implemented in pure lambdascript but transparently interacts with Python would start turning some heads. It's not hard to imagine potentially getting some speedups over Python. Though that's a tougher bar to clear; you've got a typesystem to think about at that point. The code is currently so small and agile it's a good time to start looking at these issues.
[1]: I no longer believe "languages don't have speeds, only implementations do". After many years of trying to speed up the many dynamic scripting languages and generally hitting a wall around 10x-20x slower than C across the board, I now consider the burden of proof that languages do not at least set lower bounds on performance to be on someone who can produce a truly fast-as-C dynamic language interpreter/compiler. (Obligatory LuaJIT callout as the closest to date.)
Presumably that instead of writing a language that targets a slow language and pays a multiplicative slowdown on top of that, you could write something that targets a relatively fast language that Python is able to run in also, and pay the multiplicative price against that instead. Go is not the fastest language and I expect it to have roughly a 2x slowdown over C indefinitely, but that still puts it in the "compiled language" camp rather than the "interpreted language" camp, which tends to run 10x-40x slowdown.
I may have misinterpreted your comment, but I think that was my point. He could have built Lambdascript directly on top of Go or some other marginally fast language instead of building on Python.
Almost everything is recompiled at least twice just to run it. Your comment is pretty ridiculous unless you had another point than the one I'm getting. What do you think executes on your CPU when you run CPython.
I personally see nothing wrong with writing a prototype in a "slow" language, especially if speed ain't a priority (yet). Sometimes you just have to get an initial implementation written in some language, and if that language happens to be some interpreted language from the 80's, then so be it.
Now, sticking to that language for a final version would be more concerning, but even that ain't really a dealbreaker; lots of "slow" languages are perfectly capable of interfacing with "fast" languages through some kind of FFI (or, at the very least/worst, IPC over some kind of pipe or socket or what have you). Gradually writing faster implementations as you run into bottlenecks is a perfectly common and reasonable development approach.
"I personally see nothing wrong with writing a prototype in a "slow" language, especially if speed ain't a priority (yet). "
Neither do I. That's why I conditionalized that point on the intent of the poster, who as of this writing has not replied to any of these comments.
I wish people would be more clear about their intentions. I have no problem with any of them, ranging from idly showing off something they did for discussion to "I intend professionals to drop everything and use this exclusively", but the feedback appropriate to the various cases along that range are very different. I don't want to blast a student for not creating something that Google can base their next 10 years on, but I don't want to provide someone trying to win over professionals with feedback about their capitalization of identifiers.
Does anybody know a widely used and open source project using literate programming? (Edit: please include only programs expressed "as a web of ideas" that Knuth mentions - apparently otherwise it's not literate programming, just loads of inline documentation?)
From what I've seen, literate programming is great in theory and worse than useless in practice. It's so far away from reality. I have had the misfortune of working on a FunnelWeb codebase, and it was one of the most incomprehensible codebases I've ever seen.
Preprocessing the code is an awful step in any language. Somehow, the "not documentation" still goes out of date. There are several other problems, explained at [0] better than I can in a comment.
Specifically about this project: While I enjoy writing blogs in markdown because of its constraints, reStructuredText (reST) is more powerful/expressive (by design). Because you can easily express semantics/relationships, the resulting links in the generated documentation make the "web" of source code easier to navigate - something Knuth was trying to do with literate programming.
Literate programming, to me, feels like commenting on steroids. But that's not necessarily a good thing. A big philosophy right now, and one that I share, is that your goal should be to make code that can be easily understood without comments through expressive code.
Literate programming goes in the opposite direction, and I don't think that's necessarily good, though it has it's use cases. I think data science with Jupyter notebooks are a great display of when it's good (and that's likely the best / most used form of literate programming out there), because in data science a lot of what you need to justify is the "why" a calculation is done the way it's done. Pretty graphs inline are just a bonus.
For ordinary code though? Comments are a fallback for complicated bits, but I feel like they shouldn't be the norm. I want people to write code that doesn't need hand holding just to understand.
I once build an acceptance testing suite using literate programming. I had been handed a nice write-up by our resident physicist with all the maths for how things should work and I didn't want to lose that, so we cobbled something together that used latex math, had embedded test code, etc. These documents then served both as spec and as tests. IMO, this was a terrific use for literate programming. But we did not adopt literate programming as an overall approach, pretty much for the reasons you mention.
If your end goal is to produce some kind of largely static artifact then literate programming is fine. For everyday code where you iterate and refactor all the time, literate techniques have all the downsides of prolific comments.
connections refused on https. http works. read through http://witheve.com/deepdives/literate.html. instead of fencing comments, code is fenced. they even admit they don't have a macro language, so isn't this just inline markdown documentation? it would seem that without macros, you don't get the webs of macros, or whatever knuth thought differentiated literal programming from documentation.
I've used noweb for a few projects at my work, and my PhD. Even without actually writing comments, the structure of the code has enabled me to return to a project six months after giving up on it and pick it up again really quickly. For wahtever reason, I don't recall that happening with any of my non-literate projects.
I've found it really useful for a big Django project, when I add some functionality, the definition of the models, views, dependencies and URL references are all adjacent in a nicely indicated section and subsections. Navigating around it in a PDF reader has been more pleasent than scrolling up and down code, and hunting for the appropriate file, as I would often find myself doing in Eclipse.
I'm also able to put output of the tests into the doc, and a full set of installation instructions from a fresh machine, and usage instructions, in the same doc with pointers to the appropriate code chunk.
Much of this can be achieved in various other ways. But the actual result is that I've been much happier in my last few literate projects, than I was in the ones before that.
Finally, I've also noticed that I don't have the encroaching feeling of my current project getting a bit big and unwieldy even though this is easily the most complicated thing I've ever done.
I think Mathematica/IPython aka. Juypter notebooks are great in this respect. I used them for my BSc and MSc. However, they aren't complex programs, and portability is an issue.
But according to Knuth, literate programming is about more than online code, it also includes the whole web-o'-macros thing.
> I'm also able to put output of the tests into the doc, and a full set of installation instructions from a fresh machine, and usage instructions, in the same doc with pointers to the appropriate code chunk.
Python has ways of doing this, and all of this just feels like overloading a document. Repos exist for this, and work fine keeping everything together.
Corrode, a C to Rust translator. I wouldn't say "widely used", but within the Rust community it's well known. Literate style seems to work well for code that's difficult to approach (how would you start building this?), and doesn't need frequent refactoring.
I'd say that's a good example. It doesn't use macros, however I do like this stackoverflow discussion [0] on if Literal Haskell even needs macros, because it's already a flexible language. Definitely a new perspective.
"Does anybody know a widely used and open source project using literate programming? (Edit: please include only programs expressed "as a web of ideas" that Knuth mentions - apparently otherwise it's not literate programming, just loads of inline documentation?)"
Funny you ask this, I ran across a @raganwald post mentioning a great example of ^literate programming^ (mixing code and documentation), "Literate CoffeeScript" [0]. You can read about how CoffeeScript does this at http://coffeescript.org
As a Python programmer, the appeal of the Haskell family seems to be correctness -- I've come to think of Python as a "surprisingly typed" language. This particular language looks like it has Haskelly syntax without the type system. I'd be most interested in the latter; I'd be willing to trade off performance in some places for stronger ability to reason about types. I envision a pure, strongly typed language with strict evaluation, that looks like Python to the rest of my code base.
I haven't yet -- type annotations in Python 3 still don't seem to have stabilized. Although I admit I haven't looked too deeply into it, I know a bunch of new stuff landed in 3.6. The appeal of a library that adds a functional layer on top of Python is that it partitions "new stuff" from "old stuff". It seems easier to tear out and reimplement if we screw it up, versus type annotations diffusing out into the rest of our codebase.
FWIW I tried it on a relatively new 10K line project and gave up, because it doesn't play well with metaprogramming. I consider metaprogramming to be part of Python, so mypy and statically typed Python are really different languages with less power IMO. And of course the mypy author knew that -- that's why he gave it a different name.
I'm not claiming my experience generalizes... if you have a lot of Java-like code you can probably use mypy. Bizarrely, some people do actually have a lot of Java-like Python code.
I've used typehints to write a compiler but with my own small homegrown annotations and it worked really well. It feels natural coming from languages with similar syntax like rust and the like. Cool integrates well with pycharm too
The other strength of Python (as I'm sure you're aware) is its ecosystem. It's not clear to me to what extent imports work with Lambdascript but access to them (including the scientific stack) seems to be an explicit goal of, say, Hylang.
I used Python for 6 years, but whenever I go back, I miss the expressiveness and safety offered by other languages. I frequently "forced" into using Python when libraries in languages I prefer (Scala, Haskell) just aren't available, but I can't justify reimplementing them.
Since at least some of the backend programmers are language designers, the vm code might be equally nice to play with, seperate from runtime performance issues.
A tail-recursive ("looped") process without an explicit termination is a typical server, terminated by killing the process (sorry, the impure outside world is sometimes cruel).
E.g. the best parts of OOP were the ideas of data encapsulation, of modularity, of interfaces separate from implementations. These live on, whether you use method dispatch and inheritance or not.
My experience is that it's a joy to use and its lambda and pipe syntax cleans up my code nicely. Coconut's TCO is handy though pattern-matching is the killer feature for me. It compiles to a Python 2,3,and PyPy-compatible code. The language puts a premium on interop with Python, so much so that Python code IS Coconut code. There's a vim plugin that yields syntax highlighting but no IDE plugins yet that I'm aware of yet.
51 comments
[ 2.3 ms ] story [ 110 ms ] threadBut kudos for adding tail recursion to a pythonic language. At last.
If it's just for comment, that's some impressive work. The compiler is fairly short and is a good demo of how you can do this sort of thing surprisingly quickly if you know what you are doing.
If you intend this as something a lot of people will use, then I'd express very serious concerns about building this on top of Python. Python is one of the slowest languages there is [1]. Putting a mismatching paradigm on top of it is going to cost you some sort of multiplicative slowdown, which could very easily be in the 5x range and could even be in the 20x-50x range. Expect another penalty in memory consumption for what is again already a bit of a memory-using language. It's not long before you've got a language on your hands that is 200-500x slower than C, and perhaps more to the point, 100x slower than Haskell itself. This will make trying to convince anyone else to use it an uphill battle, because while performance isn't everything (Python is perfectly suitable for many uses, of course), insufficient performance is a critical blocker for using it for some project, and performance this bad will be "insufficient performance" for a lot of projects.
Let me be clear that I am NOT pointing this out as a "criticism" of the project. If you are doing this to learn or share some fun stuff, this is irrelevant to you, and let me re-emphasize that you're off to what looks like a good start! I mean this in the spirit of warning you about what may be your biggest problem, depending on your unspecified goals, so you can make good engineering decisions about what to look out for and what to tackle.
One possible solution you could consider is rather than targeting Python AST, which I think your code is doing?, you might consider targeting the bytecode directly. While you're still going to experience a certain amount of paradigm mismatch, you might find this cuts perhaps a 2x out of the multiplicative slowdown stack, gives you more options that Python directly does, and especially in the case of PyPy, may give you the option of examining what bytecode constructs PyPy is better able to optimize and give you some wiggle room in what you emit that you can exploit.
Even if this is just a learning project you may also find you learn useful things by doing this; "emitting bytecode" is a popular option nowadays, with things like LLVM. From emitting bytecode for Python you may even find it not to be too big a leap to emit native code with LLVM, too. A language that automatically works natively when code is implemented in pure lambdascript but transparently interacts with Python would start turning some heads. It's not hard to imagine potentially getting some speedups over Python. Though that's a tougher bar to clear; you've got a typesystem to think about at that point. The code is currently so small and agile it's a good time to start looking at these issues.
[1]: I no longer believe "languages don't have speeds, only implementations do". After many years of trying to speed up the many dynamic scripting languages and generally hitting a wall around 10x-20x slower than C across the board, I now consider the burden of proof that languages do not at least set lower bounds on performance to be on someone who can produce a truly fast-as-C dynamic language interpreter/compiler. (Obligatory LuaJIT callout as the closest to date.)
It seems the submitter is also the author of that project, so I guess this is just a forgotten "Show HN:" prefix in the title.
Now, sticking to that language for a final version would be more concerning, but even that ain't really a dealbreaker; lots of "slow" languages are perfectly capable of interfacing with "fast" languages through some kind of FFI (or, at the very least/worst, IPC over some kind of pipe or socket or what have you). Gradually writing faster implementations as you run into bottlenecks is a perfectly common and reasonable development approach.
Neither do I. That's why I conditionalized that point on the intent of the poster, who as of this writing has not replied to any of these comments.
I wish people would be more clear about their intentions. I have no problem with any of them, ranging from idly showing off something they did for discussion to "I intend professionals to drop everything and use this exclusively", but the feedback appropriate to the various cases along that range are very different. I don't want to blast a student for not creating something that Google can base their next 10 years on, but I don't want to provide someone trying to win over professionals with feedback about their capitalization of identifiers.
From what I've seen, literate programming is great in theory and worse than useless in practice. It's so far away from reality. I have had the misfortune of working on a FunnelWeb codebase, and it was one of the most incomprehensible codebases I've ever seen.
Preprocessing the code is an awful step in any language. Somehow, the "not documentation" still goes out of date. There are several other problems, explained at [0] better than I can in a comment.
Specifically about this project: While I enjoy writing blogs in markdown because of its constraints, reStructuredText (reST) is more powerful/expressive (by design). Because you can easily express semantics/relationships, the resulting links in the generated documentation make the "web" of source code easier to navigate - something Knuth was trying to do with literate programming.
[0] http://akkartik.name/post/literate-programming
Literate programming goes in the opposite direction, and I don't think that's necessarily good, though it has it's use cases. I think data science with Jupyter notebooks are a great display of when it's good (and that's likely the best / most used form of literate programming out there), because in data science a lot of what you need to justify is the "why" a calculation is done the way it's done. Pretty graphs inline are just a bonus.
For ordinary code though? Comments are a fallback for complicated bits, but I feel like they shouldn't be the norm. I want people to write code that doesn't need hand holding just to understand.
If your end goal is to produce some kind of largely static artifact then literate programming is fine. For everyday code where you iterate and refactor all the time, literate techniques have all the downsides of prolific comments.
I've found it really useful for a big Django project, when I add some functionality, the definition of the models, views, dependencies and URL references are all adjacent in a nicely indicated section and subsections. Navigating around it in a PDF reader has been more pleasent than scrolling up and down code, and hunting for the appropriate file, as I would often find myself doing in Eclipse.
I'm also able to put output of the tests into the doc, and a full set of installation instructions from a fresh machine, and usage instructions, in the same doc with pointers to the appropriate code chunk.
Much of this can be achieved in various other ways. But the actual result is that I've been much happier in my last few literate projects, than I was in the ones before that.
Finally, I've also noticed that I don't have the encroaching feeling of my current project getting a bit big and unwieldy even though this is easily the most complicated thing I've ever done.
But according to Knuth, literate programming is about more than online code, it also includes the whole web-o'-macros thing.
> I'm also able to put output of the tests into the doc, and a full set of installation instructions from a fresh machine, and usage instructions, in the same doc with pointers to the appropriate code chunk.
Python has ways of doing this, and all of this just feels like overloading a document. Repos exist for this, and work fine keeping everything together.
https://github.com/jameysharp/corrode/blob/master/src/Langua...
[0] https://stackoverflow.com/questions/16869050/is-literate-pro...
Funny you ask this, I ran across a @raganwald post mentioning a great example of ^literate programming^ (mixing code and documentation), "Literate CoffeeScript" [0]. You can read about how CoffeeScript does this at http://coffeescript.org
[0] blog post: http://raganwald.com/2013/03/27/literate-coffeescript.coffee...
[1] raw text: https://raw.githubusercontent.com/raganwald/raganwald.github...
Many of these Python built languages seem to flip it around and build on the weaknesses of Python and not build on Python's strengths.
http://www.oilshell.org/blog/2016/11/30.html
update here: http://www.oilshell.org/blog/2017/01/06.html
I'm not claiming my experience generalizes... if you have a lot of Java-like code you can probably use mypy. Bizarrely, some people do actually have a lot of Java-like Python code.
- Lambdascript runs on Python3.x
- Grumpy runs on Python2.7
I can only hope I guess ...
I think there are some much better ideas surfacing soon.
E.g. the best parts of OOP were the ideas of data encapsulation, of modularity, of interfaces separate from implementations. These live on, whether you use method dispatch and inheritance or not.