I think JS and PHP have a special status here and don’t count as an answer to what GP was asking. Many people hate JS and/or PHP, often for legacy reasons.
But it's not only JS, I used it as an example because these people are doing websites, yet don't even want to learn JS. They also hate to write SQL and are stuck with ORMs...
Not even at a language level. There are Java developers who only want to work with particular frameworks (Spring) and don't know anything else or want to learn anything else.
I know right around seven languages I would be comfortable claiming I knew on a job application. I am probably more opinionated about language features, but less picky about the specific language for it.
The article is not suggesting to use Python under all circumstances.
Here's the conclusion, which summarizes the essay:
> All of this is to say while some companies do extract massive value from squeezing every CPU cycle out of their code, those companies also typically build data centres. So if you don't need a dedicated building to host your machines, please consider doing the math to see if it's truly worth making your developers less productive in the name of computational efficiency when you don't even know if that perceived efficiency is even necessary.
As an example, the kernel of my software is only a few hundred lines long. It's in C with AVX2 intrinsics to eek out the last bit of performance. The Python overhead is <1%, which means that even swapping out Python with a 1,000x faster language won't be noticeable. This is the '"glue" code' mentioned in the section on 'Use language bindings'.
Had I started with "It needs to be fast therefore I can't use Python", then it probably would have been a lot more work to do. (See, "numerous anecdotes of where someone implemented something in Python in 1/3 the time a competing team did creating the same thing in e.g. C++ or Java".)
> The article is not suggesting to use Python under all circumstances.
I would say the article is very much doing that, it's just stopping short from spelling it out clearly.
It starts from the flawed premises that Python is much more effective to come up with actual production code than other languages [1].
The author then moves on to telling you that you should try at least a couple of times to code something in python before obvious performance shortcomings are glaring, then keep python as a binding language.
At no point does the author acknowledge that there may be other reasons to pass over python than performance.
From the way he approaches the subject, the author seems to live in the paradise where you only have to sling half-baked prototypes to prod, and then move on to another project while poor souls handle maintenance.
The take on why performance matters [2] also completely overlooks the fact that optimizing for infra costs is likely the least common reason to write for performance. Sometimes, if you don't reach a given threshold, your code simply doesn't work. Sometimes, your performance is your users' time (and sometimes, these users are even developers). Such companies don't necessarily build datacenters.
[1]: "This is what leads to numerous anecdotes of where someone implemented something in Python in 1/3 the time a competing team did creating the same thing in e.g. C++ or Java." -- of course, it's just anecdotes and not a claim, so the author won't defend it, but you're still expected to believe it.
[2]: "All of this is to say while some companies do extract massive value from squeezing every CPU cycle out of their code, those companies also typically build data centres."
I agree with those who say "software engineering" is not "engineering." We as a field are more comfortable with our beliefs, supported by anecdotal evidence, than with empirical results.
That said, the few and limited empirical results which do exist, like Prechelt's empirical work from 20 years ago, support the author's premise.
As another example, Boehm's work across a range of projects (<100KLoc) shows that code size has the biggest impact on development effort, and slightly non-linear scaling in KLoC (KLoC^~1.1 in COCOMO); and various publications show Python generally requires fewer lines of code.
(As one example picked semi-arbitrarily from a Google Scholar search, http://jucs.org/jucs_19_3/a_comparison_of_five/jucs_19_03_04... on a small project has Python at the smallest effective LOC, smallest effective byte size, and nearly the smallest (after Java) eByte/eLOC size.)
Modern C++ is a much more compact (and complicated!) language than 20 years ago, so of course all of these previous results must be reviewed and reconsidered.
As a field, we have not provided that funding for solid empirical research!
> At no point does the author acknowledge that there may be other reasons to pass over python than performance.
Shrug. Okay. And from that you conclude the author is saying that Python should be used in all cases? "I'm going to talk about why you should eat more vegetables" doesn't mean "you should only ever eat vegetables." Similarly, "I'm going to talk about why you shouldn't reject Python from the get-go for a project you think is performance intensive" doesn't mean "you should only ever use Python."
The author even talks about Rust in an earlier blog entry, from August of this year, at https://snarky.ca/introducing-the-python-launcher-for-unix/ saying "And so over 3 years ago I set out to re-implement the Python Launcher for Unix in Rust. On July 24, 2021, I launched 1.0.0 of the Python Launcher for Unix".
So the evidence is that Cannon does not believe that Python is the best language for all projects.
> Sometimes, your performance is your users' time
Cannon's essay is about the case where people select a programming language based strongly on expected computational cost, and forget other factors in the cost model. Here's the relevant quote:
] I think the jump to selecting a programming language based on potential performance needs often comes from a place where people think their computation costs are more important to optimize for than their developer time. I don't think that always holds, though, as software developers are expensive.
In your model, people are choosing a language based in part of the cost of users' time, which means they they are not primarily focused on computation costs, and so are outside the scope of this essay. That doesn't contradict the essay, only show that it's incomplete. Which we know already from the text itself.
> Sometimes, if you don't reach a given threshold, your code simply doesn't work.
And sometimes you can reach a given threshold with an off-the-shelf use of Pandas, rather than build your own analysis routine. If you start off thinking "Python must be slow" and don't even know what your thresholds or performance characteristics are, then selecting (say) assembly over Python is probably premature optimization.
Cannon didn't even argue that if you aren't building data centers then you must use Python. Your [2] trims the full paragraph I quoted earlier. You left out: a) the "please consider", which makes this a request, and not a rejecti...
I took python more as a motivating example than an absolute. As far as I can tell the article suggests developer speed over application performance. Then FFI to a "fast" language to overcome your performance problems. That way you only pay the overhead of a high performance, slow to develop language where it actually matters. Which is usually much less than the whole program, so not paying for performance where it isn't needed is a big win.
It'd be great if we could start measuring how much $ it costs to run our services vs how much developer salary $ it costs to maintain and build them.
If Rust or something really saves money because it means we don't need 50 extra web workers for the same TPS then I don't think it's "premature optimization" - unless the dev salary is too much!
As it stands we're all just saying stuff with no data or evidence to back it up.
It's not going to change any time soon, IMHO. Programmers have too big of a culture of individuality, blogging, and intellectual daydreaming to actually try to come to consensus on terms of art. On top of which businesses are loathe to share data because it's tied very closely to their competitive advantage. Programming culture is more about memetic and shared ideas and uses tradition to select winning strategies. Until this changes, it'll continue to be discordant voices talking over each other. Ideology [1] is a great talk about this.
> Programming culture is more about memetic and shared ideas and uses tradition to select winning strategies.
Sounds exactly like business management culture. Makes sense though, developer productivity is first and foremost a business management issue so discussions about it ought to follow the same path as business management. And business management love their anecdotes, great person citations and jargon.
With silicon valley salaries maybe the trade-off is more between Developer time vs. Devops time. Those 50 extra web workers might not matter much on their own, but someone has to set them up, orchestrate them and deal with all weird interactions that arise.
And of course the further you get from silicon valley the more the hardware/service costs matter. A developer in Sofia easily earns an order of magnitude less than one in San Francisco, which shifts the whole optimization tradeoff a lot.
Silicon valley companies tend to have more users and therefore more servers, so saying they mostly don't need to care about server costs should be wrong. I worked at a small team at Google that had many thousands of servers, those servers cost way more to run than the engineers working on them. Compare that to a small company I worked at where one server with an extra as fallback was more than enough, there performance didn't matter at all even though salaries were much smaller.
My team does a ton of large-scale processing (most of which is ETL), and we have TONS of servers at our disposal. We effectively throw hardware at our compute, and even then, jobs can take 5-10 hours to run. No big deal - just run the job before you call it a day and it's done when you start tomorrow morning. Or when you have daily jobs, you schedule it and it's just done whenever it's done.
I can't tell you the number of times I've looked at a job someone wrote, made a 5 LOC change in 2 minutes, and cut processing from 8 hours to 5 hours. I literally had one that went from 5 hours to 5 minutes.
These extremely simple investments save a lot of money, but it's a problem that's spread across hundreds of different jobs, and it takes so much effort to convince people that we should invest in it. A daily job that costs $1500/mo to run (based on amortized hardware costs) needs only be optimized once, and those savings are yours forever. But I've had the conversation that is effectively: my time is more costly than a measly $1500 job, so it's okay if the job is expensive.
Very true. People discriminate against dynamic languages with usually two measurements: runtime errors and performance.
But we often overlook the fact that it's possible to scale those two, incrementally. With today's convenience in interop, we can always start with a prototype friendly technology and later switch tech for real bottlenecks.
Yeah, I remember working on one of the Project Euler problems, getting the right answer, then seeing someone else had solved it with a pocket calculator in less time than I took to write the program.
It was a rather embarrassing realization for me that the Fibonacci sequence has a closed-form expression and can be calculated without an algorithm. I had always seen it done in pedagogical fashion as an exercise in explaining recursion and memoization and just assumed it had to be done that way. The downside of using toy/straw problems...
I had that with the geohash algorithm I implemented naively as binary search until I came across https://mmcloughlin.com/posts/geohash-assembly that ultimately doesn't even compute it but read it from the in-memory floating point representation.
This blog post assumes Python is more productive than other languages. People like to make this claim about not only Python, but many other dynamic programming languages, specially Ruby, PHP and Lisp... but there's very little evidence to support that... programmers tend to be most productive in whatever language they know best. If they know both Python and Java equally well, I would bet they would be almost exactly as productive in either.
Granted, people come from different backgrounds, and "the right tool for the right job" always applies. But it's undeniable that generally speaking, ie. not considering any specific cases or requirements, some languages are definitely more productive than others.
I doubt it, though if anyone has data I’d love to see it. At the moment I’m quite comfortable in both javascript and rust. But I find javascript (/ typescript) noticeably more productive for small to medium projects. I can get more done with less work. For a variety of reasons it seems to take more work to write good rust code than good javascript code.
This isn’t a knock on rust - I just think rust trades off programmer productivity for correctness and performance. And it shows, on both sides.
> I doubt it, though if anyone has data I’d love to see it.
I recall a study being quoted in my university courses, which described the amount of code needed to get certain things done between different languages, which showed that Python, Ruby and others are on the less verbose side, the reasoning being that on average they'd also be more productive.
This seems to coincide with my personal experience, where JavaScript with React was much easier to work with in smaller projects, whereas using TypeScript with React lead to much slower development because of all the typing that needed to be handled, especially in cases of union types. Now, one can say that it's worth the effort to be more confident in your code doing what you want it to both now and after X months, much like you could sometimes prefer the type systems of Java or .NET over Python, Ruby or PHP, but in my eyes those tradeoffs always come with slower development velocity.
The sad thing, however, is that DuckDuckGo (and possibly other search engines) failed to return the original study or anything like it, only resulting in low quality blog content:
Anyone have any better search queries for this? Any idea why the search result quality is generally so low? Any ideas which study it might have been referencing?
The research on this topic has been discussed on HN multiple times. It is indeed hard to look up previous discussions, but the conclusion is always inconclusive... one thing everyone seems to agree nowadays is that measuring LOC is a bad proxy for programmer productivity, specially when you take code maintenance into consideration (which very few research papers do), and that types do have a small, but measurable effect on improving quality (though whether that's at the cost of productivity is still unclear as far as I know - yes, programmers need to spend a bit more time to declare their types, but without them they have to spend more time every time they need to call anything).
There's all sorts of work which cite that paper, like "An empirical investigation of the effects of type systems and code completion on api usability using typescript and javascript in ms visual studio".
For fun, here are some of those citation which themselves use the phrase "An empirical study" or similar in the title:
"An empirical study on the impact of static typing on software maintainability"
"An empirical study of the influence of static type systems on the usability of undocumented software"
"Do developers benefit from generic types? An empirical comparison of generic and raw types in Java"
"An empirical study on C++ concurrency constructs"
"An empirical study on the factors affecting software development productivity"
"An empirical study to revisit productivity across different programming languages"
> But I find javascript (/ typescript) noticeably more productive for small to medium projects.
This seems like a pretty apples-to-oranges comparison. Rust is intended to do things you couldn't/wouldn't use JavaScript for.
If you're talking about a small-to-medium project where you don't care much about mishandling memory, of course JavaScript is going to be more productive. Rust is forcing you to tell the compiler a lot of things that JavaScript assumes you don't care about (and is, most of the time, correct).
You’d bet wrong. Say I want to add two numbers and ship it to prod. In Python, that’s one line and an scp. In Java, that’s at least three, a compile step, and an scp. By definition I’m more productive in Python due to the language tradeoffs. That’s reducing to absurdity but the principle holds to a solution of any complexity, particularly in SRE/ops.
This really rears its head when Go and Rust shops say “oh let’s do all our operational scripting in the same language, too.” It seems nice in theory but is a complete mess in operations. As an SRE manager, I have data to back up the assertion that a compile step gets directly in the way, particularly in an outage situation (we need to recompile the script with a flag to change it - and we are usually pushed toward CI/CD for that outcome). If you are asking this of your operations teams, they absolutely hate you regardless of their experience with the language.
I have 21 years writing Java and three writing Python and I’ll reach for Python for “move files from directory to other directory based on heuristics” loooooong before I touch Java for the same purpose, and that’s a high bar to even get there beyond doing it in a shell. Reducing this discussion to familiarity with a language is a complete red herring and dismisses every argument the other side has with “if you were just better at Java this wouldn’t be an issue,” which is flat out wrong.
Right tool, right job. I see I’m -2 before I’m finished editing typos which I think speaks to the article’s point about language ideology more than anything.
> Say I want to add two numbers and ship it to prod. In Python, that’s one line and an scp.
This example ignores what is actually involved in production code. Yes there is a compile step involved in things like Java. In python you have to manage your dependencies on the deployment box.
In Java (minus the runtime), Go, Rust, etc they all support managing dependencies on the client side. In my experience the messiness of that in python and ruby far outstrips the complexities of a compile step in the makefile or CI pipeline.
Correct. I reduced the comparison to absurdity and intentionally overlooked several concerns, and pointed that out, to note that binary yes/no is more nuanced here and that the alternative opinion can be built from first principles without invoking familiarity with a language. I apologize that I led you to respond to an absurd argument. I’ll try harder to note it as an absurd argument in the future - should I use a better word than absurdity?
Yes, you can write your 2 line program faster, no one is disputing that.
But that's not the real world, in real world you have to change and debug large amounts of existing code, and collaborate with colleagues. There's little conclusive research on it, but so far everything hints at typed languages being superior for that. Heck, untyped "script-like" languages tend to evolve into the direction of added types and build systems (e.g. Typescript, Python type annotations, Babel, Webpack etc.)
Actually, the person I responded to did dispute exactly that by claiming the limiting effect of language choice is gated by experience. Read what they said again. That’s also reducing the argument to absurdity, which I pointed out, and which apparently did nothing to dissuade you from taking it as my actual position.
You’re also explaining a single sector of software usage to me (i.e., the revenue SaaS of the company) as “the real world,” which itself dismisses every other aspect of software development in a shop. Again, including SRE and ops scripts, which are almost always inappropriate in a mainline, industrial, compiled language. Please try to avoid explaining your point with the beginning assertion that those who disagree with you do not live in the real world.
I am OP you're talking about... you misinterpreted my point completely.
I was saying that Python and other dynamic languages are not demonstrably more productive for writing general purpose software... as I did not mention scripting at all, I thought it would be obvious that this is what I meant.
> Read what they said again. That’s also reducing the argument to absurdity, which I pointed out, and which apparently did nothing to dissuade you from taking it as my actual position.
This is absurd. What "they" said was _programmers tend to be most productive in whatever language they know best_. How the hell is this disputing "you can write your 2 line program faster (in Python), no one is disputing that."??
With all the due respect, you need to work on your text interpretation skills. Do you understand that "programmers tend to be more productive" is not the same as "programs are always, without exception, more productive"?
It’s pretty ironic that I was referring to me reducing the argument to absurdity and you’re going after me “with all due respect” for my text interpretation skills. You didn’t think it weird I was talking about pointing something out and making an argument in your point, by your interpretation, before you launched off on this crusade to defend your honor?
With all due respect, read better and save the lecture.
I used to work at a C++ for prod, Python for prototype place. Time and time again, a 1-2kloc Python prototype, the bulk of which was written in a week, would take months to translate to C++, at times longer than the whole process of research+prototype.
I think it was 2 things: one, it is genuinely harder to write a valid C++ program. The language imposes non-trivial constraints on the programmer, for better and worse, and you have to navigate them for the thing to even compile. But two, C++ invites pedantry. I heard endless discussions about "ooh, template meta-programming", "struct or class", "but is this idempotent", "you should be using move semantics", "why on earth is this a unique_ptr".
I'm not saying C++ is worse, or that its trade-offs are not worth it, but for sure, from my experience, translating an algorithm into code is done far more quickly in Python than in C++. YMMV
Ah and also, this was definitely not a question of deficient C++ coders. The hiring standards for C++ were so high we were always short of devs. Meanwhile, Python prototypes were usually written by part-time ex-academia Python dabblers.
Although that way may not be obvious at first unless you're Dutch.
—The next line
This ZoP line is by far the most misused cliché of all things Python. How is it determine that “prototyping in C++ and rewriting in C++” is not the obvious “one” way? Because you don’t like it? A Zen is intentionally self contradicting to allow introspection, not judge others :)
I think a large part of the difference is the prototype vs prod. I have written prototypes in C++ in a few weeks that still took months to go to production. It is just that production-ready code require significant more ceremony, more config files, more complex error handling, code reviews and of course feature creep.
I still agree that Python can be much more efficient to write than C++. I just wish it wasn't so unbearably slow.
I would mostly agree, though not entirely. Continuing with the anecdata, at times I had enough waiting and put my prototype into prod, with significant hardening. I can't say I always got it right, but usually 3-5 days of very focused poking made the thing at least reliably run at the desired cadence, with rare but loud and graceful failures. For sure though, there is a tax for doing it all "properly".
Even then though, I have the impression that Python just makes it easier to write it all more quickly. It just doesn't invite you (as much) down rabbit holes of doing things even more properly.
I wonder how your C++ would've gone if you had a C++ library of Python-like data types. Something like a 'variant' type that could hold arbitrary integers, strings, and dicts or lists of those (or anything else you commonly used).
You would initially trade some of the C++ performance, but the code shouldn't be more complicated or difficult to write or compile than Python. Then, you'd only need to change the performance sensitive parts to use real C++ arrays or structs.
C++ is C. We used to optimize with assembly but nowadays with a processor's superpipelined architecture and the compiler's global register optimizations and so forth your inlined assembly is more likely to throw a monkey wrench into the whole works and make everything run slower. It's a better use of your time to look for a better algorithm.
No, because you wouldn't have to pay the virtual machine overhead (among others). Another possibility would be using Cython - write the prototype in pure Python, then Cythonize the parts which need to run more quickly.
But two, C++ invites pedantry. I heard endless discussions about "ooh, template meta-programming", "struct or class", "but is this idempotent", "you should be using move semantics", "why on earth is this a unique_ptr".
I wonder what the result would have been in eg. Go or Rust, where that level of conversation doesn't exist. True, others would have taken place instead, but at a higher level nearer the design, where they should have been happening anyway.
> Time and time again, a 1-2kloc Python prototype, the bulk of which was written in a week, would take months to translate to C++, at times longer than the whole process of research+prototype.
I've heard reports like this for many years.
My pet theory is that the discrepancy here is ~10% static types and ~90% memory management. Having a GC (or refcounting or whatever) fully baked into the language so that you don't have to think at all about how values are passed around and stored is a monumental productivity boost. Possibly the biggest win in software engineering productivity in the history of the field.
Absolutely. I recently translated Microsoft's example code for using the Windows crypto API from C++ to Python, and huge chunks of the code melted away as it was mainly concerned with freeing buffers. Exceptions also helped.
> But two, C++ invites pedantry. I heard endless discussions about "ooh, template meta-programming", "struct or class", "but is this idempotent", "you should be using move semantics", "why on earth is this a unique_ptr".
Interestingly, I've consistently had the opposite experience. In C++, when I ask, "how do I do X" there will be a few different options, but pretty much any of them will do. As long as you avoid things that are definitely UB, you're fine. In Python, I'll find endless religious arguments about which is the most "Pythonic" method. Whenever I try to just hack out Python code to just get things done, I always feel like I'm being judged, because usually the quick-and-dirty method is far from the most Pythonic.
I think it all depends on who does your code reviews. I agree with GP that c++ has religious adherents to the one-true-undocumented-in-their-head standard who will make writing it a nightmare. In python the philosophy is supposed to be - there is an obvious way to do things and that’s the right way. However this does fail from time to time.
Ruby on the other hand embraces many different syntaxes to do the same thing, but they’re all accepted as correct in the traditional view.
I think different languages do have sweet-spots, and it often has to do with library availablity (for me at least).
I consider myself as someone who knows "far too much about C++", but recently when I had to spider some webpages and read some values out of them to populate a database, I did that in Python because it's a handful of lines due to high quality libraries that all work nicely together. I honestly wouldn't know how to do that in a short amount of time in C++.
I admit that Python is more productive than C++, but the reason is mostly to do with memory management... a language with a GC, like Java or JavaScript, would use a very similar level of abstraction as Python and therefore, should "cost" about the same to write.
Python and other dynamic languages don't use static types and that may give them a (very) small advantage in productivity as well, but only for very small programs... as program size increases, in my experience, statically typed languages take the lead in productivity... as OP is about writing general software, not just small scripts, I really have a hard time agreeing that Python is either the best tool for the job, or the most productive tool at all.
That takes advantage of the fact that Python puts a bunch of relatively obscure functionality in scope by default... thus increasing the chance that programs will use the wrong method or rely on stuff they don't really intend to. Case manipulation isn't something most modern programs should even be doing at all, since it's almost impossible to internationalize it.
In fact it's really mostly name space management. Python doesn't require you to pull in "upper" because it starts with a relatively big name space. It doesn't require you declare the main program because you're relying on the fact that Python executes a file on loading it (and that reliance is arguably bad Python style). All of the "std::" stuff is a name space management choice.
... and the rest has nothing to do with static typing either. The way the code is indented on multiple lines is a stylistic convention. The transform primitive requiring you to specify bounds is a library choice, related to the language's apparent lack of a universal way to map over a container. I'm not sure why you need the lambda. The rewrite in place is memory model stuff inherited from C.
In Haskell, which is fully compiled and is about the most rigid static typed language you could ever imagine:
import Data.Char(toUpper)
main = putStrLn (toUpper <$> "hello")
If you'd asked for something that didn't require oddball functionality, then that could probably also have been a one-liner.
int main() {
std::cout << ToUpper("hello");
return 0;
}
but this would be no good! You have chosen an API that forces your users to be gratuitously inefficient. Maybe that is OK for your personal project, but an API like that has no business in the standard library.
No one is stopping you from writing your C++ inefficiently if you choose to. I am skeptical that spending a bit of time thinking about ownership/lifetimes is really such a big hit to productivity.
I've done big desktop program in python. For some parts, we just used modules written in C, because code was easily 20 times faster than python, I was even surprised at how fast simple blits could be on modern hardware without any graphics card acceleration. But python version was several times faster to write.
> Python and other dynamic languages don't use static types and that may give them a (very) small advantage in productivity as well, but only for very small programs... as program size increases, in my experience, statically typed languages take the lead in productivity... as OP is about writing general software, not just small scripts, I really have a hard time agreeing that Python is either the best tool for the job, or the most productive tool at all.
As a counter point to this - Python has a large number of C-based libraries with excellent bindings. To take Numpy has an example, you can the static-types and speed of Numpy to do all your maths, whilst Python acts as the manager. Using Numpy "feels" like normal Python, you don't really feel like you're working at native-level, but you get all the advantages of native speeds and memory usage.
And to your other point, I've worked on large C++-based projects which were a complete mess, especially when you had to make any changes to existing code. I'm not sure the language has as much an affect here as just using the correct design principles.
There's a great deal of evidence that supports that, it's just not scientifically organized. If you just look at the amount of high quality high polished projects out there for the dynamic programming languages it absolutely dwarfs those of static languages. And then add to that the fact that most of those are by solo devs in their free time.
It's not just that they are more popular, Java reigned supreme for years when Ruby and PHP and later Node.js outpaced it's communities by miles. And you can't say Java developers are not open source oriented, because they 100% are.
And the argument that programmers are simply most productive in the language they know best is trivially untrue. When I first learned of Ruby, I could dream in C#, I was absolutely fluent, knew the standard library by heart. I switched to Ruby and only looked back whenever I needed tight performance. And this is not just an anecdote, the whole Ruby on Rails movement was basically Java developers fleeing to greener pastures.
I don't know a single highly experienced multi-lingual developer who does not reach for a dynamic language when they need to deliver something quick and easy.
The only exception I know off is using Go for small networked services, which is quite comfortable and intuitive for a static language. Also outside that niche Go quickly loses its productivity edge.
> I don't know a single highly experienced multi-lingual developer who does not reach for a dynamic language when they need to deliver something quick and easy.
If you're picking from a popular language, the language itself rarely matters for this at all. Every popular language does roughly the same things. Writing speed (length of keywords, for example) is a non-issue. The ecosystem is by far the most important thing.
It doesn't matter how efficient I am in Go if I'm missing a crucial library that I'll now have to write myself.
JavaScript is a classic example where, even if you are pretty fast at writing your code, you will be dramatically slowed down by: 1) needing to add a new package every 5 min to do something trivial, 2) looking through five half-dead libraries to find one that seems maintained and usable, and 3) finding out that some of the libraries you chose are buggy.
So for a quick prototype, I'd weight a language's qualities as follows:
- ecosystem: 90%
- static analysis/tooling: 5%
- stdlib: 3%
- syntax: 2%
And for a long-term, complex project, I'd weight them closer to:
> I don't know a single highly experienced multi-lingual developer who does not reach for a dynamic language when they need to deliver something quick and easy.
This argument is not addressing the claim for general purpose software, only for "quick and easy" software.
To people in this thread: please stop to think before responding to the "wrong point". I think we can all agree that dynamic, scripting languages are more adequate for "quick and easy" software, but this is not what the point of the discussion is. The point of the discussion is, or should be in my opinion, whether it's true that for general purpose software, written by teams, that is not trivial to write by oneself in 5 minutes, that scripting languages are more productive than statically typed, compiled languages.
Where do you draw the line? Is GitHub quick and easy software? Is Shopify? Were LinkedIn, Twitter? Are things built on Tensorflow?
I think you misunderstand the scripting language revolution of the mid 2000's. It's not that we suddenly realized scripting languages were the best for quick and easy projects. We realized scripting languages were suitable for a whole lot more than some data processing. We could much more effectively build huge scalable software platforms.
It got so bad that the sentiment flipped, and people like Joel Spolsky had to go out of their way writing blog posts that you could in fact build successful modern web platforms in C#. And then he went building the world's most popular project management tool in Node.js anyway.
> I don't know a single highly experienced multi-lingual developer who does not reach for a dynamic language when they need to deliver something quick and easy.
Maybe this shines light on your social circles more than actual language impact on proficiency?
The "our language is so much more productive" myth exists in many languages, including the one I currently use. But when you dig deeper, there are almost always other social or technical factors that explain the productivity gap. That kind of gap exists even between people using the same tool.
The reality of it is that assessing what makes a developer productive is incredibly hard, and people doing so to claim their language of choice is better rely on anecdata and couldn't explain what "methodology" means to save their life.
> I don't know a single highly experienced multi-lingual developer who does not reach for a dynamic language when they need to deliver something quick and easy.
Hi, highly experienced, multi-lingual developer here (Python, Ruby, PHP, JavaScript, Java, Go, C#, F#, OCaml, Elixir/Erlang, ReasonML, SQL, Smalltalk, Objective-C, Swift, leaving off quite a few prior Web 2.0 ones for brevity) nice to meet you. As you can see, I've done just about all of it: every paradigm, every syntax. It all honestly blurs together after a certain point, and it just becomes easier and easier to pick up a new language the more you learn.
Anyways, these days, I never reach for a dynamic language when I need to deliver something quick and easy. The difference is just too marginal. Not worth the downsides (and the downsides are immense!).
Because, as my experience has taught me, invariably one of those quick and easy ones will turn into something that becomes business critical, lives on for years after you are gone, and will be much more difficult for other, typically more junior, developers to update or enhance your code. Static typing is not only marginally less productive these days with all the great tools and IDE's out there (not to mention Go which has one of the least obtrusive static type checkers I've seen, or, even better if the org allows you, a language with a HM type system), but for a marginal improvement in productivity, you pay a heavy long term cost. Not worth it.
Dynamic languages are great for rapid prototypes. After that, convert it to a static language. Your junior devs that join after you, who aren't familiar with the entire ecosystem your work has will thank you.
I don't think you can have a hard and fast rule like this. It will depend on the situation. In many instances, a "prototype" built in Python or similar is perfectly fine.
In addition, one can make a horrendous mess in statically typed languages as well. One of the absolute worst projects I ever worked on was written in a popular statically typed language. This is of course an anecdote and not to say statically typed languages are worse, but they're not automatically better either.
Just a note too that I'm a big fan of, e.g., Rust & TypeScript, and I often use type hints in Python, so I'm not anti-static typing.
Way easier to refactor a that statically typed mess.
There is a floor for how bad you can write static typed code.
There is no floor to how bad of JavaScript or Python or Ruby you can write. The madness can descend to the inner most circles of coding hell. Only Perl exceeds it.
Go in the general case, Java if complex domain modeling is required (or requires "enterprise grade" B.S - SOAP, WSDLs, or other crazy XML specified madness like Adobe extension stuff, etc), and after that: anything functional if I am allowed.
Explained:
Go's type system is generally less rigid. It strikes a good balance of strict enough. A lot of Go's converts aren't from "systems" languages like it targeted originally, but rather former Python/Ruby/PHP/JavaScript backend devs. I love the performance and low level levers I can pull with Go (although to be fair Java is quite fast enough). But finally, Go is easy to learn (26 language keywords?), the standard library is mostly great, and the worst developers I've seen still write mostly maintainable code that builds fast, which is what I optimize the most for these days.
Java, for all its warts and legacy cruft, these days you can write fairly good java, utilizing modern libraries. I love most things from Codahale, who in turn I think pushed orgs like Spring to write better libraries, so now everything's pretty good. Plus all the legacy stuff comes in handy when you have to deal with arcane government or financial systems, something I have to interface with frequently.
But if I had my choice, I'd use something where you can express functional programming concepts intuitively, without fighting the language or having to do it at a heavy performance cost, like Rust or better yet, just a full fledged FP language like OCaml (whom I understand heavily inspired Rust)
If I'm prototyping something like a log file analyzer or web scraper I use a dynamic language because it gives me a debugger REPL. I can interactively develop code with immediate results and copy & paste straight from the terminal into the editor.
When I want to develop further, I put a debugger statement right where I want to pick up, where all the data is available, and develop from execution.
Some static languages have REPLs but few are as good as dynamic languages for mutating existing code in the middle of a debugging session.
If you don't write much code which involves exploration of data or unknown APIs, then this mode of development may not be as useful to you, but it's a significant productivity advantage for me and the reason I reach for a dynamic language. Ruby is my go-to, with binding.pry as my debugger REPL.
Python, like all dynamic languages, works best to bash out a new project into production ASAP, get a bonus and move on, while poor folks maintaining after you are trying to untangle it.
Current Python doesn't reflect this philosophy though. You have lots of options for strings and string formatting. There is pattern matching and if statements. The standard library often isn't the best option for stuff (like HTTP requests) so you use another library. Package management and deployment is far from solved, with lots of different tools.
> You have lots of options for strings and string formatting.
“Strings and string formatting” is a very broad domain.
For most specific tasks, there is one low-impedance approach and it takes very little (but not zero) reflection and/or experience to find it.
Most new Python features directly address specific tasks for which there are currently multiple relatively high-impedance approaches taken because there is not one obviously correct way.
Python's “one obviously correct way” is not “one possible way” (the latter approach is closer to Go.)
(BTW, the TL;DR is people that know more than one language admit they all fall short one way or another, but there are some languages that are more productive than others).
> programmers tend to be most productive in whatever language they know best
This is true, but most software is built by multiple people (for corporations) and will eventually be maintained by multiple other people. Whether an individual is "most productive" in a language is rarely important.
I spent 7 years working with C#, I like C# and I think what they’ve done in the past few years is amazing.
I’m much more productive in Python. Not as in “I feel” more productive, but as in measurably more productive.
Where Python falls short of something like .Net and C# is that I probably wouldn’t have been more productive in Python if I didn’t have 7 years experience with a rather strict environment, and as far as using Python on major projects, well let’s just say that there is a reason people use TypeScript instead of JavaScript and Python doesn’t fall into that category.
But for most programming and for most minor systems or services, Python is just wildly good.
I also spent many years working with C# before working with Python professionally. I've also spent a lot of time with Lua, though not for paid work.
The second point matches my experience with dynamic languages as well. To use them well really takes a certain level of discipline, but it pays off. It's why I'm a bit skeptical of Python as a beginners language, which it sometimes has been touted as.
About large projects, presently I am working with Python on a somewhat large project (currently 53k sloc, I guess large is relative). Its been great, though we are pretty strict about almost absolutely everything being type-hinted.
How many people are writing 1k SLOC/day? And doing so consistently? If you're banging out that much code, I'd be severely concerned for the quality of the code itself as well as overall design.
Senior developers (really seniors, not 3 year seniors) can trivially produce such amount of code, because they wrote almost same code dozen times already.
My point isn't about who _can_ do it but who _does_. If anyone on my team was writing anywhere near 1k LOC/day, I'd think it was a serious problem. If the whole team was pumping out 1k LOC/day/person, I have to imagine I'd be getting as far away from the entire team/org as possible.
Maybe not the best example, but I used to work with a guy who had pretty large scripts that he was writing for one of our projects. It turns out it was mostly copy-paste, so it sort of got the job done -- except he fixed bugs in one place but not in the other. The whole thing was a huge mess to understand and maintain. And he was a senior engineer at the time (now principal, to my great amazement).
Developers at fixed-price contracts. Fast work + low number of bugs = good margin. It's exhausting, but, IMHO, it's better to work hard for 6 months and then relax for few months than to slowly push project for 2 years with red eyes and headache.
Why are they re-writing the same code instead of re-using it? Either convert to a library, or copy&tweak the existing code.
Why are they writing the same code again instead of writing different code? Why aren't they learning about new APIs for topics they aren't so experienced in? What makes them "senior developers" and not "senior transcriptionists"?
Who is tasked with digging into 10 year old code, written by an ex-employee, to find a subtle bug? Who is identifying and fixing performance issues?
How are these "senior developers" able to write 1KLoC/day plus documentation and developer tests? I find test code and documentation each take as much time as writing the code in the first place.
Because previous code is owned by previous client.
For example, 1M+ LoC of code, including build system, CI, test cases, built-in documentation, in 6 months is about 7kLoC per day, divided by 5 developers it's about 1400LoC per day per developer.
I completed 30+ projects already, I have 20+ years of experience. Few years ago I was able to close up to 20 tickets per two week sprint, when I worked with same level senior developers and dedicated PM, product owner, and QA team, at large outsource company at fixed-price projects.
It's easy when tickets are properly sized and described by PM, when PO is responsive and easy to reach, and when QA covers your back for complex test cases. 6 month project + 1-2 months to recover after, and then another such project.
Are your fixed-price estimates based on LoC?! Else, why aren't you embedding that knowledge in a corporate library ("contractors tools") which you license to all your future customers? Twenty years of playing one song over and over for years, even done well, doesn't make someone a concert musician.
Those numbers are ridiculous, and I say this having 25+ years of professional experience.
The typical industry numbers are under 100 LoC/day.
"Improving Speed and Productivity of Software Development: A Global Survey of Software Developers" at https://uweb.engr.arizona.edu/~ece473/readings/9-Improving%2... (Fig 6) has Lines-of-Code per Total Man Months at about 1,750, so about 81 lines per day (assuming 21.62 work days per month).
"A Practical Approach to Software Metrics" at https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=819938&... says "Many industry rules of thumb describe programmer productivity in terms of lines of code, for example 350 [noncomment source statements] per engineering-month of effort." That's 16 lines per days.
That said, it's really easy to distort LoC measurement. And I mean beyond the "put in 1,000 lines of the form 'a=1'" cheat.
For example, LoC traditionally refers to source code, not test, CI, etc. that you use. Why did you use that non-standard definition?
Test code, for example, may contain a lot of data records and autogenerated code. This doesn't require the same work as a source line of code.
Consider SQLite. https://sqlite.org/testing.html says the library is 143.4 KSLOC, while the test suite is 91911.0 KSLOC - nearly 100 million lines of test code! These tests were not all written by hand. At that ratio of test code to source code you would have about 1,400 lines of source code.
And it's possible to mis-measure source code. A few years ago I added about 400,000 lines of code to my project in a few weeks. These were auto-generated when I replaced a very confusing set of C preprocessor directives with a homebrew template system to compile specialized versions of functions across my parameter space.
I then added some Cython projects, which generates C files about 50x larger than the original pyx files.
Counting auto-generated code makes LoC a worthless measure.
You also wrote you have a QA team. I assume their test cases are in your repo, and counted in your 1M+ LoC count, but you didn't include their time.
The post-release defect rate per KLoC is about 7.47 (average) and 4.3 (median). See An Overview of Software Defect Density: A Scoping Study at ht...
You numbers are right. I just checked my stats on current project: 101 LoC per day. On this project I have ) incompetent PM, so ticket descriptions are vague thus I need to manage ticket queue by myself, ) hard to reach product owner, so I must figure lot of things at my own, *) no dedicated QA team, so I need to write test scenarios in addition to automatic test cases and then perform them manually, etc.
> Are your fixed-price estimates based on LoC?!
I don't now. I was not a part of sales.
> Else, why aren't you embedding that knowledge in a corporate library ("contractors tools") which you license to all your future customers?
Because each customer has different requirements, different goals, different language and framework. Nobody wanted to pay for a library.
> If you write 1KLoC/day then you're likely generating 2 post-release defect bugs per day. That's in addition to bugs your QA team catches.
In practice, I had about 1 serious bug slipped per 1-2 weeks. Professional QA team known a lot of corner cases to test already. Professional developers know them too, even when writing in a different language using different framework. Moreover, we know how to prepare good CI, efficient automated tests, how to write efficient documentation, who is responsible for what, etc.
> Your numbers are so far outside of industry standards and academic findings that, with 20+ years of experience, you must know they are exceptional and difficult for anyone else to accept on your simple say-so.
How much I will be paid if you will accept my story? :-)
Same here. I do game development in C# and Python is my goto for sketching POCs for complex mechanics, despite never formally learning the language. It could easily be the difference between spending the whole day vs two hours on the same issue.
> there is a reason people use TypeScript instead of JavaScript and Python doesn’t fall into that category.
I thought the dynamic vs static typing debate was pointless until I joined a large Python project. Now I pretty much consider anyone that can keep up with one superhuman.
I think part of the problem is that many people who are used to static type analysis get lazy with naming and interface design. Toss one of them into your team and they'll "prove" the need for static type analysis.
Possibly. IME it’s quick Python POCs that didn’t bother with good practices that turn into critical production services without anybody going back and applying good coding standards.
Sometimes you just want to throw entirely differently shaped data into the same array or dictionary and be done with it instead of creating a "proper" type-safe design which most likely means writing lots of pointless boilerplate code. IMHO that's the whole point of a scripting language like Python, to write quick'n'dirty scripts, not "real programs".
I've been programming for nearly 40 years now and I've used all the popular languages in that time (popular for microcomputers) and I've mastered a few. I just started Python for the first time last year. Am I a Python expert? No. But I can tell you right now I can get a project done faster in Python than using any language in which I'm an expert. It's that productive.
Does that mean Python has all the performance you would ever need? No - we know it doesn't, but we also know that you rarely need that kind of performance and when you do it's usually localized to a very narrow portion of your code. Take that portion of code and implement in C and optimize to your heart's content. Even with that you'll still get your project done much faster.
Python is the tool of choice for those developers who just want to get stuff done.
I agree with you on Clojure, but I'm not going to be able to get my team to adopt Clojure - unless you're going to tell me how you convinced your team to adopt Clojure! :)
I mean it is not just mentioning a specific experience to refute a generalization when there is also this: "Python is the tool of choice for those developers who just want to get stuff done."
Admittedly that last bit was a bit inflammatory, but I've gotten tired of the constant arguing over languages at where I work. I'm on a bit of a rogue team that's just adopted Python and are getting stuff done. Other people are just now starting to pay attention because we're getting so much done and now work is headed our way as a result. Wait until they find out we're not using any of the languages they're bickering about and are using lowly Python! Still, as awesome as my team is I think they'd draw the line at Clojure.
What features of clojure do you feel make you more productive?
I’m primarily a python user, but I spent time writing code in Ocaml to understand the potential benefits, but… (I would love to have my mind changed)… it feels like many of the features people touted about Ocaml have already made their way into python…?
1. Immutability / referential transparency seem more like nice-to-haves for a codebase, rather than the real reason people tout FP…?
2. Sum/product types and pattern matching are being added soon to python
3. Mypy is starting to gradually enable python typing, although I assume it’s still a work in progress
4. Python allows us to use map/reduce, and to pass functions around as arguments…?
I want to understand the potential benefits of FP more, but my experience with Ocaml hasn’t shown me great improvements yet. Open to having my mind changed.
I love Clojure. As with Python, Clojure comes with batteries included. It's dynamic typing and REPL makes development really quick and the JVM runtime allows to achieve better performance than Python in many cases. You can even go as far as compiling to native executable with GraalVM.
I'm a C++ programmer and a python programmer. I assure you your conclusion is incorrect. Python with type checking is by far more productive. Many people who know python and another language can testify to this general truth.
C++ is probably the least productive non-joke language ever created: it has deterministic destruction which requires (manual) ownership tracking, it has an extremely slow compiler with horrible error messages, it has massive performance differences between debug and optimized builds, it's focus on performance brings lots and lots of extraneous concepts to its libraries when just seeking to get something working (std::allocator, std::string_traits and many other similar examples).
Comparing Python to Java, C#, Haskell, Erlang, Go, maybe even C or Rust, would show a much smaller performance difference.
RAII is exactly what I mean by "manual ownership tracking and deterministic destruction". RAII is better than completely manual memory management (C style malloc/free), but it still requires you to design your program such that every piece of memory is owned by some pointer with the correct properties. You have to choose between copying, bare pointers/references, unique_ptr and shared_ptr whenever two pieces of data are related to each other, or whenever you pass a piece of data to another function.
Rust is a little better since at least the ownership concept is known to the compiler and automatically enforced.
(Tracing/Copying/Compacting) GC is much easier since this entire concept goes away. You always pass references to data, and the physical storage is "owned" by the GC itself. It's also much faster for certain workflow patterns, though it always consumes more memory than deterministic destruction schemes.
Problem is that if you follow that strategy then it will likely never be as performant as if you did "Make It Fast, Make it work". So if you know that performance is a key factor to success then you should follow this path instead.
And by "make it fast, make it work", I mean benchmark first. Kind of like test first, you write the benchmark before the implementation and constantly look at how fast things runs, and ensues every single bit you add is fast.
I don't think the "Make it Right" part should or can reasonably be left out of the equation, otherwise I could just compute something if all you care about is speed. That said, I have used a performance-first approach several times using own old and new solutions and libraries and have found early benchmarks to be a great tool to weed out untenable (i.e. order-of-magnitude slower) stuff. This then can mean I have to write fewer tests to make plausible my own or a 3rd party solution does in fact what I expect it to do. At a certain point performance becomes correctness.
Just been thinking through this... I'm working on a personal project (podcast indexing/search) that involves parsing a lot of RSS feeds. Some years ago I had built the feed checker in Elixir but now when I tried to get it going again I was having too much trouble with version and compatibility changes in Phoenix. I eventually just did that part in PHP cause that's my day-to-day language. Wouldn't have made sense to block all development until I re-learned how to query a database in Phoenix. Plus later I can re-implement that feed checking part as as service in Elixir or another language. First make it work..
I quite believe that every programming language has its strengths & weaknesses, and a few languages is a must in every developer's accoutrements.
Want to build an I/O utility writing to a DB? Sure C can do it, but Python is better suited. Want to write a toy compiler? You don't want to waste your time trying to wrangle on CPython extensions. C works out of the box.
> _'if you select a programming language based on your preconceived notions of how a language performs, you will never know if the language that might be a better, more productive fit'_
Part of the CS education is not about recognizing homeruns but understanding trade-offs. The experience gain is about learning how tools work & which tools to choose to work in tandem. Modern systems use a variety of languages - JS in the webpage, SQL DBMS for queries, C++ to run the performance bits, Python for ML, introperations - maybe even Rust in the security bits of late. In that sense, the title was unfortunately misleading to me, since author tried to demonstrate a lot of usecases with Python.
Python is great - but there has to be a reason why other languages co-exist. Not just for bankers, military or some enthusiastic hobbyist.
People are too shy on hardware costs. Many of my professional colleagues develop and optimize software full time that runs on a server that is as fast as the phone in their pocket.
I don't like how he makes it a choice between Python and C++/Rust. There is very many languages that are more similar to Python in convenience and yet run reasonably fast (and you can actually optimize some procedures when you need to, because there is a compiler). Go, Julia, C#, F#, Scala, Kotlin, even recent Java... all much faster than Python and much less pain to work with than C++.
And the interoperability is not as awesome as it's painted in the article, it's always more pain to have more languages in a project that need to talk together
You have to look at resources too. You probably find 10 Java developers before you find 1 Scala developer, even if they are related technically. Especially on long term projects you have at least some turnover of people.
I am less productive in Python vs say Go. So by default use Python would be a bad heuristic for people like me. Has nothing to do with runtime performance for the most part.
Julia one of the target is solving the "two-language problem"
"Julia seemed to have solved the “two-language problem”—a conundrum often facing Python programmers, as well as users of other expressive, interpreted languages. You write a program to solve a problem in Python, enjoying its pleasant syntax and interactivity. The program works on a test version of your problem, but when you try to scale it up to something more realistic, it’s too slow. This is not your fault. Python is inherently slow—something that doesn’t matter for some types of applications but does matter for your big simulation. After applying various techniques to speed it up but only realizing modest gains, you finally resort to rewriting the most time-consuming parts of the calculation in C (most commonly). Now it’s fast enough, but now you also need to maintain code in both languages, hence the two-language problem."
What's the tl;dr of how Julia is solving this? Looking around it seems the answer is "multiple dispatch". Which seems suspect considering many languages have already tried this (Common Lisp, for example).
> Clearly, multiple dispatch, or some other way around the expression problem, is necessary for the kind of fluent composability that I’ve described above—but it is not sufficient. Julia has enjoyed an explosive degree of uptake in the scientific community because it combines this feature with several others that make it very attractive to numericists.
That's incredibly handwavy. So what's the special sauce?
There is no such thing as a free lunch when it comes to dynamic vs. static. It also seems like Julia is trading off expressiveness and easy of use in favor of efficiency, based on comments from people that have used Julia. It's one thing to be faster than any inherently slow language (Ruby, Python, Smalltalk, etc.), but keeping that flexibility and being as fast as C/C++ is a rather bold claim. Most languages hit some middle ground between the two, such as Java. But no one is under the delusion that trade-offs weren't made to get there.
Julia makes a number of (in my opinion) really good tradeoffs here.
1. You can't add fields to a type (struct) after definition. This means that Julia's structs have no overhead and are essentially equivalent to structs in C (although they are parametric)
2. No local eval. Eval in Julia only happens in the global scope and results of eval are only visible the next time you visit the global scope. This may sound kind of unintuitive, but in practice people don't generally use this for good reasons. This allows Julia to never need to de-optimize code. Once a method is compiled that code remains valid.
3. Macros. Julia has really good macros and other code manipulation (since it is basically a Lisp). This makes it possible to generate very complicated but fast code that you would never write yourself. The tradeoff here is that it makes the language more complex, but that's a pretty good tradeoff. (especially compared to the C/Fortran land of using a preprocessor that works on text).
4. Just-In-Time (just ahead of time). Julia at it's core runs as if it were highly templated C++ code. If everything got compiled ahead of time, Julia would be generating terabytes of compiled code and never finish compiling. Instead, Julia makes the tradeoff of only compiling for the argument types that are actually used in the program, which means that it only compiles a reasonable amount of code. The tradeoff here is that compiling small binaries with Julia is very difficult (not possible to do automatically yet).
The TLDR is that most expressive languages started by giving away as much expressiveness as possible, and then looked at how they could be sped up. Julia started by being a modern fast language and looked to see how much expressiveness could be added without slowing the language down.
Your instinct that Julia is making tradeoffs is indeed correct, however I don't think it actually limits the expressiveness of the language. I happen to think that Julia is a more expressive language than Python. However, it does require learning new patterns and paradigms and someone who tries to write Python code in Julia is probably bound to eventually get frustrated.
A huge part of the design considerations for Julia essentially boiled down to "what sorts of dynamism and language semantics can we disallow while keeping the the good parts of dynamism"
The two biggest things that had to go in order to make Julia fast was
1) the ability to change the memory layout of a struct in a running session
2) the ability to eval in the local scope (our eval always occurs in the global scope)
These two things are huge performance problems. We might oneday solve 1) with Revise.jl (though it'll mean recompiling all your code if you do change the layout) but 2) is basically just a very bad idea and likely to never happen. Instead of a locally scoped eval, we have macros, multiple dispatch, parametric types, and generated functions. These give an incredibly powerful suite of metaprogramming tools that are beyond anything available in Python.
Julia does a few things differently then Common Lisp, though they both offer multiple dispatch.
One of the key things in CL is that it has its metaobject protocol which forces a lot of decisions on what gets executed to runtime. There are ways to speed it up, but if you have something like:
Then CL won't call foo specialized on number when given an integer, but will call foo :before specialized on number. It determines this at runtime by searching for all applicable methods based on the type (at least as a first pass, you can cache this to speed it up but then you also have to have cache invalidation if a definition is changed).
Julia doesn't have that aspect of CL's MOP. So this helps to simplify the search for applicable methods and dispatch. Even if it did all its dispatch at runtime, it would still be simpler. The other thing Julia does is aggressive JIT compilation. So if you wrote something like (with the Julia equivalent of foo from above):
function bar(x,y)
foo(x)
foo(y)
end
And, only considering floats and integers, later called it with each pair of float and integer then Julia would compile specialized versions for those 4 combinations. Now when you call bar it still has to properly dispatch it, but once inside bar the search for the correct foo can be bypassed because the types will be known. CL, again thanks to the MOP, doesn't make that as easy to achieve.
I guess the best way to put it is that Julia encourages a style where 90%+ of code can go through paths that are static.
Personally, I think Julia starts off as easy as python, but to get C++ or Fortran speed, you can't just code naively. Things go into a steep learning curve at that point, but perhaps there isn't yet as much know how about how to code "professional Julia" yet. There needs to be a book like Fluent Python or Effective C++ for Julia, or perhaps a condensed version of the Julia manual (see the 1 page zig manual for inspiration).
The other problem I have with Julia right now is lack of static type checkers. "modern python" (e.g, python in production in the last 5 years) tends to leverage the large ecosystem of things that hook into mypy (I'm taking about tools like pydantic) to reduce the inherent brittleness of the language. Ruby, php, and every other dynamic language has also seen that trend.
Right now, I've barely seen that with Julia, and it needs this badly for higher uptake in industry. It's why for example, perhaps you see a lot of Julia packages written for people's phds right now.
There are trade-offs made. This discussion of "why Julia" describes how multiple dispatch + type stability is what gives the speed, but the trade-offs and edge cases associated with that.
Where does the author imply something like "Pick python no matter the .. etc.?"
I read it as the much more milder "don't reject Python because of vague concerns about run-time performance".
I can't find anywhere which suggests the author things people should use Python to, for example, code up their web app front-ends or to implement a 'hard' real-time operating system.
"Guys, our Python prototype is finished. It works really well and has proved our concept. I know we've put a ton of work into it but now I suggest we scrap it and rewrite it in a language that isn't dog slow."
We run pylint as part of our build, including as a prerequisite of merging a branch into master.
Our codebase is well over 1M lines of C++. We have about 100k lines of python. Running pylint takes the same order of magnitude of time (half as much IIRC) as running a full optimizing build + linking of the C++ code base. We run pylint over all cores, while we run the C++ build only on a subset of cores on the build machine.
I would call that dog slow, unless you think python is not the appropriate language write pylint. And no, it is not fine.
Dropbox is the biggest user of Python that I know of, and they went as far as writing their own JIT compiler before giving up and switching to faster languages.
>Selecting a programming language can be a form of premature optimization
>Prototype in Python.
Stopped reading here.
Wait, what? The only thing I know about python is that it's indentation-sensitive, no idea about syntax or libraries. Suggesting me to use python is premature optimisation.
My (perhap controversial) take is that static typing in Python is also a form of premature optimisation. Code should be written without static types first, and static types should only be added when absolutely necessary. 99% of the time, it never will be.
Why though? Static types make it easier to write code overall, so you're slowing yourself down for no real benefit.
You could just as easily say "Descriptive variable names are a form of premature optimisation. Code should be written with single-letter names, and descriptive names only added when absolutely necessary."
Selecting a language is like selecting a tool. It would be worse to select a language that on first sight fits the problem if you don't have any experience with it.
To be honest, today there are many viable solutions with different languages. I would not recommend to start with a plan where you have to reimplement the system in another language at some point. Experience tells me that almost none of such projects survive.
Actually, writing a POC or a MVP is quite different to writing a "long run" production product. In the first case, coding speed is surely a must - and any untyped (or loosely typed) language might be good enough - but in the last case, when dev teams have to maintain some code in the long run, during multiple versions, with turnover, typed (or pedantic) language might be easier to work with because the language already include some kind of documentation (types) and automatic checks (type verification).
Moreover, the availability of "average" (and "cheap") programmers matters a lot in the long run: if only genius can maintain your system, then you'll have problem in the long run because you'll either need to keep them at all price or need a lot of time to replace them. So, in the long run, you should better use a wide audience language with a lot of available programmer (even if they are "average") than a specific language requiring good programmers. However, for an MVP, you can recruit a genius programmer using the fastest tool for the job.
Obviously, some domain are more oriented toward some language... and for ML for example, python is quite a good choice because of the libs (as Java could be for - lets say web servers)
So it matters a lot what your system will be used for and how much time you will require it to run before needing to rewrite it from scratch
Over time I've come to conclude we are not optimizing for language features, we are optimizing for the community around a programming language or stack.
Sure, features are important and if critical ones are missing it might be a show stopper. So there is an initial thresshold that all candidates must pass.
But problem solving is not a one-off exercise. It tends to be both dynamic (=facing unpredictable challenges) and recurring over long time horizons. Which means having a healthy, engaged, resourced community that will invest in adapting / solving future requirements is essential.
So the "optimization" problem includes quite a bit more than the presently known developer team, its software stack its hardware and current problem definition / user requirement.
I think you see this dynamic in several cases (including python) where you might not think that it makes rational sense.
319 comments
[ 3.0 ms ] story [ 289 ms ] threadVisual Basic, PHP and Java come to mind. Probably C# as well.
Here's the conclusion, which summarizes the essay:
> All of this is to say while some companies do extract massive value from squeezing every CPU cycle out of their code, those companies also typically build data centres. So if you don't need a dedicated building to host your machines, please consider doing the math to see if it's truly worth making your developers less productive in the name of computational efficiency when you don't even know if that perceived efficiency is even necessary.
As an example, the kernel of my software is only a few hundred lines long. It's in C with AVX2 intrinsics to eek out the last bit of performance. The Python overhead is <1%, which means that even swapping out Python with a 1,000x faster language won't be noticeable. This is the '"glue" code' mentioned in the section on 'Use language bindings'.
Had I started with "It needs to be fast therefore I can't use Python", then it probably would have been a lot more work to do. (See, "numerous anecdotes of where someone implemented something in Python in 1/3 the time a competing team did creating the same thing in e.g. C++ or Java".)
I would say the article is very much doing that, it's just stopping short from spelling it out clearly.
It starts from the flawed premises that Python is much more effective to come up with actual production code than other languages [1].
The author then moves on to telling you that you should try at least a couple of times to code something in python before obvious performance shortcomings are glaring, then keep python as a binding language.
At no point does the author acknowledge that there may be other reasons to pass over python than performance.
From the way he approaches the subject, the author seems to live in the paradise where you only have to sling half-baked prototypes to prod, and then move on to another project while poor souls handle maintenance.
The take on why performance matters [2] also completely overlooks the fact that optimizing for infra costs is likely the least common reason to write for performance. Sometimes, if you don't reach a given threshold, your code simply doesn't work. Sometimes, your performance is your users' time (and sometimes, these users are even developers). Such companies don't necessarily build datacenters.
[1]: "This is what leads to numerous anecdotes of where someone implemented something in Python in 1/3 the time a competing team did creating the same thing in e.g. C++ or Java." -- of course, it's just anecdotes and not a claim, so the author won't defend it, but you're still expected to believe it.
[2]: "All of this is to say while some companies do extract massive value from squeezing every CPU cycle out of their code, those companies also typically build data centres."
That said, the few and limited empirical results which do exist, like Prechelt's empirical work from 20 years ago, support the author's premise.
As another example, Boehm's work across a range of projects (<100KLoc) shows that code size has the biggest impact on development effort, and slightly non-linear scaling in KLoC (KLoC^~1.1 in COCOMO); and various publications show Python generally requires fewer lines of code.
(As one example picked semi-arbitrarily from a Google Scholar search, http://jucs.org/jucs_19_3/a_comparison_of_five/jucs_19_03_04... on a small project has Python at the smallest effective LOC, smallest effective byte size, and nearly the smallest (after Java) eByte/eLOC size.)
Modern C++ is a much more compact (and complicated!) language than 20 years ago, so of course all of these previous results must be reviewed and reconsidered.
As a field, we have not provided that funding for solid empirical research!
> At no point does the author acknowledge that there may be other reasons to pass over python than performance.
Shrug. Okay. And from that you conclude the author is saying that Python should be used in all cases? "I'm going to talk about why you should eat more vegetables" doesn't mean "you should only ever eat vegetables." Similarly, "I'm going to talk about why you shouldn't reject Python from the get-go for a project you think is performance intensive" doesn't mean "you should only ever use Python."
The author even talks about Rust in an earlier blog entry, from August of this year, at https://snarky.ca/introducing-the-python-launcher-for-unix/ saying "And so over 3 years ago I set out to re-implement the Python Launcher for Unix in Rust. On July 24, 2021, I launched 1.0.0 of the Python Launcher for Unix".
So the evidence is that Cannon does not believe that Python is the best language for all projects.
> Sometimes, your performance is your users' time
Cannon's essay is about the case where people select a programming language based strongly on expected computational cost, and forget other factors in the cost model. Here's the relevant quote:
] I think the jump to selecting a programming language based on potential performance needs often comes from a place where people think their computation costs are more important to optimize for than their developer time. I don't think that always holds, though, as software developers are expensive.
In your model, people are choosing a language based in part of the cost of users' time, which means they they are not primarily focused on computation costs, and so are outside the scope of this essay. That doesn't contradict the essay, only show that it's incomplete. Which we know already from the text itself.
> Sometimes, if you don't reach a given threshold, your code simply doesn't work.
And sometimes you can reach a given threshold with an off-the-shelf use of Pandas, rather than build your own analysis routine. If you start off thinking "Python must be slow" and don't even know what your thresholds or performance characteristics are, then selecting (say) assembly over Python is probably premature optimization.
Cannon didn't even argue that if you aren't building data centers then you must use Python. Your [2] trims the full paragraph I quoted earlier. You left out: a) the "please consider", which makes this a request, and not a rejecti...
If Rust or something really saves money because it means we don't need 50 extra web workers for the same TPS then I don't think it's "premature optimization" - unless the dev salary is too much!
As it stands we're all just saying stuff with no data or evidence to back it up.
[1]: https://www.destroyallsoftware.com/talks/ideology
Sounds exactly like business management culture. Makes sense though, developer productivity is first and foremost a business management issue so discussions about it ought to follow the same path as business management. And business management love their anecdotes, great person citations and jargon.
And of course the further you get from silicon valley the more the hardware/service costs matter. A developer in Sofia easily earns an order of magnitude less than one in San Francisco, which shifts the whole optimization tradeoff a lot.
My team does a ton of large-scale processing (most of which is ETL), and we have TONS of servers at our disposal. We effectively throw hardware at our compute, and even then, jobs can take 5-10 hours to run. No big deal - just run the job before you call it a day and it's done when you start tomorrow morning. Or when you have daily jobs, you schedule it and it's just done whenever it's done.
I can't tell you the number of times I've looked at a job someone wrote, made a 5 LOC change in 2 minutes, and cut processing from 8 hours to 5 hours. I literally had one that went from 5 hours to 5 minutes.
These extremely simple investments save a lot of money, but it's a problem that's spread across hundreds of different jobs, and it takes so much effort to convince people that we should invest in it. A daily job that costs $1500/mo to run (based on amortized hardware costs) needs only be optimized once, and those savings are yours forever. But I've had the conversation that is effectively: my time is more costly than a measly $1500 job, so it's okay if the job is expensive.
But we often overlook the fact that it's possible to scale those two, incrementally. With today's convenience in interop, we can always start with a prototype friendly technology and later switch tech for real bottlenecks.
Or even doing it at all.
I like the busywork. Less spooky than 'cultic ritual'.
This isn’t a knock on rust - I just think rust trades off programmer productivity for correctness and performance. And it shows, on both sides.
I recall a study being quoted in my university courses, which described the amount of code needed to get certain things done between different languages, which showed that Python, Ruby and others are on the less verbose side, the reasoning being that on average they'd also be more productive.
This seems to coincide with my personal experience, where JavaScript with React was much easier to work with in smaller projects, whereas using TypeScript with React lead to much slower development because of all the typing that needed to be handled, especially in cases of union types. Now, one can say that it's worth the effort to be more confident in your code doing what you want it to both now and after X months, much like you could sometimes prefer the type systems of Java or .NET over Python, Ruby or PHP, but in my eyes those tradeoffs always come with slower development velocity.
The sad thing, however, is that DuckDuckGo (and possibly other search engines) failed to return the original study or anything like it, only resulting in low quality blog content:
Anyone have any better search queries for this? Any idea why the search result quality is generally so low? Any ideas which study it might have been referencing?The first result for "program language comparison", at least when I view https://scholar.google.com/scholar?q=programming+language+co... is: Prechelt, Lutz. "An empirical comparison of seven programming languages." Computer 33.10 (2000): 23-29.
There's all sorts of work which cite that paper, like "An empirical investigation of the effects of type systems and code completion on api usability using typescript and javascript in ms visual studio".
For fun, here are some of those citation which themselves use the phrase "An empirical study" or similar in the title:
"An empirical study on the impact of static typing on software maintainability"
"An empirical study of the influence of static type systems on the usability of undocumented software"
"Do developers benefit from generic types? An empirical comparison of generic and raw types in Java"
"An empirical study on C++ concurrency constructs"
"An empirical study on the factors affecting software development productivity"
"An empirical study to revisit productivity across different programming languages"
This seems like a pretty apples-to-oranges comparison. Rust is intended to do things you couldn't/wouldn't use JavaScript for.
If you're talking about a small-to-medium project where you don't care much about mishandling memory, of course JavaScript is going to be more productive. Rust is forcing you to tell the compiler a lot of things that JavaScript assumes you don't care about (and is, most of the time, correct).
This really rears its head when Go and Rust shops say “oh let’s do all our operational scripting in the same language, too.” It seems nice in theory but is a complete mess in operations. As an SRE manager, I have data to back up the assertion that a compile step gets directly in the way, particularly in an outage situation (we need to recompile the script with a flag to change it - and we are usually pushed toward CI/CD for that outcome). If you are asking this of your operations teams, they absolutely hate you regardless of their experience with the language.
I have 21 years writing Java and three writing Python and I’ll reach for Python for “move files from directory to other directory based on heuristics” loooooong before I touch Java for the same purpose, and that’s a high bar to even get there beyond doing it in a shell. Reducing this discussion to familiarity with a language is a complete red herring and dismisses every argument the other side has with “if you were just better at Java this wouldn’t be an issue,” which is flat out wrong.
Right tool, right job. I see I’m -2 before I’m finished editing typos which I think speaks to the article’s point about language ideology more than anything.
This example ignores what is actually involved in production code. Yes there is a compile step involved in things like Java. In python you have to manage your dependencies on the deployment box.
In Java (minus the runtime), Go, Rust, etc they all support managing dependencies on the client side. In my experience the messiness of that in python and ruby far outstrips the complexities of a compile step in the makefile or CI pipeline.
But that's not the real world, in real world you have to change and debug large amounts of existing code, and collaborate with colleagues. There's little conclusive research on it, but so far everything hints at typed languages being superior for that. Heck, untyped "script-like" languages tend to evolve into the direction of added types and build systems (e.g. Typescript, Python type annotations, Babel, Webpack etc.)
You’re also explaining a single sector of software usage to me (i.e., the revenue SaaS of the company) as “the real world,” which itself dismisses every other aspect of software development in a shop. Again, including SRE and ops scripts, which are almost always inappropriate in a mainline, industrial, compiled language. Please try to avoid explaining your point with the beginning assertion that those who disagree with you do not live in the real world.
I was saying that Python and other dynamic languages are not demonstrably more productive for writing general purpose software... as I did not mention scripting at all, I thought it would be obvious that this is what I meant.
> Read what they said again. That’s also reducing the argument to absurdity, which I pointed out, and which apparently did nothing to dissuade you from taking it as my actual position.
This is absurd. What "they" said was _programmers tend to be most productive in whatever language they know best_. How the hell is this disputing "you can write your 2 line program faster (in Python), no one is disputing that."??
With all the due respect, you need to work on your text interpretation skills. Do you understand that "programmers tend to be more productive" is not the same as "programs are always, without exception, more productive"?
With all due respect, read better and save the lecture.
I think it was 2 things: one, it is genuinely harder to write a valid C++ program. The language imposes non-trivial constraints on the programmer, for better and worse, and you have to navigate them for the thing to even compile. But two, C++ invites pedantry. I heard endless discussions about "ooh, template meta-programming", "struct or class", "but is this idempotent", "you should be using move semantics", "why on earth is this a unique_ptr".
I'm not saying C++ is worse, or that its trade-offs are not worth it, but for sure, from my experience, translating an algorithm into code is done far more quickly in Python than in C++. YMMV
Ah and also, this was definitely not a question of deficient C++ coders. The hiring standards for C++ were so high we were always short of devs. Meanwhile, Python prototypes were usually written by part-time ex-academia Python dabblers.
-PEP 20 -- The Zen of Python
—The next line
This ZoP line is by far the most misused cliché of all things Python. How is it determine that “prototyping in C++ and rewriting in C++” is not the obvious “one” way? Because you don’t like it? A Zen is intentionally self contradicting to allow introspection, not judge others :)
-- Django
-- Perl
-- assembler
-- fpga
Back at you.
-hdl
This seems like a monumental level of operational waste. I'd love to hear more about this particular setup
I still agree that Python can be much more efficient to write than C++. I just wish it wasn't so unbearably slow.
Even then though, I have the impression that Python just makes it easier to write it all more quickly. It just doesn't invite you (as much) down rabbit holes of doing things even more properly.
You would initially trade some of the C++ performance, but the code shouldn't be more complicated or difficult to write or compile than Python. Then, you'd only need to change the performance sensitive parts to use real C++ arrays or structs.
I wonder what the result would have been in eg. Go or Rust, where that level of conversation doesn't exist. True, others would have taken place instead, but at a higher level nearer the design, where they should have been happening anyway.
I've heard reports like this for many years.
My pet theory is that the discrepancy here is ~10% static types and ~90% memory management. Having a GC (or refcounting or whatever) fully baked into the language so that you don't have to think at all about how values are passed around and stored is a monumental productivity boost. Possibly the biggest win in software engineering productivity in the history of the field.
Interestingly, I've consistently had the opposite experience. In C++, when I ask, "how do I do X" there will be a few different options, but pretty much any of them will do. As long as you avoid things that are definitely UB, you're fine. In Python, I'll find endless religious arguments about which is the most "Pythonic" method. Whenever I try to just hack out Python code to just get things done, I always feel like I'm being judged, because usually the quick-and-dirty method is far from the most Pythonic.
Ruby on the other hand embraces many different syntaxes to do the same thing, but they’re all accepted as correct in the traditional view.
I consider myself as someone who knows "far too much about C++", but recently when I had to spider some webpages and read some values out of them to populate a database, I did that in Python because it's a handful of lines due to high quality libraries that all work nicely together. I honestly wouldn't know how to do that in a short amount of time in C++.
Python and other dynamic languages don't use static types and that may give them a (very) small advantage in productivity as well, but only for very small programs... as program size increases, in my experience, statically typed languages take the lead in productivity... as OP is about writing general software, not just small scripts, I really have a hard time agreeing that Python is either the best tool for the job, or the most productive tool at all.
In fact it's really mostly name space management. Python doesn't require you to pull in "upper" because it starts with a relatively big name space. It doesn't require you declare the main program because you're relying on the fact that Python executes a file on loading it (and that reliance is arguably bad Python style). All of the "std::" stuff is a name space management choice.
... and the rest has nothing to do with static typing either. The way the code is indented on multiple lines is a stylistic convention. The transform primitive requiring you to specify bounds is a library choice, related to the language's apparent lack of a universal way to map over a container. I'm not sure why you need the lambda. The rewrite in place is memory model stuff inherited from C.
In Haskell, which is fully compiled and is about the most rigid static typed language you could ever imagine:
If you'd asked for something that didn't require oddball functionality, then that could probably also have been a one-liner.As a counter point to this - Python has a large number of C-based libraries with excellent bindings. To take Numpy has an example, you can the static-types and speed of Numpy to do all your maths, whilst Python acts as the manager. Using Numpy "feels" like normal Python, you don't really feel like you're working at native-level, but you get all the advantages of native speeds and memory usage.
And to your other point, I've worked on large C++-based projects which were a complete mess, especially when you had to make any changes to existing code. I'm not sure the language has as much an affect here as just using the correct design principles.
Recent-ish commentary (July 2021) about it at https://renato.athaydes.com/posts/revisiting-prechelt-paper-... , HN commentary at https://news.ycombinator.com/item?id=28108806 .
Google Scholar gives about 476 paper which cite Prechelt's work. I have not followed other work in that field.
Just saw that, nice. I like Norvig because he really cuts to the chase.
It's not just that they are more popular, Java reigned supreme for years when Ruby and PHP and later Node.js outpaced it's communities by miles. And you can't say Java developers are not open source oriented, because they 100% are.
And the argument that programmers are simply most productive in the language they know best is trivially untrue. When I first learned of Ruby, I could dream in C#, I was absolutely fluent, knew the standard library by heart. I switched to Ruby and only looked back whenever I needed tight performance. And this is not just an anecdote, the whole Ruby on Rails movement was basically Java developers fleeing to greener pastures.
I don't know a single highly experienced multi-lingual developer who does not reach for a dynamic language when they need to deliver something quick and easy.
The only exception I know off is using Go for small networked services, which is quite comfortable and intuitive for a static language. Also outside that niche Go quickly loses its productivity edge.
If you're picking from a popular language, the language itself rarely matters for this at all. Every popular language does roughly the same things. Writing speed (length of keywords, for example) is a non-issue. The ecosystem is by far the most important thing.
It doesn't matter how efficient I am in Go if I'm missing a crucial library that I'll now have to write myself.
JavaScript is a classic example where, even if you are pretty fast at writing your code, you will be dramatically slowed down by: 1) needing to add a new package every 5 min to do something trivial, 2) looking through five half-dead libraries to find one that seems maintained and usable, and 3) finding out that some of the libraries you chose are buggy.
So for a quick prototype, I'd weight a language's qualities as follows:
- ecosystem: 90%
- static analysis/tooling: 5%
- stdlib: 3%
- syntax: 2%
And for a long-term, complex project, I'd weight them closer to:
- ecosystem: 50%
- static analysis/tooling: 46%
- stdlib: 3%
- syntax: 1%
This argument is not addressing the claim for general purpose software, only for "quick and easy" software.
To people in this thread: please stop to think before responding to the "wrong point". I think we can all agree that dynamic, scripting languages are more adequate for "quick and easy" software, but this is not what the point of the discussion is. The point of the discussion is, or should be in my opinion, whether it's true that for general purpose software, written by teams, that is not trivial to write by oneself in 5 minutes, that scripting languages are more productive than statically typed, compiled languages.
I think you misunderstand the scripting language revolution of the mid 2000's. It's not that we suddenly realized scripting languages were the best for quick and easy projects. We realized scripting languages were suitable for a whole lot more than some data processing. We could much more effectively build huge scalable software platforms.
It got so bad that the sentiment flipped, and people like Joel Spolsky had to go out of their way writing blog posts that you could in fact build successful modern web platforms in C#. And then he went building the world's most popular project management tool in Node.js anyway.
Maybe this shines light on your social circles more than actual language impact on proficiency?
The "our language is so much more productive" myth exists in many languages, including the one I currently use. But when you dig deeper, there are almost always other social or technical factors that explain the productivity gap. That kind of gap exists even between people using the same tool.
The reality of it is that assessing what makes a developer productive is incredibly hard, and people doing so to claim their language of choice is better rely on anecdata and couldn't explain what "methodology" means to save their life.
Hi, highly experienced, multi-lingual developer here (Python, Ruby, PHP, JavaScript, Java, Go, C#, F#, OCaml, Elixir/Erlang, ReasonML, SQL, Smalltalk, Objective-C, Swift, leaving off quite a few prior Web 2.0 ones for brevity) nice to meet you. As you can see, I've done just about all of it: every paradigm, every syntax. It all honestly blurs together after a certain point, and it just becomes easier and easier to pick up a new language the more you learn.
Anyways, these days, I never reach for a dynamic language when I need to deliver something quick and easy. The difference is just too marginal. Not worth the downsides (and the downsides are immense!).
Because, as my experience has taught me, invariably one of those quick and easy ones will turn into something that becomes business critical, lives on for years after you are gone, and will be much more difficult for other, typically more junior, developers to update or enhance your code. Static typing is not only marginally less productive these days with all the great tools and IDE's out there (not to mention Go which has one of the least obtrusive static type checkers I've seen, or, even better if the org allows you, a language with a HM type system), but for a marginal improvement in productivity, you pay a heavy long term cost. Not worth it.
Dynamic languages are great for rapid prototypes. After that, convert it to a static language. Your junior devs that join after you, who aren't familiar with the entire ecosystem your work has will thank you.
I don't think you can have a hard and fast rule like this. It will depend on the situation. In many instances, a "prototype" built in Python or similar is perfectly fine.
In addition, one can make a horrendous mess in statically typed languages as well. One of the absolute worst projects I ever worked on was written in a popular statically typed language. This is of course an anecdote and not to say statically typed languages are worse, but they're not automatically better either.
Just a note too that I'm a big fan of, e.g., Rust & TypeScript, and I often use type hints in Python, so I'm not anti-static typing.
There is a floor for how bad you can write static typed code.
There is no floor to how bad of JavaScript or Python or Ruby you can write. The madness can descend to the inner most circles of coding hell. Only Perl exceeds it.
Explained:
Go's type system is generally less rigid. It strikes a good balance of strict enough. A lot of Go's converts aren't from "systems" languages like it targeted originally, but rather former Python/Ruby/PHP/JavaScript backend devs. I love the performance and low level levers I can pull with Go (although to be fair Java is quite fast enough). But finally, Go is easy to learn (26 language keywords?), the standard library is mostly great, and the worst developers I've seen still write mostly maintainable code that builds fast, which is what I optimize the most for these days.
Java, for all its warts and legacy cruft, these days you can write fairly good java, utilizing modern libraries. I love most things from Codahale, who in turn I think pushed orgs like Spring to write better libraries, so now everything's pretty good. Plus all the legacy stuff comes in handy when you have to deal with arcane government or financial systems, something I have to interface with frequently.
But if I had my choice, I'd use something where you can express functional programming concepts intuitively, without fighting the language or having to do it at a heavy performance cost, like Rust or better yet, just a full fledged FP language like OCaml (whom I understand heavily inspired Rust)
When I want to develop further, I put a debugger statement right where I want to pick up, where all the data is available, and develop from execution.
Some static languages have REPLs but few are as good as dynamic languages for mutating existing code in the middle of a debugging session.
If you don't write much code which involves exploration of data or unknown APIs, then this mode of development may not be as useful to you, but it's a significant productivity advantage for me and the reason I reach for a dynamic language. Ruby is my go-to, with binding.pry as my debugger REPL.
Popularity and expansive use may have nothing to do with quality, and a lot to do with financial influence on peoples agency.
Business wants people templating out directories of performant code, not generating syntactic art for the ages.
“Strings and string formatting” is a very broad domain.
For most specific tasks, there is one low-impedance approach and it takes very little (but not zero) reflection and/or experience to find it.
Most new Python features directly address specific tasks for which there are currently multiple relatively high-impedance approaches taken because there is not one obviously correct way.
Python's “one obviously correct way” is not “one possible way” (the latter approach is closer to Go.)
"People who know one language think it's the greatest in the world. People who know more than one think they all suck."
The Blub paradox: http://www.paulgraham.com/avg.html
(BTW, the TL;DR is people that know more than one language admit they all fall short one way or another, but there are some languages that are more productive than others).
This is true, but most software is built by multiple people (for corporations) and will eventually be maintained by multiple other people. Whether an individual is "most productive" in a language is rarely important.
I’m much more productive in Python. Not as in “I feel” more productive, but as in measurably more productive.
Where Python falls short of something like .Net and C# is that I probably wouldn’t have been more productive in Python if I didn’t have 7 years experience with a rather strict environment, and as far as using Python on major projects, well let’s just say that there is a reason people use TypeScript instead of JavaScript and Python doesn’t fall into that category.
But for most programming and for most minor systems or services, Python is just wildly good.
The second point matches my experience with dynamic languages as well. To use them well really takes a certain level of discipline, but it pays off. It's why I'm a bit skeptical of Python as a beginners language, which it sometimes has been touted as.
About large projects, presently I am working with Python on a somewhat large project (currently 53k sloc, I guess large is relative). Its been great, though we are pretty strict about almost absolutely everything being type-hinted.
Maybe not the best example, but I used to work with a guy who had pretty large scripts that he was writing for one of our projects. It turns out it was mostly copy-paste, so it sort of got the job done -- except he fixed bugs in one place but not in the other. The whole thing was a huge mess to understand and maintain. And he was a senior engineer at the time (now principal, to my great amazement).
Why are they writing the same code again instead of writing different code? Why aren't they learning about new APIs for topics they aren't so experienced in? What makes them "senior developers" and not "senior transcriptionists"?
Who is tasked with digging into 10 year old code, written by an ex-employee, to find a subtle bug? Who is identifying and fixing performance issues?
How are these "senior developers" able to write 1KLoC/day plus documentation and developer tests? I find test code and documentation each take as much time as writing the code in the first place.
Why aren't some of these senior developers producing negative LoC? https://www.folklore.org/StoryView.py?story=Negative_2000_Li...
For example, 1M+ LoC of code, including build system, CI, test cases, built-in documentation, in 6 months is about 7kLoC per day, divided by 5 developers it's about 1400LoC per day per developer.
I completed 30+ projects already, I have 20+ years of experience. Few years ago I was able to close up to 20 tickets per two week sprint, when I worked with same level senior developers and dedicated PM, product owner, and QA team, at large outsource company at fixed-price projects.
It's easy when tickets are properly sized and described by PM, when PO is responsive and easy to reach, and when QA covers your back for complex test cases. 6 month project + 1-2 months to recover after, and then another such project.
Those numbers are ridiculous, and I say this having 25+ years of professional experience.
The typical industry numbers are under 100 LoC/day.
Eg, slide 20 of https://www.slideshare.net/ddskier/calculating-the-cost-of-m... ("A world-class developer (e.g. Facebook or Google senior engineer) will write 50 LOC per day")
"Improving Speed and Productivity of Software Development: A Global Survey of Software Developers" at https://uweb.engr.arizona.edu/~ece473/readings/9-Improving%2... (Fig 6) has Lines-of-Code per Total Man Months at about 1,750, so about 81 lines per day (assuming 21.62 work days per month).
"A Practical Approach to Software Metrics" at https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=819938&... says "Many industry rules of thumb describe programmer productivity in terms of lines of code, for example 350 [noncomment source statements] per engineering-month of effort." That's 16 lines per days.
Going the other way, in the COCOMO estimator at https://strs.grc.nasa.gov/repository/forms/cocomo-calculatio... your 1 MLoC for an "organic" project estimates 3390 person months. Not the 30 months you mentioned.
That said, it's really easy to distort LoC measurement. And I mean beyond the "put in 1,000 lines of the form 'a=1'" cheat.
For example, LoC traditionally refers to source code, not test, CI, etc. that you use. Why did you use that non-standard definition?
Test code, for example, may contain a lot of data records and autogenerated code. This doesn't require the same work as a source line of code.
Consider SQLite. https://sqlite.org/testing.html says the library is 143.4 KSLOC, while the test suite is 91911.0 KSLOC - nearly 100 million lines of test code! These tests were not all written by hand. At that ratio of test code to source code you would have about 1,400 lines of source code.
And it's possible to mis-measure source code. A few years ago I added about 400,000 lines of code to my project in a few weeks. These were auto-generated when I replaced a very confusing set of C preprocessor directives with a homebrew template system to compile specialized versions of functions across my parameter space.
I then added some Cython projects, which generates C files about 50x larger than the original pyx files.
Counting auto-generated code makes LoC a worthless measure.
You also wrote you have a QA team. I assume their test cases are in your repo, and counted in your 1M+ LoC count, but you didn't include their time.
The post-release defect rate per KLoC is about 7.47 (average) and 4.3 (median). See An Overview of Software Defect Density: A Scoping Study at ht...
> Are your fixed-price estimates based on LoC?!
I don't now. I was not a part of sales.
> Else, why aren't you embedding that knowledge in a corporate library ("contractors tools") which you license to all your future customers?
Because each customer has different requirements, different goals, different language and framework. Nobody wanted to pay for a library.
> If you write 1KLoC/day then you're likely generating 2 post-release defect bugs per day. That's in addition to bugs your QA team catches.
In practice, I had about 1 serious bug slipped per 1-2 weeks. Professional QA team known a lot of corner cases to test already. Professional developers know them too, even when writing in a different language using different framework. Moreover, we know how to prepare good CI, efficient automated tests, how to write efficient documentation, who is responsible for what, etc.
> Your numbers are so far outside of industry standards and academic findings that, with 20+ years of experience, you must know they are exceptional and difficult for anyone else to accept on your simple say-so.
How much I will be paid if you will accept my story? :-)
> Because each customer has different requirements, different goals, different language and framework
Sure, but that doesn't match your earlier statement that the senior developers 'wrote almost same code dozen times already'.
That is, I don't see how "almost the same code" is able to handle different requirements, etc.
How much would you pay me to accept your story? ;)
> there is a reason people use TypeScript instead of JavaScript and Python doesn’t fall into that category.
I thought the dynamic vs static typing debate was pointless until I joined a large Python project. Now I pretty much consider anyone that can keep up with one superhuman.
Does that mean Python has all the performance you would ever need? No - we know it doesn't, but we also know that you rarely need that kind of performance and when you do it's usually localized to a very narrow portion of your code. Take that portion of code and implement in C and optimize to your heart's content. Even with that you'll still get your project done much faster.
Python is the tool of choice for those developers who just want to get stuff done.
I’m primarily a python user, but I spent time writing code in Ocaml to understand the potential benefits, but… (I would love to have my mind changed)… it feels like many of the features people touted about Ocaml have already made their way into python…?
1. Immutability / referential transparency seem more like nice-to-haves for a codebase, rather than the real reason people tout FP…?
2. Sum/product types and pattern matching are being added soon to python
3. Mypy is starting to gradually enable python typing, although I assume it’s still a work in progress
4. Python allows us to use map/reduce, and to pass functions around as arguments…?
I want to understand the potential benefits of FP more, but my experience with Ocaml hasn’t shown me great improvements yet. Open to having my mind changed.
In python, most of these "features" are done half right and duct-taped on at the last second as an afterthought.
Comparing Python to Java, C#, Haskell, Erlang, Go, maybe even C or Rust, would show a much smaller performance difference.
Have you tried RAII?
Would you like to know more? https://stackoverflow.com/questions/2321511/what-is-meant-by...
Rust is a little better since at least the ownership concept is known to the compiler and automatically enforced.
(Tracing/Copying/Compacting) GC is much easier since this entire concept goes away. You always pass references to data, and the physical storage is "owned" by the GC itself. It's also much faster for certain workflow patterns, though it always consumes more memory than deterministic destruction schemes.
Not if you don't go looking for it: http://www.norvig.com/java-lisp.html
ETA: And don't get me started on Design Patterns: https://norvig.com/design-patterns/design-patterns.pdf
If one of these steps requires a change of programming language, I'm never too lazy to recode :)
And by "make it fast, make it work", I mean benchmark first. Kind of like test first, you write the benchmark before the implementation and constantly look at how fast things runs, and ensues every single bit you add is fast.
Want to build an I/O utility writing to a DB? Sure C can do it, but Python is better suited. Want to write a toy compiler? You don't want to waste your time trying to wrangle on CPython extensions. C works out of the box.
> _'if you select a programming language based on your preconceived notions of how a language performs, you will never know if the language that might be a better, more productive fit'_
Part of the CS education is not about recognizing homeruns but understanding trade-offs. The experience gain is about learning how tools work & which tools to choose to work in tandem. Modern systems use a variety of languages - JS in the webpage, SQL DBMS for queries, C++ to run the performance bits, Python for ML, introperations - maybe even Rust in the security bits of late. In that sense, the title was unfortunately misleading to me, since author tried to demonstrate a lot of usecases with Python.
Python is great - but there has to be a reason why other languages co-exist. Not just for bankers, military or some enthusiastic hobbyist.
And the interoperability is not as awesome as it's painted in the article, it's always more pain to have more languages in a project that need to talk together
Julia one of the target is solving the "two-language problem"
"Julia seemed to have solved the “two-language problem”—a conundrum often facing Python programmers, as well as users of other expressive, interpreted languages. You write a program to solve a problem in Python, enjoying its pleasant syntax and interactivity. The program works on a test version of your problem, but when you try to scale it up to something more realistic, it’s too slow. This is not your fault. Python is inherently slow—something that doesn’t matter for some types of applications but does matter for your big simulation. After applying various techniques to speed it up but only realizing modest gains, you finally resort to rewriting the most time-consuming parts of the calculation in C (most commonly). Now it’s fast enough, but now you also need to maintain code in both languages, hence the two-language problem."
https://arstechnica.com/science/2020/10/the-unreasonable-eff...
> Clearly, multiple dispatch, or some other way around the expression problem, is necessary for the kind of fluent composability that I’ve described above—but it is not sufficient. Julia has enjoyed an explosive degree of uptake in the scientific community because it combines this feature with several others that make it very attractive to numericists.
That's incredibly handwavy. So what's the special sauce?
There is no such thing as a free lunch when it comes to dynamic vs. static. It also seems like Julia is trading off expressiveness and easy of use in favor of efficiency, based on comments from people that have used Julia. It's one thing to be faster than any inherently slow language (Ruby, Python, Smalltalk, etc.), but keeping that flexibility and being as fast as C/C++ is a rather bold claim. Most languages hit some middle ground between the two, such as Java. But no one is under the delusion that trade-offs weren't made to get there.
1. You can't add fields to a type (struct) after definition. This means that Julia's structs have no overhead and are essentially equivalent to structs in C (although they are parametric)
2. No local eval. Eval in Julia only happens in the global scope and results of eval are only visible the next time you visit the global scope. This may sound kind of unintuitive, but in practice people don't generally use this for good reasons. This allows Julia to never need to de-optimize code. Once a method is compiled that code remains valid.
3. Macros. Julia has really good macros and other code manipulation (since it is basically a Lisp). This makes it possible to generate very complicated but fast code that you would never write yourself. The tradeoff here is that it makes the language more complex, but that's a pretty good tradeoff. (especially compared to the C/Fortran land of using a preprocessor that works on text).
4. Just-In-Time (just ahead of time). Julia at it's core runs as if it were highly templated C++ code. If everything got compiled ahead of time, Julia would be generating terabytes of compiled code and never finish compiling. Instead, Julia makes the tradeoff of only compiling for the argument types that are actually used in the program, which means that it only compiles a reasonable amount of code. The tradeoff here is that compiling small binaries with Julia is very difficult (not possible to do automatically yet).
The TLDR is that most expressive languages started by giving away as much expressiveness as possible, and then looked at how they could be sped up. Julia started by being a modern fast language and looked to see how much expressiveness could be added without slowing the language down.
A huge part of the design considerations for Julia essentially boiled down to "what sorts of dynamism and language semantics can we disallow while keeping the the good parts of dynamism"
The two biggest things that had to go in order to make Julia fast was
1) the ability to change the memory layout of a struct in a running session
2) the ability to eval in the local scope (our eval always occurs in the global scope)
These two things are huge performance problems. We might oneday solve 1) with Revise.jl (though it'll mean recompiling all your code if you do change the layout) but 2) is basically just a very bad idea and likely to never happen. Instead of a locally scoped eval, we have macros, multiple dispatch, parametric types, and generated functions. These give an incredibly powerful suite of metaprogramming tools that are beyond anything available in Python.
One of the key things in CL is that it has its metaobject protocol which forces a lot of decisions on what gets executed to runtime. There are ways to speed it up, but if you have something like:
Then CL won't call foo specialized on number when given an integer, but will call foo :before specialized on number. It determines this at runtime by searching for all applicable methods based on the type (at least as a first pass, you can cache this to speed it up but then you also have to have cache invalidation if a definition is changed).Julia doesn't have that aspect of CL's MOP. So this helps to simplify the search for applicable methods and dispatch. Even if it did all its dispatch at runtime, it would still be simpler. The other thing Julia does is aggressive JIT compilation. So if you wrote something like (with the Julia equivalent of foo from above):
And, only considering floats and integers, later called it with each pair of float and integer then Julia would compile specialized versions for those 4 combinations. Now when you call bar it still has to properly dispatch it, but once inside bar the search for the correct foo can be bypassed because the types will be known. CL, again thanks to the MOP, doesn't make that as easy to achieve.- multiple dispatch
- parametricity
- lightweight subtyping
- staged programming
In interesting ways.
felleisen's class talks (partially) about it here: https://felleisen.org/matthias/4400-s20/lecture15.html
I guess the best way to put it is that Julia encourages a style where 90%+ of code can go through paths that are static.
Personally, I think Julia starts off as easy as python, but to get C++ or Fortran speed, you can't just code naively. Things go into a steep learning curve at that point, but perhaps there isn't yet as much know how about how to code "professional Julia" yet. There needs to be a book like Fluent Python or Effective C++ for Julia, or perhaps a condensed version of the Julia manual (see the 1 page zig manual for inspiration).
The other problem I have with Julia right now is lack of static type checkers. "modern python" (e.g, python in production in the last 5 years) tends to leverage the large ecosystem of things that hook into mypy (I'm taking about tools like pydantic) to reduce the inherent brittleness of the language. Ruby, php, and every other dynamic language has also seen that trend.
Right now, I've barely seen that with Julia, and it needs this badly for higher uptake in industry. It's why for example, perhaps you see a lot of Julia packages written for people's phds right now.
http://ucidatascienceinitiative.github.io/IntroToJulia/Html/...
As for expressiveness, this then leads to different programming styles which I explained in a blog post:
https://www.stochasticlifestyle.com/type-dispatch-design-pos...
Also the author: Pick python no matter the problem, the team, the libraries, the deployment targets, etc.
I read it as the much more milder "don't reject Python because of vague concerns about run-time performance".
I can't find anywhere which suggests the author things people should use Python to, for example, code up their web app front-ends or to implement a 'hard' real-time operating system.
Looked to me like a "Python isn't than bad for perf" article.
- Nobody
Our codebase is well over 1M lines of C++. We have about 100k lines of python. Running pylint takes the same order of magnitude of time (half as much IIRC) as running a full optimizing build + linking of the C++ code base. We run pylint over all cores, while we run the C++ build only on a subset of cores on the build machine.
I would call that dog slow, unless you think python is not the appropriate language write pylint. And no, it is not fine.
>Prototype in Python.
Stopped reading here.
Wait, what? The only thing I know about python is that it's indentation-sensitive, no idea about syntax or libraries. Suggesting me to use python is premature optimisation.
You could just as easily say "Descriptive variable names are a form of premature optimisation. Code should be written with single-letter names, and descriptive names only added when absolutely necessary."
To be honest, today there are many viable solutions with different languages. I would not recommend to start with a plan where you have to reimplement the system in another language at some point. Experience tells me that almost none of such projects survive.
Moreover, the availability of "average" (and "cheap") programmers matters a lot in the long run: if only genius can maintain your system, then you'll have problem in the long run because you'll either need to keep them at all price or need a lot of time to replace them. So, in the long run, you should better use a wide audience language with a lot of available programmer (even if they are "average") than a specific language requiring good programmers. However, for an MVP, you can recruit a genius programmer using the fastest tool for the job.
Obviously, some domain are more oriented toward some language... and for ML for example, python is quite a good choice because of the libs (as Java could be for - lets say web servers)
So it matters a lot what your system will be used for and how much time you will require it to run before needing to rewrite it from scratch
Sure, features are important and if critical ones are missing it might be a show stopper. So there is an initial thresshold that all candidates must pass.
But problem solving is not a one-off exercise. It tends to be both dynamic (=facing unpredictable challenges) and recurring over long time horizons. Which means having a healthy, engaged, resourced community that will invest in adapting / solving future requirements is essential.
So the "optimization" problem includes quite a bit more than the presently known developer team, its software stack its hardware and current problem definition / user requirement.
I think you see this dynamic in several cases (including python) where you might not think that it makes rational sense.
Just because you can program a new feature quickly doesn't mean it is engineered well for the long-term & larger scales.