The data for the paper the article talks about is online [1], and it says [2] that the benchmarks are "implemented in 28 different languages (exactly as taken from the Computer Language Benchmark Game)".
The LISP evaluated by the Benchmark Game - and, apparently, this paper - is indeed SBCL [3], which seems about on par with Java.
“Lisp, on average, consumes 2.27x more energy (131.34J) than C, while taking 2.44x more time to execute (4926.99ms), and 1.92x more memory (126.64Mb) needed when compared to Pascal.”
I think this is what is so cool about Common Lisp. You can literally get pretty darn close to C in performance and still be at the highest levels of development efficiency/prototyping speed. I don't think too many other languages can say so. Take Python. It has really fast development time, but very slow performance.
C and Haskell were about what I expected, but those Pascal results have me scratching my head, given that it performed about as well as C and used even less RAM.
Pascal and C are roughly equivalent in my eyes, at least as far as what they spit out in the end. Both have pretty similar feature sets and development methodologies.
What surprised me is how much of a hit the OO programs took. I had thought they would compile down to something reasonably similar to the imperative languages, but C++ ended up being 50% slower.
Interesting results, but I think the farther you go down the list the more the fact that you're using the Computer Language Benchmark Game affects what you're seeing, as not all languages get the same attention.
For example, Javascript and Typescript. Theoretically, I would assume those to be very close, since one compiles to the other. And for memory usage, they are very close. But for running time, Typescript is an order of magnitude slower. That looks suspiciously like entirely different algorithms were used in each implementation (with one being obviously superior in terms of performance), and if that's happening in this specific case, where else is it also causing problems in the analysis?
submit a better implementation. it is not possible for any one entity to produce optimal implementations for many languages, it is up to us. or, if you dont like their rules, make your own benchmark system with new rules.
I don't use Typescript, and use Javascript only for what I have to. I don't care about the ranking of either of them, I'm just pointing out a possible flaw in the methodology that should be taken into account when looking at the numbers presented, and what I suspect is a concrete example of that.
This also isn't a criticism of the benchmark game, it's well known that not every implementation is equivalent in the time and effort put into optimizing it. It serves its purpose about as well as can be expected. Unfortunately, using it as the base of further calculations can lead to some of the known quirks of the benchmarks being exaggerated into results that are not always obviously an artifact of the underlying system, as I suspect this is. Making that obvious by pointing it out can be useful.
I agree. I just wanted to point it out in case people took the analysis as entirely accurate without looking a little closer. I suspect the entries towards the top of the list are fairly accurate, by nature of their competitive standing in the benchmark games and their focus as languages on performance. It's much easier for an implementation difference to hide in the natural drift you see in the languages with VMs and interpreters that don't receive as much attention.
That's actually one of the reasons why the TypeScript/Javascript divide jumped out at me. They were mentioning that the bottom of the list was dominated by interpreted languages, and mentioned TypeScript by name (which surprised my given the focus JavaScript VMs have gotten), and then when I reviewed JavaScript's standing (which was more in-line with what I expected), I noticed the difference between it and TypeScipt was very pronounced, which is odd when (to my knowledge) TypseScript compiles to JavaScript, and not because it's doing a lot of convenience stuff that would slow it down. That said, I don't use TypeScript, so maybe I'm overlooking something.
It took me about a minute looking at their published detailed data to notice the problem. They should have noticed a factor-of-15ish outlier and checked why on earth it was there.
It's not really about how the game works, it's whether the study is using sane data. In this case, it very much isn't and it should have been obvious to them the data is bad.
If we assume they understood the relationship between JavaScript and TypeScript then maybe it should have been noticed.
However, the original research has been posted multiple times to proggit and HN since 2017; and I don't recall whether or not anyone noticed this problem until now --
It's a study presented at some conference so while not exactly the Higgs boson, they're showing other people data and the conclusions they derived from it. It's 100% their job to understand what their data measures and to notice that one of the measurements is completely bogus for their purposes. The fact other people hadn't necessarily noticed on messageboards before is mildly curious but it's not really their job.
afaict the evidence is - not - that "one of the measurements is completely bogus".
On the contrary; we can see from archived web pages that other measurements showed the same relatively-poor performance, with those old versions of TypeScript.
What do the archived pages have to do with this study? The study starts with a snapshot of benchmark game sources. That's a perfectly sensible way to get a bunch of implementations to bootstrap the study. But some of those implementations might be unsuitable for the study, just as (at least) one was, in their case. They don't seem to have noticed that. What's a good, benign explanation that they didn't?
Because it's a different implementation that happens to be 15 times slower than the implementation used in the straight JS version. That's fine for the benchmark game, it's a garbage input to a 'how energy efficient are these languages' study.
Let's say I want to measure the 'energy efficiency' of x86 assembly and JS. I'll use sorting an array of 1000 integers. In my JS implementation, I call Array.sort. In my x86 implementation, I randomly shuffle the array and check if it's sorted, if not repeat until it is. Does measuring the execution times of these tell me anything about the 'energy efficiency' of Javascript vs x86 assembly?
> a different implementation that happens to be 15 times slower
A few TypeScript versions later, that happens to be only 1.6 times slower.
There are good questions to ask about how to handle possible outliers in a study that takes a snapshot of a changing situation and then seeks to make more general claims.
It doesn't matter what happened in the benchmark game later, it's not a study about the benchmark game. It's not 'good questions', it's a huge fuckup by the authors of the study. You're simply wrong to keep saying otherwise.
> Unfortunately, using it as the base of further calculations can lead to some of the known quirks of the benchmarks being exaggerated
You seem not to have considered the possibility that the authors may have simply made a mistake, unrelated to the origin of the programs.
The authors presented at an Oct 2017 conference. Archived benchmarks game web-pages from 2017 do not show the 10x fannkuch-redux differences that the authors report --
> Archived benchmarks game web-pages from 2017 do not show the 10x fannkuch-redux differences that the authors report
I think the Sep 1st 2017 benchmark does, though.[1] At that point it's 1,204.93 seconds, compared to the 131.39 seconds on September 18th you referenced. That makes sense, since the paper could have been finished quite a bit prior to the conference.
That’s an interesting discrepancy, I might have had the same gut reaction. But you’re using that assumption to cast slippery-slope doubt on the whole project without knowing anything specific. It’s possible you’re right, but maybe find out which algorithms are being used first? Perhaps there is a reason that TypeScript actually is an order of magnitude slower than JavaScript. The Benchmark Game project was specifically set up to allow people to scrutinize the algorithms used and make fair comparisons, so it might be unwise to start with the assumption that the most basic aspect of the project failed spectacularly and that something really obviously stupid and easy to fix is happening. I am at least giving benefit of the doubt and wondering what might be wrong with TypeScript. It’s not implausible since other languages take just as long.
You can write literally any JavaScript program with TypeScript (falling back to the `any` type if you really need to), so this doesn't really work in this case.
Also, typical TypeScript programs are faster than typical JavaScript programs, because JavaScript JITs like predictable object shapes and monomorphic functions for the same reasons that other language implementations require them. Libraries like lodash and bluebird take advantage of this fact without using TypeScript, but TypeScript steers you towards these patterns.
>typical TypeScript programs are faster than typical JavaScript programs
Person who works on js engines here :)
You would think this is the case, but in actuality, js and ts are about on par in performance (assuming similarly written code). This is in fact due to the fact engines optimize for idiomatic js patterns, not idiomatic typescript patterns. often these will align, but in some cases (usually revolving around generics and inheritance) well written js will actually fall through the optimization pipeline faster due to following more patterns that have specific optimization checks.
Any articles/resources on writing high-performance JS that you would recommend? I don't use JS for work, but just for random fun stuff. So just curious to learn more about it.
Check out gl-matrix.js, that’s one designed for being fast. It’s a simple math library, but the main way it achieves fast is by not allocating memory.
After having worked on JS for some large web apps that need good graphics performance, the two rules of thumb in my head for making JS fast are: 1- avoid dynamic memory allocation, and 2- avoid using the functional primitives like map.
The first one is more or less true in all languages, memory allocation always costs a lot, and if you can pre-allocate memory and/or re-use memory along the way, the code will run faster. This means paying attention to what things in JavaScript will allocate memory under the hood, use of dicts, use of 3rd party library and framework functions, etc.
The second one is a bummer; I love using the functional primitives. But map is slower than a for loop, all else being equal. It has gotten relatively faster over time. I mostly use functional everywhere, and only resort to for loops in performance critical code.
> 2- avoid using the functional primitives like map
I understand that map requires creating a new array, and that is already included in point 1. What overhead are functional primitives subject to apart from memory allocation? e.g. forEach
Using continuations. The function call itself and the local scope wrapped up into it can be varying degrees of expensive. When little state, because it’s inline or nearby for example, it might be optimized down to near what a for loop does. But if there’s a function call at all once it’s executed, that alone is slower than the for loop. Remember it’s a function call per element. If the function is further away with more scope, it can be more expensive both memory wise and time wise.
BTW, it's easy to test the basic primitives. I use Chrome snippets.
test = (name, fn) => {
const timeLimitMs = 1000
let start = Date.now(), count = 0
while (Date.now() - start < timeLimitMs) { fn(); count++ }
console.log(name, count)
}
var N = 1000000
let a = new Array(N)
test('for loop', _=> { for (var i = 0; i < N; i++) a[i] = i })
test('map', _=> { a = a.map((x,i) => i) })
test('forEach', _=> { a.forEach((v,i,a) => a[i] = i) })
for loop 941
map 37
forEach 69
This is on my Mac in Chrome. So forEach is faster than map, but for loop is more than 10x faster than forEach. That's for loops with trivial work, of course. If the inside of the loop is expensive, the loop/map ratio will be lower.
If I modify that code to pre-initialise an array with the values 0 to N - 1, and then copy the value from that array to a new array (rather than using the loop index), then both map and forEach are faster than the for loop for me.
That sounds fairly surprising, considering map has to allocate, and allocate is very expensive, but please share your code & I’ll try it.
Like so?
const N = 1000000
let a = new Array(N), b = new Array(N)
for (let i = 0; i < N; i++) a[i] = i
test('copy loop', _=> { for (var i = 0; i < N; i++) b[i] = a[i] })
test('copy map', _=> { b = a.map((x,i) => a[i]) })
test('copy forEach', _=> { a.forEach((v,i,a) => b[i] = a[i]) })
I get: copy loop 973, copy map 38, copy forEach 49. Same as before, but this time I tried Chrome in Windows.
Are you using a different browser? I know that sometimes other browsers have very different results.
In any case, it's somewhat irrelevant if there are cases that optimize and cases that don't. When idiomatic functional code is sometimes up to 30x slower than a for loop, it can't be used in perf critical sections. Even if it's only Chrome and only certain cases. The forEach perf needs to be always reliably performant before I can use it without worry.
My best guess: probably written by someone that doeant know how to write performance TS. I say this with no skin in the game. I write neither JS nor TS. What I have observed in language benchmarks over the years, is that the benchmarks are rarely written by an expert, but usually by someone with cursory knowledge of the language. E.g. just enough to be dangerous.
Often times, these sorts of benchmarks are done with prejudice (not necessarily malice). The benchmarks are written by someone with something to prove: my chosen tech stack performs better, and let me show you why. A favorite of mine is Perl vs Python comparisons, where you see an idiomatic Perl implementation vs a non idiomatic Python implementation (or other way around). Typically in a head-to-head comparison, the benchmarks are developed by the same individual whom likely has above average knowledge in their favorite and below average in the target they're trying to show as inferior.
You'll see this time and time again in internet benchmarks comparing performance. Unless you can see the code from all benchmarks involved, my suggestion is to avoid them. I mean, for all I know, the author of the benchmark was unaware of the built in sort and instead bubble sorted.
I thought the benchmark game is set up so every language's advocates can tune their language's programs. The only chance at a fair comparison is if every language gets the best implementation it can find for the challenges. There's nothing else that approaches fair comparison of apples and oranges.
This is unfortunately pure speculation on top of pure speculation, which is the problem I have with the top comment. You’re assuming incompetence when you could just go look it up. Why assume it’s someone who doesn’t know? Why use that to wander off into rant land about prejudices and make broad claims that internet benchmarks are bad, when you admit to having zero idea what the actual specific problem here is?
The test that lowered TypeScript’s score in the paper is called fannkuch-redux, and here are the sources in question:
They are both contributed by the same person, and there is no bubble sort involved. So now you know.
I don’t see an obvious reason one would be slower, but they’re also quite different. Maybe the algorithmic complexity is different. Maybe the cross-compilation is doing something bad with memory allocation. Note the input sizes for this test are very small, it would be easy for a difference in temporary variables the compiler injects to cause a serious problem.
What is not obvious is any prejudice, malice, or incompetence.
> Why use that to wander off into rant land about prejudices and make broad claims that internet benchmarks are bad
What are you talking about? Where did that happen?
> when you admit to having zero idea what the actual specific problem here is?
This is the comment section for a submission about an article referencing the paper. I brought it up for discussion. It is perfectly valid to bring up a question that you don't know the answer to.
> What is not obvious is any prejudice, malice, or incompetence.
Please stop.
Edit: From another comment, and some deeper digging of my own from that, you might find the archived results of the fannkuch-redux interesting. From 2017-08-01[1] to 2017-09-18[2], the benchmark changed from a running time of 1,204.93 second to a running time of 131.39 seconds. The paper was released in October 2017.
> What are you talking about? Where did that happen?
I was responding directly to @hermitdev. Did you get your threads crossed? What I'm talking about happened immediately above in the parent comment, beginning with "Often times, these sorts of benchmarks are done with prejudice" https://news.ycombinator.com/item?id=19527057
"You'll see this time and time again in internet benchmarks comparing performance."
> It is perfectly valid to bring up a question that you don't know the answer to.
I agree. It's a bummer that's not really what happened here.
>> What is not obvious is any prejudice, malice, or incompetence.
> Please stop.
The parent comment explicitly stated an assumption of both incompetence and prejudice and I responded directly to that.
From the HN guidelines: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."
If you'd like me not to call out speculation, then please assume good faith and don't speculate next time.
> From 2017-08-01[1] to 2017-09-18[2], the benchmark changed from a running time of 1,204.93 second to a running time of 131.39 seconds.
Yes! Now we are getting somewhere. It appears that would change the outcome of the paper. Perhaps it was a mistake. That might mean it was nothing more than an oversight that already got fixed. It doesn't mean there is any other coloring of the study at all, nor that there was any intention or agenda to make TypeScript look bad, right?
> The parent comment explicitly stated an assumption of both incompetence and prejudice and I responded directly to that.
Perhaps I misinterpreted what you said. You started the paragraph referring to the top level comment, which is me. I took the "you're" in "You’re assuming incompetence when you could just go look it up." to be a general "you", and commentary on my original comment.
> From the HN guidelines: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."
I actually looked this up before the GP comment, and almost included it myself. I can see now that you were implicating the comment you replied to. I didn't think that was the case, because I apparently didn't interpret that comment remotely in the same way you did.
> Perhaps it was a mistake. That might mean it was nothing more than an oversight that already got fixed. It doesn't mean there is any other coloring of the study at all, nor that there was any intention or agenda to make TypeScript look bad, right?
I never implied it was. For that matter, I didn't really interpret the comment in question as stating that either. The more charitable interpretation is not that they are trying to make another language look bad, but that they are trying to make their favorite language look good. That doesn't require purposefully tanking one benchmark, it just requires them to be much better versed in optimizing one language than another and a lack of awareness about this. As they say, never attribute to malice what can be explained by incompetence. In fact, if you read the comment carefully, they even call out to this with the "not necessarily malice" remark.
> The more charitable interpretation is not that they are trying to make another language look bad, but that they are trying to make their favorite language look good.
The project doesn’t talk about favorites or seem to want to make certain languages look good. Jumping to the conclusion that bias is involved isn’t the good faith interpretation, even if you state with a positive sounding framing. The good faith interpretation is to take the stated project goals at face value, and assume that the participants have done a good job.
> The project doesn’t talk about favorites or seem to want to make certain languages look good. Jumping to the conclusion that bias is involved isn’t the good faith interpretation
I didn't see anywhere that the comment in question called any project bias into question, but instead noted that in a situation where work is crowd sourced, people with their own intentions and motivations will put out bad benchmarks, either in the case of the benchmarks game, or a specific benchmark or comparison put forth in an article or blog. I've personally been witness to the latter multiple times just from HN submissions.
I just want to end with, as someone that's brought up viewing comments in an uncharitable light, you seem to have done a lot of that in this discussion. You've repeatedly taken your interpretation of a comment, rephrased it in a harsher way, and the stated it as what the other person was saying as fact, and then responded to that. I would think actually trying to find a charitable interpretation should at least include a question at the beginning to confirm whether what you think is being said is entirely correct. Note that I started with that when I thought you were attributing statements to me that I did not say. My first words were a solicitation "What are you talking about? Where did that happen?" to confirm what was going on. You've been doing this from your first response to my top level commend, when you stated "But you’re using that assumption to cast slippery-slope doubt on the whole project without knowing anything specific." That's a very uncharitable rephrasing of what you think I was doing, and it certainly wasn't my intention. I've already outlines in specific exactly what I was trying to do and why, and in doing so I also stated that I felt you were misinterpreting me. There's a clear trend here as I see it, and you repeatedly bringing up good faith assumptions just puts it into clear highlight.
I think we've covered about all there is to say on this (these) topics. I'll let you have to the last word if you wish. I'll read and promise to consider any points you raise, but I don't think me responding would be very fruitful, and this discussion has digressed far enough.
> Do you agree that those very different times were measurements of the same TypeScript fannkuch-redux program?
Yes.
> How should we now assess your "suspiciously like entirely different algorithms were used in each implementation" comment?
The suspicion was incorrect. That's why it was presented as a suspicion, not as fact. I have no reason to defend it if it's incorrect, but I still defend that it was valid to raise questions, given the facts on the ground. We've now shown there was something that changed very drastically at that time, and while it's less likely it's the benchmarks themselves (unless one or both of those are fairly out of date Node versions)[1], it still points towards something to be aware of in the results presented. Namely, they rely on a lot of underlying assumptions which should be looked at if you care about the numbers.
1: Also, I imagine the V8 devs probably considered the performance of TypeScript in that case to be a bug, given how horrible the performance regression from JavaScript is and that it's still javaScript running. It's possible that TypeScript was doing something really odd, but given the exposure and Microsoft's backing and developer time, I think that's a less likely scenario than some optimization that should have been triggered was missing, which happens quite often.
Please add a correction to your original comment, to prevent readers from being misled. (If it's closed to edits, I'm sure HN staff will open it when you ask).
> I still defend that it was valid to raise questions
Of course, it's valid to question a measurement that looks strange but your comment went further than that -- your comment, without evidence, assumed a cause; and, without evidence, implied that assumed cause led to widespread problems with the analysis.
> Please add a correction to your original comment, to prevent readers from being misled.
Corrections are for facts. I put forth a theory. People being misled by a theory are not something I have limited power to affect. People representing theories read on the internet as fact have larger problems that that will solve.
This discussion is the correction, and a better one than someone would be willing to read. Were it within the 2 hour edit window, I would through in an edit, I've done so numerous times in the past. I will ask Hn to amend it's rules so I can correct a statement I made about something I suspected.
> Of course, it's valid to question a measurement that looks strange but your comment went further than that -- your comment, without evidence, assumed a cause
This is incorrect. I had evidence, I had numbers that did not line up with my understanding of how things should have been given my knowledge of the subject. I presented that as a theory, by using the word "suspect". All I implied is that if that theory was correct, which I made sure to not assert as fact, then it might affect some other languages. I did not assume a cause, I assumed a possible cause, and presented it as such.
I am very particular with my language. I try not to state things as fact when they are not. I try my absolute hardest (and I believe I succeed) to always speak in good faith, where I'm trying to raise a point I think is worthwhile or ask a question where I think there is benefit. I'm actually rather bothered by how some people interpreted my words and intentions, and that includes you. I'm bothered by how you've interpreted my words. Since you're not the only one (although I do believe you're in the minority), I'll assume there's something I could have done better to represent my point. I don't think all the blame lays with me though. There should be some way for me to posit a question and advance a theory without people assuming bad faith, so my question to you is, what way is that? How could I have expressed concern over the results without triggering that interpretation from you? Because I don't think doing personal research on a problem is an acceptable prerequisite for raising a question. In this case, I could have spent hours looking into something I was unfamiliar with and come away with more answers, but many people may not have the knowledge to do so but have enough to think something is wrong. Should they just keep their mouths shut? Are we in a time where raising a concern that turns out to be unfounded (or in this case, just more complicated and slightly misdirected) is unacceptable under any circumstance? I refuse to accept that.
> you’re using that assumption to cast slippery-slope doubt on the whole project without knowing anything specific.
No! I think this is a very useful project and analysis. I just think that some languages might have extremely optimized versions (or possibly more likely, some languages don't quite yet have that extremely optimized version that has propagated throughout the others) and that might be affecting specific languages in the analysis.
I think the first 5-10 entries are probably very accurate, as they are generally with very performance centric and often optimized for performance languages. As the languages and VMs/interpreters do more and are optimized less, it's much easier to miss a performance difference caused by an benchmark submission and attribute it to an inherent aspect of the language.
> I am at least giving benefit of the doubt and wondering what might be wrong with TypeScript.
I did no such thing. Note how I used the phrases "Theoretically, I would assume" and "That looks suspiciously like". I simply raised an issue of concern, in a way where it was obvious that I did not know if my concern was correct, and wondered that if it was, what else it might affect.
> I did not know if my concern was correct, and wondered if it was, what else it might affect.
That’s exactly what I mean by casting doubt. If the concern might not be correct, why lead into speculation about further concerns?
Let’s find out what the actual reason that TypeScript is measured slower, rather than guess as what else could be wrong if unverified theoretical assumptions might be correct.
> rather than guess as what else could be wrong if unverified theoretical assumptions might be correct.
You mean, I should not have explored the ramifications of what my suspicions might mean to so people might think it's worth actually looking into? What's wrong with that?
I can't help but feel that you feel compelled to defend your original position that I believe was based on a misinterpreting my point and intention.
I saw what I believed might be a problem. I noted it. I noted why. I noted what it might mean to the analysis because if my suspicion as to the reason was correct, it might not be isolated and other items might need a closer look. I did so in a way where I was sure not to claim something as factual when I wasn't certain. What part of that do you think is an inaccurate assessment of what I did or was unwarranted?
Edit: Changed misconstruing to misinterpreting, as that's what I was trying to express, and misconstruing might be interpreted as a purposeful action, which is not what I was trying to express.
You saw what you believed might be a problem. You did not investigate whether or not it was a problem. Instead you speculated about what it might mean to the analysis.
The part that was unwarranted was the negative speculation.
You seem not to have considered the possibility that the authors may have simply made a mistake, unrelated to the origin of the programs.
The authors presented at an Oct 2017 conference. Archived benchmarks game web-pages from 2017 do not show the 10x fannkuch-redux differences that the authors report --
Many benchmarks have no TS implementations, TS/JS results are about the same except for fannkuch-redux which is about a zillion times slower in the TS implementation. When looking at that kind of massive discrepancy in similar languages with the same runtime, the guess kbenson made was a perfectly sensible one. The authors of the study should have examined that kind of crazy outlier more closely and it's, as pointed out, not that unusual when using the benchmark game as a starting point.
I’ve noticed similar discrepancies between Perl performance in the real world and in the suite they are using.
The issue is that one of the metrics in the suite is lines of code, so people write fantastically obscure and concise functional programs in Perl when the imperative one would be 2x the LOC, but much, much faster.
(This is from a spot check years ago. Maybe they’ve fixed this somehow).
Javascript in many cases runs on v8 which is a just in time compiler. Just in time compilers provide substantial performance benefits for lines of code executed multiple times. Perhaps v8 team (google) doesn't support typescript (microsoft). https://softwareengineering.stackexchange.com/questions/2754...
I'd be very interested in seeing a few energy benchmarks not here, for example: energy to route 1B hits to a web endpoint, energy to parse a Json, energy to grab an auth token from a web request and forward to an auth server. Energy to forward 1M 1k files from the filesystem to the network, etc.
Well, I think the reason they don't do that, and focused on the benchmark game (which I think is a good solution to their problem of needing representative sample, it just comes with its own caveats), is because much of that is often handled by libraries (possible written in C), code in the Interpreter or VM (also possibly written in C), or handled by the OS (in the case of connection handling and file access). Those are useful things to know, but don't inherently speak to the features of the language itself.
Sure but even a language neutral ballpark could be useful to compare relative sizes. Is network or disk energy used insignificant wrt CPU or ram ? Is it roughly equivalent ?
The language and the runtime are basically one package. I don’t care that node is fast because it is all C under the hood. To the end user, it’s just fast.
> Theoretically, I would assume those to be very close, since one compiles to the other.
I don't think that's a safe assumption. The Javascript results would be a lower bound (assuming the same algorithm, etc.), but there's no telling what "extra" Javascript (and thus overhead) might be inserted by the Typescript compiler.
As far as individual results go, there's no point jumping to conclusions when the results and code are available online and easy to try for yourself:
Typescript is still 2-3x slower than Node in several of the benchmarks, and the results could have been different in May 2018 when the article was written.
This may be a dumb question as I know nothing about Typescript, but is it idiomatic to do that, though? You can, for the most part, use C in C++. But if C was beating the pants off of C++ in benchmarks it wouldn't tell me anything useful if someone submitted a C program as the C++ benchmark. If I wanted to use idiomatic C, I'd use C. I expect a C++ vs. C comparison to compare the encouraged features of C++ with what's available in C.
Idiomatic TypeScript and idiomatic JavaScript are nearly the same; it's not like the C vs. C++ situation. A design goal of TypeScript is to make it possible to define types for idiomatic JavaScript programs, and I think they meet that goal pretty well. It's one reason the type system is much more advanced and flexible than, say, Java. You might see little differences here and there due to limitations in typechecking and the enum feature, but fundamentally TypeScript is just about adding structure and safety to your code rather than changing the way you code.
The benchmark is flawed. It should compare the fastest possible C++ implementation and a more idiomatic C++ implementation. The entire purpose of the language is to allow both to coexist in the same code-base.
I would be very surprised if the fastest C++ and C are actually different.
> but there's no telling what "extra" Javascript (and thus overhead) might be inserted by the Typescript compiler.
That's true, but the fact we're talking about an order of magnitude running time difference is what lead me to think it's not as easily explained as that.
> there's no point jumping to conclusions
Respectfully, I don't think I did. I put out a theory, with my reasoning, fully acknowledging where it might be wrong. As someone that doesn't use TypeScript (and I wouldn't consider myself in expert in JavaScript either), I don't feel qualified to assess the respective algorithms for those languages in the benchmarks game, so I brought the issue to a larger audience so someone else might take a look if so inclined.
> Typescript is still 2-3x slower than Node in several of the benchmarks, and the results could have been different in May 2018 when the article was written.
That doesn't really invalidate my theory, as it's possible those are also instances where it's lagging in a good submission. Given that there are several tests where it actually beats the fastest JavaScript algorthim submitted, that seems at least plausible.
> there's no telling what "extra" Javascript (and thus overhead) might be inserted by the Typescript compiler
TypeScript doesn't do this. With only a few exceptions, the way to turn TypeScript into JavaScript is to remove the type annotations, which leaves you with JavaScript. This is an important design goal of TypeScript: it doesn't change your code, it's just JavaScript with types. Like Babel, TypeScript can be used to compile to earlier JS versions like ES5, but I'd consider that a misconfiguration in an environment like this.
Saying that TypeScript is slower than JavaScript is almost as silly as saying "C++ with comments is slower than C++ without comments". If you see a C++ program with comments that's slower than another one without comments, it's almost certainly because they're different programs, not because the comments have any effect. From a glance through your links, it seems likely that all of the running time differences are just because it's different code being run.
...which also seems to be reflective of how much programmers using language X care about (execution) efficiency, so I'd say these results are not far off from reality.
C and C++ are another interesting pair --- proponents of the latter always love to claim "zero cost abstractions" that allow you to write very abstract code which they say the compiler can then optimise (or clean up the mess, depending on your viewpoint...) to the same output as if you did it manually in C, but the results show a very different picture.
I haven't seen the code, but maybe c&p-ing the C entrant into the C++ box to get performance parity would make the C++ entrant run afoul of typical C++ coding conventions.
(You can then have a debate about whether you want to see idiomatic C++ in benchmarks or purely the most performant C++ possible. Personally if I'm shopping for a language for a project then I'm far more likely to be interested in the former.)
I'm curious: I've used PHP (not by choice) for the past two years and I strongly dislike it. (I could wax eloquent on this subject, but I don't want a flame war. :) What do you see of value in PHP? Is it the language itself? Its ecosystem or community?
You can throw money at electricity use, and COBOL is compiled to machine code so the power usage is much less of a concern than it would be in an interpreted language.
Much higher on the list of concerns is finding capable COBOL developers, ongoing maintenance, and security. Those are all things that you can't just throw money at.
Not sure how serious they think this is, and how much is this rather a joke. If it's not a joke, they clearly not see the big picture. For example if you can write a software in a higher level language in a fraction of the time which will spare energy somewhere else in the world, it will use less electricity overall.
i suppose every attempt to figure out which is the best text editor, unix-like os, programming language etc is even more futile than most human endeavor :)
It depends. If you're writing a math kernel that will be deployed to a billion IoT devices each with a lifetime of 10 years, you may want to spend some more developer time to shave a few cycles in assembly language. If you're writing some single-use script to automate a single job, do it in the easiest high level language with the library support that you need.
How does less dev time save energy “somewhere else in the world”?
Google & AWS pay money to develop & acquire technologies that save energy because of their scale. The energy used in dev is scratch compared to the energy used to run programs at scale. Google, for example, has a PHP compiler that makes all PHP web pages execute in a fraction of the energy usage of running the PHP interpreter.
Not everyone works for FAANG. Energy usage doesn’t even come up at most small shops. Cloud hosting costs might, but only if they are really out of whack.
You might be misunderstanding. AWS and Google and a few others are hosting everyone’s code, all the big shops and all the small shops and everyone in between. They do things to optimize everybody’s code, because it saves literally millions on their power bills.
Yes I can see that. As a small shop developer, I will make my choice of language based on programmer convenience rather than energy usage. I have never heard any developer talk about "kilowatts" when choosing tech (outside of bitcoin mining perhaps).
So I hear your point but don’t understand why you think it’s arguing against the need for this article? What’s wrong with drawing attention to something you and your peers weren’t thinking about, or doing something to help inform people’s choices?
Perhaps awareness and concrete data of energy usage has to come first? Yes, the article is pointing out something that all the developers you talk to might not know. Doesn’t that make it a good thing? Maybe in the near future you will hear them start to talk about energy. Maybe now that you’re becoming aware, you can be the first among the people you know to talk about the energy usage of your software choices.
Anecdotally, my own experience is that I didn’t hear a lot or think a lot about energy usage until I switched from games & web development to working at a hardware company. My peers now do talk about energy, even though most only write software.
You’re right that most people do choose based on convenience, especially when they lack any other reasons to make the choice. Historically, choosing solely on convenience has contributed to global environmental problems, and people globally are only just becoming aware of the environmental costs of their choices. Separately, Moore’s law only just recently stopped working, so it’s not surprising to me that energy usage has suddenly become more important. Reducing energy use is now one of the primary ways we can increase speed & efficiency, unlike the recent past.
Assembly language is the lower bound here. This is a 100% correct fact that nothing can use less energy than assembly language. No such statement can be made about other languages.
Right but if I write a program that converts code into assembly and does the optimizations automatically then I can get something better or equal to a compiler.
Not sure what to make of this research. It's not like I can switch out JS in favor of these other languages (many of which are compiled binaries) in my web apps. The performance of for example web servers written in the different languages entirely would seem more interesting to compare.
The JS/TS comparison was directly fishy. I found no mention in the paper of how they managed to "run" the tasks in Typescript, which AFAIK is not possible, or at least not how one would do it in a real life.
Instead of a binary. You're running whatever the language compiles to. So then it depends on how efficient the WASM compiler is compared to the binary on whatever platform, and compared to JS.
Just use it as another ballpark reference of what trade-offs are being made when going with one language or another. Not really any different than any other language benchmark/shootout comparison.
Unless you have very large numbers of [users|servers|whatever] to amortize the cost of going one way vs another over, it's probably not worth even considering trying to change anything on your end. More likely, your personal or team productivity is the much more important metric to optimize for anyways. If you're doing things at FAANG scale, it's an entirely different story.
>Not sure what to make of this research. It's not like I can switch out JS in favor of these other languages
Nope but for iot devices or anything in a scenario with scarce energy resources or difficulty to remove heat it's an interesting datapoint. Although I think the results conform mostly with intuition. I doubt anybody would voluntarily compile a lot of haskell on their space satellite
> A faster language is not always the most energy efficient
My rule of thumb now is that memory access, not compute, is the primary consumer of energy. That would tend to confirm the above statement while also supporting the data that scripting languages aren’t super energy efficient, since they tend to do a lot more dynamic allocation than compiled languages, when generally broadly speaking about common programming practices in each language.
In the GPU case, memory access costs sometimes 10x more than compute, meaning that minimizing average (not peak) memory traffic is more or less the only path to significantly reduced energy. (See figs 7 & 10 in the linked paper)
I think practically, for useful analysis in business it would be wattage per unit of useful work (or datum processed as you suggest). For instance, I work in finance, so what is the wattage per option price calculation?
The reason I asked was that it wasn't clear to me if the parent comment was measuring the same way or not re: memory accesses. It seemed pretty expected that a memory access would be more expensive per byte (it takes like hundreds of clock cycles...) but it wasn't obvious to me if it would be so per clock so I wanted to clarify which was meant.
I absolutely agree that it wasn't clear. I don't think wattage per clock cycle is a meaningful measure here - after all, all languages will have the same wattage per clock cycle, if they're executing the same instructions.
What's important is wattage per unit of useful end product, in my example pricing an option. For others, it might be wattage per web page served or anything else.
Yes, I was looking for "carefully optimised Asm" on that list too. Even if it's not faster it will pretty much always be smaller. (Compilers are surprisingly easy to beat at size optimisation.)
Their typescript compiler may not be targeting the correct ecmascript version so the typescript generated could be using polyfills instead of native features.
This was said half in jest I suspect, but it's not a completely off the wall concern. For an infrequently used program the energy burned in its development could easily be a nontrivial amount of the total energy used over its lifetime. If you can write a 5 line Perl script that gets the job done (and is only run a couple dozen times ever), then you're probably saving energy over the 100 line C program despite the massive differences in runtime energy consumption.
Really silly to state how much electricity "languages" use without specifying implementations. You can specify for unstandardized languages, but languages don't have performance characteristics in the typical sense, implementations do, and this can very greatly between implementation.
I'm a little surprised at where Erlang features in the list. I've never used it personally, but have heard so much about it being an amazing, performant language. I wonder why it's so time/energy/memory inefficient for the algorithms used in this exercise.
I’d certainly put Erlang (and Elixir) in the class of scripting languages. In that category it beats out most of the other classic scripting languages like Python, Ruby, Perl, and Lua at execution speed and energy usage, aside from JavaScript and oddly PHP.
For being a mostly pure functional language it performs pretty well in that category. To the parent comment’s question, the reputation for solid performance likely comes from Ruby/Python developers. When dealing with web server applications it really shines due to how it handles concurrency. Also NIFs are really easy to write, especially in Rust.
It's built for massively distributed systems and is efficient at that I believe. It's not really designed to be a replacement for numeric languages like C/C++/Fortran/Ada...etc. It's a high level language, so what you gain in user development efficiency, you lose in performance. Someone knowledgeable in Erlang could confirm though as I'm being general.
BEAM has a 'busy wait' feature. Schedulers with no jobs to do will remain active so they can respond to new jobs faster. You can turn this down so that the schedulers go to sleep quicker, reducing CPU usage.
+sbwt none|very_short|short|medium|long|very_long
Sets scheduler busy wait threshold. Defaults to medium. The threshold
determines how long schedulers are to busy wait when running out of work
before going to sleep.
Can anyone extract what the impact of a JIT compiler is on these figures? The initial performance hit is eventually eroded by the runtime optimizations, so will a long-running Java program eventually close the energy performance gap on a C program?
JavaScript usually runs quite a bit faster than other dynamic languages (after JIT warmup), not because it's fundamentally any faster, just because modern JS engines are so good. Browser competition has meant that the amount of engineering work put into JS performance is much, much more than just about any other language.
Almost 20 years ago I took courses under a professor who was very into reversible computation. As I recall it was mainly on the language side, though.
Glad that it's still moving, sad that it's at an even slower pace than the "functional/immutable will save the parallellization worries" stuff I encountered in the same period.
This study is a bit silly, I had to check whether it was an April Fool's joke. The chipset running the instructions is going to have a far higher impact on energy usage than the language compiling those instructions.
For instance, switching to ASICs or even FPGAs that are optimized to handle certain types of computations.
Just look at datacenters or mobile devices. Sure, optimizations are made to runtime environments to improve performance, and increase battery life. But you are going to see bigger gains through chip architecture. And that is what vendors focus on.
To the naysayers: these languages are either compiled, or will be JIT'ed at runtime. What they will be compiled to will be chip specific binaries-- x86, ARM, whatever. The energy usage will be markedly different depending on which chip architecture is chosen, and will no doubt be inconsistent between different architectures and languages.
Or to put it another way. Why do you think mobile devices almost exclusively use ARM? Why do datacenter operators invest heavily in R&D for ASICs instead of engineering new languages or runtimes? Because languages play a secondary role in energy consumption.
This study is just like countless of other meaningless benchmarks seen in language war flame posts. Pointless and misses the big picture.
With regard to datacenter operators, I would argue that a simpler answer to that observation is that they are investing heavily in the are in which they can actually effect change. That is, they can swap in-and-out their hardware, but language of choice is largely up to their customers - the dev. They can advocate and champion a particular language, but really enforcing it in any meaningful way is narrow casting their own user base.
It is not about stack rank. Your measurements in joules is going to be affected by another variable which was omitted in your experiments! This type of omission would be pretty obvious spot in a scientific paper citing lab experiments.
You will see different energy outputs per language based on their compiled results depending on chipset architecture. And you will see different deltas between them; e.g. language x may use 10% more energy than language y on chipset A, but use 20% more energy on chipset B. And usage will vary depending on which kind of task is performed on which chip architecture, and with what language.
So you did a poor job setting your controls. The effect of the programming language was not well isolated in your benchmark trials, so setting chosen language as an independent variable is flawed. Who knows whether the overall ranking of languages would be affected had the benchmarks been done differently.
For example on a server the CPU will likely be set at full thrust even if it has nothing to do. So I don't think this matters until you need to scale, where you would need less servers/power with a more optimized program.
266 comments
[ 2.5 ms ] story [ 332 ms ] threadThe LISP evaluated by the Benchmark Game - and, apparently, this paper - is indeed SBCL [3], which seems about on par with Java.
“Lisp, on average, consumes 2.27x more energy (131.34J) than C, while taking 2.44x more time to execute (4926.99ms), and 1.92x more memory (126.64Mb) needed when compared to Pascal.”
[1] https://sites.google.com/view/energy-efficiency-languages/ho...
[2] https://github.com/greensoftwarelab/Energy-Languages
[3] https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
What surprised me is how much of a hit the OO programs took. I had thought they would compile down to something reasonably similar to the imperative languages, but C++ ended up being 50% slower.
For example, Javascript and Typescript. Theoretically, I would assume those to be very close, since one compiles to the other. And for memory usage, they are very close. But for running time, Typescript is an order of magnitude slower. That looks suspiciously like entirely different algorithms were used in each implementation (with one being obviously superior in terms of performance), and if that's happening in this specific case, where else is it also causing problems in the analysis?
This also isn't a criticism of the benchmark game, it's well known that not every implementation is equivalent in the time and effort put into optimizing it. It serves its purpose about as well as can be expected. Unfortunately, using it as the base of further calculations can lead to some of the known quirks of the benchmarks being exaggerated into results that are not always obviously an artifact of the underlying system, as I suspect this is. Making that obvious by pointing it out can be useful.
That's actually one of the reasons why the TypeScript/Javascript divide jumped out at me. They were mentioning that the bottom of the list was dominated by interpreted languages, and mentioned TypeScript by name (which surprised my given the focus JavaScript VMs have gotten), and then when I reviewed JavaScript's standing (which was more in-line with what I expected), I noticed the difference between it and TypeScipt was very pronounced, which is odd when (to my knowledge) TypseScript compiles to JavaScript, and not because it's doing a lot of convenience stuff that would slow it down. That said, I don't use TypeScript, so maybe I'm overlooking something.
So as a regular writer of TypeScript I had the exact same question as you.
https://web.archive.org/web/20070503205039/http://shootout.a...
However, the original research has been posted multiple times to proggit and HN since 2017; and I don't recall whether or not anyone noticed this problem until now --
https://news.ycombinator.com/item?id=15249289
https://www.google.com/search?q=energy+efficiency+programmin...
It's a study presented at some conference so while not exactly the Higgs boson, they're showing other people data and the conclusions they derived from it. It's 100% their job to understand what their data measures and to notice that one of the measurements is completely bogus for their purposes. The fact other people hadn't necessarily noticed on messageboards before is mildly curious but it's not really their job.
On the contrary; we can see from archived web pages that other measurements showed the same relatively-poor performance, with those old versions of TypeScript.
Unsuitable because?
Let's say I want to measure the 'energy efficiency' of x86 assembly and JS. I'll use sorting an array of 1000 integers. In my JS implementation, I call Array.sort. In my x86 implementation, I randomly shuffle the array and check if it's sorted, if not repeat until it is. Does measuring the execution times of these tell me anything about the 'energy efficiency' of Javascript vs x86 assembly?
A few TypeScript versions later, that happens to be only 1.6 times slower.
There are good questions to ask about how to handle possible outliers in a study that takes a snapshot of a changing situation and then seeks to make more general claims.
You seem not to have considered the possibility that the authors may have simply made a mistake, unrelated to the origin of the programs.
The authors presented at an Oct 2017 conference. Archived benchmarks game web-pages from 2017 do not show the 10x fannkuch-redux differences that the authors report --
https://web.archive.org/web/20170918163900/http://benchmarks...
I think the Sep 1st 2017 benchmark does, though.[1] At that point it's 1,204.93 seconds, compared to the 131.39 seconds on September 18th you referenced. That makes sense, since the paper could have been finished quite a bit prior to the conference.
1: https://web.archive.org/web/20170901020804/http://benchmarks...
5 July, Node 8.1.3, TypeScript 2.4.1
https://web.archive.org/web/20170715120038/http://benchmarks...
1 Sep, Node 8.4.0, TypeScript 2.5.2
https://web.archive.org/web/20170922144419/http://benchmarks...
----
How should we now assess your "suspiciously like entirely different algorithms were used in each implementation" comment?
Also, typical TypeScript programs are faster than typical JavaScript programs, because JavaScript JITs like predictable object shapes and monomorphic functions for the same reasons that other language implementations require them. Libraries like lodash and bluebird take advantage of this fact without using TypeScript, but TypeScript steers you towards these patterns.
Person who works on js engines here :)
You would think this is the case, but in actuality, js and ts are about on par in performance (assuming similarly written code). This is in fact due to the fact engines optimize for idiomatic js patterns, not idiomatic typescript patterns. often these will align, but in some cases (usually revolving around generics and inheritance) well written js will actually fall through the optimization pipeline faster due to following more patterns that have specific optimization checks.
After having worked on JS for some large web apps that need good graphics performance, the two rules of thumb in my head for making JS fast are: 1- avoid dynamic memory allocation, and 2- avoid using the functional primitives like map.
The first one is more or less true in all languages, memory allocation always costs a lot, and if you can pre-allocate memory and/or re-use memory along the way, the code will run faster. This means paying attention to what things in JavaScript will allocate memory under the hood, use of dicts, use of 3rd party library and framework functions, etc.
The second one is a bummer; I love using the functional primitives. But map is slower than a for loop, all else being equal. It has gotten relatively faster over time. I mostly use functional everywhere, and only resort to for loops in performance critical code.
I understand that map requires creating a new array, and that is already included in point 1. What overhead are functional primitives subject to apart from memory allocation? e.g. forEach
BTW, it's easy to test the basic primitives. I use Chrome snippets.
This is on my Mac in Chrome. So forEach is faster than map, but for loop is more than 10x faster than forEach. That's for loops with trivial work, of course. If the inside of the loop is expensive, the loop/map ratio will be lower.Like so?
I get: copy loop 973, copy map 38, copy forEach 49. Same as before, but this time I tried Chrome in Windows.Are you using a different browser? I know that sometimes other browsers have very different results.
In any case, it's somewhat irrelevant if there are cases that optimize and cases that don't. When idiomatic functional code is sometimes up to 30x slower than a for loop, it can't be used in perf critical sections. Even if it's only Chrome and only certain cases. The forEach perf needs to be always reliably performant before I can use it without worry.
Often times, these sorts of benchmarks are done with prejudice (not necessarily malice). The benchmarks are written by someone with something to prove: my chosen tech stack performs better, and let me show you why. A favorite of mine is Perl vs Python comparisons, where you see an idiomatic Perl implementation vs a non idiomatic Python implementation (or other way around). Typically in a head-to-head comparison, the benchmarks are developed by the same individual whom likely has above average knowledge in their favorite and below average in the target they're trying to show as inferior.
You'll see this time and time again in internet benchmarks comparing performance. Unless you can see the code from all benchmarks involved, my suggestion is to avoid them. I mean, for all I know, the author of the benchmark was unaware of the built in sort and instead bubble sorted.
The test that lowered TypeScript’s score in the paper is called fannkuch-redux, and here are the sources in question:
https://github.com/greensoftwarelab/Energy-Languages/blob/ma...
https://github.com/greensoftwarelab/Energy-Languages/blob/ma...
They are both contributed by the same person, and there is no bubble sort involved. So now you know.
I don’t see an obvious reason one would be slower, but they’re also quite different. Maybe the algorithmic complexity is different. Maybe the cross-compilation is doing something bad with memory allocation. Note the input sizes for this test are very small, it would be easy for a difference in temporary variables the compiler injects to cause a serious problem.
What is not obvious is any prejudice, malice, or incompetence.
What are you talking about? Where did that happen?
> when you admit to having zero idea what the actual specific problem here is?
This is the comment section for a submission about an article referencing the paper. I brought it up for discussion. It is perfectly valid to bring up a question that you don't know the answer to.
> What is not obvious is any prejudice, malice, or incompetence.
Please stop.
Edit: From another comment, and some deeper digging of my own from that, you might find the archived results of the fannkuch-redux interesting. From 2017-08-01[1] to 2017-09-18[2], the benchmark changed from a running time of 1,204.93 second to a running time of 131.39 seconds. The paper was released in October 2017.
1: https://web.archive.org/web/20170901020804/http://benchmarks...
2: https://web.archive.org/web/20170918163900/http://benchmarks...
I was responding directly to @hermitdev. Did you get your threads crossed? What I'm talking about happened immediately above in the parent comment, beginning with "Often times, these sorts of benchmarks are done with prejudice" https://news.ycombinator.com/item?id=19527057
"You'll see this time and time again in internet benchmarks comparing performance."
> It is perfectly valid to bring up a question that you don't know the answer to.
I agree. It's a bummer that's not really what happened here.
>> What is not obvious is any prejudice, malice, or incompetence. > Please stop.
The parent comment explicitly stated an assumption of both incompetence and prejudice and I responded directly to that.
From the HN guidelines: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."
If you'd like me not to call out speculation, then please assume good faith and don't speculate next time.
> From 2017-08-01[1] to 2017-09-18[2], the benchmark changed from a running time of 1,204.93 second to a running time of 131.39 seconds.
Yes! Now we are getting somewhere. It appears that would change the outcome of the paper. Perhaps it was a mistake. That might mean it was nothing more than an oversight that already got fixed. It doesn't mean there is any other coloring of the study at all, nor that there was any intention or agenda to make TypeScript look bad, right?
Perhaps I misinterpreted what you said. You started the paragraph referring to the top level comment, which is me. I took the "you're" in "You’re assuming incompetence when you could just go look it up." to be a general "you", and commentary on my original comment.
> From the HN guidelines: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."
I actually looked this up before the GP comment, and almost included it myself. I can see now that you were implicating the comment you replied to. I didn't think that was the case, because I apparently didn't interpret that comment remotely in the same way you did.
> Perhaps it was a mistake. That might mean it was nothing more than an oversight that already got fixed. It doesn't mean there is any other coloring of the study at all, nor that there was any intention or agenda to make TypeScript look bad, right?
I never implied it was. For that matter, I didn't really interpret the comment in question as stating that either. The more charitable interpretation is not that they are trying to make another language look bad, but that they are trying to make their favorite language look good. That doesn't require purposefully tanking one benchmark, it just requires them to be much better versed in optimizing one language than another and a lack of awareness about this. As they say, never attribute to malice what can be explained by incompetence. In fact, if you read the comment carefully, they even call out to this with the "not necessarily malice" remark.
The project doesn’t talk about favorites or seem to want to make certain languages look good. Jumping to the conclusion that bias is involved isn’t the good faith interpretation, even if you state with a positive sounding framing. The good faith interpretation is to take the stated project goals at face value, and assume that the participants have done a good job.
I didn't see anywhere that the comment in question called any project bias into question, but instead noted that in a situation where work is crowd sourced, people with their own intentions and motivations will put out bad benchmarks, either in the case of the benchmarks game, or a specific benchmark or comparison put forth in an article or blog. I've personally been witness to the latter multiple times just from HN submissions.
I just want to end with, as someone that's brought up viewing comments in an uncharitable light, you seem to have done a lot of that in this discussion. You've repeatedly taken your interpretation of a comment, rephrased it in a harsher way, and the stated it as what the other person was saying as fact, and then responded to that. I would think actually trying to find a charitable interpretation should at least include a question at the beginning to confirm whether what you think is being said is entirely correct. Note that I started with that when I thought you were attributing statements to me that I did not say. My first words were a solicitation "What are you talking about? Where did that happen?" to confirm what was going on. You've been doing this from your first response to my top level commend, when you stated "But you’re using that assumption to cast slippery-slope doubt on the whole project without knowing anything specific." That's a very uncharitable rephrasing of what you think I was doing, and it certainly wasn't my intention. I've already outlines in specific exactly what I was trying to do and why, and in doing so I also stated that I felt you were misinterpreting me. There's a clear trend here as I see it, and you repeatedly bringing up good faith assumptions just puts it into clear highlight.
I think we've covered about all there is to say on this (these) topics. I'll let you have to the last word if you wish. I'll read and promise to consider any points you raise, but I don't think me responding would be very fruitful, and this discussion has digressed far enough.
5 July, Node 8.1.3, TypeScript 2.4.1
https://web.archive.org/web/20170715120038/http://benchmarks...
1 Sep, Node 8.4.0, TypeScript 2.5.2
https://web.archive.org/web/20170922144419/http://benchmarks...
----
How should we now assess your "suspiciously like entirely different algorithms were used in each implementation" comment?
Yes.
> How should we now assess your "suspiciously like entirely different algorithms were used in each implementation" comment?
The suspicion was incorrect. That's why it was presented as a suspicion, not as fact. I have no reason to defend it if it's incorrect, but I still defend that it was valid to raise questions, given the facts on the ground. We've now shown there was something that changed very drastically at that time, and while it's less likely it's the benchmarks themselves (unless one or both of those are fairly out of date Node versions)[1], it still points towards something to be aware of in the results presented. Namely, they rely on a lot of underlying assumptions which should be looked at if you care about the numbers.
1: Also, I imagine the V8 devs probably considered the performance of TypeScript in that case to be a bug, given how horrible the performance regression from JavaScript is and that it's still javaScript running. It's possible that TypeScript was doing something really odd, but given the exposure and Microsoft's backing and developer time, I think that's a less likely scenario than some optimization that should have been triggered was missing, which happens quite often.
> I still defend that it was valid to raise questions
Of course, it's valid to question a measurement that looks strange but your comment went further than that -- your comment, without evidence, assumed a cause; and, without evidence, implied that assumed cause led to widespread problems with the analysis.
In other words -- innuendo.
Corrections are for facts. I put forth a theory. People being misled by a theory are not something I have limited power to affect. People representing theories read on the internet as fact have larger problems that that will solve.
This discussion is the correction, and a better one than someone would be willing to read. Were it within the 2 hour edit window, I would through in an edit, I've done so numerous times in the past. I will ask Hn to amend it's rules so I can correct a statement I made about something I suspected.
> Of course, it's valid to question a measurement that looks strange but your comment went further than that -- your comment, without evidence, assumed a cause
This is incorrect. I had evidence, I had numbers that did not line up with my understanding of how things should have been given my knowledge of the subject. I presented that as a theory, by using the word "suspect". All I implied is that if that theory was correct, which I made sure to not assert as fact, then it might affect some other languages. I did not assume a cause, I assumed a possible cause, and presented it as such.
I am very particular with my language. I try not to state things as fact when they are not. I try my absolute hardest (and I believe I succeed) to always speak in good faith, where I'm trying to raise a point I think is worthwhile or ask a question where I think there is benefit. I'm actually rather bothered by how some people interpreted my words and intentions, and that includes you. I'm bothered by how you've interpreted my words. Since you're not the only one (although I do believe you're in the minority), I'll assume there's something I could have done better to represent my point. I don't think all the blame lays with me though. There should be some way for me to posit a question and advance a theory without people assuming bad faith, so my question to you is, what way is that? How could I have expressed concern over the results without triggering that interpretation from you? Because I don't think doing personal research on a problem is an acceptable prerequisite for raising a question. In this case, I could have spent hours looking into something I was unfamiliar with and come away with more answers, but many people may not have the knowledge to do so but have enough to think something is wrong. Should they just keep their mouths shut? Are we in a time where raising a concern that turns out to be unfounded (or in this case, just more complicated and slightly misdirected) is unacceptable under any circumstance? I refuse to accept that.
The honest question is -- Can someone please confirm that those programs implement the same algorithm?
No! I think this is a very useful project and analysis. I just think that some languages might have extremely optimized versions (or possibly more likely, some languages don't quite yet have that extremely optimized version that has propagated throughout the others) and that might be affecting specific languages in the analysis.
I think the first 5-10 entries are probably very accurate, as they are generally with very performance centric and often optimized for performance languages. As the languages and VMs/interpreters do more and are optimized less, it's much easier to miss a performance difference caused by an benchmark submission and attribute it to an inherent aspect of the language.
> I am at least giving benefit of the doubt and wondering what might be wrong with TypeScript.
I did no such thing. Note how I used the phrases "Theoretically, I would assume" and "That looks suspiciously like". I simply raised an issue of concern, in a way where it was obvious that I did not know if my concern was correct, and wondered that if it was, what else it might affect.
That’s exactly what I mean by casting doubt. If the concern might not be correct, why lead into speculation about further concerns?
Let’s find out what the actual reason that TypeScript is measured slower, rather than guess as what else could be wrong if unverified theoretical assumptions might be correct.
You mean, I should not have explored the ramifications of what my suspicions might mean to so people might think it's worth actually looking into? What's wrong with that?
I can't help but feel that you feel compelled to defend your original position that I believe was based on a misinterpreting my point and intention.
I saw what I believed might be a problem. I noted it. I noted why. I noted what it might mean to the analysis because if my suspicion as to the reason was correct, it might not be isolated and other items might need a closer look. I did so in a way where I was sure not to claim something as factual when I wasn't certain. What part of that do you think is an inaccurate assessment of what I did or was unwarranted?
Edit: Changed misconstruing to misinterpreting, as that's what I was trying to express, and misconstruing might be interpreted as a purposeful action, which is not what I was trying to express.
The part that was unwarranted was the negative speculation.
The authors presented at an Oct 2017 conference. Archived benchmarks game web-pages from 2017 do not show the 10x fannkuch-redux differences that the authors report --
https://web.archive.org/web/20170918163900/http://benchmarks...
https://sites.google.com/view/energy-efficiency-languages/ho...
Many benchmarks have no TS implementations, TS/JS results are about the same except for fannkuch-redux which is about a zillion times slower in the TS implementation. When looking at that kind of massive discrepancy in similar languages with the same runtime, the guess kbenson made was a perfectly sensible one. The authors of the study should have examined that kind of crazy outlier more closely and it's, as pointed out, not that unusual when using the benchmark game as a starting point.
In fact, on the benchmarks game website, those same programs do not show a "massive discrepancy" --
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
The issue is that one of the metrics in the suite is lines of code, so people write fantastically obscure and concise functional programs in Perl when the imperative one would be 2x the LOC, but much, much faster.
(This is from a spot check years ago. Maybe they’ve fixed this somehow).
The benchmarks game does not measure LoC --
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
In that case, the characteristics of the system will dominate.
I don't think that's a safe assumption. The Javascript results would be a lower bound (assuming the same algorithm, etc.), but there's no telling what "extra" Javascript (and thus overhead) might be inserted by the Typescript compiler.
As far as individual results go, there's no point jumping to conclusions when the results and code are available online and easy to try for yourself:
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Typescript is still 2-3x slower than Node in several of the benchmarks, and the results could have been different in May 2018 when the article was written.
I would be very surprised if the fastest C++ and C are actually different.
You can see 4 or 5 C++ fannkuch-redux programs "compared"
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
That's true, but the fact we're talking about an order of magnitude running time difference is what lead me to think it's not as easily explained as that.
> there's no point jumping to conclusions
Respectfully, I don't think I did. I put out a theory, with my reasoning, fully acknowledging where it might be wrong. As someone that doesn't use TypeScript (and I wouldn't consider myself in expert in JavaScript either), I don't feel qualified to assess the respective algorithms for those languages in the benchmarks game, so I brought the issue to a larger audience so someone else might take a look if so inclined.
> Typescript is still 2-3x slower than Node in several of the benchmarks, and the results could have been different in May 2018 when the article was written.
That doesn't really invalidate my theory, as it's possible those are also instances where it's lagging in a good submission. Given that there are several tests where it actually beats the fastest JavaScript algorthim submitted, that seems at least plausible.
TypeScript doesn't do this. With only a few exceptions, the way to turn TypeScript into JavaScript is to remove the type annotations, which leaves you with JavaScript. This is an important design goal of TypeScript: it doesn't change your code, it's just JavaScript with types. Like Babel, TypeScript can be used to compile to earlier JS versions like ES5, but I'd consider that a misconfiguration in an environment like this.
Saying that TypeScript is slower than JavaScript is almost as silly as saying "C++ with comments is slower than C++ without comments". If you see a C++ program with comments that's slower than another one without comments, it's almost certainly because they're different programs, not because the comments have any effect. From a glance through your links, it seems likely that all of the running time differences are just because it's different code being run.
...which also seems to be reflective of how much programmers using language X care about (execution) efficiency, so I'd say these results are not far off from reality.
C and C++ are another interesting pair --- proponents of the latter always love to claim "zero cost abstractions" that allow you to write very abstract code which they say the compiler can then optimise (or clean up the mess, depending on your viewpoint...) to the same output as if you did it manually in C, but the results show a very different picture.
(You can then have a debate about whether you want to see idiomatic C++ in benchmarks or purely the most performant C++ possible. Personally if I'm shopping for a language for a project then I'm far more likely to be interested in the former.)
Something like 90% of all credit card transactions.
Much higher on the list of concerns is finding capable COBOL developers, ongoing maintenance, and security. Those are all things that you can't just throw money at.
Google & AWS pay money to develop & acquire technologies that save energy because of their scale. The energy used in dev is scratch compared to the energy used to run programs at scale. Google, for example, has a PHP compiler that makes all PHP web pages execute in a fraction of the energy usage of running the PHP interpreter.
Perhaps awareness and concrete data of energy usage has to come first? Yes, the article is pointing out something that all the developers you talk to might not know. Doesn’t that make it a good thing? Maybe in the near future you will hear them start to talk about energy. Maybe now that you’re becoming aware, you can be the first among the people you know to talk about the energy usage of your software choices.
Anecdotally, my own experience is that I didn’t hear a lot or think a lot about energy usage until I switched from games & web development to working at a hardware company. My peers now do talk about energy, even though most only write software.
You’re right that most people do choose based on convenience, especially when they lack any other reasons to make the choice. Historically, choosing solely on convenience has contributed to global environmental problems, and people globally are only just becoming aware of the environmental costs of their choices. Separately, Moore’s law only just recently stopped working, so it’s not surprising to me that energy usage has suddenly become more important. Reducing energy use is now one of the primary ways we can increase speed & efficiency, unlike the recent past.
But HipHop is certainly another good example that demonstrates that this matters in practice.
The JS/TS comparison was directly fishy. I found no mention in the paper of how they managed to "run" the tasks in Typescript, which AFAIK is not possible, or at least not how one would do it in a real life.
Unless you have very large numbers of [users|servers|whatever] to amortize the cost of going one way vs another over, it's probably not worth even considering trying to change anything on your end. More likely, your personal or team productivity is the much more important metric to optimize for anyways. If you're doing things at FAANG scale, it's an entirely different story.
Nope but for iot devices or anything in a scenario with scarce energy resources or difficulty to remove heat it's an interesting datapoint. Although I think the results conform mostly with intuition. I doubt anybody would voluntarily compile a lot of haskell on their space satellite
My rule of thumb now is that memory access, not compute, is the primary consumer of energy. That would tend to confirm the above statement while also supporting the data that scripting languages aren’t super energy efficient, since they tend to do a lot more dynamic allocation than compiled languages, when generally broadly speaking about common programming practices in each language.
This is mainly colored by GPU usage and a paper/presentation some friends made: https://www.researchgate.net/publication/324217073_A_Detaile...
In the GPU case, memory access costs sometimes 10x more than compute, meaning that minimizing average (not peak) memory traffic is more or less the only path to significantly reduced energy. (See figs 7 & 10 in the linked paper)
What's important is wattage per unit of useful end product, in my example pricing an option. For others, it might be wattage per web page served or anything else.
https://sites.google.com/view/energy-efficiency-languages/se...
If for the same job i need 100 manhours in python and 1000 manhours in C I’d indefinitely use python.
Interesting but useless report.
Rust was specifically created to use the paralell hardware resources well, and it excels at it.
These benchmarks are all single-threaded, pure computation. Erlang is pretty slow at that (alas).
For being a mostly pure functional language it performs pretty well in that category. To the parent comment’s question, the reputation for solid performance likely comes from Ruby/Python developers. When dealing with web server applications it really shines due to how it handles concurrency. Also NIFs are really easy to write, especially in Rust.
From http://erlang.org/doc/man/erl.html:
https://spectrum.ieee.org/computing/hardware/the-future-of-c...
Glad that it's still moving, sad that it's at an even slower pace than the "functional/immutable will save the parallellization worries" stuff I encountered in the same period.
For instance, switching to ASICs or even FPGAs that are optimized to handle certain types of computations.
Just look at datacenters or mobile devices. Sure, optimizations are made to runtime environments to improve performance, and increase battery life. But you are going to see bigger gains through chip architecture. And that is what vendors focus on.
Or to put it another way. Why do you think mobile devices almost exclusively use ARM? Why do datacenter operators invest heavily in R&D for ASICs instead of engineering new languages or runtimes? Because languages play a secondary role in energy consumption.
This study is just like countless of other meaningless benchmarks seen in language war flame posts. Pointless and misses the big picture.
Facebook and Google both have in house chip designers these days, specifically for their data centers.
You will see different energy outputs per language based on their compiled results depending on chipset architecture. And you will see different deltas between them; e.g. language x may use 10% more energy than language y on chipset A, but use 20% more energy on chipset B. And usage will vary depending on which kind of task is performed on which chip architecture, and with what language.
So you did a poor job setting your controls. The effect of the programming language was not well isolated in your benchmark trials, so setting chosen language as an independent variable is flawed. Who knows whether the overall ranking of languages would be affected had the benchmarks been done differently.