Interesting but correlates with the number of files and libraries the interpreter must read or parse. Anything that's quite self-contained is fast; anything that needs to look up and read lots of files is slow.
All I can say is that Java startup time is abysmal. Still, and after fifteen years. Java was dead slow in the 90's and it's dead slow these days when compared to anything else. I love Clojure but Java startup time rules it out for anything else but long-running processes or, when coding, using repl via swank.
For Java long start time is because it initializes big amount of RAM at the start, but thanks to this creating objects latter is fast.
Additionally measuring of start of program is meaningless, what this say to us? Nothing. OK, maybe something, something like "write programs/scripts which execution is less than seconds in languages like ....".
But for most of programs it doesn't matter.
What the difference if program starts 3.1-3.2 seconds and 0.05 second? If execution time is longer than minute human will not see difference.
I can understand people decrying results of .05 seconds apart or so, but 3 second start ups? That would be infuriating. That kind of UX would take me back quite a few decades...
Garwk. I've helped write whole operating systems that boot in less time. [Atari ST. Check. Newton OS. Check. Other stuff I can't talk about: tens of milliseconds, mostly waiting for external crap like USB to initialize and settle down.]
If xterm, less, ls, vi and others would take 3 seconds to start, I would never get anything done. I think it's sad that people think that 3 seconds is fast (unless the program is loading some big data of course).
I'd like to know what version (and vendor, especially in the case of Java) of each he was running. I've heard time and time again that "Java isn't as slow any more" and maybe that is the case with the "newer" Sun (and Oracle) versions.
However, as one of the later mails[1] says, "such a short program doesn't give the JIT time to warm up"—"Hello world" is a poor comparison to real-world performance.
That depends on what you mean by "real-world performance". The board on which this conversation takes place compares Java to Python, bash and C. Their "real-world" probably consists of a bunch of short-lived small programs on the command line, lots of i/o pipelines etc etc. I think the benchmark gives a meaningful measure of "fit-for-purpose" for that kind of workload.
There is the nailgun project (http://www.martiansoftware.com/nailgun/) which seems to initialize a JVM in the background so that tasks can execute more quickly, but it's not being maintained.
I know it's great fun, and fashionable to bash Java, but the Java startup time is slow, because the default settings are optimized for long running processes.
If you want a quicker execution time for a short lived program, just tweak the settings. But then Java may not be the best choice for short lived 'scripting' anyway.
So yes, it rules it out, unless you know what you're doing.
Interestingly, Java and Clojure have decidedly different startup times on my machine.
A trivial hello-world in Java takes ~130 ms on my Debian sid box with « OpenJDK Runtime Environment (IcedTea6 1.8.7) (6b18-1.8.7-4) / OpenJDK 64-Bit Server VM (build 16.0-b13, mixed mode) ». This is with all default runtime options.
The program « (ns hello (:gen-class)) (defn -main [] (println "Hello, world!")) » compiled with Clojure 1.2.0 (again from Debian sid) takes ~1700 ms. Using -Xbootclasspath takes it down to ~800 ms. -client doesn't do anything obvious. Unpacking the Clojure JAR increases the time.
So presumably there's something about pulling in the Clojure runtime pieces that is very slow, but I have no idea what it is.
Java is indeed slow to start. So slow, in fact, that Maven created a "Maven Shell" project to avoid re-launching a JVM for each build: http://shell.sonatype.org/
The primary disadvantage of Java/.NET compared to C/C++ is the memory overhead. Generational garbage collectors can require up to 5 times as much memory as their explicitly-managed equivalents. [1]
Space overhead doesn't matter for most applications. But it makes Java a no-go for embedded devices and data-intensive domains.
I couldn't access the paper itself, but the excerpts are indeed interesting.
Another disadvantage of using a Garbage collector, is that it's hard to predict when the JVM will "freeze" to collect garbage memory. This makes standard Java relatively ill-suited for real-time applications. For these, you need to consider technologies such as http://en.wikipedia.org/wiki/Real_time_Java
The problem with those GC freezes, is that they can be quite long in big memory-intensive applications. This is a reason for the work on alternative parallel GCs. The complexity and relative mysteriousness of the GC makes it hard to tune. Dhanji, a Google Wave engineer, who wrote the "Dependency Injection" book, and is the lead developer of sitebricks, wrote a great article on this subject : http://dhanji.posterous.com/taking-out-the-trash-and-other-c...
All this being said, I don't know if it's fair to compare the approach mentioned in your link with "classical" manual memory management. How often do you see this kind of (de-)allocation patterns in a standard app? Wouldn't a programmer often write "less than ideal" de-allocation code? (honest question there, I sadly never had the opportunity to write in a language with manual memory-management)
"We present a novel experimental methodology that lets us treat unaltered Java programs as if they used explicit memory management. Our system generates exact object reachability information by processing heap traces with the Merlin algorithm [34, 35]. It then re-executes the program, invoking free on objects just before they become unreachable. Since this point is the latest that a programmer could explicitly free objects, our approach conservatively approximates explicit memory management."
All this being said, I don't know if it's fair to compare the approach mentioned in your link with "classical" manual memory management. How often do you see this kind of (de-)allocation patterns in a standard app? Wouldn't a programmer often write "less than ideal" de-allocation code? (honest question there, I sadly never had the opportunity to write in a language with manual memory-management)
You're on the right track, but with a twist: Explicit memory management works the opposite way around. Freeing an object when it goes out of scope, like the paper's oracle, is the worst case scenario. A real programmer is free to optimize and release memory before that, even if the object still has references.
In practice, I doubt the difference matters. The primary focus of memory management in C/C++ is avoiding leaks from objects that were never freed and can never be freed because all their references are out-of-scope. Programmers have a small window to free memory, before the pointers are out of scope, but only once the object isn't needed. There's not much room for optimization.
There's RTSJ (or jrts) specifically designed for real-time use. It has an incremental GC that is overall less efficient, but is guaranteed to have bounded wait times.
"Suppose you had an automatic tool that optimizes Java code for your target platform before execution. Free of resource constraints inherent to run time optimizers, it processes all the code and optimizes it well.
You feed the application's classes to the tool, which optimizes them for performance and saves the results to disk. Now, you can deploy your Java application in optimized form so that it runs fast from the start on target systems.
...
Using the Excelsior JET Optimizer, you convert the classes into highly optimized x86 code and create a native executable for Windows or Linux. This technique is called Ahead-Of-Time (AOT) compilation. "
Unless I'm mistaken, the "Bash" benchmark is actually just measuring the amount of time it takes a C program to read an argument and print, instead of just a C program printing.
Assuming the bash program was: `echo "hello, world!"`
Edit: I likely am mistaken. Chances are this was done using the builtin 'echo', not '/bin/echo'. The builtin is always faster (though putting them in a file (as is the only fair way to do it) slows them both down of course).
I'm not really sure if I understand your question. People have runtime speed demands for their applications and when they start a project they get to choose a language. Runtime speed is one of several pertinent pieces of information that is relevant when choosing which language to use.
Not that this benchmark is actually indicating anything about which language to use in any of those circumstances (Java has a bad startup but pretty good during runtime, and startup time on these orders is generally irrelevant for any nontrivial application), but that isn't to say that language benchmarks in general are irrelevant.
from as little coding experience i have i can say that I have seen blazingly fast V1.0 of applications which doomed to sluggish responses because of poor maintenance. So, i really don't buy the speed thing. I'll always design with a note that "There will be no awesome-coders to maintain this". A well-maintained version of Java app will beat a struggling C app anyday. So, i differ. And, ofcourse if you know what you are doing, these graphs do make sense and its worth an effort to dig out the flipping bits on registers.
"A well-maintained version of Java app will beat a struggling C app anyday."
And a poorly maintained C program will outperform an equally poorly maintained Java program...
The things that make programs dramatically slower is poor algorithmic complexity (and often a poor understanding of the architecture, be it the JVM or the metal). That's not what we're talking about here though, this discussion is pretty explicitly about start-up times. We're talking constant-time issues.
With disk caches primed, the main culprit may be the dynamic linker. Some of those environments (Java, Perl, Python, PHP that others mention in this thread) always load substantial number of dynamic libraries before doing anything useful. That doesn't come free. Numberous symbols have to be resolved, some static data initialized. There's non-trivial VirtualMemory to set up for such libraries, too.
EDIT: a micro-benchmark between static and dynamic C program:
gcc -- static a.c
time (for i in `seq 1024`; do ./a.out > /dev/null; done; )
real 0m0.958s
####
gcc a.c # that's dynamic linking
time (for i in `seq 1024`; do ./a.out > /dev/null; done; )
real 0m1.292s
...and that's just libc alone! Those times were quite stable across several runs, so it's not some random one-time fluctuation.
The code:
#include <stdio.h>
int main(void) {
puts("hello world!");
return 0;
}
This is one more reason why I love dynamic linking, fork/exec for static binaries is much faster than for dynamic ones, even for simple C programs that only use libc.
What a misleading example. That's not GNU hello world, that's Linux hello world. The difference: it does not use GNU libc nor any other parts of GNU project -- except for the `unistd.h' -- but instead interfaces with the kernel directly.
Another matter is that it's not a C program, as it doesn't have int main(...) :P
That you used GCC is irrelevant; other compilers would give pretty much the same result. GCC is not required by the program by any means. Nothing of GCC is part of the program itself due to the -nostartfiles flag. The program doesn't use GCC, you (the software developer) do :-)
For sake of example, consider Linux (the kernel); usually compiled with GCC -- but ICC works as well -- and distributed under GPL, yet is not part of GNU project.
Moreover, the program in question can be ran directly on bare Linux, with no GNU stuff present anywhere, if passed via `init=PATH_TO_THE_PROGRAM' boot param. Of course the kernel would panic right after the syscall(__NR_exit, 0)... so I'm bailing out of nitpick mode right now ;-)
Uh, if you think gcj is going to beat sun's javac/java, you're mistaken.
Again, with the rapid unexplained downvotes... seriously, did you see the article about speeding up Eclipse? The suggestion was to make sure you're not using gcj and others commented that they agree'd. It's well known the gcj is just flat slow compared to Oracle's/OpenJDK.
lol, whatever guys, pile on, I must be missing the joke!
The OP was talking about program startup time and measuring that by comparing hello world programs. When I run the above hello world program compiled with gjc is 25% slower that the C program. Compare that with the OP's measured version being 300 times slower than the C version when compiled with Oracle's JDK.
Eclipse is not representative of a typical command line program. Write "ls" with the OpenJDK and tell me you don't pull you hair out waiting for it to start up when using it.
I guess java is so slow because you have the initial over-head of starting the jvm. I'm guessing that subsequent prints of "Hello world" would be somewhat faster.
This article is basically trolling java, and it seems a number of people have taken the bait. Different languages have different strengths and weaknesses. You'll choose the one most appropriate to the task required. If that's printing "Hello World" to the console then fine, but I doubt many people would think "java" when presented with this requirement.
It depends how often I run the commands. Subversion -> darcs, git, etc. was like a breath of fresh air. I know that was going out to the network but still latency can be a killer.
Ignorance and stupidity is running amok. This kind of trivial benchmark is not only incredibly dumb but is ACTIVELY HARMFUL to the programming community. Consider the conclusion drawn by author a few posts down in the thread
"Astonishing indeed. The whole thing confirmed my hunch that Bash,
Chicken, and C provide a complete toolset to tackle 99.9% of
programming needs."
No no no! Try running anything non-trival in bash or python and compare that to the java. Obviously if all you ever want to do with programming is write little boring scripts then you shouldn't be using a language like java in the first place. For anything mildly interesting and performance intensive Java is pretty much a no brainer only behind c/c++ amongst the major languages. For a site with REAL benchmarks which actually mean anything unlike the language fanboyism in this posted article checkout http://shootout.alioth.debian.org/u64q/which-programming-lan...
There is no need for hysteria; it depends on your use case. In the Unix shell, since every basic operation spawns a process, and performing such operations only takes a few milliseconds, startup time is extremely important. If it were written in Java, it would probably not be architected this way, but the startup time of C makes it acceptable to do so; it is an issue if commands written in languages like Perl and Python feel slower than the rest of the system.
But "if all you ever want to do with programming is write little boring scripts" it might help to understand that 9 hundredths of a second delay is below the threshold that someone using the boring little script will notice.
And "if all you ever want to do with programming is write little boring scripts" simply using Java as an interpreter with -Xint might be faster.
For example, that old benchmarks game n=200 "hello world"
To my great astonishment, Ruby's performance was much closer to bash than Python in my own test: 0.572sec for 101 runs in Ruby, versus 4.276sec with Python and 0.318sec with bash. I now feel slightly less bad for all those shell scripts I've written in Ruby instead of bash.
70 comments
[ 2.3 ms ] story [ 153 ms ] threadAll I can say is that Java startup time is abysmal. Still, and after fifteen years. Java was dead slow in the 90's and it's dead slow these days when compared to anything else. I love Clojure but Java startup time rules it out for anything else but long-running processes or, when coding, using repl via swank.
Why would this take so much time?
However, as one of the later mails[1] says, "such a short program doesn't give the JIT time to warm up"—"Hello world" is a poor comparison to real-world performance.
http://lists.nongnu.org/archive/html/chicken-users/2011-03/m...
That depends on what you mean by "real-world performance". The board on which this conversation takes place compares Java to Python, bash and C. Their "real-world" probably consists of a bunch of short-lived small programs on the command line, lots of i/o pipelines etc etc. I think the benchmark gives a meaningful measure of "fit-for-purpose" for that kind of workload.
If you want a quicker execution time for a short lived program, just tweak the settings. But then Java may not be the best choice for short lived 'scripting' anyway.
So yes, it rules it out, unless you know what you're doing.
Do SSDs magically fix this?
A trivial hello-world in Java takes ~130 ms on my Debian sid box with « OpenJDK Runtime Environment (IcedTea6 1.8.7) (6b18-1.8.7-4) / OpenJDK 64-Bit Server VM (build 16.0-b13, mixed mode) ». This is with all default runtime options.
The program « (ns hello (:gen-class)) (defn -main [] (println "Hello, world!")) » compiled with Clojure 1.2.0 (again from Debian sid) takes ~1700 ms. Using -Xbootclasspath takes it down to ~800 ms. -client doesn't do anything obvious. Unpacking the Clojure JAR increases the time.
So presumably there's something about pulling in the Clojure runtime pieces that is very slow, but I have no idea what it is.
More details: http://ericmiles.wordpress.com/2010/03/23/intro-to-maven-she...
Note that, startup time aside, Java's overall performance is actually quite good, due to the JIT compiler: http://stackoverflow.com/questions/145110/c-performance-vs-j...
Space overhead doesn't matter for most applications. But it makes Java a no-go for embedded devices and data-intensive domains.
[1] http://lambda-the-ultimate.org/node/2552
Another disadvantage of using a Garbage collector, is that it's hard to predict when the JVM will "freeze" to collect garbage memory. This makes standard Java relatively ill-suited for real-time applications. For these, you need to consider technologies such as http://en.wikipedia.org/wiki/Real_time_Java
The problem with those GC freezes, is that they can be quite long in big memory-intensive applications. This is a reason for the work on alternative parallel GCs. The complexity and relative mysteriousness of the GC makes it hard to tune. Dhanji, a Google Wave engineer, who wrote the "Dependency Injection" book, and is the lead developer of sitebricks, wrote a great article on this subject : http://dhanji.posterous.com/taking-out-the-trash-and-other-c...
All this being said, I don't know if it's fair to compare the approach mentioned in your link with "classical" manual memory management. How often do you see this kind of (de-)allocation patterns in a standard app? Wouldn't a programmer often write "less than ideal" de-allocation code? (honest question there, I sadly never had the opportunity to write in a language with manual memory-management)
"We present a novel experimental methodology that lets us treat unaltered Java programs as if they used explicit memory management. Our system generates exact object reachability information by processing heap traces with the Merlin algorithm [34, 35]. It then re-executes the program, invoking free on objects just before they become unreachable. Since this point is the latest that a programmer could explicitly free objects, our approach conservatively approximates explicit memory management."
Finally, according to the following (2005) article by Brian Goetz, using a garbage collector may yield more opportunities for optimization, at the cost of predictability: http://www.ibm.com/developerworks/java/library/j-jtp09275/in...
All this being said, I don't know if it's fair to compare the approach mentioned in your link with "classical" manual memory management. How often do you see this kind of (de-)allocation patterns in a standard app? Wouldn't a programmer often write "less than ideal" de-allocation code? (honest question there, I sadly never had the opportunity to write in a language with manual memory-management)
You're on the right track, but with a twist: Explicit memory management works the opposite way around. Freeing an object when it goes out of scope, like the paper's oracle, is the worst case scenario. A real programmer is free to optimize and release memory before that, even if the object still has references.
In practice, I doubt the difference matters. The primary focus of memory management in C/C++ is avoiding leaks from objects that were never freed and can never be freed because all their references are out-of-scope. Programmers have a small window to free memory, before the pointers are out of scope, but only once the object isn't needed. There's not much room for optimization.
You feed the application's classes to the tool, which optimizes them for performance and saves the results to disk. Now, you can deploy your Java application in optimized form so that it runs fast from the start on target systems.
...
Using the Excelsior JET Optimizer, you convert the classes into highly optimized x86 code and create a native executable for Windows or Linux. This technique is called Ahead-Of-Time (AOT) compilation. "
http://www.excelsior-usa.com/jetinternals.html
http://lists.nongnu.org/archive/html/chicken-users/2011-03/m...
So it looks like PHP is a good deal slower than python, and certainly perl. 22 times slower than C..
Assuming the bash program was: `echo "hello, world!"`
Edit: I likely am mistaken. Chances are this was done using the builtin 'echo', not '/bin/echo'. The builtin is always faster (though putting them in a file (as is the only fair way to do it) slows them both down of course).
Not that this benchmark is actually indicating anything about which language to use in any of those circumstances (Java has a bad startup but pretty good during runtime, and startup time on these orders is generally irrelevant for any nontrivial application), but that isn't to say that language benchmarks in general are irrelevant.
And a poorly maintained C program will outperform an equally poorly maintained Java program...
The things that make programs dramatically slower is poor algorithmic complexity (and often a poor understanding of the architecture, be it the JVM or the metal). That's not what we're talking about here though, this discussion is pretty explicitly about start-up times. We're talking constant-time issues.
For some more info: http://harmful.cat-v.org/software/dynamic-linking/
EDIT: a micro-benchmark between static and dynamic C program:
...and that's just libc alone! Those times were quite stable across several runs, so it's not some random one-time fluctuation.The code:
Hm... huh?
http://en.wikipedia.org/wiki/Irony_punctuation
What a misleading example. That's not GNU hello world, that's Linux hello world. The difference: it does not use GNU libc nor any other parts of GNU project -- except for the `unistd.h' -- but instead interfaces with the kernel directly.
Another matter is that it's not a C program, as it doesn't have int main(...) :P
[/nitpick mode]
On the contrary:
- The program is compiled with GCC
- "GCC development is a part of the GNU Project..."
For sake of example, consider Linux (the kernel); usually compiled with GCC -- but ICC works as well -- and distributed under GPL, yet is not part of GNU project.
Moreover, the program in question can be ran directly on bare Linux, with no GNU stuff present anywhere, if passed via `init=PATH_TO_THE_PROGRAM' boot param. Of course the kernel would panic right after the syscall(__NR_exit, 0)... so I'm bailing out of nitpick mode right now ;-)
Before your kernel panic out of nitpick mode, shouldn't you at least have compiled and run that hello world program with ICC? ;-)
Again, with the rapid unexplained downvotes... seriously, did you see the article about speeding up Eclipse? The suggestion was to make sure you're not using gcj and others commented that they agree'd. It's well known the gcj is just flat slow compared to Oracle's/OpenJDK.
lol, whatever guys, pile on, I must be missing the joke!
Eclipse is not representative of a typical command line program. Write "ls" with the OpenJDK and tell me you don't pull you hair out waiting for it to start up when using it.
But I can see how compiling a simple program with gcj could make it faster.
This article is basically trolling java, and it seems a number of people have taken the bait. Different languages have different strengths and weaknesses. You'll choose the one most appropriate to the task required. If that's printing "Hello World" to the console then fine, but I doubt many people would think "java" when presented with this requirement.
When "writing shell scripts / cron jobs / random commandline utilities" I believe my priorities are
1) Does it work? 2) How easy is this to write? 3) How easy is this to read?
In that order.
You can disable it using the -S option if/when you care about startup time.
Here are some numbers on my MacBook Pro:
$ ./run.sh bash hello.sh
median 0.002
$ ./run.sh python hello.py
median 0.016
$ ./run.sh python -S hello.py
median 0.008
"Astonishing indeed. The whole thing confirmed my hunch that Bash, Chicken, and C provide a complete toolset to tackle 99.9% of programming needs."
No no no! Try running anything non-trival in bash or python and compare that to the java. Obviously if all you ever want to do with programming is write little boring scripts then you shouldn't be using a language like java in the first place. For anything mildly interesting and performance intensive Java is pretty much a no brainer only behind c/c++ amongst the major languages. For a site with REAL benchmarks which actually mean anything unlike the language fanboyism in this posted article checkout http://shootout.alioth.debian.org/u64q/which-programming-lan...
That does not seem to be the use case that's being examined.
The use case that's being examined is - "writing shell scripts / cron jobs / random commandline utilities".
http://shootout.alioth.debian.org/gp4/benchmark.php?test=hel...
But "if all you ever want to do with programming is write little boring scripts" it might help to understand that 9 hundredths of a second delay is below the threshold that someone using the boring little script will notice.
And "if all you ever want to do with programming is write little boring scripts" simply using Java as an interpreter with -Xint might be faster.
For example, that old benchmarks game n=200 "hello world"