Interesting article, but do many people use curl for binary downloads? I thought most sysadmins used it for viewing headers and/or parsing raw text. For binary downloads wget seems to be favoured. So with that in mind, I'd be curious to see how wget compared with the other solutions in those benchmarks.
DISCLAIMER: I'm not trying to imply that my way is "better" nor the only way HTTP GETs should be called. I'm only commenting on how I generally work and the trend I've anecdotally observed. Though I'd be interested to see if many others on here do prefer curl over wget for pulling binary files.
> I'd be interested to see if many others on here do prefer curl over wget for pulling binary files.
This is how I ended up consistently prefering curl:
Last time I looked closely, I had better control with curl, and curl had better support for stuff. For example HTTP 1.1 was supported by curl, but not wget.
This makes curl a far better choice for testing HTTP servers.
And next, since I now know curl pretty well, and have it installed all over the place, it is my go-to HTTP command line client. I don't know of any reason to use both, or any reason to prefer wget, although in some situations maybe "wget <url>" acts slightly more like you want than "curl <url>". (Which is of course easy to work around by using "curl -OL <url>", if that's what you want, but convenient defaults are still convenient)
> "Last time I looked closely, I had better control with curl, and curl had better support for stuff. For example HTTP 1.1 was supported by curl, but not wget.
> "This makes curl a far better choice for testing HTTP servers."
Well yes. You're just reiterating what I said about curl being great for server testing. But this article is about download performance. When just downloading a largish binary file (eg a tarball), it's easier to just:
wget example.com/source.tar.gz
With that, you get a progress bar, it saves the file to disk and basically just does all the sane options you'd want for downloading by default (which I think is why most INSTALL / README docs tend to recommend wget in their instructions).
Again, I'm not saying one is better than the other, but the convenience of wget's defaults (which I think you also highlighted later in you post) does tend to make me favour it for downloading binary content (with curl being my go-to for any server testing or text resources).
> "And next, since I now know curl pretty well, and have it installed all over the place, it is my go-to HTTP command line client."
wget also comes as part of the basic install with almost all Linux and other UNIX-like OSs as well. In fact wget actually pre-dates curl (albeit they're both >= 15 years old, so there's not really much between the two relatively speaking) so I'd have expected even your oldest live systems would still have wget.
I can totally relate to the "know[ing] curl pretty well" point though. There is definitely an argument for reusing the tools that you're already familiar with.
We seem to be in full agreement. I only wanted to spell out my own reasoning for using curl and not wget, since you explicitly said you were interested in this information, and I suppose there are others that think like me in this regard as well :)
One popular UNIX-like OS that comes with curl and not wget is OS X. That might also drive adoption.
(I seem to remember having had to install both curl and wget on Ubuntu, but I am unable to verify this now)
> "We seem to be in full agreement. I only wanted to spell out my own reasoning for using curl and not wget, since you explicitly said you were interested in this information, and I suppose there are others that think like me in this regard as well :)"
I'm grateful for your input. Sorry if I came across as elitist - that wasn't my intention but in reflection, in does read a little that way.
> "One popular UNIX-like OS that comes with curl and not wget is OS X. That might also drive adoption."
Ahh interesting point. I guess that would have an impact.
Yeah, I've spent some time playing with libcurl as well. However I hadn't considered your point about wget being out of scope because this is about libraries rather than utilities. Good point :)
That said, including JVM startup time is a very odd way to benchmark HTTP libraries. Is it a common enough use case to make Java apps that just start up and download a single file?
[13:17:18] i13:klausa:~% uname -a
Darwin i13.local 12.4.0 Darwin Kernel Version 12.4.0: Wed May 1 17:57:12 PDT 2013; root:xnu-2050.24.15~1/RELEASE_X86_64 x86_64
[13:17:19] i13:klausa:~% curl
curl: try 'curl --help' or 'curl --manual' for more information
[13:17:22] i13:klausa:~% wget
wget: Command not found.
A demonstration about how OS X ships curl but not wget. Essentially he's just running
which {curl,wget}
to demonstrate that curl is in $PATH where as the shell cannot find wget (since we're talking about base installs it's safe to assume that the aforementioned utilities should be in $PATH by default - if included at all).
Curl is the default on most systems I've used. The only place wget seems to be a default is on Ubuntu (and probably some other flavors of Linux).
But most places where I need such a tool: My desktop (Mac), the load-balancers (OpenBSD), the database servers (FreeBSD), the hypervisors (SmartOS), the NFS servers (FreeBSD), basically all the infrastructurey stuff, curl is the default. I wouldn't swear to it, but I don't think wget is installed on most of those systems.
I use curl across the board. I can't even begin to count the amount of times that I have seen script kiddies use some pre-made script to hack {Wordpress, Drupal, Joomla, whatever other PHP script} and try to use wget to download their new code. wget isn't installed on my servers, so it fails. curl is installed, but they never seem to reach for that.
Keeping wget off my servers has stopped countless attacks, yes the PHP scripts should be updated... but shared hosting isn't all that great as a sys-admin.
It's an interesting subject, I just wish his benchmarks were much more indepth.
I would like to how the amount of sequential iterations of the code, the HTTP Server, JVM, OS, kernel, latency, jitter all affect the performance.
Yes. JVMs are slow for short-lived program executions compared to native code. By default, Hotspot does not compile a method until it is executed 10K times. Most libraries in the Java ecosystem are written to optimize performance over a long execution lifespan at the possible expense of startup time. The Apache HTTP libraries allow pooling of threads, HTTP keep-alive as an implementation detail, buffer re-use, etc. So, evaluating these libraries by starting up a new JVM for each run inherently tests them under a circumstance for which they were not designed/optimized. Gradle (a build tool) sort of solves the problem of short-run invocations by forwarding command line requests to a long-running Java background daemon.
For those (like me) who didn't know what NIO was, and got lost in the second paragraph, from Wikipedia:
"The APIs of NIO were designed to provide access to the low-level I/O operations of modern operating systems. Although the APIs are themselves relatively high-level, the intent is to facilitate an implementation that can directly use the most efficient operations of the underlying platform."
> NIO has significant impact on HTTP Servers not on stand alone single instance clients.
This is true for the asynchronous network IO parts of NIO, but definitely not all of it. It also enables memory mapping and DMA, two things that can have significant impact on stand alone software.
Thanks for doing this! A very useful set of results!
I'd be curious to see how these each behave when needing to manage concurrent HTTP connections, fetching files from an ultra-fast server such as Netty, nginx, or Undertow.
22 comments
[ 3.3 ms ] story [ 54.7 ms ] threadDISCLAIMER: I'm not trying to imply that my way is "better" nor the only way HTTP GETs should be called. I'm only commenting on how I generally work and the trend I've anecdotally observed. Though I'd be interested to see if many others on here do prefer curl over wget for pulling binary files.
This is how I ended up consistently prefering curl:
Last time I looked closely, I had better control with curl, and curl had better support for stuff. For example HTTP 1.1 was supported by curl, but not wget.
This makes curl a far better choice for testing HTTP servers.
And next, since I now know curl pretty well, and have it installed all over the place, it is my go-to HTTP command line client. I don't know of any reason to use both, or any reason to prefer wget, although in some situations maybe "wget <url>" acts slightly more like you want than "curl <url>". (Which is of course easy to work around by using "curl -OL <url>", if that's what you want, but convenient defaults are still convenient)
> "This makes curl a far better choice for testing HTTP servers."
Well yes. You're just reiterating what I said about curl being great for server testing. But this article is about download performance. When just downloading a largish binary file (eg a tarball), it's easier to just:
With that, you get a progress bar, it saves the file to disk and basically just does all the sane options you'd want for downloading by default (which I think is why most INSTALL / README docs tend to recommend wget in their instructions).Again, I'm not saying one is better than the other, but the convenience of wget's defaults (which I think you also highlighted later in you post) does tend to make me favour it for downloading binary content (with curl being my go-to for any server testing or text resources).
> "And next, since I now know curl pretty well, and have it installed all over the place, it is my go-to HTTP command line client."
wget also comes as part of the basic install with almost all Linux and other UNIX-like OSs as well. In fact wget actually pre-dates curl (albeit they're both >= 15 years old, so there's not really much between the two relatively speaking) so I'd have expected even your oldest live systems would still have wget.
I can totally relate to the "know[ing] curl pretty well" point though. There is definitely an argument for reusing the tools that you're already familiar with.
One popular UNIX-like OS that comes with curl and not wget is OS X. That might also drive adoption.
(I seem to remember having had to install both curl and wget on Ubuntu, but I am unable to verify this now)
I'm grateful for your input. Sorry if I came across as elitist - that wasn't my intention but in reflection, in does read a little that way.
> "One popular UNIX-like OS that comes with curl and not wget is OS X. That might also drive adoption."
Ahh interesting point. I guess that would have an impact.
https://www.gnu.org/software/wget/manual/html_node/Recursive...
And libcurl is used quite extensively to download all sorts of things, including binaries. Git, for example, makes heavy use of curl for http remotes.
But most places where I need such a tool: My desktop (Mac), the load-balancers (OpenBSD), the database servers (FreeBSD), the hypervisors (SmartOS), the NFS servers (FreeBSD), basically all the infrastructurey stuff, curl is the default. I wouldn't swear to it, but I don't think wget is installed on most of those systems.
Keeping wget off my servers has stopped countless attacks, yes the PHP scripts should be updated... but shared hosting isn't all that great as a sys-admin.
"The APIs of NIO were designed to provide access to the low-level I/O operations of modern operating systems. Although the APIs are themselves relatively high-level, the intent is to facilitate an implementation that can directly use the most efficient operations of the underlying platform."
http://en.wikipedia.org/wiki/New_I/O
The only time you'd use a NIO client, is on a server - for example, when building a caching proxy server.
The expected benefits are higher throughput of the proxy (lower CPU/RAM consumption allows multiple clients to operate in parallel).
Individual downloads being faster was never expected to be a benefit
This is true for the asynchronous network IO parts of NIO, but definitely not all of it. It also enables memory mapping and DMA, two things that can have significant impact on stand alone software.
I'd be curious to see how these each behave when needing to manage concurrent HTTP connections, fetching files from an ultra-fast server such as Netty, nginx, or Undertow.
Say 64, 128, 256, maybe even more simultaneously.