Software can't give you faster download speeds. If I get 10 Mbps down from my ISP I'm not going to somehow download at 50 Mbps. yt-dlp, steam, and firefox already max out my connection when they download something.
To nitpick: Software can give you faster effective download speeds. Compression is software. Parallelization across multiple hosts is achieved in software.
It really depends on the cause of the bottleneck. Sometimes you need capabilities on both ends though.
Also, while I'm nitpicking: "xdm" is and will always be the X Display Manager. :(
> Multi-threaded fragment downloads: Download multiple fragments of m3u8/mpd videos in parallel. Use --concurrent-fragments (-N) option to set the number of threads used
Not all sites are willing to max out your pipe with a single connection. Making multiple connections to those sites often gets around that.
It can't get you faster than your ISP's throttling, but it can get you faster than the server you download from's throttling.
I've been stuck with a throttled 10mbit fiber connection before, and still had times where sites would throttle me more than that. Now that I have a gigabit connection, it's a much more common problem.
The bottleneck is typically at the server level, not the pipe.
Steam uses BitTorrent, which is built to parallelize downloads - effectively implementing what download managers do. I think Firefox these days also does clever stuff behind the scenes. The result is downloads that complete faster than before, even though the theoretical max speed of any given connection is the same.
Download managers are somewhat old-fashioned simply because modern software learnt from them and now tries their tricks out of the box.
Sometimes the bottleneck is on the server end, and often they throttle throughput per connection. Some download mangers allow you to download the same file through multiple connections and range request. Having said that I think it's a little bit rude to work around the intended throughput limitation.
I used a lot of download managers and accelerators back in the dial-up times, they seemed like magic to me, plus being able to resume a download after somebody picked up the phone and dropped my internet connection was a plus!
How is it that they work, that is the accelerator part? My guess is an algo to find the optimal amount of parallel downloads?
Edit: also with parallel downloads, is it possible to do a HTTP GET with an bytes offset? With video I guess it’s possible to select what part to stream, but with regular http download I’m not sure
> an algo to find the optimal amount of parallel downloads
An algo...? Most download managers I tried simply had a set amount of threads they tried to open; the smartest ones will close some of those connections if they think they are being throttled, but that's rare.
That's just all there is to it: single-threaded downloads are severely inefficient in the age of multithreaded CPUs (and of operating systems without a finely-tuned network stack), so by leveraging parallelization anywhere you get a chance, you multiply the speed.
> the smartest ones will close some of those connections if they think they are being throttled, but that's rare.
And some of them reset and restart the download, in case that the bottleneck is neither in the machine nor the server (for example, some routers will throttle long-lived connections).
If the app uses a single execution thread, the OS will likely be unable to distribute work across different CPU threads. Whereas if you create multiple threads, the OS can max out CPU and I/O resources efficiently.
Nope. Starting at the NIC, it's not "single-threaded." Intel NIC drivers on most platforms service the multiple buffers using several available physical cores. Even if it were single-threaded, memory bandwidth over a decade ago was around 3GB/sec.
Download "acceleration" is snake oil and all "acceleration" does is consume unnecessary resources on both ends.
Download "managers" that can queue up a list of downloads, schedule downloads (say for when you're asleep), automatically retry/partial-resume a big download, throttle a download to a low rate, and so on? That's useful for some folks.
Edit: it's written in java. I'm laughing my ass off.
> Download "acceleration" is snake oil and all "acceleration" does is consume unnecessary resources on both ends.
If there are any network bottlenecks, including at the host, that are enough to stop you saturating your own last-mile link with a single stream, that is simply not true. Or if the sending end sets per-stream limits. Downloading multiple streams from the same source can significantly speed up the process. Maybe it often doesn't, but that doesn't make it snake-oil.
> memory bandwidth over a decade ago was around 3GB/sec
Memory bandwidth is practically irrelevant if the main bottleneck is the remote network or something between you and it.
Not sure the Latin alphabet is wide enough for all the acronyms we need in recent years. :)
Edit: reminds me of a story. Supposedly (!) the reason "Xkcd" is named so is because at the time of creation there were no Google results for that letter combination.
Together with the browser extension, I've successfully used it to download hundreds of videos. There are many other popular video downloader browser extensions, but often they contain spyware. Being open source gives me slightly more confidence in XDM.
I uses Video DownloadHelper (Firefox addon) with JDownloader2 (Java standalone app), I only use Video DownloadHelper to grab the video header and send it off to JD2 to do the download and conversion (via FFmpeg).
The video platform that I couldn't find a way to download is JW Player. I tried various extensions (Firefox/Chrome) and standalone apps, they couldn't download the video stream from JW Player. The only thing they could grab is m3u/m2u8 file and that it.
46 comments
[ 3.0 ms ] story [ 91.5 ms ] threadIt really depends on the cause of the bottleneck. Sometimes you need capabilities on both ends though.
Also, while I'm nitpicking: "xdm" is and will always be the X Display Manager. :(
> Multi-threaded fragment downloads: Download multiple fragments of m3u8/mpd videos in parallel. Use --concurrent-fragments (-N) option to set the number of threads used
Not all sites are willing to max out your pipe with a single connection. Making multiple connections to those sites often gets around that.
I've been stuck with a throttled 10mbit fiber connection before, and still had times where sites would throttle me more than that. Now that I have a gigabit connection, it's a much more common problem.
Steam uses BitTorrent, which is built to parallelize downloads - effectively implementing what download managers do. I think Firefox these days also does clever stuff behind the scenes. The result is downloads that complete faster than before, even though the theoretical max speed of any given connection is the same.
Download managers are somewhat old-fashioned simply because modern software learnt from them and now tries their tricks out of the box.
No, it downloads encrypted chunks from Steam's CDN.
How is it that they work, that is the accelerator part? My guess is an algo to find the optimal amount of parallel downloads?
Edit: also with parallel downloads, is it possible to do a HTTP GET with an bytes offset? With video I guess it’s possible to select what part to stream, but with regular http download I’m not sure
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ra...
e.g. https://explainshell.com/explain?cmd=wget+-c
Also interesting to note that it's not just HTTP it supports but other protocols I haven't heard of like DASH, HDP, HLS [1].
[0] https://github.com/subhra74/xdm/blob/master/app/src/main/jav...
[1] https://github.com/subhra74/xdm/tree/master/app/src/main/jav...
An algo...? Most download managers I tried simply had a set amount of threads they tried to open; the smartest ones will close some of those connections if they think they are being throttled, but that's rare.
That's just all there is to it: single-threaded downloads are severely inefficient in the age of multithreaded CPUs (and of operating systems without a finely-tuned network stack), so by leveraging parallelization anywhere you get a chance, you multiply the speed.
And some of them reset and restart the download, in case that the bottleneck is neither in the machine nor the server (for example, some routers will throttle long-lived connections).
Is the CPU typically the bottleneck when downloading things from the Internet?
Twenty year old CPUs had 1600MB/sec memory. 4-5 years later memory speed doubled. In 2010 it doubled again.
The Ryzen in my current desktop has memory that tops out at 25GB/sec.
Download "acceleration" is snake oil and all "acceleration" does is consume unnecessary resources on both ends.
Download "managers" that can queue up a list of downloads, schedule downloads (say for when you're asleep), automatically retry/partial-resume a big download, throttle a download to a low rate, and so on? That's useful for some folks.
Edit: it's written in java. I'm laughing my ass off.
If there are any network bottlenecks, including at the host, that are enough to stop you saturating your own last-mile link with a single stream, that is simply not true. Or if the sending end sets per-stream limits. Downloading multiple streams from the same source can significantly speed up the process. Maybe it often doesn't, but that doesn't make it snake-oil.
> memory bandwidth over a decade ago was around 3GB/sec
Memory bandwidth is practically irrelevant if the main bottleneck is the remote network or something between you and it.
* https://en.wikipedia.org/wiki/XDM_(display_manager)
* https://en.wikipedia.org/wiki/X_display_manager
Not sure the Latin alphabet is wide enough for all the acronyms we need in recent years. :)
Edit: reminds me of a story. Supposedly (!) the reason "Xkcd" is named so is because at the time of creation there were no Google results for that letter combination.
The video platform that I couldn't find a way to download is JW Player. I tried various extensions (Firefox/Chrome) and standalone apps, they couldn't download the video stream from JW Player. The only thing they could grab is m3u/m2u8 file and that it.
Nice to see there are another Java downloader.
Want to add a UI? Take a look at WebUI-Aria2: https://github.com/ziahamza/webui-aria2
It is also very easy to integrate with othersoftware via the RPC interface https://aria2.github.io/manual/en/html/aria2c.html?highlight.... You can find wrappers in some languages e.g. the JS bindings: https://github.com/sonnyp/aria2.js
Apparently it is being rewritten from Java to .Net: https://github.com/subhra74/xdm/discussions/768