Good question. I've already shut down the EC2 instance I used for testing, but I'll fire up another one later today and get back on this. Great point though, performance characteristics will almost certainly vary quite a bit between different filesystems.
No, it isn't. My motivation for running on EC2 was that benchmarking on a laptop is somewhat noisy due to thermal constraints [1]. I specifically used a bare-metal EC2 instance in an attempt to have a less noisy environment (since VM performance is less predictable and depends on the other workloads running on the same hardware) with proper cooling.
I totally buy your explanation. Consumer devices are extremely noisy. Back when I worked on embedded, the only tests that I could trust were running on the boxes
I lost the link, but I remember reading an article in the last year or so about tricks for benchmarking on a laptop. Various things you could do to avoid the thermal throttling. The catch was that it was only possible to do approximate, comparative benchmarks, but never get a result that showed the "true" unconstrained performance.
together with disabling noisy background services that's the most significant reduction in noise. There are further steps that can be taken but for CPU-bound microbenchmarks they only provide incremental improvements in my experience.
IO benchmarks are trickier. SSDs have thermal throttling too, there's the OS write cache, the SLC cache performance cliff and other factors.
This discussion is starting to make me think that the industry needs benchmarks that take into account modern hardware behavior better, somehow. The only benchmarking I've done is on deployed software for comparative performance tuning, determining if a given change runs faster or slower as deployed. I've long been a skeptic of benchmarks that try to test across heterogenous systems. My dad, who was a systems programmer, helped teach me how benchmarks could be deceptive.
While that's correct, it's misleading in the big picture.
On full load, any machine hits thermal constraints, so, considering clock changes a problem, a desktop is problematic as well. A desktop will certainly have a higher ceiling, and/or a laptop may have too much of a low one, but it's incorrect to assume that the problem is inherent to laptops.
The solution is therefore, which is canonical in serious benchmarks and it's not mentioned, is to ensure that the machine runs on a fixed clock rate. This is easily accomplished, at least on desktops, by changing the BIOS. I wasn't personally able to do this from the O/S (Linux).
Using a machine without control on the clock (which a cloud one can be, even if bare metal) is subject to generate inaccurate benchmark results.
Thanks! I'll check the reference document to search for something that works. In the past, I had tried (to no avail) with the `userspace` governor, `cpufreq-set`, `cpupower` and also `/sys/*/cpufreq/scaling_max_freq`; hopefully, I'll find other info in the document.
Inspired by many of the amazing new-wave Unix tools that have come about over the past several years (fzf, ripgrep, fd, lsd, etc.), I set out to create a more modern and performant version of the classic Unix cp command. Written in Rust due to the excellent ergonomics and performance characteristics of the language, fcp copies large files and directories in a fraction of the time that cp does.
Feel free to ask me any questions about the project!
_Why_ is it faster? (particularly on Linux, I see you're using syscalls on mac that mac's cp doesn't, since apple doesn't really maintain their unix userspace)
The primary reason is that it uses all of your computer's available cores to walk through directories and copy files, effectively speeding things up by a factor directly proportional to how many cores you have.
On macOS, this is augmented by regular files being copied using the OS's fclonefileat and fcopyfile mechanisms under the hood, which allows even very large files to be copied near-instantly (when compared to copying them block-by-block as classic cp does).
No, it is not. I would've submitted a Homebrew formula for it, but that would go directly against Homebrew's guidelines, which frown upon authors submitting formulas for their own tools [1].
This works out assuming that file copying is CPU bound. Traditionally this hasn't been the case but maybe things have changed with widespread fast SSDs and usage of disk crypto.
Traditionally the reason to use threads in small-requests i/o is to get multiple outstanding requests in flight in the hardware io stack which some hardware can use to get you more IOPS throughput. (They might end up hitting different disks in RAID, or taking advantage SSD internal parallelism, or whatever tricks spinning disks use to get more throughput with TCQ)
(Re your MacOS remark, Linux has support for copy-on-write file copies as well, see reflink and FICLONE / copy_file_range)
> Traditionally the reason to use threads in small-requests i/o is to get multiple outstanding requests in flight in the hardware io stack which some hardware can use to get you more IOPS throughput
I would assume that is the case here too. The threads are likely hiding the latency of the IOPS.
I can't see any CPU load statistics, so it is hard to say, if it is indeed CPU bound or not.
A quick run with time shows user+sys being only slightly higher than real, so with 6 threads for my 6 cpus, it uses a single cpu. (and practically no time in user-space)
Does this software gain any speed by sacrificing any safety? As in, does it switch off fsync in any way? Are the files guaranteed durable by the time the software terminates?
> The primary reason is that it uses all of your computer's available cores to walk through directories and copy files, effectively speeding things up by a factor directly proportional to how many cores you have.
CPU cores can't read/write files, those are IO operations by the storage device. The CPU could have 64 cores but on a system with a single hard disk that can do just one concurrent operation.
My only gripe with those new unix tools is that many of them don't come with a man page, even though they have essentially the same information elsewhere
This assumes README.md lists all the command-line options like in a manpage, which is not often the case. Also, manpages follow a conventional structure. You can expect the content in a manpage. Most READMEs are free-style. You don't know what is there. md2man doesn't solve the problem.
Well tldr (or tealdeer) or whatever is a nice example of another modern (semi-)alternative to man. tldr usually just gives a handful of explained example commands rather than an actual manual, but it's often enough information to solve the problem. I bet these some of these tools have pages on that at least.
(I agree that a real manual is necessary too. Maybe you could convert one from the online docs?)
Not a question, but I made a similar tool in c++[1], for Linux only using io_uring, and a blog post explaining its internals [2]. I'll definitely have a look some time soon, I'd be interested to see how performance compares (I gathered from some other comments here that you're using blocking io in threads?)
Sounds super interesting, I'll definitely check out the blog post. Yes, fcp is indeed using blocking io in threads. In an earlier prototype I was using async (i.e. M on N threading), but was running into issues with exhausting file descriptors. Because of this (and because writing async Rust is quite difficult) I settled on using blocking io in threads. Perhaps in a future version I'll try to implement the io_uring approach for Linux.
Ah, I also tried unlocking the file descriptor 1024 limit, which worked fine most of the time, but didn't actually give me a perf boost in any of my tests. I got rid of it in the end because it caused problems when copying from an smb mount. iirc I exhausted the file descriptor limit on the samba server.
He might've gotten confused by newer filesystems like btrfs which let you create links to files instantly which effectively work exactly like a copy.
The second file however isn't actually created until a change has occurred, so the linking is only an entry in the catalog to the inodes i believe.
The only thing I'd like "new" in cp is an optional progress indicator, for large files
Btw, copying files is mostly bound by disk speed (or network if share), not so much by CPU, so it doesn't seem to be a valid reason to rewrite it in Rust for cpu performance...
It looks like this is using multiple threads to copy files. That's fine for NVMe which tends to achieve more throughput with higher queue depths. But it will degrade performance when copying large files on spinning rust.
On a SATA hard drive using ext4, I copied a directory with a large hierarchy of file and 216MB overall. After 2 warm-ups, I launch this simple benchmark.
time cp -r moodle X
13.76s s (0.06s u + 2.44s k) 18% CPU
time fcp-0.1.0-x86_64-unknown-linux-gnu moodle X
46.12s s (0.15s u + 2.47s k) 3% CPU
I emptied linux cache before each run with `echo 1 > /proc/sys/vm/drop_caches`, but this does not remove the HDD buffering. I suspect that fcp would be worse in real life conditions (no warm-up).
This does not mean the fcp is bad or not performant. But it some cases, it can be 3x slower.
Since you've already started, can you do a benchmark of a pair of copies started at same time?
One copy operation is the little files and deep hierarchy. The other copy operation is just a big file. Both operations start at same time and use the same tool.
I'm curious what cp and fcp do to the cache as they operate.
What do you mean by "After 2 warm-ups"? I've seen this expression in the context of benchmarks before and it puzzles me.
Does it mean you are doing this operation enough times to trick the kernel to cache everything in RAM? Or you run it 3 times and chose the fastest timings?
I'm not sure about the specific context here, but benchmark warm-ups are often used to make sure that the program's run time is nothing but that, and doesn't include things like setting up an environment or compiling to a cache.
I think there are sometimes arguments to be made that, for the user, that is run time. When using benchmarks to compare programs like this, though, I think isolating the "actual" (i.e. useful) program execution is productive.
Hah, this is amazing! I wish they taught that in schools. I don't know who they are but they're like a mix between Bill Nye, Carl Sagan, and the Mythbusters. Cool duo.
Great old video. Thanks for sharing. Terrifying goose though.
This was definitely an oversight on my part; the benchmarks I ran were all conducted on systems with SSDs (as noted in the README). Especially considering the testing done in some of the comments below this one, I'll update the README to mention that fcp is geared towards SSDs. As a developer tool though, I still think this is the right choice, as at least anecdotally the vast majority of developers I know have an SSD in their primary machine.
Unfortunately I find it's not enough to copy large files, you need to verify them as well. I don't trust the hard drive controller or the nvme controller to have not f'ed something up.
You're in the wrong abstraction layer. There are a whole bunch of intervening levels between the layer where you've imagined that you "need to verify them as well" and the layer where the "hard drive controller" that you don't trust lives.
Those layers may conceal a real problem (if there is one) so that you can't see it even though it exists, or they can create problems, which you will then mistakenly blame on the "hard drive controller" even if it was not at fault.
So you've created a situation where you can at best confuse yourself into not understanding what's actually going on, and at worst satisfy yourself nothing is wrong when actually there's a catastrophe awaiting you. The worst of all worlds.
For SSDs this is probably the right thing (maybe use uring tho?)
For hard drives this is the opposite of what you want. For hard drives, especially when copying large trees, you only want one thread to access the disk, and preferably sort its stat()s and open/read/close() by inode.
Also this tool seems to lack most options that cp(1) has.
> preferably sort its stat()s and open/read/close() by inode
A better approach would be to use information about location of data extents. Luckily there's already a crate that can help with that: https://github.com/the8472/platter-walk
That's better for reading the data, but doesn't work for ordering stat()s because looking up extent information requires reading the inode, but that's what we're trying to make linear-ish at that point.
you can optimize for both. obtain directory entries, sort by inode, do a batch stat (ideally via io_uring) and fiemap and then sort again by extent order.
And if you only need the `d_type` instead of the full stat struct, then you get that for free from getdents(2) on some filesystems, this is what platter_walk does.
OK, this is something that's bugged me for a long time. I've been in the field forever, but working on Windows. So I've seen this kind of thing many times before, particularly when referencing man, and it seems completely cryptic.
The way linux people occasionally put a parenthetic number after a generic command seems to me one of the ways that normal civilians, and even pros like me, are intimidated into staying away from serious linux usage.
So what gives? What the heck is the parenthetic number for?
The number denotes what section of the manual the command (or function, etc.) comes from. Aside from that, it also helps clear up potential confusion about what you’re talking about—a good example is stat(1) (the command) and stat(2) (the system call). Without the number, you’d have to rely on context clues to infer which stat was the relevant one.
https://unix.stackexchange.com/questions/3586/what-do-the-nu...
I can’t lie, if you spend most of your time in Windows, there’s probably diminished reasons to understand manpages, but for those that do any degree of meaningful work on a Linux or Unix (e.g. macOS) system, not knowing how to navigate man pages separates entire classes of developer professionalism to me.
It’s night and day working with people who know how to be approachable in development, but also realize you need to understand some basic flags for everyday tools and not turn to Google or Stack Overflow for what is directly in your manpages.
Admittedly those who cannot stay approachable concoct all sorts of shell hell for everyone, even those who are experienced.
Excellent question, by the way. I wish we’d get some more candid questions like these on HN when relevant to the thread, like this one. It’s a learning opportunity for all readers.
What's wrong with Stack over man pages? Most man pages don't you give advice or context, just the raw reference, but sometimes the best way to do something is not even the tool you originally looked up (say, scp vs rsync).
Stack combines the reference (which flags) with the expertise (of educated guesses by sometimes wiser users).
I would never trust a man page over a good Stack answer. (edit: as a web dev who uses *nix all day, but has no desire to become a full fledged sysadmin... Just give me the tool I need to get the job done without sacrificing performance or security. I don't need to know every possible gnu tool and flag and distro dependent options...)
But I have a bigger issue with the fact that developers who use Stack Overflow are more often than not just going to copy and paste and not understand what they’re doing.
And that’s predominantly how the site is used.
Yeah, there are plenty of manpages that are verbose, but frankly a lot of them are easier to parse than reading hacked together Stack Overflow answers that don’t explain how the poster arrived at their answer.
one of the things that surprised me about large cluster filesystems (although it's obvious in retrospect) is that the solution to scaling performance is many threads, not one, because the backend can handle many concurrent requests. In fact at one place I worked, the cluster filesystem wouldn't even take you seriously until you hit it with GB/s of requests.
Definitely valid points, as I mentioned in a comment below I'll be updating the README to mention this is geared towards systems with SSDs.
As for lacking most of cp(1)'s options, that's a deliberate choice. My goal with fcp is to make the most common use case of cp(1) (i.e. copy some files/directories with no options) fast, not to replace cp(1) completely.
“rewrite (the easy parts of) $battle-tested-ancient-coreutil in rust” seems interesting as an exercise or as a way to learn a new language, but i’m not convinced of it’s broader utility.
i don't think it was ever meant to be "exactly GNU/coreutils grep", and I don't think it ever even tried. The API is totally different, i don't think it's ever been the same, it scratches a similar-but-not-identical itch.
I would argue that "the easy parts" of cp account for a very significant fraction of its usage (i.e. copy some files/directories without any other options). The idea with fcp is to make that common use case very fast, not to replace cp and all the options it handles.
I definitely can understand the fatigue with "rewrite it in Rust" though, as there is a lot of that.
Something you might be interested in is that people have actually gone through the effort of rewriting the actual coreutils to spec in Rust (https://github.com/uutils/coreutils).
There was a tool for Solaris called "mtwrite" that would use LD_PRELOAD to intercept writes and farm them out to threads. This way, it would work with not just cp, but also tar extracts to files, etc.
summary: this is a nice cow/cloning wrapper for use on a single ssd partition!
-- but cloning not copying between volumes or partitions is .. and when you actually want to copy with fcp, it can be very slow as detailed in other comments.
I worked with a large cluster filesystem and it had some interesting properties. In particular, files were effectively append-only and immutable when closed, but the FS supported "snapshot for append" which allowed to to make a snapshot of the file, and append to it. Under the hood it managed all the pointers but when the FS designers found out we used the feature heavily they got worried.
Why do people think it is acceptable for utility programs like file copies to launch many threads and saturate my CPU? It's usually the case that there are many things being done on the system, and I don't want one task take up all the resources I have. Most classic Unix command line tools only use a single thread, so I know they won't bog down the entire system without any effort on my part (adjusting niceness, cgroups, ulimit, etc).
I'm afraid the only way there can be a faster file copying tool is by sacrificing contiguousness and increasing fragmentation. Isn't this the case?
I want fragmentation to be as little as possible so it wouldn't be too hard to manually recover a file if I delete something accidentally or damage the file system.
86 comments
[ 2.0 ms ] story [ 167 ms ] thread[1] https://lemire.me/blog/my-sayings/
Don't take my word for it, Prof. Lemire is the expert on performance.
IO benchmarks are trickier. SSDs have thermal throttling too, there's the OS write cache, the SLC cache performance cliff and other factors.
On full load, any machine hits thermal constraints, so, considering clock changes a problem, a desktop is problematic as well. A desktop will certainly have a higher ceiling, and/or a laptop may have too much of a low one, but it's incorrect to assume that the problem is inherent to laptops.
The solution is therefore, which is canonical in serious benchmarks and it's not mentioned, is to ensure that the machine runs on a fixed clock rate. This is easily accomplished, at least on desktops, by changing the BIOS. I wasn't personally able to do this from the O/S (Linux).
Using a machine without control on the clock (which a cloud one can be, even if bare metal) is subject to generate inaccurate benchmark results.
https://www.kernel.org/doc/html/v4.12/admin-guide/pm/intel_p...
E.g., if you're using the Intel pstate driver in active mode, you can disable turbo via:
(and re-enable by writing "0".)Feel free to ask me any questions about the project!
On macOS, this is augmented by regular files being copied using the OS's fclonefileat and fcopyfile mechanisms under the hood, which allows even very large files to be copied near-instantly (when compared to copying them block-by-block as classic cp does).
[1] https://docs.brew.sh/Acceptable-Formulae#niche-or-self-submi...
Traditionally the reason to use threads in small-requests i/o is to get multiple outstanding requests in flight in the hardware io stack which some hardware can use to get you more IOPS throughput. (They might end up hitting different disks in RAID, or taking advantage SSD internal parallelism, or whatever tricks spinning disks use to get more throughput with TCQ)
(Re your MacOS remark, Linux has support for copy-on-write file copies as well, see reflink and FICLONE / copy_file_range)
I would assume that is the case here too. The threads are likely hiding the latency of the IOPS. I can't see any CPU load statistics, so it is hard to say, if it is indeed CPU bound or not.
A quick run with time shows user+sys being only slightly higher than real, so with 6 threads for my 6 cpus, it uses a single cpu. (and practically no time in user-space)
CPU cores can't read/write files, those are IO operations by the storage device. The CPU could have 64 cores but on a system with a single hard disk that can do just one concurrent operation.
So, perhaps, take a look at https://github.com/rust-cli/man or something
(I agree that a real manual is necessary too. Maybe you could convert one from the online docs?)
1: https://github.com/wheybags/wcp
2: https://wheybags.com/blog/wcp.html
That said, neither of these are truly the same as an edit to one will impact the other.
i.e. btrfs snapshots use this feature.
When copying between different filesystems it still makes sense to add the --sparse=auto flag to keep a copy of a sparse file sparse, too.
My alias for cp is:
Btw, copying files is mostly bound by disk speed (or network if share), not so much by CPU, so it doesn't seem to be a valid reason to rewrite it in Rust for cpu performance...
On a SATA hard drive using ext4, I copied a directory with a large hierarchy of file and 216MB overall. After 2 warm-ups, I launch this simple benchmark.
I emptied linux cache before each run with `echo 1 > /proc/sys/vm/drop_caches`, but this does not remove the HDD buffering. I suspect that fcp would be worse in real life conditions (no warm-up).This does not mean the fcp is bad or not performant. But it some cases, it can be 3x slower.
One copy operation is the little files and deep hierarchy. The other copy operation is just a big file. Both operations start at same time and use the same tool.
I'm curious what cp and fcp do to the cache as they operate.
Does it mean you are doing this operation enough times to trick the kernel to cache everything in RAM? Or you run it 3 times and chose the fastest timings?
I think there are sometimes arguments to be made that, for the user, that is run time. When using benchmarks to compare programs like this, though, I think isolating the "actual" (i.e. useful) program execution is productive.
Second run: fcp binary is cached by the OS in RAM so it starts up faster.
Third run: just to make sure :)
This is just one of the things of a type of cache that can be warmed up.
I learned this the hard way when dismantling and abusing a dead HDD. Had to vacuum all the tiny shards from the floor.
https://youtu.be/g1JlUcFKm5o?t=537
Great old video. Thanks for sharing. Terrifying goose though.
Those layers may conceal a real problem (if there is one) so that you can't see it even though it exists, or they can create problems, which you will then mistakenly blame on the "hard drive controller" even if it was not at fault.
So you've created a situation where you can at best confuse yourself into not understanding what's actually going on, and at worst satisfy yourself nothing is wrong when actually there's a catastrophe awaiting you. The worst of all worlds.
For hard drives this is the opposite of what you want. For hard drives, especially when copying large trees, you only want one thread to access the disk, and preferably sort its stat()s and open/read/close() by inode.
Also this tool seems to lack most options that cp(1) has.
A better approach would be to use information about location of data extents. Luckily there's already a crate that can help with that: https://github.com/the8472/platter-walk
And if you only need the `d_type` instead of the full stat struct, then you get that for free from getdents(2) on some filesystems, this is what platter_walk does.
OK, this is something that's bugged me for a long time. I've been in the field forever, but working on Windows. So I've seen this kind of thing many times before, particularly when referencing man, and it seems completely cryptic.
The way linux people occasionally put a parenthetic number after a generic command seems to me one of the ways that normal civilians, and even pros like me, are intimidated into staying away from serious linux usage.
So what gives? What the heck is the parenthetic number for?
Roughly, 1 is unprivileged shell commands, 2 is syscalls, 3 is library functions, and 8 is administrative commands.
It’s night and day working with people who know how to be approachable in development, but also realize you need to understand some basic flags for everyday tools and not turn to Google or Stack Overflow for what is directly in your manpages.
Admittedly those who cannot stay approachable concoct all sorts of shell hell for everyone, even those who are experienced.
Excellent question, by the way. I wish we’d get some more candid questions like these on HN when relevant to the thread, like this one. It’s a learning opportunity for all readers.
Stack combines the reference (which flags) with the expertise (of educated guesses by sometimes wiser users).
I would never trust a man page over a good Stack answer. (edit: as a web dev who uses *nix all day, but has no desire to become a full fledged sysadmin... Just give me the tool I need to get the job done without sacrificing performance or security. I don't need to know every possible gnu tool and flag and distro dependent options...)
But I have a bigger issue with the fact that developers who use Stack Overflow are more often than not just going to copy and paste and not understand what they’re doing.
And that’s predominantly how the site is used.
Yeah, there are plenty of manpages that are verbose, but frankly a lot of them are easier to parse than reading hacked together Stack Overflow answers that don’t explain how the poster arrived at their answer.
They literally don’t show their work.
As for lacking most of cp(1)'s options, that's a deliberate choice. My goal with fcp is to make the most common use case of cp(1) (i.e. copy some files/directories with no options) fast, not to replace cp(1) completely.
i don't think it was ever meant to be "exactly GNU/coreutils grep", and I don't think it ever even tried. The API is totally different, i don't think it's ever been the same, it scratches a similar-but-not-identical itch.
https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pos...
I definitely can understand the fatigue with "rewrite it in Rust" though, as there is a lot of that.
Something you might be interested in is that people have actually gone through the effort of rewriting the actual coreutils to spec in Rust (https://github.com/uutils/coreutils).
http://www.maier-komor.de/mtwrite.html
summary: this is a nice cow/cloning wrapper for use on a single ssd partition!
-- but cloning not copying between volumes or partitions is .. and when you actually want to copy with fcp, it can be very slow as detailed in other comments.
GNU Coreutils' cp, the thing you use on most GNU/Linux systems, is not the "Classic Unix cp".
It has options for copy-on-write cloning via kernel-specific methods.
COW copying is opt-in probably because it's not a true copy. If either file sustains damage, both are toast.
I want fragmentation to be as little as possible so it wouldn't be too hard to manually recover a file if I delete something accidentally or damage the file system.