I agree that other tools may be more user friendly, but if users are blindly copying and not really understanding what commands do then that is partly on them and lack of documentation. Arch Wiki provides a ton of options to copy a disk image and so it is one place that does show other options.
I honestly didn't know that dd just did the same thing as cp or cat. I thought you had to use it for making bootable USBs because it was doing some kind of special arcane magic behind the scenes .. this intuition was reinforced by the fact that so many GUI tools purporting to make bootable USBs are so unreliable. If literally all you have to do is copy bytes from one place to another, why the fuck do those tools fail so often?
Doing this kind of thing is a) rare and b) usually you have more things on your mind at the time, so you tend to be quite conservative and not think to experiment.
It doesn't really do anything particularly arcane (unless you count EBCDIC conversions) but it does things that are useful and often necessary that other equally-standard utilities can't. Isn't that good enough?
> Doing this kind of thing is a) rare
Probably not as rare as you think. Maybe it's not all that useful for writing applications, but for the quite large number of people who have to do provisioning (either bare-metal or virtual) and such it's a different matter. There's a lot to be said for tools that are ubiquitous, well standardized, and flexible.
I’d say that a big part of it is a lack of discoverability. The reason people copy blindly, especially if they’re new to Unix-likes, is that there are lots of commands with lots of flags, each of which you just kinda have to know about. And they work together reasonably well (except when they don’t), but going from “I need to accomplish this task” to a particular command is generally going to require a Google.
But that’s the issue — discoverability. How to get from “I want to search” to man grep or “I plugged in my USB drive and need to get at the files” to man mount/man fstab/man something else. man pages are a decent reference once you know what you’re looking for (assuming they’re up to date, correct, and present on your system) but they aren’t anything like a replacement for Google.
On each of the two machines I have immediate access to:
$ apropos search
apropos: Command not found.
Aside from the obvious, man pages just aren’t the answer most of the time.
Part of the issue is that a man page is a reference. This is a necessary kind of documentation, but sometimes I want a tutorial and sometimes I want a cookbook. References are great for some things, but discovering new tools isn’t generally one of them. Obviously, you don’t have to write man pages that way, but almost all of the ones I see in the wild are.
Another part is that man pages as a resource have really atrophied in my experience. Lots of new CLI tools don’t have them at all, and lots of systems don’t have them installed even for older/core tools. The why is harder. I suspect that it’s a mix of tooling (e.g. needing to learn groff/troff to write them), search (your man pages aren’t indexed by Google by default like your GitHub readme), and culture. But I’m not sure.
> But who cares? Why not just let the command figure out the right buffer size automatically?
Well, because it doesn't. At least on the Linux version I used in the past, it defaulted to 512 byte blocks or something similarly small and it commits every block individually, leading to really slow performance with USB sticks with controllers not smart enough to bundle writes together. I wouldn't be surprised if that incurs some heavy write amplification at flash level too. Perhaps it's smart enough to figure a better block size now but this is where that habit comes from.
Another thing, creating sparse files with the seek option (simply put files containing zeros that are not actually written to the disk nor taking space, but do turn up all these zeroes when you read the file). Also something not duplicated with cat or head.
What I like about dd is that it can do pretty much all disk operations in one simple tool. Definitely worth learning all of it IMO.
I don't read it that way. He calls out that 'weird' command specifically. But indeed he doesn't specify.
I wonder what cat does in terms of buffers, I kinda doubt it has any special optimisations though I would guess the shell redirect might have. As that's really the thing doing the work there, not cat. Edit: Nope, I'm wrong there!!
Also that command does more than just specify the memory buffer like he says. That's my point, it's useful for tuning which can be super helpful with huge images.
It can also lead to some dangerous gotchas as well when working with files. But with full disk images these don't apply generally.
> though I would guess the shell redirect might have. As that's really the thing doing the work there, not cat.
No? The shell redirection is just
int tmp_fd = open("/dev/sdb", O_WRONLY|O_TRUNC, 0666);
dup2(tmp_fd, STDOUT_FILENO);
close(tmp_fd);
(plus error handling and whatnot)
`cat` ends up with a file descriptor directly to the block device, same as `dd` does; the only difference is whether the `open()` call comes before or after the `execve()` call.
Right, and the parent comment is saying that `cat` doesn't figure it out very well.
I recall that GNU cat used a constant value that was tuned for read performance on common hard drives, giving no consideration to write performance. Looking at the GNU cat sources today, it doesn't look at all how I remember; I'd have to study it a bit to tell you what it does.
Edit: Hrmm, it seems I was mis-remembering. Perhaps I was thinking of the minimum values (32KiB, 64KiB, 128KiB below)? Or perhaps I was thinking of a BSD cat? Anyway:
How GNU cat sizes its buffers, by version:
- >=9.1 (2022-04-15) : Use the `copy_file_range()` syscall and let the kernel figure it out
No, I was talking about dd. As far as I know it does not have any smart tuning of block sizes. It definitely didn't in the past and I doubt it was added.
Not sure what cat does but like I said in my other comment its not really cat itself that does the disk writing in that command but rather the shell redirect. Edit: Nope, I'm wrong there!!
Ah, I misunderstood you. You're correct, `dd`'s default block size was and is 512, which is specified by POSIX.
I interpreted "the command" in the original article as "the command you end up running, whether it be `dd` or `cat` or something else."
But you're mistaken about the shell redirect. In either scenario it's the command (`dd` or `cat`) making the write() syscall to the device. The shell passes the file descriptor to the command, then gets out of the way.
Indeed I was mistaken about the shell, sorry about that. The redirect method I forgot a really long time ago and my memory made it into something it was not.
In that case I guess dd does call a sync() on every output block? Because it's definitely slower and the LED pattern on a USB stick is also much more 'flashy' when using 512 bytes.
For context, choosing the block size has implications on the speed at which data gets written to a USB stick, so not being able to tune that on cat can be a problem.
I guess that made sense when the standard HDD block size was 512b. Then it went to 512/4kb then pure 4kb, not sure what SSD's do maybe also 4kb? Experimenting with speeds and block sizes in the past has shown very quickly diminishing returns increasing the block size. As long as the CPU can keep the queue depth over 1 the device should be flat out.
Generally they do a logocal 4kb, but it's usually physically larger pages of more than 1M per write. a good ssd will helpfully cache and batch up small writes but if it gets it wrong then it'll amplify the wear and kill a drive quicker than needed. That's another reason to do dd with a larger block size, since it'll make it a lot less likely that you write multiple blocks for a single update
Good point. I guess there's so much going on with various types of caching and wear levelling that 'let the device figure it out' is best. And the queue can be on the device now with NVME not on the host so its not a dumb queue any more.
I don't have data at hand but if you choose the right value it's meaningfully better and can also lead to more efficient patterns in bash scripts that are more complicated than `cat in > out`
Unfortunately dd has not just footguns but foot cannons that are amplified by the mistakes people often make with string escaping, odd file names, loops, null checking, and conditionals in bash.
It uses a syscall per block write (which would slow it down if you use 512 byte blocks instead of 8M for example), but the OS does the file buffering and final writing to the device, unless you pass the fsync or fdatasync options to dd.
Edit:
here's writing to a old and slow stick and you can see that dd is fast and then the OS has to transfer the buffered data onto the stick:
dd if=some-2GB-data status=progress of=/dev/sdX bs=8M
1801256960 bytes (1,8 GB, 1,7 GiB) copied, 7 s, 257 MB/s
0+61631 records in
0+61631 records out
2019556352 bytes (2,0 GB, 1,9 GiB) copied, 378,59 s, 5,3 MB/s
And the stick is placed in a USB 3.2. port on a fast machine ;-0
> Can also do `oflag=direct` and it'll just skip as much of the caching as it can.
Correct, and that's one more point for dd compared to head/tail (which are fine commands by themselves).
But wouldn't help much in my example, where I used an (very) old "high speed usb 2.0 stick" with 4 MByte/s write speed to demonstrate the difference between buffering and actual writing.
Thats when it matters, with cat it would cache the entire write. If you don't want it to cache everything and have no idea how much time is left or how fast it is writing, and then wait an hour for it to unmount instead then cat is fine.
> Dd is super useful for incrementally writing boot loader to /dev device endpoints when offsets are required (multiple images).
How exactly does one use cat to overwrite only a specific offset? Say, I have a 512 bootloader that needs to be placed starting at 2048 bytes into the image; how would I invoke cat to do that?
No it doesn't, what's being done here is inserting a binary block into the middle of a binary file (or disk) without overwriting the data before or after it
Yea I came here to write the same. The author obviously works in a different environment where such operations are not standard.
The ability to specify that an exact amount of bytes should be written or read in a location is an important task. More so on special files, like device files. Even provisioning an SD card with the image block aligned is crucial, and that is not speaking of very early boot loaders that need to be available in a specific byte offset.
This discussion hopefully educates the public that this kind of tools are foundational and their use cases should be learned:
You never know when you need to read a special byte offset in a hacky one liner in a debug environment. Analysis productivity matters, and knowing analysis tools is key for that.
As mentioned elsewhere in this post this leads to dreadful performance. You are better off using iflag/oflag options seek_bytes, skip_bytes and count_bytes to treat these options as bytes and using a large block size.
Option 1 will run at Kilobytes per second as it is transferring 1 byte at a time (my test gives me 933kB/s)
Option 2 will run a 100's MB per second as it is transferring 4MB at a time (my test gives me 202MB/s).
Edit: What I should have said at the top of the post was "If you want to extract data from a file whose size is not an integer block multiple". In the above example you could have used a Blocksize of 10MB for the same result. You cannot do this for an oddly size extraction (say 1234567bytes).
I only found out because I use dd 1000's of times each day and have had to prototype some really esoteric data pipelines using it. I have even compiled a few custom versions to do various tasks.
One other nuggets of wisdom which is very poorly documented in the man page: if you have a long running dd process (I often have 12+ hr dd processes reading 10+TB tapes) you can send a USR1 signal to check the progress. Very useful to check if your tape drive has had a hardware failure.
Information on both this and the byte counts are in my man page (GNU coreutils 8.30 on Linux Mint 20) around lines 130.
Edit: I just found this in the coreutils 9.0 changelog:
dd now counts bytes instead of blocks if a block count ends in "B".
For example, 'dd count=100KiB' now copies 100 KiB of data, not
102,400 blocks of data. The flags count_bytes, skip_bytes and
seek_bytes are therefore obsolescent and are no longer documented,
though they still work.
The "pv" command they recommend instead is, by contrast, not standard and is nowhere near universally available. Also, as just about any not-a-noob knows, the author's "cat" suggestion will not work on devices that expect whole blocks, except perhaps by accident ... and it's not a good habit to rely on accidents. Options like skip and seek also allow you to do things like write a single block at an arbitrary location, which can be very useful sometimes and is something cat/head/etc. can't do.
Reacting to anything one doesn't understand, without even trying to understand it, as strange or obscure or weird (all their words) is a very deeply incurious attitude. That kind of crap doesn't even belong here, per guidelines.
P.S. Hey kid, learn how to use a proper link instead of a Wikipedia title.
To the extent that "standard in the colloquial sense" even makes sense, it's still wrong because "head" (the proposed alternative) is no more common or well understood than dd, and "pv" is far less so.
That probably depends on the user. I use head every day, I can't think of the last time I used dd. Agree with you on pv, I don't believe I've ever used it - I've seen it around occasionally but it's never top of mind the rare time I might want it
Using standard as an adjective to mean "used or accepted as normal or average" or "usual, common, or customary" makes a lot of sense. I just came back from touring my buddy's new woodshop. When people ask how it looks I will say "he has your standard set of woodshop tools." Nobody is going to wonder if it's an ISO, or ANSI standard set of woodshop tools.
The point here is that dd is the weird one out that has a different command line syntax than everything else. Not about whether it's installed by default. (Let's not get distracted by the pv bit; dd's status=progress isn't POSIX either)
I've been using Unixes of various flavours since the 1980s and had never heard of it until I read this article. It's certainly more obscure to me – as in not discoverable at all – than dd status=progress.
Apparently I've been slumming it by watching long jobs using Activity Monitor on macOS.
the cult of cat? the cult of pv? Really, we just need a progress meter that says "don't bother getting up up", "get lunch now", or "come back tomorrow", but I don't know that there is one.
For bulk copying of /dev/zero to /dev/null, `dd` is fastest (block sizes of 256kB up to 4MB are about the same speed), then `pv` (block sizes of 256kB up to 4MB are about the same speed), then `head`.
But it's true, unless you're dealing with an actual tape drive, there aren't a lot of things in the world anymore that depend on a specific block size; the idea that you'll get a faster transfer if you (say) exactly match the erase block size of your SD card doesn't seem to hold much water anymore.
And debian's bash-completion of `dd` has been broken for several releases, I despair of it ever working right again.
> Really, we just need a progress meter that says "don't bother getting up up", "get lunch now", or "come back tomorrow", but don't know that there is one.
It feel like this is gonna lead to another thing like thefuck...
On BSD systems you can press ^T for SIGINFO and that prints some progress information, at least for tools like dd and cp, not sure about cat and I don't have a BSD system to test. Linux doesn't have this unfortunately. GNU tools often support it (when run on BSD), it's just that the kernel doesn't.
> The idea that you'll get a faster transfer if you (say) exactly match the erase block size of your SD card doesn't seem to hold much water anymore.
No but if you use a much lower block size with cheap flash it does get horribly slow.
Makes sense because cheap eMMC does not have cache RAM to combine write commands and no NCQ (native command queueing) either. So it has to execute each write as it gets it. I bet you can kill flash pretty quickly this way with a block size of 1. The write amplification will be huge.
The problem with the cat method is that you don't really know what it's doing under the hood in terms of write sizes. Probably it will pick something smart but it depends on the OS there and possibly the shell.
I believe the way it goes is that USR1 kills BSD dd.
The Linux kernel doesn't have SIGINFO. GNU dd uses SIGINFO on platforms that have it, or SIGUSR1 otherwise. The default action for SIGUSR1 is to kill the process. So it makes sense that on platforms that do have SIGINFO no one would bother to override that default SIGUSR1 behavior.
I think it's USR1 to bsd dd? SIGINFO doesn't exist on Linux from what I remember, but I think the default behavior for processes is to terminate when they receive a signal they don't catch, and I can't imagine many programs have explicit handlers for USR1, especially if SIGINFO is an option
I find it very strange that you think it's not obvious that cat works with binary data. It's just a "dumb" program that reads from some files and sends it to stdout, it doesn't care what the data is!
One of the intended use cases of cat is reassembling files after they have been split with the "split" command!
Same here, a few times over the years. Another useful trick is piping input from echo to xxd -r -ps to dd so you can specify hex bytes directly in the echo arguments rather than reading input from a file. Quite handy if it's an embedded system that you can't easily transfer files to, but can get a shell on.
That's going to be very slow though, as it will read and write everything one byte at a time. The problem of course is that you can't specify seek/skip/count values as bytes unless your block size is 1 byte.
If you don't need seek then you can at least use ibs=1 instead, as skip/count operate on input blocks, but this will still read one byte at a time even though it will aggregate the reads into larger output blocks.
It would be really nice if we had a dd2 tool that offered similar options but defaulted the block size to a new "auto" value, treated seek/skip as byte values, and treated count as a byte value if the input block size is "auto".
dd allows you to specify your seek, skip and count values as bytes instead of blocks using the iflag/oflag options seek_bytes, skip_bytes and count_bytes.
So to read the first MB of data 100GB into a file you can:
Edit 2 : I just found this in the coreutils 9.0 changelog:
dd now counts bytes instead of blocks if a block count ends in "B".
For example, 'dd count=100KiB' now copies 100 KiB of data, not
102,400 blocks of data. The flags count_bytes, skip_bytes and
seek_bytes are therefore obsolescent and are no longer documented,
though they still work.
One of the nice things about `dd` is that the arguments can be in any order, they're explicitly given mnemonics. You can have `cp` ordering with `dd if=<source> of=<destination>` or `ln` ordering with `dd of=<destination> if=<source>` and it doesn't matter, `if` is always the input file and `of` is always the output file. The other POSIX tools don't all have consistent syntax anyway, `dd` is at least more memorable than `find` or `tar`.
Actually, ln takes arguments in the same order as cp and mv. It's always `cmd existing-file... new-name`.
ln only gets confusing because the file that exists is the target of the link, so it may be natural to think of the link as going from the last name on the command line to the earlier ones, but really the mental model should be in names that either exist or are to be created.
To me dd's noerror flag as well as seek are very comforting to know of whenever I'm on a quest to preserve some precious data from any medium currently in failure mode ... any extra read might be your last.
dd is a beautiful little swiss army knife and I'm not ashamed to be part of the "cult" lol. I have had to use ddrescue too though. It's handy to have in the toolbox if you have a disk with errors and want to get data off of it.
If you look at https://news.ycombinator.com/item?id=33334043 it turns out rascul was right and it appears to be only zsh that handles the reading of the redirected input without some other command to hand it off to.
I've since tried in two versions of Bash (3.2 and 5.1) and in five different varieties of ksh (oksh, mksh, ksh93, pdksh, GNU ksh). It does seem to be a zshism. All the others do the redirect into the destination, but only zsh provides the content of the redirect from source without some other command to read it.
The zsh way seems more properly fitting with the philosophy of the Unix shell to me. It would be an uphill battle getting everyone else to change that behavior, though.
> This is a strange program of [obscure provenance] ... that somehow, still manages to survive in the 21st century.
Perhaps it "survives" because people have been using it for decades for specific tasks for which it still still works just fine. Just because there are other ways to achieve the same results for some of the tool's use cases doesn't mean that said tool should be done away with. If you don't like it no one is forcing you to use it, and calling those who still use dd "The Cult of Dd" is just ridiculous, in my opinion.
Seeking a file descriptor is a basic, useful operation, and dd is the only POSIX tool that can seek an fd without actually paying the read / write costs.
Suppose you have 2 hours of UHD RGBA32 video, and need 5 minutes of footage from the halfway mark:
But tail might read up to pipe buffer size more data than is actually required. Also tail + head approach have an overhead of copying data between processes.
That’s why tail | head isn’t a reliable way to seek an fd — it will seek past the desired offset by reading and then failing to write up to PIPEBUF bytes.
Also, as in your example, it's a million times easier to make bs your frame size so you can use skip and count to slice frames as you wish. I use this all of the time working with raw video files.
The advantage of dd in this case is that dd is designed for exactly this use case of having a file (or character device, ie. magnetic tape) with some kind of fixed-size records.
Not to detract from the point, but if you are using ffmpeg to process raw video files, you can just use the "-ss" parameter on input and "-t" on output to get what you want. It'll seek and avoid unnecessary reads.
A weird personal discovery was that `openssl x509` used as a conversion tool actually reads the first PEM certificate and leaves stdin open and positioned just past it (rather than silently discarding any trailing data the way I always thought it did), so if you pipe or redirect a bunch of certs into a shell loop (as opposed to a command inside the loop condition) you can actually decode them one by one.
Without the direct flag dd is pretty pointless. However, with the direct flag, you do un-bufferred I/O with precisely controlled block size. It the fastest way to ensure the data really go where you want it to go, in the fashion you like it to be, right at the moment.
Apparently Linux has a facility for 'optimal IO size' of block devices (see 'lsblk -o NAME,OPT-IO') but on my system I only have a value for Linux md RAID devices (which happen to be RAID0, and the OPT-IO value is the stride). All of the other devices have OPT-IO 0.
Perhaps more work needs to be done to bubble some value up from the hardware.
Meh, "cult" is mean and unnecessary. So people use it as a swiss army knife? Does it fail? bs=4M is useful will work fairly universally on various systems +/-10% variance of time. It's always nice to know new ways but why insult people who like dd?
240 comments
[ 5.5 ms ] story [ 288 ms ] threadDoing this kind of thing is a) rare and b) usually you have more things on your mind at the time, so you tend to be quite conservative and not think to experiment.
It doesn't really do anything particularly arcane (unless you count EBCDIC conversions) but it does things that are useful and often necessary that other equally-standard utilities can't. Isn't that good enough?
> Doing this kind of thing is a) rare
Probably not as rare as you think. Maybe it's not all that useful for writing applications, but for the quite large number of people who have to do provisioning (either bare-metal or virtual) and such it's a different matter. There's a lot to be said for tools that are ubiquitous, well standardized, and flexible.
Part of the issue is that a man page is a reference. This is a necessary kind of documentation, but sometimes I want a tutorial and sometimes I want a cookbook. References are great for some things, but discovering new tools isn’t generally one of them. Obviously, you don’t have to write man pages that way, but almost all of the ones I see in the wild are.
Another part is that man pages as a resource have really atrophied in my experience. Lots of new CLI tools don’t have them at all, and lots of systems don’t have them installed even for older/core tools. The why is harder. I suspect that it’s a mix of tooling (e.g. needing to learn groff/troff to write them), search (your man pages aren’t indexed by Google by default like your GitHub readme), and culture. But I’m not sure.
Well, because it doesn't. At least on the Linux version I used in the past, it defaulted to 512 byte blocks or something similarly small and it commits every block individually, leading to really slow performance with USB sticks with controllers not smart enough to bundle writes together. I wouldn't be surprised if that incurs some heavy write amplification at flash level too. Perhaps it's smart enough to figure a better block size now but this is where that habit comes from.
Another thing, creating sparse files with the seek option (simply put files containing zeros that are not actually written to the disk nor taking space, but do turn up all these zeroes when you read the file). Also something not duplicated with cat or head.
What I like about dd is that it can do pretty much all disk operations in one simple tool. Definitely worth learning all of it IMO.
i think he's talking about cat being the command that figures it out automatically
I wonder what cat does in terms of buffers, I kinda doubt it has any special optimisations though I would guess the shell redirect might have. As that's really the thing doing the work there, not cat. Edit: Nope, I'm wrong there!!
Also that command does more than just specify the memory buffer like he says. That's my point, it's useful for tuning which can be super helpful with huge images.
It can also lead to some dangerous gotchas as well when working with files. But with full disk images these don't apply generally.
No? The shell redirection is just
(plus error handling and whatnot)`cat` ends up with a file descriptor directly to the block device, same as `dd` does; the only difference is whether the `open()` call comes before or after the `execve()` call.
I really doubt cat is smart enough to figure out a suitable block size though. At least with dd you can specify one.
It seems to try, at least
https://github.com/coreutils/coreutils/blob/master/src/cat.c...
- >=9.1 (2022-04-15) : Use the `copy_file_range()` syscall and let the kernel figure it out
- >=8.23 (2014-07-18) : max(128KiB, st_blksize(infile), st_blksize(outfile))
- >=8.17 (2012-05-10) : max(64KiB, st_blksize(infile), st_blksize(outfile))
- >=7.2 (2009-03-31) : max(32KiB, st_blksize(infile), st_blksize(outfile))
- at least as far back as 1996 : max(st_blksize(infile), st_blksize(outfile))
(In my psuedo-code, `st_blksize(fd)` is the `ST_BLKSIZE(buf)` of the result of `fstat(fd, buf)`.)
I recall that GNU cat used a constant value that was tuned for read performance on common hard drives, giving no consideration to write performance. Looking at the GNU cat sources today, it doesn't look at all how I remember; I'd have to study it a bit to tell you what it does.
Edit: Hrmm, it seems I was mis-remembering. Perhaps I was thinking of the minimum values (32KiB, 64KiB, 128KiB below)? Or perhaps I was thinking of a BSD cat? Anyway:
How GNU cat sizes its buffers, by version:
- >=9.1 (2022-04-15) : Use the `copy_file_range()` syscall and let the kernel figure it out
- >=8.23 (2014-07-18) : max(128KiB, st_blksize(infile), st_blksize(outfile))
- >=8.17 (2012-05-10) : max(64KiB, st_blksize(infile), st_blksize(outfile))
- >=7.2 (2009-03-31) : max(32KiB, st_blksize(infile), st_blksize(outfile))
- at least as far back as 1996 : max(st_blksize(infile), st_blksize(outfile))
(In my psuedo-code, `st_blksize(fd)` is the `ST_BLKSIZE(buf)` of the result of `fstat(fd, buf)`.)
Not sure what cat does but like I said in my other comment its not really cat itself that does the disk writing in that command but rather the shell redirect. Edit: Nope, I'm wrong there!!
I interpreted "the command" in the original article as "the command you end up running, whether it be `dd` or `cat` or something else."
But you're mistaken about the shell redirect. In either scenario it's the command (`dd` or `cat`) making the write() syscall to the device. The shell passes the file descriptor to the command, then gets out of the way.
In that case I guess dd does call a sync() on every output block? Because it's definitely slower and the LED pattern on a USB stick is also much more 'flashy' when using 512 bytes.
Only if you tell `dd` `oflag=sync` or `oflag=dsync`.
https://git.savannah.gnu.org/cgit/coreutils.git/tree/src/iob...
Unfortunately dd has not just footguns but foot cannons that are amplified by the mistakes people often make with string escaping, odd file names, loops, null checking, and conditionals in bash.
It uses a syscall per block write (which would slow it down if you use 512 byte blocks instead of 8M for example), but the OS does the file buffering and final writing to the device, unless you pass the fsync or fdatasync options to dd.
Edit:
here's writing to a old and slow stick and you can see that dd is fast and then the OS has to transfer the buffered data onto the stick:
And the stick is placed in a USB 3.2. port on a fast machine ;-0Correct, and that's one more point for dd compared to head/tail (which are fine commands by themselves).
But wouldn't help much in my example, where I used an (very) old "high speed usb 2.0 stick" with 4 MByte/s write speed to demonstrate the difference between buffering and actual writing.
"Who cares?" --> people that care are people that want to have control over that option, as not every tool is written intelligently.
This is very common with u-boot and eMMC/microSD devices on SBC (pi clones, etc).
It’s also generally valuable to be precise and explicit when messing with boot loader data in general.
I think the author is under appreciating these use cases.
works, fyi
https://www.unix.com/302462639-post2.html?s=26c3fb7cf64a858c...
> Dd is super useful for incrementally writing boot loader to /dev device endpoints when offsets are required (multiple images).
How exactly does one use cat to overwrite only a specific offset? Say, I have a 512 bootloader that needs to be placed starting at 2048 bytes into the image; how would I invoke cat to do that?
The ability to specify that an exact amount of bytes should be written or read in a location is an important task. More so on special files, like device files. Even provisioning an SD card with the image block aligned is crucial, and that is not speaking of very early boot loaders that need to be available in a specific byte offset.
This discussion hopefully educates the public that this kind of tools are foundational and their use cases should be learned:
You never know when you need to read a special byte offset in a hacky one liner in a debug environment. Analysis productivity matters, and knowing analysis tools is key for that.
cat can't do that.
> dd if=/tmp/file1 bs=1 skip=50000000 count=10000000
or you can use any block size you want and treat skip and count as byte counts, not block counts.
> dd if=/tmp/file1 bs=4M skip=50M count=10M iflag=skip_bytes,count_bytes
Option 1 will run at Kilobytes per second as it is transferring 1 byte at a time (my test gives me 933kB/s)
Option 2 will run a 100's MB per second as it is transferring 4MB at a time (my test gives me 202MB/s).
Edit: What I should have said at the top of the post was "If you want to extract data from a file whose size is not an integer block multiple". In the above example you could have used a Blocksize of 10MB for the same result. You cannot do this for an oddly size extraction (say 1234567bytes).
Why doesn't the man page say anything about these flags? They are completely obscure becasue of that.
One other nuggets of wisdom which is very poorly documented in the man page: if you have a long running dd process (I often have 12+ hr dd processes reading 10+TB tapes) you can send a USR1 signal to check the progress. Very useful to check if your tape drive has had a hardware failure.
Information on both this and the byte counts are in my man page (GNU coreutils 8.30 on Linux Mint 20) around lines 130.
Debian testing, coreutils 9.1.
It is also in Coreutils 8.22 in RHEL7.
https://man7.org/linux/man-pages/man1/dd.1.html
Edit: I just found this in the coreutils 9.0 changelog:
So you can simply do something like:
To get m bytes from offset nSure sign that the post wasn't researched at all, since dd is one of the very oldest UNIX utilities and is even in the POSIX standard.
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/d...
The "pv" command they recommend instead is, by contrast, not standard and is nowhere near universally available. Also, as just about any not-a-noob knows, the author's "cat" suggestion will not work on devices that expect whole blocks, except perhaps by accident ... and it's not a good habit to rely on accidents. Options like skip and seek also allow you to do things like write a single block at an arbitrary location, which can be very useful sometimes and is something cat/head/etc. can't do.
Reacting to anything one doesn't understand, without even trying to understand it, as strange or obscure or weird (all their words) is a very deeply incurious attitude. That kind of crap doesn't even belong here, per guidelines.
P.S. Hey kid, learn how to use a proper link instead of a Wikipedia title.
Apparently I've been slumming it by watching long jobs using Activity Monitor on macOS.
For bulk copying of /dev/zero to /dev/null, `dd` is fastest (block sizes of 256kB up to 4MB are about the same speed), then `pv` (block sizes of 256kB up to 4MB are about the same speed), then `head`.
But it's true, unless you're dealing with an actual tape drive, there aren't a lot of things in the world anymore that depend on a specific block size; the idea that you'll get a faster transfer if you (say) exactly match the erase block size of your SD card doesn't seem to hold much water anymore.
And debian's bash-completion of `dd` has been broken for several releases, I despair of it ever working right again.
It feel like this is gonna lead to another thing like thefuck...
Doesn't `status=progress` work? (The "obscure option to GNU dd" that the article mentions)
No but if you use a much lower block size with cheap flash it does get horribly slow.
Makes sense because cheap eMMC does not have cache RAM to combine write commands and no NCQ (native command queueing) either. So it has to execute each write as it gets it. I bet you can kill flash pretty quickly this way with a block size of 1. The write amplification will be huge.
The problem with the cat method is that you don't really know what it's doing under the hood in terms of write sizes. Probably it will pick something smart but it depends on the OS there and possibly the shell.
https://www.ivarch.com/programs/pv.shtml
2) (GNU) dd has support for progress bar and even if not enabled, one can get statistics by sending USR1 signal to it.
And the BSDs have siginfo, so at least most of the Free OSs are covered.
The Linux kernel doesn't have SIGINFO. GNU dd uses SIGINFO on platforms that have it, or SIGUSR1 otherwise. The default action for SIGUSR1 is to kill the process. So it makes sense that on platforms that do have SIGINFO no one would bother to override that default SIGUSR1 behavior.
The OP knows that. The article says that with pv, you have something that works for every command instead of needing to remember dd-specific syntax.
One of the intended use cases of cat is reassembling files after they have been split with the "split" command!
If you don't need seek then you can at least use ibs=1 instead, as skip/count operate on input blocks, but this will still read one byte at a time even though it will aggregate the reads into larger output blocks.
It would be really nice if we had a dd2 tool that offered similar options but defaulted the block size to a new "auto" value, treated seek/skip as byte values, and treated count as a byte value if the input block size is "auto".
dd allows you to specify your seek, skip and count values as bytes instead of blocks using the iflag/oflag options seek_bytes, skip_bytes and count_bytes.
So to read the first MB of data 100GB into a file you can:
> dd if=/tmp/file1 bs=2M skip=100G count=1M iflag=count_bytes,skip_bytes
It is also in Coreutils 8.22 in RHEL7.
Edit: https://man7.org/linux/man-pages/man1/dd.1.html
Edit 2 : I just found this in the coreutils 9.0 changelog:
blah | sudo sh -c 'cat > /some/file'
Though if you’re simply initializing from /dev/zero you’re probably better off with truncate to create a sparse file: https://man7.org/linux/man-pages/man1/truncate.1.html
Can’t beat a program that doesn’t even write the bytes.
ln only gets confusing because the file that exists is the target of the link, so it may be natural to think of the link as going from the last name on the command line to the earlier ones, but really the mental model should be in names that either exist or are to be created.
The `help` text for GNU coreutils `cp` starts with `Usage: cp [OPTION]... [-T] SOURCE DEST`.
The `help` text for GNU coreutils `ln` starts with `Usage: ln [OPTION]... [-T] TARGET LINK_NAME`.
The man pages & help text aren't conducive to building the `cmd old new` mental model. `dd` makes it more explicit.
I just happened to explore this on an M1, so I had a shell that would do things this way.
Update: Back at the computer now, and I cannot replicate it with bash:
I'm curious what shell you did that with.The zsh way seems more properly fitting with the philosophy of the Unix shell to me. It would be an uphill battle getting everyone else to change that behavior, though.
Perhaps it "survives" because people have been using it for decades for specific tasks for which it still still works just fine. Just because there are other ways to achieve the same results for some of the tool's use cases doesn't mean that said tool should be done away with. If you don't like it no one is forcing you to use it, and calling those who still use dd "The Cult of Dd" is just ridiculous, in my opinion.
Suppose you have 2 hours of UHD RGBA32 video, and need 5 minutes of footage from the halfway mark:
This will be a lot faster than pointlessly catting the first 3 terabytes!Here's a variation. Suppose the video file has a 1M header you need to skip:
The first dd invocation does nothing more than seek stdin ahead 1M, so the second can operate on full 1 second chunks of video. Useful!Now, are there Uncalled For Uses of dd, just like there are Useless Uses of cat? You bet.
tail -c +STARTOFFSET $FILE | head -c MAXLENGTH
There are more than one way to do stuff with UNIX tools.
But tail might read up to pipe buffer size more data than is actually required. Also tail + head approach have an overhead of copying data between processes.
You could seek to second, keyframe, etc, and it would continue to work for formats that don't have fixed frame sizes.
It's true that it is quite a lifehack if you often seek to frame in raw though.
Apparently Linux has a facility for 'optimal IO size' of block devices (see 'lsblk -o NAME,OPT-IO') but on my system I only have a value for Linux md RAID devices (which happen to be RAID0, and the OPT-IO value is the stride). All of the other devices have OPT-IO 0.
Perhaps more work needs to be done to bubble some value up from the hardware.
dd if=/dev/urandom count=1024 bs=1 | base64
Need it to be in a particular subset of characters?
dd if=/dev/urandom count=1024 bs=1 |tr -dc 'A-Za-z0-9'
Note: if you're on a mac, its tr is kind of broken. Do export LC_ALL=C first.
head -c 1024 < /dev/urandom
head [OPTION]... [FILE]...
head -c 1024 /dev/urandom | base64
Can you even use ‘>’ io redirects with sudo?