I mean, are they bad instructions? They work for most people, and they're what most people are familliar with. If I ask a random *ix user what's wrong with my shell command, they're more likely to know about dd.
Interesting assertion. Can you show me a shell invocation without using dd that cuts off the first 16 bytes of a binary file, for example? This is a common reason I use dd.
To expand, `-c` tells tail to start on the nth (starts counting at 1) byte. So +1 starts at the beginning, +17 starts after the first 16 bytes. `-n` is lines, `b` is 512-byte blocks.
This nearly tells you all you need to know. The other bit of info you'll want to note is that head -c +N produces as many bytes as you ask. So if you try to get the prefix using "head -c +N" and the suffix using "tail -c +N" then you'll have 1 byte of overlap.
(dd's corresponding options do not suffer from this problem.)
What about the `seek` argument which skips over some blocks at the beginning but still allocates them (unix "holes")?
Also note that there are still unix systems out there which do not support byte-level granularity of access to block devices. On those devices you must actually use a buffer of exactly the size of the blocks on the device. Heck, linux was like this until at least v2.
Also keep in mind that specifying the block size can be important, especially for efficiently reading data. standard shell tools don't just "figure it out" automatically. They guess, and sometimes those assumptions can be incorrect resulting in lower (orders of magnitude) performance.
The fact that it was added relatively recently is exactly why it's so obscure. Unlike if, of, bs and count, I haven't had status=progress drilled into my head by every single dd command I've read out of a manual or tutorial, so even now I still forget whether it's "status=progress" or "progress=status" or something else.
Also it's a victim of dd's bizarre non-Unix syntax - an option like "--status" or "--progress" would be more in keeping with expectations.
>One thing I'll often use dd for is recovering data from a failing drive.
Funnily enough, I ended up using it to accidentally name the wrong drive in the argument, and lost years of photos, music, video etc. though I suppose I can't blame dd for that :)
I think there's a rule that you're not really qualified to discuss command line tools in public until you've used dd to inadvertently eradicate an entire partition.
Personally, I'd add the use of a trailing full-stop in an rsync command in the wrong directory (e.g. in /home/user) as an alternative qualification to your rule.
I now use full paths for destination as well as source.
Use noerror, but forget sync? Corrupt output file if there is an error. Use a bigger bs so it's not slow as treacle? A single faulty sector blows away a whole bs of data, and your output image may get unwanted padding appended to the end. Recoverable error? dd's not going to retry.
Use ddrescue or FreeBSD's recoverdisk(1). They're faster, they're safer, they're more effective, and they're easier to use.
GNU ddrescue[1] and not dd_rescue[2]. I'm adding this precision because depending on the linux distros dd_rescue package name may be ddrescue and GNU ddrescue package name may be gddrescue.
I'll point out that dd also allows you to control lots of other filesystem and OS-related things that other tools do not. See: fsync/fdatasync. I'm not aware of any shell tools that allow you to write data like that.
OP, your alternatives to DD are more complicated, not less complicated. I shouldn't need to pipeline two commands together just to cut off the first 100MB of a file.
A counterpoint: dd survives not because it's good or makes sense, but explicitly because it doesn't.
You wanna format a usb key? Google this, copy/paste these dd instructions, it works, move on with your life.
You wanna format a usb key using something related to cat you once saw and didn't fully understand? Have fun.
Both approaches have their weak points, but in any OS the answer to "How do I format a usb key" should not start with "Oh boy, let's have a Socratic dialog over 10 years on how to do that."
Definitely this. I have found many times that I'm an offender of these "bad practices", and usually that's because a certain pattern I learned way back in the beginnings of my Linux days still hangs around.
Embarrassingly, it took me a long time before I started reaching for man pages instead of Google. That has probably has had the biggest effect on tightening up my command line fu.
find is another tool that seems to get only one specific use case that ignores its rather large and useful toolset.
I learned Linux this way a decade and a half ago when it was far (and still is imho!) more convenient to quickly search a man page than google something. (with slow internet start times, browser startup times, etc)
Now, sometimes when people watch me work in a shared session they comment on my "peculiar" (to them) usage of flipping between -h --help and man $command, because there's a whole lot of switches I have memorized over time, but even more that I just have good reference points for.
But, bar none, what I've noticed among my peers is that the people that have always bowed to quick google solutions never really have taken the time to learn what they're doing. They almost always seems to be the 'quick fix', 'get it working now, sort it out later' types.
A counterpoint: dd survives not because it's good or makes sense, but explicitly because it doesn't.
You wanna format a usb key? Google this, copy/paste these dd instructions, it works, move on with your life.
You wanna format a usb key using something related to cat you once saw and didn't fully understand? Have fun.
Both approaches have their weak points, but in any OS the answer to "How do I format a usb key" should not start with "Oh boy, let's have a Socratic dialog over 10 years on how to do that."
I think dd is primarily so popular because it is used in mostly dangerous operations. Sure, using cat makes logicial sense, but if we are talking about writing directly to disk devices here I'll trust the command I read from the manual and not explore commands I think would work.
dd's "highly nonstandard syntax" comes from the JCL programming language, but it's really just another tool to read and write files. At the end of the day it's not more complex or incompatible than other unix tools. For example, you can also use tools like `pv` with dd no problem to get progress statements.
I always thought dd stood for disk destroyer, only ever used it for making low level copies of whole disks or shredding them with if=/dev/random. This thread has been informative and terrifying as I learn cat and cp are every bit as dangerous as dd! I never would expect something like cp xxx /dev/sda to actually work. Thinking about it, why should cp even support something like that? I'll copy files but I'll also DESTROY YOUR SHIT if you say so?
> I never would expect something like cp xxx /dev/sda to actually work. Thinking about it, why should cp even support something like that? I'll copy files but I'll also DESTROY YOUR SHIT if you say so?
That's the beauty of Unix.
Everything is a file. Thus every program that can work with files, can in fact work with everything.
That's the Unix way: The customer, eh, user is always right.
I once wanted to clean up backup files created by emacs (they end in the tilde character) by typing "rm <asterisk>~" - except what I did type was "rm <asterisk> ~".
(On the upside, I learned a valuable lesson that day.)
There is some truth to the fact that (if you basically already know dd like I do) then reserving it for dangerous operations is a good way to "signal" to yourself "slow down here and pay attention"
An even easier solution: don't make people fall into the command line to format a USB reliably.
The command line should be reserved for times where you need the fine grain control to do something that DD is meant to do. A GUI should implement everything else in a reliable way that doesn't break half the time or crash on unexprected input.
"a paintbrush should be reserved for times when you need fine grain control to paint your bedroom. A hired painter could do it all reliably in a way that doesn't risk you painting over the crown moulding or falling off a ladder."
I've setup/configured bind a few times, and every time I wish there was a nice gui for it.. even thinking to myself I should make one.. but shortly after I've refreshed my memory of how to do something, and by then, I'm done and leave it alone for some while.
This is a good point as well. On BSD, we don't have `pv`, but we do have ^T. This will print some sort of status for just about any long running process. It prints very specialized status for certain programs aware of it.
pv is available on at least FreeBSD as a package like it is in Linux. I'd be quite shocked if the other BSD's didn't have a port of it as well. There is also mbuffer.
$ dd if=/dev/urandom of=foo count=1000 bs=1000000
load: 1.76 cmd: dd 80097 running 0.00u 0.89s
11+0 records in
11+0 records out
11000000 bytes transferred in 0.947316 secs (11611752 bytes/sec)
load: 1.76 cmd: dd 80097 running 0.00u 1.68s
22+0 records in
22+0 records out
22000000 bytes transferred in 1.746013 secs (12600134 bytes/sec)
load: 1.76 cmd: dd 80097 running 0.00u 2.28s
31+0 records in
31+0 records out
31000000 bytes transferred in 2.392392 secs (12957742 bytes/sec)
load: 1.76 cmd: dd 80097 running 0.00u 2.83s
38+0 records in
38+0 records out
....
There's one good (?) reason to use dd with devices: it specifies target in the same command. For devices, writing to them usually requires root privileges, so it's easy to:
sudo dd .... of=/dev/...
But there's no trivial cat equivalent:
sudo cat ... > target
Will open target as your current user anyway. You can play around with tee and redirection of course. But that's getting more complicated than the original.
This is admittedly somewhat esoteric, but it seems like a stretch to say `dd` does not have some place, especially when transferring binary data in very specific ways.
Since we're sharing shell tricks: The "sudo tee > /dev/null" may be baroque, but I find it useful whenever I start editing stuff in /etc in vim, only to find that I cannot write my changes because I'm not root. In that case,
:w !sudo tee %
does the trick. (What "w!" does is send the buffer into the given shell command as stdin.)
Basically, you can use cp wherever you use dd, as long as you're not changing any low-level parameters (e.g. starting 500 bytes into the file or something).
Yes. Yes it does. Reminds me of a Sunday evening in the late 90ies when I stopped working as root all the time:
cp backup.tar.bz /dev/sda
Nowadays I would know enough to at least get the contents of the backup.tar.bz back. Back then, this was the end of both my / partition (or any other partition) and the backup of my music collection.
Still, that didn't end my love affair with Unix. It did make me a whole lot more careful though.
Well that's strictly a bug caused by mistaken use, as strings are not expanded lazily and heredocs are just another string syntax. How can one use the unix shell without string interpolation? Also, a similar programme would give the wrong result in, say, Ruby or Perl too.
And I state again: if you're a future doctor and your biggest regret is that you could be $50k richer right now, I'm not inclined to do much weeping. Basically everyone has been a broke student.
True, but it feels worse to have had the opportunity in one's hands and thrown it away than never to have touched it at all. That's just how humans work - go figure.
But who cares? Why not just let the command figure out the right buffer size automatically?
Because it can be a lot slower. dd is low level, hence powerful and dangerous.
And, if we are going down that rabbit hole, you don't need cat[1]
“The purpose of cat is to concatenate (or "catenate") files. If it's only one file, concatenating it with nothing at all is a waste of time, and costs you a process.”
Yep, my thought was that the UUOC critique doesn't apply to most attempts to substitute cat for dd, because typically those are copying from one (regular or special) to another, and you can't simply use redirection to accomplish this in the absence of a reader.
The biggest counterexample to this that some people have experienced is accidentally swapping if= and of=, thus backing up their target onto their source rather than vice versa.
Don't cat a file and pipe it into pv. Use "pv file" as a replacement for "cat file" and it will show you the progress as a percentage. When it's in the middle of a pipeline, it doesn't know the total size (unless you tell it with -s), so it can only show the throughput.
It's an orthogonal issue, yes, but calling stat or fstat on any block device whether from stdin or argv will return .st_size == 0, so your progress bar won't display the correct answers (or could display better answers if it used the ioctl).
It makes a decent interview question tho', "explain the difference between cat file|./prog and ./prog <file". It doesn't even matter if they get it wrong, that they even know there is a difference is a very good sign.
People who have come into SA work via being C programmers usually figure it out, they make the best SAs because they are mentally equipped to reason about a system from first principles.
177 comments
[ 468 ms ] story [ 5212 ms ] thread(dd's corresponding options do not suffer from this problem.)
Grab the first N bytes vs. grab everything starting from the Nth byte.
Also note that there are still unix systems out there which do not support byte-level granularity of access to block devices. On those devices you must actually use a buffer of exactly the size of the blocks on the device. Heck, linux was like this until at least v2.
An essential tool for low level repair, like when you can guess the partition table values but there is no partition table anymore.
Also it's a victim of dd's bizarre non-Unix syntax - an option like "--status" or "--progress" would be more in keeping with expectations.
As far as I'm concerned, dd is lower-level than most of the other utilities and provides more control over what's happening.
The author does have a point that the syntax is strange though.
Funnily enough, I ended up using it to accidentally name the wrong drive in the argument, and lost years of photos, music, video etc. though I suppose I can't blame dd for that :)
I now use full paths for destination as well as source.
Use noerror, but forget sync? Corrupt output file if there is an error. Use a bigger bs so it's not slow as treacle? A single faulty sector blows away a whole bs of data, and your output image may get unwanted padding appended to the end. Recoverable error? dd's not going to retry.
Use ddrescue or FreeBSD's recoverdisk(1). They're faster, they're safer, they're more effective, and they're easier to use.
[1]: http://www.gnu.org/software/ddrescue/ddrescue.html [2]: http://www.garloff.de/kurt/linux/ddrescue/
I certainly agree the syntax of the arguments is strange, due to its age, but I don't agree that learning it is difficult or a waste of time.
All I've learned is that the author doesn't like dd well enough to learn it.
* Using "cat source > target" instead of "cp source target"
* Using "cat source | pv > target" instead of "pv source > target"
* Using "head -c 100MB /dev/zero > target" instead of "truncate -s 100MB target"
You wanna format a usb key? Google this, copy/paste these dd instructions, it works, move on with your life.
You wanna format a usb key using something related to cat you once saw and didn't fully understand? Have fun.
Both approaches have their weak points, but in any OS the answer to "How do I format a usb key" should not start with "Oh boy, let's have a Socratic dialog over 10 years on how to do that."
"Why do we do it like that? I dunno, that's how I learned how, how do you do it?"
Embarrassingly, it took me a long time before I started reaching for man pages instead of Google. That has probably has had the biggest effect on tightening up my command line fu.
find is another tool that seems to get only one specific use case that ignores its rather large and useful toolset.
Now, sometimes when people watch me work in a shared session they comment on my "peculiar" (to them) usage of flipping between -h --help and man $command, because there's a whole lot of switches I have memorized over time, but even more that I just have good reference points for.
But, bar none, what I've noticed among my peers is that the people that have always bowed to quick google solutions never really have taken the time to learn what they're doing. They almost always seems to be the 'quick fix', 'get it working now, sort it out later' types.
You wanna format a usb key? Google this, copy/paste these dd instructions, it works, move on with your life.
You wanna format a usb key using something related to cat you once saw and didn't fully understand? Have fun.
Both approaches have their weak points, but in any OS the answer to "How do I format a usb key" should not start with "Oh boy, let's have a Socratic dialog over 10 years on how to do that."
dd's "highly nonstandard syntax" comes from the JCL programming language, but it's really just another tool to read and write files. At the end of the day it's not more complex or incompatible than other unix tools. For example, you can also use tools like `pv` with dd no problem to get progress statements.
It better :-)
But it all comes from the unix idea of everything is a file.
That's the beauty of Unix.
Everything is a file. Thus every program that can work with files, can in fact work with everything.
It's actually very liberating.
What happens when you did a sparse file? And cp?
https://wiki.archlinux.org/index.php/sparse_file
C.f. fallocate(1,2)
That's the Unix way: The customer, eh, user is always right.
I once wanted to clean up backup files created by emacs (they end in the tilde character) by typing "rm <asterisk>~" - except what I did type was "rm <asterisk> ~".
(On the upside, I learned a valuable lesson that day.)
2. "UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things."
- Doug Gwyn
The command line should be reserved for times where you need the fine grain control to do something that DD is meant to do. A GUI should implement everything else in a reliable way that doesn't break half the time or crash on unexprected input.
The better half of my computer use happens in the command line interface, way more efficient use of my time.
Linux, not. I wonder why.
But it looks like the answer is just "it was complicated to implement so Linux didn't add it."
SIGINFO works on gnu dd last I tried it.
I have an ARM/linux installer script that writes the u-boot image to a specific offset before the first partition:
This is admittedly somewhat esoteric, but it seems like a stretch to say `dd` does not have some place, especially when transferring binary data in very specific ways.Basically, you can use cp wherever you use dd, as long as you're not changing any low-level parameters (e.g. starting 500 bytes into the file or something).
Still, that didn't end my love affair with Unix. It did make me a whole lot more careful though.
Ouch. That just hurts seeing that line.
"One of the biggest regrets of my life."
And if you use dd then you probably should specify a bigger block size than the default of 512 bytes.
But yeah, most usage is obsolete.
so you get the best block size for reads and writes. I can't speak to what the shell does, though.
Because it can be a lot slower. dd is low level, hence powerful and dangerous.
And, if we are going down that rabbit hole, you don't need cat[1]
“The purpose of cat is to concatenate (or "catenate") files. If it's only one file, concatenating it with nothing at all is a waste of time, and costs you a process.”
[1]http://porkmail.org/era/unix/award.html#cat
(This sounds like a zen koan somehow.)
In a UUOC avoidance case, it's the current process which reads, generally via stdin. Say, the shell, or dd itself with an 'if=' parameter.
Which I strongly suspect you know.
:-D
If you mess up the syntax on a dd invocation, a nice thing happens: nothing.
Use a shell command and pipes, and your command better be perfect before you hit return.
Usually it's not past the easily-reproducable system partition yet or on a data disk that is backed up regularly so I can recover in an hour or so..
Though fio is better because it can work in parallel.
But I guess they do...
http://stackoverflow.com/questions/1734243/in-c-how-do-i-pri...
I've seen that kind of brokenness from programs trying to find their binary image on disk. Don't do it, it's bad.
It's not the same thing as trying to walk the FS to look for the filename is silly.