Somewhat relevant: the default sed that ships on all macos is freebsd sed.
The basic featureset and syntax is about the same, but the more advanced parameters and features have different syntax (or not implemented) when compared to gnu sed.
This caused me quite a headache in the past figuring out why certain scripts works (or breaks!) on local machine but not in others; I started taking closer look on all tool versions since then.
Ah...yeah, I was probably over generalizing when I said that, though it's probably fair to say that gawk is the default awk for many popular Linux distros.
Have you flipped the final one? Both my Buster and Bullseye installation only have mawk.
Trying to dig in deeper makes my head explode though. mawk is marked required yet with update-alternatives priority of 5, and gawk is marked optional but with a priority 10. I guess the intention is that mawk is good enough for base installs, but you probably want gawk.
It's definitely hard to track down. Mawk is often way faster than gawk, but gawk has more features (sockets even...crazy) and maybe fewer arbitrary limits.
IIRC (no Ubuntu VM handy right now) mawk is the default with a minimal LTS install, using alternatives to point awk at it. When you apt install gawk it has a higher priority in alternatives and flips the awk symlinks over to gawk automagically.
> gawk it has a higher priority in alternatives and flips the awk symlinks over to gawk automagically
That must be what happened. I just checked my other machine that had a pretty new install of 20.04 on it, and it does have mawk. I'm happy to see I'm not crazy.
True, although sometimes mawk is installed, as others mention.
If you need gawk, put gawk in your shebang, and if nothing else the end user has a chance at understanding why they’re seeing a crash. Also it will crash on line 1, nice and loud.
Another plus of Perl is that it’s a little easier to specify tabs new lines and other escaped characters than in sed in my experience. In Perl I’ll usually get it right the first time, but in sed I might have to do something dumb like sticking a literal newline in a variable or something.
Problem for me is I use more than just MacOS and Linux. I would have install GNU sed on every OS I use. I make small OS images for booting from USB. I use a non-busybox multi-call binary as the initial userland to conserve space, part of filesystem embedded in the kernel (sort of like initrd on Linux I guess). Early in the boot and setup process I use sed in scripts. Thus I would have to make sure there was a copy of GNU sed installed, preferably added to the multi-call binary. Not sure the work is really worth it just for a few convenience features. I can do anything I need to do with BSD sed just as well as with GNU sed.
Instead, the solution I chose is to write sed scripts for NetBSD's sed. These work on Linux, MacOS, FreeBSD, OpenBSD, Plan9, etc., old or new. There is one feature it has that GNU does not: "-a"; it can be used to do true (but limited) "in-place editing" without creating a temp file. Not sure why but NetBSD has since changed their sed to be more FreeBSD and GNU-compatible, e.g., adding "-i" and "-g/-G", to enable automatic temp file creation then removal^1 and GNU regex, respectively. However if we start using these "features" everywhere, then our scripts may not work with older userlands.
1. This is often called "in-place" editing, but if one examines the source code one will see there is still a temp file (tmpfname) created. We are being spared the step of manually creating then removing a temp file, but we still need the requisite filesystem space available for one.
Is there a way to automatically set dd's ibs= and obs= depending on the block and/or stripe sizes of the disks involved? It's weird that optimizing a large transfer involves manually looking up and then setting parameters that the computer itself already knows.
(Though I suppose it only "knows" them through proprietary APIs that a program like dd might not want to build into itself. Still, that's what wrapper scripts are for, no?)
Speaking of, is this something sendfile(2) thinks about? Trying to optimize between time spent reading blocks from the disk and time spent writing packets to the NIC?
A lot of use cases of `dd` are better served by `head -c $bytes`. `dd` does provide a lot more control if you need it, but when you don't just use head.
I don't know what you're using dd(1) for where `head -c` would serve (reading the MBR off a disk?) Seems kind of a niche use of dd(1) to me, compared to e.g. mirroring one disk to another disk.
Personally, my own favorite use of dd(1) that I've stumbled upon recently (when making backups of database servers) is that dd(1) functions as a (many times) faster cp(1), when copying single large files between disks with large stripe sizes (e.g. when one or both volumes is backed by a RAID0ed NVMe pools with stripe≥256MiB.)
I think dd(1) is still below optimal throughput here, though, because it's not taking advantage of the huge number of IOPS these disks can put out. (I.e. it's not trying to saturate the IO queue by reading+writing the file's blocks as parallel operations.) Is there a tool that does do that? Like one of the many "parallel rsync" tools, but for single-file use? (Basically, BitTorrent, but for disk:disk rather than disk:network?)
(For my original use case, backing up a database, I had lots of individual 1GB heap files, so GNU parallel + dd(1) actually was saturating the IOPS capacity of the src+dest disk pools. But ever-so-slightly-different use-cases, e.g. moving a tarball of said database around, wouldn't be so amenable.)
The better answer to your question is that you don't /want/ dd to use the filesystem block size because this isn't optimal.
Generally speaking, the larger bs= you use, the faster things will be as it minimizes the syscall overhead. Don't worry about the particulars of the underlying filesystem or block device. Just set the buffers as big as you can reasonably afford given available memory.
And yes, sendfile or io_uring are designed to avoid this.
That supposes a fast block device (i.e. one that can accept/queue reads/writes in its controller faster than you can context switch to the kernel to submit those reads/writes.)
For a slow block device, with no controller-side queue/buffering — say, a floppy disk, the original use-case for dd(1) — you really want to try to ensure that each read(2) or write(2) is for exactly a single disk sector; otherwise you'll end up suffering from read/write amplification which could give you a 2x-or-more penalty on your copy. (Or you rely on the kernel to buffer and coalesce your writes. But that presupposes you're writing to a filesystem, not directly to a disk.)
(Now that I think about it, I'm kind of surprised dd(1) doesn't default to doing "adaptive" block sizes, like TCP's window size — starting low and adjusting up until it finds the optimum, and adjusting back down if it ever notices writes doing patterned straddling slowdowns representing IO "packets" getting split.)
>cut doesn't work with fields separated by arbitrary whitespace. It's often better to use awk
I was working on a script to provide cut like syntax using awk, mainly for awk's default field separation and regexp delimiters. It isn't ready, but I just created a repo with what I have so far: https://github.com/learnbyexample/regexp-cut
Nice tool. I just found that if you have fields separated by arbitrary whitespaces, piping that through xargs trims multiple whitespaces into a single whitespace.
>cut doesn't work with fields separated by arbitrary whitespace. It's often better to use awk
I use `cut` precisely when I want this behaviour. If something actually is tab-separated, I don't want `cut -f3` to give me the 4th column just because the value in column 1 was the empty string.
The context of the article is the Gnu coreutils. The coreutils version of echo accepts options which is not allowed according to the link you referenced. Therefore within the context of the article stating echo is non-portable is correct.
GNU coreutils isn't a standard. It conforms to and extends the POSIX standard. If it was just talking about GNU echo, then that `echo` would work the same on every machine. It even explicitly mentions other `echo` implementations, like the shell builtins. `echo` itself is portable, none of its flags are.
> You seem to have a very narrow definition of "portable."
Can you explain how wanting to include more programs into the definition of portable is narrow?
> Can you link to the definition of the jargon term you appear to be using here?
The same definition I've always seen in every community discussing shell-scripting: if it's specified by the POSIX (Portable Operating System Interface) standard. POSIX does have a definition of `echo`. GNU echo does not conform to it by default, but can be configured to.
> Also, have you considered that some words have colloquial meanings too, and that it doesn't make sense to label them as "not correct"?
But it makes sense to label `echo` itself as not portable? The argument can be made that `GNU echo` is not portable (even though it can be with an environment variable), however that should be specified, instead of mentioning there are several implementations where it is non-standard.
`echo -e` should print "-e\n" according to POSIX but the most common implementation will print "". As far as I can tell there is no way to work around this.
It makes relying on meaningful options passed to `echo` non-portable. You can always rely on `echo` being on the system (thanks to POSIX), which is what makes it portable. Relying on the behavior of -e/-n/etc is not.
But that is my point. Print the string "-e\n" in a portable way. I can't think of a way. For GNU coreutils you would need to do `echo - -e` but on a complaint echo you would need to do `echo -e`. So maybe it is portable for some strings. But it can't be used for all cases. Even something as simple as outputting a user entered string can't be done because you don't know if it is handled specially.
You can with the POSIXLY_CORRECT environment variable:
$ /bin/echo --version
echo (GNU coreutils) 8.30
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Brian Fox and Chet Ramey.
$ POSIXLY_CORRECT=1 /bin/echo -e
-e
Which is why I say relying on `echo` is fine, since its standard. Relying on `echo` to use options is not. That's also why the author is correct that you should use `printf`, because POSIXLY_CORRECT is not set, then you can run into non-standard behavior, like you showed.
This reminds me of the time I tried to replace a region in an image I created with dd and forgot to add the `notrunc` option. Which resulted in the image file being truncated and instead of the few hundred gigabytes only a few megabytes were left.
Worth considering that shells might have builtins for things like kill, echo, test, time, and others. As an example, bash has its own builtin kill command and it isn't documented by the kill(1) man page.
42 comments
[ 3.0 ms ] story [ 37.9 ms ] threadThe basic featureset and syntax is about the same, but the more advanced parameters and features have different syntax (or not implemented) when compared to gnu sed.
This caused me quite a headache in the past figuring out why certain scripts works (or breaks!) on local machine but not in others; I started taking closer look on all tool versions since then.
(The solution is to just use gnu sed with https://formulae.brew.sh/formula/gnu-sed)
Edit: As mentioned below, awk is mawk for some Linux distros, gawk for others, and something else on still others :)
Edit: just checked and my vps than runs 20.10 uses gawk
Edit 2: 18.04 also comes with gawk, at least the install I have. I know I have worked with mawk on a linux machine- could it have been Ubuntu 16.04?
Debian Jessie (circa 2016) used gawk.
Debian Stretch (circa 2017) used mawk.
Today, Debian uses gawk.
Trying to dig in deeper makes my head explode though. mawk is marked required yet with update-alternatives priority of 5, and gawk is marked optional but with a priority 10. I guess the intention is that mawk is good enough for base installs, but you probably want gawk.
That must be what happened. I just checked my other machine that had a pretty new install of 20.04 on it, and it does have mawk. I'm happy to see I'm not crazy.
If you need gawk, put gawk in your shebang, and if nothing else the end user has a chance at understanding why they’re seeing a crash. Also it will crash on line 1, nice and loud.
Perl is another great option for portability.
Find is the big one - so much so that i "cargo install fd-find" on my boxes these days and use the rust version across all hosts.
I briefly flirted with using zsh's enhanced glob abilities, e.g.
vs.Problem for me is I use more than just MacOS and Linux. I would have install GNU sed on every OS I use. I make small OS images for booting from USB. I use a non-busybox multi-call binary as the initial userland to conserve space, part of filesystem embedded in the kernel (sort of like initrd on Linux I guess). Early in the boot and setup process I use sed in scripts. Thus I would have to make sure there was a copy of GNU sed installed, preferably added to the multi-call binary. Not sure the work is really worth it just for a few convenience features. I can do anything I need to do with BSD sed just as well as with GNU sed.
Instead, the solution I chose is to write sed scripts for NetBSD's sed. These work on Linux, MacOS, FreeBSD, OpenBSD, Plan9, etc., old or new. There is one feature it has that GNU does not: "-a"; it can be used to do true (but limited) "in-place editing" without creating a temp file. Not sure why but NetBSD has since changed their sed to be more FreeBSD and GNU-compatible, e.g., adding "-i" and "-g/-G", to enable automatic temp file creation then removal^1 and GNU regex, respectively. However if we start using these "features" everywhere, then our scripts may not work with older userlands.
1. This is often called "in-place" editing, but if one examines the source code one will see there is still a temp file (tmpfname) created. We are being spared the step of manually creating then removing a temp file, but we still need the requisite filesystem space available for one.
(Though I suppose it only "knows" them through proprietary APIs that a program like dd might not want to build into itself. Still, that's what wrapper scripts are for, no?)
Speaking of, is this something sendfile(2) thinks about? Trying to optimize between time spent reading blocks from the disk and time spent writing packets to the NIC?
Personally, my own favorite use of dd(1) that I've stumbled upon recently (when making backups of database servers) is that dd(1) functions as a (many times) faster cp(1), when copying single large files between disks with large stripe sizes (e.g. when one or both volumes is backed by a RAID0ed NVMe pools with stripe≥256MiB.)
I think dd(1) is still below optimal throughput here, though, because it's not taking advantage of the huge number of IOPS these disks can put out. (I.e. it's not trying to saturate the IO queue by reading+writing the file's blocks as parallel operations.) Is there a tool that does do that? Like one of the many "parallel rsync" tools, but for single-file use? (Basically, BitTorrent, but for disk:disk rather than disk:network?)
(For my original use case, backing up a database, I had lots of individual 1GB heap files, so GNU parallel + dd(1) actually was saturating the IOPS capacity of the src+dest disk pools. But ever-so-slightly-different use-cases, e.g. moving a tarball of said database around, wouldn't be so amenable.)
Generally speaking, the larger bs= you use, the faster things will be as it minimizes the syscall overhead. Don't worry about the particulars of the underlying filesystem or block device. Just set the buffers as big as you can reasonably afford given available memory.
And yes, sendfile or io_uring are designed to avoid this.
For a slow block device, with no controller-side queue/buffering — say, a floppy disk, the original use-case for dd(1) — you really want to try to ensure that each read(2) or write(2) is for exactly a single disk sector; otherwise you'll end up suffering from read/write amplification which could give you a 2x-or-more penalty on your copy. (Or you rely on the kernel to buffer and coalesce your writes. But that presupposes you're writing to a filesystem, not directly to a disk.)
(Now that I think about it, I'm kind of surprised dd(1) doesn't default to doing "adaptive" block sizes, like TCP's window size — starting low and adjusting up until it finds the optimum, and adjusting back down if it ever notices writes doing patterned straddling slowdowns representing IO "packets" getting split.)
I was working on a script to provide cut like syntax using awk, mainly for awk's default field separation and regexp delimiters. It isn't ready, but I just created a repo with what I have so far: https://github.com/learnbyexample/regexp-cut
```$ echo "words are far apart" | xargs```
Although, it does have its caveats. https://stackoverflow.com/a/12973694
I use `cut` precisely when I want this behaviour. If something actually is tab-separated, I don't want `cut -f3` to give me the 4th column just because the value in column 1 was the empty string.
6 years ago https://news.ycombinator.com/item?id=10648255
`echo` is portable: https://pubs.opengroup.org/onlinepubs/009695399/utilities/ec... Any options passed to `echo` are not. Although I agree `printf` should be used instead.
Using common sense, this is probably what they mean by 'non portable' isn't it?
Also, have you considered that some words have colloquial meanings too, and that it doesn't make sense to label them as "not correct"?
Can you explain how wanting to include more programs into the definition of portable is narrow?
> Can you link to the definition of the jargon term you appear to be using here?
The same definition I've always seen in every community discussing shell-scripting: if it's specified by the POSIX (Portable Operating System Interface) standard. POSIX does have a definition of `echo`. GNU echo does not conform to it by default, but can be configured to.
> Also, have you considered that some words have colloquial meanings too, and that it doesn't make sense to label them as "not correct"?
But it makes sense to label `echo` itself as not portable? The argument can be made that `GNU echo` is not portable (even though it can be with an environment variable), however that should be specified, instead of mentioning there are several implementations where it is non-standard.
`echo -e` should print "-e\n" according to POSIX but the most common implementation will print "". As far as I can tell there is no way to work around this.
$ /bin/echo --version echo (GNU coreutils) 8.30 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
Written by Brian Fox and Chet Ramey. $ POSIXLY_CORRECT=1 /bin/echo -e -e
Which is why I say relying on `echo` is fine, since its standard. Relying on `echo` to use options is not. That's also why the author is correct that you should use `printf`, because POSIXLY_CORRECT is not set, then you can run into non-standard behavior, like you showed.