for f in *.txt; do sed 's/i/n/g' "$f" >"$f".tmp && cat "$f".tmp >"$f"; done
you can
sed -i.tmp 's/i/n/g' *.txt
Instead of running awk multiple times, awk can output to files given in input (or from variables etc.), e.g. if the filename you want to print to is in the third column then `{print $1,$2 >>$3}` puts cols 1 and 2 in that file
And if you have more processing in the loop, you can pipe that into one sed call:
for f in *.txt; do thing $f | sed; done
becomes
for f in *.txt; do thing $f; done | sed
which will start `sed` once instead of once per file in the loop.
However, depending on your shell and the exact loop you use this might cause it to be run in a subshell, which means you no longer get to influence the state outside in form of variables and e.g. jobs to wait for (ran into that once, had a `thing &` in the loop and a `wait` outside and was super confused that it couldn't wait). See https://mywiki.wooledge.org/BashFAQ/024.
And if `thing` takes a list of files, you can do `thing *.txt | sed` to start `thing` only once.
Also you can often use the shell's string replacement - `echo ${foo/bar/baz}` instead of `echo $foo | sed s/bar/baz`.
"Instead, what causes shell scripts problems is the cost of starting separate programs. Sed may transform text very fast and sort may sort data very fast, but starting sed or sort is comparatively expensive."
Obviously using shell builtins as much as possible is one way to mitigate this slowdown. Another is to create a multicall binary.
NetBSD (and its fork OpenBSD) have crunchgen allowing easy creation of custom multicall, static binaries. Linux has something similar called busybox.
A single binary contains all the utilities I need for text processing, e.g., sed and sort are the same program. I create hard links for all the different program names, but all programs are contained in the same binary.
One program "swallows" all the work, to use the author's terminology.
This only makes the disk space smaller and saves loading the executable image from disk, which the kernel already magics away anyway.
This does nothing for what the author was talking about which is the cost of creating and destroying a process. The kernel has to do a lot of setup and management work for every process.
Busybox would only help with this IF busybox's sh had special fork handling that recognized any built in commands and did not actually spawn a new instance of busybox for every | or `` or ().
If you write an sh script using no external binaries at all, 100% built in keywords, you will still have a very slow script if it has a lot of filelist=`echo *` because even though that echo was a sh builtin (just like a busybox builtin), a new instance of sh was spawned just to run that echo, then destroyed.
And that happens a lot. It's quite a bit more difficult to avoid that, and is exactly what the author means by writing contorted code just to avoid that.
Myself I cheat a little and allow myself to use bash as the shell, and then use every possible bash feature. For instance a huge bash feature for avoiding subshells is "printf -v" Which places the oitput of a printf command into a variable instead of stdout. Not to mention that printf itself is builtin and has several very useful features for data manipulation just from converting values with the % formatters, including another biggie is onr of the % codes it supports is %()T, which gives you essentially the date command without running date.
Just a few bashisms like that above strict posix sh makes it possible to avoid nearly all subshells.
Possible, and more performant, BUT, your program is arcane and tortured. Difficult to write, difficult to read, and full of otherwise bad practises like using a sea of global variables and state simply to avoid having to pass data around by foo=`generatefoo`, abusing the language/line parser to attain array operations without an actual split(), doing piles of work all within a huge indented wile read loop, etc.
Yet the time cost of starting UNIX userland programs, including a crunched sh, busybox or simply a static dash, is perceptively less than starting Python.
There is a lot of ridiculous criticism about shell scripts on the web in recent times. The author admits that it is unlikely a scripting language will outperform sort, sed and so on. However if I share something I wrote entirely in sed that can also be done in, say, Python with fewer LOC (because the work is being done by libraries) or in some other, large, popular scripting language, virtually no one commenting will appreciate the speed benefits of using sed.
I can deliberately avoid pipelines, I can store transformed data in temporary files, but most times I want to use pipes. They do not slow things down, they speed things up. That's because I do not write long, convulated shell scripts.
Many people want to write larger, more complex scripts. If so, then by all means use Python or whatever. But for smaller, simpler scripts, there is nothing wrong with the shell. Pipes are incredibly useful. People make shell scripts slow and then try to blame the shell.
Also, you forget to mention using {} which should not create a subshell.
I was only explaining how busybox does not do what the previous commenter thought.
The authors entire point about sort was to show just how little it's speed matters, because the job is rarely 90% within sort, or any other util like that (I don't count awk because awk is a fully functional language like sh, you could do the entire job in awk). How fast sort sorts 100 lines of text doesn't matter next to the cost of forking to execute sort, especially if it will happen repeatedly to do small jobs, treating it like a function.
The reason it's bad is that there are so many and so small such things, and it adds up, and the shell is specifically designed to hide the cost of forking a process by making keywords and executables indistiguishable. And so you end up with a zillion forks just to get the current time or to strip a few bytes off of a few bytes long string, or even to do nothing but simply store a value in a variable (date, basename/dirname, foo=`generatefoo`)
Sure, that doesn't mean don't use sh, nor does it mean do the contortions necessary to write in sh but avoid that problem, all that means is don't write non-trivial apps in sh. ... well unless you need the ubiquity more than the performance or readability. So it doesn't even necessarily mean that automatically.
temp files - It would make no sense in most cases to go through pains to avoid forks only to use temp files instead. Creating and destroying a file, even in tmpfs(ram), is as bad or maybe even worse than a process.
{} - I didn't forget {}, it had no special bearing on an explaination of what the author was talking about that makes shell scripts slow, and why busybox does not change that. It's just a branching structure like any other.
> you can easily write a shell script that appears to perform well enough in your test environment but has clear problems when run for real in environments with significantly more items
This is far from unique to shell scripts. To many times I get sent "nasty SQL" optimisation problems with the phrase "works fine in test?" or "it is really fast on my local dev environment" as part of the description. While some common constructs (loops that spawn many processes per iteration) that can make shell scripts particularly prone to this, you can't blame the language for people just not nothing to test at, or even think about, production scale.
Honestly I do not see any performance issue for a reason: shell scripts are not HPC, high load network services etc, they should/must be just small-ish stuff for doing small kind of automation. They exists for that very reason in unix.
Surely ancient systems, with a single language and end-user programming concept can be far more efficient and productive, but that's another story...
This post is spot on, I've seen work load spikes caused by crontab spawning of for/while bash loops of grep|awk|sed etc.. processing large input files where the machine would crawl at periodic intervals.
The article points out the strength of well placed shell utilities - they often do their job efficiently. The pressure of misusing these tools inefficiently comes from the absurd difficulty of most *nix shell languages.
I think the classic clutch is using `sort | tail` for finding just the largest element among millions of lines.
17 comments
[ 4.2 ms ] story [ 93.7 ms ] threadHowever, depending on your shell and the exact loop you use this might cause it to be run in a subshell, which means you no longer get to influence the state outside in form of variables and e.g. jobs to wait for (ran into that once, had a `thing &` in the loop and a `wait` outside and was super confused that it couldn't wait). See https://mywiki.wooledge.org/BashFAQ/024.
And if `thing` takes a list of files, you can do `thing *.txt | sed` to start `thing` only once.
Also you can often use the shell's string replacement - `echo ${foo/bar/baz}` instead of `echo $foo | sed s/bar/baz`.
$f = $f.tmp in this case.
Obviously using shell builtins as much as possible is one way to mitigate this slowdown. Another is to create a multicall binary.
NetBSD (and its fork OpenBSD) have crunchgen allowing easy creation of custom multicall, static binaries. Linux has something similar called busybox.
A single binary contains all the utilities I need for text processing, e.g., sed and sort are the same program. I create hard links for all the different program names, but all programs are contained in the same binary.
One program "swallows" all the work, to use the author's terminology.
This does nothing for what the author was talking about which is the cost of creating and destroying a process. The kernel has to do a lot of setup and management work for every process.
Busybox would only help with this IF busybox's sh had special fork handling that recognized any built in commands and did not actually spawn a new instance of busybox for every | or `` or ().
If you write an sh script using no external binaries at all, 100% built in keywords, you will still have a very slow script if it has a lot of filelist=`echo *` because even though that echo was a sh builtin (just like a busybox builtin), a new instance of sh was spawned just to run that echo, then destroyed.
And that happens a lot. It's quite a bit more difficult to avoid that, and is exactly what the author means by writing contorted code just to avoid that.
Myself I cheat a little and allow myself to use bash as the shell, and then use every possible bash feature. For instance a huge bash feature for avoiding subshells is "printf -v" Which places the oitput of a printf command into a variable instead of stdout. Not to mention that printf itself is builtin and has several very useful features for data manipulation just from converting values with the % formatters, including another biggie is onr of the % codes it supports is %()T, which gives you essentially the date command without running date.
Just a few bashisms like that above strict posix sh makes it possible to avoid nearly all subshells.
Possible, and more performant, BUT, your program is arcane and tortured. Difficult to write, difficult to read, and full of otherwise bad practises like using a sea of global variables and state simply to avoid having to pass data around by foo=`generatefoo`, abusing the language/line parser to attain array operations without an actual split(), doing piles of work all within a huge indented wile read loop, etc.
There is a lot of ridiculous criticism about shell scripts on the web in recent times. The author admits that it is unlikely a scripting language will outperform sort, sed and so on. However if I share something I wrote entirely in sed that can also be done in, say, Python with fewer LOC (because the work is being done by libraries) or in some other, large, popular scripting language, virtually no one commenting will appreciate the speed benefits of using sed.
I can deliberately avoid pipelines, I can store transformed data in temporary files, but most times I want to use pipes. They do not slow things down, they speed things up. That's because I do not write long, convulated shell scripts.
Many people want to write larger, more complex scripts. If so, then by all means use Python or whatever. But for smaller, simpler scripts, there is nothing wrong with the shell. Pipes are incredibly useful. People make shell scripts slow and then try to blame the shell.
Also, you forget to mention using {} which should not create a subshell.
I was only explaining how busybox does not do what the previous commenter thought.
The authors entire point about sort was to show just how little it's speed matters, because the job is rarely 90% within sort, or any other util like that (I don't count awk because awk is a fully functional language like sh, you could do the entire job in awk). How fast sort sorts 100 lines of text doesn't matter next to the cost of forking to execute sort, especially if it will happen repeatedly to do small jobs, treating it like a function.
The reason it's bad is that there are so many and so small such things, and it adds up, and the shell is specifically designed to hide the cost of forking a process by making keywords and executables indistiguishable. And so you end up with a zillion forks just to get the current time or to strip a few bytes off of a few bytes long string, or even to do nothing but simply store a value in a variable (date, basename/dirname, foo=`generatefoo`)
Sure, that doesn't mean don't use sh, nor does it mean do the contortions necessary to write in sh but avoid that problem, all that means is don't write non-trivial apps in sh. ... well unless you need the ubiquity more than the performance or readability. So it doesn't even necessarily mean that automatically.
temp files - It would make no sense in most cases to go through pains to avoid forks only to use temp files instead. Creating and destroying a file, even in tmpfs(ram), is as bad or maybe even worse than a process.
{} - I didn't forget {}, it had no special bearing on an explaination of what the author was talking about that makes shell scripts slow, and why busybox does not change that. It's just a branching structure like any other.
This is far from unique to shell scripts. To many times I get sent "nasty SQL" optimisation problems with the phrase "works fine in test?" or "it is really fast on my local dev environment" as part of the description. While some common constructs (loops that spawn many processes per iteration) that can make shell scripts particularly prone to this, you can't blame the language for people just not nothing to test at, or even think about, production scale.
Surely ancient systems, with a single language and end-user programming concept can be far more efficient and productive, but that's another story...
I think the classic clutch is using `sort | tail` for finding just the largest element among millions of lines.