I got excited when I saw the 'f' and 'count' commands, but they're just scripts he has on his system. Like doing grep 'plover' blah.log | cut -d ' ' -f 11 | sort | uniq -c | sort -n. Personally I'd prefer to use the ubiquitous commands that work everywhere than rely on having custom scripts on my system, but they are nice.
That's the whole point of shell scripting, to take a series of minimal programs and tie them together into something that does a more complex task. There's no reason to distrust a shell script simply because it is a script any more than there is to trust a binary simply because it's a binary.
Sure, but relying on custom shell scripts as unix primitives can be problematic if you find yourself frequently managing/troubleshooting systems that you don't own, and you don't want to (or aren't allowed to) put those handy scripts in place. Then when you're on any given system, you forget whether you can use "f", or if you have to fall back on awk.
I think it's less about not trusting custom scripts than it is about ensuring that your unix muscle memory doesn't atrophy.
I would be mad at any admin that would dare to deploy his helper shell scripts like “addup” and “count” on a machine other than his laptop.
And if you meant he could just have these things in his home, then it defeats the purpose of the original comment: trouble shooting and administering machines forces you to often switch user, machine, etc.
If a company has recurring troubleshooting issues, eg. "we need to know which process is taking all the CPU/RAM/IO", why wouldn't they add a script when they provision their machines with puppet or whatever tool they use?
Then instead of having to remember how to check all these things, they just run "find_resource_hogs.sh" and voila. It also enables other people to troubleshoot without specific knowledge.
Of course you don't want to put anything in there just because it might save 10 seconds, but then https://xkcd.com/1205/
I spend a lot of time moving around different machines, processes, configuration files, logs, etc. And I stopped maybe 10 years ago to use anything that is not available on a base system.
I don’t use fancy shells, I don’t use aliases, I don’t write local shortcut scripts, etc.
I just use regular bash, combine base utilities in one liners, and live with it.
Maybe I loose 1s here and there when writing one liners compared to someone with a library of wrapper utilities. But that gives me an immense benefit: I am at home on any machine, any distribution, everywhere, without any configuration, with any user.
There's always something like Ansible (or even a lower-tech solution) to at least give you the basic toolset that helps you double or triple your performance.
> that helps you double or triple your performance.
Sure, in a quest of productivity, I should also use Vagrant and Packer to create docker images with a development environment so that I can run Serverless troubleshooting containers on all my machines.
These scripts will surely help me triple the performance of my bash one liners.
I’ve heard Eclipse has good shell completion and support for oh-my-zsh too.
That’s the theory but frankly the syntax is so cumbersome, irregular and needs so many googling for "easy" things like conditional, substring, etc. that I now use a real programming language if a script needs to be anything more than a list of commands without any logic (besides variables substitution).
> That’s the theory but frankly the syntax is so cumbersome, irregular and needs so many googling for "easy" things like conditional, substring, etc. that I now use a real programming language if a script needs to be anything more than a list of commands without any logic (besides variables substitution).
You are basically describing modern programming.
Script Language (or scripting) is a programming language.
And about the "real" programming language you can also trap yourself googling and installing yet another library (did you read the code?) and/or reimplementing existing tools from the unix programming environment.
You are overly pedantic on a detail point that doesn’t matter: yes shell scripting is technically a programming language, but my point is that it is a terrible one worth ditching for any non trivial task. Perl was created precisely more than 3 decades ago to address this problem. Nowadays there are alternatives such as Python, Powershell or even scripting wrapper for compiled languages (such as C#) that allow to do the same job very well, with less surprising behavior and that can be refactored later more easily.
I disagree with you wholeheartedly and without condition.
Not only are you discounting how much time it takes to learn how to program effectively in a real 'glue' language you are high handed in ignoring the ubiquity, working archive and efficacy of a shell script. Not to mischaracterize but I find this type of attitude most frequently in 'lead' individuals with less than 10 years experience: typically 20's and 30's in age. I'm curious if this is your case?
Shell may be super useful to learn but that doesn't mean it's full of horrible footguns that greatly hamper your ability to determine if your code is doing the right thing or not once you get to a significant body of code: just take a look at https://mywiki.wooledge.org/BashPitfalls and compare with other languages that don't have some of the more egregious easy-to-make mistakes (e.g. anything to do with word splitting).
I agree, and as TFA mentions, shell scripting is an "incoherent mess of composable paraphernalia" where you have to memorize a “weird mishmash of trivia” -- but if you have memorized it, then you can do some neat stuff without a so-called "real" language installed (that you have to upgrade and have to worry about if you have a modern version installed)
Most people who use Unix directly build up some stuff in ~/bin (often a misnomer because it's shell scripts and not binaries, although mine is less of a misnomer than most because so much is in C rather than shell). The trick is to build them out of the standard portable components that exist everywhere. (This means, among other things, no #!/bin/bash.)
/bin/bash won't usually ship with a BSDish OS because of the license, so it is not generally portable to use bash-isms. (HPUX, IRIX, SunOS, Solaris, etc. I don't reckon would have had bash either)
I know this but usually don't do it. Been burnt by it too. Reprogramming your brain is hard.
[edit]
Would there be any negative consequences to having an automated process go through and change it? Maybe the size difference might cause some issues with some things doing black magic with data in the file.
> [...] but they're just scripts he has on his system. Personally I'd prefer to use the ubiquitous commands that work everywhere than rely on having custom scripts on my system [...]
Is okay for one to have their own tools.
$ f() { printf "\$%s" "$1"; }
$ echo a b c | awk '{ print $(f 2) }'
His system is not very different from mine or yours. He just chose to combine the tools in a specific way.
'sort | uniq -c | sort -n' is an interesting pipeline. It will always work and does a great job with large cardinality data on low memory systems.
However, if you have the ram, or know the data set has a low cardinality (like, http status codes or filesnames instead of ip addresses) then something that works in memory will be much more efficient.
I threw 144,000,000 'hello' and 'world' into a file:
justin@box:~$ ls -lh words
-rw-r--r-- 1 justin justin 824M Jan 7 15:21 words
justin@box:~$ wc -l words
144000000 words
justin@box:~$ time (sort <words|uniq -c)
72000000 hello
72000000 world
real 0m22.831s
user 0m32.999s
sys 0m4.675s
Compared to doing it in memory with awk:
justin@box:~$ time awk '{words[$1]++} END {for (w in words) printf("%s %d\n", w, words[w])}' < words
hello 72000000
world 72000000
real 0m10.639s
user 0m9.736s
sys 0m0.876s
This is because in the first example you are invoking two programs. The first one sort the content of the file, the second count how many lines are equal.
While in the awk example it is creating a hash table with all words and incrementing by the key and then printing.
There is no sorting plus printing may be buffered.
He's comparing apples to oranges and reaching the conclusion that... yes, apples and oranges are different things. He's quite aware of this, and even points out the tradeoff -- `sort | uniq -c` still works if your dataset doesn't fit into RAM.
Not exactly. sort (at least GNU sort) will end up doing external merge sort on temporary files if you give it more data than you have memory. Which, if you give it 100GB of 5 different strings, ends up being a huge waste.
I posted a comment on how 'sort | uniq -c | sort -n' is an interesting and very capable pipeline, but often misused and slower than other alternatives.
> you are comparing
Yes, I am comparing two methods of accomplishing the same thing. That is how comparing things works.
> Please, "huge waste"? How do you sort something that does not fit in memory?
Note how the full sentence included "if you give it 100GB of 5 different strings". If your input is 100GB of 5 different strings, then the hash table will easily fit in memory, and sorting the entire data set only to pass it to 'uniq -c' is indeed a 'huge waste'.
There are tons of large data sets that only have a small number of unique values in particular fields. protocols, ports, http status codes, hour of the day, etc. 'sort | uniq -c | sort -n' will work for all of them, but not nearly as efficient a hash table.
Even working in memory, there are different efficiencies for different methods.
Awk includes an asort() function which can sort an array, such that it would be possible to create a similar process entirely within awk to the sort | uniq -c pipeline:
In this case, sort | uniq is the fastest option. But the all-in-memory sort + separate tabulation of unique values in awk is notably slower (running in 143% of the time) than the also all-in-memory hash accumulator.
As I bump up the dataset size (20,000 records) that discrepency increases, roughly 0.052s sort|uniq, 0.065s hash, and 0.217s ask sort-unique.
TL;DR: test your assumptions, especially regarding performance.
Note: Data were generated with a simple bash loop:
for i in {1..2000}; do echo $((RANDOM%10)); done > data
With super small dataset sizes like that it'll fit in the L2 cache and can behave differently. I did test this though, and for 20,000 items generated with your loop:
sort | uniq -c takes .017s (fastest out of a few runs)
the awk command I used above takes .013s
A trivial implementation I have in go takes .08s
Additionally, using this 'protos' file which is 1,000,000 lines of tcp,udp,icmp:
$ time (sort protos|uniq -c)
5915 icmp
332003 tcp
662082 udp
real 0m0.232s
user 0m0.739s
sys 0m0.100s
$ # fixed to count 'lines' and not the first column, which makes it faster.
$ time awk '{lines[$0]++} END {for (l in lines) printf("%s %d\n", l, lines[l])}' < protos
icmp 5915
udp 662082
tcp 332003
real 0m0.194s
user 0m0.190s
sys 0m0.004s
$ time ./c < protos
662082 udp
332003 tcp
5915 icmp
real 0m0.088s
user 0m0.084s
sys 0m0.005s
And, more significantly, and as you've confirmed, not all in-memory processing is equivalent. There are faster and slower all-in-memory algorithms and implementations
Hrm... Maybe a bogounique implementation might be appropriate here....
Another nice thing about /usr/bin/time is the --verbose flag which gives:
Command being timed: "ls"
User time (seconds): 0.00
System time (seconds): 0.00
Percent of CPU this job got: 0%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 1912
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 112
Voluntary context switches: 1
Involuntary context switches: 1
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
If the time reserved word precedes a pipeline, the elapsed as well
as user and system time consumed by its execution are reported when
the pipeline terminates.
man time:
Some shells may provide a builtin time command which is similar
or identical to this utility. Consult the builtin(1) manual page.
This is very likely because without the full path your shell is using the `time` builtin function of your shell as opposed to using the binary.
The shell's builtin keyword for `time` is more limited in nature than the full `time` binary. This is true of a number of other common unix commands as well, e.g. `echo`. The manpage for your shell should describe the builtins functions.
"What if Unix had less compositionality but I could use it with less memorized trivia? Would that be an improvement? I don't know."
The answer is "no" here, because the alternative doesn't exist. Could it be created? Maybe in theory, but I suspect that the amount of stuff that you'd need to memorize (or learn to look up) to use it effectively would be about the same for any system that allowed a similar variety of work to be accomplished. If you are willing to trade off functionality for simplicity, then sure, it can be done. You can get it today by just not using all these tools at all, I suppose.
There would be less trivia to memorize if the command behaviors and options were more consistent. You may not be able to achieve that at the edges, where new commands and options are added, but you can always go back and clean things up.
For example, the cut(1) command is intended to do precisely what his f script does. But it's inconvenient because unlike many other commands it (1) doesn't obey $IFS and (2) the -d delimiter option only takes a single character. This could and should be remediated with a new, simple option.
I think the only thing preventing that change is that there's not enough interest in moving POSIX forward faster; certainly not like JavaScript.
Another problem are GNU tools. They have many great features but OMG are they a nightmare of inconsistency. BSD extensions tend to be much better thought through, perhaps because GNU tools tend to be lead by a single developer while BSD tools tend to be more team oriented.
So the way forward isn't to replace the organic evolution, it's to layer on processes that refine the proven extensions. And we already have some of those processes in place; we just need to imbue them with more authority, and that starts by not rolling our eyes at standardization and portability.
Authority is the problem...not standardization and portability. Everyone is willing and able to tell you the best way to do your work if you use their tools. Straitjacketing implementation in the name of order is a surefire way to dissuade people from using your tools.
Then there is the use of perl and its system command in "count". As with seq, why is perl needed. No explanation. Why not just put the entire pipeline into a perl system command.
This is a good point about the second half of the article (compositonality), but the author started the article by saying this was a command he "sometimes runs", presumably indicating he has it saved somewhere.
> The appearance of the TIME=… assignment at the start of the shell command disabled the shell's special builtin treatment of the keyword time, so it really did use /usr/bin/time. This computer stuff is amazingly complicated. I don't know how anyone gets anything done.
65 comments
[ 6.0 ms ] story [ 166 ms ] threadI think it's less about not trusting custom scripts than it is about ensuring that your unix muscle memory doesn't atrophy.
You start thinking about packaging.
I would be mad at any admin that would dare to deploy his helper shell scripts like “addup” and “count” on a machine other than his laptop.
And if you meant he could just have these things in his home, then it defeats the purpose of the original comment: trouble shooting and administering machines forces you to often switch user, machine, etc.
Then instead of having to remember how to check all these things, they just run "find_resource_hogs.sh" and voila. It also enables other people to troubleshoot without specific knowledge.
Of course you don't want to put anything in there just because it might save 10 seconds, but then https://xkcd.com/1205/
I spend a lot of time moving around different machines, processes, configuration files, logs, etc. And I stopped maybe 10 years ago to use anything that is not available on a base system.
I don’t use fancy shells, I don’t use aliases, I don’t write local shortcut scripts, etc.
I just use regular bash, combine base utilities in one liners, and live with it.
Maybe I loose 1s here and there when writing one liners compared to someone with a library of wrapper utilities. But that gives me an immense benefit: I am at home on any machine, any distribution, everywhere, without any configuration, with any user.
Sure, in a quest of productivity, I should also use Vagrant and Packer to create docker images with a development environment so that I can run Serverless troubleshooting containers on all my machines.
These scripts will surely help me triple the performance of my bash one liners.
I’ve heard Eclipse has good shell completion and support for oh-my-zsh too.
You are basically describing modern programming.
Script Language (or scripting) is a programming language.
And about the "real" programming language you can also trap yourself googling and installing yet another library (did you read the code?) and/or reimplementing existing tools from the unix programming environment.
Now that I saw tingletech comment I got your point :)
[edit] Would there be any negative consequences to having an automated process go through and change it? Maybe the size difference might cause some issues with some things doing black magic with data in the file.
Is okay for one to have their own tools.
His system is not very different from mine or yours. He just chose to combine the tools in a specific way.However, if you have the ram, or know the data set has a low cardinality (like, http status codes or filesnames instead of ip addresses) then something that works in memory will be much more efficient.
I threw 144,000,000 'hello' and 'world' into a file:
Compared to doing it in memory with awk: so, half the time and 1/3 the cpu.While in the awk example it is creating a hash table with all words and incrementing by the key and then printing.
There is no sorting plus printing may be buffered.
Also, you don't need to spawn a subshell nor feed sort via stdin in the first example :)
Heck, forget about RAM, the output of both programs don't even match.
That awk is pretty efficient and fast is no surprise ;)
Please, "huge waste"? How do you sort something that does not fit in memory?
I posted a comment on how 'sort | uniq -c | sort -n' is an interesting and very capable pipeline, but often misused and slower than other alternatives.
> you are comparing
Yes, I am comparing two methods of accomplishing the same thing. That is how comparing things works.
> Please, "huge waste"? How do you sort something that does not fit in memory?
Note how the full sentence included "if you give it 100GB of 5 different strings". If your input is 100GB of 5 different strings, then the hash table will easily fit in memory, and sorting the entire data set only to pass it to 'uniq -c' is indeed a 'huge waste'.
There are tons of large data sets that only have a small number of unique values in particular fields. protocols, ports, http status codes, hour of the day, etc. 'sort | uniq -c | sort -n' will work for all of them, but not nearly as efficient a hash table.
Programming is about paying the bare minimum attention to the details.
> [...] two methods of accomplishing the same thing [...]
Absolutelly not.
one prints:
the other Now try both examples against a file with more than one column to understand what I'm talking about ;)Awk includes an asort() function which can sort an array, such that it would be possible to create a similar process entirely within awk to the sort | uniq -c pipeline:
As compares with a hash-based counter: On a 2,000 value test dataset with 10 unique values:sort | uniq -c takes 0.019s (8 runs averaged)
awk hash takes 0.023s (8 runs averaged)
awk-implemented sort + unique takes 0.33s (8 runs averaged)
In this case, sort | uniq is the fastest option. But the all-in-memory sort + separate tabulation of unique values in awk is notably slower (running in 143% of the time) than the also all-in-memory hash accumulator.
As I bump up the dataset size (20,000 records) that discrepency increases, roughly 0.052s sort|uniq, 0.065s hash, and 0.217s ask sort-unique.
TL;DR: test your assumptions, especially regarding performance.
Note: Data were generated with a simple bash loop:
sort | uniq -c takes .017s (fastest out of a few runs)
the awk command I used above takes .013s
A trivial implementation I have in go takes .08s
Additionally, using this 'protos' file which is 1,000,000 lines of tcp,udp,icmp:
so yes, I do test my assumptions.And, more significantly, and as you've confirmed, not all in-memory processing is equivalent. There are faster and slower all-in-memory algorithms and implementations
Hrm... Maybe a bogounique implementation might be appropriate here....
(By analogy to bogosort: https://en.wikipedia.org/wiki/Bogosort)
Another nice thing about /usr/bin/time is the --verbose flag which gives:
:)man bash:
man time:The shell's builtin keyword for `time` is more limited in nature than the full `time` binary. This is true of a number of other common unix commands as well, e.g. `echo`. The manpage for your shell should describe the builtins functions.
The answer is "no" here, because the alternative doesn't exist. Could it be created? Maybe in theory, but I suspect that the amount of stuff that you'd need to memorize (or learn to look up) to use it effectively would be about the same for any system that allowed a similar variety of work to be accomplished. If you are willing to trade off functionality for simplicity, then sure, it can be done. You can get it today by just not using all these tools at all, I suppose.
> I don't know. I rather suspect that there's no way to actually reach that hypothetical universe.
For example, the cut(1) command is intended to do precisely what his f script does. But it's inconvenient because unlike many other commands it (1) doesn't obey $IFS and (2) the -d delimiter option only takes a single character. This could and should be remediated with a new, simple option.
I think the only thing preventing that change is that there's not enough interest in moving POSIX forward faster; certainly not like JavaScript.
Another problem are GNU tools. They have many great features but OMG are they a nightmare of inconsistency. BSD extensions tend to be much better thought through, perhaps because GNU tools tend to be lead by a single developer while BSD tools tend to be more team oriented.
So the way forward isn't to replace the organic evolution, it's to layer on processes that refine the proven extensions. And we already have some of those processes in place; we just need to imbue them with more authority, and that starts by not rolling our eyes at standardization and portability.
This is slower than not running seq and just using builtins.
If you don't want the inefficiencies of seq, bash has:
which is a lot more idiomatic than constructing a for loop out of a while loop.https://wiki.ubuntu.com/DashAsBinSh
Then there is the use of perl and its system command in "count". As with seq, why is perl needed. No explanation. Why not just put the entire pipeline into a perl system command.
1) start timer
2) start deciding which commands to pipeline together
3) run the commands
4) stop timer
a lot of times the decision is the long pole.
in this authors case it included:
5) try a couple more variants of steps 2 and 3
6) write a blog post
:)
Indeed.