They do not seems to warrant further investigation, the examples you find about these implementations are overly simplified bordering useless. If you are going that way you might as well go with something like PowerShell, and that's a horrible scriping language.
We use node at work now. The fact is it's not worth it trying to get everyone to write good bash scripts. Js doesn't have nearly as many foot guns and there's a library for just about anything you could want to do.
True for async code, but if you use something like ShellJS [1] (which would be an easy way to rewrite a bash script that was getting out of hand), all commands are synchronous, so error handling is just try/catch.
If only everyone used promises...I can dream, right?
Regardless, I'd actually prefer Haskell's do-notation for both asynchronous commands and shell-scripting tasks. Even though Haskell isn't my favorite language, do-notation is quite nice for this.
For context, since I think some people don't know this and are down voting you, this blog post is from Jane Street (probably the largest commercial OCaml deployment?). As another poster noted Haskell has many tools/libraries for writing scripts which makes dtoma's question very reasonable.
It's happened! We have some internal magic that lets us run ml directly: ./my_bash_script_replacement.ml, and it makes replacements fairly painless.
But we don't do it often. Each language has its strengths and weaknesses, and the constraints of a small script tend not to change that much over time. The script either starts other executables and pipes some text around, or it does some computation. The choice of which to use is often (but not always) clear.
I'm not sure I agree with the "don't do it often". Having jane-script (which is our OCaml scripting system) has allowed us to greatly reduce our dependence on Bash.
That said, there are still little things we use Bash for. But our tolerance for large bash scripts has diminished greatly over the years.
Yes, subshells can be tricky. Don't rely on stuff like pipefail, rather check return codes and react accordingly. Bash also provides a way to read return codes of piped commands. See PIPESTATUS in man bash.
Slightly more useful than `set -e` is `set -E` and `trap exit ERR`.
But still there's basically no way to make this consistently useful.
Even if you religiously set -e/E in every scope just in case, if you're anywhere in a scope inside the non-final operand of a bunch of &&/||s, or as the conditional expression in a control structure or whatever, -e/E will just do nothing, you can't turn it any more on, you just don't get early termination on errors no matter how many nested function calls you're actually removed from the original ||. It's not great.
As an aside, I like reading Jane Street's blog. They're one of the few companies in our space (I'm also in "automated trading") that discusses even non-proprietary stuff openly. When you work in an information vacuum, it's comforting to know that presumably similar people face the same challenges.
When the author wrote "a particular production bash script (if that doesn't sound horrifying, hopefully it will by the end of this post)," I couldn't help but smile...
It's a great blog, they usually strike a great balance on the practical/theoretical scale for the articles. Would love for them to up the frequency of the posts, maybe they will once they find the writer they're looking for.
Do you happen to have some recommendations for similar blogs?
They are fast, ubiquitous, 0-dependencies, short, espressive... most of the time. The fact is that we don't have anything better to glue commands together (afaik).
"Glueing commands together" is called programming. That's what programming languages are for. Bash etc are just really bad programming languages that lure you into a trap of unnecessary complexity with their superstitial expressiveness for the happy path in short programs, when you don't need anything besides the built-ins. Decades of ubiquitous UNIX crap tricked you into believing that bash's properties are advantages rather that disadvantages. Sure, you can "glue together" a few commands quickly, but only at the price of horrible edge cases and hours of StackOverflowing for doing the simplest of things.
>but only at the price of horrible edge cases and hours of StackOverflowing for doing the simplest of things.
Or seconds, after you learn the search term "Perl one-liner" and add it to your queries for all your edge cases.
Start thinking of perl -we as a Bash built-in function.
Suddenly edge cases go from hours to your typing speed into Google.
Don't forget to leave a comment with a link to where you found the solution, what it does according to that site, and an apology for the unreadable line noise. If you make changes to what you copy and paste, explain your changes in a comment.
(Just in case it sounds sarcastic or ironical, this comment is serious and how I really work. I am really glad for my comments whenever I revisit old scripts.)
Actually, quite the opposite. All that shell scripts do is glue together other programs they depend on.
That makes them not portable. You can't just move a bash script from Linux to macOS because it uses ancient pre-GPLv3 versions of bash and all the other GNU utils.
This is not so unreasonable a question, but the answer is simple: because it's ubiquitous, fairly standardised, and well supported.
Also, other languages suck at doing small things quickly and simply. Try rewriting a shell script in go with little experience and see how long it takes.
Yeah, in the OP why wouldn't you make the same LDAP query using any of the numerous libraries available for other languages? I'll take the tiniest little bit of complexity of having to make sure Python and PiP are installed on my environment, over a Bash script falling apart at the seams from an edge case.
One reason is that any System Administrator can be expected to be able to understand and maintain Bash scripts.
I recently built a fairly complex system entirely in Bash, not because it was the "best" language for the job (it would have been much easier to do in Ruby or Python), but because the client didn't have any permanent staff that could maintain Ruby or Python scripts.
As a contractor, one of the major factors in the technical decisions I make is: how supportable is this technology for the client, once I have left? Of course the answer to that question will vary on a case by case basis, but Bash is usually a good lowest common denominator in a Linux environment.
I highly recommend ShellCheck[0] if you're writing any bash. With the warnings and stylistic advice it provides, I feel like I can actually be confident that my scripts are doing what I think they're doing.
However, don't let it give you a false sense of security.
Shellcheck probably won't say anything about the "biting" parts from the article –
because they are valid, just behaving a bit different than the user expects…
That goes for any linter or static analyser i guess.
Thanks! That's a great tip. I've been writing a large amount of shell scripts (mainly sh not so much bash) and already made some improvements thanks to this!
I recently started having to do a lot of shell work, and shellcheck has helped immensely. It was unfortunate that there was no precompiled version of GHC for my OSX version on homebrew though, so it took 5 hours to build GHC. I ended up turning off all the power save and auto sleep features on my laptop to make sure it kept going.
Well, shell scripts are extremely convenient. It's easy to write very simple ones since they're a natural extension of shell usage.
So it's really easy to go from
$ command1
$ command2
to:
#!/bin/bash
command1
command2
As a result, everybody "likes" bash scripts since they're so easy to write, initially.
And these shell scripts will work well in 80% of situations and you can get them to 90-95% just by sprinkling a few ifs around.
Once thes script becomes longer and more complex and you want to make them easier to read and DRYer, that's when the pain begins. Or when you need to handle a bit of logic which would be trivial to do with better data structures such as arrays or dictionaries or sets...
You can work around these issues, of course, but almost every workaround is either ugly or a hack (as someone was joking, "elegant hacks" for older Unix hackers, "gross hacks" for younger ones :) ).
I agree - in my experience bash is extraordinarily easy to write the first time. It becomes a nightmare to maintain, because most people aren't actually that familiar with how bash error handling, flow logic etc. actually works. It's bad because it fails in surprising, non-conventional ways.
I just over-log absolutely everything. If it fails log it. If it succeeds and you might need to know about it, log it. That's not great error handling, but it helps with debugging.
then why not use python? Installing another shell just for running another script seems wrong unless it is only for you personally and you are ok with this.
When I see very long bash scripts I get angry. There is a reason why perl & later python were invented and they are both in 98% of cases available on Unix compatible systems.
Not that I'm not ok with short bash scripts or for systems were you only have bash... but if your Unix compatible system only has bash there is something else wrong with your system in my point of view.
Python and perl are not always the best option. For example, if I am executing a lot of system commands (sometimes through ssh), checking outputs with grep/awk, reading files, calling make, why use a language that is going to be far more verbose, difficult to modify, and show problems because it's not executing the shell in the correct profile or things like that? For interacting with the system, there's nothing better than shell scripts. You mainly only have to take care with the quotes to avoid word separation and globbing, and use the *nix commands to avoid recreating the wheel.
I'll second this. My Python is far better than my bash but there's no point writing more complicated Python if you're working with system commands. That said, quite often I'll break non-system stuff out into python scripts and wrap them in bash if I need to do a bit of both.
I'd much rather have 200 lines of bash than the 300-400 lines of python to replace it.
I have recently rewritten a lot of Python and Perl (over 10kLOC) to a small collectiom of 10-ish bash scripts that weigh in at about 700LOC total. There are some tasks where Bash is the only appropriate tool.
Much simpler code, much easier to read, and a lot faster too.
Bash lets you glue best-of-breed tools together, as long as they all talk on stdin / stdout / files. The tools themselves can be written in the best language for their task.
If you're measuring code quality / maintainability by how many lines of code there are, you're doing it wrong. I'd much rather see 400 lines of well-structured python than 200 lines of tangled bash. For the rewrite you did, how much of the benefit was because it was rewritten, rather than being rewritten _in bash_?
I'd also go as far as saying that if you're using bash because it's faster than python, you're also using the wrong tool for the job.
The quality of the bash far outshined the quality of the python and perl, because Python and Perl was being used to do a bash scripts job. When what you need is to call a bunch of binaries and redirect output around, then Python and Perl just don't cut it.
The code is much, much easier to read now. You can see a script on a single screen height, which works wonders for overview. The 3x performance gain spawned from enlightenment this overview granted us.
But don't get me wrong, in the very same rewrite, I rewrote 100 lines of bash + 300 lines of Python into a single 200 lines of Python, which when combined with PyPy, was 50 times faster than the original setup. Python doesn't do bash's job very well, and bash doesn't do Python's job very well either.
I agree that for the simple job of "start process, maybe echo some stuff", then bash is superior to python. However, my difficulty always arises the moment you want to start writing 'if', or handle some sort of error condition.
Once you go down that road, you're into magic switches at the top of your script (which TFA demonstrates aren't well understood), and subtle control issues where bash's flow control doesn't quite do what you think it should, and (for me) this is where bash stops being superior.
I don't think flow control is a problem, as long as it's just to complete a simple purpose. Checking if a file is present, running some code for every file or every line, etc.
But bash is definitely not something where complicated logic belongs.
Although it's been a long time since I last did perl programming, it has always been my understanding that perl is particularly well designed for piping and mangling input and output from binaries. It's been the successor of awk, sed, grep and cut, all contained in one, relatively consistent and well documented language. Are you saying Larry Wall failed in this design attempt? Could it be that Perl wasn't used to its fullest potential?
Not the best way to do that sort of thing (useless use of cat for instance), I was just trying to come up with an executable example. run returns success or failure of the pipeline as the return code, which I don't show here. I don't think you can pick out which command failed.
I also find that once you're in a real programming language, you have much less need to have deep pipelines anyhow. Consider the pure-perl equivalent
open PW, "<", "/etc/passwd" or die;
while (<PW>) {
next if $_ =~ m|:/bin/bash$|;
$count++;
}
print "$count\n";
Of course "real perl" would suggest some form of "use strict" and the corresponding need to declare the $count var, but if we're discussing shell script contexts it is at least fair to consider ignoring that, since it would be a silly objection that the perl equivalent doesn't turn on checks that shell scripting doesn't even have.
But if you're in a real programming language you don't need to shell out to wc just to get a line count, or write complicated awk scripts, etc. It's much more common for me in those cases to execute a single program and control its STDIN/STDOUT/STDERR and watch its exit code.
I am very much against ever shelling out from an application. Either write a shell script, or a proper application. Nothing bugs me more than shell scripts written in Python, Perl, or even gasp C++ (he was later fired).
However, most bash scripts I interact with will take at least 2-4 times the amount of code to replicate as a proper application. The line count example is a good pro-shell one, with 6 not particularly nice lines of code replacing a single, simple line:
egrep -v ':/bin/bash$' /etc/passwd | wc -l
"Complicated" AWK scripts are also much more compact than their equivalent. Say, `xyz | awk '/thing/ { getline; print $0; }'` to get the line after a pattern. I wouldn't write an AWK script much more complicated than that, though, and for me, Perl and AWK belong in the same bucket.
My objection to shell is that it obtains that line advantage by cheating. It has terribly poor error handling, it uses an incredibly sloppy serialization format that frequently breaks at the scales people attempt to apply it at, its performance characteristics are terrible, its debugging story is nonexistant, perhaps worst of all it affords a programming style that blinds people to how much compromise they are making with their data because of the aforementioned serialization format, and on it goes. Of course programming languages can't get as concise as shell; even the sloppiest programming language communities there are have the bar for quality set substantially above what shell can meet.
Your example isn't even equivalent to my rather sloppy Perl, since, for instance, it will do something quite different if /etc/passwd doesn't exist. Shell only has the size advantage as long as everything goes perfectly correctly.
Shell is only good for two cases: 1. You don't much care about the integrity of your input or output, and you don't much care about what happens if something goes wrong. 2. When you don't care about the aforementioned issues because you're right there on the spot and can fix things if they do go wrong, because you're using shell interactively.
Having fiddled around with designing a "new shell" every so often my current conclusion is that there is no way to bridge the gap between the optimal interactive use shell and the optimal shell script with one language, and modern shell scripting, for all the features they have apparently for shell scripts, are firmly for the interactive case.
My example is still very well defined, and very easy to reason about. It will output '0' on stdout if no file is found, and an error message on stderr.
Bash is very good for the things its good at, but you still need to know what you're doing to not fuck it up. Some people don't know how to consider error handling when they code, but that's the fault of the programmer.
I used to believe that. I don't any longer. Your second paragraph basically agrees with me that bash affords bad code that ignores errors. I think it's past time for the programming world to take that seriously. We've had 60-some years of programmers insisting they can heroically just remember all the things they need to do to write correct code. And they are observably wrong. To disagree with me on that is basically to assert that software is nearly universally of a high quality and generally robust to all reasonably forseeable error conditions.
Shell goes even beyond that, though; as evidenced by articles like this you can't even get good agreement on how you should write safe shell. When even experts can't agree on what's safe, that's just something unsuitable to any serious task.
None of my previous nor future paragraphs agree with you. You must have misread.
You are presenting a false analogy. I was not, am not and will never argue that one needs to be an oracle that foresees all errors (which would be absurd), but I am arguing that poor error handling is a programmer error due to the situation not being Bash specific. To reword my statement from the previous comment: If your error handling in Bash is insufficient, your error handling would also be insufficient in most other languages with implicit error handling (e.g., those with unchecked exceptions).
Once again, your Perl blob is a great example. You manually added the optional `or die` to deal with the case where the file wasn't present - perl's equivalent of `|| exit 1`. If you had not remembered to add that check, then your program would fail silently, with $count being an undefined variable. You could try to enforce better checking in the local block scope with some hacky header at the beginning of the script ("use strict;"), but that wouldn't stop you from having poor code in a module, and it would only ensure that count was defined, not that the program was well-behaved. That sounds awfully sloppy, doesn't it? Almost like what you were complaining about for Bash!
Very few languages provide you with any assistance to remind your of error checking (failing either silently or by crashing), mostly due to the awful concept of exceptions (unchecked, specifically, but checked exceptions are only useful if unchecked don't exist). Go, Rust, and - ironically - C helps you by forcing you to think about error handling at every single call-site, with lazy error handling sticking out like a sore thumb (explicitly ignored return values, which look weird in these languages), while C++, Java, Perl, Python and Bash all expect you to decide what you feel like handling today.
People always recommend Perl over a shell script but I've never seen a situation where it makes sense. Having written very little Perl, it seems like its only strength is that it's present everywhere and has a large library.
Language-wise, Python is far superior and has a large ecosystem and these days it has ubiquity too.
I'd be interested in the argument for Perl over shell scripts that does not apply to just writing the program in a modern language.
Have you ever done string manipulation (also RegEx) over many files in Perl? I think it's in this regard superior than anything else, even python when looking at syntax.
I don't know how sed and perl compare, but I know these things:
Use a C local (e.g. LANG=C or LC_ALL=C) for dramatic speed up of tools like grep/sort and probably also sed. If you don't do that, usually the UTF-8 parser kicks in, which is a lot slower.
Perl is actually well known to have a non-optimal regex implementation: https://swtch.com/~rsc/regexp/regexp1.html That said, I don't know how fast it is (what optimizations it has) at simple string substitutions.
I have not. I usually use `sed` in such instances but admitting the superiority of Perl in such instances surely this is insufficient on its own to use Perl? Seems like a rare use-case and if that script grows large then Python has superior language features but if the script stays small, then a Bash script with `sed` will do just as well.
the problem with Python where I work is that developers are openly hostile toward it. We're predominantly .NET with just a couple of products running on Linux servers. I'm not a .net guy despite the pressure, and have been teaching myself python in my own time, but the the rest of the staff including management are never going to look for Python as a skill when hiring. EVER. But of course 2.7 is sitting there on the CentOS servers never used, while bash scripted cron jobs run amok and frequently fail (because they were written by the wrong people imo).
I would love to see Python utilized, and maybe with VSCode and MS integrating open source into the .NET world it will eventually come to enterprise (where I work).
it's not about what MS put in VS and VSCode, it's about changing the minds of .NET developers who used are to C# syntax/grammar and hate Python because it doesn't fit their world-view. "Already in VS" is meaningless in a .NET enterprise company.
I feel sorry for you. Fanatism towards any language/eco-system, no matter if .net, Java, Ruby, Python (also have seen the same with Python developers) is something I really don't like.
This is the first time I have seen anoybody hold up perl as the superior option. I don't doubt that it will work, but I need the next admin to be able to read my code.
Also, every system may have some Python installed, bud by the time I've dealt with versions (2.6 is the CentOS 6 default, 3.6 is the current version) and dependencies (virtualenv? system wide pip? PYTHONPATH? RPM Python modules?) I'd better get a significant improvement.
I was just thinking how cool would it be to have a module in python to really ease generating bash style scripts, with less overhead than normal `subprocess` methods have... And there was it just a Google search away: https://pypi.python.org/pypi/sh/1.12.13
Author of sh.py here. It seems to be used by many people in production around the world. I've actively maintained it since 2011, and it supports python 2.6-3.6, inclusive. Most of the gotchas have been worked out by now, but the FAQ covers the most common stumbling blocks: http://amoffat.github.io/sh/sections/faq.html
A related annoyance is that when you write a for-loop in a Makefile, every iteration gets run even if one of them fails, due to how the shell calculates exit codes of for-loops.
So most of the time you should do "set -e; for XXX". Otherwise your Makefile loops will "succeed" incorrectly.
Not that people should be expected to know this, but here's an idiom that does work:
if res="$(ldap-query-for-valid-users)"; then
echo "($res)" > "/tmp/all-users.sexp"
else
handle_failure
fi
I second the recommendation to use https://github.com/koalaman/shellcheck – you really shouldn't be writing shell scripts without it – but in this case it doesn't seem to handle the issue (with default settings at least).
This doesn't mention one of my favorite bash gotchas, which is using `set -e` with `pipefail` at all. Try this:
set -euo pipefail
yes | head
That will consistently exit because `yes` gets sigpipe and quits. Which is expected, but triggers a script exit. But more exciting is that something like:
generate_data | head
only _sometimes_ fail. It's a race that depends on whether generate_data is able to stuff all of its data into the pipe buffer before head calls close().
-c Read commands from the command_string operand instead of from the standard input. Special parameter 0 will be set from the command_name operand and the positional parameters ($1, $2, etc.) set from the remaining argument operands.
/tmp generally has more open permissions than the rest of the file system, and if you're using a known file name, you're open to that file being abused by an attacker.
Yeah, naively using errexit won't get you the whole way. BashFAQ, ShellCheck and StackOverflow have quite a bit of information, and you can often write things that are pretty robust if you take in all the information. Particular cases that come to mind are:
1) command substitution. you can catch this with an explicit error handler inherited by child processes.
2) pipefail SIGPIPE false positives. pretty hairy, command dependent whether this is a "real" error. often not, so you can work around it by ignoring SIGPIPE
3) process substitution. As far as I know, there is no way to workaround this whilst still using the convenience syntax. You have to carefully use explicit named pipes and carefully use wait on the PIDs (carefully!). Maybe still with races...
In my experience, you can write moderately robust shell scripts if you care enough and use all these flags and linters. But by the time you are at this stage you probably shouldn't be using shell scripting. More like training to spot problems in other people's code.
142 comments
[ 3.1 ms ] story [ 141 ms ] threadocaml isn't that far off, just even less popular...
[bos]: http://erratique.ch/software/bos
[1] https://www.npmjs.com/package/shelljs
Regardless, I'd actually prefer Haskell's do-notation for both asynchronous commands and shell-scripting tasks. Even though Haskell isn't my favorite language, do-notation is quite nice for this.
But we don't do it often. Each language has its strengths and weaknesses, and the constraints of a small script tend not to change that much over time. The script either starts other executables and pipes some text around, or it does some computation. The choice of which to use is often (but not always) clear.
That said, there are still little things we use Bash for. But our tolerance for large bash scripts has diminished greatly over the years.
set -e
eg:
set -euo pipefail
foo() {
variable=$(foo) # failsfoo | do_stuff # fails
My preference is to handle things as streams
But still there's basically no way to make this consistently useful.
Even if you religiously set -e/E in every scope just in case, if you're anywhere in a scope inside the non-final operand of a bunch of &&/||s, or as the conditional expression in a control structure or whatever, -e/E will just do nothing, you can't turn it any more on, you just don't get early termination on errors no matter how many nested function calls you're actually removed from the original ||. It's not great.
Alternatively,
if they prefer a nonzero exit code to a message to stderr.When the author wrote "a particular production bash script (if that doesn't sound horrifying, hopefully it will by the end of this post)," I couldn't help but smile...
I interviewed there, but unfortunately didn't make the cut.
Do you happen to have some recommendations for similar blogs?
Or seconds, after you learn the search term "Perl one-liner" and add it to your queries for all your edge cases.
Start thinking of perl -we as a Bash built-in function.
Suddenly edge cases go from hours to your typing speed into Google.
Don't forget to leave a comment with a link to where you found the solution, what it does according to that site, and an apology for the unreadable line noise. If you make changes to what you copy and paste, explain your changes in a comment.
(Just in case it sounds sarcastic or ironical, this comment is serious and how I really work. I am really glad for my comments whenever I revisit old scripts.)
Actually, quite the opposite. All that shell scripts do is glue together other programs they depend on.
That makes them not portable. You can't just move a bash script from Linux to macOS because it uses ancient pre-GPLv3 versions of bash and all the other GNU utils.
For scripts where you need something unstandardized, say, imagemagick, you would wind up with up with a dependency in most other languages as well.
Also, other languages suck at doing small things quickly and simply. Try rewriting a shell script in go with little experience and see how long it takes.
I recently built a fairly complex system entirely in Bash, not because it was the "best" language for the job (it would have been much easier to do in Ruby or Python), but because the client didn't have any permanent staff that could maintain Ruby or Python scripts.
As a contractor, one of the major factors in the technical decisions I make is: how supportable is this technology for the client, once I have left? Of course the answer to that question will vary on a case by case basis, but Bash is usually a good lowest common denominator in a Linux environment.
[0]: https://github.com/koalaman/shellcheck
However, don't let it give you a false sense of security.
Shellcheck probably won't say anything about the "biting" parts from the article – because they are valid, just behaving a bit different than the user expects…
That goes for any linter or static analyser i guess.
E.g. "if (x = 1)" is valid C but i'd be pretty unimpressed if a linter didn't flag it up!
Perhaps it's time for a new shell language that is less arcane, with fewer gotchas and simply less bug potential?
Unless you mean something entirely different than POSIX shell family, e.g. Powershell, Python, Ruby, or JS.
After the meeting I said to colleagues, 'I quite like bash scripts, actually', and they all said 'I thought that too...'
So it's really easy to go from
$ command1
$ command2
to:
#!/bin/bash
command1
command2
As a result, everybody "likes" bash scripts since they're so easy to write, initially.
And these shell scripts will work well in 80% of situations and you can get them to 90-95% just by sprinkling a few ifs around.
Once thes script becomes longer and more complex and you want to make them easier to read and DRYer, that's when the pain begins. Or when you need to handle a bit of logic which would be trivial to do with better data structures such as arrays or dictionaries or sets...
You can work around these issues, of course, but almost every workaround is either ugly or a hack (as someone was joking, "elegant hacks" for older Unix hackers, "gross hacks" for younger ones :) ).
POSIX sh doesn't have these, but BASH actually does have arrays, and even dicts (see https://stackoverflow.com/questions/688849/associative-array...)
Or you could just learn Perl...
Or if you're just stringing commands together, traditional simpler Posix sh (note: 'bash --posix' isn't).
[0]: http://xon.sh/
[1]: https://william-droz.com/xonsh-a-modern-shell-that-enable-py...
Not that I'm not ok with short bash scripts or for systems were you only have bash... but if your Unix compatible system only has bash there is something else wrong with your system in my point of view.
I have recently rewritten a lot of Python and Perl (over 10kLOC) to a small collectiom of 10-ish bash scripts that weigh in at about 700LOC total. There are some tasks where Bash is the only appropriate tool.
Much simpler code, much easier to read, and a lot faster too.
Other than that, the simpler design also allowed greater parallelism.
I'd also go as far as saying that if you're using bash because it's faster than python, you're also using the wrong tool for the job.
The code is much, much easier to read now. You can see a script on a single screen height, which works wonders for overview. The 3x performance gain spawned from enlightenment this overview granted us.
But don't get me wrong, in the very same rewrite, I rewrote 100 lines of bash + 300 lines of Python into a single 200 lines of Python, which when combined with PyPy, was 50 times faster than the original setup. Python doesn't do bash's job very well, and bash doesn't do Python's job very well either.
Once you go down that road, you're into magic switches at the top of your script (which TFA demonstrates aren't well understood), and subtle control issues where bash's flow control doesn't quite do what you think it should, and (for me) this is where bash stops being superior.
But bash is definitely not something where complicated logic belongs.
But doing stuff like executing commands with standard output or standard error redirection, with pipes, etc. is something it's not strong at.
That's what a shell script is strong at.
To clarify, I'd: use bash to create an execution pipeline, which itself may contain a Perl script which munges some output.
Bash in this case is more of a "pipeline and execution orchestrator", and Perl is used to "properly munge the output" to make stuff happen.
I also find that once you're in a real programming language, you have much less need to have deep pipelines anyhow. Consider the pure-perl equivalent
Of course "real perl" would suggest some form of "use strict" and the corresponding need to declare the $count var, but if we're discussing shell script contexts it is at least fair to consider ignoring that, since it would be a silly objection that the perl equivalent doesn't turn on checks that shell scripting doesn't even have.But if you're in a real programming language you don't need to shell out to wc just to get a line count, or write complicated awk scripts, etc. It's much more common for me in those cases to execute a single program and control its STDIN/STDOUT/STDERR and watch its exit code.
However, most bash scripts I interact with will take at least 2-4 times the amount of code to replicate as a proper application. The line count example is a good pro-shell one, with 6 not particularly nice lines of code replacing a single, simple line:
"Complicated" AWK scripts are also much more compact than their equivalent. Say, `xyz | awk '/thing/ { getline; print $0; }'` to get the line after a pattern. I wouldn't write an AWK script much more complicated than that, though, and for me, Perl and AWK belong in the same bucket.Your example isn't even equivalent to my rather sloppy Perl, since, for instance, it will do something quite different if /etc/passwd doesn't exist. Shell only has the size advantage as long as everything goes perfectly correctly.
Shell is only good for two cases: 1. You don't much care about the integrity of your input or output, and you don't much care about what happens if something goes wrong. 2. When you don't care about the aforementioned issues because you're right there on the spot and can fix things if they do go wrong, because you're using shell interactively.
Having fiddled around with designing a "new shell" every so often my current conclusion is that there is no way to bridge the gap between the optimal interactive use shell and the optimal shell script with one language, and modern shell scripting, for all the features they have apparently for shell scripts, are firmly for the interactive case.
Bash is very good for the things its good at, but you still need to know what you're doing to not fuck it up. Some people don't know how to consider error handling when they code, but that's the fault of the programmer.
Shell goes even beyond that, though; as evidenced by articles like this you can't even get good agreement on how you should write safe shell. When even experts can't agree on what's safe, that's just something unsuitable to any serious task.
You are presenting a false analogy. I was not, am not and will never argue that one needs to be an oracle that foresees all errors (which would be absurd), but I am arguing that poor error handling is a programmer error due to the situation not being Bash specific. To reword my statement from the previous comment: If your error handling in Bash is insufficient, your error handling would also be insufficient in most other languages with implicit error handling (e.g., those with unchecked exceptions).
Once again, your Perl blob is a great example. You manually added the optional `or die` to deal with the case where the file wasn't present - perl's equivalent of `|| exit 1`. If you had not remembered to add that check, then your program would fail silently, with $count being an undefined variable. You could try to enforce better checking in the local block scope with some hacky header at the beginning of the script ("use strict;"), but that wouldn't stop you from having poor code in a module, and it would only ensure that count was defined, not that the program was well-behaved. That sounds awfully sloppy, doesn't it? Almost like what you were complaining about for Bash!
Very few languages provide you with any assistance to remind your of error checking (failing either silently or by crashing), mostly due to the awful concept of exceptions (unchecked, specifically, but checked exceptions are only useful if unchecked don't exist). Go, Rust, and - ironically - C helps you by forcing you to think about error handling at every single call-site, with lazy error handling sticking out like a sore thumb (explicitly ignored return values, which look weird in these languages), while C++, Java, Perl, Python and Bash all expect you to decide what you feel like handling today.
Language-wise, Python is far superior and has a large ecosystem and these days it has ubiquity too.
I'd be interested in the argument for Perl over shell scripts that does not apply to just writing the program in a modern language.
Great for a {poor man's,simple and easy to understand} basic template system to be used inside a bash script ;)
Use a C local (e.g. LANG=C or LC_ALL=C) for dramatic speed up of tools like grep/sort and probably also sed. If you don't do that, usually the UTF-8 parser kicks in, which is a lot slower.
Perl is actually well known to have a non-optimal regex implementation: https://swtch.com/~rsc/regexp/regexp1.html That said, I don't know how fast it is (what optimizations it has) at simple string substitutions.
I would love to see Python utilized, and maybe with VSCode and MS integrating open source into the .NET world it will eventually come to enterprise (where I work).
And it's a dream in VSCode + the most popular python extension https://github.com/DonJayamanne/pythonVSCode
Also, every system may have some Python installed, bud by the time I've dealt with versions (2.6 is the CentOS 6 default, 3.6 is the current version) and dependencies (virtualenv? system wide pip? PYTHONPATH? RPM Python modules?) I'd better get a significant improvement.
Related HN thread from almost 5 years ago: https://news.ycombinator.com/item?id=4530897
EDIT: just learned that PBS is now sh.py, so I removed it from the comment.
http://www.haskellforall.com/2015/01/use-haskell-for-shell-s...
So most of the time you should do "set -e; for XXX". Otherwise your Makefile loops will "succeed" incorrectly.
EDIT: I seemed to remember sharing this bug not too long ago, and indeed I did. pixelbeat responded with some interesting links: https://news.ycombinator.com/item?id=13940628
As another poster mentioned, set -e and set -o pipefail are crutches and tell me the script it sloppy.
-c Read commands from the command_string operand instead of from the standard input. Special parameter 0 will be set from the command_name operand and the positional parameters ($1, $2, etc.) set from the remaining argument operands.
Please use mktemp(1).
I know of no good reason to not use mktemp.
1) command substitution. you can catch this with an explicit error handler inherited by child processes. 2) pipefail SIGPIPE false positives. pretty hairy, command dependent whether this is a "real" error. often not, so you can work around it by ignoring SIGPIPE 3) process substitution. As far as I know, there is no way to workaround this whilst still using the convenience syntax. You have to carefully use explicit named pipes and carefully use wait on the PIDs (carefully!). Maybe still with races...
In my experience, you can write moderately robust shell scripts if you care enough and use all these flags and linters. But by the time you are at this stage you probably shouldn't be using shell scripting. More like training to spot problems in other people's code.
Everything bash can do, ansible can do it better.