Once I type a square bracket in a shell script, I know it's time to bring out Python. I don't think it's humanely possible to write a test command without shooting yourself in the foot in an exciting new way.
> don't think it's humanely possible to write a test command without shooting yourself in the foot in an exciting new way
That's a weird claim. Could it be that you have never really learned shell programming or read the man pages? Or can you give an example? We have lot of production code with many test statements and I don't remember when we were bitten last time by some "surprising feature" of test.
I agree that test works differently than comparisions in many programming languages. So you need to understand how it is different. And you need to be disciplined about quoting, otherwise surprises will happen when you have user-controlled input and the shell starts to do word-splitting. But if you use shellcheck you will be reminded of quoting everywhere. It's not a problem specific to test anyway.
I have written dozens of of bash utilities at a single company. Probably 5000 lines of bash in total. I still don't know the whole syntax for if/for/case and have to look it up every single time.
If I'm modifying an existing script, I generally have no problem doing that.
But the tweet quoted in the article rings very true to me:
>The opposite of "it's like riding a bike" is "it's like programming in bash".
>A phrase which means that no matter how many times you do something, you will have to re-learn it every single time.
I and probably a lot of programmers write bash scripts just enough to remember the vague shape of what we want to do, yet still infrequently enough to require looking up the exact characters every single time (-z? ! -z? one bracket? two brackets? quotes? no quotes? something about a semicolon and "then", right? where can you add newlines, again? oh yeah, it's not "end" like Ruby/Lua but "fi" right? what's "else if", again?).
I'm sure if we were working with bash scripts daily, we'd memorize it, but it's just rare and just different enough that it's basically day 1 all over again every time. Shellcheck helps as a safety net, but it doesn't necessarily save me a lot of time and cognitive load.
If I'm writing a new script from scratch and I realize there's going to be some branching or prompts or complex argument handling, I just write it in Python. It saves time and effort, it lets other people on my team more quickly modify and debug it, and it's more predictably portable ("oops, only sh, no bash"). And if it ever needs to do something more advanced in the future, it's way easier to extend.
Or put much more succinctly, by 'kristaps in another comment:
>Or I can use a sane language that let's me get my stuff done faster and with less opportunities to shoot myself in the foot.
It's nice for quickly writing something that's just wrapping multiple commands, though.
One of the big points of linters is that they change developer behavior for the better over time.
Once you get used to using shellcheck all the time your code style starts to match what it requires. You start thinking about quoting and other intricacies, and coding much more defensively.
This happens with other checkers as well - >90%+ of the time the python I write passes pep8 and black cleanly after having used those tools for a year or two.
A linter should make you a better coder, whether or not you code frequently, and also serves as a secondary "belt and suspender" check for code sanity and ease of future reading/refactoring.
> -z? ! -z? one bracket? two brackets? quotes? no quotes? something about a semicolon and "then", right? where can you add newlines, again? oh yeah, it's not "end" like Ruby/Lua but "fi" right? what's "else if", again?
really, most of these questions have only one correct answer. [ ! is never correct, neither is ! [ -z or ! [ -n. one bracket in /bin/sh, two in bash. quote every variable expansion unless you have a very good reason not to. add new lines in sensible places. nobody (sensible) complains that you can't write
for
num
in
mylist
:
in Python, so don't write it in shell either.
the problem is there is so much cargo culting garbage. the same problem exists in "C/C++", where people write "int* buf = (int* )malloc(num*sizeof(int))" to be "compatible", or do gratuitous pointer aliasing with undefined behavior, despite the fact that any modern C implementation with any optimizations at all will inline small fixed-size memcpy.
I tend to apply much the same rule as emptyparadise - once conditional logic appears, it's time for a rethink. Part of my rationale for this is that I assume that I'm dealing with the thin end of the wedge - if there's enough complexity in the task to warrant conditionals, there's likely a whole bunch more (or there will be in future) that I haven't encountered or grasped yet.
I'm thus willing to take the time-hit on rewriting in Python, because I'm betting that resulting code will be easier to adapt and easier to troubleshoot as this extra complexity gradually comes into view.
I just can't fathom this as I've never really had compatibility issues/quirks with good old #!/bin/bash. Only issues I've had have been with sed on Mac vs Linux. And I've written thousands of lines with lots of logic, generally breaking statements that become nested into separate functions. Sometimes some things become too complicated for Bash, like parsing JSON, in which case I prefer to break out and call a Python or Node script instead of wrapping my head around jq.
But my scripts are also either for personal use, or for internal use (5 person team) at work, so I know who will be running the scripts, why, and where (which OS/versions). Suppose it's a bit like developing for Apple where you can remain aware of the use (and edge) cases.
Awk is a viable replacement for Bash for scripting. A long established standard and installed on pretty much every POSIX system. It has more robust facilities for handling variables, arrays, regex search & substitution, etc.
But... neither is bash. I am sympathetic to the argument that awk is theoretically superior to bash in almost all ways (and it is of course nearly as available as bash). But... something still makes me reluctant to switch to using it in the cases I write a bash script (which I'm already trying to minimize, as is the OP). I'm not really sure why though.
(Fun-for-mostly-me fact, my very first programming job ever was writing awk. It was a while ago. I certainly don't remember any awk now).
Awk is great for snippets that are meant to be copy+pasted into a shell, and is the only reasonable option sometimes if you're stuck in a Busybox environment, but for almost everything else you'd be better off with Python.
In an embedded system (== very limited resources), wouldn't you want to avoid any interpreted language, even shell? We have some very nice languages that compile to machine code these days
I am pretty sure the PET-2001 didn't do that. 8 KB was all it had, about 1 KB was used by the operating system and 7 KB where available for programming. Actually something you would call byte code today, every basic command was 1 Byte. Just don't use labels with 4 or 5 characters, that wastes a lot of space!
The first models came even with 4 KB RAM / 3 KB usable. Have never seen one of those, though.
If we're talking Linux embedded and not something like an MCU, then you already likely have a shell installed. So you might as well use it for the things which shell excels at. Shell scripts are much smaller in size than the equivalent C code, for those things that shell is typically used for.
I was talking about embedded Linux here, so 512 MB RAM is not really uncommon, many systems have even more. No problem to run even bash. dash is sometimes preferred for license reasons, busybox ash is smaller, but then programming can probably get more annoying already, because useful features are probably missing. I use dash regularly for scripting (interactive use is a pain), haven't had to use ash for a while.
Of course there are much smaller embedded systems. But I don't think they would run Linux.
Rob Landley is working on implementing a full Bash clone (called toysh[1]) in his Toybox project[2]. I'm excited for it to be released - it will be a huge improvement vs. Busybox ash for embedded environments. We'll be able to write scripts that use arrays!
I agree with the general sentiment, though I set the bar a bit higher (or lower, depending on your perspective). Bash does okay with positional arguments, error handing, and even loops. You can get a lot done with that, and there are a lot of circumstances where a little bash script is just what you need. Where I draw the line is at arrays. If you're starting to think about data structures, bash isn't the right tool for the job.
Well, it depends. I had a co-admin who used to rewrite all my Bash scripts in Python. Most of them were a bunch of os.system() calls. He was a bit upset when I pointed out they are actually buggier than their Bash originals and noticeably slower.
Indeed. Shell scripting can easily get overcomplicated and I agree that moving to a real programming language is better at that point. However, Python isn't necessarily the ideal language for it either, because managing child processes with its standard library functionality can get overcomplicated and verbose, particularly once you start dealing with functionality like pipes. So people often resort to generic os.system calls instead and then you get a strange hybrid style.
Agreed. Not with the programming language per se, but I agree with the general premise.
However, in cases where portability is important, bash is would be my choice. If I can pass someone a bash script and not ask them to also install Python and whatever libs I used, it makes my life so much easier.
It's a very nice template, some great gems in there, like cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 .
My general issue, and it applies here too I guess, is that by the time I have 99 lines of Bash just as boilerplate,, +logic I might as well write it in Python. But if I just have to write Bash, I might use it one day.
tbh the first three lines are more than enough if you dont need arg parsing or special handling for interrupts. CI and the like where they are (repeated) one shots.
Notably the line you highlighted I do as SCRIPT_ROOT="$(cd $(dirname "${BASH_SOURCE[0]}") && pwd)"
This preserves the cwd of the invoker (for relative files etc) wholst giving you access to your asset files etc.
Yeah: I almost never want that... if you remarkably want to source files relative to the script location instead of the working directory, I will argue you usually want to do that with a variable for the directory, except in very even-more-special circumstances (in which case I have done something similar to this article).
Big use case - you want to run a helper script that is located relative to your original script + this is a package of scripts that you move between machines and/or share with co-workers
I do something similar with scripts that are within projects. There are scripts named the same but vary due to each project being different so I can't use $PATH (unless I export it every single time I change projects and open a terminal for it)
Commands are relative and sometimes these can be executed by other things. Other scripts, cron, etc. depending on the environment and the use. So, first step, cd into the correct directory and then run everything relative to it.
I mean, whatever works for you, obviously - there's no true way - but this sounds fragile - and perhaps more importantly - like working against the grain of shell/Unix.
I would generally say that state belongs in environment variables (and arguments) - and/or files/input streams. Not partially hard-coded in partial scripts.
It doesn’t make sense for generalized scripts that automate routine tasks. But for some maintenance, build, test, or deploy scripts that only ever perform a specialized job from a single location, it could be handy. I appreciate that Maciej provided it. Always easier to remove it than have to go find it yourself in the odd script you might need it.
It's minimal in the sense that it's a small template. It's not "minimal" in the sense that you can't do without every single line of the template. Just remove the parts you don't want. It would defeat its purpose if there were multiple templates that each aim at different things and then you have to decide which template you have to choose. Having two minimal templates is not minimal because of the redundancy.
I want this feature and use it sometimes in my own scripts, usually in scripts that I’m not passing relative path args to. It’s common enough for me that I like having the script_dir in there for a bash script template, and always trivially easy to comment out the cd command. Did the author do that already? I noticed the article had a sentence about changing directory struck out.
You're missing `errtrace` (-E) which means "any trap on ERR is inherited by shell functions, command substitutions, and commands executed in a subshell environment"
You probably haven't used it much if you haven't been using `trap`
I often set up a temporary directory at the beginning and then clean it up in an EXIT trap. Adding ERR and errtrace (-E) would remove the tmpdir or the first failure anywhere, no matter if it's a recoverable one.
So I'm sticking with just set -eu and trap EXIT...
> E If set, any trap on ERR is inherited by shell functions, command substitutions, and commands executed in a subshell environment. The ERR trap is normally not inherited in such cases.
set -eu is at least 30 years old. Not sure whether it makes sense to rewrite established code.
Yes! I read this somewhere a little while back and have been doing it exclusively since. Makes tons of sense.
Usually I have to look up what the long options are, since I've generally only memorized the short ones, but it helps immensely with readability, particularly for someone who hasn't memorized the short ones, who can now just semantically understand the options.
I look forward to the slow but steady reinvention of various coreutils in Rust or Go. Rg-ripgrep is better than grep. Fd-find is better than find. Exa is better than ls/tree. Fzf has no equivalent. I'm still looking for a ping that lets me timeout less than 1 second.
There are so many assumptions baked in to the early tools that a complete rewrite just bypasses.
Only because I didn't bother to strip the binary. The binary you're linking to is statically linked, which includes PCRE2 and Rust's regex engine, among other things. The GP's point is that if you look at all of grep's pieces, it comes in at a similar size.
Overall, I don't even know what the point of this kind of questioning even is.
That's a single benchmark. Before regex-redux was regex-dna, and Rust's regex engine was faster than PCRE2 on that benchmark. So how does that change your conclusion?
> Could somebody please educate me on why bashing it would be preferable?
Bash is a much better glue language than python. If all your script does is to call other programs that do all the work, it is very likely you can write it as a single line shell script. In python, you'll have to import libraries, call functions with many arguments, and whatnot.
If you have to parse output from one binary as input to another, I find it is way less error prone to use bash. There are subtleties to flushing output from processes in python that can lead to surprises.
The bash step is fast to write as it subsumes all the process calling / management.
JS (or python) is a reasonable next step, often just a tight cli wrapper around a function to be called in bash that is hard to write in bash / is from a library. I find its normally easier to pack JS dependencies for these one offs than python, but both work.
Next step is perf/testing focused. Js works as a great first step here as the Promises & event loop let you get a whole bunch of perf naturally that is a pita to arrange in python.
Now you have a program laid out and usage tested that can be re-written in something else for real or percieved benefits (go, bash, c++, rust)
In general "use the right tool for the job" but arguments for Bash:
* Succinctness: When you mostly glue together other programs, Bash will often be shorter than Python.
* Familiarity/Ubiquity: If you're using a Unix-like you're already using Bash (probably; or another shell that's similar) as your main "REPL to the system". When you wanna automate a thing you can start off with what you've already been doing for that task and go from there. Similar story for experimenting when writing a script, though Python also has a REPL for that.
If serious about creating shell script "apps", I think standard POSIX shell is the way to go. The Bashism is seldom needed, even though it is powerful.
Why? Because shell scripting is often used in DevOps, and containers very often do not come with Bash, but with (busybox) ash or dash instead.
Say you are going FROM nginx to FROM nginx:alpine (to get that smaller image size), woha! your bash script is not working anymore.
> Using `shellcheck --shell sh` is a good start :)
> If adhering to POSIX it should run the same in bash, ash and dash.
Not so. Shellcheck will tell you about syntactically invalid bashisms, but will not warn you about the zillions of things that are under or unspecified in posix. Pertinent minimal example:
Good luck finding a person who can use trap in a way that actually works as intended across different posix conform shells without consulting stack overflow first.
Sometimes of course, you just want to run something on SIGINT and not on normal exit, SIGHUP or whatever. But maybe the main use case for trap is implementing try/finally logic and you really want it to fire whenever the shell exits, not matter how (to clean up resources like temporary files). Bash's `trap EXIT` mostly does what you want ("run this when the shell exits"), but writing something that behaves this way across shells (and doesn't fire multiple times etc). is amazingly painful.
trap EXIT, does work for me in ash/dash/bash, though, when just letting the code flow exit with and without errors.
Not sure your use case, but I can for sure concur that traps are very painful to get right, especially when getting into child/grandchild processes land, taking terminal session, etc in regard.
It would be nice if bash were not a moving target. I have run into several bash incompatibilities in real life. Bash on macOS differs substantially from bash on Linux because it's much older. But bash is also generally somewhat poorly designed, leading to lots of aggravating and puzzling behavior for "corner cases" and this behavior not infrequently changes between versions.
So for example set -u for the longest time used to trigger on defined but empty arrays (which is hardly an obscure edge case), but what workarounds are possible differs quite a bit between versions:
I was listening to the Command Line Heroes podcast the other day, about Bash. First version was stored on tape. So it has some legacy, hehe. Already then the requirement was to be backwards compatible with sh, so yeah, corner cases are around for sure.
If writing bash I do aim at version 3, because as you say they got that on MacAttack.
I'll target bash (version checks are easy enough if necessary) instead of whatever happens to be /bin/sh. It's no problem for me to install bash wherever I need it, it provides some features I use, and I don't have to concern myself with POSIX implementation differences.
If you need some bash specific features, I'm not going to argue with that.
In my experience, having portability of the script is quite nice when not necessarily controlling the environment it is going to run in, and aiming for ash/dash/bash does take it a long way.
I wish that were true. But posix shell lacks two fairly vital features: sane trap and process redirection. In an ideal world the posix comittee would just fix that, but I won't hold my breath.
I totally agree Bash has some powerful features, some I go out of my way to reimplement in sh (as for example string operations [0]). But not all can ofc be replicated then bash it is.
I was going through the changes introduced in Bash 5.1, and to my surprise, it mentioned[1] that process substitution is now enabled in POSIX mode. I wondered if that means POSIX sh now supports it, but I haven't been able to find evidence for that, neither in the POSIX spec nor in the bug tracker of POSIX mailing list archives.
Similarly, I remember reading about POSIX sh supporting arrays at some point in the future, but for the life of me, I can't find any evidence for this either.
Arrays in shells generally suck (and behave quite different between different shells, e.g. zsh arrays are nothing like bash arrays), so I hope they don't make it into posix. OTOH, process substitution is a perfectly elegant and nice feature that fits well into the original design of bourne shell, removing an obvious limitation of pipes. It also behaves pretty much the same across shells that have it, so it should be much easier to standardize.
What license is the author releasing it under? I had a look on Maciej's blog and the gist, but couldn't see. I want to save it to possibly use in my projects.
POSIX has multiple Turing complete utilities that can be used in tandem. Its not a very fair competition. Many of the primitives involved are simply not primitives of higher level languages. Write a python script to remove blank lines from standard input. In awk it takes 5 non-whitespace characters (including 'awk'). Use awk to remove blank lines from your python code when you need to paste it into a terminal. Many python developers have a terrible time from the command line because they do not use the shell to help themselves out.
If your argument for using the shell (as opposed to a real programming language such as Python) is portability, writing a shell script that is not POSIX compliant literally defeats the point.
Also, 100 lines of UX-related bloat hardly classifies as "minimal".
> writing a shell script that is not POSIX compliant literally defeats the point.
Your argument is invalid. Even if the script isn't posix compliant you are comparingb portability with regards to the availability of bash vs python binaries on your target machine. Do you have data to support python is more likely to exist in linux machines than bash?
> Do you have data to support python is more likely to exist in linux machines than bash?
That's not what I'm saying. I never claimed Python was more widely supported that Bash, in terms of this argument you should use neither, that's the whole point.
What I'm saying is that it does not make sense to use Bash over e.g. Python in favor of supporting a wider range of machines while at the same time limiting support to Bash environments when you could use POSIX instead which is a subset of Bash and therefore more widely supported than Bash and Python.
Do you have data to prove that POSIX shell less likely to be supported than Bash and Python respectively? Otherwise I don't see how my argument is invalid.
Of course Bash support is larger than Python support (that's what you seem to be saying), and therefore even a Bash-specific script is more portable. What I'm suggesting is that it's contradictory to stop there when you could simply drop the bashisms and go for full POSIX compliance for even more portability.
That said, if I'm misunderstanding you, please point out how so.
There's also the fact that while Bash is common on Linux, it's not generally the default shell for the BSDs or other UNIXes. But most of them will have Python installed. Also it's easier to use Python on Windows than a POSIX shell, though neither is present by default.
I have a lot of experience writing Bash scripts and I regret It.
Now that I have decent enough mastery of python to be dangerous I realize that you will get to a point where you wish you had access to all the niceties of a decent programming language.
> Now that I have decent enough mastery of python to be dangerous I realize that you will get to a point where you wish you had access to all the niceties of a decent programming language.
What decent language is it, then? Or are you implying that it doesn't exist?
I think its almost hilarious that something calling itself a "safe script template" doesn't even spend a couple of lines dealing with the possibility of race conditions (e.g. a locking process so that if, as is almost inevitable, my script gets put in a cronjob, it doesn't run when already running).
The four lines the script wastes on setup_colors() - something that has nothing to do with "safety" - could easily be used for a basic locking process.
While I find the "almost hilarious" kind of overly snarky, I have my own version of this that has the race condition check. Basically uses `mkdir $THISBIN.lock` line that gives me an atomic way to create or check for existence of a logical semaphore, and provides me with a scratch area. The trap'd cleanup function removes it (which when debugging, and I use that scratch area, is inconvenient, but <shrug>)
exec >>$log_file 2>&1
if ! flock --exclusive --nonblock 1
then
log "unable to get exclusive lock, aborting"
exit 1
fi
Which redirects standard output to a log file, then tries to get an exclusive lock on it. The lock is created by the flock subprocess, but continues to exist until the script exits and closes that file descriptor.
This means there can only be one process writing to a given log file at a time. For me, that is the right granularity of locking - i have several cron jobs running this script, but with different arguments, and writing to different log files.
Well, I agree with your cron argument. On the other hand - sometimes I have some utility scripts, for example to trigger some processing, that I want to run multiple times at the same time, with different arguments. So lock is not a universal thing for me. (I'm the article author btw)
This is my feeling, too. Race conditions, misunderstanding globbing, whitespace in path names - these are the main sources of errors in Bash scripts I come by.
By default where? I think it comes with coreutils, so for instance it'd be pretty hard to get a Debian installation without it, but maybe you have a more exotic system?
There is a lot of "better write this in python"-ism here, and while I agree with the general sentiment that a more complex script shouldn't be written in bash, there are a ton of use cases where it is preferable to have it written in bash. So thank you for this post. It was very educational.
This website is an excellent example why I browse the web with JS off by default these days. When I turned it on, an obnoxious icon appeared, refusing to go away. When I hovered on it, another box appeared with an x in the corner. I moved the mouse to close it, and the box disappeared. The obnoxious icon, however, stays as before.
I'm really glad we are still able to turn JavaScript off these days. I'm pretty sure in the next years more and more so-called "developers" switch to text rendering by JS, rendering more websites unusable.
Of course, but it hasn't become the norm yet. I'm afraid this day will eventually arrive as most user-hostile techniques depends on JS being on, so they need to find a way to force it on users.
This script is too much boilerplate and opiniated for me. I maintain a snippet with the most important settings for safe and robust shell script execution: https://gitlab.com/-/snippets/1968838 I'm now thinking of adding the trap cleanup boilerplate to my snippet as well, as it tends to get overlooked too often.
Honestly I think I prefer the bash version to `DIR=${0:h}`. Although Powershell is superior with simply "$PSScriptRoot" (On the other hand, Powershell's strict mode is a mess and "catch errors" in Powershell is still an extra setting that doesn't actually catch errors half the time.)
I much prefer this approach (finding the script directory and explicitly using it) over changing into the directory of the script. It makes it much easier to use paths relative to the working directory when you want to.
That may cause errors or undesirable behaviour. As way of example, dealing with arrays: `read -a` in bash is `read -A` in zsh, and the former’s arrays are zero-based while the latter’s are one-based.
My gripe is about being able to work on a bash script that needs to work "everywhere". Bash 4 was release over a decade ago, so it's a reasonable expectation that it will be present on most servers with a shell.
Homebrew works around that, but their refusal over GPL taint for having the binary present still doesn't seem right.
Same, I was scratching my head, wondering why the author of the article doesn't just use readlink... and then a few hours later, a coworker of mine asked why my bash script isn't working on her Mac! Turns out readlink on OS X doesn't have the same flags. I ended up substituting it out for the hackier but more portable method that the article uses.
This is the real minimal template. But it should be pwd -P at the end there or you introduce path resolving bugs. Or you could use cd -P, or cd -e -P to be extra fail-safe (bashism).
IMO this solution solves the wrong problem. The “right” problem to solve is the coupling between the script and the directory layout. If the script needs to operate on data in some directory, the path to that directory should be part of the script’s interface and handled accordingly.
What about if the script wants to source sibling files in the samedirectory? I.e. it is a bash library of sorts with modules spread over files. This isn’t “operating on data” abe it would be weird for a library to have to receive its own location as an argument in order to have to load its own modules.
I don't think there's much weird about providing a script a path to its dependencies. If it's the case that the path to the dependencies location is the same as the path to the script's location, that's fine. We're asking distinct questions (where is my data, where are my deps, where is my script) here, the unnecessary coupling is precisely the assumption they have the same answer. If they happen to have the same answer, that's not weird, that sort of thing happens all the time.
In many cases one might not want a full stand-alone program with arguments, usage and such. Rather a "script" i.e. something that can be simply executed to run a series of tasks. For example it's quite silly for a build script to require the path of the project it resides in.
It's really not silly. A build script could easily move around inside a project. It could go from the project root, to /root/bin, to /root/bin/build, etc etc. If the inconvenience of arguments troubles you, write a wrapper or alias it or something like that.
Yes, it is silly. You've just added extra complexity to a project for little to no benefit. If you decide to move the script you take the same steps as if you moved anything else i.e update the references.
It's unnecessary complexity for the user. As opposed to simply having the build instructions as `Run bin/build` you're suggesting "Run bin/build <project root>". But there's zero reason for requiring the user to supply the project root when we already know it.
This is exactly the kind of implicit coupling that makes software unmaintainable in the long run. You _are not passing the project root_, you are passing the value of a variable which currently should be set to the project root. The constraints on this value are not that it must be the project root, they are other constraints that are satisfied by the project root right now.
You said earlier that "If the inconvenience of arguments troubles you, write a wrapper". That's precisely what a script like this is. A wrapper. I can run one simple, static command, similar to just about any build tool e.g. `cd project && make`.
Really scripts should avoid called `cd` or `pushd`/`popd` wherever possible. It's a lot better to just work with absolute directories, i.e. as someone else pointed out:
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
No, not "just like." Whatever you think of JS the tweet above this quote doesn't apply-- I don't constantly have to look up how a function callback works, or how to do math.
To be "just like" shell scripting, you'd have to get JS devs to start regularly using `eval` in lots of unnecessary places, plus using arbitrary, inconsistent use of single-letter aliases for some of the core methods
I mean...
> set -Eeuo pipefail
Upper and lowercase! And the flag meanings aren't even explained in this article!
Edit: To be honest, this reflects my own mental model for shell commands: commandName -garblegooklegarblegookle myArg1 myArg2 etc..
$ help set
...
-e Exit immediately if a command exits with a non-zero status.
-E If set, the ERR trap is inherited by shell functions.
-u Treat unset variables as an error when substituting.
-o option-name
Set the variable corresponding to option-name:
pipefail the return value of a pipeline is the status of
the last command to exit with a non-zero status,
or zero if no command exited with a non-zero status
Hey, the flag meanings are explained in the article I linked in the post:
> For details on what options exactly set -Eeuo pipefail changes and how they will protect you, I refer you to the article I have in my bookmarks for a few years now.
> The opposite of "it's like riding a bike" is "it's like programming in bash".
> A phrase which means that no matter how many times you do something, you will have to re-learn it every single time.
Couldn't phrase it better myself. After wasting way too much time, I tend to go with Python for anything more than a couple of lines, even trivial stuff. That's, of course, unless there's a very compelling reason to use bash.
Every time a python script throws an exception at me (pretty much every time I invoke a python script for the first time), my first debugging step is to read it and attempt to rewrite it in bash.
9 times out of 10, the result is 10-100x times shorter, with zero external dependencies, and is also easier to debug.
In the other case, the python thing is some non-trivial monstrosity that should have been implemented in a compiled language with static type checking.
Python is readable, awk is not. I'd rather spend an extra 5 minutes writing something that I don't have to burn 20 minutes understanding in just a few months time.
I think the popularity of Python for scripting is pretty good evidence that PG was dead wrong on the importance of brevity in programming languages. Further evidence can be found in Python's development of type hints.
Another counterpoint to readability: the benefit is diluted if the reader isn’t familiar with the dependencies. I know this goes both ways, though in some circles (say, devops) there’s more familiarity with Unix tools than Python. So the cognitive overhead of getting familiar with something you’re not IMO outweighs readability in either case.
I believe it is a matter of taste and experience. Some people prefer to read & edit 3-4 concise lines, and some prefer ten pages of prose.
> I think the popularity of Python for scripting is pretty good evidence that PG was dead wrong on the importance of brevity in programming languages.
IMO Python has a fair number of brevity constructs, which was it's one of the selling points, besides keeping other things readable. If you look at the published code from the research community, many of it resembles Matlab or Mathematica's style.
This is true, but if you already know Python, then "awk + Python" has greater total complexity than "just Python". So the question is does awk add enough value to be worth the incremental cost of learning it in addition to Python? I think for many, the answer is "no".
> Give me $1 for every time I see a whole python script with loads of dependencies for the simplest sed/awk/jq task with pipes.
I'll add good old Make to the list. It boggles the mind how some people prefer to reinvent the wheel instead of pulling a standard tool from a standard tool belt.
You can do sed/awk/jq trivially in Python with no dependencies, other than Python itself. Which is a fewer number of dependencies than a script gluing together sed+awk+jq, each of those being a dependency managed implicitly by your distribution's package manager.
except with sed, awk and jq, I don't have to worry about it being present on a production system and someone leveraging it to execute arbitrary, sophisticated code.
You can't just assume that it's okay to have your preferred scripting language everywhere.
I don't understand what you're arguing here. You don't run sed, awk, and jq directly, you run them via Bash, your preferred scripting language that can execute arbitrary, sophisticated code.
Personally I don't use Python much for these use cases. For anything beyond trivial scripts I write Go programs, compiled into statically-linked executables that I can scp onto a host and run, the only dependency required at that point being libc. Or ideally the remote host is running inside a container, which would allow me to reproduce the dependencies locally (but also has a bunch of other complexity).
But saying "just use Bash" and then patting ourselves on the back is not really solving the problem, it's just kicking the can down the road, possibly not even very far.
Bash can't be made to bind a socket that listens and receives instructions from the internet without also having something like socat or netcat or inetd and there's no privilege-free packaging system to rapidly get there.
Python and other powerful scripting languages can.
Having production servers that don't have Python/Ruby/Perl/etc unless they absolutely must is decades-old advice at this point.
The smart ones take that one step further: no production server should have any binaries that aren't absolutely necessary. This is why we use containers.
> Bash can't be made to bind a socket that listens and receives instructions from the internet without also having something like socat or netcat or inetd
Not listen, but it can certainly connect out and ask for instructions...
Your point seemed like confirmation bias to me. I've been given to use just as many broken bash script as Python. (I'll admit I'm saying that in a false attempt at being fair. In my experience, Python script tend to work better. Then again, I tend to work in mixed env which includes Windows OS.)
You're mentioning exception errors as if having an explicit error message on failure is a problem, not a plus.
What happens when a bash script fails? Or worse, silently fails and leave you with a half-working setup at best and a seriously broken one due to some intermediary step failure being ignored?
If you happen to notice the error that is... like the OP pointed out, almost all bash shell script ignore silently errors and figuring which of the hundred of lines fails on your particular setup is a fun exercise.
And then... which bash debugger to you use to fix problem in bash shell script you been handed? Ah no, it's long reading and echo and trial and errors.
> If you happen to notice the error that is... like the OP pointed out, almost all bash shell script ignore silently errors and figuring which of the hundred of lines fails on your particular setup is a fun exercise.
> And then... which bash debugger to you use to fix problem in bash shell script you been handed? Ah no, it's long reading and echo and trial and errors.
Or you could just add "-x" to the infamous "set -Eeuo pipefail" at the top of the script before running it, in which case it'll let you know exactly which of the hundreds of lines failed.
Alternatively, if you're using the "script template" in the submission, you can just pass in the "-v" option when you run the script and it'll take care of setting it for you.
I am very skeptical of this claim...in practice I've found it to rarely be the case. Usually shell scripts have more dependencies, and even worse, they are implicit since Bash provides no mechanism for managing dependencies. Programs like sed, awk, grep, cut, paste, curl, jq, gzip, tar, etc. These are all separate dependencies, usually in separate packages (some are grouped together into coreutils), supporting different feature sets between distributions and OSes. All of those can be replaced with the Python standard library.
Bash itself is pretty anemic and isn't very useful without gluing together external programs, which is its raison d'etre after all.
Are there any systems that run Bash but do not include CoreUtils? I think not...
A shell script is literally intended to "glue together" external programs, including all of CoreUtils. If you are doing more than just "gluing together" other programs, then you are programming, not scripting, and you'd be far better off reaching for an actual programming language instead of a shell script of any kind.
Python has plenty of warts too. You can't depend on it being installed on any given system, can't depend on the version even (Python 2.x or 3.x?), which libraries are installed, environment, venv support, etc.
What version of coreutils? Try comparing the behavior of cp on macOS vs. a modern Linux distribution. They are very different.
I'm not saying Python isn't without its problems too. I don't even like using Python that much for scripting. But we should be clear about what tradeoffs we're making when we write Bash scripts, and not pretending like we have "zero dependencies".
> Try comparing the behavior of cp on macOS vs. a modern Linux distribution
Not many people are running macOS servers... and servers tend to be where most shell scripts are being used.
macOS has notoriously out of date versions of all system programs, CoreUtils and Bash among them, due to licensing issues with Apple's closed-source proprietary OS. BSD and Linux based systems, generally don't have this issue.
That being said, a regular Shell Script (not a Bash Script) will run unmodified on macOS. When people say "shell script", they usually mean an actual `sh` script instead of a Bash script.
At my current gig, our infra team - who all use linux - regularly break the developer workflow for the rest of the org, which is around 80% macOS - by using unsupported bash or coreutils features. When I look at the scripts I want to pull my hair out, their use of bash goes far beyond glue code and rivals the complexity of some of our smaller micro services.
Into the weeds - but sounds like your developers need a better development environment, one that more closely matches the production environment.
Thinking of macOS as "just another *NIX" really doesn't fly these days. Crazily outdated standard systems utilities is just the tip of the ice berg, really.
On the contrary - we use kubernetes heavily and our development environment matches production about as closely as possible for the architecture we have. But we’ve got to bootstrap minukube and seed configuration and data to uninteresting dependencies somehow. Using different scripts for that in development and production would be moving in the wrong direction, but I do often wish they were written in Python.
Has your team considered using a standardized VM to develop in?
Your team could then freely use whatever computer/OS they wanted, but also keep everyone's development systems consistent with each other, as well as avoid any strange Apple-induced issues that come with running Apple hardware as a developer in 2020.
You're really fighting an uphill battle with macOS these days... as sad as that might be for some.
You've spelled it out in a few different posts - macOS breaks everything for your team. So why fight it? Get a different OS to develop in... or dumb down everything because macOS uses crusty, old utilities.
> Has your team considered using a standardized VM to develop in?
How is that not what we're already doing? Minikube runs in a VM. But we have to set it up somehow. Running a minikube VM inside a maually configured linux VM makes no sense - if we did that, now we'd need a scipt to boottrap the linux VM.
> You're really fighting an uphill battle with macOS these days... as sad as that might be for some.
I can definitely see this becoming more and more true in many fields, but this hans't been an issue for us.
> How is that not what we're already doing? Minikube runs in a VM. But we have to set it up somehow.
I was implying a development environment inside a VM, not just a VM for testing.
You posted several times about how macOS isn't compatible with your production environments scripts and what-not. The solution is to either not use macOS, or use a standardized VM every developer on the team uses to develop inside of, so that your scripts work "out of the box".
> I can definitely see this becoming more and more true in many fields, but this hans't been an issue for us.
Your posts seem to contradict this. Perhaps you were exaggerating the pain it causes the team when production scripts don't work on developer systems. Either way - it seems the writing is already on the wall. Apple isn't going to make it easier any time soon - so ditch the mac's.
> You posted several times about how macOS isn't compatible with your production environments scripts and what-not. The solution is to either not use macOS, or use a standardized VM every developer on the team uses to develop inside of, so that your scripts work "out of the box".
Another solution would be to write those scripts against a language and standard library that doesn't have wildly varying behaviour between its underlying platforms. One of my side projects has a bunch of supporting scripts in Python doing similar things and they run exactly the same on Linux and macOS. They even run fine on Windows with only minimal modifications.
> Your posts seem to contradict this. Perhaps you were exaggerating the pain it causes the team when production scripts don't work on developer systems. Either way - it seems the writing is already on the wall. Apple isn't going to make it easier any time soon - so ditch the mac's.
With this statement I was referencing specifically the complaints that have surfaced in the past few years about new OS changes and security features breaking tooling. I don't count our infra team using new bash features in this because the issue with outdated bash/unix tooling is a long-standing one.
I can't speak for others, but for me personally, the usability of macOS for almost all of the things I do on a day-to-day basis is just day-and-night better than anything else, even if I have to hack up a bash script to work correctly from time to time. I will say the difference is less than it used to be - I use it occasionally and Linux has come a long way in the past 10 years.
This is basically what writing a Python script is. Python code is compiled into Python bytecode and interpreted by the Python VM, after all.
(I know what you mean by VM, but I don't think the distinction is very important here, you get the same benefits either way, and using Python is arguably a lot simpler than using a full VM.)
What you describe is 100% real and is why `brew install coreutils findutils gnu-sed` has always been a prerequisite for any developer whose dev environment I've been tasked with supporting.
That’s definitely one solution and I have resorted to it from time to time, but it’s far from perfect - now anything that detects macOS and uses different options is broken.
So what you can do, though, is not link them in and your scripts can use their Homebrew directory to get ahold of the GNU binaries. It is still a script change you have to make, but it's one you can make at a top level and then your bash wranglers are very unlikely to break them.
(Though I use those as daily-driver packages on Macs and have actually never had something break? GNU tools tend to support BSD options.)
Maybe this is easier with the way homebrew does things (I prefer MacPorts) but then you have to manage a bunch of symlinks to bring in the stuff you DO want to use by default.
I actually hadn't thought to try this approach, though. The next time this happens I will definitely give it a shot.
Python is by default installed on RHEL, CentOS, Debian and many more, the noteble exception are BSDs [0]. Python 2.4 was released in 2004, by writing python 2.4 compatible code you should cover everything besides ancient mainframes. While python 2.4. does not contain all niceties of modern python, it has lists, sets, dicts and the subprocess module which makes writing bash script comparable scripts very doable.
I think bash is marginally more portable, yet python has some wonderful things in the standard library that make it a way more generalized language that will go further. os.path, argparse and os.walk I use a lot.
I am on dev team that isn't fast to upskill, and have been rewarded for writing my build-test-release solutions in bash because my team is not generally able to self-manage their developer python environments. Without needing to install anything, it is very lucrative for us to use such an unpopular language.
That's my way to do things now. I still have my 500 lines bash script file to backup multiple databases, with dynamic filter, on multiple remote servers... to remind me that I can do that much easier in Python.
284 comments
[ 4.4 ms ] story [ 252 ms ] threadThat's a weird claim. Could it be that you have never really learned shell programming or read the man pages? Or can you give an example? We have lot of production code with many test statements and I don't remember when we were bitten last time by some "surprising feature" of test.
I agree that test works differently than comparisions in many programming languages. So you need to understand how it is different. And you need to be disciplined about quoting, otherwise surprises will happen when you have user-controlled input and the shell starts to do word-splitting. But if you use shellcheck you will be reminded of quoting everywhere. It's not a problem specific to test anyway.
Or I can use a sane language that let's me get my stuff done faster and with less opportunities to shoot myself in the foot.
But the tweet quoted in the article rings very true to me:
>The opposite of "it's like riding a bike" is "it's like programming in bash".
>A phrase which means that no matter how many times you do something, you will have to re-learn it every single time.
I and probably a lot of programmers write bash scripts just enough to remember the vague shape of what we want to do, yet still infrequently enough to require looking up the exact characters every single time (-z? ! -z? one bracket? two brackets? quotes? no quotes? something about a semicolon and "then", right? where can you add newlines, again? oh yeah, it's not "end" like Ruby/Lua but "fi" right? what's "else if", again?).
I'm sure if we were working with bash scripts daily, we'd memorize it, but it's just rare and just different enough that it's basically day 1 all over again every time. Shellcheck helps as a safety net, but it doesn't necessarily save me a lot of time and cognitive load.
If I'm writing a new script from scratch and I realize there's going to be some branching or prompts or complex argument handling, I just write it in Python. It saves time and effort, it lets other people on my team more quickly modify and debug it, and it's more predictably portable ("oops, only sh, no bash"). And if it ever needs to do something more advanced in the future, it's way easier to extend.
Or put much more succinctly, by 'kristaps in another comment:
>Or I can use a sane language that let's me get my stuff done faster and with less opportunities to shoot myself in the foot.
It's nice for quickly writing something that's just wrapping multiple commands, though.
Once you get used to using shellcheck all the time your code style starts to match what it requires. You start thinking about quoting and other intricacies, and coding much more defensively.
This happens with other checkers as well - >90%+ of the time the python I write passes pep8 and black cleanly after having used those tools for a year or two.
The issue is the frequency. Due to not coding in it often, you don't retain it.
Probably around once a week.
A linter should make you a better coder, whether or not you code frequently, and also serves as a secondary "belt and suspender" check for code sanity and ease of future reading/refactoring.
really, most of these questions have only one correct answer. [ ! is never correct, neither is ! [ -z or ! [ -n. one bracket in /bin/sh, two in bash. quote every variable expansion unless you have a very good reason not to. add new lines in sensible places. nobody (sensible) complains that you can't write
in Python, so don't write it in shell either.the problem is there is so much cargo culting garbage. the same problem exists in "C/C++", where people write "int* buf = (int* )malloc(num*sizeof(int))" to be "compatible", or do gratuitous pointer aliasing with undefined behavior, despite the fact that any modern C implementation with any optimizations at all will inline small fixed-size memcpy.
I'm thus willing to take the time-hit on rewriting in Python, because I'm betting that resulting code will be easier to adapt and easier to troubleshoot as this extra complexity gradually comes into view.
But my scripts are also either for personal use, or for internal use (5 person team) at work, so I know who will be running the scripts, why, and where (which OS/versions). Suppose it's a bit like developing for Apple where you can remain aware of the use (and edge) cases.
If you expect that someone else can read and maintain your code with minimal learning I don't think it's the tool of choice.
(Fun-for-mostly-me fact, my very first programming job ever was writing awk. It was a while ago. I certainly don't remember any awk now).
This previous comment of mine goes into a bit more detail: https://news.ycombinator.com/item?id=23242753
The first models came even with 4 KB RAM / 3 KB usable. Have never seen one of those, though.
I was talking about embedded Linux here, so 512 MB RAM is not really uncommon, many systems have even more. No problem to run even bash. dash is sometimes preferred for license reasons, busybox ash is smaller, but then programming can probably get more annoying already, because useful features are probably missing. I use dash regularly for scripting (interactive use is a pain), haven't had to use ash for a while.
Of course there are much smaller embedded systems. But I don't think they would run Linux.
[1] https://github.com/landley/toybox/blob/master/toys/pending/s...
[2] http://landley.net/toybox/about.html
Also pipe/subprocess heavy scripts often become sad complex monstrosities in non-shell languages.
However, in cases where portability is important, bash is would be my choice. If I can pass someone a bash script and not ask them to also install Python and whatever libs I used, it makes my life so much easier.
Or we are both the same kind of horrifying :D
My general issue, and it applies here too I guess, is that by the time I have 99 lines of Bash just as boilerplate,, +logic I might as well write it in Python. But if I just have to write Bash, I might use it one day.
Notably the line you highlighted I do as SCRIPT_ROOT="$(cd $(dirname "${BASH_SOURCE[0]}") && pwd)"
This preserves the cwd of the invoker (for relative files etc) wholst giving you access to your asset files etc.
Eh? This breaks all relative path arguments:
If you need to reference relative path data internally then that should be hidden from the caller by that same logic, but used only internally.I do something similar with scripts that are within projects. There are scripts named the same but vary due to each project being different so I can't use $PATH (unless I export it every single time I change projects and open a terminal for it)
Commands are relative and sometimes these can be executed by other things. Other scripts, cron, etc. depending on the environment and the use. So, first step, cd into the correct directory and then run everything relative to it.
I would generally say that state belongs in environment variables (and arguments) - and/or files/input streams. Not partially hard-coded in partial scripts.
I suppose I could see a "framework" like:
But I'd still prefer setup and friends to accept a path (not PATH) as first argument or whatever. Rather than assuming a relative path.Maybe even factor most of this into something that could live under /opt/bin or whatever.
Especially for cron I'd rather see:
/opt/projX/scripts/mangle /opt/projX
Rather than just the first part with an implicit argument of "parent of containing folder".
It doesn’t make sense for generalized scripts that automate routine tasks. But for some maintenance, build, test, or deploy scripts that only ever perform a specialized job from a single location, it could be handy. I appreciate that Maciej provided it. Always easier to remove it than have to go find it yourself in the odd script you might need it.
Apparently this is for some category of utility that is a "bash script", not "a unix style command implemented in bash".
Make small utilities, call them for each other, or chain them with pipes. 99% of the time that will make more sense.
You probably haven't used it much if you haven't been using `trap`
So I'm sticking with just set -eu and trap EXIT...
> E If set, any trap on ERR is inherited by shell functions, command substitutions, and commands executed in a subshell environment. The ERR trap is normally not inherited in such cases.
set -eu is at least 30 years old. Not sure whether it makes sense to rewrite established code.
Usually I have to look up what the long options are, since I've generally only memorized the short ones, but it helps immensely with readability, particularly for someone who hasn't memorized the short ones, who can now just semantically understand the options.
Could somebody please educate me on why bashing it would be preferable?
Bash is a prototype to the app in Python that is a prototype to the app in Golang.
There are so many assumptions baked in to the early tools that a complete rewrite just bypasses.
Overall, I don't even know what the point of this kind of questioning even is.
The regex crate has many more benchmarks: https://github.com/rust-lang/regex/blob/master/bench/log/07/... and https://github.com/rust-lang/regex/blob/master/bench/log/07/... for example.
And then there are things like this which can impact the performance of real world use cases: https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pcr...
Please don't draw sweeping conclusions about the comparative performance of regex engines from a single benchmark.
Bash is a much better glue language than python. If all your script does is to call other programs that do all the work, it is very likely you can write it as a single line shell script. In python, you'll have to import libraries, call functions with many arguments, and whatnot.
The bash step is fast to write as it subsumes all the process calling / management.
JS (or python) is a reasonable next step, often just a tight cli wrapper around a function to be called in bash that is hard to write in bash / is from a library. I find its normally easier to pack JS dependencies for these one offs than python, but both work.
Next step is perf/testing focused. Js works as a great first step here as the Promises & event loop let you get a whole bunch of perf naturally that is a pita to arrange in python.
Now you have a program laid out and usage tested that can be re-written in something else for real or percieved benefits (go, bash, c++, rust)
* Succinctness: When you mostly glue together other programs, Bash will often be shorter than Python.
* Familiarity/Ubiquity: If you're using a Unix-like you're already using Bash (probably; or another shell that's similar) as your main "REPL to the system". When you wanna automate a thing you can start off with what you've already been doing for that task and go from there. Similar story for experimenting when writing a script, though Python also has a REPL for that.
At least a comment. If you use CI that should run shellcheck to check all scripts.
Why? Because shell scripting is often used in DevOps, and containers very often do not come with Bash, but with (busybox) ash or dash instead.
Say you are going FROM nginx to FROM nginx:alpine (to get that smaller image size), woha! your bash script is not working anymore.
If adhering to POSIX it should run the same in bash, ash and dash.
Bash is kind of a moving target in it self, since it comes with many new features.
You have to be aware of which version you are targeting. For instance when using arrays, associative arrays, there is a diff between version 3 and 4.
Version 5 is even more competent and it's easy start digging to deep into the cookie jar and getting your hand stuck in there :)
Not so. Shellcheck will tell you about syntactically invalid bashisms, but will not warn you about the zillions of things that are under or unspecified in posix. Pertinent minimal example:
Good luck finding a person who can use trap in a way that actually works as intended across different posix conform shells without consulting stack overflow first.Comes with a lot trial and error.
The above will work as (tested in ash/dash/bash):
But to your point, yes, how to actually know that without doing trial n' error & stackoverflow.Thanks for the `sleep 1h`, didn't know that you could do hour as unit :)
(edited: turned out INT is enough to trap)
Sometimes of course, you just want to run something on SIGINT and not on normal exit, SIGHUP or whatever. But maybe the main use case for trap is implementing try/finally logic and you really want it to fire whenever the shell exits, not matter how (to clean up resources like temporary files). Bash's `trap EXIT` mostly does what you want ("run this when the shell exits"), but writing something that behaves this way across shells (and doesn't fire multiple times etc). is amazingly painful.
trap EXIT, does work for me in ash/dash/bash, though, when just letting the code flow exit with and without errors.
Not sure your use case, but I can for sure concur that traps are very painful to get right, especially when getting into child/grandchild processes land, taking terminal session, etc in regard.
So for example set -u for the longest time used to trigger on defined but empty arrays (which is hardly an obscure edge case), but what workarounds are possible differs quite a bit between versions:
https://gist.github.com/dimo414/2fb052d230654cc0c25e9e41a965...
I was listening to the Command Line Heroes podcast the other day, about Bash. First version was stored on tape. So it has some legacy, hehe. Already then the requirement was to be backwards compatible with sh, so yeah, corner cases are around for sure.
If writing bash I do aim at version 3, because as you say they got that on MacAttack.
https://stackoverflow.com/questions/11376975/is-there-a-mini...
In my experience, having portability of the script is quite nice when not necessarily controlling the environment it is going to run in, and aiming for ash/dash/bash does take it a long way.
I wish that were true. But posix shell lacks two fairly vital features: sane trap and process redirection. In an ideal world the posix comittee would just fix that, but I won't hold my breath.
I totally agree Bash has some powerful features, some I go out of my way to reimplement in sh (as for example string operations [0]). But not all can ofc be replicated then bash it is.
Maybe then: sh->bash->python/perl
[0] https://gitlab.com/space-sh/string/-/blob/master/Spacefile.s...
Similarly, I remember reading about POSIX sh supporting arrays at some point in the future, but for the life of me, I can't find any evidence for this either.
[1]: https://git.savannah.gnu.org/cgit/bash.git/tree/CHANGES#n538
Also, 100 lines of UX-related bloat hardly classifies as "minimal".
Your argument is invalid. Even if the script isn't posix compliant you are comparingb portability with regards to the availability of bash vs python binaries on your target machine. Do you have data to support python is more likely to exist in linux machines than bash?
That's not what I'm saying. I never claimed Python was more widely supported that Bash, in terms of this argument you should use neither, that's the whole point.
What I'm saying is that it does not make sense to use Bash over e.g. Python in favor of supporting a wider range of machines while at the same time limiting support to Bash environments when you could use POSIX instead which is a subset of Bash and therefore more widely supported than Bash and Python.
Do you have data to prove that POSIX shell less likely to be supported than Bash and Python respectively? Otherwise I don't see how my argument is invalid.
Of course Bash support is larger than Python support (that's what you seem to be saying), and therefore even a Bash-specific script is more portable. What I'm suggesting is that it's contradictory to stop there when you could simply drop the bashisms and go for full POSIX compliance for even more portability.
That said, if I'm misunderstanding you, please point out how so.
Now that I have decent enough mastery of python to be dangerous I realize that you will get to a point where you wish you had access to all the niceties of a decent programming language.
Any shell script over 50 lines is suspect.
What decent language is it, then? Or are you implying that it doesn't exist?
The four lines the script wastes on setup_colors() - something that has nothing to do with "safety" - could easily be used for a basic locking process.
This means there can only be one process writing to a given log file at a time. For me, that is the right granularity of locking - i have several cron jobs running this script, but with different arguments, and writing to different log files.
And more generally, the arcane rules for quoting, substitutions, parameter parsing, running other processes, and how all of these interact.
I'm really glad we are still able to turn JavaScript off these days. I'm pretty sure in the next years more and more so-called "developers" switch to text rendering by JS, rendering more websites unusable.
This has been done for years on lots of sites.
[1] https://weblogs.asp.net/soever/returning-an-exit-code-from-a...
[2] https://octopus.com/blog/powershell-exit-codes
[3] https://community.idera.com/database-tools/powershell/powert...
Catalina shipped with zsh 5.7 whereas 5.8 is now the latest.
Homebrew works around that, but their refusal over GPL taint for having the binary present still doesn't seem right.
I also tack on [ "${DEBUG:-0}" = "1" ] && set -x
not cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1
with a matching popd at the end
Otherwise it can screw up parent scripts
Put it on GitHub and curl it from there with checksum. Cache the file somewhere and lookup cache first.
No, not "just like." Whatever you think of JS the tweet above this quote doesn't apply-- I don't constantly have to look up how a function callback works, or how to do math.
To be "just like" shell scripting, you'd have to get JS devs to start regularly using `eval` in lots of unnecessary places, plus using arbitrary, inconsistent use of single-letter aliases for some of the core methods
I mean...
> set -Eeuo pipefail
Upper and lowercase! And the flag meanings aren't even explained in this article!
Edit: To be honest, this reflects my own mental model for shell commands: commandName -garblegooklegarblegookle myArg1 myArg2 etc..
Link: https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_p...
About "just like JavaScript" - it was sarcasm ;-) I write in TypeScript daily and still make fun of JS. But I agree, Bash is worse!
I updated the script template applying the most commonly mentioned things, like removing "cd" to the directory. See details in the article.
"Just remember that the cleanup() can be called not only at the end but as well having the script done any part of the work. "
> A phrase which means that no matter how many times you do something, you will have to re-learn it every single time.
Couldn't phrase it better myself. After wasting way too much time, I tend to go with Python for anything more than a couple of lines, even trivial stuff. That's, of course, unless there's a very compelling reason to use bash.
Every time a python script throws an exception at me (pretty much every time I invoke a python script for the first time), my first debugging step is to read it and attempt to rewrite it in bash.
9 times out of 10, the result is 10-100x times shorter, with zero external dependencies, and is also easier to debug.
In the other case, the python thing is some non-trivial monstrosity that should have been implemented in a compiled language with static type checking.
I think the popularity of Python for scripting is pretty good evidence that PG was dead wrong on the importance of brevity in programming languages. Further evidence can be found in Python's development of type hints.
You can write a simplifying function, but programs rarely do.
This is why I use the shell overall, but when I need to process something that's better expressed in python, I use pypyp.
> I think the popularity of Python for scripting is pretty good evidence that PG was dead wrong on the importance of brevity in programming languages.
IMO Python has a fair number of brevity constructs, which was it's one of the selling points, besides keeping other things readable. If you look at the published code from the research community, many of it resembles Matlab or Mathematica's style.
I actually find awk very readable. It's a far simpler language than Python really.
The exception is perhaps if you need to do something complex, like parse a csv.
It's like all of the bash scripts that we see that don't handle failures and traps or print a usage block.
This is true, but if you already know Python, then "awk + Python" has greater total complexity than "just Python". So the question is does awk add enough value to be worth the incremental cost of learning it in addition to Python? I think for many, the answer is "no".
I'll add good old Make to the list. It boggles the mind how some people prefer to reinvent the wheel instead of pulling a standard tool from a standard tool belt.
You can't just assume that it's okay to have your preferred scripting language everywhere.
Personally I don't use Python much for these use cases. For anything beyond trivial scripts I write Go programs, compiled into statically-linked executables that I can scp onto a host and run, the only dependency required at that point being libc. Or ideally the remote host is running inside a container, which would allow me to reproduce the dependencies locally (but also has a bunch of other complexity).
But saying "just use Bash" and then patting ourselves on the back is not really solving the problem, it's just kicking the can down the road, possibly not even very far.
Python and other powerful scripting languages can.
Having production servers that don't have Python/Ruby/Perl/etc unless they absolutely must is decades-old advice at this point.
The smart ones take that one step further: no production server should have any binaries that aren't absolutely necessary. This is why we use containers.
Not listen, but it can certainly connect out and ask for instructions...
You're mentioning exception errors as if having an explicit error message on failure is a problem, not a plus.
What happens when a bash script fails? Or worse, silently fails and leave you with a half-working setup at best and a seriously broken one due to some intermediary step failure being ignored?
If you happen to notice the error that is... like the OP pointed out, almost all bash shell script ignore silently errors and figuring which of the hundred of lines fails on your particular setup is a fun exercise.
And then... which bash debugger to you use to fix problem in bash shell script you been handed? Ah no, it's long reading and echo and trial and errors.
> And then... which bash debugger to you use to fix problem in bash shell script you been handed? Ah no, it's long reading and echo and trial and errors.
Or you could just add "-x" to the infamous "set -Eeuo pipefail" at the top of the script before running it, in which case it'll let you know exactly which of the hundreds of lines failed.
Alternatively, if you're using the "script template" in the submission, you can just pass in the "-v" option when you run the script and it'll take care of setting it for you.
"Debian Almquist shell. A small POSIX-compliant shell that is faster than bash."
I am very skeptical of this claim...in practice I've found it to rarely be the case. Usually shell scripts have more dependencies, and even worse, they are implicit since Bash provides no mechanism for managing dependencies. Programs like sed, awk, grep, cut, paste, curl, jq, gzip, tar, etc. These are all separate dependencies, usually in separate packages (some are grouped together into coreutils), supporting different feature sets between distributions and OSes. All of those can be replaced with the Python standard library.
Bash itself is pretty anemic and isn't very useful without gluing together external programs, which is its raison d'etre after all.
A shell script is literally intended to "glue together" external programs, including all of CoreUtils. If you are doing more than just "gluing together" other programs, then you are programming, not scripting, and you'd be far better off reaching for an actual programming language instead of a shell script of any kind.
Python has plenty of warts too. You can't depend on it being installed on any given system, can't depend on the version even (Python 2.x or 3.x?), which libraries are installed, environment, venv support, etc.
I'm not saying Python isn't without its problems too. I don't even like using Python that much for scripting. But we should be clear about what tradeoffs we're making when we write Bash scripts, and not pretending like we have "zero dependencies".
Not many people are running macOS servers... and servers tend to be where most shell scripts are being used.
macOS has notoriously out of date versions of all system programs, CoreUtils and Bash among them, due to licensing issues with Apple's closed-source proprietary OS. BSD and Linux based systems, generally don't have this issue.
That being said, a regular Shell Script (not a Bash Script) will run unmodified on macOS. When people say "shell script", they usually mean an actual `sh` script instead of a Bash script.
Thinking of macOS as "just another *NIX" really doesn't fly these days. Crazily outdated standard systems utilities is just the tip of the ice berg, really.
Your team could then freely use whatever computer/OS they wanted, but also keep everyone's development systems consistent with each other, as well as avoid any strange Apple-induced issues that come with running Apple hardware as a developer in 2020.
You're really fighting an uphill battle with macOS these days... as sad as that might be for some.
You've spelled it out in a few different posts - macOS breaks everything for your team. So why fight it? Get a different OS to develop in... or dumb down everything because macOS uses crusty, old utilities.
How is that not what we're already doing? Minikube runs in a VM. But we have to set it up somehow. Running a minikube VM inside a maually configured linux VM makes no sense - if we did that, now we'd need a scipt to boottrap the linux VM.
> You're really fighting an uphill battle with macOS these days... as sad as that might be for some.
I can definitely see this becoming more and more true in many fields, but this hans't been an issue for us.
I was implying a development environment inside a VM, not just a VM for testing.
You posted several times about how macOS isn't compatible with your production environments scripts and what-not. The solution is to either not use macOS, or use a standardized VM every developer on the team uses to develop inside of, so that your scripts work "out of the box".
> I can definitely see this becoming more and more true in many fields, but this hans't been an issue for us.
Your posts seem to contradict this. Perhaps you were exaggerating the pain it causes the team when production scripts don't work on developer systems. Either way - it seems the writing is already on the wall. Apple isn't going to make it easier any time soon - so ditch the mac's.
Another solution would be to write those scripts against a language and standard library that doesn't have wildly varying behaviour between its underlying platforms. One of my side projects has a bunch of supporting scripts in Python doing similar things and they run exactly the same on Linux and macOS. They even run fine on Windows with only minimal modifications.
> Your posts seem to contradict this. Perhaps you were exaggerating the pain it causes the team when production scripts don't work on developer systems. Either way - it seems the writing is already on the wall. Apple isn't going to make it easier any time soon - so ditch the mac's.
With this statement I was referencing specifically the complaints that have surfaced in the past few years about new OS changes and security features breaking tooling. I don't count our infra team using new bash features in this because the issue with outdated bash/unix tooling is a long-standing one.
I can't speak for others, but for me personally, the usability of macOS for almost all of the things I do on a day-to-day basis is just day-and-night better than anything else, even if I have to hack up a bash script to work correctly from time to time. I will say the difference is less than it used to be - I use it occasionally and Linux has come a long way in the past 10 years.
(I know what you mean by VM, but I don't think the distinction is very important here, you get the same benefits either way, and using Python is arguably a lot simpler than using a full VM.)
(Though I use those as daily-driver packages on Macs and have actually never had something break? GNU tools tend to support BSD options.)
I actually hadn't thought to try this approach, though. The next time this happens I will definitely give it a shot.