That's not dash, it's bash in POSIX compatibility mode. dash does not implement all the POSIX stuff in this "bible".
Imagine using this "bible" to write a "pure sh" script thinking "this will be highly portable", only to have it fail because the Linux scripting shell is not bash in POSIX compatibility mode, it's dash. It would fail on NetBSD, too, whose scripting shell is the version of ash from which dash is derived.
Because if you are not 100% compatible with bourne shell you shouldn't automatically execute script made for bourne shell.
I have no problem with people using #!/bin/dash shebang for their scripts and using dash as their default shell but it shouldn't pass itself as something else. It is fine with bash because it is bourne shell compatible.
> Because if you are not 100% compatible with bourne shell you shouldn't automatically execute script made for bourne shell.
The same can be said about bash.
> It is fine with bash because it is bourne shell compatible.
I was running an HPC cluster when we upgraded Debian during the bash->dash change, and there were all sorts of problems: IIRC most folks simply changed their scripts from /bin/sh to /bin/bash.
Do you have any specific examples to cite? The snippets are meant to be run in POSIX-compliant shells like dash/yash/ash/etc, not just Bash in compatibility mode.
In fact, there used to be a section which listed workarounds[1] for bugs in dash, but they have since been fixed[2]. If you are still using an old version of dash, you may need to use them.
Perhaps dash now has these features but does NetBSD sh or FreeBSD sh have them.
What's the point of "pure sh" if it's restricted to specific versions of shells. Opinions may differ but I'd rather learn the "lowest common denominator". I use the same scripts on both Linux and BSD so I need portability. I do not use bash for scripting. It's larger and slower.
I'm using the latest version of dash. I just tested it and it does have these ternary operators. However, I'm not going to use them in scripts unless I know they'll work on older dash versions, OpenWRT sh, Android sh, NetBSD sh, FreeBSD sh, etc. The process of updating these shells is generally very slow. It can take years.
"Pure sh" implies there is some advantage over "bashisms". I thought maybe it was portability but perhaps it is something else.
I like to write posix sh scripts for the sake of portability and, funnily enough, future proofing as I don't like having to maintain stuff against changes that break compatibility, which is something bash does (archlinux is still on an older bash even though debian stable has the latest, because bash 5.2 broke some of archlinux's own scripts. This is why you should not write bash scripts.), but bash being slow isn't one of the good reasons. If your scripts do that much work that your shell's speed matters, you should reconsider writing shell scripts and start thinking about using something like perl, python or ruby. I'd suggest perl, if only because unlike the latter, perl doesn't constantly pull the rug under you and make you do busy work to make sure your scripts work on the latest runtimes.
It is nothing as major as what Python does, but there are routine paper cuts through deprecation and breaking changes across versions, the most notable being 1.9 and 3.0.
But the language itself isn't the worst offender, the whole ecosystem around it tends to not even make reasonable attempts at stable APIs. Which isn't helpful with the way dynamically typed languages typically lack in tooling to safely, properly deal with refactoring - there are exceptions like smalltalk I've heard, but overall the cavalier attitude toward delivering a constant treadmill isn't pleasant to me. Of course, some may argue that the reason Perl doesn't suffer as much from it is because it is "dead", but I'll take dead over busywork.
I must say, I don't like Java the language, but Java the platform almost feels like the promised land compared to this. I don't use it to develop myself, but when I ran into an old unmaintained jar and ran it on a current JVM and it just works.. it feels.. fantastic. Absolutely wonderful. The same sort of feeling I also have when I run old games on Wine and it just works. win32 is the stable API I always wished linux had for GUI apps but will never have because the culture is all about CADT.
I had the misfortune of having to maintain an 18'000 line bash 3 script. Array handling broke in many subtle ways between bash 3 and 4.
> If your scripts do that much work that your shell's speed matters, you should reconsider writing shell scripts and start thinking about using something like perl, python or ruby.
Usually yes.
But there is a small subset of use cases which need a really portable solution. Once I would have recommended to just write it in perl, because it was on virtually all systems. But now perl is getting phased out on some OSs/distros, and with python you never know if you get 2 or 3. Or embedded systems which don't have python/perl in the first place.
My go to solution for slow scripts is usually to limit subshells/forks as much as possible, and/or run it with busybox with the "exec prefers applets" option
edit: it's actually the "run 'nofork' applets directly" option. Can give quite a speed boost compared to bash if you have to call "external" utilities like grep/head/etc.
> I had the misfortune of having to maintain an 18'000 line bash 3 script
My hats off to you, I do not think I have the fortitude to stomach.. this.
> run it with busybox with the "exec prefers applets" option
> edit: it's actually the "run 'nofork' applets directly" option. Can give quite a speed boost compared to bash if you have to call "external" utilities like grep/head/etc.
Nifty trick, I had no idea about that, thanks for sharing this.
Yes and yes. My FreeBSD machine has not even been updated in 5 years, if that helps.
> What's the point of "pure sh" if it's restricted to specific versions of shells.
The aforementioned features have been implemented for a very long time. The issues with old versions of dash I mentioned were straight-up crashes for very simple things, not some fancy new-fangled feature that was yet to be implemented. Even then, the bible specifically listed workarounds for them.
dylanaraps has quite a prolific collection of shell programs, and they run on a variety of operating systems. Most notably, pfetch[1] runs on Linux, Android, NetBSD, FreeBSD, OpenBSD, Minix, Haiku, macOS, Solaris and IRIX. I assure you that he is fully aware of the importance of compatibility.
His pure Bash bible even has very thorough warnings for Bash versions required, since macOS uses Bash 3.2 (released in 2006).
There are people commenting on HN who do not understand the difference between compatibiity and portability. This doesn't stop them from commenting on someone else's preference for portability.
> That's not dash, it's bash in POSIX compatibility mode.
I was under the impression that bash's POSIX mode was 'leaky': that a lot of non-POSIX stuff/extensions are available even though it's called as /bin/sh.
I know this caused a lot of our users issues when we upgraded Debian/Ubuntu during its switchover to dash. We told folks you could either rewrite those parts of the code or call /bin/bash.
One major reason that Debian chose dash is plainly written on the bash manual page:
$ man bash | sed -n '/BUGS/,/^$/p'
BUGS
It's too big and too slow.
Another major reason is adherence to the POSIX standard; bash was written a decade prior to standardization, and has interesting issues because of this.
> bash was written a decade prior to standardization, and has interesting issues because of this.
bash can do whatever it wants to do if it's called as "bash". But it should only do specific things when called as "sh". This is true of any shell:
> When interpreted with dash instead of bash, the same script will fail. This is because dash is much stricter than bash in following the sh standard. Since dash is designed as a minimal implementation of the sh standard, it has to be stricter. The double brackets [[ … ]] are a ‘bashism,’ or a feature only available in bash and other, later shells such as ksh and zsh.
> Even though zsh also interprets most of these bashisms, zsh in sh compatibility mode is also stricter than bash and will error.
POSIX.2: Shell and Utilities (IEEE Std 1003.2-1992) is a few decades old now: regardless of when bash development started, I would think that by the 2010s it would have managed to better compartmentalize compatibility and extensions.
I would argue reaching for the shell to do your scripting is never a good idea, unless your problem is trivial to begin with (e.g. fits in less than a hundred lines of extremely readable and straightforward code). My rule of thumb is: "can I fit it in a one-liner?" - then I refactor it until it's readable, or otherwise use something else.
Once you need to do fancy array manipulation, hash tables, data structures, etc you're very likely much better off with Python, Perl, Ruby, Lua, Tcl, JS, or literally anything else. Notably, one or both of Python and Perl are available out of the box on Debian (and derivatives), macOS, most (all?) BSDs, and so on. Targeting Python 3.6+ will probably cover 99%+ of the systems you can find in the wild (e.g. Ubuntu 18.04 and newer).
I have also had some minor success using Go for that purpose. The toolchain can trivially cross-compile executables for ca 40 different OS/arch combinations; you need to consider the os/arch if you're running something different than Linux/x86-64 but that will again likely put you in the 1%. I'm working on some tooling to make this approach more straightforward.
One-liners are meant to be thrown away, so it does not matter if it's pretty or not. If you care enough to keep it, you should consider rewriting it for clarity. That last step does not require using any particular language; e.g. my PS1 started off as an increasingly convoluted one-liner, until I rewrote it in Go.
Depends on what you're trying to do. If you're shelling out to git(1) or docker(1), rather than e.g. recursively checking for the presence of .git in parent directories, or inspecting ~/.docker/config.json, then the fork+exec overhead is already quite significant. Next if you're parsing ~/.docker/config.json in shell, you're most likely either asking for trouble or (again) shelling out to jq. Writing it all in an interpreted language means you're paying the cost of interpreter startup, which on underpowered systems can take hundreds of milliseconds even when idle. OTOH loading a static binary to memory happens only once, and with Go you can trivially cross-compile.
I also have a fallback shell one-liner, without any of the fanciness like displaying the current git branch:
Less critical then yesterday because of ChatGPT, I always expected to become fluent in bash at some point, but the AI became too good at writing my scripts now...
Also you can use the chat as a learning tool, it's really a game changer for whom want to really learn it.
as someone who is late to this party, can you tell me how you are using it? In an IDE? what sort of way are you using it to learn rather than just give answers? thx.
Just use the website. You can explain (in english or even native language if relevant) your algorithm and for small functions it's giving you correct answer. You still need to concatenate them.
For learning in my case I needed to learn python and I'm already expert in javascript for example. So you can ask the prompt for the equivalent in python every time you feel stuck. But using the chat is so more precise and quick, than google for this kind of questions. Also without all the noise of google. In particular without this "noise" and time and concentration to find the response, it's a game changer when you really want to just learn and memorize.
I think this is a trap for engineers inexperienced with Bash. You can (and should of course) use ChatGPT to be more productive in writing bash scripts, but as was said in the previous discussions, bash contains so many traps which you won't be able to spot.
Bash's dangers might be exacerbated by using ChatGPT, since you'll rely on the tool and you might not be able to understand what's wrong if shit hits the fan.
I'd recommend not using bash for anything other than the simplest tasks, as per Google's Shell style guide [0].
What could go wrong? Use a technology that is notorious for outputting wrong information to write a script that is really hard to get it right and then later reviewed by someone that can’t really understand it.
You say that likes writing code doesn’t involve experimentation in general. You can’t trust the code that that Chatgpt gives you, but you also can’t trust the code that even you write until you test it. It is always an iterative process. Why does it matter that some use AI tools as a starting point?
Either it gives you so much that you never learn anything of use from those iterations or it gives you too little that you might as well write it from scratch (and get better quickly).
> Why does it matter that some use AI tools as a starting point?
I rather learn (or iterate) using a proper source of information (like a book, a blog post written by someone with actual knowledge, or even with a more experience coleague). At least for me it seems more productive.
Try pasting a piece of code you don't understand and ask LLM to explain it to you.
Then try the same and find a human to explain it to you.
Then compare the price (including money and time) and performance of both.
I won't believe you made an objective assessment if your conclusion is that either party is the best solution all the time. The point is right now the number of occasions that AI has better price performance is large enough for it to be generally useful.
> I won't believe you made an objective assessment
Why even ask then?
I have tried to use chatGPT a couple of time and many of those times it was wrong,
to a point that if I ever use again I'd definitely double check the
output(which makes me not bother using it in the first place).
Cute that some people find it useful, I don't, it has no use for me in the
state it is today.
I point this out because I’m pretty sure lazy reliance on spellcheck and autocorrect are to blame with about half the internet using then when they should be using than.
I shudder to think what a few years of ChatGPT is going to do.
Is the fastest way of becoming proficient with shell scripting. Whether it's pure posix shell, or bash and its extensions, they're small languages and can be learned in a week end if you're proficient in another programming language. The main pain point is that there's a lot of "surprises" and non-intuitive behaviors, that are all listed here. The first ones, about parameter expansions, quoting variables and leading dashes are the most common issue with newbies in shell scripting.
There's a lot of amazing tooling these days to make sure your shell scripts are fine too.
First, shellcheck:
And finally, checkbashisms if you intend on making pure posix scripts that are compatible with debian/ubuntu's dash. It is part of the debian's devscripts suite, but is often individually packaged in other distros.
> Also you can use the chat as a learning tool
Or you could learn from a guide written by people who have suffered decades of experience of the pitfalls of shell scripting and have shared their woes.
If only to note, say, the section(s) of the Bash manpage or GNU Info Manual in which topics are discussed.
Taking the first-listed example, the '%%', '##', etc., substitutions are covered in Parameter Expansion.
I tend to go first to manpages, though I'm well aware that GNU deprecates these (exceedingly unwisely IMO) in favour of the Info Doc format (and its dedicated single-purpose reader). I'll note that the manpage alone renders to 120 pages (man bash | pr | less) onscreen, which is fairly formidable.
People all too often forget that there is often a lot of extra documentation hidden in /usr/share/doc/ when you install -doc packages, at least on deb systems, and not just info and manpages. For example, the bash manual you cite is in /usr/share/doc/bash/bashref.pdf along with its html version as bashref.html. There is also an /example/ folder filled with commented programs and functions to serve as reference. And a README that suggests looking for "Advanced Bash-Scripting Guide" which is a massive online book filled with all sorts of esoteric knowledge. You can also find things like the release changelogs and one that details incompatibilities from previous versions.
While there are exceptions, most packages have useful goodies in there to check first before one has to think about looking things up online.
What you'll find is both a set of pages & hierarchy of documentation as well as search (via swish++) through that space. And yes, there is an absolute trove of information available:
- Manpages
- Info pages
- Default package information
- Supplementary package documents, if installed
- HOWTOs from the Linux Documentation Project.
- Debian-specific documentation (installation, administration, security, policy, and other manuals)
- RFCs (divided into several groups, for manageability / relevance)
- Resources such as the Linux Gazette
There's also a commandline utility, dwww, which will invoke your browser of choice. Point that at a terminal-based browser (w3m, lynx, links, elinks, etc.) and you can quickly scan through docs from your terminal.
Depending on your job description it is completely fine to not write shell scripts.
For anything bigger than a screenful of lines or something that would need proper unit testing or something that other people who are likely to be less versed in shell scripting need to maintain, think twice about using shell scripts.
Unixoid machines can use anything as scripts and using a language that breathes the programming language designs of the 1970s is not the best choice.
I typically agree, but I don't think its as simple as a "screenful of lines" criteria.
I find it useful when most of your work is wiring other command line tools together. When that's most of your work, shell scripting feels like it has the best affordances.
Anything that gets too fancy beyond this, with anything that resembles an algorithm or computation, you'll feel real pain :) Bash doesn't even support floating point!
For me this is is it, if you're stringing together a load of other command line tools then a large well structured shell script is perfectly fine.
A typical task in my day job is extracting metadata from and transcoding large AV archives, I always use a bash for this because I'm mainly stringing together tools like ffmpeg or sox, making directories, tempfiles, creating chesksum manifests, etc.
Also I typically do this on large CPU machines with many cores, I can parallelize operations in one line using xargs and a bash function. Doing stuff like this in python is actually a real pain in comparison.
There's a library for Node called Zx (https://github.com/google/zx) that provides a more shell-liked interface for Node.
I've been working on something in Python, inspired by Zx and Xonsh (https://xon.sh/), with syntax like this:
from pathlib import Path
from pyshell import Pipeline
args = ["123", "456"]
f = Path("out.txt")
with Pipeline() as p:
p.exec("a") | p.exec("b", *args) | p.exec("c", "x", "y") >> f
Personally, the above would be hugely useful to me in reducing my temptation to write shell scripts, and significantly reducing the annoyance of doing these things in Python.
Is this something you'd use? It's slow going for me because it basically involves writing a compiler and I have approximately zero idea of what I'm doing. Knowing that other people would find it useful helps me stay motivated to keep working on it! Or if someone who theoretically is the primary audience doesn't find it useful, it might mean that I should spend my energy on other things.
This looks cool, especially if one needed to do a number of pipe operations from python, I'm not sure of an elegant way to do that already.
Honestly I might be the wrong audience though, I've spent many years shell scripting and I find the flexibility of using the command line tools so great I choose to use it first despite the inadequacies of bash. I wonder if users with my point of view would ever be tempted away.
Also the shell is an interactive repl, which to me is where a lot of the power lies. The number of times I've asked a junior things like what the total duration of a set of audio files is, or the average length or something. They often come back with a python script or worse a jupyter notebook and pandas. The answer can also often be found by combining things like grep and awk in a few seconds once you've become acquainted with the tools, I guess I do try and demonstrate / preach this where I can.
Yeah every invocation of bc is a call to a command. So if you do it in a loop, its going to be very slow.
I had to do this once. It was so slow, I decided to use integers in kind of fixed-point math where I just multiply all the floats by 100 and work with ints.
There was a really awful and amazing floating point implementation posted here¹ a couple of years ago, with a few interesting comments too². I remember it in part because I was surprised there wasn't more discussion.
Yes. Can someone please write an LLM that can translate shell scripts to Python, for those cases when my shell script starts exceeding a screenful of lines.
To be fair for most of the shell scripts I write it would be more work to put it into an LLM and then verify what it output is correct than just retyping it in python.
I tend towards python these days if I need more than 1 variable since at that point I'm probably in the weeds already.
If it's just a ton of commands in succession though, I leave it bash. I have a few scripts in bash and PowerShell for weird ffmpeg operations I generally don't want to remember, stuff like loudness normalisation and converting to specific formats with specific parameters using ffmpeg.
I'd love to see a little language which compiles to _readable_ bash. Could it be so hard? There's a dead project that comes up now and then called bish.
One of my current hobby projects is a scripting language which compiles to POSIX sh (for maximum portability). However, it's far from complete, the output is not exactly readable and POSIX sh is so limited that I might need to change course and target something else.
EDIT: I didn't know of Bish, but from its code samples we seem to have converged on similar syntax :)
Maybe this is a bit of a silly question, but what's the point?
I do think it would be a really fun project. But is there really a practical use for it? Bash is portable but not that portable if you want to care about version compatibility. I understand that it's not always possible to install a different interpreted language like Perl or Python. Maybe the use case is for platforms where you can't install other language interpreters and can't cross-compile a compiled language? It seems like a very specific situation.
> If I get good enough at bash, will it get better?
Bash itself will not get better because you'll be stuck writing scripts targeting some ancient version to ensure wider compatibility.
It's still a powerful skill to develop. It's a sweet spot between a one-off shell pipeline and something that involves actual logic.
Bash scripts should not be used for their logic. Think of them as a way to document a series of commands that you do not want to manually type. That can be including long argument names, generating includes or classpaths, or generating log file names based on the current time.
A good rule of thumb is if your script involves any if-statement other than parameter validation or basic on-success-of-x-do-y, it probably should be in a real language.
I don't agree on -e: see my comment higher up in the thread. It might be ok if you're hyper-aware, but recommending it to others will leave them baffled and confused in situations like that in my example where a non-zero exit is within the bounds of normal behavior of the script.
I like bash, zsh and other shell languages as well. However, unless you're really experienced, they shouldn't be the product you ship, whether that is internally or externally to you or your company.
Shell languages is the final layer of configuration and customization you apply to the setup, it should almost never be the foundation of what you build.
Especially for a company, unless you've got really, really skilled engineers on the scripts, it becomes a complete mess. And even then it can become almost impossible to grok.
The way we develop infrastructure automation, ci, and developer tools is binaries first and foremost, then plugins for our infrastructure (also binaries, or libraries) and then some zsh on top to provide a nice developer experience.
But we've had to heavily scale back our use of zsh, it simply turned into a ball of mud, like all software that isn't maintainable, upgrade-able, or simple enough to parse.
This isn't a dig on zsh, bash etc, we've just found out that we can't maintain zsh as a product to supply to our engineers, and as such have to correct.
At least you've discovered Zsh and hopefully its significant improvements over Bash. The only Bash benefit is readarray/mapfile, Z Shell is better in just about every other way.
Unless I'm misreading your comment, zsh does provide similar functionality via zsh/mapfile¹. It, and a lot of other cool stuff, is documented in zshmodules(1).
Most commercial products that include an installer on *NIX are written in shell. No other method allows you to have a universal installer on every platform that will always work.
> Especially for a company, unless you've got really, really skilled engineers on the scripts, it becomes a complete mess. And even then it can become almost impossible to grok.
Dude. It's really not that hard a language. Is amateur shell script ugly? verbose? redundant? bug-riddled? Absolutely. But it works despite that, and has fewer dependencies than other solutions, so in practice, it's fine.
That said, use whatever you know. Wanna use Python? Use it. Wanna use a compiled language? Do it. Just make something. The biggest difficulties you will have in producing a product will not be the language.
You don't have to avoid shell just because people on HN (who've probably never read the man page) keep repeating that it's terrible. With Shellcheck, almost anyone can write good scripts now. If you can write a concise, bug-free script that works, then do it.
No other method allows you to have a universal installer on every
platform that will always work.
No shell will guarantee that either. Last week I spent a few hours debugging an installer (bash script) that was failing with a rather puzzling error message (variable not defined). Non-Linux, bash 5.x. Worked on nearly everything under the sun including OSX with its ancient GPLv2 bash.
Turns out on one non-Linux operating systems, one of the utilities used for string manipulation, while POSIX compliant, behaved differently than the same utility in nearly every other operating system.
has fewer dependencies than other solutions
Wrong, wrong, wrong. Nearly everything you do in a shell script is an external dependency. And that's the biggest problem with shell scripts. The actual language is so narrowly scoped that you're dependent upon external utilities that are not so easy to upgrade.
Count how many programs are in /bin. Those are your dependencies. Unlike e.g. python or ruby dependencies, you'll have a much harder time upgrading /bin. Having to fork a new process for trivial tasks is also one of the reasons why shell scripts are so slow.
Sure, bash has builtin versions of some of those utilities like [, but not every system has bash.
This seems to be a general problem for what I normally refer to orchestration software. I.e. calling a lot of external dependencies, io and in general very slim on business logic. You've simply got to deal with too many dependencies, which may or may not have different versions, flavors and whatnot. It becomes impossible to test every permutation, so instead you rely on brute force to weed out bugs, simply fix the things once a bug report comes in.
It helps having good integration/e2e tests, but they only go so far, customers/developers customize and mess up their systems all the time.
Which is why I've choose binaries instead, they are mostly self-contained (except for various things about gnu and musl), and in general just rely on a higher level of built in functionality (because of the statically linked nature).
It is probably the closest we're gonna get to a universal language, until we get wasm deeper in the os' or something. It is kind of the javascript of operating systems, which is both good and bad.
It is also quite bad that we've got to vet bash installation scripts before running them, as you may be required to run them as root. IMO, this isn't a huge issue for me personally, as we already are doing this for a lot of stuff, homebrew, linux package managers etc. We generally run a lot of untrusted code on our machines.
> Dude. It's really not that hard a language. Is amateur shell script ugly? verbose? redundant? bug-riddled? Absolutely. But it works despite that, and has fewer dependencies than other solutions, so in practice, it's fine.
Bash, zsh whatever, depend heavily on what is installed on the os, and what type of flavor of base tools are provided. Need base64 in a certain way, don't forget that line wrapping is different between bsd and linux. (a pain in the butt to test too).
Need to download something, better hope you're not running on a busybox or something with curl, or only have wget installed, now you gotta build support for both.
A lot of scripts, especially installation scripts are a product of blood, sweat and tears, mostly tears tho. Each line can be interpreted in a multitude of ways, under different conditions and versions of bash, os and so on, and the script will have to be robust enough for that.
> That said, use whatever you know. Wanna use Python? Use it. Wanna use a compiled language? Do it. Just make something. The biggest difficulties you will have in producing a product will not be the language.
It may not be the biggest difficulty, but it certainly is something we should consider. We've chosen our values, and shell scripts doesn't align with those, I'd rather not fight my environment every step of our development journey.
I am not building a new product, I am maintaining and expanding a product, as such I am heavily biased towards getting as few support tickets as humanly possible, while building an awesome experience. We are also mostly software engineers, and not sysadmins, as such we prefer compiled binaries. We then choose to distribute them through curated and secure channels.
We get the benefit that we can actually test parts of our tools in isolation, benchmark them if necessary, and version our stuff in a nice concise way. If I want to run our onboarding installation, I can simply fetch our current list of binaries expected for developers and run it, because I know they don't depend on external tools from the environment. This is a lot of work, but in a large development organization, this means a lot.
> You don't have to avoid shell just because people on HN (who've probably never read the man page) keep repeating that it's terrible. With Shellcheck, almost anyone can write good scripts now. If you can write a concise, bug-free script that works, then do it.
Just to reiterate, I am not digging on shell scripts, and I don't necessarily think they're terrible, they have their time and place, and for us, it is customization and extensibility. We setup our tools such that they work awesome on the cli, and can be composed however, developers like, but that is up to them, and how they want to form their workflows.
We've seen that developers really struggle to maintain what was previously built (myself included), so now we've got a more homogeneous tech stack, which comes with its own trade offs, but whose values align more with our engineering culture.
With LLMs/ChatGPT, i would prefer fine-tuning a model and have it auto-generate bash commands with my natural language description.
I remember, I used to keep a diary with landline phone numbers of relatives etc. I used to remember at least a dozen of those numbers for emergency. With mobile and digital contacts, now that number is down to just 2.
Wonder, what the programming is going to look like in 10 yrs. If no-one dedicates so much of their time and effort to document, the programmers then would feel helpless. right?
1. Or at that point, LLMs will be intelligent enough to read the source code and figure out the commands.
2. They can generate data by running commands, and train themselves using this simulated data.
What other scenarios will occur once we stop generating data using human effort because we all are so dependent on LLMs to generate those commands?
There's a middle ground alternative where the LLM can read the source, give you a high level description (aka documentation), plus cite the relevant parts of the code. Fingers crossed.
I always hear people talk about writing portable, POSIX compliant shell scripts so they'll work in more environments (assuming those environments are needed, of course), but how often do you run into an environment where you don't have a fairly complete version of Bash available? Genuine open question, I'd be really interested to know what that situation is. Lean container images, embedded, older systems? I'm really curious.
Containers/docker/kubernetes and embedded systems that use e.g busybox is what I’ve seen. Also macOS (used to?) ship ancient bash versions, but there’s recent zsh so that’s mostly fine.
It's so tempting to implement in bash. I made the mistake _again_ this week! It's so seductive to be able to start subprocesses with no ceremony, use output redirection, use jq, etc.
If you have even the slightest hesitation regarding whether the task in front of you would be better done in a real programming language, the answer is do it in a real programming language.
A lot of what's in the Pure Bash Bible is horrifically slow. Many of those things are substantially faster, even when paying the cost of starting a new process, when you use an external and commonly available tool. I wrote a bash performance profiler that outputs data in a format that flamegraph.pl recognizes - it really helped identify where we could improve the performance of ZFSBootMenu.
105 comments
[ 2.5 ms ] story [ 164 ms ] threadhttps://news.ycombinator.com/item?id=30363506 (291 points | Feb 16, 2022 | 96 comments)
https://news.ycombinator.com/item?id=21013150 (746 points | Sept 19, 2019 | 248 comments)
See also: Pure Sh Bible https://news.ycombinator.com/item?id=35765707 (406 points | 3 months ago | 133 comments)
Imagine using this "bible" to write a "pure sh" script thinking "this will be highly portable", only to have it fail because the Linux scripting shell is not bash in POSIX compatibility mode, it's dash. It would fail on NetBSD, too, whose scripting shell is the version of ash from which dash is derived.
Why?
I have no problem with people using #!/bin/dash shebang for their scripts and using dash as their default shell but it shouldn't pass itself as something else. It is fine with bash because it is bourne shell compatible.
The same can be said about bash.
> It is fine with bash because it is bourne shell compatible.
I was running an HPC cluster when we upgraded Debian during the bash->dash change, and there were all sorts of problems: IIRC most folks simply changed their scripts from /bin/sh to /bin/bash.
Then it doesn't sound fit for scripting.
In fact, there used to be a section which listed workarounds[1] for bugs in dash, but they have since been fixed[2]. If you are still using an old version of dash, you may need to use them.
[1] https://github.com/dylanaraps/pure-sh-bible/commit/70f410ebb...
[2] https://github.com/dylanaraps/pure-sh-bible/issues/13
Perhaps dash now has these features but does NetBSD sh or FreeBSD sh have them.
What's the point of "pure sh" if it's restricted to specific versions of shells. Opinions may differ but I'd rather learn the "lowest common denominator". I use the same scripts on both Linux and BSD so I need portability. I do not use bash for scripting. It's larger and slower.
One source I consult is https://www.in-ulm.de/~mascheck/bourne/
It is sometimes submitted to HN, but not IMO enough
https://news.ycombinator.com/from?site=in-ulm.de
I'm using the latest version of dash. I just tested it and it does have these ternary operators. However, I'm not going to use them in scripts unless I know they'll work on older dash versions, OpenWRT sh, Android sh, NetBSD sh, FreeBSD sh, etc. The process of updating these shells is generally very slow. It can take years.
"Pure sh" implies there is some advantage over "bashisms". I thought maybe it was portability but perhaps it is something else.
I like to write posix sh scripts for the sake of portability and, funnily enough, future proofing as I don't like having to maintain stuff against changes that break compatibility, which is something bash does (archlinux is still on an older bash even though debian stable has the latest, because bash 5.2 broke some of archlinux's own scripts. This is why you should not write bash scripts.), but bash being slow isn't one of the good reasons. If your scripts do that much work that your shell's speed matters, you should reconsider writing shell scripts and start thinking about using something like perl, python or ruby. I'd suggest perl, if only because unlike the latter, perl doesn't constantly pull the rug under you and make you do busy work to make sure your scripts work on the latest runtimes.
https://www.ruby-lang.org/en/news/2019/12/12/separation-of-p... https://nuclearsquid.com/writings/ruby-1-9-what-s-new-what-s...
But the language itself isn't the worst offender, the whole ecosystem around it tends to not even make reasonable attempts at stable APIs. Which isn't helpful with the way dynamically typed languages typically lack in tooling to safely, properly deal with refactoring - there are exceptions like smalltalk I've heard, but overall the cavalier attitude toward delivering a constant treadmill isn't pleasant to me. Of course, some may argue that the reason Perl doesn't suffer as much from it is because it is "dead", but I'll take dead over busywork.
I must say, I don't like Java the language, but Java the platform almost feels like the promised land compared to this. I don't use it to develop myself, but when I ran into an old unmaintained jar and ran it on a current JVM and it just works.. it feels.. fantastic. Absolutely wonderful. The same sort of feeling I also have when I run old games on Wine and it just works. win32 is the stable API I always wished linux had for GUI apps but will never have because the culture is all about CADT.
> If your scripts do that much work that your shell's speed matters, you should reconsider writing shell scripts and start thinking about using something like perl, python or ruby.
Usually yes.
But there is a small subset of use cases which need a really portable solution. Once I would have recommended to just write it in perl, because it was on virtually all systems. But now perl is getting phased out on some OSs/distros, and with python you never know if you get 2 or 3. Or embedded systems which don't have python/perl in the first place.
My go to solution for slow scripts is usually to limit subshells/forks as much as possible, and/or run it with busybox with the "exec prefers applets" option
edit: it's actually the "run 'nofork' applets directly" option. Can give quite a speed boost compared to bash if you have to call "external" utilities like grep/head/etc.
My hats off to you, I do not think I have the fortitude to stomach.. this.
> run it with busybox with the "exec prefers applets" option
> edit: it's actually the "run 'nofork' applets directly" option. Can give quite a speed boost compared to bash if you have to call "external" utilities like grep/head/etc.
Nifty trick, I had no idea about that, thanks for sharing this.
Yes and yes. My FreeBSD machine has not even been updated in 5 years, if that helps.
> What's the point of "pure sh" if it's restricted to specific versions of shells.
The aforementioned features have been implemented for a very long time. The issues with old versions of dash I mentioned were straight-up crashes for very simple things, not some fancy new-fangled feature that was yet to be implemented. Even then, the bible specifically listed workarounds for them.
dylanaraps has quite a prolific collection of shell programs, and they run on a variety of operating systems. Most notably, pfetch[1] runs on Linux, Android, NetBSD, FreeBSD, OpenBSD, Minix, Haiku, macOS, Solaris and IRIX. I assure you that he is fully aware of the importance of compatibility.
His pure Bash bible even has very thorough warnings for Bash versions required, since macOS uses Bash 3.2 (released in 2006).
[1] https://github.com/dylanaraps/pfetch
I was under the impression that bash's POSIX mode was 'leaky': that a lot of non-POSIX stuff/extensions are available even though it's called as /bin/sh.
I know this caused a lot of our users issues when we upgraded Debian/Ubuntu during its switchover to dash. We told folks you could either rewrite those parts of the code or call /bin/bash.
* https://wiki.ubuntu.com/DashAsBinSh
* https://lwn.net/Articles/343924/
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
bash can do whatever it wants to do if it's called as "bash". But it should only do specific things when called as "sh". This is true of any shell:
> When interpreted with dash instead of bash, the same script will fail. This is because dash is much stricter than bash in following the sh standard. Since dash is designed as a minimal implementation of the sh standard, it has to be stricter. The double brackets [[ … ]] are a ‘bashism,’ or a feature only available in bash and other, later shells such as ksh and zsh.
> Even though zsh also interprets most of these bashisms, zsh in sh compatibility mode is also stricter than bash and will error.
* https://scriptingosx.com/2020/06/about-bash-zsh-sh-and-dash-...
* https://zsh.sourceforge.io/Doc/Release/Invocation.html#Compa...
POSIX.2: Shell and Utilities (IEEE Std 1003.2-1992) is a few decades old now: regardless of when bash development started, I would think that by the 2010s it would have managed to better compartmentalize compatibility and extensions.
If you alias p=printf in a #!/bin/sh script, it will work; it will fail in a #/bin/bash script, unless POSIX mode is forced.
Once you need to do fancy array manipulation, hash tables, data structures, etc you're very likely much better off with Python, Perl, Ruby, Lua, Tcl, JS, or literally anything else. Notably, one or both of Python and Perl are available out of the box on Debian (and derivatives), macOS, most (all?) BSDs, and so on. Targeting Python 3.6+ will probably cover 99%+ of the systems you can find in the wild (e.g. Ubuntu 18.04 and newer).
I have also had some minor success using Go for that purpose. The toolchain can trivially cross-compile executables for ca 40 different OS/arch combinations; you need to consider the os/arch if you're running something different than Linux/x86-64 but that will again likely put you in the 1%. I'm working on some tooling to make this approach more straightforward.
I also have a fallback shell one-liner, without any of the fanciness like displaying the current git branch:
https://github.com/rollcat/etc/tree/master/cmd/prompter#i-li...
Also you can use the chat as a learning tool, it's really a game changer for whom want to really learn it.
For learning in my case I needed to learn python and I'm already expert in javascript for example. So you can ask the prompt for the equivalent in python every time you feel stuck. But using the chat is so more precise and quick, than google for this kind of questions. Also without all the noise of google. In particular without this "noise" and time and concentration to find the response, it's a game changer when you really want to just learn and memorize.
Bash's dangers might be exacerbated by using ChatGPT, since you'll rely on the tool and you might not be able to understand what's wrong if shit hits the fan.
I'd recommend not using bash for anything other than the simplest tasks, as per Google's Shell style guide [0].
[0] https://google.github.io/styleguide/shellguide.html#s1.2-whe...
At this point just use something else.
I rather learn (or iterate) using a proper source of information (like a book, a blog post written by someone with actual knowledge, or even with a more experience coleague). At least for me it seems more productive.
Then try the same and find a human to explain it to you.
Then compare the price (including money and time) and performance of both.
I won't believe you made an objective assessment if your conclusion is that either party is the best solution all the time. The point is right now the number of occasions that AI has better price performance is large enough for it to be generally useful.
Why even ask then?
I have tried to use chatGPT a couple of time and many of those times it was wrong, to a point that if I ever use again I'd definitely double check the output(which makes me not bother using it in the first place).
Cute that some people find it useful, I don't, it has no use for me in the state it is today.
Than. Not “then,” than.
I point this out because I’m pretty sure lazy reliance on spellcheck and autocorrect are to blame with about half the internet using then when they should be using than.
I shudder to think what a few years of ChatGPT is going to do.
This page :
https://mywiki.wooledge.org/BashPitfalls
Is the fastest way of becoming proficient with shell scripting. Whether it's pure posix shell, or bash and its extensions, they're small languages and can be learned in a week end if you're proficient in another programming language. The main pain point is that there's a lot of "surprises" and non-intuitive behaviors, that are all listed here. The first ones, about parameter expansions, quoting variables and leading dashes are the most common issue with newbies in shell scripting.
There's a lot of amazing tooling these days to make sure your shell scripts are fine too. First, shellcheck:
https://www.shellcheck.net/
I systematically run this on my scripts before using them.
Then shfmt for nice, reliable autoformatting: https://github.com/mvdan/sh
And finally, checkbashisms if you intend on making pure posix scripts that are compatible with debian/ubuntu's dash. It is part of the debian's devscripts suite, but is often individually packaged in other distros.
> Also you can use the chat as a learning tool
Or you could learn from a guide written by people who have suffered decades of experience of the pitfalls of shell scripting and have shared their woes.
https://mywiki.wooledge.org/BashGuide
chatGPT is good at coming up with magical solutions, but teaching you about all the corner cases?
If only to note, say, the section(s) of the Bash manpage or GNU Info Manual in which topics are discussed.
Taking the first-listed example, the '%%', '##', etc., substitutions are covered in Parameter Expansion.
I tend to go first to manpages, though I'm well aware that GNU deprecates these (exceedingly unwisely IMO) in favour of the Info Doc format (and its dedicated single-purpose reader). I'll note that the manpage alone renders to 120 pages (man bash | pr | less) onscreen, which is fairly formidable.
The PDF-formatted GNU bash manual (a fine format for a reference of this sort) runs 196 pages: <https://www.gnu.org/software/bash/manual/bash.pdf>
While there are exceptions, most packages have useful goodies in there to check first before one has to think about looking things up online.
What you'll find is both a set of pages & hierarchy of documentation as well as search (via swish++) through that space. And yes, there is an absolute trove of information available:
- Manpages
- Info pages
- Default package information
- Supplementary package documents, if installed
- HOWTOs from the Linux Documentation Project.
- Debian-specific documentation (installation, administration, security, policy, and other manuals)
- RFCs (divided into several groups, for manageability / relevance)
- Resources such as the Linux Gazette
There's also a commandline utility, dwww, which will invoke your browser of choice. Point that at a terminal-based browser (w3m, lynx, links, elinks, etc.) and you can quickly scan through docs from your terminal.
For anything bigger than a screenful of lines or something that would need proper unit testing or something that other people who are likely to be less versed in shell scripting need to maintain, think twice about using shell scripts.
Unixoid machines can use anything as scripts and using a language that breathes the programming language designs of the 1970s is not the best choice.
<https://github.com/kward/shunit2>
I find it useful when most of your work is wiring other command line tools together. When that's most of your work, shell scripting feels like it has the best affordances.
Anything that gets too fancy beyond this, with anything that resembles an algorithm or computation, you'll feel real pain :) Bash doesn't even support floating point!
A typical task in my day job is extracting metadata from and transcoding large AV archives, I always use a bash for this because I'm mainly stringing together tools like ffmpeg or sox, making directories, tempfiles, creating chesksum manifests, etc.
Also I typically do this on large CPU machines with many cores, I can parallelize operations in one line using xargs and a bash function. Doing stuff like this in python is actually a real pain in comparison.
There's a library for Node called Zx (https://github.com/google/zx) that provides a more shell-liked interface for Node.
I've been working on something in Python, inspired by Zx and Xonsh (https://xon.sh/), with syntax like this:
Personally, the above would be hugely useful to me in reducing my temptation to write shell scripts, and significantly reducing the annoyance of doing these things in Python.Is this something you'd use? It's slow going for me because it basically involves writing a compiler and I have approximately zero idea of what I'm doing. Knowing that other people would find it useful helps me stay motivated to keep working on it! Or if someone who theoretically is the primary audience doesn't find it useful, it might mean that I should spend my energy on other things.
Honestly I might be the wrong audience though, I've spent many years shell scripting and I find the flexibility of using the command line tools so great I choose to use it first despite the inadequacies of bash. I wonder if users with my point of view would ever be tempted away.
Also the shell is an interactive repl, which to me is where a lot of the power lies. The number of times I've asked a junior things like what the total duration of a set of audio files is, or the average length or something. They often come back with a python script or worse a jupyter notebook and pandas. The answer can also often be found by combining things like grep and awk in a few seconds once you've become acquainted with the tools, I guess I do try and demonstrate / preach this where I can.
But bc (and dc) does, and shell authors deemed this was good enough.
And that's part of the problem with the "Unix philosophy"
I had to do this once. It was so slow, I decided to use integers in kind of fixed-point math where I just multiply all the floats by 100 and work with ints.
zsh supports float though
If you prefer rigid adherence to standards, use awk.
¹ https://github.com/clarity20/shellmath
² https://news.ycombinator.com/item?id=26250743
I tend towards python these days if I need more than 1 variable since at that point I'm probably in the weeds already.
If it's just a ton of commands in succession though, I leave it bash. I have a few scripts in bash and PowerShell for weird ffmpeg operations I generally don't want to remember, stuff like loudness normalisation and converting to specific formats with specific parameters using ffmpeg.
https://github.com/tdenniston/bish
EDIT: I didn't know of Bish, but from its code samples we seem to have converged on similar syntax :)
I do think it would be a really fun project. But is there really a practical use for it? Bash is portable but not that portable if you want to care about version compatibility. I understand that it's not always possible to install a different interpreted language like Perl or Python. Maybe the use case is for platforms where you can't install other language interpreters and can't cross-compile a compiled language? It seems like a very specific situation.
https://github.com/babashka/babashka
set -e
Which forces the script to exit on first error.
This line be in the bible as Genesis 1:1
That should be line two of every bash script.
I like JavaScript. Because I'm good enough at it to ignore the bad parts. If I get good enough at bash, will it get better?
Bash itself will not get better because you'll be stuck writing scripts targeting some ancient version to ensure wider compatibility.
It's still a powerful skill to develop. It's a sweet spot between a one-off shell pipeline and something that involves actual logic.
Bash scripts should not be used for their logic. Think of them as a way to document a series of commands that you do not want to manually type. That can be including long argument names, generating includes or classpaths, or generating log file names based on the current time.
A good rule of thumb is if your script involves any if-statement other than parameter validation or basic on-success-of-x-do-y, it probably should be in a real language.
You might want to be in the dash POSIX shell for raw speed and portability.
1. Pipefail is a bashism, and doesn't exist on Ubuntu's /bin/sh or Busybox /bin/sh (common on embedded Linux systems)
2. Pipefail invalidates some common shell patterns like "grep _something_ | xargs -r CMD".
Pipefail is nice in some circumstances for sure. But it's probably not great advice to automatically copypasta into the top of every shell-script.
I wasn't suggesting for a generic posix shell. It's explicitly for bash.
If you're going to dumb down to pure posix, it's even more painful as you can't even have local function variables.
> 2. Pipefail invalidates some common shell patterns like "grep _something_ | xargs -r CMD".
That type of situation is the exception, not the rule. The more common situation is "foo some-input.txt | bar" and you want it to fail if foo fails.
> Pipefail is nice in some circumstances for sure. But it's probably not great advice to automatically copypasta into the top of every shell-script.
I'd much rather have that type of failure than an inadvertent continuation of a script when something early in a pipeline failed.
The defaults of bash make more sense for terminal use than scripting.
Powershell is a bit further towards the scripting end of the spectrum- part of why it annoys people coming from bash.
Compiled languages would be at the scripting end.
function main {
}main
I only mention this because there are folks who don't like set -e.
Shell languages is the final layer of configuration and customization you apply to the setup, it should almost never be the foundation of what you build.
Especially for a company, unless you've got really, really skilled engineers on the scripts, it becomes a complete mess. And even then it can become almost impossible to grok.
The way we develop infrastructure automation, ci, and developer tools is binaries first and foremost, then plugins for our infrastructure (also binaries, or libraries) and then some zsh on top to provide a nice developer experience.
But we've had to heavily scale back our use of zsh, it simply turned into a ball of mud, like all software that isn't maintainable, upgrade-able, or simple enough to parse.
This isn't a dig on zsh, bash etc, we've just found out that we can't maintain zsh as a product to supply to our engineers, and as such have to correct.
¹ https://zsh.sourceforge.io/Doc/Release/Zsh-Modules.html#The-...
But this is cool too.
> Especially for a company, unless you've got really, really skilled engineers on the scripts, it becomes a complete mess. And even then it can become almost impossible to grok.
Dude. It's really not that hard a language. Is amateur shell script ugly? verbose? redundant? bug-riddled? Absolutely. But it works despite that, and has fewer dependencies than other solutions, so in practice, it's fine.
That said, use whatever you know. Wanna use Python? Use it. Wanna use a compiled language? Do it. Just make something. The biggest difficulties you will have in producing a product will not be the language.
You don't have to avoid shell just because people on HN (who've probably never read the man page) keep repeating that it's terrible. With Shellcheck, almost anyone can write good scripts now. If you can write a concise, bug-free script that works, then do it.
Turns out on one non-Linux operating systems, one of the utilities used for string manipulation, while POSIX compliant, behaved differently than the same utility in nearly every other operating system.
Wrong, wrong, wrong. Nearly everything you do in a shell script is an external dependency. And that's the biggest problem with shell scripts. The actual language is so narrowly scoped that you're dependent upon external utilities that are not so easy to upgrade.Count how many programs are in /bin. Those are your dependencies. Unlike e.g. python or ruby dependencies, you'll have a much harder time upgrading /bin. Having to fork a new process for trivial tasks is also one of the reasons why shell scripts are so slow.
Sure, bash has builtin versions of some of those utilities like [, but not every system has bash.
It helps having good integration/e2e tests, but they only go so far, customers/developers customize and mess up their systems all the time.
Which is why I've choose binaries instead, they are mostly self-contained (except for various things about gnu and musl), and in general just rely on a higher level of built in functionality (because of the statically linked nature).
It is also quite bad that we've got to vet bash installation scripts before running them, as you may be required to run them as root. IMO, this isn't a huge issue for me personally, as we already are doing this for a lot of stuff, homebrew, linux package managers etc. We generally run a lot of untrusted code on our machines.
> Dude. It's really not that hard a language. Is amateur shell script ugly? verbose? redundant? bug-riddled? Absolutely. But it works despite that, and has fewer dependencies than other solutions, so in practice, it's fine.
Bash, zsh whatever, depend heavily on what is installed on the os, and what type of flavor of base tools are provided. Need base64 in a certain way, don't forget that line wrapping is different between bsd and linux. (a pain in the butt to test too).
Need to download something, better hope you're not running on a busybox or something with curl, or only have wget installed, now you gotta build support for both.
A lot of scripts, especially installation scripts are a product of blood, sweat and tears, mostly tears tho. Each line can be interpreted in a multitude of ways, under different conditions and versions of bash, os and so on, and the script will have to be robust enough for that.
> That said, use whatever you know. Wanna use Python? Use it. Wanna use a compiled language? Do it. Just make something. The biggest difficulties you will have in producing a product will not be the language.
It may not be the biggest difficulty, but it certainly is something we should consider. We've chosen our values, and shell scripts doesn't align with those, I'd rather not fight my environment every step of our development journey.
I am not building a new product, I am maintaining and expanding a product, as such I am heavily biased towards getting as few support tickets as humanly possible, while building an awesome experience. We are also mostly software engineers, and not sysadmins, as such we prefer compiled binaries. We then choose to distribute them through curated and secure channels.
We get the benefit that we can actually test parts of our tools in isolation, benchmark them if necessary, and version our stuff in a nice concise way. If I want to run our onboarding installation, I can simply fetch our current list of binaries expected for developers and run it, because I know they don't depend on external tools from the environment. This is a lot of work, but in a large development organization, this means a lot.
> You don't have to avoid shell just because people on HN (who've probably never read the man page) keep repeating that it's terrible. With Shellcheck, almost anyone can write good scripts now. If you can write a concise, bug-free script that works, then do it.
Just to reiterate, I am not digging on shell scripts, and I don't necessarily think they're terrible, they have their time and place, and for us, it is customization and extensibility. We setup our tools such that they work awesome on the cli, and can be composed however, developers like, but that is up to them, and how they want to form their workflows.
We've seen that developers really struggle to maintain what was previously built (myself included), so now we've got a more homogeneous tech stack, which comes with its own trade offs, but whose values align more with our engineering culture.
I remember, I used to keep a diary with landline phone numbers of relatives etc. I used to remember at least a dozen of those numbers for emergency. With mobile and digital contacts, now that number is down to just 2.
Wonder, what the programming is going to look like in 10 yrs. If no-one dedicates so much of their time and effort to document, the programmers then would feel helpless. right?
1. Or at that point, LLMs will be intelligent enough to read the source code and figure out the commands.
2. They can generate data by running commands, and train themselves using this simulated data.
What other scenarios will occur once we stop generating data using human effort because we all are so dependent on LLMs to generate those commands?
The solution is, do as little with shell scripts as possible to bootstrap a more workable environment.
If you have even the slightest hesitation regarding whether the task in front of you would be better done in a real programming language, the answer is do it in a real programming language.
https://github.com/zbm-dev/zfsbootmenu/releases/tag/v1.12.0
Don't fall in the trap of thinking things have to be written entirely in bash; it's okay to use other tools to help fill in the gaps.