As much as people use the shell, it’s regrettable how few have actually read the manual. Anything you use for extended periods of time on a regular basis is worth sitting down and reading the manual for.
With a manual as labyrinthine and disorganized as Bash's, in my mind they are forgiven for not being able to struggle through it. It makes even the Zsh manual seem delightfully lucid by comparison.
The Bash manual page is concise because it is a Unix manual page. Unix manual pages are supposed to be concise, because they are meant to be reference documents, not tutorials. In the GNU project, this is what the Info documentation is for. However, the bash project has elected not to maintain such a document, so the above documents from TLDP (The Linux Documentation Project) are the next best thing.
The bash man page is simply way too long, and IMHO not organized very clearly. And man pages don't even have a table of contents, never mind hyperlinks. The info version is naturally better in those respects, though.
I think that's a testament to how successful shell scripting is. Somehow you can be really productive and write useful tools without really ever diving deep into the docs.
I've written literally thousands of lines of shell scripts (maybe even 10k+) across dozens of scripts over the years and I've only really looked at the docs a couple of times. Of course I'm Googling various syntax things that are humanly impossible to remember, but I never once really sat there and combed through all of the docs or even read a book on Bash.
You can't do that with most programming languages.
Some of us actually have to USE shell scripts written by other people who like to brag about only looking at the doc a couple of times, who just copy and paste things they found in outdated pages on google into hodge-podges of frankenscripts, and who then run them as root at regular intervals on unattended servers.
It may be care free, fun, and effortless for you, but try being on the receiving end of that behavior some time.
RTFM! And if the manual sucks (like the Bash manual), then learn to use another language.
Instead of learning to write shell scripts better, people should learn NOT to write shell scripts, and learn a decent scripting language instead.
You will find that most decent scripting languages like Python are actually a hell of a lot easier to learn and program and maintain than shell scripts.
Just to add, I generally prefer shell scripts over Python because it is supported in all my use-cases out-of-the-box.
However, I recently discovered the power of Python’s Fabric scripts, and holy cow... the ability to run functions on multiple computers at the same time based on roles is amazing (even if the “join” syntax is cumbersome as hell).
I think in general, simple system things should be written as scripts. The moment you realize “this is going to take a LOT of scripting” it’s time to look at Python because chances are there’s a module for that.
If you like writing things that you can execute on multiple machines take a look at Ansible. It's a Python based configuration management tool where a majority of what you write is YAML and Jinja 2 templates.
But on that note, a majority of my current shell scripts are being used on my dev box to improve my day to day. Things like this: https://github.com/nickjj/invoice
Have you ever looked through the Linux source code to see how many lines of code and instructions and data structures and libraries and memory allocations and mappings and system calls and context switches are required to search a path for an executable and fork off and run a Unix process?
It really breaks my heart to see stuff like this pointlessly wasting electricity by running on a computer that has its own "mul" instruction -- like forking off a "bc" and sending it a stream of decimal digits over a pipe just to multiply a number by 100:
Don't tell me you prefer writing shell scripts that do simple arithmetic because it's more convenient than using Python.
Calling out to awk to do trivial string manipulation may impress you as "amazing" after you've hit the wall trying to convert a string to decimal hours with bash and bc, but do you really enjoy all that noisy cryptic bash syntax with all those carefully orchestrated backslashes, double dollar signs, triple angled brackets, echos and pipes and quotes, nested parens and braces, and mysteriously abbreviated dash-letter parameters, to typing "100 * default_hourly"?
At least Python's "*" operator doesn't require a "-l" parameter to figure out how to multiply a number by 100.
The process id is not your Linux high score. There's no need to run it up by forking so many processes all the time.
> Don't tell me you prefer writing shell scripts that do simple arithmetic because it's more convenient than using Python.
That's not a fair comparison. The code you pasted isn't just doing the calculation. It's dynamically generating a human readable help menu that shows both the input and output of what the script will do. That requires quite a bit more characters and escaping.
The "real" calculation to get the hourly is:
amount="$(echo "${total_hours} * ${hourly_rate}" | bc -l)"
Seems pretty straight forward to me. Plus, that's a really cherry picked example in Python's favor. You should evaluate the whole script's purpose. It's mostly just reading and filtering text output from a file and then performing basic math on the filtered text. Seems like a textbook case to use Bash IMO.
Again, look at the Linux kernel and userland source code and give me a ballpark estimate of how many instructions it requires to multiply a number by 100 by forking off bc or sed, instead of executing a single "mul" instruction.
And Python code doesn't require a page of comments to explain why it has to go to so much effort and work around so many pitfalls just to convert a couple of strings to decimal hours.
Why, Python even has named function parameters -- now THAT is TRULY "amazing".
And in the number of keystrokes it took you to write the following comments (and the number of trials and errors it took you to finally get it right), you could have imported a hell of a lot of powerful Python modules, and read a lot of the interactive built-in help documentation about their APIs.
Your time would be much better spent getting used to using Python rather than sed.
to_decimal_hours () {
local hours="${1}"
local minutes="${2}"
# [h: ] uses the "h", ":" or " " character as a split delimiter.
#
# If the first split argument ends with "m", then we're not dealing with
# hours at all, only minutes, so now $1 becomes minutes and we divide by
# 60 because you can't have more than 60 minutes in 1 hour.
#
# Otherwise, we have both hours ($1) and minutes ($2). The hours can be
# taken as is, and like before the minutes get divided by 60.
#
# It's also important to send both the hours and minutes in without a
# space as input, otherwise values like "30m" will end up being "30m "
# which makes $1 no longer end with an "m" and it will incorrectly report
# back 30.00 instead of 0.5.
#
# awk will automatically remove the "h" and "m" characters when it comes
# time to do the addition.
#
# Yep, awk is amazing. I'm just getting used to using it!
awk -F '[h: ]' '{ if ($1 ~ /m$/) printf("%.2f\n", ($1 / 60));
else printf("%.2f\n", $1 + ($2 / 60)) }' <<< "${hours}${minutes}"
}
Look, if you want to sit there and rip into a stranger on the internet for open sourcing a Bash script (on their free time) then by all means knock yourself out.
For the record I've written more Python than I have Bash (by a substantial amount). I wrote that script in Bash because it took literally 3 minutes to prototype it on the command line with grep, cut and bc so I spent the extra hour turning it into its current form. It does exactly what I want it to do (nothing less, nothing more).
Also the code is MIT licensed. Free free to fork it, rewrite it in Python, test it on 50+ real invoices, write extensive test coverage, write ~1500+ words of documentation and then make a ~20 minute video showing to use it.
You're the one who offered up that code as an example in the first place, to prove your point about bash being really great for people who don't want to bother reading documentation.
Being open source doesn't magically make your code great or efficient or bug-free or easy and irresistible for millions of eyeballs to examine and audit (in spite of what ESR's fallacious "Linux's Law" might claim), or obligate anyone to complement it or use it or take it and run with it.
If you make your code open source, then post links to it in public forums, right after bragging about only reading the language documentation a couple of times, you'd better get over being thin skinned about criticism.
I'm not criticizing your code, or your decision to open source it, or which license you chose, or your pride in only reading the documentation twice, or your decision to cherry pick and share that script as the best example you could come up with of bash's superiority. I'm criticizing your decision to write the code in bash instead of Python.
Can you at least please answer this one simple question: Name one thing about that code that is easier in bash than in Python?
>I've written literally thousands of lines of shell scripts (maybe even 10k+) across dozens of scripts over the years and I've only really looked at the docs a couple of times. Of course I'm Googling various syntax things that are humanly impossible to remember, but I never once really sat there and combed through all of the docs or even read a book on Bash.
At least you admit 1) you've only really looked at the docs a couple of times, and 2) bash syntax things are humanly impossible to remember.
Have you ever written any 10k+ line bash scripts? No, because it doesn't scale (and machine generated configure scripts don't count). I have a 16,078 line Python script in my emacs buffer right now that I wrote over more than a decade (which is just one facet of a big system written in many other languages).
My point (which is just a corollary of JWZ's Law of Software Envelopment and Greenspun's Tenth Rule) is that programs always grow over time, and you never realize how big and complex your programs are going to grow or how far they're going to evolve when you first write them.
So never assume you can get away with using a language that's so lame it has to call out to two other different languages just to perform simple string manipulation and arithmetic.
Good grief, even TCL can perform string manipulation and arithmetic without forking a bunch of new processes.
You should invest more time using languages with docs you can actually read, and with syntax you can actually remember. You might like it.
> You're the one who offered up that code as an example in the first place, to prove your point about bash being really great for people who don't want to bother reading documentation.
I posted a link to it next to the sentence that talked about the types of tools I use it for, such as things on my dev box to help me in my day to day.
> Have you ever written any 10k+ line bash scripts? No, because it doesn't scale (and machine generated configure scripts don't count)
I've written a bunch of 1k+ line scripts. They aren't a problem to maintain and a number of them are being actively used and maintained by other people. These are for real production systems too. Some are single files, others are broken up into multiple files.
I haven't personally written a 10k+ line script (the need for that has never happened), but I've seen a number of multiple thousand line scripts out in the wild that are very well maintained and are part of mission critical systems like setting up a PKI realm for a bunch of servers. That PKI script was something like 4,800 total lines of code but it being 10k+ wouldn't have made a difference for the worse.
Shellcheck helps a lot with writing pretty well formed shell scripts. It's a really good linter.
I just wanted to be clear that I'm not just making this stuff up, and you really did write what I quoted (claiming to have written 10k+ lines of shell scripts, that you only really looked at the docs a couple of times, and that bash syntax is humanly impossible to remember), but you ninja-edited it out later, after I quoted what you originally wrote:
>I've written literally thousands of lines of shell scripts (maybe even 10k+) across dozens of scripts over the years and I've only really looked at the docs a couple of times. Of course I'm Googling various syntax things that are humanly impossible to remember, but I never once really sat there and combed through all of the docs or even read a book on Bash.
Only really looking at the docs a couple times isn't something to brag about, and doing that doesn't make bash good.
Nor is it good for languages to be designed so you don't have to read the manual (or else you get COBOL or HyperTalk, and then you still have to read the manual).
Nor is it good for languages to have syntax things that are humanly impossible to remember.
All those claims you deleted actually support my point about bash being terribly inferior to Python (whose syntax is extremely simple and easy to learn and remember and most importantly read, and it even has built-in interactive "help" documentation), which is why I quoted what you originally said in the first place.
Again, what is it about that script you offered as an example that is easier, more elegant, or more maintainable and understandable to do in bash than in Python?
If you don't have an answer to that, can you show me any other bash code that was better off written in bash than in Python? And explain how it's better, please.
The current year is 2019. It's not as if every modern Unix based operating system didn't already include or even require at least one goddamned Python ecosystem.
Nobody's running such an old version of Ultrix on their MicroVAX with a big yellow thick wire Ethernet cable and vampire tap and static /etc/hosts file derived from HOSTS.TXT downloaded weekly from SRI-NIC.ARPA without checking certificates for man-in-the-middle attacks and running unencrypted rlogin and telnet demons instead of ssh any more, that they can't easily and securely update Python and install modules.
Also: You think the Python "ecosystem" is goddamned? Remember that bash by its very nature depends on a terribly goddamned inconsistent "ecosystem" consisting of all the essential utilities it requires to do the most elementary of tasks, including awk, sed, dc, cut, grep, yes, and many others, whose locations and apis and features and parameters and behavior and performance vary widely from platform to platform. Bash is much more tightly coupled with the operating system distribution that Python.
While most of those essential features (like coughcough string manipulation and arithmetic rolls eyes, among others) are built into Python or available as standard plug-in modules (like os, system, threading, multiprocessing, pexpect, mmap, readline, Beautiful Soup, SQL Alchemy, json, yaml, etc) that are identical across all platforms.
And most of those Python modules are vastly more efficient than their bash counterparts, because they don't require forking separate processes instead of making local procedure calls, communicating over 1-dimensional pipes instead of shared memory, serializing and deserializing text parameters and data instead of passing pointers to live data structures.
What good is a scripting language these days that doesn't have built-in json and yaml parsers (not to mention a nice way to parse html and talk to databases)? I see so many half-assed bash scripts desperately trying to parse field separated values and process output with regular expressions, grep, cut, awk and sed, by forking off hundreds of ridiculously inefficient processes.
Yes I know there are xpath/jsonpath utilities you can execute from bash scripts that look up elements and keys in XML documents and json dictionaries -- that's my whole point, that there NEEDS to be something like that! God forbid you need to process 150 million records.
And as to what the grandparent poster actually said: I explained in the cousin posting below that I quoted and addressed what he ORIGINALLY said, and then he NINJA-EDITED it after I replied, to substantially change the meaning and remove several ridiculous, indefensible statements.
Unless he's just gaslighting and actually MEANT for you and others to mistakenly believe that I was putting words in his mouth (in which case we should expect his radio silence to continue), I'm still waiting for him to acknowledge that fact, and finally reply to the simple straightforward question that he kept avoiding:
Name one thing about that script that's easier to do in bash than in Python.
But I'm not holding my breath. Do you care to chime in with an answer instead?
>People who aren't stupid and know other languages still whip out little shell scripts; we should try to understand why that is.
Well the grandparent's excuse (which he deleted) for whipping out little shell scripts was that he "only really looked at the docs a couple of times". I don't think that's a legitimate excuse, myself, and apparently neither did he, so he ninja-edited it out, instead of admitting it wasn't a good defense of bash to brag about only reading the manual twice.
I'm not really arguing for bash or awk as a surrogate for Python. Just wanted to remark on some of these comments. (Also, I'm not the person you replied to, FWIW.)
> The current year is 2019. It's not as if every modern Unix based operating system didn't already include or even require at least one goddamned Python ecosystem.
If you're just talking about Linux, this is kind of true (Fedora's dnf and yum before it are built on Python). But AFAIK Debian/Ubuntu apt-get is written in C/C++ and does not require Python.
Non-Linux unixes like, say, FreeBSD, do not have Python in the base distribution (nor bash). It's available in ports, which might be what you're getting at.
That said, my experience has been that non-Linux "posix" Python is a second class citizen to Python on Linux. Just for example, signal/thread semantics happen to align in a certain nonspecified-by-POSIX way on Linux and differently on FreeBSD. IIRC, FreeBSD delivers nonspecific process signals (e.g., external INT or TERM) to the most recent thread, whereas Linux delivers signals to the oldest thread. Or vice versa, but Python has its own weird signal semantics it tries to impose on top of this that works better on Linux's model than FreeBSD.
As a second example, Python fork+exec invokes close() on every possible integer between N and the file descriptor limit. This is typically tolerable for naive python programs on Linux because the default fd limit is usually 1024. FreeBSD scales the default fd limit with memory, often resulting in a much larger limit (e.g., ulimit -n = 939258 on this 32GB system). For years we have tried to get Python to adopt the closefrom(2) API, which accomplishes the same goal much more cheaply. But because most Python developers are on Linux, they simply don't feel the pain of a subprocess.Popen() taking noticeable fractions of a second.
> Also: You think the Python "ecosystem" is goddamned? Remember that bash by its very nature depends on a terribly goddamned inconsistent "ecosystem" consisting of all the essential utilities it requires to do the most elementary of tasks, including awk, sed, dc, cut, grep, yes, and many others
No disagreement there.
> What good is a scripting language these days that doesn't have built-in json and yaml parsers (not to mention a nice way to parse html and talk to databases)?
jq is a wonderful tool for parsing and manipulating JSON in sh, though it has its limitations. I certainly wouldn't attempt to parse HTML in (ba)sh. I don't encounter YAML very often.
> Name one thing about that script that's easier to do in bash than in Python.
"ls -l"? There are lots of quick and sloppy path manipulation things that are easier to bang out in shell than in Python -- that's why most people use (ba)sh et al as interactive shells, instead of python.
Anyway, Python is powerful, and quite possibly more consistent platform to platform than bash. No disagreement there. I'd also argue it's a much better tool for writing programs that handle adversarial input than bash is. The ways Python differs between platforms are mostly areas bash doesn't attempt to cover at all. And the datastructures available in Python are much more powerful than in bash.
Knowing the basics of shell scripting is not optional. As a dev on a unix system, it's just a question of time before you have to read and tweak other people's scripts.
Knowing how to change a diaper when you're a parent is not optional. But you shouldn't shit your own pants when you don't absolutely need to.
You may think you're clever enough to sneak out a little one-liner that nobody will notice, but all too often you underestimate the task, and it blossoms into a big messy shart script that other people have to deal with.
Around 1992 I wrote a CGI at Cygnus for reporting the top high priority and important bugs, who was responsible and how old they were. It was used for years, and people would load that page several times a day as they did not want to have their name at the top of the list!
"the manual" is a form of communication for humans. Something nix has been de-prioritizing for thirty years! By experience, the nix culture has been "if you do not understand this, you should not be here" and a low-level hostility towards questions and learning, comes from some science gestalt in the 1800's German or something.
*nix is friendly, it is just selective in who its friends are" .. is a dot-signature from that era
A manual is a reading experience, a working reference system, and relies on the visual context it is presented in. There are vast differences in the quantitative and qualitative contents of manuals, on the same content! The BASH man page is .. improving ?
Specifically compare an alphabetized list of every option, including "change the preferred shortcut" and "let the sea-water in, thereby killing everyone if you are underwater" .. are right next to each other .. to modern manuals in say, the Python culture. Specifically compare say, huge amounts of text supplied for some obscure option that is not at all needed, to a terse one-liner for something crucial that is used everyday.
The parent comment cheerfully "blames the victim" with implied guilt for "not reading the manual" as if more time and effort on the part of the user, is the answer to the communication problem posed with such a dense and subtle realm.
Last thing in this rant is, that a small interesting article on something specific is an antidote to the largely unsolvable challenge here, so good on that.
I don't think you can generalize that. E.g. some of the BSDs pride themselves in great documentation. The shell manuals are not just alphabetical lists, but sort-of guides to their programming languages. etc.
When I had to start dealing with bash a lot about 5 years ago I consulted "Greg's Bash Wiki" at https://mywiki.wooledge.org/BashGuide and it pretty comprehensively treats best practices for writing bash scripts. I often go back to consult the guide.
> Something nix has been de-prioritizing for thirty years! By experience, the nix culture has been "if you do not understand this, you should not be here"
This is an open-source thing. Hardly anyone wants to write documentation that isn’t paid to do so. There’s no glory in it. I first learned Unix on SunOS/Solaris and the man pages were excellent. You could start with “man intro” and learn your way around the entire system from there.
BTW, if you think you’re any good at documentation, open-source needs your help. It’s a great way to contribute.
> Hardly anyone wants to write documentation that isn’t paid to do so. There’s no glory in it.
There actually is glory in it. Many (granted, not all) popular, celebrated open source projects got that way by having a great user experience with intuitive, accessible docs.
I actually think that most programmers (even many very good ones) are simply bad at writing docs. They actually can't do it well. Or it would be a monumental effort for them to do it well.
> Is writing docs really that different a skill from writing readable code?
It is correlated (though probably not perfectly) with writing readable code. But that is a different skill from writing useful, correct software. That is, you can write useful, correct software whose code is difficult for others to understand and maintain.
Bash is painful and awful, IMO. The last thing I want to do (if I were even capable of comprehending the man pages) is really learn it and internalize its awfulness (same with, say, Vim); I want to be able to do what I want to with some syntax quickly glued on from StackOverflow.
Actually, I don't want to do that either.
What I really want to do is use a Python script instead. :3
But using Python is very painful as well for shell use-case. Do you really want to open files, subprocesses, and create fifo manually instead of typing "cmd1 | cmd2 > output"?
Then you have Xonsh, but in my opinion, switching between the two contexts, shell and python, at each part of a command was too difficult. I was totally lost even at writing very basic commands.
The bash manpage ... runs longish. 110 pages as 'man bash | pr". I do read it, frequently, though it's hard to assimilate in total andI constantly find new or unexpected things (after 20+ years).
Checking zsh by comparison: 6 pages, though a long "SEE ALSO" list with 10 zsh-specific entries, totalling 374 pages in all:
I specified both result and methodology. Yes, a different method gives a different result. Using pr allows rapid verification of numerous manpages as in the zsh manpages example shown, for validation.
I'm aware that generating postscript is possible, and results in fewer pages. But that doesn't readily permit comparisons with simple shell tools.
On the flip side, I added this to my dot files quite a while ago but I didn't originally find out about it from the docs.
I ran into a situation where I thought "wow it would have been useful if I knew when this command was run" while going through a massive amount of history on a foreign server.
So then I Googled for "add timestamp to bash history" and immediately found it.
Interesting list (not very surprising to me personally though saw $REPLY mentioned anywhere after a long time). One that surprised me recently (in a how-come-I-didn't-know-about-it-for-so-long way) was $PIPESTATUS e.g. if you are running a pipeline "cmd1 | cmd 2" and would like to know the return status of cmd1, you can use ${PIPESTATUS[0]} to get that. Useful, e.g. when you are tee-ing the output etc.
I would recommend against using $REPLY, for two related reasons:
1. It’s confusing to do a plain “read” and later have a magic $REPLY variable show up, seemingly from nowhere. It has shades of Perl, and I mean nothing positive with that statement.
2. Names are good. By setting a name to the thing which is read, you make the later code easier to read. It’s easier to understand “echo $filename” than “echo $REPLY”, so if you’re reading filenames, do “read filename”. (Or, rather, you probably want to do “read -r filename”, but that’s a different subject.)
I don't do a lot of bash programming, but I imagine using $REPLY also increases the danger of having the value overwritten before you use it as your code changes.
Now that you mentioned it, I never hit the case where I would read two different times in the same function/subshell scope and would never have considered this a problem. But maybe I'm just preempting this halfway subconciously applying meticulous scoping?
Yeah I don't use it in practice - I remembered it because it was part of my bash trivia collection when I started learning more about shell scripts many years go. Named variables are obviously a cleaner choice.
Why do you find $PIPESTATUS useful? It is in theory, but in practice I've never found a situation where I wouldn't just want to fail the whole pipe if one part of it failed, which is what "set -o pipefail" is for.
It's good practice to start your scripts with "set -euo pipefail" and then list exceptions to that scoped with "set +..." as needed. That's the "unofficial bash strict mode"[1].
I don't use the IFS recommendation in that article. I prefer to just keep one set of shell inanities in my head, and being used to the non-default IFS behavior doesn't help when debugging other people's shellscripts.
Because sometimes you want to take some action in case of non-zero $PIPESTATUS instead of just failing everything immediately. This provides full control.
>4) SHLVL: [...] This can be very useful in scripts where you’re not sure whether you should exit or not, or keeping track of where you are in a nest of scripts.
If you're writing recursive shell scripts where you're not sure whether you should exit or not, you shouldn't be writing shell scripts.
Leave it to bash to go out of its way to make it easier to write scripts that break encapsulation and implement spooky mysterious action-at-a-distance by implicitly depending on how they were run, and purposefully behave differently whether they're invoked from a command line or another script, and are so confused about what they should do that they have to make guesses about whether or not to exit.
What about a simple "die" function that echoes a message and return an error code? Could be used both in scripts and in the interactive shell. It would use "exit" when SHLVL is more than 1, and "return" otherwise.
It's not about the best way to behave differently at runtime and sometimes call exit depending on how the script was invoked, it's about the idea that a scripting language would SUPPORT and ENCOURAGE writing that kind of script, with a special magic built-in variable and a section in the already-complex documentation encouraging its use.
Scripts should not behave differently depending on if they were invoked from other scripts, and modular reusable libraries should never take it upon themselves to call "exit" when there's an error.
The way to control the flow of recursive functions is by passing explicit parameters (like the recursion depth, or a data structure to walk, or an error callback), not reflecting on the depth of the runtime stack or looking at how the function was called.
If bash is too weak to handle errors, or exceptions, or recursion, or passing functional callback parameters properly, or even file names with spaces in them, then use a better scripting language, don't just throw up your hands and exit.
> This is similar to the problems seen when the dot folder is not included in the more familiar PATH variable.
Is this implying that you should put `.` on PATH? I've always heard that to be a security problem.
What's interesting is that so many of these variables are incredibly tersely named. What's the reasoning behind not using a full name for them to make Bash actually readable? Surely not the minor performance scrape?
Yes, with `.` in the PATH you have potential security issues: If your working directory happens to be world-writable (i.e. while doing cleanup in /tmp) an attacker could put evil replacements into your PATH. "cd /tmp; sudo rm some_file" might execute /tmp/sudo and do really evil stuff.
Sure, on an otherwise 100% secure system accessed only by trusted users there is no problem, but not having . in PATH is a trivial mitigation.
> Sure, on an otherwise 100% secure system accessed only by trusted users there is no problem, but not having . in PATH is a trivial mitigation.
All systems are secure and sufficiently hardened until a botnet demonstrates otherwise, aren't they? And a botnet is the friendly hostile takeover because it'll just deploy a bot or a crypto miner and might even patch your system.
Just don't put . on your PATH. And if you do, append it - <PATH=$PATH:.>. This way, /usr/bin/foo will win against ./foo.
Having a dot at the end of the path also trivially mitigates that particular scenario while still providing the convenience of being able to run an executable without having to always type dot-slash. I guess the reasons not to put the dot at the end are mainly about common typo attacks, but I’m partial to dot at the end.
I use it in combination with a trap on DEBUG signal to compute information about each interactive command (start/stop timestamps, type, working dir, return code, and more) and save it in a big log file. I can then transform the data into pretty charts!
It's also used by the `z` command line tool to maintain its directory "frecency" list.
If you set $CDPATH, then for goodness’ sake don’t export it. It changes the behaviour of the cd command to make it output the absolute path of the directory changed to, which breaks a common shell-scripting pattern for converting relative directory paths to absolute paths, viz:
absolute=$(cd "$relative" && pwd)
Conversely, if you’re writing a bash script and it needs to be robust against people who do export CDPATH, you can do it like this instead:
That’s a very Linux-centric view. Neither of these works on BSD. It’s probably fine if you’re writing scripts that only need to run on Linux, but I wouldn’t call it the “right method”.
Sure, readlink exists. And it lets you read symlinks. But it can’t be used to convert relative to absolute directory paths, which I thought was the use you were proposing.
> if the -f option is specified, the output is canonicalized by following every symlink in every component of the given path recursively. readlink will resolve both absolute and relative paths, and return the absolute pathname corresponding to file. In this case, the argument does not need to be a symbolic link.
This is the case for *BSD, GNU/Linux, and Irix. If macOS would prioritize being consistent with the other Nix OSs, it would be supported too, but alas Apple doesn’t.
`$()` `cd` and `pwd -P` are all part of the POSIX spec. While that doesn't guarantee they're implemented in all shells, it's a lot more likely than things like readlink which are known to not be portable with the flags required to resolve a symlink.
`readlink -f` while not POSIX proper, works on FreeBSD, OpenBSD, NetBSD, and most Linux distros. If you really need to support a wider range than that, reality says you will likely have conditional OS checks in your script anyway, so I’m not sure your point other than being pedantic?
`readlink -f` on GNU canonicalises the path.
`readlink -f` on a *BSD is for formatting the output
`readlink -f` on macOS throws an error because it's an supported option.
> if the -f option is specified, the output is canonicalized by following every symlink in every component of the given path recursively. readlink will resolve both absolute and relative paths, and return the absolute pathname corresponding to file. In this case, the argument does not need to be a symbolic link.
My problem with bash is that things are not very discoverable. There is so much cool stuff you find out even after ten years of using it daily. I wonder if there is a way to have all its features in a less obscure language.
Could that be a property of command line things in general? Is that why GUIs are so popular with non-professionals that don't want to take the time to learn to use a system properly? There is so much in terms of data manipulation that you can do with basic tools, but people will open op Excel to find a graphical button they need instead of using, I don't know, maybe `man -K` or `apt search` to find the right tool.
For years I've been trying to figure out what exactly the difference is. Terminals are seen as difficult and outdated, yet the more nerdy you get, the more likely it is that you end up using (and the more time you will spend in) a terminal. There has to be a reason terminals haven't replaced GUIs altogether, and GUIs haven't replaced terminals altogether, despite the near-complete overlap in things you can do with them. The only thing that really needs a mouse/touchscreen is when you have something spatial like photo editing, but conversely, the extreme end of the power user tools almost always use a terminal and can't really work with a GUI.
Possible. I would say that the shell is even worse than most command line tools. I have started using PowerShell in my Mac and I find it easier to find stuff there then bash. As soon as I don’t use bash for a month I forget all the little obscure tricks.
WARNING: this page has a dangerous typo. do not put '.' in your path.
"This is similar to the confusion I felt when I realised the dot folder was not included in my more familiar PATH variable… but you should [not] do that in the PATH variable because you can get tricked into running a ‘fake’ command from some downloaded code."
Today, the threat model for a personal computer makes this a lot less important: to exploit this someone needs to be able to create a file on your computer and set its mode to +x: but if someone has that kind of access to one’s personal computer, it’s already game over.
Things Are Going to Get Lot Worse Before Things Get Worse
“I liken starting one’s computing career with Unix, say as an undergraduate, to being born in East Africa. It is intolerably hot, your body is covered with lice and flies, you are malnourished and you suffer from numerous curable diseases. But, as far as young East Africans can tell, this is simply the natural condition and they live within it. By the time they find out differently, it is too late. They already think that the writing of shell scripts is a natural act.” -Ken Pier, Xerox PARC
Whitelisting is safer than blacklisting. Mount /home noexec, mount ~/bin exec. If you're going to go this route.
I don't go this far yet, but do strongly recommend nodev and nosuid except where absolutely needed. This is also a strong argument for multiple partitions rather than single-partition formatting.
This is really interesting. I mount ~/Downloads anyways, because I put it on a nfs mount, but I’ve never considered some scheme like what you’re talking about. Maybe I’ll go for this on my next Linux machine.
Also, there’s a better argument for not putting . in $PATH: if your $PATH is relatively static, you have less to think about when looking through your shell’s history for a particular command: if . is on the PATH, you always have to wonder whether a particular history entry is running a system-wide command or a command in $PWD
It's too late, the damage is irreparable. By now, hundreds of people have probably already put . in their path, mounted their repos and /home noexec nodev and nosuid, and mounted a separate file system on ~/Downloads, just to spite you. ;) Or were you trying to trick them into doing that all along, so you could hack into their systems? I can never tell with you slippery cyber security types...
A simple search gave me the apparent original source for the quote; an article titled Bash — The GNU Shell, first published in the Linux Journal in 1994:
I have to say that the LINENO variable does not have the expected behavior in process substitutions, especially when using multi-line commands (with backslashes at the end, or using multi-line strings).
LINENO will use the last line number of a multi-line "$(...)" command, offsetting all the commands' line numbers within the substitution. Since multi-line strings are concatenated by Bash internally, LINENO will only increase by one for those.
You end up with line numbers greater than the number of lines in your file, or overlapping with empty lines or comments. In this matter, Zsh's LINENO is way better as it will have the expected value (it will always be the line where the command was started, not an offset).
I find it surprising that the fellow that coined the word POSIX, agreed to censoring the POSIX_ME_HARDER variable.
It's also surprising that the first version of bash committed source code suicide via a bug in globbing, which makes it's name seem more poignant, and was why there was a file named "-i" in it's source tree for many years.
124 comments
[ 1.2 ms ] story [ 117 ms ] threadI often describe my bash book as an attempt to carefully demystify the man page so that you can approach it with confidence.
http://tldp.org/guides.html#abs
Beginners might prefer the Bash Guide for Beginners:
http://tldp.org/guides.html#bbg
The Bash manual page is concise because it is a Unix manual page. Unix manual pages are supposed to be concise, because they are meant to be reference documents, not tutorials. In the GNU project, this is what the Info documentation is for. However, the bash project has elected not to maintain such a document, so the above documents from TLDP (The Linux Documentation Project) are the next best thing.
https://www.worldcat.org/title/from-bash-to-zshell-conquerin...
https://www.worldcat.org/title/learning-the-unix-operating-s...
I've written literally thousands of lines of shell scripts (maybe even 10k+) across dozens of scripts over the years and I've only really looked at the docs a couple of times. Of course I'm Googling various syntax things that are humanly impossible to remember, but I never once really sat there and combed through all of the docs or even read a book on Bash.
You can't do that with most programming languages.
It may be care free, fun, and effortless for you, but try being on the receiving end of that behavior some time.
RTFM! And if the manual sucks (like the Bash manual), then learn to use another language.
Instead of learning to write shell scripts better, people should learn NOT to write shell scripts, and learn a decent scripting language instead.
You will find that most decent scripting languages like Python are actually a hell of a lot easier to learn and program and maintain than shell scripts.
However, I recently discovered the power of Python’s Fabric scripts, and holy cow... the ability to run functions on multiple computers at the same time based on roles is amazing (even if the “join” syntax is cumbersome as hell).
I think in general, simple system things should be written as scripts. The moment you realize “this is going to take a LOT of scripting” it’s time to look at Python because chances are there’s a module for that.
But on that note, a majority of my current shell scripts are being used on my dev box to improve my day to day. Things like this: https://github.com/nickjj/invoice
It really breaks my heart to see stuff like this pointlessly wasting electricity by running on a computer that has its own "mul" instruction -- like forking off a "bc" and sending it a stream of decimal digits over a pipe just to multiply a number by 100:
Don't tell me you prefer writing shell scripts that do simple arithmetic because it's more convenient than using Python.Calling out to awk to do trivial string manipulation may impress you as "amazing" after you've hit the wall trying to convert a string to decimal hours with bash and bc, but do you really enjoy all that noisy cryptic bash syntax with all those carefully orchestrated backslashes, double dollar signs, triple angled brackets, echos and pipes and quotes, nested parens and braces, and mysteriously abbreviated dash-letter parameters, to typing "100 * default_hourly"?
At least Python's "*" operator doesn't require a "-l" parameter to figure out how to multiply a number by 100.
The process id is not your Linux high score. There's no need to run it up by forking so many processes all the time.
> Don't tell me you prefer writing shell scripts that do simple arithmetic because it's more convenient than using Python.
That's not a fair comparison. The code you pasted isn't just doing the calculation. It's dynamically generating a human readable help menu that shows both the input and output of what the script will do. That requires quite a bit more characters and escaping.
The "real" calculation to get the hourly is:
Seems pretty straight forward to me. Plus, that's a really cherry picked example in Python's favor. You should evaluate the whole script's purpose. It's mostly just reading and filtering text output from a file and then performing basic math on the filtered text. Seems like a textbook case to use Bash IMO.You're the one who got to cherry pick the script to use as an example, so don't complain that I cherry picked the most ridiculous line of that code.
Python can generate a human readable help menu with much less trouble.
Tell me one thing about that script that's easier in bash than in Python.
Forking off sed? Python doesn't need to, it has string manipulation. Forking off bc? Python doesn't need to, it has arithmetic.
And why are you using sed regular expressions just to do arithmetic? Now you have two problems!
https://blog.codinghorror.com/regular-expressions-now-you-ha...
Again, look at the Linux kernel and userland source code and give me a ballpark estimate of how many instructions it requires to multiply a number by 100 by forking off bc or sed, instead of executing a single "mul" instruction.
And Python code doesn't require a page of comments to explain why it has to go to so much effort and work around so many pitfalls just to convert a couple of strings to decimal hours.
Why, Python even has named function parameters -- now THAT is TRULY "amazing".
And in the number of keystrokes it took you to write the following comments (and the number of trials and errors it took you to finally get it right), you could have imported a hell of a lot of powerful Python modules, and read a lot of the interactive built-in help documentation about their APIs.
Your time would be much better spent getting used to using Python rather than sed.
Look, if you want to sit there and rip into a stranger on the internet for open sourcing a Bash script (on their free time) then by all means knock yourself out.
For the record I've written more Python than I have Bash (by a substantial amount). I wrote that script in Bash because it took literally 3 minutes to prototype it on the command line with grep, cut and bc so I spent the extra hour turning it into its current form. It does exactly what I want it to do (nothing less, nothing more).
Also the code is MIT licensed. Free free to fork it, rewrite it in Python, test it on 50+ real invoices, write extensive test coverage, write ~1500+ words of documentation and then make a ~20 minute video showing to use it.
Being open source doesn't magically make your code great or efficient or bug-free or easy and irresistible for millions of eyeballs to examine and audit (in spite of what ESR's fallacious "Linux's Law" might claim), or obligate anyone to complement it or use it or take it and run with it.
https://en.wikipedia.org/wiki/Linus%27s_Law
If you make your code open source, then post links to it in public forums, right after bragging about only reading the language documentation a couple of times, you'd better get over being thin skinned about criticism.
I'm not criticizing your code, or your decision to open source it, or which license you chose, or your pride in only reading the documentation twice, or your decision to cherry pick and share that script as the best example you could come up with of bash's superiority. I'm criticizing your decision to write the code in bash instead of Python.
Can you at least please answer this one simple question: Name one thing about that code that is easier in bash than in Python?
>I've written literally thousands of lines of shell scripts (maybe even 10k+) across dozens of scripts over the years and I've only really looked at the docs a couple of times. Of course I'm Googling various syntax things that are humanly impossible to remember, but I never once really sat there and combed through all of the docs or even read a book on Bash.
At least you admit 1) you've only really looked at the docs a couple of times, and 2) bash syntax things are humanly impossible to remember.
Have you ever written any 10k+ line bash scripts? No, because it doesn't scale (and machine generated configure scripts don't count). I have a 16,078 line Python script in my emacs buffer right now that I wrote over more than a decade (which is just one facet of a big system written in many other languages).
My point (which is just a corollary of JWZ's Law of Software Envelopment and Greenspun's Tenth Rule) is that programs always grow over time, and you never realize how big and complex your programs are going to grow or how far they're going to evolve when you first write them.
So never assume you can get away with using a language that's so lame it has to call out to two other different languages just to perform simple string manipulation and arithmetic.
Good grief, even TCL can perform string manipulation and arithmetic without forking a bunch of new processes.
You should invest more time using languages with docs you can actually read, and with syntax you can actually remember. You might like it.
https://en.wikipedia.org/wiki/Jamie_Zawinski#Principles
>Zawinski's law of software envelopment (also known as Zawinski's law) comments on the phenomenon of software bloating with popular features:
>Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.
https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule
>Greenspun's tenth rule of programming is an aphorism in computer programming and especially programming language circles that states:
>Any sufficiently complicated C or Fortran program contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.
I posted a link to it next to the sentence that talked about the types of tools I use it for, such as things on my dev box to help me in my day to day.
> Have you ever written any 10k+ line bash scripts? No, because it doesn't scale (and machine generated configure scripts don't count)
I've written a bunch of 1k+ line scripts. They aren't a problem to maintain and a number of them are being actively used and maintained by other people. These are for real production systems too. Some are single files, others are broken up into multiple files.
I haven't personally written a 10k+ line script (the need for that has never happened), but I've seen a number of multiple thousand line scripts out in the wild that are very well maintained and are part of mission critical systems like setting up a PKI realm for a bunch of servers. That PKI script was something like 4,800 total lines of code but it being 10k+ wouldn't have made a difference for the worse.
Shellcheck helps a lot with writing pretty well formed shell scripts. It's a really good linter.
>I've written literally thousands of lines of shell scripts (maybe even 10k+) across dozens of scripts over the years and I've only really looked at the docs a couple of times. Of course I'm Googling various syntax things that are humanly impossible to remember, but I never once really sat there and combed through all of the docs or even read a book on Bash.
Only really looking at the docs a couple times isn't something to brag about, and doing that doesn't make bash good.
Nor is it good for languages to be designed so you don't have to read the manual (or else you get COBOL or HyperTalk, and then you still have to read the manual).
Nor is it good for languages to have syntax things that are humanly impossible to remember.
All those claims you deleted actually support my point about bash being terribly inferior to Python (whose syntax is extremely simple and easy to learn and remember and most importantly read, and it even has built-in interactive "help" documentation), which is why I quoted what you originally said in the first place.
Again, what is it about that script you offered as an example that is easier, more elegant, or more maintainable and understandable to do in bash than in Python?
If you don't have an answer to that, can you show me any other bash code that was better off written in bash than in Python? And explain how it's better, please.
It's free of Python dependency hell: easier on the downstream consumers.
Python would be a nice language if it were more like Awk: a single executable with a few satellite files and no goddamned "ecosystem".
https://imgs.xkcd.com/comics/python_environment.png
In all other regards, of course proper data processing in one language is vastly superior to some awkward string chopping through external processes.
> You should invest more time using languages with docs you can actually read, and with syntax you can actually remember. You might like it.
Didn't grandparent say: "I've written more Python than I have Bash (by a substantial amount)".
People who aren't stupid and know other languages still whip out little shell scripts; we should try to understand why that is.
Nobody's running such an old version of Ultrix on their MicroVAX with a big yellow thick wire Ethernet cable and vampire tap and static /etc/hosts file derived from HOSTS.TXT downloaded weekly from SRI-NIC.ARPA without checking certificates for man-in-the-middle attacks and running unencrypted rlogin and telnet demons instead of ssh any more, that they can't easily and securely update Python and install modules.
Also: You think the Python "ecosystem" is goddamned? Remember that bash by its very nature depends on a terribly goddamned inconsistent "ecosystem" consisting of all the essential utilities it requires to do the most elementary of tasks, including awk, sed, dc, cut, grep, yes, and many others, whose locations and apis and features and parameters and behavior and performance vary widely from platform to platform. Bash is much more tightly coupled with the operating system distribution that Python.
While most of those essential features (like cough cough string manipulation and arithmetic rolls eyes, among others) are built into Python or available as standard plug-in modules (like os, system, threading, multiprocessing, pexpect, mmap, readline, Beautiful Soup, SQL Alchemy, json, yaml, etc) that are identical across all platforms.
And most of those Python modules are vastly more efficient than their bash counterparts, because they don't require forking separate processes instead of making local procedure calls, communicating over 1-dimensional pipes instead of shared memory, serializing and deserializing text parameters and data instead of passing pointers to live data structures.
What good is a scripting language these days that doesn't have built-in json and yaml parsers (not to mention a nice way to parse html and talk to databases)? I see so many half-assed bash scripts desperately trying to parse field separated values and process output with regular expressions, grep, cut, awk and sed, by forking off hundreds of ridiculously inefficient processes.
Yes I know there are xpath/jsonpath utilities you can execute from bash scripts that look up elements and keys in XML documents and json dictionaries -- that's my whole point, that there NEEDS to be something like that! God forbid you need to process 150 million records.
And as to what the grandparent poster actually said: I explained in the cousin posting below that I quoted and addressed what he ORIGINALLY said, and then he NINJA-EDITED it after I replied, to substantially change the meaning and remove several ridiculous, indefensible statements.
https://news.ycombinator.com/item?id=19898052
Unless he's just gaslighting and actually MEANT for you and others to mistakenly believe that I was putting words in his mouth (in which case we should expect his radio silence to continue), I'm still waiting for him to acknowledge that fact, and finally reply to the simple straightforward question that he kept avoiding:
Name one thing about that script that's easier to do in bash than in Python.
But I'm not holding my breath. Do you care to chime in with an answer instead?
>People who aren't stupid and know other languages still whip out little shell scripts; we should try to understand why that is.
Well the grandparent's excuse (which he deleted) for whipping out little shell scripts was that he "only really looked at the docs a couple of times". I don't think that's a legitimate excuse, myself, and apparently neither did he, so he ninja-edited it out, instead of admitting it wasn't a good defense of bash to brag about only reading the manual twice.
> The current year is 2019. It's not as if every modern Unix based operating system didn't already include or even require at least one goddamned Python ecosystem.
If you're just talking about Linux, this is kind of true (Fedora's dnf and yum before it are built on Python). But AFAIK Debian/Ubuntu apt-get is written in C/C++ and does not require Python.
Non-Linux unixes like, say, FreeBSD, do not have Python in the base distribution (nor bash). It's available in ports, which might be what you're getting at.
That said, my experience has been that non-Linux "posix" Python is a second class citizen to Python on Linux. Just for example, signal/thread semantics happen to align in a certain nonspecified-by-POSIX way on Linux and differently on FreeBSD. IIRC, FreeBSD delivers nonspecific process signals (e.g., external INT or TERM) to the most recent thread, whereas Linux delivers signals to the oldest thread. Or vice versa, but Python has its own weird signal semantics it tries to impose on top of this that works better on Linux's model than FreeBSD.
As a second example, Python fork+exec invokes close() on every possible integer between N and the file descriptor limit. This is typically tolerable for naive python programs on Linux because the default fd limit is usually 1024. FreeBSD scales the default fd limit with memory, often resulting in a much larger limit (e.g., ulimit -n = 939258 on this 32GB system). For years we have tried to get Python to adopt the closefrom(2) API, which accomplishes the same goal much more cheaply. But because most Python developers are on Linux, they simply don't feel the pain of a subprocess.Popen() taking noticeable fractions of a second.
> Also: You think the Python "ecosystem" is goddamned? Remember that bash by its very nature depends on a terribly goddamned inconsistent "ecosystem" consisting of all the essential utilities it requires to do the most elementary of tasks, including awk, sed, dc, cut, grep, yes, and many others
No disagreement there.
> What good is a scripting language these days that doesn't have built-in json and yaml parsers (not to mention a nice way to parse html and talk to databases)?
jq is a wonderful tool for parsing and manipulating JSON in sh, though it has its limitations. I certainly wouldn't attempt to parse HTML in (ba)sh. I don't encounter YAML very often.
> Name one thing about that script that's easier to do in bash than in Python.
"ls -l"? There are lots of quick and sloppy path manipulation things that are easier to bang out in shell than in Python -- that's why most people use (ba)sh et al as interactive shells, instead of python.
Anyway, Python is powerful, and quite possibly more consistent platform to platform than bash. No disagreement there. I'd also argue it's a much better tool for writing programs that handle adversarial input than bash is. The ways Python differs between platforms are mostly areas bash doesn't attempt to cover at all. And the datastructures available in Python are much more powerful than in bash.
Has anyone tried coding it and creating a locally applied patch?
Knowing the basics of shell scripting is not optional. As a dev on a unix system, it's just a question of time before you have to read and tweak other people's scripts.
You may think you're clever enough to sneak out a little one-liner that nobody will notice, but all too often you underestimate the task, and it blossoms into a big messy shart script that other people have to deal with.
I wrote it in Bash.
https://www.donhopkins.com/home/catalog/unix-haters/x-window...
(Well that, and installing Solaris.)
https://www.donhopkins.com/home/catalog/unix-haters/slowlari...
Code can be astonishingly durable.
*nix is friendly, it is just selective in who its friends are" .. is a dot-signature from that era
A manual is a reading experience, a working reference system, and relies on the visual context it is presented in. There are vast differences in the quantitative and qualitative contents of manuals, on the same content! The BASH man page is .. improving ?
Specifically compare an alphabetized list of every option, including "change the preferred shortcut" and "let the sea-water in, thereby killing everyone if you are underwater" .. are right next to each other .. to modern manuals in say, the Python culture. Specifically compare say, huge amounts of text supplied for some obscure option that is not at all needed, to a terse one-liner for something crucial that is used everyday.
The parent comment cheerfully "blames the victim" with implied guilt for "not reading the manual" as if more time and effort on the part of the user, is the answer to the communication problem posed with such a dense and subtle realm.
Last thing in this rant is, that a small interesting article on something specific is an antidote to the largely unsolvable challenge here, so good on that.
An oddly specific comparison.
When I had to start dealing with bash a lot about 5 years ago I consulted "Greg's Bash Wiki" at https://mywiki.wooledge.org/BashGuide and it pretty comprehensively treats best practices for writing bash scripts. I often go back to consult the guide.
This is an open-source thing. Hardly anyone wants to write documentation that isn’t paid to do so. There’s no glory in it. I first learned Unix on SunOS/Solaris and the man pages were excellent. You could start with “man intro” and learn your way around the entire system from there.
BTW, if you think you’re any good at documentation, open-source needs your help. It’s a great way to contribute.
There actually is glory in it. Many (granted, not all) popular, celebrated open source projects got that way by having a great user experience with intuitive, accessible docs.
I actually think that most programmers (even many very good ones) are simply bad at writing docs. They actually can't do it well. Or it would be a monumental effort for them to do it well.
It really is a different skill set.
Or are these "great programmers" not actually that good at that?
It is correlated (though probably not perfectly) with writing readable code. But that is a different skill from writing useful, correct software. That is, you can write useful, correct software whose code is difficult for others to understand and maintain.
Actually, I don't want to do that either.
What I really want to do is use a Python script instead. :3
Then you have Xonsh, but in my opinion, switching between the two contexts, shell and python, at each part of a command was too difficult. I was totally lost even at writing very basic commands.
Checking zsh by comparison: 6 pages, though a long "SEE ALSO" list with 10 zsh-specific entries, totalling 374 pages in all:
I'm aware that generating postscript is possible, and results in fewer pages. But that doesn't readily permit comparisons with simple shell tools.
(The output is decidedly pretty though.)
I ran into a situation where I thought "wow it would have been useful if I knew when this command was run" while going through a massive amount of history on a foreign server.
So then I Googled for "add timestamp to bash history" and immediately found it.
FWIW I don't think I've ever seen this done in any system anywhere. My usual expectation is that local executables are run as "./foo".
1. It’s confusing to do a plain “read” and later have a magic $REPLY variable show up, seemingly from nowhere. It has shades of Perl, and I mean nothing positive with that statement.
2. Names are good. By setting a name to the thing which is read, you make the later code easier to read. It’s easier to understand “echo $filename” than “echo $REPLY”, so if you’re reading filenames, do “read filename”. (Or, rather, you probably want to do “read -r filename”, but that’s a different subject.)
I know it because I once needed to strip whitespace and only found out through IRC; the difference is only somewhat hinted at in the manual.
It's good practice to start your scripts with "set -euo pipefail" and then list exceptions to that scoped with "set +..." as needed. That's the "unofficial bash strict mode"[1].
I don't use the IFS recommendation in that article. I prefer to just keep one set of shell inanities in my head, and being used to the non-default IFS behavior doesn't help when debugging other people's shellscripts.
1. http://redsymbol.net/articles/unofficial-bash-strict-mode/
If you're writing recursive shell scripts where you're not sure whether you should exit or not, you shouldn't be writing shell scripts.
Leave it to bash to go out of its way to make it easier to write scripts that break encapsulation and implement spooky mysterious action-at-a-distance by implicitly depending on how they were run, and purposefully behave differently whether they're invoked from a command line or another script, and are so confused about what they should do that they have to make guesses about whether or not to exit.
Scripts should not behave differently depending on if they were invoked from other scripts, and modular reusable libraries should never take it upon themselves to call "exit" when there's an error.
The way to control the flow of recursive functions is by passing explicit parameters (like the recursion depth, or a data structure to walk, or an error callback), not reflecting on the depth of the runtime stack or looking at how the function was called.
If bash is too weak to handle errors, or exceptions, or recursion, or passing functional callback parameters properly, or even file names with spaces in them, then use a better scripting language, don't just throw up your hands and exit.
https://stackoverflow.com/questions/14199689/how-can-i-handl...
https://github.com/texane/stlink/issues/634
Is this implying that you should put `.` on PATH? I've always heard that to be a security problem.
What's interesting is that so many of these variables are incredibly tersely named. What's the reasoning behind not using a full name for them to make Bash actually readable? Surely not the minor performance scrape?
Sure, on an otherwise 100% secure system accessed only by trusted users there is no problem, but not having . in PATH is a trivial mitigation.
All systems are secure and sufficiently hardened until a botnet demonstrates otherwise, aren't they? And a botnet is the friendly hostile takeover because it'll just deploy a bot or a crypto miner and might even patch your system.
Just don't put . on your PATH. And if you do, append it - <PATH=$PATH:.>. This way, /usr/bin/foo will win against ./foo.
It's also used by the `z` command line tool to maintain its directory "frecency" list.
https://www.freebsd.org/cgi/man.cgi?query=readlink&sektion=1
So how is that a Linux-centric view again?
> if the -f option is specified, the output is canonicalized by following every symlink in every component of the given path recursively. readlink will resolve both absolute and relative paths, and return the absolute pathname corresponding to file. In this case, the argument does not need to be a symbolic link.
This is the case for *BSD, GNU/Linux, and Irix. If macOS would prioritize being consistent with the other Nix OSs, it would be supported too, but alas Apple doesn’t.
> if the -f option is specified, the output is canonicalized by following every symlink in every component of the given path recursively. readlink will resolve both absolute and relative paths, and return the absolute pathname corresponding to file. In this case, the argument does not need to be a symbolic link.
[0] https://www.freebsd.org/cgi/man.cgi?query=readlink&sektion=1
Update:
And here’s the OpenBSD man page from 1997 refuting your claim it’s for formatting as well:
https://man.openbsd.org/OpenBSD-2.2/readlink.1
Update 2:
OpenBSD (1997): http://www.openbsd.org/cgi-bin/cvsweb/src/usr.bin/readlink/r...
GNU/Linux (2003): https://lists.gnu.org/archive/html/bug-gnu-utils/2003-02/msg...
So it definitely did exist in at least one *BSD before Linux.
Maybe next time understand what does and doesn't work before you complain about a solution that is both in POSIX and works on every platform.
[0] http://pubs.opengroup.org/onlinepubs/9699919799/
A bit of scripting is then wanted to coerce the output of 'dirs' into a readable format.
https://github.com/clvv/fasd
For years I've been trying to figure out what exactly the difference is. Terminals are seen as difficult and outdated, yet the more nerdy you get, the more likely it is that you end up using (and the more time you will spend in) a terminal. There has to be a reason terminals haven't replaced GUIs altogether, and GUIs haven't replaced terminals altogether, despite the near-complete overlap in things you can do with them. The only thing that really needs a mouse/touchscreen is when you have something spatial like photo editing, but conversely, the extreme end of the power user tools almost always use a terminal and can't really work with a GUI.
"This is similar to the confusion I felt when I realised the dot folder was not included in my more familiar PATH variable… but you should [not] do that in the PATH variable because you can get tricked into running a ‘fake’ command from some downloaded code."
https://web.mit.edu/~simsong/www/ugh.pdf
Unix-Haters Handbook Preface
Things Are Going to Get Lot Worse Before Things Get Worse
“I liken starting one’s computing career with Unix, say as an undergraduate, to being born in East Africa. It is intolerably hot, your body is covered with lice and flies, you are malnourished and you suffer from numerous curable diseases. But, as far as young East Africans can tell, this is simply the natural condition and they live within it. By the time they find out differently, it is too late. They already think that the writing of shell scripts is a natural act.” -Ken Pier, Xerox PARC
I don't go this far yet, but do strongly recommend nodev and nosuid except where absolutely needed. This is also a strong argument for multiple partitions rather than single-partition formatting.
Also, there’s a better argument for not putting . in $PATH: if your $PATH is relatively static, you have less to think about when looking through your shell’s history for a particular command: if . is on the PATH, you always have to wonder whether a particular history entry is running a system-wide command or a command in $PWD
https://en.wikipedia.org/wiki/Wikipedia:Don%27t_stuff_beans_...
It's too late, the damage is irreparable. By now, hundreds of people have probably already put . in their path, mounted their repos and /home noexec nodev and nosuid, and mounted a separate file system on ~/Downloads, just to spite you. ;) Or were you trying to trick them into doing that all along, so you could hack into their systems? I can never tell with you slippery cyber security types...
https://books.google.com/books?id=KJQRAwAAQBAJ&pg=PA36&lpg=P...
http://tldp.org/LDP/abs/html/exit-status.html (HTML)
http://tldp.org/LDP/abs/abs-guide.pdf#page=57 (PDF)
https://www.linuxjournal.com/article/2800
The latest revision of that text seems to be available here:
https://tiswww.case.edu/php/chet/bash/article.pdf
One very useful one not mentioned on this page, possibly because considered well-known: $RANDOM
http://tldp.org/LDP/abs/html/randomvar.html
LINENO will use the last line number of a multi-line "$(...)" command, offsetting all the commands' line numbers within the substitution. Since multi-line strings are concatenated by Bash internally, LINENO will only increase by one for those.
You end up with line numbers greater than the number of lines in your file, or overlapping with empty lines or comments. In this matter, Zsh's LINENO is way better as it will have the expected value (it will always be the line where the command was started, not an offset).
Here is a function to convert HH:MM:SS to seconds:
[0]: https://www.gnu.org/software/bash/manual/html_node/Bash-Vari...It's also surprising that the first version of bash committed source code suicide via a bug in globbing, which makes it's name seem more poignant, and was why there was a file named "-i" in it's source tree for many years.