Excellent talk. She seems to be a very likable person. She is right about Bash being full of "gotchas" and trivia and memorizing them all is very hard, but I think it is nice to memorize some trivia. For instance, I tended to forget the order of the arguments of the find command, and I would lose time trying to remember its syntax when I'm in front of a machine with no readily available internet connection. So I committed to learning and memorizing the most common command line tools and some of their "gotchas". I used Anki for that, and some mnemonics, and the return on the investment has been worth it I think.
I maintain a file with commands that I don't use often (ex: increase volume with ffmpeg, add a border to an image with convert, etc). I even have a shortcut that'll add the last executed command to this file and another shortcut to search from this file.
Oh, that's a great idea. I have a doc that I maintain by hand, either via ">>" or editing directly. Time to go and make a shortcut. Do you do any annotation to help with the search?
Usually no annotations, as I typically search by command name. But sometimes I edit the file to add comments if there are many examples for the same command.
A large Justfile (https://just.systems/) of random recipes might be a way to make it both executable and searchable (at least on zsh, you can get an autocomplete list of completions from the command line).
Can you think of a single CLI tool that would let me commit a past incantation to a file and retrieve it later? Especially one that syncs well across devices.
The best I can think of is Atuin (https://github.com/atuinsh/atuin) but I wasn't super interested in using it - I kind of want something more lightweight.
Pretty much the same, though I usually just keep the file open in a side terminal. I want to use stuff like cheat.sh (ex. curl cheat.sh/grep) but I never remember.
If you’re already putting them in a file, you might as well put them in a shell script on $PATH: at a certain point I started writing shell scripts and little utilities for relatively infrequently used commands and other tasks (e.g. clone this repo from GitHub to a well-known location and cd to it)
Keeping a ~/bin directory with all your personal shell shortcut scripts has been my go-to for years. I tend to make a lot of project-specific shortcut scripts for anything that I want to remember/becomes a common task.
You're right that you often can't modify your current environment by invoking a shell script. That's because it's executed in a sub-shell.
For cases where you need to modify your current environment (setting environment variables, changing directories, etc), you need to run the script using the "source" built-in. That will execute the script in the current shell rather than a sub-shell.
So instead of
./some-script.sh
you'd run
source some-script.sh
or use the dot (".") shorthand
. some-script.sh
In cases where I need to source a script, I generally create an alias or a shell function for it. Otherwise I may forget to source it.
Nobody asked but I'd like to chime in with my method.
I have a lot of aliases, for example to start my QEMU VM with my development stuff in it, I make an alias for 'qemu-system-x86_64 [...]' with all the switches and devices and files required, called 'startvm'. I have another that takes me to my current project's folder and pulls the repo. And a third that creates a new folder called 'newproject', creates a small set of folders and empty files with specific names, and finally makes a git repo in it. I am a serial abandoner of projects so I use this more often than I care to admit.
It's not pretty, but functional; and since I always copy my dotfiles when I change computers, I've kept these small helpers with me for a while now.
If you don't mind, it would be awesome to see your cheatsheet. I think this would be a great thing for people to share - like their dotfiles. But maybe they already do and I don't pay much attention to it because I'm lazy - like their dotfiles.
I like `fzf`'s default override of Ctrl+R backwards search for this purpose, along with the fish shell's really good built in autocompletion.
I've been thinking about updating the GIFs in my fzf tutorial to show off fish, but I think I'd rather leave them with ish just so I don't dilute the pedagogical message.
I wrote an Emacs package that works fairly well for saving commands but also making them reusable from its file manager without the need to tweak input or output file paths https://github.com/xenodium/dwim-shell-command
While Emacs isn’t everyone’s cup of tea, I think the same concept can be applied elsewhere. Right click on file(s) from macOS Finder or Windows Explorer and apply any of those saved commands.
I tended to forget the order of the arguments of the find command, and I would lose time trying to remember its syntax when I'm in front of a machine with no readily available internet connection.
The man pages are readily available.
The bash man page is huge and hairy, but comprehensive, I've found it pretty valuable to be familiar with the major sections and the visual shape of the text in the man page so I can page through it quickly to locate the exact info I need. This is often faster than using a Internet search engine.
I'm not a fan of man pages. Or any documentation that focuses on textual explanations rather than examples in code (looking at you aws).
I recently found https://tldr.sh/ and found it more convenient. I ended up writing myself a vscode extension to have a quick lookup at my fingertips, since I am at least 60% of the time looking at a terminal in vscode
It might be better to invest in something more general like better docs/cheatsheets (the bad old man pages which you could convert to a text editor friendly format, or something better like tldr, or something like Dash) so you don't depend on the internet, but also don't have to memorize bad designs (since find wouldn't be the only one)
I came here to say Anki is my lifeline for grokking difficult things like DNS.
It was in fact on jvns.ca's book recommendation that I got Michael W. Lucas's _Networking for System Administrators_, and strip mined it for Anki cards containing both technical know-how and more than a little sysadmin wisdom.
It might be one of the highest ROI books I've ever read, considering I actually remember how to use things like nectat and tcpdump to debug transport layer issues at a moment's notice now.
TiL: The shell does not exit if the command that fails is a part of any command executed in a && or || list except the command following the final && or ||.
`grep` should terminate with an empty output on `stdout` and an error message on `stderr`, then `sort` will successfully sort the empty contents of `stdout`
"Fails" is a higher-level concept than the shell is concerned with. Failure conditions and reactions are entirely at the discretion of the programmer and are not built as an assumption into the shell.
The only thing /bin/false does is return 1. Is that a failure? No, that's how it was designed to work and literally what it is for. I have written hundreds of shell scripts and lots of them contain commands which quite normally return non-zero in order to do their job of checking a string for a certain pattern or whatever.
Programs are free to return whatever exit codes they want in any circumstance they want, and common convention is to return 0 upon success and non-zero upon failure. But the only thing that the shell is concerned with is that 0 evaluates to "true" and non-zero evaluates to "false" in the language.
It would be pretty inconvenient if the shell exited any time any program returned non-zero, otherwise if statements and loops would be impossible.
If a script should care about the return code of a particular program it runs, then it should check explicitly and do something about it. As you linked to, there are options you can set to make the shell exit if any command within it returns non-zero, and lots of beginner to intermediate shell script writers will _dogmatically_ insist that they be used for every script. But I have found these to be somewhat hacky and full of weird hard-to-handle edge cases in non-trivial scripts. My opinion is that if you find yourself needing those options in every script you write, maybe you should be writing Makefiles instead.
> It would be pretty inconvenient if the shell exited any time any program returned non-zero, otherwise if statements and loops would be impossible.
In another life I worked as a Jenkins basher and if I remember correctly I had this problem all the time with some Groovy dsl aborting on any non zero shell command exit. It was so annoying.
There are more arcane things to learn about shell, at some point one has to go shrug, it's a fine tool for getting quick results but not for writing robust programs.
In my opinion, it is one of the biggest flaws in the shell language design, because it means, that a function can lead to different results independent of the arguments, but depending of the context from which you call it. And it even overrides explicitly setting `set -e` within a function.
Well said! I'm in manufacturing, not web tech, so the hard things I work with are very different (RS-274 G-code, servo motion systems, PLC/SCADA network connections are a few examples), but it's interesting to see the overlap in causes of difficulty like trivia, gotchas, frequency of use, and visibility. And disheartening to see that the web tech community is so open and friendly and communicative, while controls engineers may rarely hang out on forums but rarely share anything even close to this outside of a $2,000 training course.
And not to nitpick, but I don't have Julia's email, but when struggling with hard things like DNS I hate to have someone bounce off this little roadblock...her link to to the demo of the DNS exploration website is pointing to the .com, when it's actually at the .net TLD:
This is a talk turned into a web page done right. It seems like it should be a simple thing to do, but often the results are confusing and hard to read. Not here, well done!
I'd love to know if jvns has a library or something she uses for this. I've seen this kind of slide thingy on https://boringtechnology.club/ too, and I'm really curious!
I really disagree strongly with the take on bash. The best solution is not to add tooling on top of bash or memorize its idiosyncrasies. It is to not use bash. That is the only way to escape its pitfalls.
That's easier said than done, and completely getting rid of bash may often not be worth the time and effort. But in general I agree, anything remotely complex I try to offload into scripts written in less idiosyncratic languages. Having some tools to help avoiding mistakes with the last 5% that stay bash scripts is super helpful in that case.
a part of me agrees with the aim of what you say: why not start afresh with something that is less prone to accidents? and i do agree with that idea. but i think it is also somewhat impractical to ignore that pretty much every server i've ever interacted with has a default, vanilla, bash installed, and if i know how to use that, it helps me get by.
not to say we can't and shouldn't try to do better.
I think this is a valid point. bash is an overly complicated tool...so I'll write another tool on top of that (with none of the decades of debugging that bash itself has undergone) to make bash...LESS complex?
The problem is with bash itself.
We tend to undervalue ease of use and overvalue "cleverness."
Case in point: git. Very clever tool. Ease of use: terrible. But Linus wrote it and Linus is clever, so it must be us that's the problem.
We get what we value. Let's value ease of use more.
It has idiosyncracies because it's not a general purpose language. Even the things that she mentions are happening for good reasons - like the fact that set -x would break the expected behavior of || and &&.
Actually what language does crash when a function returns false? I mean some throw exceptions but isn't "false" a valid thing to return?
I find the same thing with makefiles - people don't understand what they're doing and expect them to work in a certain way because they haven't ever thought about build systems very deeply. Recursive assignment in Make catches almost everyone out e.g.
FLAGS=-b
COMPILE=compile $(FLAGS)
$(info compile command=$(COMPILE))
FLAGS=-a
myfile:
echo $(COMPILE) $? -o $@
outputs:
t43562@rhodes:~ make -f t.mk
compile command=compile -b
echo compile -a -o myfile
compile -a -o myfile
Despite this, making all assignments immediate to match other programming languages would take a VERY useful tool away. The more you understand these tools the more you know where to bother using them and how much effort to put into it.
I have yet to find a proper replacement for bash. Especially for scripts.
The two most common alternatives are 1) using some of the newer shells people have created, like Oil shell [0], or 2) using programming language like Python, JavaScript, or PHP.
The problem with using a newer shell is that you'll have to install the new shell anywhere you want to use the script. Meanwhile bash is ubiquitous. Unless you're the only one maintaining the script, you're requiring others to learn the other shell to maintain the script.
The problem with using another programming language is that they rarely have good ergonomics for doing what bash does: stringing together commands, command input, command output, and files. If you try to do that in another programming language, things suddenly get a lot more complicate or at least more verbose.
So I still use bash, but I recognize that it's strength is in running other commands and dealing with I/O. If I'm doing complicated logic that doesn't involve that, then I'll offload my work to another language. Sometimes that just means calling a python script from bash, not avoiding bash completely.
If people found that they work better by taking other approaches, the please share them.
I'll still write bash, but only if it's very trivial (~ <= 10-ish lines, no super-complex conditionals, etc). For anything else, it's worth stepping up to just about any more-robust language. If you don't like the hoops that something like Python makes you go through for basic shell-like operations, maybe try Perl? For middling-complexity scripts, perl can be a nice win over bash, without all the development overhead of Python. Perl5 is pretty much installed everywhere and universally compatible for anything but the very latest language features.
I usually resort to running inline scripts using python3 or node (using only the standard library) if I need to do something that I can't express in bash easily. That means avoiding stuff that have to look up and I won't remember or be able to debug in a month.
I like the idea of running inline scripts. This means that all the logic is in a single file so you don't have multiple files to copy around (e.g. a .sh file and a .py file).
I've done this with awk and jq, but haven't done it with node or python. But why not? It sounds like a good approach for some cases.
I love bash for anything that doesn't need to scale. The reason is reliability. I can't tell you how many times I've run into problems with python or similar on small library implementations. If I need to make an http request in my code and I'm using curl in bash and something fails, I can pretty much guarantee the problem is not curl itself. It's battle tested and deterministic. However the python script that forgets to wrap a try catch or configure proper tls auth or some other quirk? I have had those eat hours/days of my time and having to resort to tcpdump to sort out what's really happening. Same with SQL libraries/ORM vs SQL cli, same with Json/xml parsing libraries vs jq etc.
> The problem with using another programming language is that they rarely have good ergonomics for doing what bash does: stringing together commands, command input, command output, and files. If you try to do that in another programming language, things suddenly get a lot more complicate or at least more verbose.
tclsh is one way around that. Use a real first-class programming language, but in a mode where calling programs is as easy as it is in shell.
Yes. While I’m a huge fan of shellcheck and am one who actually have used and know bash deeply. No amount of linters or other tooling on top of bash can fix it.
Best solution is to just stay away.
For real. Just stop. Don’t try to be macho. The whole model of the language is fundamentally broken. I mean, stringly typed, global mode switches, one-character flags for fundamental comparison operators, defaulting to ignoring errors at every corner you look, functions especially. Each such idiosyncrasy on its own is enough to dismiss such a language, bash has them all plus more.
This is a great description of things that seem like they shouldn't be so difficult but can have many complications. The SQL part seems to double-down on a conceptual failure rather than demystifying it though.
A query's logic is declarative which defines the output. It's the query plan that has any sense of execution order or procedural nature to it. That's the first thing to learn. Then one can learn the fuzzy areas like dependent subqueries etc. But being able to see the equivalence between not-exists and an anti-join enables understanding and reasoning.
Using an analogy such as procedurally understanding of written queries only kicks the can further down the road, then when you're really stuck on something more complicated have no way to unravel the white lies.
> The SQL part seems to double-down on a conceptual failure rather than demystifying it though.
She talked about a mental model to help her understand the query (it can be useful), and mentioned that it probably is not how the database actually processes the query.
My point is that there should be two mental models. One for getting the correct results. Then another for doing so performantly. Being able to write many different forms of obtaining the same correct results is where this leads to combined understanding and proficiency.
An example of where muddling these ends up with real questions like "how does the db know what the select terms are when those sources aren't even defined yet?" By 'yet' they mean lexically but also procedurally.
I suspect that Julia is solely using the first kind of mental model (getting the correct result), and completely ignoring query planning. But even this model has an order to it! Three examples of how this order can manifest, that should all agree with each other:
1. The explanatory diagrams that Julia drew for the talk. These wouldn't make sense if they were in a different order.
2. The order of operations you would perform if you developed a proof of concept SQL implementation that completely ignored performance. In this example the order would be: "cats, filter, group, filter, map, sort". This is exactly the order that Julia's explanation showed.
3. The relational logic expression for this query. There should be a correspondence between this expression and this ordered list of operations, though it's somewhat annoying to state. I think it's that, assuming all the operators in the relational logic expression are binary, if you reverse the order that a subset of the operators are written in, then the operators in the tree occur in the same order as the ordered list of operations. (I don't actually know relational logic, so I'm making a prediction here. This prediction is falsifiable: you can't put the operators in the tree in an arbitrary order.)
(Side note: the order isn't completely fixed. The last two steps --- SELECT and ORDER BY --- could happen in either order.)
that was fantastic. i didn't watch the talk, i opened up the transcript and thought to myself i would read a few slides and see if i enjoyed it. that was maybe 20-30 mins ago and i finished it all. really well done, i enjoyed it from start to finish.
i like the way she broke it down. i feel like the author addressed ways to learn and communicate effectively and also grounded it in very concrete terms. i don't want to get too much into psychology or anything which i really am not qualified to talk about but i feel like going through these types of ways of learning and communicating is a continued exercise in ego deflation and in pragmatic problem solving.
i liked the SQL order-of-query-operations thing, the bash and shellcheck thoughts (i will use `-o all` from now on), i am curious to play with the DNS tool (the article links to https://messwithdns.com/ but the actual URL - taken from the text - is https://messwithdns.net/ , FYI), and i enjoyed the part about HTTP (i was hoping she would talk about SPDY and whatnot, i have not even begun to explore such things and am curious what that is all about).
i am going to read the two posts linked the behind-the-scenes on "hello, world".
I expected there to be an abstracted, general, repeatable tldr that can be reapplied as a mental model. I haven't digested the whole thing, but after skimming, I can't identify what it is.
it's kind of more like the beginning of a pattern language for why things seem hard to novices but simple to experts, and ways to mitigate those effects - a different mitigation is appropriate for each pattern
And that is why I like the book Accelerated C++. Instead of explaining each and everything. It helps just the right things, thus preparing you for further adventures.
Another literature that comes closer is Designing Data Intensive Applications and SICP.
Better/worse is one of those subjective things that comes with experience. No one can definitively answer such a question in a way that will apply to every circumstance.
But as someone who rarely writes bash scripts but just enough to be dangerous ... I use `set -xe` at the top of every single shell script I write. The -x flag causes the script to echo every command that is executed to stdout which can be very valuable if I am debugging a shell script that is running on an external environment (like a CI). I've seen this habit suggested many times and it has served me well.
I'd have to look up exactly what's what, but if you're inclined to `-e` you probably want the others too. (Off the top of my head I think E is the same thing in functions, u is bail out if a variable is used without being defined (instead of treating it as empty string), and pipefail is similar for when you pipe to something else like `this_errs | grep something`.)
I don't like `-x` personally, I find it way too verbose; more confusing than helpful.
I have no idea why but I wanted to hate this article. Maybe jvns shows up on HN too often and I was in a bad mood. But this is a great article and as someone with 20 years of development experience is about as true as any meta-level discussion on programming could be.
The selective vision thing is so true, both for `dig` and for `man` pages. I can't count the number of times have I `man <cmd>` and just felt overwhelmed by the seemingly endless pages of configuration options and command line flags. One tip I use for `man` is use vim style search functions triggered with `/`. For example, if I want to find how to output the line number of each match in grep and I can't remember how - I'll just `man grep` then type `/line` and hit enter and it will search for any occurrence of the word "line" in the man page. Next match is just `/<enter>`.
I'm also a bit sad to hear that Strange Loop is now finished? I only found them last year or so and it seemed like so many of the talks were exceptional quality.
> I'm also a bit sad to hear that Strange Loop is now finished? I only found them last year or so and it seemed like so many of the talks were exceptional quality.
You might want to watch Alex Miller's talk that has been uploaded recently:
To make hard things easy you have to find the right way to abstract them so you hold only some bits of the hard things in your head and all the frequently-used details too (maybe), and everything else you have to look up as needed. That's what I do, and that's roughly what TFA says.
The problem is that people don't necessarily bother to form a cognitive compression of a large topic until they really have to. That's because they already carry other large cognitive burdens with them, so they (we!) tend to resist adding new ones. If you can rely on someone else knowing some topic X well, you might just do that and not bother getting to know topic X well-enough. For those who know topic X well the best way to reduce help demand is to help others understand a minimal amount of topic X.
> So, bash is a programming language, right? But it's one of the weirdest programming languages that I work with.
Yes, `set -e` is broken. The need to quote everything (default splitting on $IFS) is broken. Globbing should be something one has to explicitly ask for -- sure, on the command-line that would be annoying, but in scripts it's a different story, and then you have to disable globbing globally, and globbing where you want to gets hard. Lots of bad defaults like that.
It's not just Bash, but also Ksh, and really, all the shells with the Bourne shell in their cultural or actual lineage.
As for SQL, yes, lots of people want the order of clauses to be redone. There's no reason it couldn't be -- I think it'd be a relatively small change to existing SQL parsers to allow clauses to come in different orders. But I don't have this particular cognitive problem, and I think it's because I know to look at the table sources first, but I'm not sure.
The problem is that we're still using these ancient shells when we have better ones. Users shouldn't be wasting time memorizing arcana like "set -e". At least we have search engines now...
I'm quite partial to the fish shell myself for this reason.
But I SSH into a lot of embedded systems these days, where you don't exactly have the luxury of installing your own shell all the time. For those times I like to whip out the "minimal safe Bash template" and `sftp` it to the server.
When explanations include superfluous detail, I find it very confusing. Like Chekhov's gun, I keep trying to fit it into the plot but it doesn't fit.
My super power is a terrible memory. So I have to understand things in order to remember them (aka a cognitive compression). I can't just learn things like normal people.
These docs are comprehensive, but most people don't want that level of detail, so having someone else test it and write something short would help!
For awhile I didn't "push" Oils because it still had a Python dependency. But it's now in pure C++, and good news: as of this week, we're beating bash on some compute-bound benchmarks!
(I/O bound scripts have always been the same speed, which is most shell scripts)
What if my SQL engine is Presto, Trino [1], or a similar query engine? If it's federating multiple source databases we peel the SQL back and get... SQL? Or you peel the SQL back and get... S3 + Mongo + Hadoop? Junior analysts would work at 1/10th the speed if they had to use those raw.
Every time a jnvs.ca article goes popular on HN I have to relive the trauma of my Stripe interview with Julia where I just could not get that simple test to pass despite having years of programming experience. Ugh! She was nice though :)
I both agree and disagree with her sentiments here. Not in a right/wrong sense, but just what I found works for me.
I agree with having helpers to understand how tooling works. Any resources that increase understanding of how to use a tool to it's full capability is productively beneficial. (Hat-tip to anything that unpacks all the `curl` switches.)
Here's where I have disagreement: an old boss of mine once said "don't worry about the tricks of the trade; learn the trade." That's very contextual, but sometimes understanding the core first makes the rest of it easy. And understanding the core takes both effort and increases cognitive load, so I understand why one might not go that route.
So, for me, I always try to keep a balance between trying to back my way into execution via those helpers, and recognizing when I need to take a step back and learn at a bit more core level.
Her down-to-earth approach is refreshing, that's for sure.
> Here's where I have disagreement: an old boss of mine once said "don't worry about the tricks of the trade; learn the trade." That's very contextual, but sometimes understanding the core first makes the rest of it easy.
Seems like you might agree more than you realize! From TFA:
> And much like when debugging a computer program, when you have a bug, you want to understand why the bug is happening if you're gonna fix it.
It sounds a lot like "don't worry about how to fix the bug; learn what the bug is."
The point Julia makes about `grep` resonate with me alot! I have the same (call it problem if you want) issue with `ps`. There is only one variation of ps I know of (`ps aux`) and if I want to change that I have to either look for the options in the man page or google it.
Oh, that's a nice one, thanks for mentioning it. I'm personally used to "ps afx", but the "u" does give out some quite useful info... and it's compatible with "f"! So I guess I'll add "ps aufx" to my repertoire.
ps is bad in general, the default view is almost never what you want and lots of minor formatting options make for a complicated man page. But it is especially a shitshow on linux.
fsf: which ps options should we use(bsd, systemv, solaris, sgi)?
also fsf: well that's a tricky one... why not all of them?
I don't think her conclusion is correct in this case (that it's all about chronological order). It's not about what happens first but about levels of abstraction. It doesn't make sense to say that we want to decide on the number of doors before specifying if we're building a LEGO car or a skyscraper in Manhattan.
The reasonable approach is to start from high-level concepts and only then deal with the details - without specifying high-level concepts the details have no particular meaning.
As a side note I have to say that I definitely prefer languages with `object.function` rather than `function(object)`, precisely because of this.
Another example: `if(foo == 5)`, not `if(5 == foo)`
Well, I'm going to disagree with her solution to bash. Bash is terrible, end of story. Just the *.txt is actually an endless pit of trap, badness and wrong behaviour.
I've not written a bash script in years. I always use Python and in my experience professionally, most people automating things nowaday use Python. I only use bash to write one-line to invoke the underlying Python script. For the record, I do the same for batch file on Windows, with a simply .bat file to invoke the underlying Python script.
Trying to improve the bash experience with tools and knowledge is throwing bad time at a problem that has a well-known solution.
Edit: to be clear, the presentation itself is awesome. I'm just disagreeing that bash needs to be better explained. I'm ready to admit that there are existing scripts out there, most of which are probably only working in the trivial, normal case and are just one snag away from exploding, and having a known source of help can help. But please, take you're bash scrip to the shed and upgrade them to Python.
These are the use cases where generative AI is great. I want to convert an mp4 to mov and resize it to 720p with ffmpeg, give me the command. I want a bash script to filter out the third column from a CSV and convert the result to a JSON.
These are not hard, but I won’t remember ffmpeg flags because I only use it once per year.
201 comments
[ 3.3 ms ] story [ 181 ms ] threadThe best I can think of is Atuin (https://github.com/atuinsh/atuin) but I wasn't super interested in using it - I kind of want something more lightweight.
https://github.com/mikemccracken/hs
your snippets are stored in a git repo that you can sync around how you like.
For cases where you need to modify your current environment (setting environment variables, changing directories, etc), you need to run the script using the "source" built-in. That will execute the script in the current shell rather than a sub-shell.
So instead of
you'd run or use the dot (".") shorthand In cases where I need to source a script, I generally create an alias or a shell function for it. Otherwise I may forget to source it.It’s more like “roll your own oh-my-zsh”
I have a lot of aliases, for example to start my QEMU VM with my development stuff in it, I make an alias for 'qemu-system-x86_64 [...]' with all the switches and devices and files required, called 'startvm'. I have another that takes me to my current project's folder and pulls the repo. And a third that creates a new folder called 'newproject', creates a small set of folders and empty files with specific names, and finally makes a git repo in it. I am a serial abandoner of projects so I use this more often than I care to admit.
It's not pretty, but functional; and since I always copy my dotfiles when I change computers, I've kept these small helpers with me for a while now.
I've been thinking about updating the GIFs in my fzf tutorial to show off fish, but I think I'd rather leave them with ish just so I don't dilute the pedagogical message.
Just today I saved a new one for trimming borders on video screenshots https://xenodium.com/trimming-video-screenshots to https://github.com/xenodium/dwim-shell-command/blob/main/dwi... (that’s my cheat sheet).
I wrote an Emacs package that works fairly well for saving commands but also making them reusable from its file manager without the need to tweak input or output file paths https://github.com/xenodium/dwim-shell-command
While Emacs isn’t everyone’s cup of tea, I think the same concept can be applied elsewhere. Right click on file(s) from macOS Finder or Windows Explorer and apply any of those saved commands.
Edit: More examples…
- Stitching multiple images: https://xenodium.com/joining-images-from-the-comfort-of-dire...
- Batch apply on file selections: https://xenodium.com/emacs-dwim-shell-command
The man pages are readily available.
The bash man page is huge and hairy, but comprehensive, I've found it pretty valuable to be familiar with the major sections and the visual shape of the text in the man page so I can page through it quickly to locate the exact info I need. This is often faster than using a Internet search engine.
True, but I find the man pages not easy and quick to parse.
Use `/` and search for the things of interest (keywords, arguments, options, etc...). Use n/N to quickly jump forward/back.
But yeah, you're right.
However, I already have this in my muscle memory: find <where> -name '<what>' -type f(file)/d(directory)
Works in 90% of situations when searching for some file in terminal, ie: find / -name 'stuff*'
The rest of the time is spent figuring out exec/xargs. :)
And once you master that, swap xargs for GNU parallel. I bet your machine has a ton of cores, don't let then sit idly. ;)
man <command> | col -b | grep "search_string"
I recently found https://tldr.sh/ and found it more convenient. I ended up writing myself a vscode extension to have a quick lookup at my fingertips, since I am at least 60% of the time looking at a terminal in vscode
gm () { man $1 | col -b | grep --color=always "$2" }
I also have something similar for grepping a command's help
gh () { $1 --help | grep --color=always $2 }
I usually try gh(grep help) and if I don't get what I'm looking for, I run gm(grep man).
It appears that I can also use tldr
It was in fact on jvns.ca's book recommendation that I got Michael W. Lucas's _Networking for System Administrators_, and strip mined it for Anki cards containing both technical know-how and more than a little sysadmin wisdom.
It might be one of the highest ROI books I've ever read, considering I actually remember how to use things like nectat and tcpdump to debug transport layer issues at a moment's notice now.
Bash is terrible.
Reference: https://www.gnu.org/software/bash/manual/bash.html#index-set
What's more pernicious is that pipelines don't cause the shell to exit (assuming set -e) unless the last command fails:
does not fail if README doesn't exist, unless you've also used `set -o pipefail`.The only thing /bin/false does is return 1. Is that a failure? No, that's how it was designed to work and literally what it is for. I have written hundreds of shell scripts and lots of them contain commands which quite normally return non-zero in order to do their job of checking a string for a certain pattern or whatever.
Programs are free to return whatever exit codes they want in any circumstance they want, and common convention is to return 0 upon success and non-zero upon failure. But the only thing that the shell is concerned with is that 0 evaluates to "true" and non-zero evaluates to "false" in the language.
It would be pretty inconvenient if the shell exited any time any program returned non-zero, otherwise if statements and loops would be impossible.
If a script should care about the return code of a particular program it runs, then it should check explicitly and do something about it. As you linked to, there are options you can set to make the shell exit if any command within it returns non-zero, and lots of beginner to intermediate shell script writers will _dogmatically_ insist that they be used for every script. But I have found these to be somewhat hacky and full of weird hard-to-handle edge cases in non-trivial scripts. My opinion is that if you find yourself needing those options in every script you write, maybe you should be writing Makefiles instead.
In another life I worked as a Jenkins basher and if I remember correctly I had this problem all the time with some Groovy dsl aborting on any non zero shell command exit. It was so annoying.
Some time ago I gave an example: https://news.ycombinator.com/item?id=22213830
And not to nitpick, but I don't have Julia's email, but when struggling with hard things like DNS I hate to have someone bounce off this little roadblock...her link to to the demo of the DNS exploration website is pointing to the .com, when it's actually at the .net TLD:
Looks like someone needs an HTML linter... :)The correct link - https://messwithdns.net/ - is actually pretty neat!
not to say we can't and shouldn't try to do better.
The problem is with bash itself.
We tend to undervalue ease of use and overvalue "cleverness."
Case in point: git. Very clever tool. Ease of use: terrible. But Linus wrote it and Linus is clever, so it must be us that's the problem.
We get what we value. Let's value ease of use more.
Actually what language does crash when a function returns false? I mean some throw exceptions but isn't "false" a valid thing to return?
I find the same thing with makefiles - people don't understand what they're doing and expect them to work in a certain way because they haven't ever thought about build systems very deeply. Recursive assignment in Make catches almost everyone out e.g.
FLAGS=-b
COMPILE=compile $(FLAGS)
$(info compile command=$(COMPILE))
FLAGS=-a
myfile:
echo $(COMPILE) $? -o $@
outputs:
t43562@rhodes:~ make -f t.mk
compile command=compile -b
echo compile -a -o myfile
compile -a -o myfile
Despite this, making all assignments immediate to match other programming languages would take a VERY useful tool away. The more you understand these tools the more you know where to bother using them and how much effort to put into it.
Many bad designs are too entrenched to be fixed with some tooling
Bash scripts are extremely useful and productive for their niche. You'd have to change a whole lot of things to match that.
That's not a very practical hill to die on. But well, you get to decide what fight you engage on.
The two most common alternatives are 1) using some of the newer shells people have created, like Oil shell [0], or 2) using programming language like Python, JavaScript, or PHP.
The problem with using a newer shell is that you'll have to install the new shell anywhere you want to use the script. Meanwhile bash is ubiquitous. Unless you're the only one maintaining the script, you're requiring others to learn the other shell to maintain the script.
The problem with using another programming language is that they rarely have good ergonomics for doing what bash does: stringing together commands, command input, command output, and files. If you try to do that in another programming language, things suddenly get a lot more complicate or at least more verbose.
So I still use bash, but I recognize that it's strength is in running other commands and dealing with I/O. If I'm doing complicated logic that doesn't involve that, then I'll offload my work to another language. Sometimes that just means calling a python script from bash, not avoiding bash completely.
If people found that they work better by taking other approaches, the please share them.
[0] https://www.oilshell.org
I've done this with awk and jq, but haven't done it with node or python. But why not? It sounds like a good approach for some cases.
tclsh is one way around that. Use a real first-class programming language, but in a mode where calling programs is as easy as it is in shell.
Best solution is to just stay away.
For real. Just stop. Don’t try to be macho. The whole model of the language is fundamentally broken. I mean, stringly typed, global mode switches, one-character flags for fundamental comparison operators, defaulting to ignoring errors at every corner you look, functions especially. Each such idiosyncrasy on its own is enough to dismiss such a language, bash has them all plus more.
A query's logic is declarative which defines the output. It's the query plan that has any sense of execution order or procedural nature to it. That's the first thing to learn. Then one can learn the fuzzy areas like dependent subqueries etc. But being able to see the equivalence between not-exists and an anti-join enables understanding and reasoning.
Using an analogy such as procedurally understanding of written queries only kicks the can further down the road, then when you're really stuck on something more complicated have no way to unravel the white lies.
She talked about a mental model to help her understand the query (it can be useful), and mentioned that it probably is not how the database actually processes the query.
An example of where muddling these ends up with real questions like "how does the db know what the select terms are when those sources aren't even defined yet?" By 'yet' they mean lexically but also procedurally.
1. The explanatory diagrams that Julia drew for the talk. These wouldn't make sense if they were in a different order.
2. The order of operations you would perform if you developed a proof of concept SQL implementation that completely ignored performance. In this example the order would be: "cats, filter, group, filter, map, sort". This is exactly the order that Julia's explanation showed.
3. The relational logic expression for this query. There should be a correspondence between this expression and this ordered list of operations, though it's somewhat annoying to state. I think it's that, assuming all the operators in the relational logic expression are binary, if you reverse the order that a subset of the operators are written in, then the operators in the tree occur in the same order as the ordered list of operations. (I don't actually know relational logic, so I'm making a prediction here. This prediction is falsifiable: you can't put the operators in the tree in an arbitrary order.)
(Side note: the order isn't completely fixed. The last two steps --- SELECT and ORDER BY --- could happen in either order.)
i like the way she broke it down. i feel like the author addressed ways to learn and communicate effectively and also grounded it in very concrete terms. i don't want to get too much into psychology or anything which i really am not qualified to talk about but i feel like going through these types of ways of learning and communicating is a continued exercise in ego deflation and in pragmatic problem solving.
i liked the SQL order-of-query-operations thing, the bash and shellcheck thoughts (i will use `-o all` from now on), i am curious to play with the DNS tool (the article links to https://messwithdns.com/ but the actual URL - taken from the text - is https://messwithdns.net/ , FYI), and i enjoyed the part about HTTP (i was hoping she would talk about SPDY and whatnot, i have not even begun to explore such things and am curious what that is all about).
i am going to read the two posts linked the behind-the-scenes on "hello, world".
thanks for the link!
edit: two things this made me think of:
1. the XKCD comic "ten thousand": https://xkcd.com/1053/
2. Mark Russinovich on git: https://twitter.com/markrussinovich/status/15784512452490526...
But as someone who rarely writes bash scripts but just enough to be dangerous ... I use `set -xe` at the top of every single shell script I write. The -x flag causes the script to echo every command that is executed to stdout which can be very valuable if I am debugging a shell script that is running on an external environment (like a CI). I've seen this habit suggested many times and it has served me well.
I'd have to look up exactly what's what, but if you're inclined to `-e` you probably want the others too. (Off the top of my head I think E is the same thing in functions, u is bail out if a variable is used without being defined (instead of treating it as empty string), and pipefail is similar for when you pipe to something else like `this_errs | grep something`.)
I don't like `-x` personally, I find it way too verbose; more confusing than helpful.
My suggestion would be to run your script like this `bash -x your_script.sh` when (and only when) you need to debug something.
The selective vision thing is so true, both for `dig` and for `man` pages. I can't count the number of times have I `man <cmd>` and just felt overwhelmed by the seemingly endless pages of configuration options and command line flags. One tip I use for `man` is use vim style search functions triggered with `/`. For example, if I want to find how to output the line number of each match in grep and I can't remember how - I'll just `man grep` then type `/line` and hit enter and it will search for any occurrence of the word "line" in the man page. Next match is just `/<enter>`.
I'm also a bit sad to hear that Strange Loop is now finished? I only found them last year or so and it seemed like so many of the talks were exceptional quality.
You might want to watch Alex Miller's talk that has been uploaded recently:
https://www.youtube.com/watch?v=suv76aL0NrA
And yes, it's sad that it ended!
However he made a very good case for why sometimes it's good for things to end. If you watch the whole talk it all makes sense.
Also I made https://github.com/kristopolous/mansnip
It lets you check the most commonly used options from your terminal, for example "tldr badblocks".
You can also press `n`
The problem is that people don't necessarily bother to form a cognitive compression of a large topic until they really have to. That's because they already carry other large cognitive burdens with them, so they (we!) tend to resist adding new ones. If you can rely on someone else knowing some topic X well, you might just do that and not bother getting to know topic X well-enough. For those who know topic X well the best way to reduce help demand is to help others understand a minimal amount of topic X.
> So, bash is a programming language, right? But it's one of the weirdest programming languages that I work with.
Yes, `set -e` is broken. The need to quote everything (default splitting on $IFS) is broken. Globbing should be something one has to explicitly ask for -- sure, on the command-line that would be annoying, but in scripts it's a different story, and then you have to disable globbing globally, and globbing where you want to gets hard. Lots of bad defaults like that.
It's not just Bash, but also Ksh, and really, all the shells with the Bourne shell in their cultural or actual lineage.
As for SQL, yes, lots of people want the order of clauses to be redone. There's no reason it couldn't be -- I think it'd be a relatively small change to existing SQL parsers to allow clauses to come in different orders. But I don't have this particular cognitive problem, and I think it's because I know to look at the table sources first, but I'm not sure.
But I SSH into a lot of embedded systems these days, where you don't exactly have the luxury of installing your own shell all the time. For those times I like to whip out the "minimal safe Bash template" and `sftp` it to the server.
https://betterdev.blog/minimal-safe-bash-script-template/
Also, we have ChatGPT now. That helps a lot.
My super power is a terrible memory. So I have to understand things in order to remember them (aka a cognitive compression). I can't just learn things like normal people.
The need to quote everything (default splitting on $IFS) is broken
Globbing should be something one has to explicitly ask for
By the way OSH runs existing shell scripts and ALSO fixes those 3 pitfalls, and more. Just add
to the top of your script, and those 3 things will go away.If anyone wants to help the project, download a tarball, test our claims, and write a blog post about it :)
Details:
https://www.oilshell.org/release/latest/doc/error-handling.h...
https://www.oilshell.org/release/latest/doc/simple-word-eval...
These docs are comprehensive, but most people don't want that level of detail, so having someone else test it and write something short would help!
For awhile I didn't "push" Oils because it still had a Python dependency. But it's now in pure C++, and good news: as of this week, we're beating bash on some compute-bound benchmarks!
(I/O bound scripts have always been the same speed, which is most shell scripts)
(Also, we still need to rename Oil -> YSH in those docs, that will probably cause some confusion for awhile - https://www.oilshell.org/blog/2023/03/rename.html )
https://lobste.rs/s/6gycoi/making_hard_things_easy#c_sjfxif
Feedback is welcome (especially based on upgrading real scripts)
We should peel off SQL and get access to the underlying layers.
[1] https://trino.io/
I agree with having helpers to understand how tooling works. Any resources that increase understanding of how to use a tool to it's full capability is productively beneficial. (Hat-tip to anything that unpacks all the `curl` switches.)
Here's where I have disagreement: an old boss of mine once said "don't worry about the tricks of the trade; learn the trade." That's very contextual, but sometimes understanding the core first makes the rest of it easy. And understanding the core takes both effort and increases cognitive load, so I understand why one might not go that route.
So, for me, I always try to keep a balance between trying to back my way into execution via those helpers, and recognizing when I need to take a step back and learn at a bit more core level.
Her down-to-earth approach is refreshing, that's for sure.
Seems like you might agree more than you realize! From TFA:
> And much like when debugging a computer program, when you have a bug, you want to understand why the bug is happening if you're gonna fix it.
It sounds a lot like "don't worry about how to fix the bug; learn what the bug is."
fsf: which ps options should we use(bsd, systemv, solaris, sgi)?
also fsf: well that's a tricky one... why not all of them?
The linux ps man page is a wild mess.
been using SQL for years and didn't stop to think about that...
The reasonable approach is to start from high-level concepts and only then deal with the details - without specifying high-level concepts the details have no particular meaning.
As a side note I have to say that I definitely prefer languages with `object.function` rather than `function(object)`, precisely because of this. Another example: `if(foo == 5)`, not `if(5 == foo)`
I've not written a bash script in years. I always use Python and in my experience professionally, most people automating things nowaday use Python. I only use bash to write one-line to invoke the underlying Python script. For the record, I do the same for batch file on Windows, with a simply .bat file to invoke the underlying Python script.
Trying to improve the bash experience with tools and knowledge is throwing bad time at a problem that has a well-known solution.
Edit: to be clear, the presentation itself is awesome. I'm just disagreeing that bash needs to be better explained. I'm ready to admit that there are existing scripts out there, most of which are probably only working in the trivial, normal case and are just one snag away from exploding, and having a known source of help can help. But please, take you're bash scrip to the shed and upgrade them to Python.
These are not hard, but I won’t remember ffmpeg flags because I only use it once per year.