Helping you how? If you actually have a problem you are trying to solve, then do just that. My experience with command line came from solving problems I had. Today I do a lot in the command line and I am learning new things all the time. However if I just wanted to get "better" at it, then I don't even know where to start because there is no clear goal.
Well said, it's an ongoing process and there is always something new to learn - some use case which we may not have come across. So, it's an ongoing process.
I used to lean on python for much of my scripting needs, mainly because the more advanced bash syntax was pretty daunting. Getting better at bash has a trickle-down effect, especially in this container age. ENV var scoping + loops and various var expansion methods really made it click for me. Shelling out to various tasks (and grabbing the results) is effortless via bash scripts. With bash on windows now it's pretty much ubiquitous. My advice is to consider why the task at hand can't be done in bash, because often times it can, with much more portability.
In both my personal and professional life, I have the opposite conclusion. Bash is only for the simplest scripts and pipelines, and everything else (assuming you're going to use it more than once) gets written in Python or Go.
The Bash syntax is not daunting, or if it is that's never been the problem with Bash. The problem is that Bash or shell programming in general gives you a million ways to shoot yourself in the foot and maybe one or two obscure, funny-looking ways to do what you want. Like iterating over the files in a directory, for example. If you think that's easy in Bash you have either have a funny definition of easy or you're forgetting a few corner cases. Or getting the output from a program—did you know that $(my_prog) or `my_prog` modifies the program output?
Fair enough! I'm more in the devops/platform space so worrying about python versions and setting GOPATH is wasted time for my purposes. What I love is the portability with minimal effort.
I'm in a similar position. Unfortunately I've seen far too many non-portable Bash scripts fail to run in production and I can't trust them! This includes simple errors like #!/bin/sh at the top, missing programs, or differences between GNU and BSD versions of programs (macOS and Linux). Chalk it up to differences between dev workstations and production. Python has some problems too but they tend to be pretty well-understood and navigated by our developers.
GOPATH isn't actually necessary except when you compile, and Python version problems mostly come up with larger programs you'd never dream of writing in Bash. 2.7 is dirt common even if you're using a slow-as-molasses-update-cycle LTS Linux.
$ mkdir "hi there"
$ mkdir "how are you"
$ ls -l
total 8
drwxr-xr-x 2 xxxxxxxx xxxx 4096 Jun 26 14:54 hi there
drwxr-xr-x 2 xxxxxxxx xxxx 4096 Jun 26 14:54 how are you
$ for f in $(ls); do echo $f; done
hi
there
how
are
you
... works better for the easy edge cases, but still probably has some issues. Personally I think klodolph called it; once you get into anything that has a few interesting edge cases bash becomes pretty unwieldy.
The easiest way is still: for file in *; do ...; done
This works fine with spaces and other strange characters, no need to come up with more complicated ways.
If you must use find, the correct way is really tricky and uses obscure bash features like settings IFS to the empty string to split on null bytes and process substitution so that the loop doesn't run in a subshell (which would prevent it from modifying variables in the current environment):
while IFS= read -r -d '' file; do
...
done < <(find /tmp -type f -print0)
I like that, I've never thought of or seen it, although it should've been intuitively obvious given how bash expansion works.
I think this demonstrates the point pretty well though - it takes a discussion among three of us before arriving at something that works moderately well, and we found four fairly different ways of doing it...
then reset it later. Perhaps store existing IFS in a variable so you can put it back. The \n vs actual line feed has to do with an edge case in cygwin and dos files.
is both much easier and much safer than piping the output of ls.
It does the correct thing with filenames containing spaces or weird characters, and the intent is much more visible, so there is really no reason to use $(ls).
One thing I would say is first check that you are running the latest version! There has been a lot of development to bash over the years. If you feel like customizing things a lot I'd check out zsh and http://ohmyz.sh/. It has a lot compatible with bash with (IMO) some saner scripting support.
Aliases are something I use a lot - it's very basic but just having "big long command with options" aliases to something easy to remember makes it much more likely I will not make mistakes, can repeat it easily in loops.
Another thing that complements using bash effectively are using other applications config files. As soon as I have a new host to interact with I add it to my ssh.config file - then any scripting I need to do I don't need to deal with any special files. Other files like ~/.netrc or ~/.pgpass make my shell sessions that much more productive. For some reason many people don't bother ever updating these and rely on the shell history to do anything more than once.
I've found the more I do anything at the shell of any complexity I end up writing a small python command line client to do any heavy lifting. Argparse (https://docs.python.org/3/library/argparse.html) makes this trivially easy and then I can use regular shell "glue" to combine a few commands together.
In addition to what others said, I can recommend just reading through the manpage once (probably in multiple sittings). Even if you don't remember the exact syntax, you will have an idea what bash can do, and know enough of the jargon to find it again in the manpage when you need it. For example, when I need to replace a substring, I used to do
FOO="Hello World"
...
BAR="$(echo "$FOO" | sed "s/World/Hacker News/")"
until I remembered that bash can do string replacement by itself. A quick search for "pattern" and "substitute" in the manpage turned up the right syntax,
I would also recommend to read Single Unix Specification on shell syntax, and
then avoid bashisms whenever possible. Bash had a history of subtly changing
its behaviour in these, which leads to scripts getting suddenly broken without
any modification on system update.
What I was hit with were subtle changes to how [[ ]] worked, along the lines
of implicit anchoring in pattern in previous version not taking place in
succeeding version. It took quite a while to debug what became a logic error.
After that I started avoiding bashisms in my scripts, especially that they
usually provide very little benefit at the cost of gambling against breaking
silently on a barely related update.
Nowadays it's also worth noting that #!/bin/sh often (Debian derivatives)
points to something that is not Bash, and people writing in shell/Bash usually
don't understand the difference.
OTOH, if you can structure your problem as a composition of pipelines, it can be quite a bit faster in bash than in a "proper" language. You get to choose optimized tools, and they run concurrently.
Writing efficient bash code forces you to think about your problem differently. It's a very similar process to thinking functionally; e.g. you don't want to deal with lines of a file one at a time in a loop, you want to do filters and maps in languages like grep, sed and awk to deal with data in a streaming fashion with a minimum of forked processes.
Is there a "proper scripting language" somewhere that supports the same first-class access to Unix programs and the same piping | syntax that shells in general do?
In most platforms you can install the rc shell from plan9. It's what I use exclusively for my shell scripts. It is only when I want to share some script with other people that I consider using /bin/sh, and even then I've gone for rc nevertheless. Here you can read about it: http://doc.cat-v.org/plan_9/4th_edition/papers/rc
Thanks for the suggestion. I've also been after a saner shell, and have been disappointed in one way or another with the approach of either using another language idiomatically (Python, Scheme, Haskell, Scala, etc.), since running commands, piping, etc. are all rather awkward; or using such languages with embedded shell-like libraries, which seem to have awkward edge-cases.
As a "real" shell, it looks like rc maintains the command, file and piping niceties of bash, whilst avoiding the edge-cases of shell-like embeddings.
Useless use of cat here just to make a point, of course.
It's easy to use Perl strings as input or output, or files, and there's also ways to interact with streams as they go along.
Generally speaking the slightly more verbose interaction with shell is made up for by the savings when you can use Perl directly to do something rather than spawn a shell process for something simple. (Shell composition can be powerful, yes, but on the other hand spawning a full process and setting up a pipe for things like "tr" or "wc" is often just silly.)
I also personally believe there's a win in the syntax; you may say "What? 'tr a-z A-Z' is way more convenient than '["tr", "a-z", "A-Z"]' but I say one of the biggest and most pervasive errors in shell is to incorrectly set up the arguments by having something interpolated in incorrectly. Having an unambiguous syntax for separating arguments has often made it much safer for me to write the code that has to use the shell. Used correctly it can even be made to work in the face of user-supplied input, something that should generally not be combined with any form of implicit argument separation. (Although bear in mind that "used correctly" encompasses more than just "separating the arguments correctly.")
I am also NOT claiming exclusivity; this is just the thing I know off the top of my head. I'm sure the other scripting languages have nice libraries, too. Though watch out for them being too nice. There's a bit of an impedance mismatch between shell and scripting language. Anything that smooths it over too well is probably either giving up critical features or introducing subtle bugs, which can become security bugs in the face of arbitrary user input.
Perl was created to do this stuff. Perl is like duct tape to hold together everything. Most people call it ugly because of that but Perl is the best language for quick dirty one liners that work. Only problem is that if you look at it later you will have problems understanding it.
Indeed. While you can get the rules regarding file name escaping right, the language does nothing to help you get it right every time, so unless you're careful someone will leave a file with a space (or, God help you, a newline) in the name in exactly the wrong place and blow it up.
How about OP does anyway? Bash as a scripting language is fucking great!
Since when is simplicity an argument against writing programs? Whether scripts or frameworks? "Hard to read" is not neccessarily an inherent trait[1] of the language, and more likely wrong on some PEBKAC level.
I have a customised environment at near 10k lines of bash in 5 projects, all of it in the correct tool for the job, aka a proper scripting language, so I can suggest another use for your thumb :-)
Bash as a scripting language is actually pretty amazing. It gives you everything you need to perform some quick-and-dirty tasks with minimal overhead. If you need only work on sequential data, files and processes, it's a perfect match.
It's not a full-fledged programming language by any stretch of the imagination (lacking structures more complex than associative arrays), but it's damn good for scripts of all sorts.
As an example, I've reimplemented a subset of Ansible (a command able to send "modules" on multiple machines via SSH and capturing+caching their output for subsequent queries) in ~150 lines of Bash. Considering that the size of Ansible, written in the more proper Python, is ~15000 LOC, I'd say Python is the much lesser scripting language.
Edit: to answer the OP's question, the documentation I've found most helpful to learn Bash is the one present on the Linux Documentation Project, with the page for arrays deserving special mention : http://tldp.org/LDP/abs/html/arrays.html. I spent a lot of time reading the manual before stumbling upon that documentation, and none of it really clicked until I had a few examples before my eyes.
Ansible has to take into account numerous edge cases, operating systems, backwards compatibility, etc. etc. Of course it's much, much bigger than your 150 lines bash script.
Typically a program like Ansible will have the majority of its use cases implemented in a minority of its code, while the rest of the code will be there to support special cases, edge cases, rare use cases, etc. So that contributes to the disparity in size.
Also, even if you manage to become better in Bash, you are bound to lose your skills at some point when you have been programming in other languages for a while.
I always have to look up how to do even basic things in Bash. I just don't use it often enough for these things to "stick".
If you have really limited resources, then bash or one of the other shells are the only tools at hand. Embedded devices might not give you enough disk space to get perl, ruby or python.
2. Conceive of use-cases you can't already solve, and see if you can find a way to do them using Bash.
3. Consider that perhaps Bash isn't the best tool for every job. (It most certainly isn't, though you can abuse it frightfully.)
4. Books. Jerry Peek's guides are getting rather dated, but they're still a good introduction.
5. Read the manpage. Frequently. Find some part of it that doesn't make sense, or that you haven't played with before, and play with it. Shell substitutions, readline editing, parameter substitution, shell functions, math, list expansions, loops, tests, are all high-payoff areas.
6. Take a hard look at zsh, which does a great deal Bash doesn't.
I wanted to get better at bash too, but instead I ended up getting everything[1] done at fish, which is cool, much better as a language, but no environment has fish pre-installed nowadays.
I'm ok at bash, but I do not default to complicated bash scripts for my needs. I make little reusable tools. For example I have a tool aliased that makes it easy to apply quick ruby code. For example
I find it's much faster to be productive like this than it is to try to do the same with ruby -e because I really only want to manipulate single incoming lines. I don't want to have to write the looping code or the code that sets variables or what have you.
Also, sometimes it gets confusing what tools are just bash functions or alias and which are scripts, so if you ever forget what a tools definition is just type:
type toolname
As for actually answering your question, look at your friend's dotfiles on their github account to learn which tools and tricks they use and when you don't know how something works ask them questions. People will usually point you in the right direction.
As an alternative, you could also look into PowerShell. it's open source and cross platform. I use it because it's really powerful on Windows.
In any programming language, you learn by practice. Given that your shell does so much, that's the easiest place to find tasks to practice on. I have been leaning on my shell scripts to do a lot of automation. The list is long and I just pick something from that list to work on for most days.
If you don't have system automation that you want to work on, then you probably have a lot of personal data that you can work with. I have scripts setup to manipulate data exports from the various services I consume and then remix that data in my own database. My shell scripts can get the data, operate on it and then shove it into a DB. Then I'll use something else to display that data.
For scripting, I recommend the rc shell from plan9, which is the one I use for my shell scripts. It is only when I want to share a script with other people that I consider using /bin/sh, and even then more often than not I've gone for rc.
The shell language itself is pretty simple and featureless. It relies on the system utilities for most things so once you know the basics of how the shell works you should probably focus on learning those rather than bash. Also I find that when it comes to interactive use having good key bindings and some nice aliases makes a lot of difference.
Flame war between bash/fish/zsh/powershell is almost meaningless to beginners, because the basic skills are common to all shells. (That said, you will love zsh once you use it)
I learned to use shell, about 7 years ago, by reading O'Reilly "Classic Shell Scripting". It is well written, and teach you something that you can hardly learn from google. But don't try to remember everything, especially those advanced string manipulation syntax, because one would usually use a scripting language such as ruby for advanced job.
The Tcl programming language is what shell scripting should be, basically. It is not just a language with all the features you need, it has explicit Unix shell alike scripting capabilities and strong DSL abilities. An example two-liner:
set files [glob /etc/*.conf]
foreach f $files {file lstat $f file_info; puts "$f: $file_info(size)"}
/etc/asl.conf: 1051
/etc/autofs.conf: 1935
/etc/dnsextd.conf: 2378
... and so forth ...
Also there is an `exec` command that supports pipes, redirections, and so forth:
set result [exec cat /etc/passwd | grep Directory]
The pipe has no special meaning in Tcl, but because of its DSL capabilities you can do things like that. Exec is a DSL basically.
194 comments
[ 2.4 ms ] story [ 247 ms ] threadThe Bash syntax is not daunting, or if it is that's never been the problem with Bash. The problem is that Bash or shell programming in general gives you a million ways to shoot yourself in the foot and maybe one or two obscure, funny-looking ways to do what you want. Like iterating over the files in a directory, for example. If you think that's easy in Bash you have either have a funny definition of easy or you're forgetting a few corner cases. Or getting the output from a program—did you know that $(my_prog) or `my_prog` modifies the program output?
For containers we do everything declaratively.
GOPATH isn't actually necessary except when you compile, and Python version problems mostly come up with larger programs you'd never dream of writing in Bash. 2.7 is dirt common even if you're using a slow-as-molasses-update-cycle LTS Linux.
Maybe I'm overlooking something...but why wouldn't this work:
for file in $(ls); do {<looped command>}; done
... works better for the easy edge cases, but still probably has some issues. Personally I think klodolph called it; once you get into anything that has a few interesting edge cases bash becomes pretty unwieldy.
This works fine with spaces and other strange characters, no need to come up with more complicated ways.
If you must use find, the correct way is really tricky and uses obscure bash features like settings IFS to the empty string to split on null bytes and process substitution so that the loop doesn't run in a subshell (which would prevent it from modifying variables in the current environment):
while IFS= read -r -d '' file; do ... done < <(find /tmp -type f -print0)
See http://mywiki.wooledge.org/BashFAQ/020
I think this demonstrates the point pretty well though - it takes a discussion among three of us before arriving at something that works moderately well, and we found four fairly different ways of doing it...
is both much easier and much safer than piping the output of ls.
It does the correct thing with filenames containing spaces or weird characters, and the intent is much more visible, so there is really no reason to use $(ls).
More details: http://mywiki.wooledge.org/ParsingLs
Aliases are something I use a lot - it's very basic but just having "big long command with options" aliases to something easy to remember makes it much more likely I will not make mistakes, can repeat it easily in loops.
Another thing that complements using bash effectively are using other applications config files. As soon as I have a new host to interact with I add it to my ssh.config file - then any scripting I need to do I don't need to deal with any special files. Other files like ~/.netrc or ~/.pgpass make my shell sessions that much more productive. For some reason many people don't bother ever updating these and rely on the shell history to do anything more than once.
CommandlineFu (http://www.commandlinefu.com/commands/browse) has some nice one liners and there's often some gems on ServerFault (https://serverfault.com/questions/tagged/bash) - just browsing those for topics matching your workflow can be very fruitful.
I've found the more I do anything at the shell of any complexity I end up writing a small python command line client to do any heavy lifting. Argparse (https://docs.python.org/3/library/argparse.html) makes this trivially easy and then I can use regular shell "glue" to combine a few commands together.
http://www.tldp.org/LDP/Bash-Beginners-Guide/html/
EDIT : Additional links -
Advanced - http://tldp.org/LDP/abs/html/
Bash programming - http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html
Next time, try to google "absg shell".
Here is the path for learning bash : https://learn-anything.xyz/operating-systems/unix/shells/bas...
learn from this wiki that has many tutorials and good examples http://wiki.bash-hackers.org/bash4
It runs about 70-80 pages, so counts as a small book.
Hrm. Make that 107 pages:
46,000 words.After that I started avoiding bashisms in my scripts, especially that they usually provide very little benefit at the cost of gambling against breaking silently on a barely related update.
Nowadays it's also worth noting that #!/bin/sh often (Debian derivatives) points to something that is not Bash, and people writing in shell/Bash usually don't understand the difference.
Anything that is not simple in bash gets hard to read and debug and probably is wrong on some subtle levels.
I have a rule of thumb that any shell script that grows beyond a screenful of lines gets redone in a proper scripting language.
Writing efficient bash code forces you to think about your problem differently. It's a very similar process to thinking functionally; e.g. you don't want to deal with lines of a file one at a time in a loop, you want to do filters and maps in languages like grep, sed and awk to deal with data in a streaming fashion with a minimum of forked processes.
Not pipefail as such, but Bash's error handling semantics (or lack there of) is pretty lame.
I would love something like that.
As a "real" shell, it looks like rc maintains the command, file and piping niceties of bash, whilst avoiding the edge-cases of shell-like embeddings.
As a rule of thumb, if a script starts to need arrays or to handle spaces in filename, I migrate it to Perl.
Error handling is also easier and more natural in Perl.
It's easy to use Perl strings as input or output, or files, and there's also ways to interact with streams as they go along.
Generally speaking the slightly more verbose interaction with shell is made up for by the savings when you can use Perl directly to do something rather than spawn a shell process for something simple. (Shell composition can be powerful, yes, but on the other hand spawning a full process and setting up a pipe for things like "tr" or "wc" is often just silly.)
I also personally believe there's a win in the syntax; you may say "What? 'tr a-z A-Z' is way more convenient than '["tr", "a-z", "A-Z"]' but I say one of the biggest and most pervasive errors in shell is to incorrectly set up the arguments by having something interpolated in incorrectly. Having an unambiguous syntax for separating arguments has often made it much safer for me to write the code that has to use the shell. Used correctly it can even be made to work in the face of user-supplied input, something that should generally not be combined with any form of implicit argument separation. (Although bear in mind that "used correctly" encompasses more than just "separating the arguments correctly.")
I am also NOT claiming exclusivity; this is just the thing I know off the top of my head. I'm sure the other scripting languages have nice libraries, too. Though watch out for them being too nice. There's a bit of an impedance mismatch between shell and scripting language. Anything that smooths it over too well is probably either giving up critical features or introducing subtle bugs, which can become security bugs in the face of arbitrary user input.
[1]: http://search.cpan.org/~toddr/IPC-Run-0.96/lib/IPC/Run.pm
Since when is simplicity an argument against writing programs? Whether scripts or frameworks? "Hard to read" is not neccessarily an inherent trait[1] of the language, and more likely wrong on some PEBKAC level.
I have a customised environment at near 10k lines of bash in 5 projects, all of it in the correct tool for the job, aka a proper scripting language, so I can suggest another use for your thumb :-)
1: https://www.reddit.com/r/commandline/comments/2kq8oa/the_mos...
It's not a full-fledged programming language by any stretch of the imagination (lacking structures more complex than associative arrays), but it's damn good for scripts of all sorts.
As an example, I've reimplemented a subset of Ansible (a command able to send "modules" on multiple machines via SSH and capturing+caching their output for subsequent queries) in ~150 lines of Bash. Considering that the size of Ansible, written in the more proper Python, is ~15000 LOC, I'd say Python is the much lesser scripting language.
Edit: to answer the OP's question, the documentation I've found most helpful to learn Bash is the one present on the Linux Documentation Project, with the page for arrays deserving special mention : http://tldp.org/LDP/abs/html/arrays.html. I spent a lot of time reading the manual before stumbling upon that documentation, and none of it really clicked until I had a few examples before my eyes.
Also, even if you manage to become better in Bash, you are bound to lose your skills at some point when you have been programming in other languages for a while.
I always have to look up how to do even basic things in Bash. I just don't use it often enough for these things to "stick".
https://github.com/EtiennePerot/parcimonie.sh/blob/48044f913...
2. Conceive of use-cases you can't already solve, and see if you can find a way to do them using Bash.
3. Consider that perhaps Bash isn't the best tool for every job. (It most certainly isn't, though you can abuse it frightfully.)
4. Books. Jerry Peek's guides are getting rather dated, but they're still a good introduction.
5. Read the manpage. Frequently. Find some part of it that doesn't make sense, or that you haven't played with before, and play with it. Shell substitutions, readline editing, parameter substitution, shell functions, math, list expansions, loops, tests, are all high-payoff areas.
6. Take a hard look at zsh, which does a great deal Bash doesn't.
[1]: https://github.com/koalaman/shellcheck
It will highlight common mistakes, and their wiki explains each one detail and how you should use an alternate, better implementation.
[1]: https://github.com/fiatjaf/react-site
echo "345.44
544.50" | rg "€#{l}: €#{(l.to_f * 3).to_i}"
Produces the following output:
€345.44: €1036
€544.50: €1633
Based on this code:
https://gist.github.com/zachaysan/4a31386f944ed31a3f8a920c85...
I find it's much faster to be productive like this than it is to try to do the same with ruby -e because I really only want to manipulate single incoming lines. I don't want to have to write the looping code or the code that sets variables or what have you.
Also, sometimes it gets confusing what tools are just bash functions or alias and which are scripts, so if you ever forget what a tools definition is just type:
type toolname
As for actually answering your question, look at your friend's dotfiles on their github account to learn which tools and tricks they use and when you don't know how something works ask them questions. People will usually point you in the right direction.
In any programming language, you learn by practice. Given that your shell does so much, that's the easiest place to find tasks to practice on. I have been leaning on my shell scripts to do a lot of automation. The list is long and I just pick something from that list to work on for most days.
If you don't have system automation that you want to work on, then you probably have a lot of personal data that you can work with. I have scripts setup to manipulate data exports from the various services I consume and then remix that data in my own database. My shell scripts can get the data, operate on it and then shove it into a DB. Then I'll use something else to display that data.
I invite you to read about it: http://doc.cat-v.org/plan_9/4th_edition/papers/rc.
I find the control structures simpler and more elegant, and overall its design feels more consistent. For example, consider an if statement in bash:
And now in rc: Or a case statement in bash: And expressed in rc: In the past, I've used it as my shell too, but now I use it only for scripting. I think you can install it in most platforms.I learned to use shell, about 7 years ago, by reading O'Reilly "Classic Shell Scripting". It is well written, and teach you something that you can hardly learn from google. But don't try to remember everything, especially those advanced string manipulation syntax, because one would usually use a scripting language such as ruby for advanced job.
I can second this recommendation. :)
Everything will take longer but imho it's the only way to get better.
http://www.commandlinefu.com
http://www.shell-fu.org