HN Explorer

Ask HN: What to use instead of Bash / Sh for scripting?

52 points by lordgroff ↗ HN
I'm at the point where I feel a certain fatigue writing Bash scripts, but I am just not sure of what the alternative is for medium sized (say, ~150-500 LOC) scripts.

The common refrain of "use Python" hasn't really worked fantastically: I don't know what version of Python I'm going to have on the system, installing dependencies is not fun, shelling out when needed is not pleasant, and the size of program always seemingly doubles.

I'm willing to accept something that's not on the system as long as it's one smallish binary that's available in multiple architectures. Right now, I've settled on (ab)using jq, using it whenever tasks get too complex, but I'm wondering if anyone else has found a better way that should also hopefully not be completely a black box to my colleagues?

105 comments

[ 1.8 ms ] story [ 36.1 ms ] thread
auslegung
I’ve seen people use Go because it’s a simpler language. It’s what I would probably use if I didn’t want to use bash.

What do you think about editing the title to include “for scripting”? I thought you were asking for recommendations like zsh or fish when I first clicked

magicpointer
Also go has the advantage of producing a single static binary. Easy to build on your machine and run on another host.
berkes
Which is why I'm writing most of my tooling in Rust.

Deployment or installation is just one scp away.

not rewriting, just when It needs large refactoring, or is currently a mess, or brand new.

brundolf
Assuming you're deploying to the same OS/architecture that you're working from, otherwise you'd have to rebuild (which Rust makes easy to do, but still)
Raed667
How easy is it to cross-compile with Rust?
brundolf
Trivial. All of the tooling, dependency management, linking, etc. is neatly bundled together under cargo. Just run "cargo build" for the current system, or specify one or more other architectures as arguments to get multiple binaries. Everything's statically-linked so you should be able to scp those wherever.

Still, when we're talking about casual "scripts" even this extra step seems a little bit burdensome

berkes
True.

Though in all those cases, for me, all alternatives are just as bad or worse.

Getting ansible running on a windows server is... unfortunate. Ensuring that your bash script runs on OSX is easier, but I've had lots of unexpected issues with tools like grep or find working just a tiny bit different. And so on.

brundolf
Node scripts, even with dependencies, are pretty much 100% equivalent across all platforms :)
cpach
You might want to experiment with Powershell. It runs on Linux.

You might not adopt it, but it can give perspective on the pros and cons that comes with Bash.

majkinetor
Just use PowerShell. You can do so much in it without any dependency (jq is native in it and many other things).

It is cross-platform so you can use your scripts anywhere.

Don't use python or go or ruby or whatever as those are not scripting languages.

IMO Bash is horror and should die already.

lordgroff
I use Powershell in Windows and am a big fan. It is a biiiiiiiiiiiiiiiiiiiig download on Linux, making it pretty impractical managing many different Linux machines.
majkinetor
> It is a biiiiiiiiiiiiiiiiiiiig download

How big really? Can't be more then 200MB? Should be less then minute to download and install? If you have docker, can you go that route?

Also, if you account the fact that you have variants of many other popular tools included OTB is that really a downside ?

That is certainly different concern, you want fast and practical script programming and why would you care if it has first time latency of couple of minutes if you are set good forever after that ?

scaladev
I just checked, if you already have .NET installed, the powershell package is about 20 MBs, which is… tolerable. However, I see that the latest version still starts very slowly compared to *nix shells (and also Python).

The shell was running from tmpfs (so basically a RAM disk). I have a decent processor and an NVMe SSD, and this is what I see (it was tested 10 times in a row to pull everything into the page cache properly, and the numbers are pretty stable):

  $ time ./pwsh <<<"" >/dev/null
  
  real 0m1.050s
  user 0m1.464s
  sys 0m0.067s
More than a second to execute an empty 'script', and even longer to get to the interactive prompt (which is harder to time properly so I'm not publishing any numbers).

It's even slower on an older Windows laptop which I have lying around (starts in about 3-4 seconds). Absolutely terrible for an interactive shell/script execution engine.

Just for comparison's sake, the same laptop can run an empty bash script in 2-3 milliseconds. This is very important when running build jobs, for example.

majkinetor
How about pwsh -NoLogo -NoProfile ?

    > Measure-Command { pwsh -NoProfile -NoLogo exit }
    
    Days              : 0
    Hours             : 0
    Minutes           : 0
    Seconds           : 0
    Milliseconds      : 254
BTW, Windows OS is often subject of Windows Defender and other AVs which are not present on Linux machines and their slowdown of the system is phenomenal.
scaladev
Thanks for the suggestion. Adding these flags halves the startup time:

  $ time ./pwsh -NoProfile -NoLogo <<<'' >/dev/null

  real 0m0.456s
  user 0m0.388s
  sys 0m0.040s
Much better, but I still wouldn't use this even as an interactive shell, as I open a few hundred terminal windows each day.

There's no defender (or anything similar) on that other machine. It also has an SSD. With an older HDD powershell started in about 10-15 seconds there (while cold startup time for bash was something like a second, and instantly afterwards).

majkinetor
I open few hundred terminals each day. There is something seriously wrong with your machines. Pwsh is not a king of speed but 10-15 is local problem. Anything more then 2 seconds is a local problem. You can maybe get to 5 with lots of modules in your profile.
bsg75
Python is not a scripting language?
pskinner
Powershell is a horror you mean, it's got the verbosity of objective c with the consistency that comes with all Microsoft designed products.

Just use a proper scripting language.

majkinetor
No, I said what I mean. Bash and its disasters like [ as executable and mega error prone and awkward language designed 50 years ago IS horror.

Verbosity of powershell is optional, I don't use it.

> consistency that comes with all Microsoft designed products.

Oh, lets not troll here.

jolmg
> its disasters like [ as executable

Why is that a disaster? What problem has it caused you that wouldn't have happened if it wasn't an executable?

[[ is a non-executable version of [. Does that solve your problem?

majkinetor
I don't have a problem - I don't use bash unless its few lines or absolute must.

Its disaster because launching process to achieve simple math expression is just ridiculous, besides being slow and besides having to terminate it with ;. Its wrong on so many levels I don't even know where to start, they must have been smoking some nasty things back in time.

michaelcampbell
> launching process to achieve simple math expression is just ridiculous

That's why you don't do that. $(( math goes here ))

majkinetor
That is also ridiculous as it starts subshell/subprocess.
[deleted]
(comment deleted)
jolmg
Neither [ nor $(()) start subshells/subprocesses. [ is built-in in pretty much every shell, and there's absolutely no reason for any shell to make a subshell/subprocess for $(()). You might be confusing it with $().

In any case, worrying about subprocesses and their slowness in a shell is completely moot. Shells aren't built to run fast, and are rather built to work with executables conveniently.

> besides having to terminate it with ;

Is that really a complaint? Does that bother you a lot? This seems so petty, I'm a bit lost for words.

[ is a regular command because the control structures of the shell are flexible enough to support any command for their conditions.

I don't know if you're asking for the shell to make a super-special exception in its syntax for that particular command just so that people won't have to type ; under certain circumstances, or if you're asking for control structures to be more restricted and allow only [ instead of any command for their conditions.

> they must have been smoking some nasty things back in time.

So far, you've given the impression of someone complaining that screwdrivers really suck for inserting nails. I wonder if the issue is that you've never seen hammers before or if you've never seen screws before. Or maybe the issue is you believe each tool should be equally fit for everything?

michaelcampbell
Demonstrably false.

And even if not, so what? Every argument you make is a shifting of a goalpost.

pskinner
Disasters like powershell returning a 0 exit code even if there is an error? Its a pile of trash that doesn't add anything new other than a big shiny MS logo.
majkinetor
That is simply not true. RTFM - $ErrorLevel is for native apps $? is for cmdlets
2OEH8eoCRo0
Perl is nice and ubiquitous.
hedora
Agreed. Also, Perl’s sweet spot is 1-500 lines of code. (Beyond that, not having types starts to hurt.)

Also, for scripting stuff, my (readable, extensible) perl is about 10% as long as other people’s python, so 500 lines of perl ends up being 5000 lines of python. That creates a huge difference in maintainability and correctness.

With code golf, my perl can be 10x shorter than that, but then it is neither readable nor extensible, and doesn’t scale to 500 (or even 50) lines before becoming more trouble than it’s worth.

elviejo
I've been wondering the same myself... I want: - reliable scripts::in my case this means strong typing. Because for example currently to see the exit status of a program one need to check the exit number 0 or non zero plus the string for the command. - crosplatform:: this eliminates interpreted languages like ruby and python - easy to interact with the environment :: is something that like bash can actually call programs installed on the computer.

As the song says: "But I still... haven't found... what I'm looking for."

hedora
Go might be a decent choice. It’s basically python, but with reasonable type and error handling.

It still has other python-style problems, like epic dependency graphs, with all the compatibility and bitrot nightmares those imply.

brundolf
I recommended Node in another comment, but if you want typing I'd recommend Deno. Similar reasons, but add the fact that you get type checks without a build step.
nailer
With typescript installed:

usr bin env ts-node

On phone but you get what I mean

openfuture
With nix you can make anything into a script, there's some people using haskell for example.

I tend to use common lisp but I'm learning guile scheme and will probably move to that instead.

hollerith
I don't understand (even though Nixos was my only OS for a couple of months).

How would Nix help me make Haskell into a script?

jiehong
I think OP means that it would allow you to pull it from anywhere you have nix installed...
exdsq
I'd say Go might be your best bet - it's easy to generate executables, you get a simple type system, and it's pretty easy for anyone to pick up and modify a program in. However another alternative is to use the language of the project so it's uniform. Most languages are pretty usable for scripting.
brundolf
I really like NodeJS for more complex system-scripting. In contrast with Python:

1) There aren't two disjoint versions to juggle

2) Dependency installation is clean and contained

3) Dependencies can easily be installed locally or globally, depending on your preference for isolation vs reuse

4) Lots of packages are available for doing cross-platform (read: Windows compatible) operations

5) JS makes it really easy to do efficient async tasks, for example high-IO scripting involving reading and writing lots of files and making network requests (without any dependencies, I might add)

At my last company I converted all our shell scripts to Node scripts when the pandemic hit, so they'd work predictably on Windows machines, and it was great. These days I'd do it that way to begin with.

lordgroff
JavaScript as a shell replacement, what a time to be alive!

I am familiar with and use quite a bit of js on the front end, but how is node for shelling out? This is something I find Python to be quite terrible at.

brundolf
I wouldn't use it as an interactive prompt personally - it's a bit too verbose - though I did know someone in college who did that :)

Calling shell commands is reasonably straightforward:

  const { exec} = require('child_process')
  const child = exec('ls', ['-lh', '/usr'])
  console.log('error', child.error)
  console.log('stdout ', child.stdout)
  console.log('stderr ', child.stderr)
However, you can also get cool packages like this to use instead: https://github.com/shelljs/shelljs

I will admit though that in most of the cases where I've used it I didn't really do any shelling-out, so I can't speak much to how that experience scales in practice. Mostly I was either 1) doing self-contained logic with files, web requests, etc, or 2) delegating out to JS build tools (which even when they have a CLI, almost always expose a nice JS API too). Ymmv, etc.

mekster
It's far easier to do it on perl.

$output = `ls -lh /usr`;

I don't think I can remember all the boilerplate of using the wrong language for the job if I only happen to write that once a month or less.

Even PHP is easy.

$output = shell_exec('ls -lh /usr');

sbacic
Let me put it this way; I recently wrote a script to parse the output of last and format it.

The Node version was some 90 LoC but if I included the dependencies it came to about 5 MB. Then I rewrote the same script in Python3 and it was 60 LoC and no dependencies that needed to be installed separately.

I will admit that the JS code looked _a lot_ cleaner than the the Python code though.

schwartzworld
Node is great for shell scripts. Async programming is a breeze in JavaScript. I wrote a quick wrapper function around child_process.exec to promisify it, and this allows me to use features like async / await and Promise.all in my scripts.
chatmasta
I tend to write a lot of bash, partially for masochistic reasons, but I would probably choose Node over Python in 2021 for basic dev tooling scripts. Setting up a node environment with nvm can be done in a few lines of code, and it’s pretty reliable across platforms. Python is just too difficult to support consistently.

Really you should pick whatever you currently have in your codebase. We happen to have both Python and Node, with the languages generally being developed separately by different devs. So we’ve got tools in both, but a Node developer is unlikely to ever need/use the Python tools (although they can all run in Docker anyway). What annoys me a bit is that pre-commit requires a Python env to bootstrap itself, but it can install nvm without any fuss.

002445
I use Fish as my shell instead of bash precisely because I really like the scripting language for my personal scripts.

It's not the best fit for everyone but it has served me well.

runjake
Just use shell or Python3 and stop nitpicking IMHO. There aren’t really any other choices.

Most OSes come with relatively modern versions of either. Pretty much everyone is or should be familiar with either. It’s not like it’s some Lispy thing.

westurner
A configuration management system may have you write e.g. YAML with Jinja2 so that you don't reinvent the idempotent wheel.

It's really easy to write dangerous shell scripts ("${@}" vs ${@} for example) and also easy to write dangerous Python scripts (cmd="{}; {}").

Sarge is one way to use subprocess in Python. https://sarge.readthedocs.io/en/latest/

If you're doing installation and configuration, the most team-maintainable thing is to avoid custom code and work with a configuration management system test runner.

When you "A shell script will be fine, all I have to do is [...]" and then you realize that you need a portable POSIX shell script and to be merged it must have actual automated tests of things that are supposed to run as root - now in a fresh vm/container for testing - and manual verification of `set +xev` output isn't an automated assertion.

westurner
> avoid custom code and work with a configuration management system test runner

ansible-molecule is a test runner for Ansible playbooks that can create VMs or containers on local or remote resources.

You can definitely just call shell scripts from Ansible, but the (parallel) script output is only logged after the script returns a return code unless you pipe the script output somewhere and tail that .

> manual verification of `set +xev` output isn't an automated assertion.

From "Bash Error Handling" https://news.ycombinator.com/item?id=24745833 : you can display the line number in `set -x` output by setting $PS4:

  export PS4='+(${BASH_SOURCE}:${LINENO}) '
  set -x
But that's no substitute for automated tests and a test runner that produces e.g. TAP output from test runner results: http://testanything.org/producers.html#shell
ThePhysicist
Perl (5). It's an incredibly capable scripting language, many things essential for Shell programming are much better integrated than in Python (e.g. regular expressions). It gets a lot of hate these days but it's still a damn fine language and perfectly suited to shell programming (and larger projects).
lordgroff
One of the things about Perl is that I can't quite believe myself how far it has fallen.

I was never a Perl monk by any means but I knew enough to get by. And why did I not even think about it? I haven't coded Perl in years, but if my memory serves me correctly, bundling extra dependencies in Perl was a heck of a lot less painful than Python too.

ksherlock
What dependencies? OP wants to replace a bash/sh script, not left pad a string.
ThePhysicist
Yes, Perl's package ecosystem CPAN (the "Comprehensive Perl Archive Network") was lightyears ahead of other systems at the time (late 90s / early 2000s), it provided modules for everything, which made you so incredibly productive. You need a DNS library? Just go to CPAN. CGI functionality? CPAN. Parsing some obscure file format? CPAN.

Really sad it lost so much traction in the last decade, I partially blame it on the failed Perl6 endeavour. I also mostly use Python these days but looking at my old Perl scripts Python really doesn't come close in terms of succintness.

ilyash
Next Generation Shell was born exactly out of the pain of using bash and Python. I'm the author and I felt the same way.

> shelling out when needed is not pleasant,

Handled properly because the language was built ground up for Ops tasks. There is even a syntax for "run a command and parse the output". Rationale and examples - https://ngs-lang.org/doc/latest/man/ngswhy.1.html

Currently NGS supports Linux and MacOS.

> smallish binary

Unfortunately there is no static build yet so can't check this checkbox

Next Generation Shell - https://github.com/ngs-lang/ngs

Alternative Shells list by @chubot - https://github.com/oilshell/oil/wiki/Alternative-Shells

max_hammer
If you are ready to invest some time then try `perl`. It is awesome.
TheGrassyKnoll
I tried using Python with import sh for awhile, but...meh.

  https://amoffat.github.io/sh/sections/faq.html
curun1r
This won’t work for many people who don’t know the language or its ecosystem, but I’ve started doing my ad-hoc scripting in Rust using rust-script [0]. I’ve got a template with lots of boilerplate for everything I might need and I just copy it and delete the parts I don’t want. Having argument parsing, dotenv init, toml configuration and such ready to go with just a few tweaks, simple user prompting with dialoguer [1] and easy delegation to other tools with cmd_lib [2] means I can often write the tool I’m envisioning in just a couple of minutes.

[0] https://rust-script.org/

[1] https://crates.io/crates/dialoguer

[2] https://crates.io/crates/cmd_lib

JohnL4
Can you be more specific than "fatigue"? That sounds like ennui, in which case maybe consider a job switch instead of a language switch (not being sarcastic or intentionally mean). Bash plus all the Unix tools (awk, grep, sed, tsort, comm, the ones I don't know, etc.) have pretty long legs, in terms of functionality and the syntax is pretty understandable. They provide most of what Perl offers, apart from OO (-ish) and line noise.

If you want strong typing (are you scripting in the large?), then, almost by definition, no scripting language provides that. If you need performance, switch to a compiled language (or a jit-compiled interpreted language) that most of your co-workers understand.

If Python brings too many dependencies, I guess you're trying to do something that is way out of the realm of possibility for bash, so you're not looking for an "alternative" to bash, you're looking for the best language for whatever project you're undertaking. And you may not be able to get away from the dependencies if you're trying something specific/sophisticated.

majkinetor
> If you want strong typing (are you scripting in the large?), then, almost by definition, no scripting language provides that.

Powershell is basically strongly typed.

> Can you be more specific than "fatigue"? That sounds like ennui

I love my job, but bash makes me wanna hang myself. I can say the same about stored procedures. Its like going back to cobol days. Some people have strong tastes I guess.

diroussel
I used “true” instead of “$True” in a power shell script. It didn’t error. It didn’t warn. It just evaluated to false.

So doesn’t behave like a strongly typed language to me.

majkinetor
RTFM - if you don't specify type, it is infered.

    [bool] $x = "true"
    MetadataError: Cannot convert value "System.String" to type "System.Boolean". Boolean parameters accept only Boolean values and numbers, such as $True, $False, 1 or 0.
JohnL4
Yeah, i forgot to add my thoughts on PowerShell. It's great for Windows and nothing else. Its power isn't syntax (which is a bit weird) but the fact that it interfaces with all the Windows APIs. I can't really imagine it on any other platform. If you're on Windows and nowhere else, PS is superior to bash. Have you been using bash on Windows? If so, why? Cross-Platform?
majkinetor
PS is superior to bash, period, no matter platform. You are talking about something else entirely, like added benefit of lots of Windows tools supporting PowerShell automation. While that is nice, the major appeal of pwsh for me is not the ecosystem but the paradigm and language - in bash, you always do something irrelevant to your current task as anything requires excessive boilerplate and all supportive utilities are reinventing stuff like parameter names and parsing, error handling, etc. Also, those utilities are sometimes different on other *nix platforms, so you can't even count on linux portability.

So yeah, PowerShell is huge on Windows and just awesome on Linux. You must also take into account that its only few years old in that space.

zbuf
I don't think the breaking point is the number of lines in the script; your pain point is probably one or two specific concepts or features? (or lack of)

Shell is likely to remain the best way to connect together other pieces of software.

The tipping point, in my experience, is folks trying to use arrays or other buffering of data. At that point they're writing in shell, but wishing for a procedural language.

Whereas, shell works excellently (and incredibly efficiently) if you can compose your task strictly into streams of data (a more 'functional' feel)

So the real answer I think is specific to the types of script you are writing!

But also it could be worth revisiting the style in which you're using shell; especially if you can call upon your own helper programs (eg. in Python) where you require buffering or non-streamed access to data.

Whilst there's a lot of documentation of sh/bash functionality, I don't think there's anything like the same momentum behind 'best practices' as other languages like Python or even C.

akagusu
The question you should ask yourself is not what use instead of bash but what is the problem you are trying to solve and what is the best way to solve it.

It's hard for us to help you with you giving us some context.

150-500 loc of bash are not that much, specially when you combine bash with standard Unix utilities.

These scripts you wrote, for what do you use them? Are they executed by humans or machines? On which platform? Linux, Mac, Windows or all?

You mentioned jq. Are you parsing json from HTTP requests or files? And why are you parsing json with bash?

Without a better context, A lot of people will say to use Python and Python is a good choice with you are running you scripts on Linux because you probably already have Python 3 installed and whatever you are writing with 500 loc of bash can be accomplished with zero dependencies with Python standard library.

If relying on the Python runtime is really a problem for you, your next best option is Go. Go is easy to learn and easy to use for building small cli apps with only the standard library, but even if you need some dependency, you will ship a single binary, so no problems with runtime compatibility or dependencies.

zzo38computer
Depends on the program. Sometimes I use shell scripts, and sometimes I use Node.js, and (more rarely) PostScript.
quickthrower2
Like others said it depends on each problem you are solving. Consider Java, nodejs, go, rust, ruby, python, c# - the one you know best is the best. Build up some Util classes for stuff you do regularly. NPM has a package for everything eg there is a cron so I like it for that reason.

Consider docker for some situations. Consider kubernetes if the need is more devops/ running a production system.

Consider powershell if on windows.

Travis ci / Jenkins / appveyor etc. for CI

Be lazy - can I get out of writing this code should be in the forefront of your mind (for any code!). The answer might be no but consider it.

No code tools can be considered - Google sheets, Wordpress, plethora of saas to do this. Paas eg aws lambda etc.

So many options. Solve the problem, rather than transliterate bash.

Effortfully
Honestly it depends on what you're solving, personally I use both.