bashumerate — iterate over files, lines, ranges, or lists with a consistent {} syntax. No for loops, no find -exec, no xargs flags to remember.
enumerate -f '*.sh' -- wc -l {}
enumerate -L a b c -- 'echo {}'
Under 150 lines of bash, pluable sources, NUL-safe.
https://github.com/wallach-game/bashumerate
Not sure I see the point of this. Remembering a bunch of options on one tool is no better than a bunch of tools/constructs? Especially if the latter are useful elsewhere. And it can't be used for scripts unless you start shipping it alongside, which… nah.
On the topic of xargs replacements, I love gnu parallel.
The --dry-run flag of parallel made me confident to do more batch processing than I ever did with xargs.
Parallel has an option for almost everything, it's almost too much.
But I have shopped around for alternatives.
The creator Ole Tange maintains a painstakingly long article of the alternatives and their differences. [0]
The gnu parallel book and reading materials [1] are excellent too.
I wanted something simpler — one consistent way to iterate over anything.
That's not actually simpler though. Simple is removing everything unnecessary. You took commands which could already do what you wanted, and added an extra program which calls them in specific ways. This will add bugs and maintenance headaches, not be portable, etc. This is added complexity.
The reason you made this script is not because you wanted simpler, you wanted easier. There's nothing wrong with that, and I'll grant you it probably is, especially for those unaccustomed to these commands. But easier != simpler. Often you'll find that simple is hard and easy is complexity deferred.
I have found that the most reliable way I like is to just construct the command externally and then pass to gnu parallel (mostly for --eta and --tmuxpane). And the great thing is, as others say, xargs -I. I prefer for shortness (and few collisoins), '@'
Good idea but strange syntax. It would be more idiomatic if the command came first and the list of globs last.
Additionally the use of "--" is not what everybody expects: here it is used to introduce one argument, the command, while it's usually meant to introduce multiple arguments without worrying if they have a leading "-".
A possible revised syntax with command as the first argument followed by a list of globs optionally introduced by "--" would allow to enumerate all files with a leading "-", which the current syntax cannot:
enumerate 'whatever {}' -- '-*'
I'm assuming "-f" for simplicity, but the same reasoning holds for "-L" too.
> Or maybe you pipe into xargs and pray your filenames don’t have spaces…
Always use -0. Most gnu utilities support it. It makes them put a null byte after every filename instead of a newline. Completely eliminates the problem of dealing with whitespace in the filenames.
> Or maybe you pipe into xargs and pray your filenames don’t have spaces…
Most of this, if not all, is fixable by adding a `export IFS=$'\n'` to your bashrc. I'm not trying to disregard your project, just point out something that took me years to learn and I currently use extensively to solve this very problem. Perhaps you didn't know about it until now... :)
No, because `wc` accepts multiple files. And the example given is incorrect for any file having a `.txt` suffix and whitespaces.
> Or this?
find . -name '*.sh' -exec wc -l {} +
No.
> These all work, but each has its own syntax, its own flags, its own quirks. I wanted something simpler — one consistent way to iterate over anything.
And therein lies the proverbial xkcd standards[0] proof.
find -print0...|xargs -0...
works for me, and i don't always want to execute something and xargs can run parallel processes. i feel like this guy never bothered to read the man page.
I don’t want to be a downer here, but the structure of this repo and the verbosity+language of the docs feel 80% vibecoded. Not that that’s wrong; I just feel kinda gullible for even clicking on this.
One uses xargs or parallel only a few times before they remember some of the quirks that but them. And then they become cautious. And then it’s muscle memory. And if it’s not an often occurrence, they learn to check the man page.
Anyways, in a world of “vibecoding” why add another “tool” to the mix when the LLMs have been trained on all our stackoverflow-posted grievances to begin with?
If you want a shell to interact with the results, you can of course just use a (sub)shell.
ls -1 ./*.sh | xargs -rd\\n sh -c 'for i in "$@" ; do ... ; done' sh
1. not strictly necessary to use -1 as I believe all common ls detect !isatty(stdout) and produce line-by-line output anyway.
2. xargs -r just doesn't run the command if there's no input, also not strictly necessary but I'm addicted to using it because it's the sensible default to me.
3. xargs -d\\n makes it collect fields as full lines, which is what you typically want, unless you're able to generate NULs.
4. use whatever shell you want of course, but I don't use bashisms, etc., by default, /bin/sh is fine for me, even if it's dash.
5. the trailing "sh" at the end is due to a quirk of `sh -c` usage, where $0 is the first non option argument, so `printf %s\\n 1 2 3 | xargs -rd\\n sh -c 'for i in "$@" ; do printf "%s " "$i" ; done ; echo'` (note the lack of trailing "sh") would only print "2 3 " as $0 is not included in "$@" ($0 is 1, $1 is 2, $2 is 3). It's very easy to just always give the shell name itself manually as $0 instead of trying to ingest "$0" into your logic.
Of course, you could `find . -maxdepth 1 -type f -name '*.sh' -print0 | xargs -r0 ...` instead, depending on what you're up to, that may be the easiest. It's definitely the simplest -- as long as your xargs has -0 support.
In this thread: bash experts with arcane knowledge, unintentionally demonstrating how awful bash is.
The obvious solution would be to use something more sane, like PowerShell or nushell, but instead old experts will always defend the skills they have honed for years, while criticizing anything that's different.
Yes, Bash is filled with footguns and is awful due to that. However, it's still incredibly useful as it's everywhere and has a far greater lifespan than almost anything else. You can write scripts in PowerShell or nushell, but then find that twenty years later they're no longer usable or you find a twenty year old machine that won't have PowerShell/nushell installed.
It's not so much about defending arcane scripting skills, but that Bash functions as a lowest common denominator and is useful because of that. If you want something that works reliably over decades, then it's best not to go for an "improved" shell as it may not still be around.
I like to think of Bash script writing as the opposite of riding a bike - you have to relearn it almost every time you write a script.
This is a sleeper option, but 'xonsh' as a Python superset is a very pleasant experience, especially when you just don't feel like learning another bespoke language for the complex stuff.
I once had to scrape the output out of something hacked together by a grad student 10 years before me, scrape yet another hacked together application, do some fancy numerics, and then plonk it into a database. I did it without tears!
As others have pointed out throughout the thread, the reason we defend the skills is that there’s a high likelihood that these tools will always be available on any machine; PowerShell and nushell, not so much.
Given the alternative proposed by OP was more of the same but with layers on top, saying "how about just using xargs and sh anyway" is a simplification -- learn 2 tiny things vs start depending on some new tool whose only purpose is to let you avoid learning those 2 tiny things by adding template language, etc., on top? I'll learn the 2.
As for widening the scope to other tools, I've written about that on here before, how I have lots of powershell experience, some nushell, etc. I haven't become a convert (definitely not on Linux, of course occasionally on Windows).
Happy to respond to any input or anything, but you should reply directly to people if you're going to criticize them -- very poor form to blanket criticize while also taking an uncharitable read.
1. Things have developed over time. Bash and other shells weren't always this way. If you have ever touched different shells, awk, sed and so on and then touch perl you see how it is basically just the glue between the other tools put into one language.
2. The majority of scripts is written in sh or bash. So these shells won't go away anytime soon and any newcomer will be confronted with them. So it's best to know all the edge cases before you do stuff you don't want to do. And yes, you could always install another shell. But that brings another dependency and opens up another can of worms. In professional environments it's not really an option to have the next exotic language.
It's like having a tmux.conf on your local machine that configures tmux exactly the way you want it. Nice to have but the moment you touch any of the other billions of systems out there that run the default configuration, you might be lost because you never learned the default and only use your modified version.
I'm just replying reflexively because suggesting Powershell on Linux is an abomination, and because I didn't know about nushell, and it looks awesome. I wonder why the documentation is still that incomplete, though.
I like Bash for its portability and pretty much no other reason. That one attribute of it is pretty hard to replicate with any other tool, at least over short time horizons
62 comments
[ 0.26 ms ] story [ 45.6 ms ] threadJust go with
The --dry-run flag of parallel made me confident to do more batch processing than I ever did with xargs.
Parallel has an option for almost everything, it's almost too much.
But I have shopped around for alternatives. The creator Ole Tange maintains a painstakingly long article of the alternatives and their differences. [0]
The gnu parallel book and reading materials [1] are excellent too.
[0] https://www.gnu.org/software/parallel/parallel_alternatives....
[1] https://www.gnu.org/software/parallel/#Tutorial
That's not actually simpler though. Simple is removing everything unnecessary. You took commands which could already do what you wanted, and added an extra program which calls them in specific ways. This will add bugs and maintenance headaches, not be portable, etc. This is added complexity.
The reason you made this script is not because you wanted simpler, you wanted easier. There's nothing wrong with that, and I'll grant you it probably is, especially for those unaccustomed to these commands. But easier != simpler. Often you'll find that simple is hard and easy is complexity deferred.
seq 1 10|xargs -I@ echo 'bash run.py @'|parallel -j 10
I know the echo is a little silly but then I can remove the |parallel and see if it's right. And if I don't want parallelism, I just pass to bash
This null termination is now a POSIX standard.
Additionally the use of "--" is not what everybody expects: here it is used to introduce one argument, the command, while it's usually meant to introduce multiple arguments without worrying if they have a leading "-".
A possible revised syntax with command as the first argument followed by a list of globs optionally introduced by "--" would allow to enumerate all files with a leading "-", which the current syntax cannot:
I'm assuming "-f" for simplicity, but the same reasoning holds for "-L" too.Always use -0. Most gnu utilities support it. It makes them put a null byte after every filename instead of a newline. Completely eliminates the problem of dealing with whitespace in the filenames.
Most of this, if not all, is fixable by adding a `export IFS=$'\n'` to your bashrc. I'm not trying to disregard your project, just point out something that took me years to learn and I currently use extensively to solve this very problem. Perhaps you didn't know about it until now... :)
> Or this?
No.> These all work, but each has its own syntax, its own flags, its own quirks. I wanted something simpler — one consistent way to iterate over anything.
And therein lies the proverbial xkcd standards[0] proof.
0 - https://xkcd.com/927/
Fails : enumerate -f '.txt' -- 'wc -l {}'
For all use cases
Correct : enumerate -f '.txt' -- 'wc -l -- {}'
One uses xargs or parallel only a few times before they remember some of the quirks that but them. And then they become cautious. And then it’s muscle memory. And if it’s not an often occurrence, they learn to check the man page.
Anyways, in a world of “vibecoding” why add another “tool” to the mix when the LLMs have been trained on all our stackoverflow-posted grievances to begin with?
2. xargs -r just doesn't run the command if there's no input, also not strictly necessary but I'm addicted to using it because it's the sensible default to me.
3. xargs -d\\n makes it collect fields as full lines, which is what you typically want, unless you're able to generate NULs.
4. use whatever shell you want of course, but I don't use bashisms, etc., by default, /bin/sh is fine for me, even if it's dash.
5. the trailing "sh" at the end is due to a quirk of `sh -c` usage, where $0 is the first non option argument, so `printf %s\\n 1 2 3 | xargs -rd\\n sh -c 'for i in "$@" ; do printf "%s " "$i" ; done ; echo'` (note the lack of trailing "sh") would only print "2 3 " as $0 is not included in "$@" ($0 is 1, $1 is 2, $2 is 3). It's very easy to just always give the shell name itself manually as $0 instead of trying to ingest "$0" into your logic.
Of course, you could `find . -maxdepth 1 -type f -name '*.sh' -print0 | xargs -r0 ...` instead, depending on what you're up to, that may be the easiest. It's definitely the simplest -- as long as your xargs has -0 support.
The obvious solution would be to use something more sane, like PowerShell or nushell, but instead old experts will always defend the skills they have honed for years, while criticizing anything that's different.
It's not so much about defending arcane scripting skills, but that Bash functions as a lowest common denominator and is useful because of that. If you want something that works reliably over decades, then it's best not to go for an "improved" shell as it may not still be around.
I like to think of Bash script writing as the opposite of riding a bike - you have to relearn it almost every time you write a script.
I once had to scrape the output out of something hacked together by a grad student 10 years before me, scrape yet another hacked together application, do some fancy numerics, and then plonk it into a database. I did it without tears!
As for widening the scope to other tools, I've written about that on here before, how I have lots of powershell experience, some nushell, etc. I haven't become a convert (definitely not on Linux, of course occasionally on Windows).
Happy to respond to any input or anything, but you should reply directly to people if you're going to criticize them -- very poor form to blanket criticize while also taking an uncharitable read.
1. Things have developed over time. Bash and other shells weren't always this way. If you have ever touched different shells, awk, sed and so on and then touch perl you see how it is basically just the glue between the other tools put into one language.
2. The majority of scripts is written in sh or bash. So these shells won't go away anytime soon and any newcomer will be confronted with them. So it's best to know all the edge cases before you do stuff you don't want to do. And yes, you could always install another shell. But that brings another dependency and opens up another can of worms. In professional environments it's not really an option to have the next exotic language.
It's like having a tmux.conf on your local machine that configures tmux exactly the way you want it. Nice to have but the moment you touch any of the other billions of systems out there that run the default configuration, you might be lost because you never learned the default and only use your modified version.
I prefer Almquist (ash), eg., NetBSD sh, Debian sh with tab completion, busybox ash, etc.
Powershell must be slow and/or bloated like Bash; someone wrote xargs in C++ for Windows:
https://github.com/idigdoug/TextTools/tree/main/wargs