Tell HN: Please don't print –-help to stderr in your CLI tools
Imagine you get a lengthy help description which then you pipe to less.. and you only get (END) in your terminal. Turns out the author decided to print help message to stderr instead of stdout. I assume newcomers will be as confused as I was when it happened to me for the first time. GNU utils use stdout for help texts, and so should you.
155 comments
[ 4.1 ms ] story [ 158 ms ] threadhttps://docs.oracle.com/en/java/javase/17/docs/specs/man/jav...
But since the more standard double-dash variants printing to stdout where added with Java 9 I would actually laud it as good backward compatibility.
(Of course, an alternative argument is that commands should fail silently but emit a nonzero return value.)
When invoked directly, as with '-h' '--help', etc., help output should write to stdout, and not stderr.
StackOverflow has tackled this question, 2nd response follows the course I suggest:
<https://stackoverflow.com/questions/1068020/app-help-should-...>
And in this case, the first response:
<https://stackoverflow.com/questions/2199624/should-the-comma...>
I'm looking for any specific guidance from, e.g., GNU but am not finding any.
More than justifiable, I'd say it's the correct thing to do in that case. Otherwise, the caller (which can be another script) may end up working with the help message thinking it was the output it expected.
The whole rule should be something like "Print to stdout if it's part of what's asked by the caller. Print to stderr if it wasn't asked but the user should know about it." So outputting it to stdout should happen when it's asked via --help, and outputting it to stderr should happen when it's part of an error.
https://www.gnu.org/prep/standards/html_node/_002d_002dhelp....
> The standard --help option should output brief documentation for how to invoke the program, on standard output, then exit successfully.
OP says please have —help post to stdout, GNU guidance posted above says exact same thing?
There are two cases when a CLI program can print its usage
1. when the `--help` option, or often also the short-option `-h`, is passed to the program
2. When the user passes a wrong option to the program, where first the error is printed and then often also the general usage.
For 1. the output always should be on stdout, but for 2. the error should be on stderr, and it might be warranted that in that case the usage might be printed on stderr too, so that all is on the same stream.
Doing 2. is not a must though, one can also go for an output like:
This avoids "hiding" the actual error in the often rather big amount of usage-text while still hinting how to get information about what options the program expects and/or accepts.This is a special case of 2, but is distinctly different since no context can be inferred.
In my opinion the program should fail successfully (as in non zero return) since no command was given. I'm highly annoyed when kubectl starts spuwing help text when I forget the command somewhere in a script.
Can we also find a "special place" for programs who always output the help text to stderr no matter what and have pages of options? I don't want to be redirecting before beging able to grep...
Since you can nest functions in Bash (did you know?), I usually have a help function within the main function that is called from both logic branches and just outputs to the right file descriptor.
Yes, but they are not scoped to the parent:
Probably the most profitable use for this is for individual functions to override some callback.But without even dynamic scope, you have no nice way of restoring the previous one, which could be one that some intermediate caller installed for itself.
It could be used to delay the definition of functions. Say that for whatever reason, we put a large number of functions into a file and don't need them all to be defined at once. There can be functions which, when invoked, define sets of functions.
A module could be written in which certain functions are intended to be clobbered by the user with its own implementation. A function which defines those functions to their original state would be useful to recover from a bad redefinition, without reloading that module.
I wish I could use Elixir as a shell scripting language without incurring the VM startup cost and losing the ability to export shell variables or create shell functions... Maybe it would make sense for someone to hack its REPL to dupe most of what one would need on a commandline
In a language that provides nothing beyond a function that lets you run system commands, you can control the child's environment via the env utility:
Chatter on success reads like a cheesy sci fi script.
A message like "incorrect arguments, use --help" can itself go to stderr. Not --help itself though.Some GNU guidance is in the GNU Coding Standards:
https://www.gnu.org/prep/standards/standards.html#Command_00...
That does say that --help and --version should go to standard output.
The document also gives a list of common options; i.e. don't invent your own name for an option, if something matches in this list.
ESR's a somewhat less reliable narrator on many topics these days, but his TAOUP remains useful, and indeed suggests "Rule of Repair: Repair what you can — but when you must fail, fail noisily and as soon as possible."
<https://www.catb.org/~esr/writings/taoup/html/ch01s06.html>
Postel's Law does not count among "repair what you can". Do not try to repair Postel's Law as ESR is doing here; it's broken beyond repair and can only be replaced.
- Rarely repair a bad input; it is optional at your discretion. Just fail.
- If you repair a bad input, do it only in order to try to diagnose more of that input; more things could be wrong, or the first failure encountered could even have a root cause in those other things.
- Remember to fail if you repaired the bad input, even if there are no more errors after the repair.
Not part of it, but not against it. It's useful to stay quiet when the program is meant for conditions and failure is normal. For example: `test`/`[`, `false`, `grep` (when no matches are found), etc. Also when the program is meant as a sort of wrapper to other programs, like `ssh localhost false`, `script -qec false /dev/null`, `true | xargs false`, etc.
> A message like "incorrect arguments, use --help" can itself go to stderr. Not --help itself though.
I don't agree that it's incorrect to save the user the step of calling --help, when it's obvious they need to see that info from an incorrect call. Once decided that including the --help message in an error is right, I don't think it's correct to include it in stdout when it's not expected.
This isn't an odd behavior either, including the --help message (or at least just the synopsis) in stderr on incorrect options is the behavior I'm seeing in utilities like GNU's `bash`, `grep`, and OpenBSD's `netcat`, for example.
The user doesn't always need help; maybe they just made a typo.
Spewing help on every error gets old, fast. Much faster than a Unix graybeard gets old.
Personal, I like it, when a program directs me to its help-command, when I give it a bad option, like
If it doesn't find a match for the specified regex, it fails silently (and exits with a return value of 1).
If the file isn't found, then grep reports that error, e.g.,
Mind that it's quite possible that that error might result from an improperly quoted or escaped multi-term regex (containing whitespace), e.g., In a well-designed bash script, you'd test the file and quote the regex, e.g., Grep is verbose on error where that's useful, but not overly so.A tool which returns a (brief) usage note is "magick" (from the ImageMagick suite):
There are others I've encountered which return a much longer help set. I believe opkg from OpenWRT is amongst those, which prints over 80 lines worth of options to stderr when given an invalid argument.There’s a “Share” link under each answer that you can use to link directly to them. In this case it’s impossible to know which answer you mean because we don’t know what’s your “Sorted by” option. But even then, the order changes over time.
Fish just uses a different order (&|): https://fishshell.com/docs/current/language.html#piping
The output of --help is not an error message, it's the legitimate and expected output of the program when invoked with that argument.
Many tools, for consistency or for laziness always print usage to stderr. But it is better than always printing it to stdout. Errors should never go to stdout, and paging stderr can easily be done with 2>&1.
Edit: and maybe, if your --help output is several pages long, consider leaving out the details to a manpage.
But yeah, agree. It's way more preferable to have surprise output on stderr than surprises mingling with stdout, and it's good to be prepared for that.
Frankly, I nearly quit piping to a pager when GUI terminal backscroll became easy and infinite.
80x25
> I set all my terminals to 22x23.
You are a very silly man and silliness should not be catered for.
> Will --help run a few quick ioctls to calculate screen size?
Your terminal can wrap text around just fine. If it can't, ask for refund.
Also my father produced tonnes of scratch paper in narrow strips. I found a way to weave them into a perpetually-growing "tractor-feed snake" which I placed in a padded crib and brought to grade school to show off. I would typically juggle a few bits and bobs to attract even more attention...
I'm sure you could dig around into my hometown's landfill for awhile. They're only 30 years sediment. I'll send you a tarp, shovel, duct tape to stick the pages back together, and my sister's old clothes to wear while you work. DM for deets.
Alternatively, tools could build in pagers for fewer pipeline surprises, like the all-encompassing systemd.
Even better, I could envision a framework where any tool that produces output can be automatically subject to pagination, with auto terminal detection and the whole bit. Think of libreadline but for output. You could thereby eliminate plenty of ad hoc hacks.
In that same vein, I wondered if that syntax was supported in zsh since modern macOS went whole hog and ... I gained one more pebble on the huge pile of steaming turds why I detest zsh
dafuq? oh, aren't you just the funniest. har. de. har.-x is the most invaluable tool in my shell-debugging toolkit. It is great to see every command evaluated and run alongside the script output itself. I use it multiple times per day at work.
But it's also really useful to be able to get full synopsis of all the options even if all you have available is the binary. Some programs have a lot of options. The "--help" output for rsync on my machine is 184 lines, and is actually pretty terse.
... and there truly is no agreed-upon idea of what constitutes a "page". Even the VT100 screen size was never dominant enough to always count. And nowadays people's windows may be of almost any size.
Windows Powershell gives you the detail of git push in scary red text.
https://stackoverflow.com/questions/12751261/powershell-disp...
You would use stderr for status messages, error messages, and other stuff that is not the primary output of the program. (In some cases, this might include help text; like another comment says, if you specified wrong arguments (not --help) then a short description might be a part of the error message.)
One program that does write error messages to stdout and that annoys me significantly is Ghostscript. (Although you can tell it to write it to stderr, doing that causes all output to be written to stderr; I want output from "print", "==", etc to be written to stdout.)
However, if you used the tool incorrectly (passed the wrong args) and you expected the usage information to go to stdout rather than stderr, I would disagree vehemently. stdout is (generally) for parseable information, whereas stderr is kind of a garbage bin of everything else.
I think you meant stdout here, not stderr.
For example, many programs will print usage/help when used incorrectly. Imagine you upgrade the "read_reactor" tool, and your usage in your "control_reactor" becomes invalid - suddenly you're piping help message data to the control rods. By sending it to stderr instead, no bogus data would be piped and, as a bonus, you would see the help message after invoking your script because (as you have experienced) stderr is not piped by default.
If you want to send it to less: read_reactor -h 2>&1 | less
As others have noted, that's an error which shouldn't go to stdout.
But help text is not an error. It's arguably the expected and primay output of the help function.
If you're following such and such standards that says it should go to stdout, it should go to stdout though. I don't take that as a given.
I agree with this remaining open after all of these years: https://github.com/commandlineparser/commandline/issues/399 OP should add 2>&1 before the pipe or replace the pipe with |& (bash) or &| (fish)
<https://www.gnu.org/prep/standards/html_node/_002d_002dhelp....>
Contradicts your argument.
This is being pedantic. If 99% of programs operate a certain way (GNU coreutils or their BSD equivalent), while not dogma, it becomes convention.
You need a good reason to break convention than whatever rationalisation you seem to have against it.
The comment in the linked github issue does not present any advantage to using stderr (lack of buffering, really?) yet they completely ignore convention. Quit being fancy, be a good GNU/BSD citizen.
These sound useful, in any event.
I'd also like to be able to pipe both STDOUT and STDERR to the next in a sequence of pipes, but eh
I'm sure there's some obscure reason why |& isn't the one people suggest first but when I learned about it recently it was hugely useful.
(And if I haven't learned the correct ordering for the > variant by now I think it's fair to assume I was never going to do so...)
But I'm glad the mnemonic works for you. :)
|& isn't POSIX. As for the redirection order, if you haven't learned it yet, learn now!
These numbers arent magic. 0 is stdin, 1 is stdout, and 2 is stderr. These are the free file descriptors you get on Unix and on Windows. Stdout (1) from previous process goes to stdin (0) in the next process in a pipeline. So, if you want less to see the previous processe's stdout (1) and stderr (2), you just need to tell shell "send stderr to wherever stdout is going right now". That's exaxtly what 2>&1 means.
A fun caveat about this syntax is the difference between these two:
The first one tells stderr to go to dev null ("where stdout is going right now"), and the second one sends stderr to the next process and stdout to dev null> |& isn't POSIX
POSIX, the obscurest of reasons! :)
While I appreciate you taking the time to explain the details, it's not that I haven't tried to learn/remember the specifics but that I tended to use it so infrequently that I'd forget in between uses--and even while knowing the specifics in terms of file descriptors & redirection syntax I would claim the ampersand placement/usage isn't exactly intuitive.
Also, my use primarily tends to be interactive rather than scripting so POSIX compatibility is less of a concern than whether I have to think about it--if I end up somewhere POSIX-compliant that isn't also bash clearly I took a wrong turn and should just through the whole computer out the window. :D
It might be a weird & non-POSIX pipe to die in but at least it's mine. :)
EPIPE
I guess at that point you'd need 4 standard FD's: STDIN, STDOUT, STDERRIN, and STDERR.
But stderr was designed to be seen on terminal regardless of piping or logging. It’s its purpose, so that a pipe user could see what’s wrong or what’s up. There may be a programmatic need to read stderr, but mixing it with stdout is only needed with programs that use these descriptors incorrectly.
Not quite the same, but I really dislike programs which log status updates and more fundamental output to the same place.
IIRC, ffmpeg is like this with everything going to stderr, making metadata parsing more difficult than it needs to be.
If you're trying to parse metadata, ffprobe has a set of options for structured output to stdout. Parsing this will be dramatically easier than whatever you're doing.
(There's more -show_stuff options, but they're probably more detailed than you want. Run ffprobe --help for details.)But we all wanted to know, does it stdout or stderr? :-D
(It prints help to stdout, and a bunch of junk to stderr)
but no, you have to mess with stderr because the programs natural output without using the flags is an error apparently.
Standard output is for program output. Standard error is something of an unfortunate name since it's actually a side channel for all non-output messages meant for the user to read.
So what's "output" anyway? Whatever the user asked the program to compute. If you pass a --help option, the help message is clearly the program's output because it's what the user asked for. If you use it incorrectly and the program prints usage information, the help message should go to standard error while the output is empty because the operation failed.
when I rerun with --help, it prints a worse than useless usage string and says "for more help, type --help-advanced"
https://github.com/sharkdp/bat#highlighting---help-messages
This highlights the help output with colors so it looks nicer, works with most help outputs, as it highlights the first part which is the flag/argument in one color and the description in another colorUse the defaults for your logging library and support a config option.
--help, when used correctly, is almost always interactive, where stdout/stderr and exit status don't matter at all. The few noninteractive uses like help2man or zsh auto parsing can trivially handle a redirect. Sure, a noob piping --help to less may be confused for a first time, that's rare and it's a good chance for them to learn about streams and redirection.
That leaves accidental noninteractive usage. Sooner or later someone will call your program with dynamic arguments from another program, and if your command accepts filenames/IDs there's always a chance to encounter one that starts with '-' and contains an 'h' (a practical example: YouTube video IDs). It's very easy to forget to add -- before the unsafe argument(s), so that it's accidentally interpreted as flags. Nonempty stdout, empty stderr and zero exit status makes it way too easy to accidentally accept the output as valid, only to discover much later.
This is not a theoretical concern, I've made this mistake myself and had it masked by -h behavior. A noob only need to learn redirection once, in a totally harmless setting. Meanwhile, even the most seasoned expert could forget -- in a posix_spawn.
At the end of the day, this is not a big deal, but as I said, if you have to make a conscious choice, make the one that makes accidental mistakes more obvious, because humans do make mistakes. This principle applies everywhere.
The GNU Project has published tools of varying quality, based on who was around to write the tool, debug it, give feedback, etc. It is not the exemplar of high quality software. (But it's far from crap.) The important bit about GNU (and any other software) is that it was written to adhere to their uses. Other people have different requirements. Telling people to "write your software like GNU writes their software" is to misunderstand personal agency and one of the major points of open source software.
Your comments sound like you're saying "Software freedom means you're free to write software the way I want you to write software."
No thank you.
On the other hand, a system that has been designed coherently is much nicer to use.
But the flip side of this is, yes, cruft.
The word you’re looking for is “condescending”.
Regardless, there is no argument about software freedom to be made here.
You’re allowed, be it open or close source, to write and publish software that defies common, well-established conventions.
You can pretend that it’s some sort of first amendment right to do so if you like, and attempt to deflect your unwillingness to write software that behaves properly as incompetence on the users’ end.
But whether anyone will be convinced by that is a separate question, and those who aren’t convinced certainly have the right to tell you, in turn, that your software sucks. This does not inflige on your rights to write broken software.
Yup, sounds pretty snotty.
Is there no value in following a convention?
This is sort of my hot button issue. After years of working on BSAFE, OpenSSL, firefox and libnss, I hate that people say "Hey. Great software. Here's a list of things you must add to it. Of course I'm not going to pay you."
Why should I change my code to adhere to someone else's conventions when they're in opposition to existing conventions?
You really do not need to be such a grumpy elitist. People are not born with Unix knowledge already in their heads. Asking questions, raising doubts, and getting answers from more knowledgeable users is a very effective way of learning new things!
With that said, can you make an example of a legitimate use of `command1 --help | command2`, where `command2` does something useful and is not `less`?
More importantly, I can easily find things by searching with less's `/` hotkey. Relying on the terminal emulator's built-in search isn't great because (a) I'm not used to it - I am more used to vim's keybindings, and the search hotkey `/` is the same in vim, and (b) that's also going to search all the output from before I ran --help (not as big of a gripe, but still somewhat annoying).
And if your brain is wired for vi, then that makes complete sense.
But... the cool thing about using the scrollwheel to scroll up to see the --help output is it's always there. If you pipe it into less, it disappears as soon as you exit less. So if you're writing a big, beefy command with lots of unfamiliar options, you can start typing down at the prompt and then scroll up to read the help output. It's annoying when you type that you immediately scroll back down to the bottom of the terminal buffer, and I think all terminal emulators default to doing this, but maybe it's a configurable behaviour.
This also works with `man <command> | cat`.
Also... how many times have I had to type out `git branch -a | cat` and tried to remember to put the `| cat` in it. I HATE that the stock git cli automagically pipes to /usr/bin/pager. If I wanted to pipe the output to /usr/bin/pager, I would type `command | /usr/bin/pager`. But now I'm just kvetching.
(Alacritty has a tonne of great features I just haven't taken the time to learn the muscle memory to use.. ctrl-j/k bindings to scroll back and some custom patterns to open as URLs is about it.)
The problem isn't that you get a usage message when you ask for it. It's that you get a usage message (written to STDOUT) when you don't. Many commands will print out the usage message when command line options specify a condition that can't be met. I find this frequently when ssh'ing into busybox based systems. Busybox's find command is much less "refined" than comparable desktop OS finds (BSD & GNU/Linux).
So if I do something like:
And expect output that looks something like a sorted line of hashes, I will be sorely disappointed.- if you run cut with no input and no parameters, it outputs an error message to STDERR
- if you run cut --help it outputs usage info to STDOUT
This is what I'd expect based on the man page. Running cut with no parameters is undefined, hence the output to STDERR. Running cut --help is defined, so the output goes to STDERR.
I think people get confused because some tools, when run without any input, output the full help info to STDERR, instead of suggesting the user run foo --help*.
So foo* and foo --help* appear to be equivalent. Until you pipe into less*.
I use the shell almost daily, and have shipped industry-leading products.
2>&1 is a vague memory because I'm not sure if I've ever done that; and I certainly shouldn't have to know some arcane shell trick to read the manual.
I use much more complicated pipe tricks than that interactively on a daily basis, and I definitely don't think of them as "arcane". As somebody who does that, it's useful to me to know which channel the data I want to pipe are going to come out on. Which is why help, which is normal requested output, should obviously go to stdout.
Usage messages issued in response to actual user errors are different, of course.
Also, if you needed to use it every day, I suspect it would be more familiar than a vague memory.
I don't pipe command output daily, though. I use the shell because often it's easier than point-and-click for a lot of operations.
The shell has plenty of use cases that don't involve piping output. Saying you shouldn't touch the shell unless you understand piping output is like saying you shouldn't touch a refrigerator unless you know the perfect temperature to store milk.
"Exceptional event" is not a useful or well defined concept. A better concept is "error" or "unexpected result".
Asking for help is a request for information. The normal, non-error, expected result is that a bunch of text will show up on the output. It is entirely reasonable that the "next command in the pipe" might want to do something with that expected output.
I shouldn't have to guess whether or not you think the output I specifically requested is "exceptional", so it's entirely reasonable to expect that programs in general consistently put user-requested help on stdout.
You are of course free to write your software any way you want. And I'm free to think it's stupid, and to not use your software.
Yes. You're unlikely to like my software. I don't recommend you use it.
Uh-huh. And then you get developers that do this (from inside a f#@<ing library, of all things):
If it's not an error, it is not exceptional, so it should go to stdout, right? Yay for personal agency!Saves a few characters compared to 2>&1