Frankly I've never liked man pages. To me they always screamed "This is how documentation was done in the 90s." The examples are often very unclear or incomplete, and the explanations often assume prior knowledge without providing links in case such knowledge is absent.
Modern documentation has gotten way better, as seen in the Stripe docs and many others, and I wish the man pages could be updated accordingly.
The quality of manpages can vary significantly, however many of them are excellent. The Git manpages stand out; most of the system manpages for OpenBSD are excellent as well.
One other nice thing is integration with an editor. I view manpages in emacs, and can yank the snippets directly into my other buffers for extra convenience.
What is your better solution ? Instead of being derogatory about a technology which works how about creating your ideal and seeing if the Internet likes it ?
It's a good ideal but it also helps to recognize the problem. Not everyone that recognizes the problem is apt or interested in offering a solution. Also I don't believe parent comment was intended to be derogatory
Indeed. When you do user testing and they tell you a certain component of your application is confusing or hard-to-use, do you castigate them for being derogatory and tell them to fix it?
I don't know. I really like the moments where Alan Kay appears out of nowhere, tells where your idea squanders the potential of computers, points to 60's-80's prior art that did it better anyway, and disappears in a puff of smoke.
The last thing our discipline needs is us thinking we got it all and it just needs polishing.
I understand what you're saying but I think it goes without saying that we've started paying more attention to quality of user experience for software in the past twenty years.
If you genuinely disagree and believe the 90s were as ripe with quality tooling and documentation as 2016, well, that's a bit strange, but you're certainly entitled to your opinion.
Yes man pages are usually upside-down; the examples should be right at the start and then lead to a drill-down into options. 9/10 times I end-up having to search the web for a basic introductory example.
But even in big corps corps with ISO9000 accreditation there is seldom self- questioning as to whether documentation is useful rather than just ticking the box for process-completeness.
Yes! For some reason I still always type `man whatever` when I need to do something with a tool... even though I'm just presented with 50 pages of gibberish that I scroll through for a few pages then go to Stack Overflow.
The problem I usually find is that man pages give equal importance to every possible flag - making really hard to figure out which elements you most likely will need to do common tasks.
In practice --help output is mostly useless—it’s neither simple enough for a quick overview nor comprehensive enough to actually know what I can do. Tried “less --help” lately? It spits out ten pages of output!
Plus, every program does --help differently. Does it spit out short output or tons of output so that I need a pager? Does it fork and execute a pager itself, or do I have to run the program again and pipe it to less manually? And then it turns out some programs print --help output to stderr, not stdout, so the pager doesn’t display any output until I do “blahblah --help 2>&1 | less”.
A good manpage beats --help all hollow. It’s simultaneously more brief and more (usefully) verbose. The Synopsis section provides a quick overview of flags, if I just need a quick reminder to jog my memory. But if I want to know the details of a particular flag, I can just scroll down to it or search for it. Mandoc, the default manpage formatter in OpenBSD, even has semantic search built in. It uses more(1)'s built‐in tags functionality to take me straight to whatever flag I want. (You can get this functionality on Linux too by installing mandoc: http://mdocml.bsd.lv/)
Really, horrendous and arcane manpages are a trope that mostly exists in the Linux/GNU world. In BSDs, especially OpenBSD and FreeBSD, clear, simple documentation is a priority. No need for --help or “bropages” when the actual manpages are perfectly clear. For example, check out OpenBSD’s signify(1) manual (http://man.openbsd.org/signify.1), used for cryptographic signing and verification. Compare that to the monstrosity that is gpg(1)… (https://www.gnupg.org/documentation/manpage.en.html)
You're presuming man pages are primarily meant to serve as a reference. But I rarely need man-pages as a reference†.
Most of the time, if I'm looking up a man-page for something, it's because I've just installed a new package that sounded like it would solve a problem and then did a dpkg-query(1) to find out what binaries came with it—or used apropos(1) to find a relevant binary already installed—and now I want to know what the uses of a given binary are and whether those uses include solving my particular problem.
† Well, except for the utilities with absolutely horrible command-line UX-design, like tar(1) or ps(1) or rsync(1), where I just memorize the options I need for my usual case, and then have to look in the man page to do anything novel.
Let me rephrase: you assume that it makes sense for manpages to continue serving primarily as a reference—that this is the primary use-case people have for the standardized program documentation that ships with their distro packages.
Shipping a reference to a binary with that binary may have made sense before the internet. But nowadays, it's the opposite.
• Complex programs with many options have (sometimes dozens of) websites documenting them thoroughly. (Try searching with any search-engine for "wget mirroring", for example; the number and complexity of the results is overwhelming.)
• Meanwhile, for the simple "corner-case" programs, you really hope that they shipped with docs—because seemingly nobody else out there on the web cares to bother documenting them. With a lot of these little programs, the only web doc you can find are, in fact, online mirrors of their man-page.
For the popular-and-complex programs, man-pages are just redundant, because everyone will document what they did to achieve whatever. But for the simple-but-weird programs—the ones for which man-pages aren't just redundancies—if the man-page doesn't give usage, then nothing is going to give usage.
Now, I can understand why man-pages for these little utilities are the way they are. These programs are usually created by a single author, so time spent writing docs is time not spent fixing bugs or scratching their itch or whatever else. And an options reference certainly is the "minimal normalized form" of documentation: it lets others brute-force combinatoric-search the space of invocations until they find some combination that Works For Them™. Basically, you can (through a lot of trial and error) generate a cookbook from an options reference. So the author probably doesn't feel a strong need to add anything beyond an options reference, because the people who really need to solve the problem their binary solves are willing to go to that effort.
But if you're a distro downstream packager, and it's your job to make your distro easy for people to use, you should have every incentive to submit upstream patches to said author, with manpage additions of cookbook example usages resulting from your trial-and-error experimentation with their program.
Annoyingly, you, as a distro packager, probably don't have time to do that trial-and-error experimentation, especially if the utility serves a niche use-case that you don't even understand. That—and not the fact that "manpages should be a reference and nothing more"—is most of the reason manpages continue to be the way they are.
Man pages are written (ideally) as the authoritative reference on your system, where you can go to find information. That sort of thing needs to exist somewhere, it has a clear use case.
An ideal man page is concise, informative, and complete. Learning to read them is like learning to read scientific literature: a pain, but once you figure it out, you're at a higher level.
>you assume that it makes sense for manpages to continue serving primarily as a reference
No, I don't. But that's what they are, and what they're meant to be. Complaining that they are is like complaining that Haskell is functional: It may not be ideal for your circumstances, and you're welcome to make that known, but there's no use complaining about it, because it's the entire point.
• The primary win for all of those utilities would be in separating their horribly-large arrays of top-level runtime switches into subcommands that each have a restricted, learnable set of runtime switches relevant only to that subcommand. (See: git, docker, lvm, ip, ufw).
• As well, for top-level switches that are really preferences—that is, switches that don't apply to a use-case, but rather to a user—read those from ~/.config/foo/foorc with defaults in /etc/foo/foorc. Don't expose them at all as runtime switches. If there are features that would be customized by different wrapper-client UIs, provide a --config-file=[file] switch that takes a config-file that sets those. And for special drivers, like Makefiles or CI setups, that have a matrix of different UI options they might want to ask for, you can provide FOO_OPTION env-vars. (See: curl, compiler toolchains, ffmpeg, apt). In ffmpeg's case, there are even config-file presets—basically little plug-ins of config options you can enable as a group. You can dump new files into the /etc/ffmpeg/presets or ~/.config/ffmpeg/presets to add them as presets.
• For switches that are exposed individually, but which are also aggregated into switch-group-setting switches, in most cases there's literally no use-case for setting the individual switches, only the switch-group-setting switch. Remove the individual switches. Just because the program could theoretically be configured to do X compatibility thing for old-arch Foo, but not Y other old compatibility thing for old-arch Foo (where any time you're talking to Foo you always need both), you don't have to expose both an X switch and a Y switch. Just expose a --foo-compat switch and be done with it. People trying to use your binary on other weird OSes shouldn't be doing so by combining tons of option-switches; they should sit down and port your program to that OS, by finding the place in your program where you've defined {arch -> shim flag set} mappings, adding a new mapping for their arch, and exposing it as a new external --bar-compat switch.
• Speaking of compat flags, try just making such behaviors always-on when you build targeting the relevant arch, rather than needing to specify them at runtime. The only real use-case for runtime compat switches is as an IPC/RPC client talking to a server that expects the weird behaviour. And even then, your client should first try to auto-detect the server's expectations, and you should only add the runtime switch if that auto-detection turns out to be unreliable. (Meanwhile, If you're the IPC/RPC server, the compat settings belong in the config file. See: Samba, NFS, Netatalk, ...)
• If your binary manipulates state (like, say, how git manipulates repos), and you change the way it does so in a backward-incompatible manner, you might be tempted to add a runtime switch to turn the old behaviour back on to allow collaboration with people using older versions. If your state-store is at all extensible, though, you should instead add a field for a schema-version and a flag for pinning the state-store to that schema-version, as well as a subcommand to pin/unpin a given state-store's schema-version. Now your binary will upgrade the schema of its state-store by default, but will respect "protected" state-stores and perform only backward-compatible operations on those. (Importantly, this guides the code architecture into failing by default instead of doing something backward-incompatible; whereas, with backcompat switches, you're always forcing a state-store schema-migration on people by default every time you add new code, unless/until you add a matching backcompat-switch.)
• quiet/verbose and interactive/batch are silly "mode switches" to have in modern programs, just like forking is a silly way to do daemons when you've got a modern init(8). P...
> • The primary win for all of those utilities would be in separating their horribly-large arrays of top-level runtime switches into subcommands that each have a restricted, learnable set of runtime switches relevant only to that subcommand. (See: git, docker, lvm, ip, ufw).
In principle I agree. In practice, all of the commands you listed are multi-tool commands. For example, git overall can be described as a "content tracker"; only its subcommands can be described as doing one thing, and even then often they do multiple duties (to a newbie, git reset apparently does three different things). In contrast, two of the commands you listed (ps and rsync) do only one thing; list processes and copy files. The third, tar, does technically have multiple modes, but most of the switches apply to multiple modes (-f, compression settings, etc). How would you break these up into subcommands?
> • As well, for top-level switches that are really preferences—that is, switches that don't apply to a use-case, but rather to a user—read those from ~/.config/foo/foorc with defaults in /etc/foo/foorc. Don't expose them at all as runtime switches. If there are features that would be customized by different wrapper-client UIs, provide a --config-file=[file] switch that takes a config-file that sets those. And for special drivers, like Makefiles or CI setups, that have a matrix of different UI options they might want to ask for, you can provide FOO_OPTION env-vars. (See: curl, compiler toolchains, ffmpeg, apt). In ffmpeg's case, there are even config-file presets—basically little plug-ins of config options you can enable as a group. You can dump new files into the /etc/ffmpeg/presets or ~/.config/ffmpeg/presets to add them as presets.
Some would argue that this is an anti-pattern and that such configuration should reside in your shell as an alias or function. Regardless, with the possible exception of ps, for the commands you listed, which options exactly would you move into a configuration file? It would likely not be a good idea, for example, to say that all tar -c commands should imply -J, since that would break lots of scripts.
> • For options that are aggregated into option-set flags, where there's literally no use for the options outside of their use in the option-set flag, remove the individual options. Just because the program could theoretically be configured to do X compatibility thing for old-arch Foo, but not Y other old compatibility thing for old-arch Foo, you don't have to support both an X and Y option. Just expose a --foo-compat option and be done with it. People trying to use your binary on other weird OSes shouldn't be doing so by combining tons of option-switches; they should sit down and port your program to that OS, by finding the place in your program where you've defined {arch -> shim flag set} mappings, adding a new mapping for their arch, and exposing it as a new external --bar-compat switch.
I don't understand what this means. Can you give a concrete example?
> • Speaking of compat flags, try just making such behaviors always-on when you build targeting the relevant arch, rather than needing to specify them at runtime. The only real use-case for runtime compat switches is as an IPC/RPC client talking to a server that expects the weird behaviour. And even then, your client should first try to auto-detect the server's expectations, and you should only add the runtime switch if that auto-detection turns out to be unreliable. (Meanwhile, If you're the IPC/RPC server, the compat settings belong in the config file. See: Samba, NFS, Netatalk, ...)
Again, I don't understand what this means.
> Those four things together should trim each subcommand to a reasonable "visible" option-set. Now just ensure that typing "man command subcommand" gets you a separate man-page written just for the subcommand, and add bash/zsh complet...
1. They commands I listed seem like "inherently multi-tool" commands because they've been refactored into multi-tool commands. ip is (mostly) a refactoring of ifconfig(1), but also of ethtool(1) and arp(1) and a few other things. A proper multi-mode refactoring of ps(1) would integrate overlapping utilities like pgrep(1) and fuser(1)/lsof(1), and then split those back down along separate lines.
A proper multi-subcommand refactoring of tar, I would think, might involve an interestingly different usage—a "fluent OOP interface" involving either a series, or pipeline, of invocations, all involving tar subcommands. For example:
tar manifest create --new-root=/ . | tar archive assemble --extended-attributes --timestamps --compression=lzma > foo.tar.xz
or, equivalently:
tar manifest create --new-root=/ . > foo.manifest
tar archive new --extended-attributes --timestamps --compression=lzma > foo.tar.xz
tar archive insert foo.tar.xz foo.manifest
then, later:
tar manifest read foo.tar.xz | grep -E '...' | sed '{ ... }' | tar archive extract foo.tar.xz
Using some file-descriptor introspection trickery, none of the steps except for the last one would actually have to stream bytes through them.
2. One interesting thing about ps is that it takes two separate sets of switches—BSD switches and GNU switches. Given that any given user will only want to use one or the other, these two interfaces should be broken into two utilities (likely using argv[0] detection, like gzip(1) and zcat(1)), with separate man-pages. Setting which one plain "ps" means when ps is called TTY-interactively should be a configuration-file thing.
3. Look at the man-page for GNU coreutils ln(1) or cp(1). They're messes of compat-switches, and compat-switch aggregates, aimed at allowing you to reproduce the default behaviour of the OS you're used to on any-and-every OS you can compile coreutils for. This configuration doesn't belong in argv[]; it belongs in a ./configure script, or in a ~/.config/coreutils/ui-rc file. (Some of the switches are maybe for things you might want to do even in your initrd before you've got the rootfs mounted; those are what env-vars are for.)
4. rsync(1) has options for copying ACLs, copying Extended Attributes, "handling sparse files efficiently", and so on. rsync should just try to do these things automatically, and fall back to not doing them if the source or dest doesn't have support for them. There's no use-case where both the source and target support EAs but you don't want them copies. There's also no use-case where one or the other doesn't support EAs and so you want to fail the sync instead of copying. There's no need for these switches, just like there's no need for an FTP PASV switch. It can be detected.
---
As an aside:
> not knowing that GNU and at least one BSD tar know how to automatically select compression format in at least some cases
For extraction, sure. But why not for creation? RAR and Zip can pick the best compression they "know how to" do at the current moment, and offer compression speed/quality presets that actually involve picking different algorithms.
The real confusion over tar's compression switches is more fundamental; it comes from the fact that a tar archive is "wrapped in" an arbitrary compression-container file-format, without there being any standard for those to allow you to detect something as a "gzipped tar" rather than "gzipped opaque data", and thus the inability of tar, or your OS, to recognize an "[unknown compressor]ed tar" file.
The gordian knot of tar(1)'s switches is for tar—already having treated the compressor as a subprocess rather than a pipe-destination for a long time now—to just begin adding a "...
> A proper multi-subcommand refactoring of tar, I would think, might involve an interestingly different usage—a "fluent OOP interface" involving either a series, or pipeline, of invocations, all involving tar subcommands. For example:
... it was my understanding that the goal was to make the interface easier, not three times as complicated.
> 2. One interesting thing about ps is that it takes two separate sets of switches—BSD switches and GNU switches. Given that any given user will only want to use one or the other, these two interfaces should be broken into two utilities (likely using argv[0] detection, like gzip(1) and zcat(1)), with separate man-pages. Setting which one plain "ps" means when ps is called TTY-interactively should be a configuration-file thing.
"bsdps" and "gnups" seem much worse than "ps" and "ps -".
> 3. Look at the man-page for GNU coreutils ln(1) or cp(1). They're messes of compat-switches, and compat-switch aggregates, aimed at allowing you to reproduce the default behaviour of the OS you're used to on any-and-every OS you can compile coreutils for. This configuration doesn't belong in argv[]; it belongs in a ./configure script, or in a ~/.config/coreutils/ui-rc file. (Some of the switches are maybe for things you might want to do even in your initrd before you've got the rootfs mounted; those are what env-vars are for.)
cp I'll kind of buy, but ln? It's only got 14 real options, none of which can be described as a "compat switch". Again, I'll have to ask you for a specific example. Here's an example of a specific example: "cp -Q does <weird thing> that HP-UX does and nobody else".
> There's no use-case where both the source and target support EAs but you don't want them copies.
Security?
> There's also no use-case where one or the other doesn't support EAs and so you want to fail the sync instead of copying.
Wait, what? "fail silently"?
> There's no need for these switches, just like there's no need for an FTP PASV switch. It can be detected.
copying ACLs isn't like PASV, it's more like ASCII vs binary mode. trying to auto-detect makes the situation worse.
> The gordian knot of tar(1)'s switches is for tar—already having treated the compressor as a subprocess rather than a pipe-destination for a long time now—to just begin adding a "this is a tar archive with a compressed stream of archive files inside it" header to the outside of the compressed stream. Even though full backcompat is still possible after defaulting to such a change (user pref files! env vars! in-file hinting that the previous binary can recognize!), everybody's too lazy to want to make that sort of change.
how about "updating the compression algorithm requires everybody to install a new (de)compressor and until they do they can't open your files"?
To me, for CLI commands, "easier" means two things:
1. more learnable, which usually translates to more discoverable, which in turn translates to more orthogonally-decomposed and regular. You want the grammars, syntaxes, sets of verbs/subcommands, names for taken parameters, etc. of all the binaries on a system to obey the Principle of Least Surprise with respect to one-another, so you can learn the syntax for one binary and then reuse that learning on another binary. (Much more likely to be done in a BSD than in Linux, but still possible.)
2. more wrappable, which usually also translates to being more orthogonally-decomposed and regular. You want your binaries to provide an API not only for humans, but for anything that wants to wrap your command in a GUI, or in a friendlier CLI UI, or in a network service, or whatever else. (If you write your binary as mostly a library with a small binary driver, this is even better—see e.g. curl's libcurl—but this is incredibly rare for some reason, and it's a bit much to demand that people refactor their entire codebases this way.)
Being wrappable means that your binary itself is 100% allowed to be "dumb" and "tedious" for experienced users (like in my tar pipeline-UX example), because aliases or scripts or entire conventional-command packages can be built around your binary.
The fact that there's no e.g. `docker gc` subcommand built into docker(1) is a feature, not a bug; the docker(1) binary just provides a set of orthogonal primitive subcommands. Someone else then wrote a docker-gc package containing a configurable script that calls docker(1) a bunch of times.
There should certainly be a high-level UX that doesn't require three separate commands to create a tar file. But tar(1), the core "engine" handling creating that archive, should not be the binary providing that convenient CISC UX. (Maybe you could follow the git model and allow people to call installed "tar-foo" binaries using `tar foo`, but even this is a bit silly.)
But personally, in my mind, tar(1) itself should be a low-level implementation-detail sort of binary anyway, like mknod(8) or losetup(8). tar(1) only handles .tar files, and that's a silly UX for any user-visible task. The only time you already know you've got a .tar file, or want a .tar file, is in a script—and in a script, you want exactly the kind of verbosity, strictness, and interposed-shell-gloop flexibility I was talking about.
Meanwhile, what a user wants 99% of the time for their TTY-interactive use, is a command equivalent to the things a GUI archive-manager does. Users don't want or need tar(1); they want an xdg-archive(1) that uses libarchive.
> Security?
Not the right layer to enforce that in; that's what the [no_]user_xattr mount option is for. (Or, alternately, hierarchical xattr namespaces, with everything coming from foreign sources tainted into a quarantine namespace.)
> Wait, what? "fail silently"?
rsync(1) actually prints a ton of warnings (one for each file, in fact) when you tell it to copy EAs but it can't manage either the read or write step. But it doesn't fail. Because, like I said, there's no use-case. rsync(1) mostly isn't used TTY-interactively; it's used as a recurring batch job. If you said you wanted data+EAs the first time, and the remote's sysadmin remounts their computer's filesystem so EAs stop being available, you don't suddenly want to stop syncing the data. You just want a syslog full of errors telling you that you missed the EAs, plus smart logic such that you can message the other sysadmin telling them to fix their box, and then rsync(1) will gracefully just add the now-available EAs to the files rather than having to re-copy the files. Which... it does. So, great!
(I mean, I can see a use-case for rsync being told "these files should have EAs" and then performing its ...
The quality of the medium is not determined by the quality of its content. GNU manpages especially are incredibly low quality (they have info(1) for extensive documentation), but not all man pages are GNU manpages. If you use man with shell job control (Ctrl-z, fg, bg), you can easily read multiple man pages simultaneously.
I got a few thousand dollars of that money as a security bug bounty.
OpenSSL fixed the problem quickly, but one year on still haven't accepted the regression test for the issue. It would be amusing if it wasn't so horrifying.
The experts writting it for themselves part seemed inaccurate given what I read in LibreSSL commits. It was one atrocity after another. Still love Ted Unganst's observation about them making surd endianess of CPU doesnt change while protocol is running. Just cant remember how often that check was performed.
i interpreted this as: this was written by security experts (cryptographers), not expert programmers. this means the algorithms are generally ok, but the implementation is wacky (and issue prone).
It was also written by cryptographers who for years asked for support, and got barely enough to keep the server running let alone live off it. Meanwhile the world and his wife joined in with feature requests and complaints about things they didn't like, but mostly without offering to help.
So it doesn't surprise me that unit testing, documentation, code review etc. weren't a top priority for spending more unpaid hours on - people literally got what they paid for.
I'll partly agree with that. Mostly even. I draw the line at expecting a security-critical library intending widespread adoption at least follow secure coding guidelines if nothing else. It really doesnt take much effort vs what was already done. Tiny fraction of it.
That plus the larger trend of developers ignoring basic, good practices is why I critique the project a bit. Plus, LibreSSL team illustrated my point nicely by doing 10x what I expected in a very short time with no pay.
If you think OpenSSL is bad, try MIRACL (only documentation I could find is a word file that's basically a list of function signatures). And OpenSSL at least generally builds fine on a vanilla Ubuntu machine.
In contrast, libsodium deserves praise for writing documentation like they want people to actually use their library.
OpenSSL is pretty bad. After reading about some of the stuff that lead the the libressl fork, I wouldn't trust it with my lunch money. Sure, the algorithms are good, but as far as the code's concerned, Heartbleed was the tip of the iceberg.
There's a saying, "don't roll your own crypto," and it's good advice.
In the case of openssl, you might be better off rolling your own. At least the vulnerabilities you end up with are different than the ones that the rest of the world has.
"For instance it doesn't have everything you need to validate certificates..."
Yet it has all the CA crap thrown in, via the overloaded openssl binary. As "examples". And according to the documentation, not even "correct" illustrations of how libssl should be used.
Encryption and authentication are two separate problems.
Just because you figured out a way to encrypt a message does not mean you have also figured out how to a way to send it to only the correct recipient... over an insecure network. (Insecure not only in the sense of "plaintext" but in the sense you are not in control of much of anything - routing, PKI infrastructure, etc.)
It seems to me that one would want to solve the authentication problem first, and then move on to encryption.
This comment shows that for proponents of using SSL on the public web, it's been the other way around. Authentication was never sorted out.
When it comes to authentication, all due respect to the OpenSSL authors, SSH has provided a better attempt at a solution than any implementation of PKI using SSL/TLS.
And one more thing, how many ciphers does a user really need? As we've heard time and again, many of them are not even "safe" to use. Some of the alternative SSL libraries have wisely removed them. But I guess OpenSSL is append only?
49 comments
[ 3.4 ms ] story [ 123 ms ] threadModern documentation has gotten way better, as seen in the Stripe docs and many others, and I wish the man pages could be updated accordingly.
One other nice thing is integration with an editor. I view manpages in emacs, and can yank the snippets directly into my other buffers for extra convenience.
The last thing our discipline needs is us thinking we got it all and it just needs polishing.
If you genuinely disagree and believe the 90s were as ripe with quality tooling and documentation as 2016, well, that's a bit strange, but you're certainly entitled to your opinion.
But even in big corps corps with ISO9000 accreditation there is seldom self- questioning as to whether documentation is useful rather than just ticking the box for process-completeness.
The problem I usually find is that man pages give equal importance to every possible flag - making really hard to figure out which elements you most likely will need to do common tasks.
Having a summary of the options right there at the top is the most valuable thing in a reference.
Plus, every program does --help differently. Does it spit out short output or tons of output so that I need a pager? Does it fork and execute a pager itself, or do I have to run the program again and pipe it to less manually? And then it turns out some programs print --help output to stderr, not stdout, so the pager doesn’t display any output until I do “blahblah --help 2>&1 | less”.
A good manpage beats --help all hollow. It’s simultaneously more brief and more (usefully) verbose. The Synopsis section provides a quick overview of flags, if I just need a quick reminder to jog my memory. But if I want to know the details of a particular flag, I can just scroll down to it or search for it. Mandoc, the default manpage formatter in OpenBSD, even has semantic search built in. It uses more(1)'s built‐in tags functionality to take me straight to whatever flag I want. (You can get this functionality on Linux too by installing mandoc: http://mdocml.bsd.lv/)
Really, horrendous and arcane manpages are a trope that mostly exists in the Linux/GNU world. In BSDs, especially OpenBSD and FreeBSD, clear, simple documentation is a priority. No need for --help or “bropages” when the actual manpages are perfectly clear. For example, check out OpenBSD’s signify(1) manual (http://man.openbsd.org/signify.1), used for cryptographic signing and verification. Compare that to the monstrosity that is gpg(1)… (https://www.gnupg.org/documentation/manpage.en.html)
Most of the time, if I'm looking up a man-page for something, it's because I've just installed a new package that sounded like it would solve a problem and then did a dpkg-query(1) to find out what binaries came with it—or used apropos(1) to find a relevant binary already installed—and now I want to know what the uses of a given binary are and whether those uses include solving my particular problem.
† Well, except for the utilities with absolutely horrible command-line UX-design, like tar(1) or ps(1) or rsync(1), where I just memorize the options I need for my usual case, and then have to look in the man page to do anything novel.
They are.
Shipping a reference to a binary with that binary may have made sense before the internet. But nowadays, it's the opposite.
• Complex programs with many options have (sometimes dozens of) websites documenting them thoroughly. (Try searching with any search-engine for "wget mirroring", for example; the number and complexity of the results is overwhelming.)
• Meanwhile, for the simple "corner-case" programs, you really hope that they shipped with docs—because seemingly nobody else out there on the web cares to bother documenting them. With a lot of these little programs, the only web doc you can find are, in fact, online mirrors of their man-page.
For the popular-and-complex programs, man-pages are just redundant, because everyone will document what they did to achieve whatever. But for the simple-but-weird programs—the ones for which man-pages aren't just redundancies—if the man-page doesn't give usage, then nothing is going to give usage.
Now, I can understand why man-pages for these little utilities are the way they are. These programs are usually created by a single author, so time spent writing docs is time not spent fixing bugs or scratching their itch or whatever else. And an options reference certainly is the "minimal normalized form" of documentation: it lets others brute-force combinatoric-search the space of invocations until they find some combination that Works For Them™. Basically, you can (through a lot of trial and error) generate a cookbook from an options reference. So the author probably doesn't feel a strong need to add anything beyond an options reference, because the people who really need to solve the problem their binary solves are willing to go to that effort.
But if you're a distro downstream packager, and it's your job to make your distro easy for people to use, you should have every incentive to submit upstream patches to said author, with manpage additions of cookbook example usages resulting from your trial-and-error experimentation with their program.
Annoyingly, you, as a distro packager, probably don't have time to do that trial-and-error experimentation, especially if the utility serves a niche use-case that you don't even understand. That—and not the fact that "manpages should be a reference and nothing more"—is most of the reason manpages continue to be the way they are.
An ideal man page is concise, informative, and complete. Learning to read them is like learning to read scientific literature: a pain, but once you figure it out, you're at a higher level.
No, I don't. But that's what they are, and what they're meant to be. Complaining that they are is like complaining that Haskell is functional: It may not be ideal for your circumstances, and you're welcome to make that known, but there's no use complaining about it, because it's the entire point.
how would you improve their design
• As well, for top-level switches that are really preferences—that is, switches that don't apply to a use-case, but rather to a user—read those from ~/.config/foo/foorc with defaults in /etc/foo/foorc. Don't expose them at all as runtime switches. If there are features that would be customized by different wrapper-client UIs, provide a --config-file=[file] switch that takes a config-file that sets those. And for special drivers, like Makefiles or CI setups, that have a matrix of different UI options they might want to ask for, you can provide FOO_OPTION env-vars. (See: curl, compiler toolchains, ffmpeg, apt). In ffmpeg's case, there are even config-file presets—basically little plug-ins of config options you can enable as a group. You can dump new files into the /etc/ffmpeg/presets or ~/.config/ffmpeg/presets to add them as presets.
• For switches that are exposed individually, but which are also aggregated into switch-group-setting switches, in most cases there's literally no use-case for setting the individual switches, only the switch-group-setting switch. Remove the individual switches. Just because the program could theoretically be configured to do X compatibility thing for old-arch Foo, but not Y other old compatibility thing for old-arch Foo (where any time you're talking to Foo you always need both), you don't have to expose both an X switch and a Y switch. Just expose a --foo-compat switch and be done with it. People trying to use your binary on other weird OSes shouldn't be doing so by combining tons of option-switches; they should sit down and port your program to that OS, by finding the place in your program where you've defined {arch -> shim flag set} mappings, adding a new mapping for their arch, and exposing it as a new external --bar-compat switch.
• Speaking of compat flags, try just making such behaviors always-on when you build targeting the relevant arch, rather than needing to specify them at runtime. The only real use-case for runtime compat switches is as an IPC/RPC client talking to a server that expects the weird behaviour. And even then, your client should first try to auto-detect the server's expectations, and you should only add the runtime switch if that auto-detection turns out to be unreliable. (Meanwhile, If you're the IPC/RPC server, the compat settings belong in the config file. See: Samba, NFS, Netatalk, ...)
• If your binary manipulates state (like, say, how git manipulates repos), and you change the way it does so in a backward-incompatible manner, you might be tempted to add a runtime switch to turn the old behaviour back on to allow collaboration with people using older versions. If your state-store is at all extensible, though, you should instead add a field for a schema-version and a flag for pinning the state-store to that schema-version, as well as a subcommand to pin/unpin a given state-store's schema-version. Now your binary will upgrade the schema of its state-store by default, but will respect "protected" state-stores and perform only backward-compatible operations on those. (Importantly, this guides the code architecture into failing by default instead of doing something backward-incompatible; whereas, with backcompat switches, you're always forcing a state-store schema-migration on people by default every time you add new code, unless/until you add a matching backcompat-switch.)
• quiet/verbose and interactive/batch are silly "mode switches" to have in modern programs, just like forking is a silly way to do daemons when you've got a modern init(8). P...
In principle I agree. In practice, all of the commands you listed are multi-tool commands. For example, git overall can be described as a "content tracker"; only its subcommands can be described as doing one thing, and even then often they do multiple duties (to a newbie, git reset apparently does three different things). In contrast, two of the commands you listed (ps and rsync) do only one thing; list processes and copy files. The third, tar, does technically have multiple modes, but most of the switches apply to multiple modes (-f, compression settings, etc). How would you break these up into subcommands?
> • As well, for top-level switches that are really preferences—that is, switches that don't apply to a use-case, but rather to a user—read those from ~/.config/foo/foorc with defaults in /etc/foo/foorc. Don't expose them at all as runtime switches. If there are features that would be customized by different wrapper-client UIs, provide a --config-file=[file] switch that takes a config-file that sets those. And for special drivers, like Makefiles or CI setups, that have a matrix of different UI options they might want to ask for, you can provide FOO_OPTION env-vars. (See: curl, compiler toolchains, ffmpeg, apt). In ffmpeg's case, there are even config-file presets—basically little plug-ins of config options you can enable as a group. You can dump new files into the /etc/ffmpeg/presets or ~/.config/ffmpeg/presets to add them as presets.
Some would argue that this is an anti-pattern and that such configuration should reside in your shell as an alias or function. Regardless, with the possible exception of ps, for the commands you listed, which options exactly would you move into a configuration file? It would likely not be a good idea, for example, to say that all tar -c commands should imply -J, since that would break lots of scripts.
> • For options that are aggregated into option-set flags, where there's literally no use for the options outside of their use in the option-set flag, remove the individual options. Just because the program could theoretically be configured to do X compatibility thing for old-arch Foo, but not Y other old compatibility thing for old-arch Foo, you don't have to support both an X and Y option. Just expose a --foo-compat option and be done with it. People trying to use your binary on other weird OSes shouldn't be doing so by combining tons of option-switches; they should sit down and port your program to that OS, by finding the place in your program where you've defined {arch -> shim flag set} mappings, adding a new mapping for their arch, and exposing it as a new external --bar-compat switch.
I don't understand what this means. Can you give a concrete example?
> • Speaking of compat flags, try just making such behaviors always-on when you build targeting the relevant arch, rather than needing to specify them at runtime. The only real use-case for runtime compat switches is as an IPC/RPC client talking to a server that expects the weird behaviour. And even then, your client should first try to auto-detect the server's expectations, and you should only add the runtime switch if that auto-detection turns out to be unreliable. (Meanwhile, If you're the IPC/RPC server, the compat settings belong in the config file. See: Samba, NFS, Netatalk, ...)
Again, I don't understand what this means.
> Those four things together should trim each subcommand to a reasonable "visible" option-set. Now just ensure that typing "man command subcommand" gets you a separate man-page written just for the subcommand, and add bash/zsh complet...
A proper multi-subcommand refactoring of tar, I would think, might involve an interestingly different usage—a "fluent OOP interface" involving either a series, or pipeline, of invocations, all involving tar subcommands. For example:
or, equivalently: then, later: Using some file-descriptor introspection trickery, none of the steps except for the last one would actually have to stream bytes through them.2. One interesting thing about ps is that it takes two separate sets of switches—BSD switches and GNU switches. Given that any given user will only want to use one or the other, these two interfaces should be broken into two utilities (likely using argv[0] detection, like gzip(1) and zcat(1)), with separate man-pages. Setting which one plain "ps" means when ps is called TTY-interactively should be a configuration-file thing.
3. Look at the man-page for GNU coreutils ln(1) or cp(1). They're messes of compat-switches, and compat-switch aggregates, aimed at allowing you to reproduce the default behaviour of the OS you're used to on any-and-every OS you can compile coreutils for. This configuration doesn't belong in argv[]; it belongs in a ./configure script, or in a ~/.config/coreutils/ui-rc file. (Some of the switches are maybe for things you might want to do even in your initrd before you've got the rootfs mounted; those are what env-vars are for.)
4. rsync(1) has options for copying ACLs, copying Extended Attributes, "handling sparse files efficiently", and so on. rsync should just try to do these things automatically, and fall back to not doing them if the source or dest doesn't have support for them. There's no use-case where both the source and target support EAs but you don't want them copies. There's also no use-case where one or the other doesn't support EAs and so you want to fail the sync instead of copying. There's no need for these switches, just like there's no need for an FTP PASV switch. It can be detected.
---
As an aside:
> not knowing that GNU and at least one BSD tar know how to automatically select compression format in at least some cases
For extraction, sure. But why not for creation? RAR and Zip can pick the best compression they "know how to" do at the current moment, and offer compression speed/quality presets that actually involve picking different algorithms.
The real confusion over tar's compression switches is more fundamental; it comes from the fact that a tar archive is "wrapped in" an arbitrary compression-container file-format, without there being any standard for those to allow you to detect something as a "gzipped tar" rather than "gzipped opaque data", and thus the inability of tar, or your OS, to recognize an "[unknown compressor]ed tar" file.
The gordian knot of tar(1)'s switches is for tar—already having treated the compressor as a subprocess rather than a pipe-destination for a long time now—to just begin adding a "...
... it was my understanding that the goal was to make the interface easier, not three times as complicated.
> 2. One interesting thing about ps is that it takes two separate sets of switches—BSD switches and GNU switches. Given that any given user will only want to use one or the other, these two interfaces should be broken into two utilities (likely using argv[0] detection, like gzip(1) and zcat(1)), with separate man-pages. Setting which one plain "ps" means when ps is called TTY-interactively should be a configuration-file thing.
"bsdps" and "gnups" seem much worse than "ps" and "ps -".
> 3. Look at the man-page for GNU coreutils ln(1) or cp(1). They're messes of compat-switches, and compat-switch aggregates, aimed at allowing you to reproduce the default behaviour of the OS you're used to on any-and-every OS you can compile coreutils for. This configuration doesn't belong in argv[]; it belongs in a ./configure script, or in a ~/.config/coreutils/ui-rc file. (Some of the switches are maybe for things you might want to do even in your initrd before you've got the rootfs mounted; those are what env-vars are for.)
cp I'll kind of buy, but ln? It's only got 14 real options, none of which can be described as a "compat switch". Again, I'll have to ask you for a specific example. Here's an example of a specific example: "cp -Q does <weird thing> that HP-UX does and nobody else".
> There's no use-case where both the source and target support EAs but you don't want them copies.
Security?
> There's also no use-case where one or the other doesn't support EAs and so you want to fail the sync instead of copying.
Wait, what? "fail silently"?
> There's no need for these switches, just like there's no need for an FTP PASV switch. It can be detected.
copying ACLs isn't like PASV, it's more like ASCII vs binary mode. trying to auto-detect makes the situation worse.
> The gordian knot of tar(1)'s switches is for tar—already having treated the compressor as a subprocess rather than a pipe-destination for a long time now—to just begin adding a "this is a tar archive with a compressed stream of archive files inside it" header to the outside of the compressed stream. Even though full backcompat is still possible after defaulting to such a change (user pref files! env vars! in-file hinting that the previous binary can recognize!), everybody's too lazy to want to make that sort of change.
how about "updating the compression algorithm requires everybody to install a new (de)compressor and until they do they can't open your files"?
1. more learnable, which usually translates to more discoverable, which in turn translates to more orthogonally-decomposed and regular. You want the grammars, syntaxes, sets of verbs/subcommands, names for taken parameters, etc. of all the binaries on a system to obey the Principle of Least Surprise with respect to one-another, so you can learn the syntax for one binary and then reuse that learning on another binary. (Much more likely to be done in a BSD than in Linux, but still possible.)
2. more wrappable, which usually also translates to being more orthogonally-decomposed and regular. You want your binaries to provide an API not only for humans, but for anything that wants to wrap your command in a GUI, or in a friendlier CLI UI, or in a network service, or whatever else. (If you write your binary as mostly a library with a small binary driver, this is even better—see e.g. curl's libcurl—but this is incredibly rare for some reason, and it's a bit much to demand that people refactor their entire codebases this way.)
Being wrappable means that your binary itself is 100% allowed to be "dumb" and "tedious" for experienced users (like in my tar pipeline-UX example), because aliases or scripts or entire conventional-command packages can be built around your binary.
The fact that there's no e.g. `docker gc` subcommand built into docker(1) is a feature, not a bug; the docker(1) binary just provides a set of orthogonal primitive subcommands. Someone else then wrote a docker-gc package containing a configurable script that calls docker(1) a bunch of times.
There should certainly be a high-level UX that doesn't require three separate commands to create a tar file. But tar(1), the core "engine" handling creating that archive, should not be the binary providing that convenient CISC UX. (Maybe you could follow the git model and allow people to call installed "tar-foo" binaries using `tar foo`, but even this is a bit silly.)
But personally, in my mind, tar(1) itself should be a low-level implementation-detail sort of binary anyway, like mknod(8) or losetup(8). tar(1) only handles .tar files, and that's a silly UX for any user-visible task. The only time you already know you've got a .tar file, or want a .tar file, is in a script—and in a script, you want exactly the kind of verbosity, strictness, and interposed-shell-gloop flexibility I was talking about.
Meanwhile, what a user wants 99% of the time for their TTY-interactive use, is a command equivalent to the things a GUI archive-manager does. Users don't want or need tar(1); they want an xdg-archive(1) that uses libarchive.
> Security?
Not the right layer to enforce that in; that's what the [no_]user_xattr mount option is for. (Or, alternately, hierarchical xattr namespaces, with everything coming from foreign sources tainted into a quarantine namespace.)
> Wait, what? "fail silently"?
rsync(1) actually prints a ton of warnings (one for each file, in fact) when you tell it to copy EAs but it can't manage either the read or write step. But it doesn't fail. Because, like I said, there's no use-case. rsync(1) mostly isn't used TTY-interactively; it's used as a recurring batch job. If you said you wanted data+EAs the first time, and the remote's sysadmin remounts their computer's filesystem so EAs stop being available, you don't suddenly want to stop syncing the data. You just want a syslog full of errors telling you that you missed the EAs, plus smart logic such that you can message the other sysadmin telling them to fix their box, and then rsync(1) will gracefully just add the now-available EAs to the files rather than having to re-copy the files. Which... it does. So, great!
(I mean, I can see a use-case for rsync being told "these files should have EAs" and then performing its ...
http://bropages.org/
Similarly, documentation seems to have apexed with the TeXbook. :(
I'm curious how much of this still stands today.
0. http://www.cvedetails.com/product/383/Openssl-Openssl.html?v...
1. http://www.tedunangst.com/flak/post/analysis-of-openssl-free...
OpenSSL fixed the problem quickly, but one year on still haven't accepted the regression test for the issue. It would be amusing if it wasn't so horrifying.
"Experts"... lol...
So it doesn't surprise me that unit testing, documentation, code review etc. weren't a top priority for spending more unpaid hours on - people literally got what they paid for.
That plus the larger trend of developers ignoring basic, good practices is why I critique the project a bit. Plus, LibreSSL team illustrated my point nicely by doing 10x what I expected in a very short time with no pay.
In contrast, libsodium deserves praise for writing documentation like they want people to actually use their library.
In the case of openssl, you might be better off rolling your own. At least the vulnerabilities you end up with are different than the ones that the rest of the world has.
"dont try and do it all on your own but trust no one else to do it for you" probably better.
An open, modular project with a wide choice of options would be a godsend and wipe out most digital crime almost overnight.
Of course, then, they couldnt use the likes of yahoo and google to read drug dealers emails.
Yet it has all the CA crap thrown in, via the overloaded openssl binary. As "examples". And according to the documentation, not even "correct" illustrations of how libssl should be used.
Encryption and authentication are two separate problems.
Just because you figured out a way to encrypt a message does not mean you have also figured out how to a way to send it to only the correct recipient... over an insecure network. (Insecure not only in the sense of "plaintext" but in the sense you are not in control of much of anything - routing, PKI infrastructure, etc.)
It seems to me that one would want to solve the authentication problem first, and then move on to encryption.
This comment shows that for proponents of using SSL on the public web, it's been the other way around. Authentication was never sorted out.
When it comes to authentication, all due respect to the OpenSSL authors, SSH has provided a better attempt at a solution than any implementation of PKI using SSL/TLS.
And one more thing, how many ciphers does a user really need? As we've heard time and again, many of them are not even "safe" to use. Some of the alternative SSL libraries have wisely removed them. But I guess OpenSSL is append only?
shock horror it shows.
i find it really quite painful that no one seems to be taking this as seriously as it deserves.
cost must be literally hundreds of billions a year now of electronic crime simply because we have been denied secure communications from day 1.