Ask HN: How should the CLI be designed today?

85 points by danfrost ↗ HN
I've long thought that the command line is just a bit rubbish, even if it is powerful.

If you were redesigning it today (2016), how would you lay it out and what it do?

E.g. collapse output. Keep more metadata on history. Built in error hints (e.g. google the error, harvest possibilities).

How should the CLI work across microservices and cloud services?

What should the graphical design be?

This is deliberately open. Looking for pet hates and ideas.

90 comments

[ 6.0 ms ] story [ 158 ms ] thread
Plan9?
I used a port of the rc shell for linux as my login shell for a couple of months. I switched back to zsh eventually because rc doesn't have job control and autocompletion. The autocompletion wasn't that big an issue because it can use readline for history completion, and it can be configured so that when you type the start of a command and hit up arrow it will find the last command that started with what you typed. This can be replicated for zsh and bash, and it's arguably as good as autocompletion. I found no convinient work-around for the lack of job control, other than starting a new terminal.

How is rc better than bash or zsh? I've read the bash man pages and I couldn't figure out exactly how bash expands commands before executing them. Every time the shell see's an unquoted variable name (e.g. $foo) it will do word splitting, which is why you have to write "$foo" instead, or "${foo}bar" to distinguish "$foo" from "$foobar", etc. Add history and filename expansion and command substitution, weird syntax for control statements...

The best feature of rc is that word splitting and globbing happens once, and it results in a list of strings. There is no longer a need to quote variables all the time. No need to use filenames without spaces.

Here's what bash does:

    $ files=*; echo "$files"; echo $files
    *
    file1.txt file2.txt file3.txt
    $ # echo $files -> echo * -> file1.txt file2.txt file3.txt
Here's what rc does:

    $ files=*; echo $files
    file1.txt file2.txt file3.txt
    $ # files=(file1.txt file2.txt file3.txt)
    $ # echo $files -> echo file1.txt file2.txt file3.txt
It's similar to the situation for dynamic vs lexical scope.

http://plan9.bell-labs.com/sys/doc/rc.html

https://github.com/rakitzis/rc

As an aside, in terms of design and little helpers (e.g. git/virtualenv) powerline shell really made my terminal much more friendly: https://github.com/banga/powerline-shell

Idea: proper graphics for data/image/video display.

I think it's an interesting idea that a terminal editor can have text only input, but the ability to draw real high-res graphical output. e.g. a canvas to draw your matplotlib diagram in the terminal output or a more graphical version of htop (with time data graphs etc).

> Idea: proper graphics for data/image/video display.

Too late. xterm already has it for a long time (I remember seeing it working as long ago as in 2003 in rxvt, and it was nowhere new thing then). See the screenshot of w3m, a lynx-like web browser: https://en.wikipedia.org/wiki/W3m

My today's rxvt-unicode also supports this mode.

People say Windows' Powershell using some sort of objects rather than a text stream is good.

For me I want better control over how my shell records and stores history. I want it to record more and probably not in a flat text file. I don't want it to write commands that fail to disk but I do want them in my history so I can correct my typo and run again.

I'd like lines to be rewrapped if I change terminal dimensions. I'd like more automatic pagination (rather than forgetting to use less and nuking your scrollback). I think is an issue with mintty/putty but I'd like its command history to be printed correctly after changing the terminal width.

Pet hates: terminal bells that I didn't explicitly ask for; middle click paste; the obscure terminfo file that sometimes controls the features a program uses.

Powershell is like the Java of shells: object-oriented, but needlessly verbose.
Eh, I like the Powershell angle here. By default everything is named in lengthy, self explaining names, but there's shorthand aliases for everything.

In reality, no one is using `get-childitem` instead of `ls` or `dir` (both of which are simply aliases for `get-childitem`).

Autocomplete globally for every command and its subcommands
While this would be great, I doubt it will ever happen. I don't really see how this possible without support from each command being completed. And not every command is going to do this.
Maybe machine learning could help?
Going through man pages would certainly help. Otherwise you're analyzing the source code of scripts or worse, binaries. I won't say it's impossible, but I'm not convinced the effort is worth it. That said, I think it could be made much easier to make completion scripts rather than all the shell-specific stuff that exists now.
--help would be a good thing to learn from too. But again, that assumes that --help or even a man page was implemented.
A simple structured list in config seems like a reasonable start.

    - git
      - blame
      - commit
      - log
      - ...
We could generate this semi-automatically for a lot of commands with a little effort. Perhaps git wasn't the best example to demonstrate a simple one.

For existing commands that don't get updated, one could just supply the list from a central repo, not unlike Homebrew Cask.

Default support for control-f searching output
OSX Terminal does this (but with Command-f).
Control+Shift+F on Konsole (in KDE).
Easier inline editing of command args i.e accidentally executing the script when trying to enter a newline.
Current best solution: set $EDITOR to your favourite editor and Ctrl-x, Ctrl-e.
Hyperlinking is good, we should do more of that.

I have my urxvt configured to open http(s) URLs in my browser, but it would be nice if I could open URLs of arbitrary protocols via handlers. Similarly, being able to distinguish valid pathnames and open them in $EDITOR/$VISUAL (or perhaps via special file-type handlers) would be a nice.

Overall, I'm not particularly dissatisfied with the state of the command line in 2016. My experience has been that I make no more (or fewer) mistakes with command-line interfaces than I do with graphical ones, and that spending a few seconds thinking about what I actually want to do prevents me from using the wrong approach (i.e., a graphical or console program) to a problem.

If we could start with having working ctrl-c ctrl-v that would be great :). (I know Windows improved on this with 10 but Linux is still lacking)

Other than that I wouldn't mind some graphical features, such as being able to display thumbnails of images for example.

You can probably map Super+C and Super+V (i.e. the "Windows Key") to copy and paste in your chosen terminal emulator.

(I have done this in KDE's Konsole, so I can paste from a web form I must often use, which somehow blocks the standard Unix/Linux selecting-to-copy, middle-click-to-paste method.)

On Linux I usually use Ctrl+Shift+C, Ctrl+Shift+V which seems to be bound by default on most terminals. Ctrl+C of course terminates the running program, usually.
Why is ctrl-C the shortcut for "terminate", anyway? I would have expected it to be something like ctrl-T.
(comment deleted)
from wikipedia (https://en.wikipedia.org/wiki/Control-C):

As many keyboards and computer terminals once directly generated ASCII code, the choice of control-C overlapped with the ASCII end-of-text character. This character has a numerical value of three, as "C" is the third letter of the alphabet. It was chosen to cause an interrupt as it is otherwise unlikely to be part of a program's interactive interface. Many other control codes, such as control-D for the end-of-transmission character, do not generate signals and are occasionally used to control a program.

Also see https://en.wikipedia.org/wiki/C0_and_C1_control_codes#STX

Type

   man ascii
and notice what is 0x40 less than the capital letter.

H is 0x48, backspace is 0x08. ^H is the same as backspace.

I is 0x49, tab is 0x09, ^I is the tab sequence.

Now you know why Windows \r in text file show up as ^M in vi and Emacs.

^D is ASCII 0x04, "EOT (end of transmission)". That's why you use that to end input.

^C is ASCII 0x03, whose name is "ETX (end of text)". That seems a reasonable choice.

(comment deleted)
(comment deleted)
This is probably one of the biggest advantages of macOS using ⌘C and ⌘V for copying and pasting instead of Ctrl-C and Ctrl-V.

(…well that, and being able to use your thumb for performing the keyboard shortcut instead of having to use your pinky finger).

> Linux is still lacking

Linux isn't lacking it. It just has already decided on a specific shortcut. Do you also consider the absence of Ctrl-C to paste in Safari (in favour of Cmd-C) a lack?

At least on Linux terminal emulators you could always remap it (at your own peril, be careful about setting up another shortcut for the actual interrupt). Pre-Windows 10, OTOH...

In Linux there's no need to Ctrl-v---just make the selection---, and Ctrl-c is instead Shift-Insert. How is that lacking; it saves 2 keystrokes?
The insert key is located in incredibly non-standard and inconvenient positions on many keyboards. For instance, right now I'd have to hold down a special mode-shift key, shift, and then find the insert key, which I don't believe I have ever typed on purpose in my life - it basically exists as a huge PITA, like the Caps-Lock key, that occasionally puts me into overwrite mode or does other things I didn't intend when I fat-finger it.
In TCC/LE (a CMD replacement for Windows by JPSoft) there is one little feature that I miss in Powershell because I got so used to it: if you just enter the name of a folder (as if it's a command), you change dir to that folder (so no need to type the 'cd').

In Powershell you just get an error that this thing is not executable. A waste of semantic space, IMO.

Would love something that showed you how the current command was going to be parsed as an argument vector before actually submitting the command (pressing return). Would help with spacing and quoting.
My primary complaint about Unix CLI is that the commands try to do two disparate things simultaneously: outputting text for humans and generating data for consumption by other commands. In a properly designed command line ecosystem (it's far, far too late for that now), there would be good human readable output in one mode, and tools would exchange structured, already escaped data amongst themselves in another mode. Alternatively, the shell would render the output for humans, and tools would just communicate in machine format.

Consider for example executing a compression command on all jpg files in a folder and below. Currently, the syntax goes something like this:

  find . -iname "*jpg" -exec gfxcompress {} \;
Naive Unix users would have expected this to be achievable by piping, but no! In reality there is a special syntax of the find command that does this, and it's not completely clear how you would deal with file names further down the line, or how you would achieve further chaining. This syntax is also completely unique to the find command. You can't just plug in another data source and perform the same operation. In a CLI where tools are designed to be chained, it would look something like this:

  find | match :name="*jpg" |> gfxcompress :name 
In this example I used an imaginary |> operator to indicate I want something performed on every item, and I can refer to the "name" field of each directory entry directly. If I wanted to filter by time, for example, I might use the :time field instead, without any need for explicit support. If I wanted to mutate the name somewhere in the chain, I could do it. If I wanted to chain additional commands, it would be obvious how. And it would just work.

The same goes for error handling. Imagine a compiler spitting out structured error information which you could use directly in other tools without having to parse and interpret it first.

Windows PowerShell seems to offer exactly what you're looking for.
I'm not a Windows user, so I'm not familiar with it. And "just move to Windows" isn't exactly the kind of solution I was hoping for, if I'm being honest. Some day someone in the vast Linux/Unix ecosystem hopefully sees this problem too and does something about it, but it's a controversial idea at this point.
The usability, robustness and the ease of composition of simple text interfaces is not accidental. Pipes survived for many years for a reason (though they are broken on Windows -- it is hard to pipe data as is without corrupting it).

SOAP, WSDL have taught us already that more "structure" does not mean better. Binary interfaces such as provided by systemd or http/2 may teach us again the importance of simplicity, human-readability and avoiding too frigid constraints failing in the face of natural evolution between different software versions.

The problem with power shell is that it misses basics such as grep that command line users take for granted.
Not really arguing against your point - however when in Powershell, as everything is an object, you don't end up using the same "string heavy" techniques that required grep with more traditional shells.
This. It would be really great to have a language like PowerShell on Linux. Being able to pass file names reliably[0] without disgusting hacks would be great, for a starter. And, unlike in current nix shells, a new shell should separate responsibilities cleanly between tools (`find` has both `-exec` and `-delete` unlike most commands; `grep` has `-c`), return reusable results (damn you `ls`!), have sensible scoping rules, stop on the first error (like `errexit`, `nounset` and `pipefail` implemented sanely), and just generally be more like a real language.

[0] http://www.dwheeler.com/essays/filenames-in-shell.html

The first feature already exists. You write

    for i in **/*.jpg; gfxcompress $i
in Zsh, or

    shopt -s globstar     # (Which you would have in your .bashrc)
    for i in **/*.jpg; do gfxcompress $i; done
in Bash.

Or, if gfxcompress supports multiple arguments, just

    gfxcompress **/*.jpg

Treating the output as objects would be an improvement. I write things like

    for id in $(< file-containing-ids ); do
        r=$(curl -Ss http://api.example.org/get/$id | jq .status)
        if [[ $r ...
        ...
    done
or the more traditional columnar output and awk, but it would be nice if the shell knew about this, enabling tab completion and avoiding a lot of boilerplate.
I was afraid someone might come in and declare this a non-problem, or school me on the example I chose in an effort to invalidate my overall argument. For every example I could conceivably come up with, there is a locally optimal arcane way of achieving a good result. That's not what this is about. Functional (im)possibility was not the message of my argument, ease of use and structural cleanliness was.

The issue here is not really finding a "better" way to implement this exact example, it's finding a better mechanism for what the example represents.

Can you make everything work with today's tools? Absolutely. Can you do it without arcane switches and weird corner cases? Mostly not. The question I'm raising is more whether this pain is inevitable or not.

A shell for loop is not arcane — that's one of the shell's most basic control structures. It uses one fewer external command (no "find"), which is a great way to avoid the weird (and common) corner cases.

But anyway, I think you missed where I wrote, "treating the output as objects would be an improvement", and presented a better example in support of your suggestion.

> But anyway, I think you missed where I wrote, "treating the output as objects would be an improvement"

We already agree about that part (at least sort of). Instead I wanted to address the part where you declared my example invalid and where we're talking about nothing else but the details of this one example. Did I choose it poorly? Was it too simplistic? Maybe, it really depends on your tastes. I content that arguing about whether or not the find command is usefully implemented is besides the point. The problem lies deeper and is more systemic than that.

I'm not arguing that a loop construct is arcane, I'm arguing that in practice such scripts turn arcane really quickly, and that there is something substantial to be gained from having tools exchange information in a structured, machine-centric format - and also from switching to a human-centric control format.

The glob breaks if you have a lot of files, or if you're in a deeply nested folder so the full filepaths are long. So the find command is better for a general-purpose script. You also didn't some variables, which can cause problems in a general-purpose tool.
You can actually do what you want with piping:

    find . -print0 | grep -z '.*.jpg$' | xargs -0 gfxcompress
(-0n1 likely, as it sounded from the example like gfxcompress is designed to operate only on one item at a time.) To make this explicit for readers: find, grep, and xargs actually already has the requested feature of "good human readable output in one mode, and [] structured, already escaped data [] in another mode", through -print0, -z, and -0, which also works great when combined with read -rd $'\0'.
On this point of using a null char to separate things: do you people think that automatically using it when I/O is not connected to a terminal is a good idea?
Sort of.

Obviously, for a great many tools that wouldn't be an option because it would break compatibility for scripts that have been running for decades.

And \0 isn't necessarily a safe delimiter for all types of lists.

But, for lists of file names (and other strings that cannot contain \0), yes, I believe that automatically using it when stdout is not connected to a terminal would be a reasonably good idea.

Code point 0 / ASCII symbol number 0 is valid in ASCII and Unicode strings, so it simply can't be used for termination.

    imaginary |> operator to indicate I want something performed on every item
xargs?
So, Powershell, basically?
You should have a separate output for humans, instead of having a flag for it, and programs which are the last in a chained series should default to it, but you could also redirect stuff to it in the middle of piping.

I'd love if the bash shell language was less clunky.

A standard method for programs to provide help, examples, and tab completion suggestions to the shell. Right now we have man, info, --help, -help, -?, etc, and tab completion is part of the shell rather than the apps. If all of that was standardized the tools could be much better.
I'd basically like to have powershell. Potentially with a better introspection and attributes so that you can build a more interactive CLI. For example output size estimation + tagging local read-only commands to allow showing some output preview as you combine the commands. On-a-side display of help for arguments.

My pet hate is that commands themselves cannot tell you things about them in a structured way. There's no reason why completion and --help are somehow separate from the execution itself.

Love: context aware shell like zsh/oh-my-zsh provide. If I'm in a git project, show me a good git summary for example. If I'm in python venv, tell me about it. If I'm in some python project, extract its name from setup.py.

It should have a daemon to centralize things such as command history across multiple terminals.
A modern protocol to communicate with the terminal would be nice.
Don't google errors. Don't network anything that isn't explicitly requested to be networked. Don't give any built-in affordances for HTTP or other poorly-designed, politically-imposed protocols. I don't even know what you're talking about with microservices, as that notion only makes sense in the context of implementation details of RPC services ("Cloud APIs" in 2016 parlance).

A modern shell would:

- Never use text as the interchange format, but rather keep binary representations in memory and link all commands in a given interaction into a single computation, doing inlining and optimization across the boundaries of what are currently separate processes communicating via pipes/IPC.

- Know about the structure of valid invocations of commands (essentially, their type signature and the algebraic type structure of their parameters), preventing any "invalid invocation" error messages a priori. Only valid options would be permissible for tab-completion, and un-tab-completable options would give error feedback directly without executing the program.

- Have a principled, data-race-free alternative to the filesystem for persistent storage of data, which maintains type information without preventing useful generic operations like bytewise encryption.

- Support unicode as the only character set for text.

- Encourage proportional fonts, so as to make box-drawing characters not work properly. An associated UI system would permits applications to refer neither to character cells nor pixels, but only to a graph of modifiable or immutable values and a set of presentation hints. Platform-specific modules would interpret these based on the capabilities of the local terminal. Full graphical applications over SSH are trivial under this model--no more "I use a shitty 1980's-era UI because that's the only thing I can export across the network consistently".

- Allow addressing both local and remote system resources (RAM, CPUs, I/O devices) via a uniform interface, and allow dynamically attaching and detaching these from running processes, both local and remote.

- From a more traditional shell perspective, it seems important to support "by" operations with good ergonomics. This is stuff like "sort all these files based on the output of applying this other file-accepting command to each" or "filter out all paragraphs in a text file containing the string 'foo', but only print the last seven characters of each one.".

These would be a start.

Looks like IPython Notebook is almost an ideal way to interact with a computer (as an information processing device, rather than, for example, a game machine).
Agree. I think the plain ipython shell can go a long way now. It will pop up a graphic, and you can access the surrounding shell. In theory you can just live in ipython; I've tried it but I stumbled too much.

Next on the spectrum would be the QT version of ipython. And then of course ipython notebook/jupyter.

What really looks promising is the Jupyter Lab, currently in alpha.

http://blog.jupyter.org/2016/07/14/jupyter-lab-alpha/

https://pypi.python.org/pypi/jupyterlab

ipython shell is great for quick jobs, and is taking over a lot of what I used to do with long pipelines of cut/grep/perl -n, etc.

Being able to grab some filenames/contents with

    foo = !ls
    bar = !cat *.txt # returns list of lines
is super handy. And the real shell is always just C-z away, if I need it.
I'd like to see a good CLI design involving sandboxing-by-default, and handling object-capabilities (e.g. file descriptors) as first-class objects.

Right now, "convert a.jpg b.png" passes the strings "a.jpg" and "b.png" to ImageMagick's convert command, but that command is free to open whatever files it wants, since there's no distinction between strings vs resources such as filenames, network sockets, etc.

You can do this with pipes in simple cases, and bash has some support for this, but it's very primitive and the syntax is cumbersome.

Black Screen has some pretty interesting ideas:

https://github.com/shockone/black-screen

Do you know what the roadmap is for this? It looks like they want to build a terminal, but I can't see how it's significantly different from what we've got at the moment.
I find CLI a great interface. Its just stuck in pre web days! Here are some thoughts on its design / UX -

- contextual help. if i write a command incorrect, it should suggest correction and link to more help.

- cloud storage of all my command history

- be more understanding. talk a bit in natural language. Eg. show my ip address -> translated to the right shell command.

- learn and grow automatically as the community builds more tools. Like, hey used this command but check this one out!

It is not feasible to upgrade all existing scripts to adhere to this but a wrapper shell that can provide this functionality should be very well possible.

I've tried using Elixir's iex as a login shell. It was an interesting experience. The pipelining in Elixir actually makes it feel a lot like bash. In Elixir, remote shell [1], autocomplete are built in. Strings are binary. Structs help structure output for human vs machine use (e.g. see HTTPoison.get("http://example.org") vs. HTTPoison.get!("http://example.org"). Processes are cheap. Functions are compostable. Rock on!

http://elixir-lang.org/docs/stable/iex/IEx.html#module-remot...

Although, through a long career, I've become accustomed to a wide variety of CLIs, I tend to agree: We've moved beyond 1999 a bit. The fact that a lot of these systems have legacy and tradition isn't sufficient. Languages like COBOL also have a long history. But now we have lots of other languages to choose from.

First off, as much as possible, I'd eliminate non-words. I tutor a lot and people think perhaps I have a speech impediment the first time I say "grep". (Yes, I know, it's not technically part of the shell. But okay how about "fi" and "esac"? Really? Why not "enod" then?)

I'd take command completion further: IDEs often have drop-downs and function signature descriptions as you move along coding. In a hypertext world with tooltips, etc, I think there's be a lot of room for those types of ideas.

I do think the philosophy of "do one thing and one thing only, but do it well" is a good one. Piping stuff together is wonderful. I wonder if it would make sense and be shorter to explain in plain English, if the "water flowed the other way" instead. For example:

"sort the output of the find result" = sort | find

as opposed to:

"take the output of find and sort it" = find | sort

(That last bit was just a random thought of the moment, just trying to think a wee bit outside of the box.)

It wouldn't be enod, it would be rof