The concept presented is something I can agree with in principle, but "transforming a filename into the content of the file" is a really thin justification for a responsibility.
By all means don't build something where you have cascading effect and need to retest an entire pipeline, but this is _not_ it.
P.S.
And if you really really want to keep it separate, just do "< access.log head -500 etc etc etc" (no I didn't forget a pipe. And yes the "< inputfile" works even if it's in front of what you're calling).
> And if you really really want to keep it separate, just do "< access.log head -500 etc etc etc" (no I didn't forget a pipe. And yes the "< inputfile" works even if it's in front of what you're calling).
Or just use `cat` and let the pipe separate the different steps. "< access.log head" is nice but it breaks this representation where each step is piped into the next one. Sure, once you’re done fiddling you can rewrite the thing to remove the "cat", but when you are constructing the thing I find it clearer to use cat.
Then the arrow would be pointing in the right direction. But then it would be unclear whether "infile" is a file or a command. Which is why people use:
cat infile | some_cmd > outfile
You can now interpret "cat" as a keyword that specifies that "infile" is a file.
> Just like you wouldn't add spaces in the middle of '2>&1' when redirecting stderr to stdout:
Mostly because if you do it doesn’t work: '2 >&1' is not the same as '2> &1' (invalid syntax) which is not the same as '2>& 1', which …is the same as '2>&1'.
It's not really a precedence relation as much as it's part of shell syntax. There are separators[0], and anything sequence of tokens between a separator is a command[1]. A command decomposes further, of course, but the only thing the programmer needs to know here is that separators are a different kind of thing than commands.
> transforming a filename into the content of the file" is a really thin justification for a responsibility
This is one of those things where I think it is until it isn't.
I sometimes second-guess myself when I think I might be over-single-responsibilifying. "Well in practice these two things are so trivial that this feels a little silly."
It often turns out to have been a good call in hindsight, especially when working with other people who aren't necessarily thinking about these things at all. If the responsibilities have been sufficiently split up, they're more likely to change only the part that needed to be changed, and less likely to complectify the two things together that really shouldn't've been. Or when I go "oh wow that thing that I thought I overly-abstracted sure composes well with this unexpected new thing!"
Hardcore separation of concerns is just another method of defensive programming.
> < access.log head -500 etc etc etc
It's too bad that the syntax is so different. Why does the first stage not end with "|"? There's space for shell syntax improvements, here. Maybe a 'cat'-like builtin that translates `cat foo | bar` into `bar <foo` so you can have the nice syntax but don't needlessly create processes would leave everyone happy.
> "transforming a filename into the content of the file" is a really thin justification for a responsibility.
Uh... I dunno, but my lizard brain thinks that the whole idea of mediating filesystem operations on storage and IPC mechanisms like pipes is a lot more complicated, magic, and deserving of a single command than merely filtering the data on stdin.
I agree with the article and the logic, and think this historic meme was basically wrong originally. You string up your chain of pipelines with the first element being "where does it come from?" and not merely whatever the first operation happens to be just because that operation allows for some kind of file input or redirection syntactically.
> so yes, "cat xyz" plays a source - and can be replaced with another source - without touching all other stuff.
This. I often have to read logs and being able to apply the same oneliner on older logs just by adding a "z" in front of "cat" to read .log.gz instead of .log files is extremely useful.
My cat recently discovered that her fur can absorb a toilet bowl full of water. This seems like it could be useful, if you need a very wet floor immediately.
Here you go. This is the second time in the past few years that cat was used to process bikers in the NW WA wilderness. The first time was fatal. I used to live about 15min from where this incident occurred.
https://www.seattletimes.com/seattle-news/cougar-attacks-cyc...
an useful use, back in time, was to assemble partial ISO file of OpenSolaris SXDE, the first open version before Ian Murdock/OpenIndiana. It's more an anecdote but considering the usefulness of netcat in many cases such "cat binary use" might still be relevant in some embedded scenarios.
I'm a bit surprised `rev` isn't in coreutils, it's ancient. Originally written because `cut` doesn't allow selecting fields from the end of a line. So to get the last field in each line, you reverse each line (right to left) with `rev`, select the first field with `cut`, and then reverse it again with `rev`.
On my Ubuntu machines, `man rev` indicates that it came with a package called util-linux, so it seems like it might actually considered even more essential than coreutils. Funny enough, that manpage lists `tac` under its "See Also" section, so maybe these two packages are tied together at the hip.
You describe a really clever use case for rev. On the surface, my intuition was that reversing, cutting, and reversing again wouldn't work if you had escape sequences, but I suppose that's a moot point with cut!
Something that "cat file | ..." advocates might be overlooking is that a redirection ("<inputfile", ">outputfile", "2>errorfile") can appear anywhere within a simple command, so these:
are all exactly the same -- and of course very similar to
cat file | command -option
If the purpose of typing "cat file | command" is to put the input file at the beginning (which does make logical sense), you can achieve the same thing with "< file command". Admittedly, it does look at bit strange if you're not accustomed to it. (It even works with csh and tcsh.)
Because that's what you get if you address the actual argument and still insist on input redirection.
Input redirection is inconsistent with every other command to retrieve data. Not only does it not have the same syntax, it's combining two actions into one step of the pipeline.
Redirection isn't a step in the pipeline; it's syntax which indicates to the shell that it should perform certain file descriptor manipulations when it arranges the pipeline.
I would say that < <(command arg) is a useless use of process substitution (UUoPS). You just want command arg |.
The redirection variant doesn't eliminate command and does not move arg out of command's argument position; it's just superfluous syntax.
Just because we want "< file" instead of "cat file |" doesn't imply that we want "< <(command arg)"
instead of "command arg |". It's not even the same rewrite pattern at all.
However, if someone wrote:
cat <(curl https://example.com/file) | next
then that now the UUoC pattern "cat file |". We can apply the transformation to eliminate cat:
< <(curl https://example.com/file) next
Now in so doing, we have moved the process substitution such that there is an obvious match for the UUoPS pattern. We apply that rewrite rule as well:
"Get data" is the pipeline step we're talking about. Using "< file" combines it with the first transformation step, instead of keeping it as its own separate step as all other such data sources require.
It isn't implementation-wise, but it is semantically. If anything, < is a useless syntax that should never have been in the shell to begin with. If you want to peephole optimize cat... just do that, just like it replaces tons of other commands with built-ins.
This blogger does not seem to know that in the POSIX shell syntax, redirections can be specified anywhere in the command:
< access.log head -n 500 | grep mail | perl -e
Now you can delete "head -n 500 |".
> If we then delete only the head processing step we’re left without a step that transforms the string access.log into the contents of the access log.
By introducing "cat access.log" we have the same problem: if we delete only the cat processing step, we're left without a step that transforms the string access.log into the contents. For the useless cat to have the nice property that you can cleanly delete it from the command line, you need:
The useless cat plays the role of ‘a process that produces a stream’, which gives you higher confidence that you can substitute it with a different ‘process that produces a stream’ - like the curl command mentioned in the article, or perhaps a server whose stdout output you want to analyse.
It should be the same as giving your pipeline a file input handle via < access.log… but why take the risk?
That's the whole mistake. Streams are not sourced by processes, but by kernel objects. You don't need a process to read bytes from a serial port, for instance. You don't need "cat /dev/ttywhatever | program". Interrupt handlers in the drivers already drive the activity of bytes being received.
I'm in this camp as well, starting with input file redirection just makes so much sense to me.
</proc/0/environ xargs -0
I also don't tend to want enormous volumes of text in my terminal scrollback so I generally view files or pipe verbose commands to `less`, then when I find what I want to send to the terminal I use the `|` less command to pipe it to `cat`.
Or to grab just a few lines for my later reference:
kubectl get po/my-pod -o yaml | less
/* find the lines I'm interest in */
-N
|^sed -n 34,35p
What do you mean? It's specifically for $(cat …) / $(< …): the latter is a faster equivalent of the former. Other than that, see my comment about $READNULLCMD.
Another reason to do `cat | foo` is because you can then quickly change it to `zcat | foo` or some other source for the data by editing the very beginning of the command.
Bash (and similar) will replace !$ with the last parameter of the previous command.
This is a trick I’ve used lots when wanting to perform a non-piping operation on a command I’ve ‘cat’ed (eg ‘rm -v !$’)
I’d never criticise anyone for “useless” use of ‘cat’ though. If the fork() overhead was really that critical then it wouldn’t be a shell command to begin with.
Even easier, press alt+. (fewer keystrokes too! cli golf is fun), it'll copy-and-paste the last parameter from the previous command. If you press . multiple times it'll go further back into your history.
Often the shortcuts that people point out in these threads are too niche for me to remember, but that seems like a pretty great one, I’ll try to remember it.
As I tend to live with set -o vi, [esc] _ gets the same. I hardly ever remember to set:
bind '"\e."':yank-last-arg
So [esc] _ does it on systems where I haven't customised my environment. A major drawback though is it doesn't go through history like alt . does.
However, when 'set -o vi' is enabled you can easily go through history and edit with familiar vi keystrokes, or press 'fc' and fix the previous command in EDITOR, or 'v' to edit the current command in EDITOR.
'v' will start the visual editor. 'fc' will edit the previous command. Both invoke EDITOR in bash's vi readline command mode.
I wasn't aware that bash and zsh did it differently, I assumed they'd both use the same readline - now I'm aware there's more to understand about it :)
Ha my bad I assumed zsh because I saw the `bind` command being used. I did not know that it was also in bash
I find that zsh's mode is actually better than readline's
For example it can handle visual mode which is why the same function is `vv` and not just `v`
Zsh can handle text objects too, very useful to be able to ci" for example
Fish's can too and is quite good IIRC
Too bad the `v` command does not work in gdb so it seems it's more of a bash thing than a readline thing
I look through a lot of logs, so I've aliased `not` to `grep -vE` and using the method you describe over several iterations i end up with a history that has a lot of
cat log | not spam1 | not spam2 | not 'spam(3|4)' | .... | less
Something sets $PAGER to “rm -rf /“ or something along those lines.
Really I’d be more worried about accidental invocations. Aliases are not scoped, so if you’re dealing with one of those programs with non standards option handling and uppercase single character switches…
As for accidental invocations, yeah I agree it feels dangerous, but in a few hundred thousand lines of shell history since I set them, never had a problem.
Haven’t had a problem, or haven’t noticed you’ve had a problem? I kid - but only partially.
What I really wish for is some sort of tool that would let me pipeline like that, but also easily examine each step in the chain for sanity. Sort of a workbook for shell.
Tons of things use environment variables like EDITOR or PAGER, and tons of things can "inject" "rm -rf ~" or whatnot. But if they can inject that, they can probably also go all sorts of other nefarious things. There is nothing special about a global alias here.
And an accidental typo is not really a big deal, certainly not an "injection possibility".
If we're going to talk about unnecessary extra processes like useless cat, we should merge the head and grep commands into a sed, and possibly just merge everything into perl:
<access.log sed -n '/mail/p; 500q' | perl -e ...
If perl is processing the file line-by-line then filtering lines by regex and stopping at line X is trivial, and you don't even need sed.
I think your point is valid and I don't dispute it, but if I had a nickel for every time I said "just do it all in perl" and regretted it, I'd be... well, perhaps not a rich man, but I'd have lunch covered for a few weeks.
Perl has the advantage of only having one implementation, unlike sed and grep (e.g. BSD or GNU) and /bin/sh (can be one of many POSIX shells), so upgrading this pipeline to 100% perl is safer in some respects. The example in the article is light on details so it's hard to comment very deeply.
I have heard snarky Perl putdowns ad nauseam at work and on HN and may have regretted using it a handful of times but I can say worse or similar for other popular tools, languages...
When you're done with the part after the &&, remove the 'last if $. > 500'
For me, the most useful use of (gnu) cat, is cat -A weird.file
It saves my day or solves weird issues (X-files) with files generated by (not so) junior sysadmins, copy/pastes, end of lines, invisible diffs, etc... many times each year.
I like this sort of blog post - simple, clear example in a short format. Of course we can nitpick the implementation but I think the principle is solid: don't be afraid to sacrifice some brevity for modularity or composability.
I do think he could have woven in the Parnas source a bit better. The first quote seems to rely a lot on the surrounding context of the essay and its never contextualized. This bit:
> The problem is that these subsets and extensions are not the programs that we would have designed if we had set out to design just that product.
Just feels a bit disjointed in the article. I get the high-level message but thought it could be woven into the blog authors narrative better. Anyways, it sounds like a good article though.
Author here. You're right. Originally I had a much longer excerpt but it got a bit rambly. I overcorrected and cut out too much; I'll see what I can do to fix it. Thanks!
It's a nitpick BTW. Of course a bit more context would help I think, but I probably appreciate your edits to keep it brief too. As I mentioned, I really appreciate how concise and to the point it was. Thanks for writing.
I actually noticed your reply because I came back to this comment to save your blog. Not just for the content but the page style. The eggshile white (or similar) color, font, foot(side)notes. It's nice.
I also use cat this way, and for me the biggest reason is it just allows for a more intuitive left-to-right reading of any pipeline.
Things like this:
head -n 500 access.log | grep ...
head -n 500 <access.log | grep ...
Feel like you start with the filename, then go leftwards to the first operation, then start reading rightward again through the pipe. At least in my brain, it feels slightly more awkward.
I wish math notation and computer programming had just settled on postfix over prefix early on, it's so much more natural to read. Of course, we kind of get it with object oriented programming, some languages have UFCS [1], F# has that pipe operator etc.
It's funny, when learning programming, I think Haskell was the language that introduced me to the pattern of having a chain of operators processing a stream to build up a result (and I'd later cover it again in SICP), and I loved how clean it looked compared to imperative code. But I now find it one of the harder to read languages due to it all being prefix, whereas Java/Kotlin/C#/Javascript now all have stream constructs that use method calls, so read left-to-right, source-to-sink
And I'm reminded that I need to give Forth a proper go sometime
Somewhere along the way it had been so long since I used cat to concatenate files that I had to relearn to do so. That felt either very silly or, “I’ve forgotten more about X than you’ll ever know”
Honestly, I've being UUOC all my life and everything is fine. Never run into a problem. I mostly use `zsh` and `< file | cmd` works easily even when you remove the `cmd` so that's fine. But honestly, it's both bought and lost me nothing, which means I shouldn't care about it.
What's really useless are all the "useless use of cat" comments (and shellcheck warnings although I take it for a shell script there could be cases where one less process may be justified [although really if you're at that point you've got other things to worry than cat, sadly]).
I use "cat ... | ... | ... " like in TFA and just like many in this thread because it simply makes sense. It's more intuitive. It's easier to read. It requires less braincycles to remember how this or that command wants its parameter passed, etc.
I think the "useless use of cat" movement made its time: it failed. Many of us are never going to give up our use(less|ful) of cat (you decide). So stop wasting your time complaining about it.
I use shellcheck a lot, and I've found so far that its complaints fall neatly into one of two buckets: "oh, whoops, nice catch" and "shut up, shellcheck".
"useless use of cat" goes into the latter bucket. Complaining at me about it does not actually improve the code; it's just a nag about a bad habit that, arguably, isn't even a bad habit.
I think you're mistaken here, and confusing two different usages of cat.
"Useless" uses of cat aren't bad habits during interactive usage, for all the reasons people mention here which I won't rehash.
For scripts, however, the story is different than for one-off commands. For one thing, it's slower due to the extra forks and copying of data across pipes, so there's at least that. For another, it prevents the command from inspecting the other end of its pipe, which can negatively impact usage in some case. (For example, if the program knows its input is from a terminal, it may flush its output on every newline it sees.) Moreover, a bunch of the arguments for the interaction case (like "it's fewer keystrokes" or whatever) don't even apply to the script case in the first place...
The end result here is that you definitely shouldn't assume some habit is just fine with scripting merely because it's fine when you're typing on the terminal, or vice-versa.
For shell scripts, I would argue quite vehemently that the most important goals should be correctness and readability, with performance being a very distant third concern. I'd even be tempted to argue that performance shouldn't be a consideration at all, except of course that argument would be misinterpreted to support some absurd edge case until I'd have to admit that of course performance is a little bit of a concern. But in any case, I can't recall a single example of a cat pipe being the root cause of an unacceptable performance problem in a shell script.
On the readability point, the example that probably irritates me most often is a cat pipe into some commands into a while loop. I much prefer this:
cat file.conf | sed -e 's/pattern/replacement/g' -e 's/reallybigolhonkinpattern/other-replacement/g' | tr... | while read line; do...
to this:
sed -e 's/pattern/replacement/g' -e 's/reallybigolhonkinpattern/other-replacement/g' wrongfile.conf | tr... | while read line; do...
or this:
sed -e 's/pattern/replacement/g' -e 's/reallybigolhonkinpattern/other-replacement/g' | tr... | while read line; do
stuff...
done < file.conf
...and that's a pretty common pattern where the edge case of reading input from a terminal doesn't apply.
So this is the kind of thing that makes me go "shut up shellcheck" instead of "thanks shellcheck!"
Fork performance is a much more severe problem on Windows (WSL1, MSYS2, etc.) than Linux, so I'm not claiming you'll personally run into it per se, but it can affect users of some scripts.
But: performance was just one of the problems I cited. I gave you more than that -- one was a correctness reason (which you do care about) and had nothing to do with performance. And, again, incorrect buffering (which can make the script literally unusable in some cases) was just one example. I've seen needless redirection interfere with Ctrl+C handling too, though I don't recall the exact example. Oh, and there's terminal coloring and ANSI escape processing too, which programs detect differently. Point is, being unable to see the end of the pipe can definitely cause an unnecessary mess in some cases.
As for readability - honestly, part of the reason you find it less readable is that you're missing something else. Namely, this:
sed -e 's/pattern/replacement/g' -e 's/reallybigolhonkinpattern/other-replacement/g' wrongfile.conf | tr... | while read line; do...
should really have been:
sed -e 's/pattern/replacement/g' -e 's/reallybigolhonkinpattern/other-replacement/g' -- wrongfile.conf | tr... | while read line; do...
which is in fact both more correct (at least when the file name isn't hard-coded, which is the common case in shell scripts) and more readable than your example; you can immediately spot where the file name is. The difference between that and cat "$blah" | sed ... is very minor at that point (and in fact you should be doing cat -- "$blah" as well...); anybody reading a command like sed without a pipe input knows to look for an input argument. The important point regarding readability here is, it's not like the code gets overly tricky if you write it one way vs. another way. It's just a matter of spending 1-2 extra seconds glancing over. So it's very much a minor thing to be prioritizing above everything else. (If the logic became harder to reason about, that'd be a different story, and it'd put more weight on the readability aspect.)
It's unclear if you missed the filename being wrongfile.conf. Embedding a -- in the middle of a long series of arguments isn't the magic pixie dust that suddenly makes the filename argument stand out the way that it does when it's the very first argument in the pipe.
Yes, I saw your other points, and I chose this example because it is an example drawn from real-world use where there is zero objective reason to wag a finger about "useless use of cat". Those other points are not relevant in this example, and piping a cat into some other commands into a while loop is pretty typical shellcode. Forcing me to move a filename argument into the middle of a long line for stylistic reasons should be obviously wrong. It is one case where shellcheck is over-reaching and being a nuisance rather than helping me catch errors.
If you deliberately chose a specific example then you forgot what this discussion was about? You're giving an existence proof. Yes, there exist situations where it's fine. But this discussion was about what constitutes good habits, not about cherrypicked counterexamples. The whole point of paying attention to good and bad habits is that they sometimes make a difference, and if your habits are bad, you'll sometimes get yourself in trouble.
And it's not like you can reasonably expect shellcheck to distinguish the benign cases from the potentially problematic ones either. It has to give you a recommendation about what constitutes good habits, and avoiding redundant calls to cat is a good habit for all the reasons I just listed, even if in some cherrypicked cases it provides ~no benefit.
I’m guessing none, since if I assume the commands are executed sequentially then, at the time of execution of killall, the cat commands are not yet running.
185 comments
[ 2.8 ms ] story [ 49.9 ms ] threadBy all means don't build something where you have cascading effect and need to retest an entire pipeline, but this is _not_ it.
P.S.
And if you really really want to keep it separate, just do "< access.log head -500 etc etc etc" (no I didn't forget a pipe. And yes the "< inputfile" works even if it's in front of what you're calling).
Or just use `cat` and let the pipe separate the different steps. "< access.log head" is nice but it breaks this representation where each step is piped into the next one. Sure, once you’re done fiddling you can rewrite the thing to remove the "cat", but when you are constructing the thing I find it clearer to use cat.
Or just… don’t?
No it doesn't?
is completely valid, and reads right-to-left as well as the cat version. IMO using stdin is preferable to either solution in TFA.Mostly because if you do it doesn’t work: '2 >&1' is not the same as '2> &1' (invalid syntax) which is not the same as '2>& 1', which …is the same as '2>&1'.
[0]: newline, ‘||’, ‘&&’, ‘&’, ‘;’, ‘;;’, ‘;&’, ‘;;&’, ‘|’, ‘|&’, ‘(’, or ‘)’.
[1]: https://www.gnu.org/software/bash/manual/bash.html#Simple-Co...
This is one of those things where I think it is until it isn't.
I sometimes second-guess myself when I think I might be over-single-responsibilifying. "Well in practice these two things are so trivial that this feels a little silly."
It often turns out to have been a good call in hindsight, especially when working with other people who aren't necessarily thinking about these things at all. If the responsibilities have been sufficiently split up, they're more likely to change only the part that needed to be changed, and less likely to complectify the two things together that really shouldn't've been. Or when I go "oh wow that thing that I thought I overly-abstracted sure composes well with this unexpected new thing!"
Hardcore separation of concerns is just another method of defensive programming.
> < access.log head -500 etc etc etc
It's too bad that the syntax is so different. Why does the first stage not end with "|"? There's space for shell syntax improvements, here. Maybe a 'cat'-like builtin that translates `cat foo | bar` into `bar <foo` so you can have the nice syntax but don't needlessly create processes would leave everyone happy.
Uh... I dunno, but my lizard brain thinks that the whole idea of mediating filesystem operations on storage and IPC mechanisms like pipes is a lot more complicated, magic, and deserving of a single command than merely filtering the data on stdin.
I agree with the article and the logic, and think this historic meme was basically wrong originally. You string up your chain of pipelines with the first element being "where does it come from?" and not merely whatever the first operation happens to be just because that operation allows for some kind of file input or redirection syntactically.
but that absolutely kills modularity / composability
so yes, "cat xyz" plays a source - and can be replaced with another source - without touching all other stuff.
This. I often have to read logs and being able to apply the same oneliner on older logs just by adding a "z" in front of "cat" to read .log.gz instead of .log files is extremely useful.
- pest control
- entertainment
- transporting small solar arrays
I definitely need to hear more about this one.
cat like sleeping in sun.
attach solar panel to cat.
cat occasionally moves to remain in sun as it naps.
From this gem of a DEFCON talk: https://youtube.com/watch?v=rJ5jILY1vlw
You describe a really clever use case for rev. On the surface, my intuition was that reversing, cutting, and reversing again wouldn't work if you had escape sequences, but I suppose that's a moot point with cut!
`Y x | Z` is Verb-Subject-Object.
That's why I prefer using cat "uselessly".
I.e.
? that's... something else entirely.Something that "cat file | ..." advocates might be overlooking is that a redirection ("<inputfile", ">outputfile", "2>errorfile") can appear anywhere within a simple command, so these:
are all exactly the same -- and of course very similar to If the purpose of typing "cat file | command" is to put the input file at the beginning (which does make logical sense), you can achieve the same thing with "< file command". Admittedly, it does look at bit strange if you're not accustomed to it. (It even works with csh and tcsh.)Input redirection is inconsistent with every other command to retrieve data. Not only does it not have the same syntax, it's combining two actions into one step of the pipeline.
I would say that < <(command arg) is a useless use of process substitution (UUoPS). You just want command arg |.
The redirection variant doesn't eliminate command and does not move arg out of command's argument position; it's just superfluous syntax.
Just because we want "< file" instead of "cat file |" doesn't imply that we want "< <(command arg)" instead of "command arg |". It's not even the same rewrite pattern at all.
However, if someone wrote:
then that now the UUoC pattern "cat file |". We can apply the transformation to eliminate cat: Now in so doing, we have moved the process substitution such that there is an obvious match for the UUoPS pattern. We apply that rewrite rule as well:"Get data" is the pipeline step we're talking about. Using "< file" combines it with the first transformation step, instead of keeping it as its own separate step as all other such data sources require.
It isn't implementation-wise, but it is semantically. If anything, < is a useless syntax that should never have been in the shell to begin with. If you want to peephole optimize cat... just do that, just like it replaces tons of other commands with built-ins.
> If we then delete only the head processing step we’re left without a step that transforms the string access.log into the contents of the access log.
By introducing "cat access.log" we have the same problem: if we delete only the cat processing step, we're left without a step that transforms the string access.log into the contents. For the useless cat to have the nice property that you can cleanly delete it from the command line, you need:
:)It should be the same as giving your pipeline a file input handle via < access.log… but why take the risk?
Or to grab just a few lines for my later reference:
[1] https://evalapply.org/posts/shell-aint-a-bad-place-to-fp-par...
I wish I could write "useless cat" without people pestering me about it.
https://zsh.sourceforge.io/Doc/Release/Expansion.html#Comman...
0: https://github.com/wofr06/lesspipe
This is a trick I’ve used lots when wanting to perform a non-piping operation on a command I’ve ‘cat’ed (eg ‘rm -v !$’)
I’d never criticise anyone for “useless” use of ‘cat’ though. If the fork() overhead was really that critical then it wouldn’t be a shell command to begin with.
Press Escape, then the number of the argument from the previous command, confirm with Ctrl+Alt+Y.
Example:
Escape, 1, Ctrl+Alt+Y gives you arg1.However, when 'set -o vi' is enabled you can easily go through history and edit with familiar vi keystrokes, or press 'fc' and fix the previous command in EDITOR, or 'v' to edit the current command in EDITOR.
I think you meant 'vv' and I wonder if it's not something that's set up by oh-my-zsh ? It's great though
I wasn't aware that bash and zsh did it differently, I assumed they'd both use the same readline - now I'm aware there's more to understand about it :)
I find that zsh's mode is actually better than readline's For example it can handle visual mode which is why the same function is `vv` and not just `v` Zsh can handle text objects too, very useful to be able to ci" for example Fish's can too and is quite good IIRC
Too bad the `v` command does not work in gdb so it seems it's more of a bash thing than a readline thing
Relevant : https://superuser.com/questions/1543120/make-readline-edit-i...
Then I can look at it before hitting return
useful shorthand
> sudo !!
for example.
Again, bash specific.
Really I’d be more worried about accidental invocations. Aliases are not scoped, so if you’re dealing with one of those programs with non standards option handling and uppercase single character switches…
As for accidental invocations, yeah I agree it feels dangerous, but in a few hundred thousand lines of shell history since I set them, never had a problem.
What I really wish for is some sort of tool that would let me pipeline like that, but also easily examine each step in the chain for sanity. Sort of a workbook for shell.
And an accidental typo is not really a big deal, certainly not an "injection possibility".
UUoC criticism, to me, belongs when one sits down to script.
https://zsh.sourceforge.io/Doc/Release/Redirection.html#Redi...
cat “filename.txt”
Up | grep “thing”
Up | grep -v “not thing”
Up | grep -v “other thing”
Etc. it’s just easier to build this way even if the initial cat is unnecessary.
> grep “thing I want” filename.txt.
…every time
I have heard snarky Perl putdowns ad nauseam at work and on HN and may have regretted using it a handful of times but I can say worse or similar for other popular tools, languages...
For me, the most useful use of (gnu) cat, is cat -A weird.file
It saves my day or solves weird issues (X-files) with files generated by (not so) junior sysadmins, copy/pastes, end of lines, invisible diffs, etc... many times each year.
I do think he could have woven in the Parnas source a bit better. The first quote seems to rely a lot on the surrounding context of the essay and its never contextualized. This bit:
> The problem is that these subsets and extensions are not the programs that we would have designed if we had set out to design just that product.
Just feels a bit disjointed in the article. I get the high-level message but thought it could be woven into the blog authors narrative better. Anyways, it sounds like a good article though.
I actually noticed your reply because I came back to this comment to save your blog. Not just for the content but the page style. The eggshile white (or similar) color, font, foot(side)notes. It's nice.
Things like this:
head -n 500 access.log | grep ...
head -n 500 <access.log | grep ...
Feel like you start with the filename, then go leftwards to the first operation, then start reading rightward again through the pipe. At least in my brain, it feels slightly more awkward.
I'm realizing now another (and potentially stronger influence) is just years of muscle memory starting pipelines with cat.
It's funny, when learning programming, I think Haskell was the language that introduced me to the pattern of having a chain of operators processing a stream to build up a result (and I'd later cover it again in SICP), and I loved how clean it looked compared to imperative code. But I now find it one of the harder to read languages due to it all being prefix, whereas Java/Kotlin/C#/Javascript now all have stream constructs that use method calls, so read left-to-right, source-to-sink
And I'm reminded that I need to give Forth a proper go sometime
[1] https://en.wikipedia.org/wiki/Uniform_Function_Call_Syntax
But probably the former.
<x Y | Z w>
It's also possible to make a much faster cat (I have, considered naming it cheetah),10%+ faster.
What I do like is doing something like:
>cat access.log | head -n 500 | grep mail | perl -e …
>we find that cat performs two responsibilities:
>1. Printing an error to stderr if the file doesn't exist
>2. Copying a file to stdout
I use "cat ... | ... | ... " like in TFA and just like many in this thread because it simply makes sense. It's more intuitive. It's easier to read. It requires less braincycles to remember how this or that command wants its parameter passed, etc.
I think the "useless use of cat" movement made its time: it failed. Many of us are never going to give up our use(less|ful) of cat (you decide). So stop wasting your time complaining about it.
"useless use of cat" goes into the latter bucket. Complaining at me about it does not actually improve the code; it's just a nag about a bad habit that, arguably, isn't even a bad habit.
"Useless" uses of cat aren't bad habits during interactive usage, for all the reasons people mention here which I won't rehash.
For scripts, however, the story is different than for one-off commands. For one thing, it's slower due to the extra forks and copying of data across pipes, so there's at least that. For another, it prevents the command from inspecting the other end of its pipe, which can negatively impact usage in some case. (For example, if the program knows its input is from a terminal, it may flush its output on every newline it sees.) Moreover, a bunch of the arguments for the interaction case (like "it's fewer keystrokes" or whatever) don't even apply to the script case in the first place...
The end result here is that you definitely shouldn't assume some habit is just fine with scripting merely because it's fine when you're typing on the terminal, or vice-versa.
For shell scripts, I would argue quite vehemently that the most important goals should be correctness and readability, with performance being a very distant third concern. I'd even be tempted to argue that performance shouldn't be a consideration at all, except of course that argument would be misinterpreted to support some absurd edge case until I'd have to admit that of course performance is a little bit of a concern. But in any case, I can't recall a single example of a cat pipe being the root cause of an unacceptable performance problem in a shell script.
On the readability point, the example that probably irritates me most often is a cat pipe into some commands into a while loop. I much prefer this:
to this: or this: ...and that's a pretty common pattern where the edge case of reading input from a terminal doesn't apply.So this is the kind of thing that makes me go "shut up shellcheck" instead of "thanks shellcheck!"
But: performance was just one of the problems I cited. I gave you more than that -- one was a correctness reason (which you do care about) and had nothing to do with performance. And, again, incorrect buffering (which can make the script literally unusable in some cases) was just one example. I've seen needless redirection interfere with Ctrl+C handling too, though I don't recall the exact example. Oh, and there's terminal coloring and ANSI escape processing too, which programs detect differently. Point is, being unable to see the end of the pipe can definitely cause an unnecessary mess in some cases.
As for readability - honestly, part of the reason you find it less readable is that you're missing something else. Namely, this:
should really have been: which is in fact both more correct (at least when the file name isn't hard-coded, which is the common case in shell scripts) and more readable than your example; you can immediately spot where the file name is. The difference between that and cat "$blah" | sed ... is very minor at that point (and in fact you should be doing cat -- "$blah" as well...); anybody reading a command like sed without a pipe input knows to look for an input argument. The important point regarding readability here is, it's not like the code gets overly tricky if you write it one way vs. another way. It's just a matter of spending 1-2 extra seconds glancing over. So it's very much a minor thing to be prioritizing above everything else. (If the logic became harder to reason about, that'd be a different story, and it'd put more weight on the readability aspect.)Yes, I saw your other points, and I chose this example because it is an example drawn from real-world use where there is zero objective reason to wag a finger about "useless use of cat". Those other points are not relevant in this example, and piping a cat into some other commands into a while loop is pretty typical shellcode. Forcing me to move a filename argument into the middle of a long line for stylistic reasons should be obviously wrong. It is one case where shellcheck is over-reaching and being a nuisance rather than helping me catch errors.
This has been argued better and to death already: https://stackoverflow.com/a/16619430, http://oletange.blogspot.com/2013/10/useless-use-of-cat.html, https://news.ycombinator.com/item?id=23341711, https://news.ycombinator.com/item?id=36116208, https://news.ycombinator.com/item?id=6367319, https://news.ycombinator.com/item?id=1116085, etc.
Terminated: 15
Which cat died first?
Fascinated to know the actual answer though.