Ahh, the (apparently?) lost art of command line voodoo. This was type of "write once, read never again" culture. Perl was a similar guy in this family, and if you really got good at it, you could of course read again these gems and produce them, too.
This is quite a different culture then "young folks" [TM] who write a unit-tested nodejs library with tons of dependencies to do the same ;-)
Of course I'm exaggerating, but there's some wisedom in these onliners, or Perl5 with it's sigils.
I wouldn't say it's a lost art, but perhaps declining in ubiquity. I'm an SRE; CLI wizardry is part and parcel of the job. jq is especially helpful/powerful for various services that output JSON.
This has haunted me working in the infrastructure automation space. It is utterly dominated by wizards quickly solving problems with one-off scripts and then committing those to Kubernetes CRDs embedded as bash script heredocs in a yaml list that are completely incomprehensible to anyone else trying to read them to figure out how a system works. It means coming onto teams that are completely dominated by whoever happened to work there first because they wrote all the custom tooling and nobody else understands it, leaving us with fires all the time where everyone else can scramble for hours or days to figure out what the heck is happening, but he shows up and fixes it in ten seconds because he wrote the bad code and immediately knows what the problem is.
Interesting -- what part of this infrastructure would you like to change? The wizards would probably want to get rid of YAML, probably even of Kubernetes and just run their scripts on bare metal. The DevOps would probably want to get rid of the bash scripts in favour of some "sane" tool written in a language which allows for testing, extension, etc.
This is one of these "it works for me" problems, the main issue is probably actually the lack of documentation. I wonder why quickly rushing wizards don't adopt the dictation technique used by (alway rushing) medicals/physicians. They outsource the transcription to lower paid assistants.
There is a "!" missing from the title[0], should be
awk '!a[$0]++'
"removes duplicates while maintaining order. a feat unachieved by uniq or sort -u which either removes only adjacent duplicates or has to break order to remove duplicates."
Also, this version[1], which works with gawks from 2013 and after, changes the file in place as it goes, where the original version just makes the array which then needs to be printed/saved:
gawk -i inplace '!a[$0]++' file
(The original version seems to only work with files up to about 2GB.)
p.s. The current misspelled title using "Awk", raises another question–I was surprised that "Awk '!a[$0]++'" works too (on Mac at least). If I "ls Awk" in \usr\bin, it says Awk is there. If I "ls awk" it says awk is also there—but it seems they're the same file, and it's only pretending Awk is a file. AWK also... I never noticed that before!
[0] I sympathize—I once tried to submit a !!Con talk.
> I was surprised that "Awk '!a[$0]++'" works too (on Mac at least). If I "ls Awk" in \usr\bin, it says Awk is there. If I "ls awk" it says awk is also there—but it seems they're the same file, and it's only pretending Awk is a file. AWK also... I never noticed that before!
I presume this is a side-effect of the macOS default filesystem being case-insensitive. I'm running macOS with a case-sensitive fs and that does not work:
8 comments
[ 5.7 ms ] story [ 35.5 ms ] threadThis is quite a different culture then "young folks" [TM] who write a unit-tested nodejs library with tons of dependencies to do the same ;-)
Of course I'm exaggerating, but there's some wisedom in these onliners, or Perl5 with it's sigils.
This is one of these "it works for me" problems, the main issue is probably actually the lack of documentation. I wonder why quickly rushing wizards don't adopt the dictation technique used by (alway rushing) medicals/physicians. They outsource the transcription to lower paid assistants.
Also, this version[1], which works with gawks from 2013 and after, changes the file in place as it goes, where the original version just makes the array which then needs to be printed/saved:
(The original version seems to only work with files up to about 2GB.)p.s. The current misspelled title using "Awk", raises another question–I was surprised that "Awk '!a[$0]++'" works too (on Mac at least). If I "ls Awk" in \usr\bin, it says Awk is there. If I "ls awk" it says awk is also there—but it seems they're the same file, and it's only pretending Awk is a file. AWK also... I never noticed that before!
[0] I sympathize—I once tried to submit a !!Con talk.
[1] https://unix.stackexchange.com/a/320128
I presume this is a side-effect of the macOS default filesystem being case-insensitive. I'm running macOS with a case-sensitive fs and that does not work:
This default is a topic of great debate.