When you make a posting on HN or anywhere else and cite a URL, that URL could go down years later. Your historic posting then has a dangling URL. You can't go back and edit old postings to fix it.
Also, some content changes. The URL is valid, but the cited material is not there. With the archive, you can point to a specific date when that was retrieved.
I get a 403 as well, no point in linking to something unaccessible just because reasons. Gdpr is hard but people in Europe should not be denied access just for the sake of immutable URIs
Just a basic example, I'd definitely use grep for something as simple as that. Only pointing out the pattern (you'll definitely have some heftier stuff in {} if you're going to bother with awk in that case)
The typical awk case is to print selected fields, or operate on them (count, sum, classify, etc). So a '/<pattern>/ { print }' idiom is fairly common, generally on the way to something more refined.
For some time I've been using both Awk and Perl for text processing, based on which offered the most convenient command for a given task.
There are two (IMO) glaring omissions in Awk:
- using the captured group(s) of a regular expression
- slicing the arguments array
Mawk supports the first functionality, but it's inconvient.
Then at some point, I've found that virtually every task I do in my text processing via Awk can be done via Perl in a comparable way... and much more - including the two functionalities above - so I've ditched awk.
The only thing inconvenient is to type something like `perl -lane` as opposed to `awk`, but this can be aliased :-)
Yes, but as soon as your awk script becomes complex, you need to switch to a more complete language like perl (or python or ruby). Switch slightly earlier and avoid learning all the subtelities of awk (the same applies to sed). You need to learn well one scripting language and to know most common use cases of sed, awk.
No you don't. There is no reason to switch. I've written many full-blown programs in AWK and I see no reason to switch them to another language -- they're fast, portable and readable, with tiny memory requirements. It all comes down to mastery: someone who's mastered AWK will have no issues with a large AWK program and therefore will feel no urge to translate that program to another language. Large AWK programs are easy to maintain because the syntax is so small that the code is always intrinsically understandable.
Try mawk. It's an incredibly fast awk implementation. If you're using gawk, setting LANG=C will make large awk programs with lots of complex regexpes run significantly faster as well.
Thinking over this, it just (finally) struck me that awk (and sed, grep, etc.) are basically just libraries (more like DSLs), and the shell is just another scripting language with a REPL/notebook environment.
When that's the case, what makes them particularly useful compared to some other language with a uniform syntax and possibly better forms of "glue" -- say Python, or lisp, or whatever? Why not allow people to acquire proficiency in just one language, and use it consistently to communicate with their computer? I don't think ubiquity/portability is a compelling answer, since (for many/most people who care about this) it is relatively easy to install the environment of their choice on their personal computing device. BTW, we must also strive to maintain that flexibility in the next generation of computing devices.
As an aside, folks who do like the task-focused nature of shell DSLs might appreciate language-oriented programming with Racket.
Awk is part of BusyBox, thus it's present by default on a wide range of embedded systems/systems with limited resources.
Some people are comfortable working in sh/sed/awk. I personally don't care for it. I'm not familiar with it because I don't use it very much, and I dislike the lack of consistency and all the gotchas in shell scripting. (I'm talking about sh+sed+awk+other command line tools, not just awk.) But I've done it at times when it's all that's available.
There is no inconsistency if one programs in straight Bourne shell from 1978, not using any additional features, and frankly, I've been programming in it for 25+ years and never needed "bashisms". ksh is also extremely consistent and portable. One of the main hidden strengths of AWK is that there is one and only one way how to do something in AWK syntax, so from that point, as long as one doesn't use GNU AWK-isms, it's programming luxury and paradise. Programs written in non-GNU AWK run everywhere (one would have to go out of one's way to use Thompson specific features, for example). As far as I'm concerned, it's a matter of understanding UNIX, not GNU/Linux, but UNIX, so that it is clear what is what and what specifically to avoid (like any GNU features). People who grew up on GNU/Linux often struggle with understanding what's what because they have a huge knowledge gap. The fact that GNU/Linux was their first UNIXoid OS is actually highly detrimental now.
I suspect shell languages see widespread use because they are optimized for interactivity. The barrier between thinking of an action and making it run is low. At a certain point there is diminishing returns for particularly complex actions, e.g. you call out to awk instead of implementing awk in a shell language all else being equal.
Python and Lisp use way more parens (which many find unergonomic) and don't have pipes. Let me qualify that - parens are cumbersome with the usual tools and even if you add pipes to your Python/Lisp project the ecosystem doesn't follow the idiom. That being said, there is a Python shell called xonsh, but that uses kind of a Python - bash hybrid AFAIK.
> and even if you add pipes to your Python/Lisp project the ecosystem doesn't follow the idiom.
That isn't quite true. While, yes, a pipe operator would probably be something new that you add to the language, Scheme which Racket originally comes from, actually comes with first-class pipe support.
Scheme makes use of a concept called ports, quite extensively. You can turn files, sockets, string, etc. into ports. Ports are most definitely idiomatic Scheme.
Racket extends ports with this:
(make-pipe [limit input-name output-name])
Which works to transfer data from one port to another in basically the same way the pipe operator works at the shell level.
To make it work the way people are more used to, you might also combine it with the infix library.
So if you wanted to create a pipe operator for some reason, everything you need is already there, it'd just by syntax sugar.
> When that's the case, what makes them particularly useful compared to some other language with a uniform syntax and possibly better forms of "glue" -- say Python, or lisp, or whatever?
Well, first:
> possibly better forms of "glue" -- say Python
That's not a better form of glue. First, consider that executables of all languages are standarized to work with text. They get text arguments, text input through stdin, text output through stdout and stderr, text input and output through other files whose paths are given as arguments, etc.
Shell languages are good glue because they work only with text. With the exception of control structures like "if", and "while", all statements in a shell languages are based on text. The command name is just text, the arguments are text. Syntax facilitates the manipulation of their standard inputs and outputs. Command substitution allows one to glue the output of one command as arguments to another. That's what makes shell languages a good glue.
Python, on the other hand, is not as good at gluing these executables written in a myriad of languages because it doesn't put enough importance on this universal interface between languages that is text, nor does it sufficiently facilitate the execution of other executables.
Another thing to consider is that shell languages are meant primarily to be used interactively. Things like putting commas between arguments and parenthesis around them would be a nuisance in interactive use. Having to quote every argument would also be a nuisance. In a not-primarily-interactive language like python, one uses variables more than literals, so it's easier to write variables than quoted strings. In a primarily-interactive language like bash, one typically uses literals more than variables, so it's the variables in bash that require the extra syntax (the sigil, '$'), while literal strings do not (most of the time).
> with a uniform syntax
I'm not sure what a more "uniform" syntax is, but I get your drift that there are warts in typical shell languages. I would also prefer that `if` statements didn't end in `fi` or `case` statements end in `esac`, but I think languages like bash and zsh offer to much to forgo them for such little details. There are more to these languages than these warts that makes them worth it.
You might be interested in the fish shell project. It's a newish shell language with a simplified ("uniform"?) syntax. Personally, I don't use it because it lacks too many features, but it's there if you like it.
> Why not allow people to acquire proficiency in just one language, and use it consistently to communicate with their computer?
That's shell languages. If you meant something like python, you can. Just do `usermod -s $(which python) $USER` to set python to be your shell. I wouldn't last long on it as my shell for the reasons I mentioned, but there's no-one not allowing us to if we wanted.
I guess shell scripting and the standard commands are convenient interfaces to a lot more functionality. The interactive use (and pipes) tend to make the linear processing super terse and easy (but lots of other things become annoying).
In a type-safe language, it’s typically impossible to specify a function which can do as many things as (eg) sort can, and which doesn’t require specifying a huge amount of default arguments on every invocation. Therefore you end up with a limited function, a huge annoying function, or many small functions. In any case, changing from one kind of sort to another (eg how to sort things, what to sort, how to order them, whether to keep unique elements, whether to be stable, whether to take multiple inputs, whether to assume the inputs are sorted (so just merge), and so on) becomes difficult whereas with a tweak-in-many-ways command like sort it is easy. In a language like CL with optional arguments and no types, this problem can be less hard but whereas sort will likely work fone for a large dataset, your favourite library function may not. (In particular it almost definitely won’t use on-disk storage when needed or support parallelism. Partly note that storing objects from your favourite language to disk is probably hard but if you only work on lines of text it’s easy).
The other big difference is the data types. The data type of shell programs is basically a sequence of lines, each of which is sometimes broken up into one or more records by some other separator. In many languages the sequence operations are all about getting the nth element, or extending them, or changing elements, or deleting them, or splitting the sequence up into other sequences. And these operations typically identify the elements by the position. In shell scripts, many of these operations are unavailable or not used. One basically only iterates forwards, processing each element, and one almost never thinks about the position of the element in the sequence. (c)split exists but isn’t commonly used.
Lots of work in a typical language is converting one data type to another, or extracting bits of it, or doing data type-specific things. In shell these don’t really exist (eg you don’t sort as dates, you write your dates like 2019-09-28 and sort them like strings), and so data is simple, and because extraction is flexible (typically a field number or byte positions) lots of the bureaucracy of changing data types is omitted.
These things all often lead to shell scripts being unreliable. But also resilient, flexible and “sufficient”. By “sufficient” I mean that they can do things well enough that the cost for a long term, thorough, or “proper” solution isn’t justified.
> (In particular it almost definitely won’t use on-disk storage when needed or support parallelism...)
The parallelism possibilities of shell pipelines was always obvious to me.
But I hadn't appreciated explicitly the trivial disk storage ability - that is important too. Nowadays, the other way may be important too - that you can create an in-memory filesystem and run scripts that work off files way faster.
Another tool I feel we don't use enough, is 'make'.
Command line tools are very terse, heavily optimized (ie fast), interactive, and easily chainable. I would never want to write a python program for quick analysis of a log file when I can grep/sed/awk/sort/uniq my way to a solution.
The other day my boss asked me to find out if there was any second where we processed more than 100 transactions a second on our server in the last 90 days. I needed to go through 100s of GB of data which mawk managed in no time and in a relatively simple one liner. I imagine my idiomatic python script would still be running
Often times, analysis is an evolving thing. As you uncover one answer, new questions arise. It’s far easier imo to tweak my command line pipelines than to modify program flow.
At the end of the day, general purpose programming languages are jacks of all trades and masters of none. They’re inherently held back by their flexibility when compared to DSLs. Text processing is so easy in awk b/c it’s a text processing domain specific language. If you really think that we should use on language for everything, you’re ignoring the value that specialization can provide you, whether in speed, power, or ease of use
> It’s far easier imo to tweak my command line pipelines than to modify program flow.
Why? Creating a throwaway file in the IDE and attaching the debugger to it is just two keyboard shortcuts away. Then i can just change one line, hit F5 and get the output instantly. Contrast with the command line where i have to be careful not to hit the up-button and get all my modifications reverted, and can't use fundamental features like selecting text using shift+left/right. When writing slightly longer shell-pipelines i even tend to first write them in the IDE and then paste them into the shell to avoid this! With a program i can also set breakpoints halfway through the script to see what has happened halfway through a pipeline.
A big factor is the speed with which I’m getting results (I’m generally processing 10gb compressed log files). I don’t want to wait around for results or spend time finding the bottleneck is that my string split is creating new strings.
Secondly, these tools are much more powerful than programming languages in their domain. Through such a terse api, they can accomplish so much because the command line interface is encapsulating a well written, thoroughly tested complex program. There is no boilerplate in awk and all of the assumptions make sense for the line based text processing it’s built for. A grep|awk|sort|uniq -c pipeline is the workhorse of my production support via structured log files.
The other day I needed to read images out of a database, resize them, and insert them back into the dB. It was trivially easy to ‘psql | xxd | convert | xxd | psql’ to extract, convert to hex, use image magick, convert back to hex, and store the new image. The solution flowed out of me within 1 minute. I guarantee you that doing that in a higher level language would require looking up the syntax for how to connect to the dB, how to convert a byte stream to hex, how to manage sub processes when calling to convert, or which library I should use for resizing images to avoid calling convert in the first place. On top of that, we’d have to perform the various necessary incantations, songs and dances, and boilerplate to like creating a dB connection, handling subprocess timeouts, etc.
On top of all of this, there is the parallelism I get for free. Then there’s the intelligent use of writing to disk as opposed to running out of RAM when dealing with giant files (sort for example).
You can of course use programming languages to do everything I’ve described. IMO the conceptual overhead is inherently higher b/c I’m left to think about many more factors.
This is true. Some reasons why sh hasn't been replaced by LISP (for example) as the default Unix shell is because:
- it was there first
- grep awk etc are heavily optimized and faster than their equivalents in most languages
- the primary job of a shell is to run other programs, not to be the program. Scripting languages don't generally let you run a program just by typing the name and hitting enter.
All that being said, you can set your login shell to be whatever you want in /etc/passwd, and you're free to login to your REPL of choice instead of sh if that's what you really want.
I have, and read, a book on awk (the AWK programming language, by guess-the-initials) and didn't see the point. If I need regexes I have emacs. For CSVs I use excel or LibreOffice.
It has been a while so could someone remind me why awk is so fab? Perhaps it's just I've never met the right use case to make me fall in love.
AWK's primary use case is processing a file line by line. It offers a few well thought out mechanisms to do so, allowing you to script relatively complex operations in just a few lines. At the same time, and acknowledging it's not pretty, it's powerful enough to run an http server (not kidding: https://github.com/kevin-albert/awkserver).
So if you want/need to write a script that requires more sophistication than cat, join and sort offer, but you don't want the hassle of setting up IO, field splitting, pattern matching in a more complex programming language, etc., AWK is your friend.
Some of examples of my own awk scripts: converting csv to json, counting trigrams, splitting text by words or sentences, getting the one line from a complex and long log file that I needed, pre-processing files before compilation, cross-tabulation, rewriting file paths, converting weird file formats and ad-hoc templating.
excel can't handle millions of lines. Every job (only 2, but still) I've had, I've had a problem at least once where I'm given millions of lines and needed to parse them for something.
That being said, I used python in those instances, but I'm willing to believe I could've done it faster and more easily with awk.
Yes, this was my feeling too, why not just use python and save the effort of learning another language? Perhaps because you can't assume python will be present in a basic *nix install.
Awk predates Excel (and therefore LibreOffice), so the original point was that...there was no GUI spreadsheet application to use instead.
32-bit Excel used to have a limit of around 65 000 rows. 64 bit Excel is just over 1 000 000 rows. Both of these limits are commonly exceeded.
Excel requires all the data to be in memory. Awk processes line by line so requires less memory.
Awk scripts can easily be incorporated in a chain of other commands.
Version comparison is easier with Awk scripts as they are text based not binary (xls) or compressed XML (xlsx) files.
Awk can be combined with sed, uniq, sort and other tools as one-liners or in shell scripts for fast and powerful data analysis or transformation. These tools are available by default on my Unix-like operating systems.
Nowadays though, scripting languages like Perl (inspired by Awk), Python or Ruby might be used instead of Awk.
Awk has been a handy tool for me professionally a number of times when I had large amounts of textual data to analyse. I have used it many times to transform data to pass to Oil and Gas engineers who were unable to handle the raw data in Excel.
For very large CSV files you cant use excel or libre office because they cannot even open the files. Instead you just use bash tools like awk, sed, grep to get what you need. Even for moderately large CSV files these tools are miles faster than GUI applications.
I don't write much Awk code any more, but at one time it was my favorite programming language, and not just for little one-liners.
A while ago I ran across a couple of my old Awk programs and uploaded them to GitHub. One is an implementation of Mark V. Shaney (the infamous Markov chain program), the other takes source code and prints it out in a nice "two up" landscape format on a LaserJet II, including graphical boxes to replace the ASCII dividers in comments like this:
//------------------
// Some comment here
//------------------
I wish I had a sample printout, but they are long gone.
Awk was the first language I used that had associative arrays built into the language, which made SHANEY.AWK almost trivial to write. Especially compared with implementations I saw in C and other languages of the day.
I think LJPII.AWK is a good example of how you can write a fairly substantial program in Awk and have it be (mostly) readable. (It was written for either MKS Awk or Thompson Awk, I forget which - so there are a few things in it that may not be available in a typical Awk today.)
I use a lot of Perl and occasionally some awk. My experience is that Perl is a lot faster (it really shows when you have millions of lines of text to chew).
For example, I compared two scripts that find the longest line in a file: Perl took about a second when awk took 21s.
Weird. The site is served by cloudflare and backed by an s3 bucket. Totally static. You shouldn't get forbidden unless cloudflare or your isp are doing something odd.
I've found the combination of sed & awk to be a really powerful duo.
One of my earliest development projects circa 2002 was to create a complex awk & sed program that took tons of database blobs with inconsistent semi-structured test from a publishing company and converted it to XML, automatically prompting the user for input (using the Pico text editor) whenever there was something that couldn't be resolved. It saved thousands of hours as part of a metadata digitization project that got their journals into ProQuest.
As only a semi-literate programmer at the time, it was significantly easier than trying the same thing with a "traditional" programming language. On occasion I still call to awk withing python code when I need to do a one-off quick & dirty <something>
For those interested, if you can find a copy, there'a truly excellent book on awk:
"The Awk Programming Language", by Alfred Aho, Brian Kernighan, and Peter J. Weinberger
Also Arnold Robbins' books Sed and Awk and Effective Awk Programming are excellent, and also detail the many features added since the latest edition of The AWK Programming Language (1988).
Awk and sed can provide a better integration with "modern" formats like JSON/YAML/Protobuf/etc. And with corresponding tools like jq/pq/rq. It would help to keep them popular still.
Once you get a hang of how shell constructs like pipes, redirections, globs, variable/command substitutions, etc, the various cli text processing tools like grep, sed, awk, perl, sort, tr, pr, paste, etc are worth to learn at least the basics. They have been through years of use and heavily optimized for performance. Just today, a friend of mine called to ask how to improve a Python script's performance for 5-10 MB text file. After confirming he isn't using any other special modules, I advised to check if the performance improves by implementing it using awk.
If anyone's interested in examples based tutorials on cli text processing tools, check out my github repo [2]
The concept of language "generations" is mostly lost now, much like how we stopped inventing names for how densely we pack transistors onto silicon once we'd gotten to VLSI (Very Large Scale Integration); the point of the essay is that the shell complete with a userspace is a programming language in itself, and one which is much more programmable and extensible than, say, getting all of your data into a database and using SQL to manipulate it. It's a programmable environment, as opposed to a data silo, where the only tools you have are the ones the silo's implementer deigned to provide.
There is a short talk by Eben Moglen [0] which is only 10 minutes long but jam packed with wisdom. One part which changed the way I think about using computers is how he describes GUI interfaces: as a "point and grunt" or "caveman" interface. Using UNIX, on the other hand, is to use language to communicate with the computer and comes with all the benefits that we have when using it to interact with people.
I wonder. Unless you have this syntax (and their semantics) memorized so that it just flies from your fingers, it does seem much easier to me to use whatever your language of choice is (Python, Java, ...) to whip up a script. Sure, awk will be terser, but in my case at least, the bottleneck will not be typing, it will be looking up awk features and syntax.
Maybe if you do this every day though. I only have need for things like this once every few months.
73 comments
[ 2.0 ms ] story [ 310 ms ] threadhttps://gregable.com/2010/09/why-you-should-know-just-little...
In this case the archive.org link is a 403 for me while the original works.
When you make a posting on HN or anywhere else and cite a URL, that URL could go down years later. Your historic posting then has a dangling URL. You can't go back and edit old postings to fix it.
Also, some content changes. The URL is valid, but the cited material is not there. With the archive, you can point to a specific date when that was retrieved.
ingve is an excellent submitter who doesn't normally do that, so I'm sure there was some reason.
https://news.ycombinator.com/item?id=1738688
https://news.ycombinator.com/item?id=2932450
The article mentions FS but this is easier to access with -F. e.g. for some quick and dirty CSV munging-
awk -F, '{print $2}'
Something else I find super useful is the built-in regex matching, e.g.-
awk '/ERROR/{print $(NF-1)}'
or maybe picking out all 503's from an web server access log-
awk '/503/{print}'
edit: formatting....wow hn is bad for this
{print} is the default action, I believe. So it would suffice to do:
Though, then you may as well just use grep for something like this.There are two (IMO) glaring omissions in Awk:
- using the captured group(s) of a regular expression
- slicing the arguments array
Mawk supports the first functionality, but it's inconvient.
Then at some point, I've found that virtually every task I do in my text processing via Awk can be done via Perl in a comparable way... and much more - including the two functionalities above - so I've ditched awk.
The only thing inconvenient is to type something like `perl -lane` as opposed to `awk`, but this can be aliased :-)
Tip: if you have quoted fields, use gawk, and
Though if you're doing a lot of csv munging, xsv would probably be a good investment.
When that's the case, what makes them particularly useful compared to some other language with a uniform syntax and possibly better forms of "glue" -- say Python, or lisp, or whatever? Why not allow people to acquire proficiency in just one language, and use it consistently to communicate with their computer? I don't think ubiquity/portability is a compelling answer, since (for many/most people who care about this) it is relatively easy to install the environment of their choice on their personal computing device. BTW, we must also strive to maintain that flexibility in the next generation of computing devices.
As an aside, folks who do like the task-focused nature of shell DSLs might appreciate language-oriented programming with Racket.
Some people are comfortable working in sh/sed/awk. I personally don't care for it. I'm not familiar with it because I don't use it very much, and I dislike the lack of consistency and all the gotchas in shell scripting. (I'm talking about sh+sed+awk+other command line tools, not just awk.) But I've done it at times when it's all that's available.
That isn't quite true. While, yes, a pipe operator would probably be something new that you add to the language, Scheme which Racket originally comes from, actually comes with first-class pipe support.
Scheme makes use of a concept called ports, quite extensively. You can turn files, sockets, string, etc. into ports. Ports are most definitely idiomatic Scheme.
Racket extends ports with this:
Which works to transfer data from one port to another in basically the same way the pipe operator works at the shell level.To make it work the way people are more used to, you might also combine it with the infix library.
So if you wanted to create a pipe operator for some reason, everything you need is already there, it'd just by syntax sugar.
Well, first:
> possibly better forms of "glue" -- say Python
That's not a better form of glue. First, consider that executables of all languages are standarized to work with text. They get text arguments, text input through stdin, text output through stdout and stderr, text input and output through other files whose paths are given as arguments, etc.
Shell languages are good glue because they work only with text. With the exception of control structures like "if", and "while", all statements in a shell languages are based on text. The command name is just text, the arguments are text. Syntax facilitates the manipulation of their standard inputs and outputs. Command substitution allows one to glue the output of one command as arguments to another. That's what makes shell languages a good glue.
Python, on the other hand, is not as good at gluing these executables written in a myriad of languages because it doesn't put enough importance on this universal interface between languages that is text, nor does it sufficiently facilitate the execution of other executables.
Another thing to consider is that shell languages are meant primarily to be used interactively. Things like putting commas between arguments and parenthesis around them would be a nuisance in interactive use. Having to quote every argument would also be a nuisance. In a not-primarily-interactive language like python, one uses variables more than literals, so it's easier to write variables than quoted strings. In a primarily-interactive language like bash, one typically uses literals more than variables, so it's the variables in bash that require the extra syntax (the sigil, '$'), while literal strings do not (most of the time).
> with a uniform syntax
I'm not sure what a more "uniform" syntax is, but I get your drift that there are warts in typical shell languages. I would also prefer that `if` statements didn't end in `fi` or `case` statements end in `esac`, but I think languages like bash and zsh offer to much to forgo them for such little details. There are more to these languages than these warts that makes them worth it.
You might be interested in the fish shell project. It's a newish shell language with a simplified ("uniform"?) syntax. Personally, I don't use it because it lacks too many features, but it's there if you like it.
> Why not allow people to acquire proficiency in just one language, and use it consistently to communicate with their computer?
That's shell languages. If you meant something like python, you can. Just do `usermod -s $(which python) $USER` to set python to be your shell. I wouldn't last long on it as my shell for the reasons I mentioned, but there's no-one not allowing us to if we wanted.
In a type-safe language, it’s typically impossible to specify a function which can do as many things as (eg) sort can, and which doesn’t require specifying a huge amount of default arguments on every invocation. Therefore you end up with a limited function, a huge annoying function, or many small functions. In any case, changing from one kind of sort to another (eg how to sort things, what to sort, how to order them, whether to keep unique elements, whether to be stable, whether to take multiple inputs, whether to assume the inputs are sorted (so just merge), and so on) becomes difficult whereas with a tweak-in-many-ways command like sort it is easy. In a language like CL with optional arguments and no types, this problem can be less hard but whereas sort will likely work fone for a large dataset, your favourite library function may not. (In particular it almost definitely won’t use on-disk storage when needed or support parallelism. Partly note that storing objects from your favourite language to disk is probably hard but if you only work on lines of text it’s easy).
The other big difference is the data types. The data type of shell programs is basically a sequence of lines, each of which is sometimes broken up into one or more records by some other separator. In many languages the sequence operations are all about getting the nth element, or extending them, or changing elements, or deleting them, or splitting the sequence up into other sequences. And these operations typically identify the elements by the position. In shell scripts, many of these operations are unavailable or not used. One basically only iterates forwards, processing each element, and one almost never thinks about the position of the element in the sequence. (c)split exists but isn’t commonly used.
Lots of work in a typical language is converting one data type to another, or extracting bits of it, or doing data type-specific things. In shell these don’t really exist (eg you don’t sort as dates, you write your dates like 2019-09-28 and sort them like strings), and so data is simple, and because extraction is flexible (typically a field number or byte positions) lots of the bureaucracy of changing data types is omitted.
These things all often lead to shell scripts being unreliable. But also resilient, flexible and “sufficient”. By “sufficient” I mean that they can do things well enough that the cost for a long term, thorough, or “proper” solution isn’t justified.
The parallelism possibilities of shell pipelines was always obvious to me.
But I hadn't appreciated explicitly the trivial disk storage ability - that is important too. Nowadays, the other way may be important too - that you can create an in-memory filesystem and run scripts that work off files way faster.
Another tool I feel we don't use enough, is 'make'.
The other day my boss asked me to find out if there was any second where we processed more than 100 transactions a second on our server in the last 90 days. I needed to go through 100s of GB of data which mawk managed in no time and in a relatively simple one liner. I imagine my idiomatic python script would still be running
Often times, analysis is an evolving thing. As you uncover one answer, new questions arise. It’s far easier imo to tweak my command line pipelines than to modify program flow.
At the end of the day, general purpose programming languages are jacks of all trades and masters of none. They’re inherently held back by their flexibility when compared to DSLs. Text processing is so easy in awk b/c it’s a text processing domain specific language. If you really think that we should use on language for everything, you’re ignoring the value that specialization can provide you, whether in speed, power, or ease of use
Yeah but I would wager very few people actually know awk. I would also wager 90+% of awk programs are pasted directly from StackOverflow.
That's why I like the idea of pawk[1] - it's like awk, except with python syntax instead of some esoteric syntax that nobody knows.
[1] https://github.com/alecthomas/pawk
Why? Creating a throwaway file in the IDE and attaching the debugger to it is just two keyboard shortcuts away. Then i can just change one line, hit F5 and get the output instantly. Contrast with the command line where i have to be careful not to hit the up-button and get all my modifications reverted, and can't use fundamental features like selecting text using shift+left/right. When writing slightly longer shell-pipelines i even tend to first write them in the IDE and then paste them into the shell to avoid this! With a program i can also set breakpoints halfway through the script to see what has happened halfway through a pipeline.
Secondly, these tools are much more powerful than programming languages in their domain. Through such a terse api, they can accomplish so much because the command line interface is encapsulating a well written, thoroughly tested complex program. There is no boilerplate in awk and all of the assumptions make sense for the line based text processing it’s built for. A grep|awk|sort|uniq -c pipeline is the workhorse of my production support via structured log files.
The other day I needed to read images out of a database, resize them, and insert them back into the dB. It was trivially easy to ‘psql | xxd | convert | xxd | psql’ to extract, convert to hex, use image magick, convert back to hex, and store the new image. The solution flowed out of me within 1 minute. I guarantee you that doing that in a higher level language would require looking up the syntax for how to connect to the dB, how to convert a byte stream to hex, how to manage sub processes when calling to convert, or which library I should use for resizing images to avoid calling convert in the first place. On top of that, we’d have to perform the various necessary incantations, songs and dances, and boilerplate to like creating a dB connection, handling subprocess timeouts, etc.
On top of all of this, there is the parallelism I get for free. Then there’s the intelligent use of writing to disk as opposed to running out of RAM when dealing with giant files (sort for example).
You can of course use programming languages to do everything I’ve described. IMO the conceptual overhead is inherently higher b/c I’m left to think about many more factors.
- it was there first
- grep awk etc are heavily optimized and faster than their equivalents in most languages
- the primary job of a shell is to run other programs, not to be the program. Scripting languages don't generally let you run a program just by typing the name and hitting enter.
All that being said, you can set your login shell to be whatever you want in /etc/passwd, and you're free to login to your REPL of choice instead of sh if that's what you really want.
It has been a while so could someone remind me why awk is so fab? Perhaps it's just I've never met the right use case to make me fall in love.
Sorry if it's a dumb question. TIA
So if you want/need to write a script that requires more sophistication than cat, join and sort offer, but you don't want the hassle of setting up IO, field splitting, pattern matching in a more complex programming language, etc., AWK is your friend.
Some of examples of my own awk scripts: converting csv to json, counting trigrams, splitting text by words or sentences, getting the one line from a complex and long log file that I needed, pre-processing files before compilation, cross-tabulation, rewriting file paths, converting weird file formats and ad-hoc templating.
That being said, I used python in those instances, but I'm willing to believe I could've done it faster and more easily with awk.
32-bit Excel used to have a limit of around 65 000 rows. 64 bit Excel is just over 1 000 000 rows. Both of these limits are commonly exceeded.
Excel requires all the data to be in memory. Awk processes line by line so requires less memory.
Awk scripts can easily be incorporated in a chain of other commands.
Version comparison is easier with Awk scripts as they are text based not binary (xls) or compressed XML (xlsx) files.
Awk can be combined with sed, uniq, sort and other tools as one-liners or in shell scripts for fast and powerful data analysis or transformation. These tools are available by default on my Unix-like operating systems.
Nowadays though, scripting languages like Perl (inspired by Awk), Python or Ruby might be used instead of Awk.
Awk has been a handy tool for me professionally a number of times when I had large amounts of textual data to analyse. I have used it many times to transform data to pass to Oil and Gas engineers who were unable to handle the raw data in Excel.
A while ago I ran across a couple of my old Awk programs and uploaded them to GitHub. One is an implementation of Mark V. Shaney (the infamous Markov chain program), the other takes source code and prints it out in a nice "two up" landscape format on a LaserJet II, including graphical boxes to replace the ASCII dividers in comments like this:
I wish I had a sample printout, but they are long gone.Awk was the first language I used that had associative arrays built into the language, which made SHANEY.AWK almost trivial to write. Especially compared with implementations I saw in C and other languages of the day.
I think LJPII.AWK is a good example of how you can write a fairly substantial program in Awk and have it be (mostly) readable. (It was written for either MKS Awk or Thompson Awk, I forget which - so there are a few things in it that may not be available in a typical Awk today.)
https://github.com/geary/awk
https://en.wikipedia.org/wiki/Mark_V._Shaney
For example, I compared two scripts that find the longest line in a file: Perl took about a second when awk took 21s.
One of my earliest development projects circa 2002 was to create a complex awk & sed program that took tons of database blobs with inconsistent semi-structured test from a publishing company and converted it to XML, automatically prompting the user for input (using the Pico text editor) whenever there was something that couldn't be resolved. It saved thousands of hours as part of a metadata digitization project that got their journals into ProQuest.
As only a semi-literate programmer at the time, it was significantly easier than trying the same thing with a "traditional" programming language. On occasion I still call to awk withing python code when I need to do a one-off quick & dirty <something>
For those interested, if you can find a copy, there'a truly excellent book on awk:
"The Awk Programming Language", by Alfred Aho, Brian Kernighan, and Peter J. Weinberger
http://gen.lib.rus.ec/book/index.php?md5=0AD0779BF3602F1EA22...
Also Arnold Robbins' books Sed and Awk and Effective Awk Programming are excellent, and also detail the many features added since the latest edition of The AWK Programming Language (1988).
http://gen.lib.rus.ec/search.php?&req=awk+robbins&sort=year&...
So I wrote a small guide which is hands on for awk
https://github.com/thewhitetulip/awk-anti-textbook
Once you get a hang of how shell constructs like pipes, redirections, globs, variable/command substitutions, etc, the various cli text processing tools like grep, sed, awk, perl, sort, tr, pr, paste, etc are worth to learn at least the basics. They have been through years of use and heavily optimized for performance. Just today, a friend of mine called to ask how to improve a Python script's performance for 5-10 MB text file. After confirming he isn't using any other special modules, I advised to check if the performance improves by implementing it using awk.
If anyone's interested in examples based tutorials on cli text processing tools, check out my github repo [2]
[1] https://blog.jpalardy.com/posts/why-learn-awk/
[2] https://github.com/learnbyexample/Command-line-text-processi...
https://www.gnu.org/software/nosql/4gl.txt
The concept of language "generations" is mostly lost now, much like how we stopped inventing names for how densely we pack transistors onto silicon once we'd gotten to VLSI (Very Large Scale Integration); the point of the essay is that the shell complete with a userspace is a programming language in itself, and one which is much more programmable and extensible than, say, getting all of your data into a database and using SQL to manipulate it. It's a programmable environment, as opposed to a data silo, where the only tools you have are the ones the silo's implementer deigned to provide.
[0] https://youtu.be/uKxzK9xtSXM
But I am wondering: in a professional setup, are there no better tools to analyse and interpret structured log messages out there?
Maybe if you do this every day though. I only have need for things like this once every few months.