I was first exposed to Awk when I started work at Bell Labs in the late 80s. Until then, I'd been using either Lisp or C exclusively and was really blown away by how simple some things were in Awk. I used it with impunity to munge all sorts of data for input into fault prediction tools I was working on. Speed was never an issue for me, so I never explored the potential improvements offered by 'awkcc'. Although perl was becoming the new hotness at that time, Awk remained my goto tool for many years.
If you are interested in learning Awk, I highly recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger. It's about the same size as the original "The C Programming Language" and is equally well-written. Previously on HN: https://news.ycombinator.com/item?id=13451454
Aho taught the programming languages course at my Uni, and he loved to tell stories about Bell Labs, and this Brian guy in particular. ‘Let me tell you, I’d walk into Brian’s office and say, Brian, you really messed that language up...’ The entire semester I was like who the f is this Brian guy? Years later I was like oh shit, Brian is Brian Kernighan, and the language is C! And I realized I missed a freaking awesome opportunity to ask Al Aho about some serious heads in CS.
Perl was basically written because Larry Wall found Awk's syntax to be a little too cryptic. In the language design business this is what we refer to as baby steps.
Also, Awk isn't great for making reports, which is why Perl 5 to this day has an awkward report creation system[1] that looks like some COBOL refugee instead of idiomatic perl code.
> Perl was basically written because Larry Wall found Awk's syntax to be a little too cryptic.
That seems ridiculous; where is it substantiated?
When Wall posted Perl to comp.sources.unix for the first time, he wrote "If you have a problem that would ordinarily use sed or awk or sh, but it exceeds their capabilities or must run a little faster, and you don't want to write the silly thing in C, then perl may be for you."
Or rather, not Larry Wall, but the apparent newsgroup moderator added that text, lifting it from the Perl manual page.
Thus he was pitching it as something that performs faster than awk and sed, with a greater range of capabilities.
On pp. 381-382, my copy of Programming Perl (1992 printing of first edition) says he was trying to build a configuration management system for 6 Vax and 6 Sun machines, and he needed to solve some problems like file replication across a 1200 baud link and approvals. So he installed B-news, the Usenet news software at the time. Then he was asked to generate some reports and:
> News was maintained in separate files on a master machine, with lots of cross references between files. Larry's first thought was "Let's use awk." Unfortunately, awk couldn't handle opening and closing of multiple files based on information in the files. Larry didn't want to have to code a special-purpose tool. As a result, a new language was born.
So that's why it's the Practical Extraction and Reporting Language. He wanted to extract data from files and generate reports.
From the link: "The lone dot that ends a format can also prematurely end a mail message passing through a misconfigured Internet mailer (and based on experience, such misconfiguration is the rule, not the exception). So when sending format code through mail, you should indent it so that the format-ending dot is not on the left margin; this will prevent SMTP cutoff."
If you're of a certain age and read that and grimace as you immediately understand why this would happen, it does rather put into perspective the misery of dealing with, say, Webpack configuration.
> From the link: "The lone dot that ends a format can also prematurely end a mail message passing through a misconfigured Internet mailer (and based on experience, such misconfiguration is the rule, not the exception). So when sending format code through mail, you should indent it so that the format-ending dot is not on the left margin; this will prevent SMTP cutoff."
Did email clients not handle "dot stuffing" back then? That is, if a line begins with a single dot, the client would automatically insert another dot right before it. Then, at the receiving end, the client would remove the extra dot at the beginning of the line.
>IIRC there's no method of putting headers or footers on pages, nor page number without manually counting the lines yourself.
Good point. The BEGIN and END patterns do work for global headers and footers, totals, etc., but not for per-page stuff. You can do it yourself with some extra awk code, but yes, you have to write it. Not difficult, though. I guess it was not designed to be a Crystal Reports-like reporting tool, with report bands and what not.
>I could be wrong though. Awk is one of those tools like vi where you can use it for years and still be discovering new features.
Agreed :) Not only new features, even new uses for existing features, because, although it is a sort of DSL or little language (but a programmable one), the area it is applicable to, pattern matching and data processing of many kinds, is vast.
As many others pointed out in this thread, the fact that the reading of input is built-in to it (whether from standard input or files given as command-line arguments), saves you a bit of boiler-plate code each time (cumulatively) you write an awk program using that feature. So does the pattern-action model, with those two defaults for missing pattern or action (match all lines, or print). And again as others have said, Perl, Ruby, etc. have that feature too (the first one).
I found awk after using perl (four was the new hotness)
Before long I was trying to figure out just what in my daily work perl was suppose to be protecting me from
The awk book is a lot of fun. I just finished reading it after seeing it recommended here a few weeks ago. The highlights for me:
1. A simple interpreter for an awk-like language called qawk. qawk is like awk except that it allows for querying by field name rather than field number. For instance, it allows doing
{ print $country, $population, $capital }
instead of the more cryptic
{ print $1, $3, $5 }
2. An awk program that takes another awk program (in their example, a sorting algorithm) and outputs a version of that program modified to include profiling statements and an END section that outputs the results of those profiling statements to some file; then, another awk program that reads the data in that file and inserts that data back into the original awk program, thereby approximating where the hotspots are.
There's a lot more in the book besides these, but to me these are the coolest programs because they are the awk-iest, by which I mean that they loop over lines of input, split the fields of those lines, and then manipulate the fields. Some of the programs in the book don't do this; instead, they consist of a single large BEGIN block with typical for-loops, arrays, etc. Used in this way, awk is just yet another dynamic language.
Thank you, I remember wanting to follow up on these more abstract constructions in the book. They seemed to be leading me somewhere amazing and very computer science-y. Programs that take programs as input and generate new code to do that thing I wanted with some data files — I’m sure this will be useful if I put the time in.
Am I right that qawk was included as a program in the text? Did they ever follow up with further uses?
Yeah, the code is all in the book, and it works! Here's the main body of the qawk interpreter:
BEGIN { readrel("relfile") }
/./ { doquery($0) }
where
- relfile is a file containing the field attributes used in various database files,
- readrel is a function that parses the relfile and stores the fields in a dictionary, and
- doquery is a function that takes a qawk query, converts it to an awk query by replacing the field names with their corresponding field numbers, and then executes the awk command.
>If you are interested in learning Awk, I highly recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger. It's about the same size as the original "The C Programming Language" and is equally well-written.
I'm a big fan of small utilities :) - as I sometimes say in my email sig; but more importantly, I'm a big fan of Kernighan et al, where by "et al" I mean the others from the core early Unix days, such as Dennis Ritchie, Rob Pike, Ken Thompson and many unnamed others, from whom I (and tons of others) learned about the Unix command-line (tools), the shell (scripting), and the Unix philosophy [1].
Had written this just a few weeks ago on HN, in the thread titled "Technical Writing: Learning from Kernighan", but worth repeating here in the context of this thread:
"If someone has already solved a problem once, don't let pride or politics suck you into solving it a second time rather than re-using."
I'd strongly argue it's overzealous. As much as I agree "reinventing the wheel" is dangerous, tempting, and can quickly spiral to yak shaving, but Unix itself, and all the good it brought, is a prime example of "solving [a problem] a second time" after Multics.
In other words, I'd restate it in Sage Speak™: "Don't do this. Except when you need to." ;P Or, just want to have fun :P
I sort of agree. I didn't quote those sections in the sense of recommending that the advice in them be followed strictly, and to the letter. Also, ESR is known to talk a bit that way - sort of overzealous, as you put it. But that is part of the fun of reading his stuff. As long as one takes it with a pinch of salt, and common sense, it is okay, and one usually gets to learn something from his writings.
>But Unix itself, and all the good it brought, is a prime example of "solving [a problem] a second time" after Multics.
Not sure about that. I mean, I know it came after Multics and was inspired by it (due to some of the early Unix people having worked on Multics), including that the name was originally Unics (I've heard, as a word play on Multics, because it was originally written by one person or was originally a single-user OS, maybe), but I am not so sure that all the good it brought was from Multics. Likely Unix brought some new stuff too. Others who know better may be able to say more.
>In other words, I'd restate it in Sage Speak™: "Don't do this. Except when you need to." ;P Or, just want to have fun :P
Good one. A bit Zennish :) Check out one of ESR's other compilations, the Unix Koans of Master Foo, if not seen already ...
Wasn't meaning that the good was from Multics. Just that Unix was after Multics, "solving [a similar problem] a second time". In fact, that Unix brought extra good doing this, actually strenghtens the thesis that reinventing the wheel may bear good fruit :)
I've heard about _The Awk Programming Language_ but could get my hands only on _The GNU Awk User's Guide_ by Arnold Robbins, since it's available as Info pages on my Linux box. It's also free online
I'm not a coder and awk was my first more serious or more complete effort to learn a programming language. What I did notice afterwards was that the syntax and semantics of C also became a lot more clearer to me.
I think Kernighan also stated somewhere that awk was (also) designed as a helper-tool to learn C.
All in all, I think it's a great first language, even if you're initially more compelled by Lisp family languages. If you have time to learn only one language, then awk is not a bad choice; It'll open many doors in the Unix/KISS-world.
I can also state that "The Awk programming language" is, among other things, an excellent introduction to computer science or "the way programmers think" in general. A remarkably well written book for general audience.
This, a hundred times over. If you actually sit down to write an AWK script, instead if trying to jam it into a one liner, you'll find that it's actually a fun, pleasant language for simple data processing tasks.
I don't want to speak for everyone, but that's a big part of it's value proposition: its brevity and ubiquity are very enabling. I don't think we'd be talking about awk right now if it wasn't useful inline in the shell.
Plenty of people that would rather drop to Perl or Python once awk gets too cumbersome for the command line.
And that's also much of what the article is about, though it's arguable if it's trying to get people more deeply interested in the language in general.
Still, I'm not certain if the original comment is complaining about the tendency toward one-liners by actually writing less code (something the article alludes to, "only the inner part of the for loop") or merely cramming what would otherwise be readable as a multi-line, well-indented program into a single line.
Some terse Awk one-liners don't do anything clever; they just use common Awk idioms such as as { print } being the implied action if one is omitted and whatnot.
I disagree. To me that is the value. If I wanted to write a script I'd use Python. If I want to do something as a oneliner I use Awk and don't worry about anything as it isn't saved.
I think you misunderstand me as we might agree. Python is great for small to medium scripts, but can be verbose for the really small stuff that works better as a combination of Linux and Awk commands. Python has a lot of stuff in the standard library that I'm going to guess is missing from Awk (Sqlite, Sets, Classes...etc).
I guess the example I gave you was a one-liner but I didn't intend to constrain it to such. Just a week or two ago I wrote a 33-line script (that seems to fall squarely in your "small-to-medium" category) to calculate the minimal set of packages I need to install on Ubuntu in order to obtain the same set of packages I have right now. 28 of those 33 lines were an awk script consisting of three /PATTERN/{} blocks and one END{} block. It seemed about as simple as it could possibly be, and doing it in Python would've probably been twice as verbose.
Isn't saved? I log my bash history forever and have tools to help with grepping. Also for any serious data processing not saving the process is unacceptable. But I guess you're talking more sysadmin type tasks and yeah, I write throwaway awks all the time.
I have a variety of dev machines at work and the history can't be counted on. Yea, reproducible data analysis is good, but some tasks are just really simple and aren't worth it. Most of my terminal stuff is just running processes and grep/awk. Any major data analysis is probably in a SQL query which I do save on our wiki page.
I really like the concept of Wolfram Mathematica Notebooks as you can essentially embed everything into a reproducible analysis (what they call a computational essay). The free and open source equivalent is Jupyter notebooks.
You could do it with a little extra effort. I used to work on a cluster and ran bash on any of hundreds of machines. Having my entire history available at all times saved me countless times.
I felt precisely the same for a long time. Then I started following Unix SO and realized that people are being taught the bad habits.
I tried answering a few awk questions being verbose and explaining the steps. But I noticed that most are not asking to learn but to get a quick fix. Soon after that came the second realization: several of the answers did not understand what they where doing.
My conclusion: We are living in a cut'n'paste world were actual understanding is less valued.
Naturally code reuse is nice. But I doubt that many users are able to construct their magic awk snippets from the ground up. Hence the prolific terse magic snippets.
You are absolutely right about that. Not just awk but it happens in almost all programming languages when you're tempted to write a very long code in a super nested `array_map(array_filter(array_keys(.. .` type of expression.
Don't know why it gives me joy to write those things even when I know it would be far better to write the same in three nested for loops (which are less error prone, more maintainble and gives more flexibility like easy to "break").
Could you elaborate on that last part? I don't understand how for loops would be more maintainable and less error prone. They have more moving parts and therefore more room for error. Isn't the value proposition of the aforementioned functions exactly avoiding easy to mess up loops?
For starters when you're in a for loop it's easy to `break` out of it. Not sure if you can do it in array_map or array_filter.
Secondly and this is a personal preference but the code looks more readable when the for statements are on different lines telling you which array is being iterated compared to the nested functions where the output of one function is piped as input to another.
Some people might think "why learn awk today when there is Python, Go, etc". But a lot of tasks (even big ones) can be done easier with this tool.
The fact that most OSs have a POSIX compliant version of it in the base system also make it very valuable, not just for the people crunching data but also to sysadmin/devops people.
Yeah, you can never be sure what version of a scripting language is going to be installed on a system. Always a little risky depending on the system version. These tools have the advantage of having not changed too much in the last few decades.
Awk works great in pipes on the commandline, unlike Python where you would be writing a multiline script to accomplish the same task.
My 99% use case is extracting particular fields from some input stream.
awk '{print $1,$5}'
That's a tremendously common problem and awk is the best tool for the job.
I once started diving deeper into awk and there is an impressive amount you can do with it, but I hit a point pretty quickly where it would be cleaner to write the script in a more traditional language where my successor won't be cursing my name for writing something complex in an obscure language.
I'll tend to do this in scripts, both for clarity and precision (i.e. when I know the specific, single delimiter).
However, for ad-hoc command-line usage, I still prefer awk, because (1) the default delimiter is white space, which means I don't have to worry about if what I'm looking at might contain tabs intermingled with spaces (or how many spaces) and (2) it makes it much easier to incrementally increase the complexity of my ad-hoc operation, such as if I decide I want to take a sum or average of one of those columns, after all.
I first learned awk precisely to replace cut. Not being able to reorder fields make it an especially narrow tool. Is there anything from cut I'm missing out on?
Although I completely agree with this for any given "one-liner" or pipeline (even if it's multiple lines), I don't believe the mere use of awk in one place in the script means using cut elsewhere isn't simpler/clearer there.
Consider a script that originally had now awk but just a
field=$(somecmd | cut -d: -f3)
but now has been modified to have awk elsewhere. Did the above just become less simple or clear? If so, is it worth changing it to the below for clarity? Does it matter if the awk predates the cut, instead?
field=$(somecmd | awk -F: '{print $2}')
I say "no" to both, although I do recognize the argument for just using the same, consistent tool everywhere, instead.
I actually wrote a sed utility to collapse arbitrary whitespace before piping into cut before I learned about Awk. I never knew about -w before, but looks like neither of the systems I'm currently using for work support it.
>I actually wrote a sed utility to collapse arbitrary whitespace
Can be done with tr, without writing a utility (if by "collapse" you mean what I think you do):
This file t:
$ cat t
the quick brown fox
jumped
over
the lazy dog
containing many combinations of spaces, tabs and newlines (whitespace) can be changed to this file t2:
$ cat t2
the
quick
brown
fox
jumped
over
the
lazy
dog
by this tr command:
tr -cs "[a-zA-Z]" "\012" < t > t2
That also makes the output more amenable to further processing, including common tasks like finding the frequencies of the words in the input, as mentioned in
the "More shell, less egg" post mentioned in this post:
Don't know. It was a while back, and in Python 2 (if you mean the Python version I wrote). I didn't take any special steps to support Unicode, so likely not. Same for the shell version I wrote.
> For example, before the POSIX standard, you had to write /[A-Za-z0-9]/ to match alphanumeric characters. If your character set had other alphabetic characters in it, this would not match them. With the POSIX character classes, you can write /[[:alnum:]]/ to match the alphabetic and numeric characters in your character set.
> Currently tr fully supports only single-byte characters. Eventually it will support multibyte characters; when it does, the -C option will cause it to complement the set of characters, whereas -c will cause it to complement the set of values. This distinction will matter only when some values are not characters, and this is possible only in locales using multibyte encodings when the input contains encoding errors.
Yeah, I usually start with `cut` - it makes the easy cases very easy - then only reach for awk if it turns out I need more power. Thinking of recent cases from my own experience, that tends to be needs like:
- outputting my selected fields in an order other than their order in the input lines: you can't `cut -f2,1`, for example.
- use a multi-character or regex field separator: -F'<space><star><comma><space><star>' [edit: rewritten in words to dodge HN's formatting] is a real winner when dealing with hand-written CSV.
- use a record separator other than \n: thanks, Windows, for all those times I've written ORS="\r\n" :P - you can even use a regex record separator, which not even perl supports; I don't need that often, but it's a godsend when I do.
- select based on criteria other than "number of fields from the start": I recently needed the last field from lines of varying field-count, and awk '{print $NF}' was just the thing.
But if you don't need anything like that, cut is a lot more succinct - especially if you need to split on tabs only, which it does by default, while awk defaults to splitting on spaces too.
It's often easy to start with tools such as cut, sed, grep, tr, etc., but as you start building a one-liner (or short script) you realise that what you've cobbled together could all be expressed in awk.
Another common usage for awk in my case is to do floating operations (*, +, eq, gt, etc) in a shell script. It avoids installing bc, which is not always present by default.
I used it for other things, like for example comparing software versions in shell scripts.
The most advanced usage of awk in my case was to do some statistic analysis (average, confidence interval, standard deviation) of network simulations.
Just be aware there are some caveats when using awk to operate on large numbers. GNU awk has the -M option to handle precision arithmetic, but where you can't use `gawk`, `bc` may be safer afterall.
> The fact that most OSs have a POSIX compliant version of it
You need to be cautious however, gnu awk can be somewhat inconsistent at times:
try the following on various distros (and also an *BSD) for example:
awk 'BEGIN {print "9.1" >= "8"}';
awk 'BEGIN {print "9.1" > "8"}';
awk 'BEGIN {print "9.1" < "8"}';
awk 'BEGIN {print "9.1" <= "8"}';
on OpenBDS, you get no output or a syntax error, on Debian, you get the correct result for < and <= in all cases, never for > (it's interpreted as a redirection), and >= only works with newer Debian versions (sid and maybe stretch).
I know that the fix is simple: just put some parenthesis (awk 'BEGIN {print ("9.1" <= "8")}'), but I was quite amused to discover that a few days ago.
I agree! I absolutely love Javascript, but in the time it'll take me to setup a project, decide whether to use JS/Typescript/Flow and to actually write the code I would have done the goddamn thing in awk, gone home and watched Ronaldo score the hat-trick.
Most of my early career was Unix, with just a little DOS and Windows thrown in (enough to persuade me to keep trying to avoid it).
Unix was mature even then. There is NO WAY you would have convinced me that 30 years later I'd still be using most of that, still gluing things together with the same shell tools. Or that most of the books would still have a place on the shelf (albeit sometimes in 2nd or 3rd editions), and turn out to be classics.
Ignoring big iron, we'd gone from Apple IIs to Amigas and workstations in just a few years, and hardware was changing as rapidly as the current fashion for JS frameworks does today.
Yet here we are, awk, grep, sed and the other tools still feel like the right, concise answer. So why not learn them, they might greatly surprise you by their staying power.
Also, awk is way faster than Python. Its patterns aren't interpreted over and over like they would be in Python; they're compiled into a state machine.
Python regexes can be saved into variables, and even if you don't do it manually it automatically maintains a pattern-cache for the last expressions you have used.
Maybe I wasn't clear about this distinction. Python is an interpreted language. Even if you are matching a "compiled" regex, you are repeatedly interpreting the line that says to match it.
Awk is a compiled language. Your Awk script is compiled once and applied to every line of your file at C-like speeds. It is way faster than Python.
If you learn to use Awk well, you will start doing things with data that you wouldn't have had the patience to do in an interpreted language.
Technically, python is compiled to pyc and not reparsed again and again. What'd more, string operationd are all implemented in c. Unless you use pypy, which is faster.
I say this with no disrespect for Python, it's just you should know the tradeoffs of the tools you use. It does not matter if the individual operations in Python are implemented in C. It will still be much slower at looping over lines of a file than a compiled language designed decades ago for that exact purpose.
I'm really unsure about my benchmarking capabilities, yet I wanted to have a general idea of the order of magnitude we are talking about. Reading your comment, I was expecting at least a X5 speed up by using awk.
I have a lot of Python file in one dir:
$ find . -iname "*.py" | wc -l
10429
Finding them and cating them all takes about 0.3 secs:
$ time find . -iname "*.py" | xargs cat {} > /tmp/cat.out
...
real 0m0.344s
user 0m0.140s
sys 0m0.175s
So to have something simple that takes a bit of time, I tried to get all lines starting with "print", and output the first thing after that.
I'm really bad at awk, so I don't know if there is a better way. I went for the most obvious thing for me:
$ time find . -iname "*.py" | xargs cat {} | awk '/^print/{print($2)}' > /tmp/awk.out
...
real 0m1.111s
user 0m1.165s
sys 0m0.368s
Now, with Python, it's definitely not as easy to type. You have to get a script like. awk wins the expressivity metrics for this use case:
import sys
for x in sys.stdin:
if x.startswith('print'):
try:
print(x.split()[1])
except IndexError:
print('') # to match awk behavior
But as for performance, I don't get the huge boost in perfs you are talking about:
$ time find . -iname "*.py" | xargs cat {} | python /tmp/test.py > /tmp/python.out
...
real 0m0.762s
user 0m0.862s
sys 0m0.347s
Thanks for running these statistics. Your experience is certainly different from mine. Is your Python actually PyPy?
Here's what I observed: I had a simple text-processing tool I needed that I call "countmerge", which just merges adjacent lines with the same key and adds up their corresponding values. I needed to run it on a lot of large files.
I first wrote it in Python, where it was a significant bottleneck compared to the steps that came before it (split, sort, uniq -c). Eventually I rewrote it in Rust [1], and it was at least 5 times faster, at the expense of a fair amount more low-level code. But then rewriting it in awk [2] turned out to be as fast as what I wrote in Rust, possibly inconclusively faster.
>>> with open('data.txt', 'w') as f:
... for l in string.ascii_uppercase:
... for x in range(0, random.randint(1, 100000)):
... f.write('Key {}\t{}\n'.format(l, random.randint(0, 100)))
With this script:
import sys
old_key = total = 0
for line in sys.stdin:
key, value = line.split('\t')
if old_key != key:
old_key = key
total = 0
print(key, value, end="")
total += int(value)
I get:
$ <data.txt time python3 test.py
Key A 2
Key B 87
Key C 58
Key D 64
Key E 29
Key F 25
Key G 2
Key H 74
Key I 17
Key J 37
Key K 97
Key L 77
Key M 19
Key N 74
Key O 33
Key P 61
Key Q 67
Key R 23
Key S 4
Key T 70
Key U 25
Key V 15
Key W 35
Key X 17
Key Y 31
Key Z 18
1.03user 0.01system 0:01.05elapsed 99%CPU (0avgtext+0avgdata 9564maxresident)k
0inputs+0outputs (0major+1100minor)pagefaults 0swaps
But I can't manage to get the awk version working. It only prints one line on Ubuntu 16.04:
You wouldn't believe how many complicated python tasks I see that run into the many hundreds of lines of code that are an awk one or two liner and a line in crontab.
And if you really want that in a repo, you can stick that in ansible trivially.
Sometimes instead I just wrap a line of awk in %x() in Ruby to save a few chunks of code.
Most programmers don't understand why I do this though. :(
> You wouldn't believe how many complicated python tasks I see that run into the many hundreds of lines of code that are an awk one or two liner and a line in crontab.
Hundred of lines of python by two awk lines ? Must be very bad python programmers.
I tend to post this whenever an awk discussion pops up, but at least for me, the reason I didn't learn awk earlier is that the gnu man page for awk is quite offputting. I spent ten years thinking "awk is too complicated for my weak mind." Then I stumbled on the man page of the plan9 implementation of awk and I learned the language in fifteen minutes - if even that.
Sure, the gnu version is more powerful, but the plan9/original awk implementation handles most cases I run into, and I can now look up the gnu extras when I need them.
Yeah, in fact that's exactly the same reason for me. I remember when I started using GNU/Linux, trying to grok the most useful shell commands, and stumbling into awk... then quickly going somewhere else.
awk is also interesting because it's not just a language, it's a "model". It isn't something that you can't write with some lines of any other language but, because the loop is embedded, makes it easier to write utilities for the command line.
ruby's -n and -p options are venerable equivalents to perl's options which were present in version 1.0 waaay back in 1987 and they were there so it could be that drop in replacement for awk
ruby inherited a lot of structure and ideas from perl which it inherited from awk and that thought makes me irrationally happy
You can easily wrap python with a small script that makes it more useful as command line filtering tool. Most of the bloat actually just comes from importing packages.
#!/usr/bin/python3
import re, sys, json, other, useful, packages
def printf(s):
print(s.format(**globals()))
for NR, line in enumerate(sys.stdin):
match = eval(sys.argv[1])
if match:
eval(sys.argv[2])
I deliberately didn't pass pattern straight into re.search but rather provides the option to do other forms of parsing/matching in that argument, for example json parsing. I just spent 2 minutes writing this script so it's obviously not perfect and you probably won't be able to use it for json out of the box but give it a few more iterations and it will most likely work.
I saw a comment just yesterday(?) on HN that mentioned awk and it got me interested (first time I heard about it). This was a good little intro that was easy to follow along with and get a sense of what it's used for.
As a sysadmin, knowing awk, cut, and join is what lets you solve problems in seconds instead of minutes. I can't tell you how many times being able to quickly slice up a log file lead me straight to the problem I was trying to solve.
Looks interesting; can you explain how the hell it works?
I tried writing a script that would extract entries from logs (e.g. if every entry started "====CRITICAL ERROR LOG 2018/06/15====", I wanted it to start there and grab every line below it until it hits the next log entry starting with the "=====" header), but I gave up. Your script might do something similar, if I can figure out what it's looking for.
Another option might be to use multi-line records (https://www.gnu.org/software/gawk/manual/html_node/Multiple-...). First pre-process the file to add an empty line before each header, then use blank lines as the record separator and newlines for the field separator.
Sure. I'll explain the code and hopefully you'd find enough clues to adapt it for your use-case which, from what you've described, is a subset.
Intent:
Given a folder with multiple text files of academic papers from PubMed, remove the contents after the '====Refs' header. Then in each file find occurrences of any term in a keywords file and extract it along with 400 chars before and after it. Print this output as a tsv.
input:
$1) A text file containing keywords of interest that need to be searched in the files separated by `|`.
$2) File containing paragraphs of lots of text with headers starting with '===='. There isn't a formatting requirement really. Just that, in my case the files happened to be so.
# Note: Awk string commands have indexes that start with 1 (not 0).
# The first Awk command just removes the newlines in file $2 and replaces them
# with space. `pat` contains the contents of the file $1: 'keyword1|kewword2|...'.
# --re-interval allows use of, well, regex intervals.
awk 'BEGIN { ORS=" "} { print $0 }' $2 | awk --re-interval -v pat="$1" '
{
# both variables are initialized with full-content of text file
# with the part after '==== Refs' removed.
cut_content = substr($0, 1, match($0, "==== Refs"))
orig_content = substr($0, 1, match($0, "==== Refs"))
# index (could be better named as offset, but life is too short to refactor).
idx = 0
# match the keywords, even if it appears between non-alphabet chars.
# e.g -keyword1. is a match.
regex = "(^|[^a-zA-Z]{1})" pat "([^a-zA-Z]{1}|$)"
# loop until matches are found.
while (match(cut_content, regex)) {
# RLENGTH is the length of the matched keyword.
if (RLENGTH > 0) {
# set the matched keyword to the variable `prot`.
# RSTART is the character position of the matched keyword in the file.
# This is the first occurrence of it in the file when read from the start.
prot = substr(cut_content, RSTART-1, RLENGTH+1)
# clean-up punctuation and/or space from the beginning and end of `prot`.
gsub(/[:punct: ]$/, "", prot)
gsub(/^[:punct: ]/, "", prot)
# print the `prot` and 400 chars before and after it using idx
# as an offset. Separate the results by tab char.
print prot, "\t", substr(orig_content, idx+RSTART-400, RLENGTH+800)
# update offset to character position right after the current matched keyword
idx += (RSTART + RLENGTH -1)
# cut-out all the text up until the position of the
# last matched keyword (including it) and on the remaining text
# repeat the above steps in the loop. This is so that the search
# continues throughout the file and doesn't keep finding the same
# keyword over and over again. See `while` condition.
cut_content = substr(cut_content, RSTART+RLENGTH)
}
}
}'
Edit: Formatting. HN doesn't wrap text in code-block.
awk is irreplaceable on the command line because there's not any other good way to extract columns (awk '{print $2, $3}') or parse/work with numbers. How else would you print a running total for select columns easily? awk '$4>3 { x+=$3; print $2, $3, x}'
It's the only tool that's aware of numbers, unlike grep and sed.
My big new power move for awk is -F. You can use any regex as a field separator! Mind blown!
One trick I came up with recently: I found myself needing to add up a column of numbers, but there were so many that I started losing precision to awk's use of floats for all numbers. I wound up augmenting with `dc`[0] - something like (borrowing your example, and working from memory):
awk '
BEGIN { print "0" }
$4 > 3 { print $3"+p" }
' | dc
Quick note: my favorite way to extract columns is:
`cut -d' ' -f1, 5`. You won't be getting a running total but for selecting columns I find it useful.
I've never really been able to keep a significant amount of awk in my head. Mostly I use it for splitting out columns in output, especially when the fields may be padded for output. So "{ print $2 }".
Awk has great abilities, but when I want to use it I always have to spend a lot of time reading.
I've always wanted to come up with something in Python that combined the awesomeness of awk with my ability to pick up and hack something together in Python, but I've never been able to come up with the right semantics.
One thing I think could be really useful is the regex lines range, where you give it two regexes and that block of code gets executed on every line in that range.
{ print $0 }, { print }, and nothing (i.e. no action given - but a pattern is given) are all equivalent, and result in printing the current line of the input (if it matches the pattern). Similarly, if no pattern is given, but an action is given, the default is to match every line (and so perform the given action on every line). Either the pattern or the action must be given, or both.
Much better. Awk is first a filter, then a processor. As others mentioned you don't even need a processor here. Using awk as just a filter is great and easily communicated by proper use of the language.
Recently, I was helping out a friend in analyzing some RNA samples for her work. These samples are huge - like nearly a gigabyte of data. There was this tool which was recommended for the job - mirexpress. It was a small job, perhaps 10 minutes worth of effort. To make my work easier, I provisioned a beefy (and costly) machine on Azure to do the job, took a quick look at the clock (it said 11 PM), ran the tool, and relaxed. The tool crashed while reading the file.
In an attempt to fix the bug, I opened mirexpress's code. And all my confidence in my programming ability vanished when I saw its innards. I understand that the code may have been written by scientists who had no experience in programming, but I have never been so utterly _disoriented_ by bad code. Anyways, after hacking away at the mess for about 3-4 hours, I realized that this was a fool's errand and thought I'll just phone it in the next day saying I couldn't do it. I went to sleep thinking that it was already late and I'd get late for work the next day.
- 5 minutes later -
I woke up with a start, recalling this nifty tool called awk. I had last used it maybe 3 years ago, and before that only in college. But I could see how awk could do some of the things which mirexpress was claiming to do. So I fire up my computer, write an awk script - 2 lines only! TWO FUCKING LINES! And it runs like a charm - eats away at megabytes of sample data and gives me results I can show. So then like any rational person, I spent the remaining hours re-discovering awk and forgot to sleep. Pissed away the whole next day (and some part of the day after that too!) :-D
It's really fascinating that this nifty little tools invented DECADES ago are still going strong, and there's been no _evolutionary_ leap in areas where tools like awk/grep/sed excel at.
At one job I had we got "help" from a local college professor on how to design our database. A lot of what he said made sense. And we were super green and databases were not our expertise or focus. Short on time, high on cash, we asked if he'd implement it for us.
I will never forget my gut wrenching feeling when I opened up one of the main provisioning scripts (in Python) to read the first two lines:
True = 0
False = -1
The rest of it was an unmaintainable disaster that may have worked. I learned so much that summer, having been forced to learn it all myself.
True is normally 1, while False is 0. This seems like specific variable declarations as well, which you generally don't need as they are built into all languages e.g. "variable = false;"
However maybe the professor needed 'true/false' to represent something to be used in his formulas so this made sense to him. But who knows.
In most programming languages, including python, true is represented by 1 and false is represented by 0.
This script is changing it to be backwards, and python is flexible enough to allow that. But I would expect this would break most python libraries, because it's such a core language feature that's being completely restructured.
It’s, IMHO, a terrible idea but the one use I could imagine is if you are doing a lot of stuff with Unix exit codes, where exit code 0 is success. That would cover setting True = 0 ...maybe but not failure which is usually a positive exit code.
No! Actually I take it back this _is_ definitely still a terrible idea and will lead to certain madness for all involved. :)
Luckily, assigning to True and False affects only the current module. If he had changed __builtins__.True and __builtins__.False, on the other hand...
(Python has a search path: it looks first on the current module, then on the built-in __builtins__ module, which is where the built-in constants and functions reside.)
Standard unix tools are so incredibly useful in bioinformatics. Almost all of our data formats are, or can be canonically transformed into, plain-text.
Learning the basics of awk, sed, etc is almost a requirement.
Nice story! I have a friend who worked in bioinformatics, but he was good at programming and went to better-paying pastures.
I wonder if the problem is twofold: 1) a lack of education compounded by 2) the rapid evolution of computer systems.
Unix is a rare beast in that not just its philosophy but even its components survive to this day and remain relevant[1]. People in unrelated fields rebuild tools that could just as well be assembled using Unix's basic components, but they're just not aware of them. And why would they look for these antiquated tools? They've been trained to reasonably expect old tools to have been replaced by newer, better, more featureful ones.
[1] AWK was created before I was born, and I'm among the more senior engineers in my 20+ team.
I think it's just that they are good at biology, not computers. I mean if they entrusted ME to tell two amino acids apart by looking at their behaviour, I'd take a month!
From what I've seen, amid all the research, failed experiments, constant fight for funding (in some cases) and "life" in general, people have much less time to learn these critical things like programming / tools so useful to them. They will do it at some point, but probably not till their life depended on it. On the other hand, if life depended on me recognizing those amino acids, we'd all be :-)
I think part of it too is that it's exploratory. Sometimes I use javascript to write music and as creative twists and turns happen the code folds in on itself in ugly ways that don't happen when solving a clearer problem
Compounded by the fact that in science once you have the result the code is probably just an artifact so there's no real reason to refactor it
Just messing around and nothing worth showing, but the web audio API gives all the low level stuff, and then hot module reload from webpack means you can change a function while the music is playing and it updates in real time. Not being bound by a GUI makes it trivial to do things like arpeggiate or automate filters, and then orchestrate that at higher levels
It's easier to break the mould if you're not bound by other people's software, but starts to look awfully sciency if you explore too far : D So it's useful to keep wiping the slate to not be bogged down by previous experiments
So are you programming on the sequence level (notes, parameter automation) instead of on the synthesis level? From a previous life as a music software geek I remember an abundance of programming environments for synthesis, and a shocking lack of options for programming one level higher.
It's actually pretty common in all kinds of fields to see others reproduce things that they have known for decades. A fun (and infamous) example is this paper (http://care.diabetesjournals.org/content/17/2/152) which rediscovers the trapezoidal rule to calculate area under the curve and gets published.
I am always impressed by the UNIX tools because of their simplicity and yet how easily you can build powerful systems due to their composability. But what I find even more amazing is that they "got it right" so many years ago. We have progressed so much in our understanding of computing and human-computer-interaction, but not many tools that have come out recently can claim similar success.
When you start looking at the code of the tools, and how older systems were design, it's really is a testament to good engineering practices because it all scales so well. Small, simple, "one task well" utilities that process data in streams means the megabyte files of the 80's and 90's are today's terabytes. Sure they require some work to get things scripted correctly but it really is pretty amazing what coreutils and a shell script can do.
After seeing what the demo scene is able to pull off in very little code makes me hope that the art of fundamentals & single responsibility principals like in Unix/Linux resonates with younger developers the ability of truly understanding a problem scope before implementing a solution. Above all else have fun with it, be curious on even a low level how your program functions at an os & hardware levels. Things like what is the von Neumann bottleneck? Or even those building boolean logic gates in minecraft is inspiring to want to know "how does it work? And why?"
A tutorial I had written for IBM developerWorks, titled "Developing a Linux command-line utility" (in C), is mentioned in the Resources section of that page ([0]) above.
But that article was archived from the IBM dW site some time ago, after being there for some years. However, I wrote to IBM and got the PDF of the article, and put it on my Bitbucket account, with the C code for the utility.
It may be of use to people who want to progress beyond using Unix / Linux command-line utilities (in C, but the principles and techniques can be adapted to other languages like Python, Ruby, etc.), to writing such utilities themselves, along with integrating them into shell scripts and pipelines.
The difference is that most of the things back in 60s, 70s and 80s were built by people who were actually passionate about computers and programming. Then the career types took over — the ones who spend more time planning sprints and following SCRUM practices and figuring out ways to exaggerate what they do than they spend writing actual code. Of course, some of the blame can be laid on the feet of the people who fancy themselves engineering manager because they happen to have an MBA — despite the fact that they can’t Engineer their way out of a paper bag.
The one thing I find unfortunate about how people learn awk is the emphasis on making it terse and unreadable, especially trying to turn everything into a shell one-liner.
Don't write one-liners if they're not that simple. Write ten-liners! Add comments! Commit them to version control! Awk can be a nice, readable little language if you're not trying to win at code golf all the time.
bwk addressed this in a video at U Nottingham a few years ago. In a nutshell, he's basically arguing that they designed it to be terse to avoid giving people an excuse to write 1000's of lines for no reason. Less code = better (at least to his mind).
I can't imagine what one would do with thousands of lines of awk, so that seems a little extreme.
Awk is a pretty terse language, and that gives you plenty of room to put in whitespace and comments and still have something short and sweet. I think that the idea of writing code to be nicely readable by your collaborators, or your future self, might not have been around in the early days of UNIX.
("But it's just a one-off thing I'm never going to do again, why should I save it to a file?", you may ask. Is what you do important? Then you'll probably have to do something like it again.)
I have a little repo called “cleversql”. It’s full of little scripts that represent hard problems that I’ve had to solve, clearly-named and well-documented. I fall back on it often, when I remember that I’ve solved a certain kind of problem before, but I cannot recall exactly how.
It’s an invaluable asset and saves me tons of time on a regular basis. I’d rather search my OWN hard drive for an example of my OWN code for how to - for instance - use a CTE to recursively populate a date dimension table, than to search Google and see someone else’s code, to refresh my memory.
I suspect a huge amount of the value of it is because it is personal, I.e. based heavily on his past experience. It means he is always just refreshing his memory rather than looking at something new.
It’s why my Programmer’s Compendium [0] may only ever be useful to me. If you try looking at some of the more complete pages there, you might think they’re useless, but they’re, in fact, all that I need to write down.
However, mine is public, and there’s not much harm in these things being public, because perhaps it can be useful to someone else one day. I would encourage people to share knowledge by default.
So while I also think making it public is a good idea, I would also say never be under pressure to make it accessible to others!
What’s good for learning is almost never good for reference.
The Perl Cookbook is just great. IIRC I have it, though not looked at it lately. (Had read most of it earlier, though.) I also have the Python Cookbook for Python 2 [1] and for Python 3 [2]. Both those are great too. All three are by O'Reilly Media. It's not only that you get ready-made recipes that you can use; you get to learn a lot about language and library features and software design too, from such books.
[1] The chapter on iterators and generators is excellent, and full of useful ideas and code snippets you can reuse and build upon. I think a large part of that chapter was written by Raymond Hettinger, who also has designed and implemented much of those features in Python, IIRC. This book is written by many contributors, for the different sections and recipes.
[2] Interestingly, the Cookbook for Python 3 takes a somewhat different approach. It still has recipes, of course, but they are presented without too much discussion. The authors expect you to (and say so explicitly up front) use more of your time and thinking to figure out how they work (and to read the relevant Python and external library docs for the background information needed), rather than giving detailed explanations (not that the explanations in the previous edition are very detailed, but they are there). David Beazley and Brian Jones are the main authors of this one, IIRC.
Both models have their merits, IMO. In fact, overall, I think the approach of the Python 3 Cookbook may be better, at least for experienced programmers, because it makes you think and do more on your own (using the book as a base), from which you grow more as a programmer.
I use it to save all sorts of clever snippets on the command line. Helps me revise some nifty commands periodically and thereby grow expertise in them with time.
Not really. I don't think it would be terribly useful to anyone else - it's a reflection of how I understand and process certain techniques... and I really don't want the mental overhead of knowing that there's the chance that strangers on the internet could be judging my work.
Furthermore, a lot of it contains stuff that I'm doing for work at any given moment. Some queries are designed to run against databases at work. I don't know what kind of hot water I could get in for letting that code loose on the internet, but I'd rather not find out.
I may consider making some sort of "best of" project and putting some stuff out there, though. It would be fun pick 5-7 concepts that I've struggled with, and spend some time writing about them and making them presentable for the public.
Oh, I think I understand the point here now. I said in my other comment that I can't imagine what one would do with thousands of lines of awk -- but maybe this means their design succeeded.
By designing a terse language, they made people want to do things with it that are short and sweet, preventing it from feature-creeping into a full-featured language where you'd write thousands of lines of spaghetti. Is that the idea?
I believe there is also an earlier reference where he said something similar. If I recall correctly, he said he was surprised when he became aware people were writing large programs in awk.
I also recall a more recent interview where when asked about his language preferences, he admitted he does very little programming anymore. When the interviewer pressed for what language he would use now (sigh), he said he would just use Python.
Build it bit by bit in a wiki. When it gets too big to type in reliably put it in version control. Add a help function and more flags for all the corner cases your coworkers discover.
For the curious, the workflow builds ML models to predict binding to unwanted proteins for new drug compounds, based on drug molecule-protein target interaction data, and is implemented with out Go-based SciPipe workflow library [0].
The rawdata is a 18GB tsv file (ExcapeDB [1]), and AWK really really shines for this.
I sometimes wish we had an SQL-like language for expressing all these computations, because we ended up doing some pretty complex join-like filtering and stuff. But from my tests with SQLite, I would not get a query answer until after like ~5 minutes with SQLite, so I gave up. With AWK, I can do a "head -n 10" on the result of the AWK operation (or on the input data file), and immediately verify that it works as expected. This seems to be one of the strongest points of the unix tool philosophy: Ability to check partial results in no time, and so iterate quicker on developing long "pipelines" of chained commands.
It also fits perfectly as a way to build scipipe workflow components, since having the component code in a separate language (AWK), rather than as inline Go, which is also possible in SciPipe, it is super-easy to send this component of to the HPC resource manager (given that we have a shared parallel filesystem): Just prepend the SLURM [2] salloc command with its parameter, before the command.
Although standard awk has the advantage of being installed on all POSIX systems, for bioinformatics there's a nifty bioawk (https://github.com/lh3/bioawk) that extends awk so that records aren't necessarily lines but fasta/fastq records as well. Pretty neat for oneliners.
> So then like any rational person, I spent the remaining hours re-discovering awk and forgot to sleep. Pissed away the whole next day (and some part of the day after that too!) :-D
Haha! I've been there!
Awk is amazing and when paired with sed it exponentially increases what one can achieve on the command line.
I used mawk for many years and wrote numerous >1000 line programs for manufacturing automation. It is just about the fastest interpreter I have ever encountered, processing multi-megabyte files in just seconds.
shell + shell tools + piping = a league of its own.
As a big fan and advocate of a low level language (C) and a high level language (Python), as a combo, being a very powerful paradigm, I still can't believe some of the things the, shall we call it, shell piping workflow, let's you accomplish in one line and in a matter of minutes.
Here's what you do.
Say you have a text log file. It's "semi structured" (like most log files) in that you can extract quite a bit of information using things like grep/sed/awk, but still not fully parse it (without a very complex parser; we don't want to get into that). What's the first thing you do with the log file? You view it:
view logfile
(view is just opening the file in vim in read-only mode, if you don't have it, you can use "vim -R"). You inspect the file and then quit. Try this instead:
cat logfile | view -
(don't forget the hyphen at the end). Again, inspect and quit (e.g., using :q). You might say, what a round-about/inefficient way to view a logfile. But. Now you can do something in the middle:
cat logfile | do something | view -
inspect and quit. Or more than one thing
cat logfile | do something | do something else | view -
and you keep adding the piped commands until you're satisfied with your output. Once you're satisfied, you can dump the output into another text file as a "report" by replacing the last "| view -" with "> reportfile".
cat logfile | many | processing | commands | later > reportfile
and you can do this multiple times to generate multiple report files of various kinds. And you can concatente some of those report files
cat file1 file2
or put some columns of some of the report side by side
To give one example from my shell history. Some times I have a dozens of pdf files open in my linux desktop (using evince pdf viewer) and I wanna restart the computer but don't want to lose track of which pdf files were open. There may be automated ways of doing this but let's say there aren't any. I start with ps ax:
ps ax | view -
A long log. I want to pick only lines that have evince:
ps ax | grep evince | view -
Now I included the 'grep evince' line too, which I don't want:
ps ax | grep evince | grep -v grep | view -
Good. But I don't care about all the columns of ps log, except for the last one (the one that shows the full path of the file). That's column 6:
Then I close all my pdfs, and reboot. And I don't know grep, awk except for very basic things like what I already did. But I know similar tidbits about many other commands. If I have to perform regex search only I use grep, but for search and replace I use sed. Sometimes cut is more handy for column selection than awk. If I don't know the command but I know what I need to do, I just google it, and 99% of the time, I can find a stackoverflow post where a linux based one-liner is mentioned which is pretty much a drop-in for my piping workflow.
Finally, if I want to traverse through each line of the entry and process one by one, I use the while loop. For example, if instead of dumping the file paths of the pdf I wanted to format it a little bit using directory name and file name, I'll do this:
This is a great technique; thanks for sharing it. While I knew about all the individual parts that you used, I don't think I ever thought of combining all the parts with the "view -" at end, although I knew that "-" as a command-line arg means standard input (to many commands that support it).
Very useful stuff, and as you say, the possibilities are endless.
I had done something sort of analogous, an experimental Python tool called pipe_controller, which is not about piping commands to each other in the traditional Unix sense; rather it is about "piping" the output of one function to a second one, and the output of the second to a third one, and so on, as many as one needs, under the control of a for loop. The net effect is like normal composition of function calls, like f(g(h(x))), but some other interesting effects can be achieved by doing it with a for loop, and changing some things at run time:
After first creating pipe_controller (which is simple, really), I played around with using it in a few different ways, and found that it can be used for at least a couple of interesting things:
- running a "pipe" (of those functions) incrementally (something like the technique you showed), and saving / viewing the output of each intermediate stage;
- swapping components of the "pipe" at run time, under program control, which again can lead to some interesting use cases.
I blogged a small series of posts about pipe_controller and such uses of it. Here are the two last or so posts, and the previous posts can be reached by following links in those posts:
Swapping pipe components at runtime with pipe_controller:
Interesting. Looks like you're trying to do shell piping in python but without the annoying nesting.
I have thought of this myself too, both in terms of python and in terms of scheme. For example, for your python function composition example f(g(h(x))), shell piping would look like this:
cat x | h | g | f | view -
while in scheme it would be like python, just a little different:
(f (g (h x)))
Essentially the point is, by using existing syntax and language facilities, can we mimic the seamless shell piping workflow (e.g., with no annoying nesting).
I think one issue that python or scheme will have is that, when something goes wrong, debugging the problem would still be very annoying in python/scheme, whereas in shell piping, you simply remove some tail-end processing commands to view and earlier output, then fix your issue, then reintroduce the tail-end commands that you removed, possibility with some modifications. So "view -" acts as a debugger, not just for visual inspection of your report.
Anyway this is an interesting area. Keep up the good work. I would just like to mention that this is related, in fact it is, part of a much larger paradigm of programming called dataflow programming, sometimes called stream processing. And also has connection with reactive programming. (which if you lookup in wikipedia [1] falls into a larger paradigm called declarative programming).
The most important feature of awk is simplicity. No worries about complex language syntax, you just have a language that seems like C without everything that makes it complex. And it excels at a simple task: reading lines of data and transforming them as needed.
Yes, which is one reason I like PowerShell so much.
As a shell, the tab completion, parameter completion, long names, makes it easier to discover, easier to understand, and easier to remember.
Then it's a high level language too so if there's something that needs scripting, it doesn't mean a complete change from shell to Python/Ruby/Perl, it stays PowerShell.
Then it's a .Net language too, so if there's something getting a bit big for it, it doesn't mean a complete change to C#/Java instead, it means a small change to PowerShell with .Net methods, then maybe PowerShell with a C# core (like Python with a C module, but still much easier to create).
Of course there's a bit of XKCD "fix having too many things by adding another thing" going on.
But, the fact that it covers the common shell tools with all their different syntaxes reasonably well, and it can be tuned to approach the speed of C# as well, makes it useful for a whole lot of situations, despite having a pretty huge syntax and list of warts, it still seems to come out well.
I once read the two or three first chapters of _ The AWK Programming Language_. Those chapters cover the whole language and I could read it in one sitting (actually standing; in a book store, but I digress).
I mostly think of awk as a query language for text that's in record format. It doesn't seem to handle unformatted text very well and I tend to use sed or some other kind of tool to clean it up before feeding it to awk.
I like the baked in logic for dealing with record/field formatted data. One thing that seems to hang people up is the lack of a concatenation operator. You just put things next to each other. It looks a little weird. Having associative arrays is a nice plus-up from shell programming.
Don't forget about `-F`, the field delimator (which is WHITE by default). Obviously an option of awk, but some people disregard some *nix tools, because there data is not space delimited (and delimiter choice flags differ between tools :( ).
For quick filtering it's really great, e.g. you can parse simple (pretty-printed) XML tags via `-F[<>] '{print $2}'` (for a quick glance on the data - of course not a good idea in production).
246 comments
[ 2.6 ms ] story [ 245 ms ] threadIf you are interested in learning Awk, I highly recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger. It's about the same size as the original "The C Programming Language" and is equally well-written. Previously on HN: https://news.ycombinator.com/item?id=13451454
That must be where the name comes from, right?
Also, Awk isn't great for making reports, which is why Perl 5 to this day has an awkward report creation system[1] that looks like some COBOL refugee instead of idiomatic perl code.
[1] https://perldoc.perl.org/perlform.html
That seems ridiculous; where is it substantiated?
When Wall posted Perl to comp.sources.unix for the first time, he wrote "If you have a problem that would ordinarily use sed or awk or sh, but it exceeds their capabilities or must run a little faster, and you don't want to write the silly thing in C, then perl may be for you."
Or rather, not Larry Wall, but the apparent newsgroup moderator added that text, lifting it from the Perl manual page.
Thus he was pitching it as something that performs faster than awk and sed, with a greater range of capabilities.
> News was maintained in separate files on a master machine, with lots of cross references between files. Larry's first thought was "Let's use awk." Unfortunately, awk couldn't handle opening and closing of multiple files based on information in the files. Larry didn't want to have to code a special-purpose tool. As a result, a new language was born.
So that's why it's the Practical Extraction and Reporting Language. He wanted to extract data from files and generate reports.
If you're of a certain age and read that and grimace as you immediately understand why this would happen, it does rather put into perspective the misery of dealing with, say, Webpack configuration.
Did email clients not handle "dot stuffing" back then? That is, if a line begins with a single dot, the client would automatically insert another dot right before it. Then, at the receiving end, the client would remove the extra dot at the beginning of the line.
Why?
I could be wrong though. Awk is one of those tools like vi where you can use it for years and still be discovering new features.
Good point. The BEGIN and END patterns do work for global headers and footers, totals, etc., but not for per-page stuff. You can do it yourself with some extra awk code, but yes, you have to write it. Not difficult, though. I guess it was not designed to be a Crystal Reports-like reporting tool, with report bands and what not.
>I could be wrong though. Awk is one of those tools like vi where you can use it for years and still be discovering new features.
Agreed :) Not only new features, even new uses for existing features, because, although it is a sort of DSL or little language (but a programmable one), the area it is applicable to, pattern matching and data processing of many kinds, is vast.
As many others pointed out in this thread, the fact that the reading of input is built-in to it (whether from standard input or files given as command-line arguments), saves you a bit of boiler-plate code each time (cumulatively) you write an awk program using that feature. So does the pattern-action model, with those two defaults for missing pattern or action (match all lines, or print). And again as others have said, Perl, Ruby, etc. have that feature too (the first one).
1. A simple interpreter for an awk-like language called qawk. qawk is like awk except that it allows for querying by field name rather than field number. For instance, it allows doing
instead of the more cryptic 2. An awk program that takes another awk program (in their example, a sorting algorithm) and outputs a version of that program modified to include profiling statements and an END section that outputs the results of those profiling statements to some file; then, another awk program that reads the data in that file and inserts that data back into the original awk program, thereby approximating where the hotspots are.There's a lot more in the book besides these, but to me these are the coolest programs because they are the awk-iest, by which I mean that they loop over lines of input, split the fields of those lines, and then manipulate the fields. Some of the programs in the book don't do this; instead, they consist of a single large BEGIN block with typical for-loops, arrays, etc. Used in this way, awk is just yet another dynamic language.
http://github.com/dkogan/vnlog
Am I right that qawk was included as a program in the text? Did they ever follow up with further uses?
- relfile is a file containing the field attributes used in various database files,
- readrel is a function that parses the relfile and stores the fields in a dictionary, and
- doquery is a function that takes a qawk query, converts it to an awk query by replacing the field names with their corresponding field numbers, and then executes the awk command.
The whole thing runs about 60 lines.
I'm a big fan of small utilities :) - as I sometimes say in my email sig; but more importantly, I'm a big fan of Kernighan et al, where by "et al" I mean the others from the core early Unix days, such as Dennis Ritchie, Rob Pike, Ken Thompson and many unnamed others, from whom I (and tons of others) learned about the Unix command-line (tools), the shell (scripting), and the Unix philosophy [1].
Had written this just a few weeks ago on HN, in the thread titled "Technical Writing: Learning from Kernighan", but worth repeating here in the context of this thread:
https://news.ycombinator.com/item?id=17163276
It's a list of his books. I guess many may not know of some of them - I know I didn't.
[1]:
The Unix Philosophy in One Lesson:
http://www.catb.org/esr/writings/taoup/html/ch01s07.html
Attitude Matters Too:
http://www.catb.org/esr/writings/taoup/html/ch01s09.html
"If someone has already solved a problem once, don't let pride or politics suck you into solving it a second time rather than re-using."
I'd strongly argue it's overzealous. As much as I agree "reinventing the wheel" is dangerous, tempting, and can quickly spiral to yak shaving, but Unix itself, and all the good it brought, is a prime example of "solving [a problem] a second time" after Multics.
In other words, I'd restate it in Sage Speak™: "Don't do this. Except when you need to." ;P Or, just want to have fun :P
Not sure about that. I mean, I know it came after Multics and was inspired by it (due to some of the early Unix people having worked on Multics), including that the name was originally Unics (I've heard, as a word play on Multics, because it was originally written by one person or was originally a single-user OS, maybe), but I am not so sure that all the good it brought was from Multics. Likely Unix brought some new stuff too. Others who know better may be able to say more.
>In other words, I'd restate it in Sage Speak™: "Don't do this. Except when you need to." ;P Or, just want to have fun :P
Good one. A bit Zennish :) Check out one of ESR's other compilations, the Unix Koans of Master Foo, if not seen already ...
http://www.catb.org/esr/writings/unixkoans/introduction.html
https://www.gnu.org/software/gawk/manual/gawk.html
It's a pretty good book which teaches you both techniques and the nitty-gritty. I recommend this.
it is available from internet archive, so i guess a legit copy.
I think Kernighan also stated somewhere that awk was (also) designed as a helper-tool to learn C.
All in all, I think it's a great first language, even if you're initially more compelled by Lisp family languages. If you have time to learn only one language, then awk is not a bad choice; It'll open many doors in the Unix/KISS-world.
I can also state that "The Awk programming language" is, among other things, an excellent introduction to computer science or "the way programmers think" in general. A remarkably well written book for general audience.
Just observe the kind of kindergarten antics some in the _sec world will get up to...
Plenty of people that would rather drop to Perl or Python once awk gets too cumbersome for the command line.
Still, I'm not certain if the original comment is complaining about the tendency toward one-liners by actually writing less code (something the article alludes to, "only the inner part of the for loop") or merely cramming what would otherwise be readable as a multi-line, well-indented program into a single line.
Interesting... to me the value (er, a value) is that you can write
instead of And another value is that it's more likely to be preinstalled than Python.I really like the concept of Wolfram Mathematica Notebooks as you can essentially embed everything into a reproducible analysis (what they call a computational essay). The free and open source equivalent is Jupyter notebooks.
I tried answering a few awk questions being verbose and explaining the steps. But I noticed that most are not asking to learn but to get a quick fix. Soon after that came the second realization: several of the answers did not understand what they where doing.
My conclusion: We are living in a cut'n'paste world were actual understanding is less valued.
Naturally code reuse is nice. But I doubt that many users are able to construct their magic awk snippets from the ground up. Hence the prolific terse magic snippets.
Don't know why it gives me joy to write those things even when I know it would be far better to write the same in three nested for loops (which are less error prone, more maintainble and gives more flexibility like easy to "break").
Secondly and this is a personal preference but the code looks more readable when the for statements are on different lines telling you which array is being iterated compared to the nested functions where the output of one function is piped as input to another.
For some tasks it's amazingly effective, and it's usually installed even if Python isn't.
(The twitter thread that follows also has some great one-liners!)
It can be really helpful in mundane data/logs analysis, for example.
If I have to look up something 2-3 times in a month it's usually committed to memory. Any less and it's probably better forgotten anyway, haha.
The fact that most OSs have a POSIX compliant version of it in the base system also make it very valuable, not just for the people crunching data but also to sysadmin/devops people.
My 99% use case is extracting particular fields from some input stream.
That's a tremendously common problem and awk is the best tool for the job.I once started diving deeper into awk and there is an impressive amount you can do with it, but I hit a point pretty quickly where it would be cleaner to write the script in a more traditional language where my successor won't be cursing my name for writing something complex in an obscure language.
However, for ad-hoc command-line usage, I still prefer awk, because (1) the default delimiter is white space, which means I don't have to worry about if what I'm looking at might contain tabs intermingled with spaces (or how many spaces) and (2) it makes it much easier to incrementally increase the complexity of my ad-hoc operation, such as if I decide I want to take a sum or average of one of those columns, after all.
It's a narrow tool. Other than the inherent advantage this brings in clarity/simplicity in a script, there's nothing else to it.
Consider a script that originally had now awk but just a
but now has been modified to have awk elsewhere. Did the above just become less simple or clear? If so, is it worth changing it to the below for clarity? Does it matter if the awk predates the cut, instead? I say "no" to both, although I do recognize the argument for just using the same, consistent tool everywhere, instead.Can be done with tr, without writing a utility (if by "collapse" you mean what I think you do):
This file t:
$ cat t
the quick brown fox jumped
containing many combinations of spaces, tabs and newlines (whitespace) can be changed to this file t2:$ cat t2
the
quick
brown
fox
jumped
over
the
lazy
dog
by this tr command:
tr -cs "[a-zA-Z]" "\012" < t > t2
That also makes the output more amenable to further processing, including common tasks like finding the frequencies of the words in the input, as mentioned in the "More shell, less egg" post mentioned in this post:
The Bentley-Knuth problem and solutions:
https://jugad2.blogspot.com/2012/07/the-bentley-knuth-proble...
(which I somehow managed to read as an awk invocation).
Re: python - I believe if using things like \w, \d or \s you should be Unicode safe.
Come to think of it, I seem to recall gnu tools should also have Unicode aware pattern/character classes, eg:
https://www.gnu.org/software/gawk/manual/html_node/Bracket-E...
> For example, before the POSIX standard, you had to write /[A-Za-z0-9]/ to match alphanumeric characters. If your character set had other alphabetic characters in it, this would not match them. With the POSIX character classes, you can write /[[:alnum:]]/ to match the alphabetic and numeric characters in your character set.
https://docs.python.org/3/library/re.html#re-syntax
[ed:
Apparently gnu tr is still out in the cold re:unicode;
https://www.gnu.org/software/coreutils/manual/html_node/tr-i...
> Currently tr fully supports only single-byte characters. Eventually it will support multibyte characters; when it does, the -C option will cause it to complement the set of characters, whereas -c will cause it to complement the set of values. This distinction will matter only when some values are not characters, and this is possible only in locales using multibyte encodings when the input contains encoding errors.
]
- outputting my selected fields in an order other than their order in the input lines: you can't `cut -f2,1`, for example.
- use a multi-character or regex field separator: -F'<space><star><comma><space><star>' [edit: rewritten in words to dodge HN's formatting] is a real winner when dealing with hand-written CSV.
- use a record separator other than \n: thanks, Windows, for all those times I've written ORS="\r\n" :P - you can even use a regex record separator, which not even perl supports; I don't need that often, but it's a godsend when I do.
- select based on criteria other than "number of fields from the start": I recently needed the last field from lines of varying field-count, and awk '{print $NF}' was just the thing.
But if you don't need anything like that, cut is a lot more succinct - especially if you need to split on tabs only, which it does by default, while awk defaults to splitting on spaces too.
You can shell out if necessary:
Also: coprocesses.Another common usage for awk in my case is to do floating operations (*, +, eq, gt, etc) in a shell script. It avoids installing bc, which is not always present by default.
I used it for other things, like for example comparing software versions in shell scripts.
The most advanced usage of awk in my case was to do some statistic analysis (average, confidence interval, standard deviation) of network simulations.
You need to be cautious however, gnu awk can be somewhat inconsistent at times:
try the following on various distros (and also an *BSD) for example:
awk 'BEGIN {print "9.1" >= "8"}';
awk 'BEGIN {print "9.1" > "8"}';
awk 'BEGIN {print "9.1" < "8"}';
awk 'BEGIN {print "9.1" <= "8"}';
on OpenBDS, you get no output or a syntax error, on Debian, you get the correct result for < and <= in all cases, never for > (it's interpreted as a redirection), and >= only works with newer Debian versions (sid and maybe stretch).
I know that the fix is simple: just put some parenthesis (awk 'BEGIN {print ("9.1" <= "8")}'), but I was quite amused to discover that a few days ago.
Unix was mature even then. There is NO WAY you would have convinced me that 30 years later I'd still be using most of that, still gluing things together with the same shell tools. Or that most of the books would still have a place on the shelf (albeit sometimes in 2nd or 3rd editions), and turn out to be classics.
Ignoring big iron, we'd gone from Apple IIs to Amigas and workstations in just a few years, and hardware was changing as rapidly as the current fashion for JS frameworks does today.
Yet here we are, awk, grep, sed and the other tools still feel like the right, concise answer. So why not learn them, they might greatly surprise you by their staying power.
Awk is a compiled language. Your Awk script is compiled once and applied to every line of your file at C-like speeds. It is way faster than Python.
If you learn to use Awk well, you will start doing things with data that you wouldn't have had the patience to do in an interpreted language.
I say this with no disrespect for Python, it's just you should know the tradeoffs of the tools you use. It does not matter if the individual operations in Python are implemented in C. It will still be much slower at looping over lines of a file than a compiled language designed decades ago for that exact purpose.
I have a lot of Python file in one dir:
Finding them and cating them all takes about 0.3 secs: So to have something simple that takes a bit of time, I tried to get all lines starting with "print", and output the first thing after that.I'm really bad at awk, so I don't know if there is a better way. I went for the most obvious thing for me:
Now, with Python, it's definitely not as easy to type. You have to get a script like. awk wins the expressivity metrics for this use case: But as for performance, I don't get the huge boost in perfs you are talking about: I do get the same output though:Here's what I observed: I had a simple text-processing tool I needed that I call "countmerge", which just merges adjacent lines with the same key and adds up their corresponding values. I needed to run it on a lot of large files.
I first wrote it in Python, where it was a significant bottleneck compared to the steps that came before it (split, sort, uniq -c). Eventually I rewrote it in Rust [1], and it was at least 5 times faster, at the expense of a fair amount more low-level code. But then rewriting it in awk [2] turned out to be as fast as what I wrote in Rust, possibly inconclusively faster.
[1] https://github.com/rspeer/countmerge
[2] https://gist.github.com/rspeer/60c87dca1ab550326f8bd6d086452...
And if you really want that in a repo, you can stick that in ansible trivially.
Sometimes instead I just wrap a line of awk in %x() in Ruby to save a few chunks of code.
Most programmers don't understand why I do this though. :(
Hundred of lines of python by two awk lines ? Must be very bad python programmers.
Sure, the gnu version is more powerful, but the plan9/original awk implementation handles most cases I run into, and I can now look up the gnu extras when I need them.
http://man.cat-v.org/plan_9/1/awk
ruby inherited a lot of structure and ideas from perl which it inherited from awk and that thought makes me irrationally happy
I tried writing a script that would extract entries from logs (e.g. if every entry started "====CRITICAL ERROR LOG 2018/06/15====", I wanted it to start there and grab every line below it until it hits the next log entry starting with the "=====" header), but I gave up. Your script might do something similar, if I can figure out what it's looking for.
Another option might be to use multi-line records (https://www.gnu.org/software/gawk/manual/html_node/Multiple-...). First pre-process the file to add an empty line before each header, then use blank lines as the record separator and newlines for the field separator.
Intent: Given a folder with multiple text files of academic papers from PubMed, remove the contents after the '====Refs' header. Then in each file find occurrences of any term in a keywords file and extract it along with 400 chars before and after it. Print this output as a tsv.
input: $1) A text file containing keywords of interest that need to be searched in the files separated by `|`.
$2) File containing paragraphs of lots of text with headers starting with '===='. There isn't a formatting requirement really. Just that, in my case the files happened to be so.
Edit: Formatting. HN doesn't wrap text in code-block.My big new power move for awk is -F. You can use any regex as a field separator! Mind blown!
Edit: Better syntax per comments.
Want to throw a few examples my way?
Disclaimer: never learned perl
Scalar value @F[0] better written as $F[0] at -e line 1.
Scalar value @F[1] better written as $F[1] at -e line 1.
In Perl 5 the array elements of @a are $a[0] $a[1] etc. However @F[0] works too because a "slice" is returned.
Awk has great abilities, but when I want to use it I always have to spend a lot of time reading.
I've always wanted to come up with something in Python that combined the awesomeness of awk with my ability to pick up and hack something together in Python, but I've never been able to come up with the right semantics.
One thing I think could be really useful is the regex lines range, where you give it two regexes and that block of code gets executed on every line in that range.
Using "200" in quotes makes sense if you're looking for an exact string match; this will fail if the datum is 0200, or 200.0.
In an attempt to fix the bug, I opened mirexpress's code. And all my confidence in my programming ability vanished when I saw its innards. I understand that the code may have been written by scientists who had no experience in programming, but I have never been so utterly _disoriented_ by bad code. Anyways, after hacking away at the mess for about 3-4 hours, I realized that this was a fool's errand and thought I'll just phone it in the next day saying I couldn't do it. I went to sleep thinking that it was already late and I'd get late for work the next day.
- 5 minutes later -
I woke up with a start, recalling this nifty tool called awk. I had last used it maybe 3 years ago, and before that only in college. But I could see how awk could do some of the things which mirexpress was claiming to do. So I fire up my computer, write an awk script - 2 lines only! TWO FUCKING LINES! And it runs like a charm - eats away at megabytes of sample data and gives me results I can show. So then like any rational person, I spent the remaining hours re-discovering awk and forgot to sleep. Pissed away the whole next day (and some part of the day after that too!) :-D
It's really fascinating that this nifty little tools invented DECADES ago are still going strong, and there's been no _evolutionary_ leap in areas where tools like awk/grep/sed excel at.
One of these days, when the mood strikes, I'll upload the matlab script I inherited from my Ph.D supervisor.
Scientific programming is, by and large, an abysmal state of affairs.
I will never forget my gut wrenching feeling when I opened up one of the main provisioning scripts (in Python) to read the first two lines:
The rest of it was an unmaintainable disaster that may have worked. I learned so much that summer, having been forced to learn it all myself.Could you please explain what is so egregious about those two lines for those people that may not see it? Asking for a friend.
However maybe the professor needed 'true/false' to represent something to be used in his formulas so this made sense to him. But who knows.
This script is changing it to be backwards, and python is flexible enough to allow that. But I would expect this would break most python libraries, because it's such a core language feature that's being completely restructured.
https://en.wikipedia.org/wiki/Boolean_data_type
Is there a valid use case for changing the numeric values of True and False?
No! Actually I take it back this _is_ definitely still a terrible idea and will lead to certain madness for all involved. :)
Python 2 has _many_ quirks like this one. Python 3 was not just "it's unicode".
[1] https://docs.python.org/3/whatsnew/3.0.html#changed-syntax
(Python has a search path: it looks first on the current module, then on the built-in __builtins__ module, which is where the built-in constants and functions reside.)
https://codesearch.debian.net/search?q=%28%3Fm%29%5E%5Cs%2AT...
Learning the basics of awk, sed, etc is almost a requirement.
Most of my time is spent on the terminal and without these I'd shoot myself.
I wonder if the problem is twofold: 1) a lack of education compounded by 2) the rapid evolution of computer systems.
Unix is a rare beast in that not just its philosophy but even its components survive to this day and remain relevant[1]. People in unrelated fields rebuild tools that could just as well be assembled using Unix's basic components, but they're just not aware of them. And why would they look for these antiquated tools? They've been trained to reasonably expect old tools to have been replaced by newer, better, more featureful ones.
[1] AWK was created before I was born, and I'm among the more senior engineers in my 20+ team.
From what I've seen, amid all the research, failed experiments, constant fight for funding (in some cases) and "life" in general, people have much less time to learn these critical things like programming / tools so useful to them. They will do it at some point, but probably not till their life depended on it. On the other hand, if life depended on me recognizing those amino acids, we'd all be :-)
Compounded by the fact that in science once you have the result the code is probably just an artifact so there's no real reason to refactor it
Could you explain what you mean by this?
It's easier to break the mould if you're not bound by other people's software, but starts to look awfully sciency if you explore too far : D So it's useful to keep wiping the slate to not be bogged down by previous experiments
pd, processing, chuck, cm/clm, or just old-school mod-tracking are some good ways to get going
It's a shame to see so much bloated, over abstracted development these days with more emphasis on delivery than doing one thing well.
https://en.wikipedia.org/wiki/Demoscene
After seeing what the demo scene is able to pull off in very little code makes me hope that the art of fundamentals & single responsibility principals like in Unix/Linux resonates with younger developers the ability of truly understanding a problem scope before implementing a solution. Above all else have fun with it, be curious on even a low level how your program functions at an os & hardware levels. Things like what is the von Neumann bottleneck? Or even those building boolean logic gates in minecraft is inspiring to want to know "how does it work? And why?"
[0] http://people.fas.harvard.edu/~lib113/reference/unix/writing...
But that article was archived from the IBM dW site some time ago, after being there for some years. However, I wrote to IBM and got the PDF of the article, and put it on my Bitbucket account, with the C code for the utility.
This post on my blog describes the article:
https://jugad2.blogspot.com/2014/09/my-ibm-developerworks-ar...
And here is the Bitbucket project for selpg, the utility used as a case study, with the C code and the article text (as PDF):
https://bitbucket.org/vasudevram/selpg
The article and all the source files are here:
https://bitbucket.org/vasudevram/selpg/src
It may be of use to people who want to progress beyond using Unix / Linux command-line utilities (in C, but the principles and techniques can be adapted to other languages like Python, Ruby, etc.), to writing such utilities themselves, along with integrating them into shell scripts and pipelines.
https://github.com/RetroBSD/retrobsd/tree/master/src/cmd/awk
https://github.com/onetrueawk/awk
Don't write one-liners if they're not that simple. Write ten-liners! Add comments! Commit them to version control! Awk can be a nice, readable little language if you're not trying to win at code golf all the time.
Awk is a pretty terse language, and that gives you plenty of room to put in whitespace and comments and still have something short and sweet. I think that the idea of writing code to be nicely readable by your collaborators, or your future self, might not have been around in the early days of UNIX.
("But it's just a one-off thing I'm never going to do again, why should I save it to a file?", you may ask. Is what you do important? Then you'll probably have to do something like it again.)
It’s an invaluable asset and saves me tons of time on a regular basis. I’d rather search my OWN hard drive for an example of my OWN code for how to - for instance - use a CTE to recursively populate a date dimension table, than to search Google and see someone else’s code, to refresh my memory.
It’s why my Programmer’s Compendium [0] may only ever be useful to me. If you try looking at some of the more complete pages there, you might think they’re useless, but they’re, in fact, all that I need to write down.
However, mine is public, and there’s not much harm in these things being public, because perhaps it can be useful to someone else one day. I would encourage people to share knowledge by default.
So while I also think making it public is a good idea, I would also say never be under pressure to make it accessible to others!
What’s good for learning is almost never good for reference.
[0] https://qasimk.gitbooks.io/programmers-compendium/content/
Maybe. But stuff like The Perl Cookbook helped me a lot in the past.
[1] The chapter on iterators and generators is excellent, and full of useful ideas and code snippets you can reuse and build upon. I think a large part of that chapter was written by Raymond Hettinger, who also has designed and implemented much of those features in Python, IIRC. This book is written by many contributors, for the different sections and recipes.
[2] Interestingly, the Cookbook for Python 3 takes a somewhat different approach. It still has recipes, of course, but they are presented without too much discussion. The authors expect you to (and say so explicitly up front) use more of your time and thinking to figure out how they work (and to read the relevant Python and external library docs for the background information needed), rather than giving detailed explanations (not that the explanations in the previous edition are very detailed, but they are there). David Beazley and Brian Jones are the main authors of this one, IIRC.
Both models have their merits, IMO. In fact, overall, I think the approach of the Python 3 Cookbook may be better, at least for experienced programmers, because it makes you think and do more on your own (using the book as a base), from which you grow more as a programmer.
I use it to save all sorts of clever snippets on the command line. Helps me revise some nifty commands periodically and thereby grow expertise in them with time.
Furthermore, a lot of it contains stuff that I'm doing for work at any given moment. Some queries are designed to run against databases at work. I don't know what kind of hot water I could get in for letting that code loose on the internet, but I'd rather not find out.
I may consider making some sort of "best of" project and putting some stuff out there, though. It would be fun pick 5-7 concepts that I've struggled with, and spend some time writing about them and making them presentable for the public.
By designing a terse language, they made people want to do things with it that are short and sweet, preventing it from feature-creeping into a full-featured language where you'd write thousands of lines of spaghetti. Is that the idea?
Either you won’t understand it later, or you’re just wasting effort.
I also recall a more recent interview where when asked about his language preferences, he admitted he does very little programming anymore. When the interviewer pressed for what language he would use now (sigh), he said he would just use Python.
It's not worth it. As soon as you go over about 3 lines, you're better off switching to Ruby, Python or even Perl.
I tend to one line most of my things in bash and tend to write a script if I have a multiple use case.
Such is the folly in bioinformatics. Most of the stuff that we use tend to be one off. :/
For the curious, the workflow builds ML models to predict binding to unwanted proteins for new drug compounds, based on drug molecule-protein target interaction data, and is implemented with out Go-based SciPipe workflow library [0].
The rawdata is a 18GB tsv file (ExcapeDB [1]), and AWK really really shines for this.
I sometimes wish we had an SQL-like language for expressing all these computations, because we ended up doing some pretty complex join-like filtering and stuff. But from my tests with SQLite, I would not get a query answer until after like ~5 minutes with SQLite, so I gave up. With AWK, I can do a "head -n 10" on the result of the AWK operation (or on the input data file), and immediately verify that it works as expected. This seems to be one of the strongest points of the unix tool philosophy: Ability to check partial results in no time, and so iterate quicker on developing long "pipelines" of chained commands.
It also fits perfectly as a way to build scipipe workflow components, since having the component code in a separate language (AWK), rather than as inline Go, which is also possible in SciPipe, it is super-easy to send this component of to the HPC resource manager (given that we have a shared parallel filesystem): Just prepend the SLURM [2] salloc command with its parameter, before the command.
[0] http://scipipe.org
[1] https://zenodo.org/record/173258
[2] https://slurm.schedmd.com/
you might be interested in datajoint:
https://github.com/datajoint/
https://tutorials.datajoint.io/
Haha! I've been there!
Awk is amazing and when paired with sed it exponentially increases what one can achieve on the command line.
As a big fan and advocate of a low level language (C) and a high level language (Python), as a combo, being a very powerful paradigm, I still can't believe some of the things the, shall we call it, shell piping workflow, let's you accomplish in one line and in a matter of minutes.
Here's what you do.
Say you have a text log file. It's "semi structured" (like most log files) in that you can extract quite a bit of information using things like grep/sed/awk, but still not fully parse it (without a very complex parser; we don't want to get into that). What's the first thing you do with the log file? You view it:
(view is just opening the file in vim in read-only mode, if you don't have it, you can use "vim -R"). You inspect the file and then quit. Try this instead: (don't forget the hyphen at the end). Again, inspect and quit (e.g., using :q). You might say, what a round-about/inefficient way to view a logfile. But. Now you can do something in the middle: inspect and quit. Or more than one thing and you keep adding the piped commands until you're satisfied with your output. Once you're satisfied, you can dump the output into another text file as a "report" by replacing the last "| view -" with "> reportfile". and you can do this multiple times to generate multiple report files of various kinds. And you can concatente some of those report files or put some columns of some of the report side by side The possibilities are endless.To give one example from my shell history. Some times I have a dozens of pdf files open in my linux desktop (using evince pdf viewer) and I wanna restart the computer but don't want to lose track of which pdf files were open. There may be automated ways of doing this but let's say there aren't any. I start with ps ax:
A long log. I want to pick only lines that have evince: Now I included the 'grep evince' line too, which I don't want: Good. But I don't care about all the columns of ps log, except for the last one (the one that shows the full path of the file). That's column 6: Looks good. Generate a report file from this: Then I close all my pdfs, and reboot. And I don't know grep, awk except for very basic things like what I already did. But I know similar tidbits about many other commands. If I have to perform regex search only I use grep, but for search and replace I use sed. Sometimes cut is more handy for column selection than awk. If I don't know the command but I know what I need to do, I just google it, and 99% of the time, I can find a stackoverflow post where a linux based one-liner is mentioned which is pretty much a drop-in for my piping workflow.Finally, if I want to traverse through each line of the entry and process one by one, I use the while loop. For example, if instead of dumping the file paths of the pdf I wanted to format it a little bit using directory name and file name, I'll do this:
Very useful stuff, and as you say, the possibilities are endless.
I had done something sort of analogous, an experimental Python tool called pipe_controller, which is not about piping commands to each other in the traditional Unix sense; rather it is about "piping" the output of one function to a second one, and the output of the second to a third one, and so on, as many as one needs, under the control of a for loop. The net effect is like normal composition of function calls, like f(g(h(x))), but some other interesting effects can be achieved by doing it with a for loop, and changing some things at run time:
After first creating pipe_controller (which is simple, really), I played around with using it in a few different ways, and found that it can be used for at least a couple of interesting things:
- running a "pipe" (of those functions) incrementally (something like the technique you showed), and saving / viewing the output of each intermediate stage;
- swapping components of the "pipe" at run time, under program control, which again can lead to some interesting use cases.
I blogged a small series of posts about pipe_controller and such uses of it. Here are the two last or so posts, and the previous posts can be reached by following links in those posts:
Swapping pipe components at runtime with pipe_controller:
https://jugad2.blogspot.com/2012/10/swapping-pipe-components...
Using PipeController to run a pipe incrementally:
https://jugad2.blogspot.com/2012/09/using-pipecontroller-to-...
The pipe_controller code is here:
https://bitbucket.org/vasudevram/pipe_controller
Generate PDF from a Python-controlled Unix pipeline:
https://jugad2.blogspot.com/2016/01/generate-pdf-from-python...
Code for recent post about PDF from a Python pipeline:
https://jugad2.blogspot.com/2016/01/code-for-recent-post-abo...
I have thought of this myself too, both in terms of python and in terms of scheme. For example, for your python function composition example f(g(h(x))), shell piping would look like this:
cat x | h | g | f | view -
while in scheme it would be like python, just a little different:
(f (g (h x)))
Essentially the point is, by using existing syntax and language facilities, can we mimic the seamless shell piping workflow (e.g., with no annoying nesting).
I think one issue that python or scheme will have is that, when something goes wrong, debugging the problem would still be very annoying in python/scheme, whereas in shell piping, you simply remove some tail-end processing commands to view and earlier output, then fix your issue, then reintroduce the tail-end commands that you removed, possibility with some modifications. So "view -" acts as a debugger, not just for visual inspection of your report.
Anyway this is an interesting area. Keep up the good work. I would just like to mention that this is related, in fact it is, part of a much larger paradigm of programming called dataflow programming, sometimes called stream processing. And also has connection with reactive programming. (which if you lookup in wikipedia [1] falls into a larger paradigm called declarative programming).
[1] https://en.wikipedia.org/wiki/Programming_paradigm
Thanks for the encouragement and links. Will check them out.
(1) https://stedolan.github.io/jq/
As a shell, the tab completion, parameter completion, long names, makes it easier to discover, easier to understand, and easier to remember.
Then it's a high level language too so if there's something that needs scripting, it doesn't mean a complete change from shell to Python/Ruby/Perl, it stays PowerShell.
Then it's a .Net language too, so if there's something getting a bit big for it, it doesn't mean a complete change to C#/Java instead, it means a small change to PowerShell with .Net methods, then maybe PowerShell with a C# core (like Python with a C module, but still much easier to create).
Of course there's a bit of XKCD "fix having too many things by adding another thing" going on.
But, the fact that it covers the common shell tools with all their different syntaxes reasonably well, and it can be tuned to approach the speed of C# as well, makes it useful for a whole lot of situations, despite having a pretty huge syntax and list of warts, it still seems to come out well.
To my surprise, it was powerful enough to write a somewhat limited implementation of Snake: https://github.com/johshoff/snawk
I like the baked in logic for dealing with record/field formatted data. One thing that seems to hang people up is the lack of a concatenation operator. You just put things next to each other. It looks a little weird. Having associative arrays is a nice plus-up from shell programming.
Awk is a great tool. Glad to see this.
For quick filtering it's really great, e.g. you can parse simple (pretty-printed) XML tags via `-F[<>] '{print $2}'` (for a quick glance on the data - of course not a good idea in production).