Understanding Awk (earthly.dev)

The funny thing is, by and large my only use case for awk is to print out whitespace delimited columns where the amount of whitespace is variable. Surprisingly hard to do with other Unix tools.

Neat discussions around that sort of thing at least here: https://news.ycombinator.com/item?id=23427479

   echo -e "foo   bar   baz" | choose -1 -2

   echo -e "foo   bar   baz" | awk '{ print $2, $3}'

  $ awk -F: '{len=length($1);if(len>max){max=len;user=$1}}END{print user,max}' /etc/passwd

  ls -l | tr -s ' ' | cut -d ' ' -f 5

  $ printf " one two  three"  | tr -s ' ' | cut -d ' ' -f 1

  $ printf " one two  three"  | awk '{print $1}'
  one

  ps ax | sed 's/^\s\+//; s/\s\+/ /g;' | cut -d ' ' -f 4

  echo -e '1\t2\t3\t4\t5' | expand -t 1 | cut -d ' ' -f 3

  $ printf "a b  c d   e\n1 2  3 4 5" | perl -lanE 'say "$F[2] $F[4]"'
  c e
  3 5

    ruby -n -a -e 'puts "#{$F[0] $F[1]}"'

    ruby -ne '
    BEGIN { $words = Hash.new(0) }

    $_.split(/[^a-zA-Z]+/).each { |word| 
    $words[word.downcase] += 1 }

    END {
        ...

I really appreciate you writing this guide. As a long time Linux user, I've always wanted to learn AWK, but it seemed too daunting. Three minutes into your guide and I immediately saw how I could use it in my day-to-day usage.

    $ man gawk | wc
       1568   13030   94207
    $ man -l /usr/share/man/man1/awk.1plan9.gz | wc 
        214    1579   10956

As someone who's never used awk before, I really enjoyed this write-up and I think it was very well written!

Yes! Another recent thread recently discussed best practice and whether something like that exist. I believe this is a good example.

For some purposes, awk+xargs can replace hours of work to write a tool to automate some process. It's my go-to for ops work that I don't expect to live very long and just needs to _happen_.

Also, happy 1337 karma day :).

I've never bothered to learn much AWK, but that's mostly because Perl is my bread and butter language and has been for 20 years, and focusing on knowledge of that seemed a better investment (especially since with a few judicious flags, Perl is a passable AWK replacement even for very small one liners).

That said, if you just want to supplement your knowledge of other shell tools and pull out something that can do some obvious text munging, AWK has always looked attractive for the task to me.

Came here to say this. Glad to see /bin getting respect.

To anyone processing huge quantities of text and text files, someone very likely had the same problem you faced back in the 1980's and there's a Unix/GNU tool for it already.

Not having to parse the output at all is even better. I really like the way Powershell can pass structured data like this.

I'm a huge Linux/Unix fan but sometimes a rethink really works out. I hope Linux will get something similar. I know Powershell is available for Linux but without an adapted userland there's not much benefit

I do a lot of work with structured data--json, yaml, etc. For me, this is how I feel about jq. One of my favorite use-cases is querying Kubernetes resources. E.g., `kubectl get secret <secret-name> -o json | jq -r '.data | map_values(@base64d)'` (fetch a secret and decode all of its values).

Yep awk is lovely and well worth the time to learn.

This is probably not important for embedded but doesn’t a pipeline of small scripts (which could be in awk) give you better threading support?

Xargs, GNU parallel or even make then scale that out really quickly.

I have used it for many thing such as extract bibtex entries and checking for missed definition of abbreviations, converting remind entry to ICS format etc. Recent example is to print a calendar like cal but with calendar dates highlighted.

This isn't your question but if your files are CSV, there is a CSV extension for gawk

    @include "csv"
    BEGIN { CSVMODE = 1 }

Tab separated values all the things

The original language specification, written by the authors, is now free online. Chapter 2 covers the whole language in a little over 40 pages.

https://archive.org/download/pdfy-MgN0H1joIoDVoIC7/The_AWK_P...

Severely underrated comment.

Having relied heavily on the (unofficial, non-GNU) gawk manpage extensively (it's quite good), I instantly started learning very useful features reading the GNU docs. (I still need to fully internalise those). Yes, the full manual is very much better than the manpage.

(Also recommend The AWK Programming Language mentioned here, though I'd suggest the GNU manual adds to that as well.)

shebangs and more complex scripts are covered in the article.

But yes, I agree that the original AWK book is really good. After covering some basics and the language reference, it has some fun projects that you can build with AWK.

I use awk for one-liners, no more.

Looking at my command history, I mostly use awk to extract a field like this:

   <something> | awk '{print $3}'

(I know "cut" is supposed to do the same thing, but it was never reliable for me - maybe tabs/spaces?)

    $ cat /bin/awkmail
    #!/bin/gawk -f

    BEGIN { smtp="/inet/tcp/0/smtp.yourco.com/25";
    ORS="\r\n"; r=ARGV[1]; s=ARGV[2]; sbj=ARGV[3]; # /bin/awkmail to from subj < in

    print "helo " ENVIRON["HOSTNAME"]        |& smtp;
    smtp |& getline j; print j
    print "mail from: " s                    |& smtp;  smtp |& getline j; print j
    if(match(r, ","))
    {
     split(r, z, ",")
     for(y in z) { print "rcpt to: " z[y]    |& smtp;  smtp |& getline j; print j }
    }
    else { print "rcpt to: " r               |& smtp;  smtp |& getline j; print j }
    print "data"                             |& smtp;  smtp |& getline j; print j

    print "From: " s                         |& smtp;  ARGV[2] = ""   # not a file
    print "To: " r                           |& smtp;  ARGV[1] = ""   # not a file
    if(length(sbj)) { print "Subject: " sbj  |& smtp;  ARGV[3] = "" } # not a file
    print ""                                 |& smtp

    while(getline > 0) print                 |& smtp

    print "."                                |& smtp;  smtp |& getline j; print j
    print "quit"                             |& smtp;  smtp |& getline j; print j

    close(smtp) } # /inet/protocol/local-port/remote-host/remote-port

    #!/usr/bin/python
    import smtplib
    from email.mime.text import MIMEText

    msg = 'hi'
    subj='read this!'

    smtp_server='mail.example.com'
    smtp_from='me@example.com'
    smtp_to='you@example.com'

    m = MIMEText(msg)

    m['To'] = smtp_to
    m['From'] = smtp_from
    m['Subject'] = subj

    s = smtplib.SMTP(smtp_server)
    s.sendmail(smtp_from, [smtp_to], m.as_string())
    s.quit()

    with open(arg.logfile) as f:
        log_contents = f.read()

    m = MIMEText(log_contents)

    from email.mime.image import MIMEImage
    from email.mime.text import MIMEText
    from email.mime.multipart import MIMEMultipart

    m = MIMEMultipart()
    m.attach(MIMEText('\n\n%s\n\n'%xkcd_img_title))
    m.attach(MIMEImage(xkcd_img))

I take it you LOVE ada :-)

There is a lot of wisdom in the things you avoid, however I would ask one question, "How often do you use it?"

For me, the best systems are those that can be wordy and prescriptive but as you get to know them you can use more short hand so they "get out of the way" as it were. A good example of that philosophy is keyboard short cuts. When I'm learning a program I'm happy to pause and sling the mouse around to find the thing I need in the labeled menu stack with an appropriate name which also tells me what the keyboard short cut is for that thing. Then as I get better I can just use the short cut and my workflow gets faster. Once I've internalized the keymap my flow is held up by how fast I can think, not by how fast I can take my hand off the keyboard, move the mouse, click and then put it back on the keyboard.

Awk is one of those things that once you internalize what it can do, you can use it for a lot of stuff, and you can do it quickly.

The thing that prevents awk from being a major part of my daily routine is that it (amazingly) has poor CSV support. Consider the following:

col1,col2,col3

1,2,3

4,"hello, \"world\"",6

"7 buckets",,9

To get the usual awk experience with this very common file format, exactly the type of thing you want to parse with awk, you first need to install gawk, then use a big FPAT regex that needs to be adjusted for any new CSV variant.

I would love to see awk with "CSV mode", where it intelligently handles formats like this if you just pass a flag. I think awk would do well to differentiate itself with excellent 2d dataset parsing functionality, but at least catchup up to the average scripting language would be great.

I'm half expecting someone to say "just pass -csv it does what you want" and if so I'll be very excited.

    @include "csv"
    BEGIN { CSVMODE = 1 }

    awk -f ./ucsv.awk -e '{print $5}'

  cat textfile.csv | csvquote | awk -f myprogram.awk | csvquote -u > output.csv

I've been doing systems work for 20 years. Here's why most of those things are actually good:

- Strings are subtly complex, but strings are not variables. You can assign a string, and later handle it as a variable, and not deal with any of the specifics of string-iness. Likewise, you can take a variable, and later treat it as a string (for loosely or not-typed variables).

- Magic switches are not magic, they are options. Virtually every program takes options. Sometimes they impact a lot of things, sometimes a little. Only the context determines how much is "too much".

- Terse/cryptic variables allow you to write complex expressions in a compact form. This allows you to read more in a small space, making it easier to reason about or form complex expressions. Human languages are flush with these, as is mathematics. But you have to balance the terse, cryptic and magical with guilelessness, or it becomes a mess.

- Terse and magical syntax is, again, a feature, not a bug. Using magical syntax I can do in a few characters what would take me many lines with a traditional language, and as we all know, increased number of lines correlates to bugs, in addition to simply making it harder to grok.

- Types aren't ignored, but they may be very loosely enforced. If you want to write a quick program to get something done, typing is a curse. If you want to write a very thorough program, typing is a blessing. In many cases, loosely or untyped programs actually work better than their typed cousins, because they allow for more unexpected behaviors without failing. Failing early and often may be a modern trend, but... it literally means things fail more, and this is often not desirable.

Caveats:

- Programs that are hard to read do indeed suck, and it takes lots of experience to make some kinds of programs easier to read. But that's not an indictment of the program, it's an indictment of the person who wrote it. We don't indict English when somebody writes a document that's impossible to comprehend.

- Interestingly, some of the more popular languages are the worst to debug. Perl is probably one of the easiest languages to debug, not inconsequently because of how good the interpreter is at suggesting to the user what the actual problem was and almost exactly how to fix it.

Yep, anything more than 1 line would be a python script. The author vaguely tries to answer that but doesn't really give anything more satisfying than "you can rewrite it in python" and "why not?".

And if it's something you don't know and are experimenting with, you can also open ipython and play around with the data (kinda like he's doing throughout the article) and keep your variables between each run without having to write intermediate values to disk or keep piping over and over.

Let alone the huge stdlib and 3rd party lib you have access to if needed.

In my day job I use Java, and all the items you list are good ideas to maintain my sanity.

But I sometimes use Awk scripts to quickly analyse a log file to get information you need or to generate a configuration file or something.

I think of the strict strongly-typed nature of a language like Java as equivalent to the formal language a lawyer might use in a legal document: There shouldn't be any place for ambiguity or misunderstanding.

Programming in Awk, on the other hand, is more like having an informal conversation with an old friend; those cryptic variable names is like the slang words that both of you understand and the magical syntax is like the dialect you share that may sound a bit strange to outsiders.

I think there is a place for both approaches in the broad space of computing.

And for the flash card type of learners it is good to see the "HANDY ONE-LINE SCRIPTS FOR AWK" page is still available. See the links in the Credits section at the bottom for more great reading:

https://www.pement.org/awk/awk1line.txt

That author also edited the "USEFUL ONE-LINE SCRIPTS FOR SED" page:

https://www.pement.org/sed/sed1line.txt

That happened to me. AWK -> Perl -> Ruby.

If that is true it could be that people are tired of the packaging/dependency hell of "other languages".

Portability.

The /usr/bin/env trick will work on a wide range of systems, in which even common utilities might have numerous locations: /bin, /sbin, /usr/bin, /usr/bin/local, /opt, or others. If you're writing scripts for portability and ohers, this has value.

That said, /usr/bin/env fails on Android/Termux AFAIU.

Nice tip, so basically like excel with tables

I've also used this to address columns by name for files with lots of columns that I'm too lazy to count: https://unix.stackexchange.com/a/359699

You can also put a similar code block at the start of a general processing entry. This applies on both flat (uniform record) and hierarchical (multiple record-type) data.

E.g.:

  {
     name = $1
     dob = $2
     grade = $3
     # ...

     # Do stuff with name / dob / grade, etc.
  }

If the data are structured, so that there are multiple record types (typically defined by prefix or some other regex) you can put variable assignments within each block.

   /^rectype1/ { var1 = $1; var2 = $2, ... }
   /^rectype2/ { varA = $1; varB = $2, ... }

I prefer to leave BEGIN blocks for defining constants or tables and such.

If you want to do that, use vnlog instead. You're 90% there already. It formalizes the idea of a header to label columns, and allows normal unix data-processing tools to just work with those labels. Your awk use case is quite literally "vnl-filter --eval"

https://github.com/dkogan/vnlog/

Is the exec needed in the pipeline?

       x(){ tr \\11 \\40;}
       x|cut -f "$1"|x

       x(){ tr \\40 \\11;}
       y(){ tr \\11 \\40;}
       x|cut -f "$1"|y