At what point should you give up writing a sed "program" and just write a few lines of Python to arrive at an answer? Seems he went way past that point.
It was even easier to just count with my finger. :)
The point was a sed challenge. Challenges are fun sometimes.
It was also a good exercise for doing things with sed. There are times when sed is the most terse way to shape text to how you want it. It's also very portable.
Is answering the challenge he answered with python actually easier? I know counting the weeks is easier, but is parsing text like a ninja far less verbose in posix-like tools?
Learning awk made me much more productive, at the very least...
cal -y | sed -E '1,2d;s/(.{20}) /\1\n/g;' | python3 -c “import sys, calendar;[print(ln.strip()) for ln in sys.stdin.readlines() if ln.strip() in calendar.month_name]”
easier to do:
python3 -c “import calendar;[print(m) for m in calendar.month_name]”
I think you've only read the title? :-) The article is about a script that prints the full calendar format, with all the days/dates in between to be able to count some weeks. Not just a list of month names.
Ruby and Perl have awk modes, activated with command-line switches, that are strictly more powerful than awk. I tried using awk recently for a text-processing task and it was a fun learning experience, but I ended up rewriting in Ruby. There's basically no advantage that awk has over Ruby or Perl, aside from portability to embedded systems, and plenty of disadvantages.
Yes, but not Python. azalemeth was wondering if Python could be more terse than sed for this type of work. I think Python tends to be the most verbose among all languages discussed for text handling.
> There's basically no advantage that awk has over Ruby
There are a few. The main advantage of awk over ruby for me is that it's more terse for some simple jobs. That's because of features like
- how unused variables can be used as numbers, strings, or associative arrays without previous declaration. For example, in the common snippet `awk '!a[$0]++'`, `a` is referenced as an array without having to declare it as such first, and the returned value is used as a number (i.e. by `++`) without having to declare that as such.
- how number strings can be implicitly used as numbers
- how records are split by default in a very useful way, with each field made available in a very terse manner with the `$` unary operator.
- how plain regexes can be used as booleans that match over the line.
- how simply passing a boolean expression means to print records that match. It also helps that records can be multi-line or be defined by regexes.
- how ranges can be specified with a couple of boolean expressions
- how files and co-processes can be implicitly opened on first use
It also has advantage in that
- It launches much quicker in comparison to most languages. I recently did a small shell DSL using many piped calls to awk to write some extensions for i3, and performance was decent, surprisingly. jq calls were a far bigger bottleneck.
- It's available on even really old systems. It can be refreshing at times when your only other options are sh, C, or 4GL.
There are many cases where awk is more convenient than ruby.
Another way to make it more challenging would be to do everything in a single `sed` call. Maybe it's for that kind of script that features like `D`'s and `P`'s "up to the first newline" detail become useful.
Wow. I've only ever used sed for simple regex operations, reading this has opened up a whole new world for me.
... I will force myself not to make use of it because you're a bad influence and I already have to stop myself from using too much grep/sed/awk magic, but this is seriously impressive, and an entertaining read. Well done!
I can’t bring myself to write blog posts, but here’s a single-precedence-level infix-to-postfix notation transformer in sed written as part of a solution for last year’s Advent of Code 18a (https://adventofcode.com/2020/day/18):
You are correct; I seemed to remember there was a catch but was too lazy to check, sorry. This isn’t a precedence parser, that was 18b, while this one solves 18a, which was just to resolve parentheses correctly and disregard precedence. (Original comment fixed.)
I've just gotten started! I think it's comfortable in the sense that I won't need to copy and paste stuff to and from the terminal and can just code from the same file and see how the result changes. That's really neat!
I imagine I'll be feeling restricted at some point, since I'm not doing custom HTML for each post, and I want to keep the org file readable on its own, so no excessive markup.
> I like the theme you built.
Thanks, though it's really almost no styling. The syntax highlighting is done by the ruby library `rouge`. The colors were just black on white, but the Dark Reader Firefox extension I have installed would change that to these off-black and off-white colors, and I liked it, so I set them in the CSS.
Thanks! Alright. I haven't looked much into RSS/Atom yet, but I'll try to add it by next week. That should ease my feeling for a need of a schedule. I was just wondering if there were RSS/Atom users that would use it.
cal ← ⎕SH 'cal -y' ⍝ get output of cal command
bymonth ← {⍵⊂⍨0=23|⍳≢⍵}⍤1↑2↓cal ⍝ partition into month groups
lines ← {⍵⊂⍨0=8|⍳≢⍵}⍤1⍉bymonth ⍝ partition into chunks in each line
months ← ~∘' '¨,⍉⊃¨lines ⍝ extract month names and remove spaces
days ← (23⍴1 0 0)⊂⍤1↑⊃,/,⌿2↓¨lines ⍝ partition into days
trimmed ← ∊⍤1⍉{⍵/⍨(' '∨.≠¨⍵)∨∧\' '∧.=¨⍵}⍤1⍉¯1↓days ⍝ remove excess spaces
⎕←53↑(¯2↓⍤1⊢trimmed) , ↑months\⍨∨/' 1 '⍷trimmed ⍝ append months to calendar where the 1s are, and print
Surprisingly tricky (but still easier than sed IMO :P)
It's too bad APL never enjoyed ubiquity on Unix-like platforms. The world would have been a better place. Maybe if Stallman or some early GNU participant had championed APL we'd all be either using APL or constantly excusing ourselves. Like Emacs did for Lisp.
I'm decent with sed. I wouldn't call myself an expert, but know how to employ addressing and hold spaces, which are sadly aracana to most people who employ sed. I ended up learning sed because it was there, proved useful, and it fit into the command-line toolkit really well. (In order of importance.) APL would never slot into that toolkit as well as other tools, but it's value-add would more than compensate. Unfortunately, the value-add isn't enough to compensate for the fact that it's not there and in your face every time you're exploring a problem and its solution space.
AFAIU APL implementations very early on supported ASCII equivalents. But maybe it was too late to either disabuse people of the notion that it required non-ASCII, or the convention proved too strong to overcome.
> (...) Richard Stallman, founder of the GNU Project, was an early adopter of APL, using it to write a text editor as a high school student in the summer of 1969.
And:
Rho, rho, rho of X
Always equals 1
Rho is dimension, rho rho rank.
APL is fun!
Richard M. Stallman, 1969
Yeah agreed, Dyalog APL started out on Unix but is now more focused on Windows, I wish I could integrate it into scripts and things but it's often difficult/impossible to do so.
In the very halcyon days of Unix, lets just say early-80's, I met so many new-to-C guys coming away from APL screaming, with their hair on fire, that I, as an innocent young mangenue, was sure it would be one of those languages that would die a horrid yet rapid death and I could therefore completely ignore it.
Yet, here we are. I'm still amazed to see people using it, and even more amazed to see someone break it out for some toy play.
I attempted to run that APL, and it didn't quite work as is on my system (Dyalog on macOS). As written, it seems to be an odd mixture of things that assume ⎕IO 0 and ⎕IO 1, in that with ⎕IO←1, the bymonth assignment only gets two columns instead of three, but with ⎕IO←0, it needs to be 0=22 and 1↓cal instead of 0=23 and 2↓cal.
Similarly, the lines assignment needed to be {⍵⊂⍨0=9|⍳≢⍵}⍤1⍉bymonth.
After those changes it worked OMM, but the month names were appended directly to the end of the first Saturday's date with no space (e.g. "1 2January"). So I changed ↑months to ↑(' ',¨months) in the last line to insert a space there (there's likely a better way to do that; I'm still pretty new at APL).
I added `-h` to make sure `cal` didn't highlight today's date, which can throw things off, and eliminated intermediate vars that were only used once; this was what I wound up with (works from a clean ws in Dyalog OMM):
I was just testing with TIO, it looks like the version of `cal` on there is different to the common one. The ⎕SE.UCMD'Calendar 2021' version should work cross platform (as it doesn't rely on anything external).
cal -y | sed -E '1,2d;s/(.{20}) /\1\n/g;' | egrep “^. .* $“
or
cal -y | sed -E '1,2d;s/(.{20}) /\1\n/g;' | python3 -c “from calendar import month_name;import sys;[print(ln.strip()) for ln in sys.stdin.readlines() if ln.strip() in month_name]”
Sorry about that. I was typing from my phone, blindly. This is a cool website, thanks. Here is the correct solution:
cal -y | sed -E '1,2d;s/(.{20}) /\1\n/g;' | python3 -c "from calendar import month_name;import sys;[print(ln.strip()) for ln in sys.stdin.readlines() if ln.strip() in month_name and len(ln.strip())]"
That's only a list of names of months, the article creates a single column view with all the dates be between to then count the number of weeks between two dates.
For those who want to never learn sed addressing modes, here's a slight cheat using only simpler shell logic. The cheat is to realize you can concatenate multiple calls to `cal -1` instead of reshaping the 3x4 grid into 12x1, so just focus on the per-month splicing task...
#!/bin/sh
g() {
read m; read n
[ $1 -eq 1 ] \
&& printf '%20.20s %s\n' "$n" "$m" \
|| printf ' %s %s\n' "$n" "$m"
sed -e 's/ *$/ /'
}
f() {
cal -1 $@ | grep '[0-9]' | g $@ | head -c -1
}
y=${1:-`date +%Y`}
for m in {1..12}; do f $m $y; done
printf "\n"
Indeed, there are improvements that could be made. I took the easy way not only by using GNU extensions, but also by calling sed multiple times instead of just once, and I even called awk at the end, though that was just because that bit didn't seem that significant. The spirit of the challenge wasn't exclusive to GNU, though. :)
I was even hoping there'd be comments that show a better solution to learn from them, GNU or not. I got an APL one at least, so that was cool.
> Heres a solution that uses only sh and portable sed. No awk. No wc.
No, it isn't! :)
The challenge was specifically about joining the months of `cal -y` into a single column with sed. It's about the challenge of working with a grid of grids, with the restrictive nature of sed, having only one hold-space to hold 2 months separately. You totally missed doing that with your solution. Of course, it's loads easier to just skip the challenge altogether!
Maybe the point of the sed challenge wasn't clear. It's about stretching the abilities of sed to its limits. Of being able to use all its features to its fullest. I think this is the first time that I had a real use for m~n, and I still haven't found a good use for `D` or `P`.
EDIT: You're using `cal -y` now, but you've still gone outside the challenge space:
> what kind of sed script I’d need in the middle of a `cal -y | ... | wc -l` pipeline to get the same answer
Meaning the working space is just substituting the ellipses for `sed` commands. It's a single `cal -y` call, a single `wc -l` call, and no other shell code except pipes.
I did an awk command in the last bit, but here is the sed equivalent:
sed -E '
/[a-z]$/{h;s/.* //;x}
/[a-z]$/!G
/\b7\b.*July/,/20.*August/!d
s/\n.*//
'
Ideally, it was supposed to be just a single sed command, but I wanted to try out the org-mode blocks for showing the process in multiple stages and also use the m~n addresses, which it seemed like I wouldn't be able to if I did the single call.
Under the heavily populated category of Solutions That Aren't, here's my approach. ^_^
Use ncal because it provides the week numbers and can start weeks on Monday (personal preference). Pipe to sed, fold, and then to Perl. (Perl is awesome.)
ncal -w -3 -y -b -h -M | sed 's/ /*/g' | fold -w 27 |
tail -n 99 | perl -w -e 'use strict;
my (@g1,@g2,@g3,@aAccum)=((),(),(),()); my $tog=1;
sub clearQ{ (@g1,@g2,@g3)=((),(),());}
sub doAccum{ @aAccum=(@g1,@g2,@g3); clearQ();
print @aAccum; @aAccum=(); }
for my $wk(<>){
my $LN=$wk; chomp($LN);
if( length($LN)==0 ){ doAccum(); next;}
if($tog==1){push @g1,$wk;}
elsif($tog==2){push @g2,$wk;}
elsif($tog==3){push @g3,$wk;} $tog++;
if($tog>3){ $tog=1;}
}
doAccum();'
In *buntu 20.04, I have cal and ncal. But yes, any common cal utility that generates week numbers is a better solution than cal -y | ... | wc -l.
I understand your preference for a lean file system. In this challenge, I needed more than awk (because of arrays) and wanted an explicit solution that works regardless of calendar language.
I know this is not in the spirit of the article, but getting the number of weeks is quite useful generally.
One can do this with the standard date util like:
Is there any sources for practice problems like this with sed and other Unix utilities? Like a leetcode, but for solving problems in a Unix environment.
I really enjoy these kinds of issues and see some wizards (like this post!) be able to do some really cool stuff, and I’d like to have it in my tool belt, but I’m not sure where I can get practice problems.
Not entirely sure if you'll find many sed posts there, but answering open/new stack overflow and related stack exchange questions is a great way to find some real-world practice and help others out in the meantime.
72 comments
[ 4.7 ms ] story [ 131 ms ] threadThe point was a sed challenge. Challenges are fun sometimes.
It was also a good exercise for doing things with sed. There are times when sed is the most terse way to shape text to how you want it. It's also very portable.
easier to do:
python3 -c “import calendar;[print(m) for m in calendar.month_name]”
Yes, but not Python. azalemeth was wondering if Python could be more terse than sed for this type of work. I think Python tends to be the most verbose among all languages discussed for text handling.
> There's basically no advantage that awk has over Ruby
There are a few. The main advantage of awk over ruby for me is that it's more terse for some simple jobs. That's because of features like
- how unused variables can be used as numbers, strings, or associative arrays without previous declaration. For example, in the common snippet `awk '!a[$0]++'`, `a` is referenced as an array without having to declare it as such first, and the returned value is used as a number (i.e. by `++`) without having to declare that as such.
- how number strings can be implicitly used as numbers
- how records are split by default in a very useful way, with each field made available in a very terse manner with the `$` unary operator.
- how plain regexes can be used as booleans that match over the line.
- how simply passing a boolean expression means to print records that match. It also helps that records can be multi-line or be defined by regexes.
- how ranges can be specified with a couple of boolean expressions
- how files and co-processes can be implicitly opened on first use
It also has advantage in that
- It launches much quicker in comparison to most languages. I recently did a small shell DSL using many piped calls to awk to write some extensions for i3, and performance was decent, surprisingly. jq calls were a far bigger bottleneck.
- It's available on even really old systems. It can be refreshing at times when your only other options are sh, C, or 4GL.
There are many cases where awk is more convenient than ruby.
I can also empathize with the desire for a single-column calendar output, but my first instinct wasn't to implement it with sed:
https://github.com/jez/calz
Substantially more lines than the sed implementation, but also more feature rich, because I can do things like
etc. Also: pretty colors.Thanks! You can access the org file by changing the URL extension from .html to .org. Maybe I should make that a link.
... I will force myself not to make use of it because you're a bad influence and I already have to stop myself from using too much grep/sed/awk magic, but this is seriously impressive, and an entertaining read. Well done!
UPD: The original version claimed this was a precedence parser. It isn’t, it just does parentheses :(
And accompanying blog post in Russian https://habr.com/ru/post/191006/.
I've just gotten started! I think it's comfortable in the sense that I won't need to copy and paste stuff to and from the terminal and can just code from the same file and see how the result changes. That's really neat!
I imagine I'll be feeling restricted at some point, since I'm not doing custom HTML for each post, and I want to keep the org file readable on its own, so no excessive markup.
> I like the theme you built.
Thanks, though it's really almost no styling. The syntax highlighting is done by the ruby library `rouge`. The colors were just black on white, but the Dark Reader Firefox extension I have installed would change that to these off-black and off-white colors, and I liked it, so I set them in the CSS.
I'm decent with sed. I wouldn't call myself an expert, but know how to employ addressing and hold spaces, which are sadly aracana to most people who employ sed. I ended up learning sed because it was there, proved useful, and it fit into the command-line toolkit really well. (In order of importance.) APL would never slot into that toolkit as well as other tools, but it's value-add would more than compensate. Unfortunately, the value-add isn't enough to compensate for the fact that it's not there and in your face every time you're exploring a problem and its solution space.
I think the major issue is that APL seems to require more than ASCII to work it.
https://en.wikipedia.org/wiki/APL_(programming_language)#GNU...
> (...) Richard Stallman, founder of the GNU Project, was an early adopter of APL, using it to write a text editor as a high school student in the summer of 1969.
And:
https://www.gnu.org/software/apl/But:
> GNU APL was written and is being maintained by Jürgen Sauermann.
and I believe it's fairly young when compared to emacs/emacs lisp.
Yet, here we are. I'm still amazed to see people using it, and even more amazed to see someone break it out for some toy play.
Still, not gonna learn it. I like my hair.
Similarly, the lines assignment needed to be {⍵⊂⍨0=9|⍳≢⍵}⍤1⍉bymonth.
After those changes it worked OMM, but the month names were appended directly to the end of the first Saturday's date with no space (e.g. "1 2January"). So I changed ↑months to ↑(' ',¨months) in the last line to insert a space there (there's likely a better way to do that; I'm still pretty new at APL).
I added `-h` to make sure `cal` didn't highlight today's date, which can throw things off, and eliminated intermediate vars that were only used once; this was what I wound up with (works from a clean ws in Dyalog OMM):
Potentially cal -y on macOS produces a different shape? What's ⍴↑cal?
Try doing cal←⎕SE.UCMD'Calendar 2021' and replace ↑2↓cal with 1↓cal - potentially that'll work?
What system are you on? On Ubuntu, cal -y | wc -l also gives me 36.
or
cal -y | sed -E '1,2d;s/(.{20}) /\1\n/g;' | python3 -c “from calendar import month_name;import sys;[print(ln.strip()) for ln in sys.stdin.readlines() if ln.strip() in month_name]”
cal -y | sed -E '1,2d;s/(.{20}) /\1\n/g;' | python3 -c "from calendar import month_name;import sys;[print(ln.strip()) for ln in sys.stdin.readlines() if ln.strip() in month_name and len(ln.strip())]"
https://tio.run/##TY1BDsIgFAWv8tJNIbGtrUvWnsIag4VaEvohwKZRz4...
#!/bin/sh
This will not work with BSD or Plan9 versions of sed
They do not support using newline characters in a substitution command.
Also, the other seds do not have the GNU "steps" extension, e.g., 2~3. The blog author's solution strategy is 100% reliant on this "steps" feature.
I was even hoping there'd be comments that show a better solution to learn from them, GNU or not. I got an APL one at least, so that was cool.
This should work with GNU sed, BSD sed and Plan9 sed.
No, it isn't! :)
The challenge was specifically about joining the months of `cal -y` into a single column with sed. It's about the challenge of working with a grid of grids, with the restrictive nature of sed, having only one hold-space to hold 2 months separately. You totally missed doing that with your solution. Of course, it's loads easier to just skip the challenge altogether!
Maybe the point of the sed challenge wasn't clear. It's about stretching the abilities of sed to its limits. Of being able to use all its features to its fullest. I think this is the first time that I had a real use for m~n, and I still haven't found a good use for `D` or `P`.
EDIT: You're using `cal -y` now, but you've still gone outside the challenge space:
> what kind of sed script I’d need in the middle of a `cal -y | ... | wc -l` pipeline to get the same answer
Meaning the working space is just substituting the ellipses for `sed` commands. It's a single `cal -y` call, a single `wc -l` call, and no other shell code except pipes.
I did an awk command in the last bit, but here is the sed equivalent:
Ideally, it was supposed to be just a single sed command, but I wanted to try out the org-mode blocks for showing the process in multiple stages and also use the m~n addresses, which it seemed like I wouldn't be able to if I did the single call.111 is the full calendar, the others are individual months.
No dependence on any GNU sed features. Will work with GNU sed, BSD sed and Plan9 sed.
Uses only sed commands and pipes, single cal -y, sed instead of wc.
There's only the problem that because of `-n`, it doesn't output anything.
Use ncal because it provides the week numbers and can start weeks on Monday (personal preference). Pipe to sed, fold, and then to Perl. (Perl is awesome.)
Use the cal that is in util-linux: https://www.kernel.org/pub/linux/utils/util-linux/
cal -m starts weeks on Monday.
cal -w displays week numbers.
Personal preference is not to install Perl unless needed, e.g., for compiling OpenSSL. Takes up precious space that can be used for other things.
I understand your preference for a lean file system. In this challenge, I needed more than awk (because of arrays) and wanted an explicit solution that works regardless of calendar language.
Now, given that this confounded problem has taken too much of my time @^_^@ I'll just say that the simplest option seems to be
ncal -w -3 -y -b -h
because it already computes the week numbers. Fiddle away with sed afterward.
By the way, the "fold" utility [1] can help here too:
ncal -w -3 -y -b -h | sed 's/ /*/g' | fold -w 27
Cheers!
[1] https://en.wikipedia.org/wiki/Fold_(Unix)
I really enjoy these kinds of issues and see some wizards (like this post!) be able to do some really cool stuff, and I’d like to have it in my tool belt, but I’m not sure where I can get practice problems.
Thanks!
LANG=c cal -y|grep '[a-z]'|egrep -v 'Su ' | xargs