HN Explorer

63 comments

[ 5.7 ms ] story [ 136 ms ] thread
daenz
I love the motivation for this tool. This looks very useful for when you just want to make something easier in an existing shell script, without fully migrating to Python. If you do want to migrate fully to Python, I encourage people to use sh[0] (disclaimer: author).

One concern that I have is that it could be so useful that now your script is filled with `| pz`, and is launching Python processes at every line . Now you have a tangled hybrid of Bash and Python.

0. https://github.com/amoffat/sh

yunruse
Dealing with a lot of `| pz` statements could be pretty quickly solved with eg a `pz` interpreter that can be explicitly given multiple sequences of inputs.

That said, it would be amazing if `pz` could take in some input arguments and output the Unixy equivalent if it can find one.

Klasiaster
Or pz could be a lightweight wrapper that speaks to a python server or starts a new one if there isn't yet. The server could end after X seconds of not getting new requests.
LeoPanthera
That is a really confusing name, since /bin/sh is the default shell in almost every unix.
dahart
This sh is a python lib, not a unix shell, so can’t be easily confused.
freedomben
For people that know Ruby, if you haven't explored Ruby's CLI abilities, you definitely, definitely should[1][2]. When I was building my (free and open source) awk course[3] I fell in love with awk. When I later found out that Ruby has some of the same features, it changed my life:

[1]: https://robm.me.uk/2013/11/ruby-enp/

[2]: https://benoithamelin.tumblr.com/ruby1line/

[3]: https://github.com/FreedomBen/awk-hack-the-planet

js2
Perl too. The -p, -n, -e switches work identically.

-n causes Perl to assume the following loop around your program, which makes it iterate over filename arguments somewhat like sed -n or awk:

    LINE:
      while (<>) {
          ...             # your program goes here
      }
-p causes Perl to assume the following loop around your program, which makes it iterate over filename arguments somewhat like sed:

    LINE:
      while (<>) {
          ...             # your program goes here
      } continue {
          print or die "-p destination: $!\n";
      }
https://linux.die.net/man/1/perlrun
p_l
Little underappreciated part of Ruby these days is that part of it grew out of Perl, and specifically supports similar kind of scripting for daily tasks. Pretty sure both thsoe specific switches and some of the sigils used in syntax came from this Perl legacy.
freedomben
Yes thanks, I should have added that perl took it (and improved it) from awk, and ruby took it from perl. Most ruby people don't like to remember that ruby has a lot of perl-isms in it, but I think perl was a great language (and probably is, though I liked ruby so much better I never looked back).
rcthompson
My mnemonic for the perl switches is "perl dash lane", i.e. `perl -lane`. And then I remember to replace "n" with "p" if I want to auto-print every line after munging it. I mainly use this as a replacement for sed/awk/grep because PCRE syntax is more familiar to me than whatever flavor(s) of regexes are used by those tools.
[deleted]
(comment deleted)
nomorecommas
Yep, ruby is a natural, pipeline friendly, command line companion out of the box.

https://github.com/learnbyexample/Command-line-text-processi...

https://learnbyexample.github.io/learn_ruby_oneliners/

asicsp
Thanks for the shout out for my Ruby one-liners ebook. The second link (https://learnbyexample.github.io/learn_ruby_oneliners/) is the latest version.
indigodaddy
Does ruby come standard on most distros (Eg mainly thinking of RHEL-based here), or you’d have to install it usually?
asicsp
Don't know about RHEL and I'm not aware of Linux distros that come with Ruby pre-installed. Perl/Python are usually available by default.
CopperWing
Suse/openSuse comes with Ruby pre-installed because Yast, their distinctive system manager tool, is written in Ruby.
freedomben
Usually not, although it's one of the top/main packages supported by Red Hat. `dnf install ruby` will work everywhere.
joseph8th
And then there's `xonsh`, "The unholy union of bash and Python."[0]

Which I've been using as my daily driver for a few years, and absolutely swear by it.

https://xon.sh

seanw444
What in the world... that's certainly interesting.
c0npr
agreed, but if I am learning a new syntax anyways I will go for ansible.
ginko
Care to explain? Xonsh is a python shell. From a cursory look at the very corporate Ansible website and wikipedia it's

"an open-source software provisioning, configuration management, and application-deployment tool enabling infrastructure as code"

Whatever that means. They seem to do very different things as far as I can tell.

joseph8th
No new syntax, unless you want to write scripts in xonsh, or mix bash and Python in the same command. There's rarely any reason to. 99/100 times I just write pure bash or pure python
pedrovhb
I've been daily driving xonsh for a couple of years too. It _sounds_ like a terrible idea for reasons you can't quite put your finger on, but other than a minor annoyance every now and then, it's been great.

I especially like iterating over files as Path objects with the simple syntax

    for f in p`.\*\.jpg`:
        if f.isdir():
            foo = $(echo @(f.absolute()))
suifbwish
What sorcery is this?! Object oriented bash? Incredible.
abnry
PowerShell has entered the chat.
meanmrmustard92
Powershell doesn't simply enter the chat. Powershell enters-its-username-and-password-and-hits-enter the chat.

[I'm usually a fan of informative command names, but surely there's a happy medium]

joseph8th
Really it's just a souped-up Python interactive shell. When you give it bash commands, it forks to bash. When you give it python it evaluates like a REPL
jamespwilliams
This seems very useful!

The behaviour of auto-imports https://github.com/CZ-NIC/pz#auto-import seems bit scary:

> Caveat: When accessed first time, the auto-import makes the row reprocessed. It may influence your global variables. Use verbose output to see if something has been auto-imported.

It sounds like this could cause some very subtle bugs. Maybe a strong and clear warning when a line is being reprocessed would help.

hauntsaninja
I wrote a similar tool a while back. My solution for this was to statically analyse the input to discover unused names. The cool part of doing transformations statically is that you're able to see the exact Python code you'd run without running it, e.g. `pyp --explain x[:5]`. https://github.com/hauntsaninja/pyp
visarga
Nice, I made a similar tool for Perl. I chain them up to three rows in terminal, if they get longer I switch to a script.
uniformlyrandom
I don't know if I like this approach. Common "regular" Bash programs have a huge advantage - they are common. When I share my 'may accidentally conjure up a monster' one-liners with my team, I am sure they can read it, understand it, and run it. I lose that advantage with the use of a very nice, but obscure tool.
ogogmad
Bash involves tons and tons of arcania. It also doesn't support features that PL editors have like Intellisense. I sometimes feel like the only people who don't see a problem with the Unix experience are people who've used it day-in day-out for a decade, and don't need to grep man pages more than they'd like to. But by then, it's Stockholm Syndrome? I think this Github project might begin to pull out the chord. I know my Python better than I know my awk/sed/bash/xargs/find/make/bc...
suifbwish
Bash has one major advantage though: it’s core libraries and and commands almost never change which mean your scripts are most likely going to work forever. Python core library development is filled with asshats who like to deprecate naming conventions in later versions rather than just improving the existing libraries and keeping the method names the same. Python is great for a lot of reasons but if you know bash well, it is far more reliable in the long term.
dotancohen
I'm 44 years old. When I read your comment, I think "Great, the Bash script will continue to work the next time I need it (which might be after Elon lands humans on another planet)". But when my 22 year old colleague reads it, he likely thinks "Great, that script is full of incomprehensible syntax from the era of my last diaper change".

Thinking long term is a skill that is often realized only after one fails to make a previous long-term investment.

fivea
> Python core library development is filled with asshats who like to deprecate naming conventions in later versions rather than just improving the existing libraries and keeping the method names the same.

I think this is a gross misrepresentation of both technologies, to the point that it goes well beyond a normal apples-to-oranges comparison.

A shell script runs well on any standard-compliant implementation, just like python3.4 scripts will run exactly like they always did when running on a python3.4 interpreter.

Python's libraries break when asshats "deprecate naming conventions" only when you somehow assume that, say, your python3.10 script would run on python2.7, or just like your shell script breaks when the program it calls happens to break it's cli.

Also, is versioning still a problem given the pervasiveness of package managers with support for version pinning?

tyingq
Most of the examples would be relatively simple Perl or AWK one-liners. The appeal of this tool seems mostly to be that you can use Python, which isn't one-line friendly out of the box.
brtknr
Probably easier to stick to bash rather than learn a new set of commands from scratch. Also bash commands are universally available whereas this needs to be installed first which is not always that straightforward. Also python is slower at some of the examples than compiled executables.
fivea
> Probably easier to stick to bash rather than the overhead of learning new set of commands that I need to learn from scratch.

That's all fine and dandy until your shell script grows beyond a dozen lines of code or so. The moment that happens, shell scripts become barely readable and terribly hard to maintain, unlike alternatives such as python.

MobiusHorizons
I have to say I have not found this to be true. Bash is an excellent glue language if what you want to do is fundamentally a shell like activity (eg composing together other more powerful primitives). I think the key (which is often forgotten) is just to remember to use the abstractions provided by the language (also shell-check, which is the most helpful linter I have ever encountered)

If you use functions to aid in abstraction, you can easily convert the function to a program in a more capable language as soon as it becomes too complicated.

In the end, I usually find that most of the things people replace good old unix tools with are better in one limited area, but miss a large part of what make the originals magic. Usually this is some combination of universal, discoverable, reduced dependencies, composable abstractions.

drran
Refactor your terrible and hard to maintain script into a well looking and maintainable script. It's not a bash fault.

My bash-modules[0] project[1] may help you with that. Typically, I perform script refactoring in these steps:

1. Split code into few subroutines with clear boundaries between them, or split a large script into multiple simpler scripts.

2. Add an error handler with transparent message to every line of code.

3. Add command line options to override hard-coded things, to be able to test things in isolation.

4. Write documentation (man-page) for script, for future self.

5. Write test cases.

6. Refactor code, following best practices.

[0]: http://vlisivka.github.io/bash-modules/

[1]: https://github.com/vlisivka/bash-modules

otikik
“Universal” is too much of a stretch. You still get tripped by things like “that flag doesn’t exist in OSX sed” or “that option in git was only available in git > 2.8.14”

And that’s how you end up with bash scripts wrapped inside Docker images, that I have been seeing for the last couple of years.

baq
Suddenly xonsh from a sibling thread doesn’t sound outrageous anymore
Naac
This looks like an unmaintained version of (the much more complete) pyp:

https://github.com/hauntsaninja/pyp

ur-whale
pyp is very cool, especially the --explain flag

Didn't know about it, thanks for this.

Oh, and small gotcha, if you want to try it, its:

    pip3 install pypyp
rather than the obvious which installs something else entirely.
aasasd
Ah, yes, the Perlton.
gpvos
That's just perl -pe or -ne.
McHankHenry
I just skip the bash etc. alltogether and write scripts entirely in python, and put the most commonly used stuff in a utility module.
j_not_j
What I would like to see (and haven't found yet), is a bash or python shell that automatically does a "more" when a command exceeds some number of lines.

So many times I accidently fill up my scroll buffer with an unintended awk or grep mistake.

throwaway984393
pz versus Perl:

  $ echo -e "example\nwikipedia" | pz 's += ".com"'
  $ echo -e "example\nwikipedia" | perl -lpe '$_.=".com"'
  
  $ echo "hello world" | pz s[6:]
  $ echo "hello world" | perl -pe '$_=substr($_,6)'
  $ echo "hello world" | perl -pe 'substr($_,0,6)=""'
  $ echo "hello world" | perl -lape '$_=$F[1]'
  
  $ tail -f /var/log/syslog | pz -f '{len(s)}: {s}'
  $ tail -f /var/log/syslog | perl -lpe '$_=length($_).": $_"'
  
  $ echo "HELLO" | pz s.lower
  $ echo "HELLO" | perl -pe '$_=lc'
  
  $ echo "hello,5" | pz 's.split(",")[1]' | pz n+7
  $ echo "hello,5" | perl -F, -lpe '$_=$F[1]+7'
  
  $ pz --findall "(https?://[^\s]+)" < file.log
  $ perl -lne 'print $1 if m{(https?://\S+)}' foo2.log
  
  $ echo -e "1\n2\n3\n4" | pz --end sum
  $ echo -e "1\n2\n3\n4" | perl -lne 'END{print $t} $t+=$_'
  
  $ echo -e "1\n2\n2\n1\n3" | pz --end "sorted(set(lines))"
  $ echo -e "1\n2\n2\n1\n3" | perl -ne 'END{print sort keys %h} $h{$_}++'
  
  $ echo -e "red green\nblue red green" | pz 'C.update(s.split())' --end C.most_common
  $ echo -e "red green\nblue red green" | perl -lae 'END{print "$_, $h{$_}" for sort keys %h} $h{$_}++ for @F'
randoglando
For people familiar with python, the first one is immediately understandable, and the second one is incomprehensible. Learning Perl may have been a better choice long-term, but this might be a good stop-gap fix for those who know Python already.
est
> Example: appending '.com' to every line.

It was so much easier in CMD.exe

mv * *.com

35mm
As someone who considers themselves ‘not a real programmer’ but uses Python, shell scripts, etc to ‘just get things done’, I really love simple utilities like this.

Maybe it’s not the ‘right’ way to get things done, but it’s quick and it works.