62 comments

[ 0.21 ms ] story [ 119 ms ] thread
So it is like the edge trigger functionality of a Digital Storage Oscilloscope for pipes? I can think of more than a handful of times this would have become handy but also seems like a very unlikely tool that I find myself reaching, especially due to the complicated usage.
But in actual fact, it is very easy to use. You might not get that from the video because it's inspired by Vi: commands are not echoed, other than colon commands and search patterns. So in the video you don't always know what is going on: did the display change because of a command you don't see or was it spontaneous? That is all painfully obvious if you're in the driver's seat.
> especially due to the complicated usage.

Said he while using find, dd, rsync, tar and others every day. But in all seriousness new cmd tools should be given a chance, it sometimes feels like people believe posix tools were perfected in the 80s and further development makes no sense.

I don’t want to watch the video on my phone, so would you please give some practical examples of what this might be used for? I read the description and I don’t know what it’s for.
I've only watched the video, so I don't know a lot about the tool. But from what I can tell, it's a really neat way to view text as it's streaming through a pipe, _and_ it keeps a line at the bottom, for you to be able to type a line, which when you press return, sends your text to the STDIN of the program that is streaming the text.

This can relieve a pain point if you ever need to work on a serial port, and as you are typing in a command, the other side sends text, which overwrites the characters you were typing (depending on how the local echo is configured). It reminds me of a line based serial terminal emulator, where the input is a separate text box, and the output is a larger box above it.

Oh, and it also seems to have methods of sampling the data as it comes in, so that it can be configured to only show the messages you are looking for, and also counting them?

The source file is also tiny: https://www.kylheku.com/cgit/pw/tree/pw.c

Very Unix

I don't like the landing page, it lists all the niceties of the program and details how it does what it does, but it doesn't say what it does.

Why do I need this? What problem does it solve for me? I know it samples pipes, I don't know why I would want something that does.

It looks like it’s ostensibly a way of being able to dynamically peak into stuff that’s being passed through a pipe.

A bit like if ‘pv’ (https://linux.die.net/man/1/pv) and a pager (eg ‘less’/‘more’/etc) had a child.

I’ve typically solved this kind of problem with ‘tee’ but this looks a much cleaner approach.

It could be a very useful tool for any Sysadmins who still find they need to occasionally SSH into a server for real time log parsing (which is a dying habit these days). Or for DevOps and Developers needing to debug.

Ah, thank you. My main question now is whether it decides what to show based on what it's already showing (ie deduplicates somehow) or is just straight-up sampling.
I wasn’t entirely sure of that myself but it does look as if you can filter results manually
Another way to simulate the behavior of this tool, if I'm understanding you correctly, is with software flow control, usually mapped as ctrl-s for XOFF and ctrl-q for XON. That lets you pause output while you look at it, and then resume once you're done.
You cannot add filters et al with them though. I guess this is comparable to the scroll back history on a terminal multiplexer
XOFF doesn't discard input; if you pause for too long, things will backlog, which could be a problem if we are reading directly from some program (rather than, say, reading the output of something disposable and disconnected from the source, like tail -f).

When pw isn't updating the display, it's still streaming and discarding the data, which is a big semantic difference. It means that the source of the data will not block, for one thing. But if you need to see all the data or search through it, then pw is the wrong tool, at least by itself. You need to make other arrangements to capture the complete data.

>(which is a dying habit these days).

Hmmm, I don't know what that says about me. Checking logs is 2nd nature to me. I guess that makes me old if the "kids these days" are doing something else, but what is the something else? Just not looking at the data and spinning wheels debugging? New tools available? I'll admit I've never looked for anything other than the location of a log of a service I'm investigating.

We use a log aggregator and watch logs from hundreds of servers in a searchable & filterable interface like Kibana. We also use Loki and Elastic extensively. If you have money to spend, I think you'd be looking at Splunk or Datadog.

As a matter of fact, we do not have SSH access to our servers. Everything happens over a web interface that has been secured.

Usually you'd aggregate those logs to a centralised location and have some kind of visibility platform to parse them like Splunk, ELK, DataDog, etc. That same pipeline would also typically share at least part of the process for streaming your logs to a tamper proof archive.

So it's not that checking logs is a dying habit, it's process of SSHing onto the server and running shell commands to query them.

Though I'm not suggesting the latter is bad practice. In some domains it's suboptimal but in other domains it is a pretty reasonable response to fault diagnosis. IT is a broad field :)

>IT is a broad field :)

Yes it is. Today's assumption seems to be that everything is being done "web scale" with hundreds of servers in multiple datacenters across the globe.

I'm not so fortunate/unfortunate (depending on one's viewpoint) of having to deal with that kind of scale. I have less than 10 servers I maintain so a lot of these new fancy viewers is overkill for me.

I'm not going to judge how you do your job because I clearly don't know anything about the problems you're tasked to solve. However I will disagree that visibility is a web scale problem. I've still found having a centralised visibility platform useful even when managing a very small number of servers. After all, you need to set up monitoring and (presumably) alerts when something fails anyway. So it's just another service on your Nagios (for example) server.

Plus if you have any kind of auditing and compliance where you need to retain tamper proof logs for any period of time then you've already done the other half the work by streaming those logs off the servers.

Even if you're not webscale, modern tooling is just so much better for visualizing website stats. Even if you have only one server, how do you make visualizations from the log file? You don't have to use something webscale and enterprise-grade like Splunk. I do recommend finding a modern tool that you like though. Something better than `awk`-ing a log file into CSV, loading that into a spreadsheet, and then fiddling with it to get a graph.
> I have less than 10 servers I maintain so a lot of these new fancy viewers is overkill for me.

There are several layers to the complexity of "web scale orchestration" however, and you definitely can get the same tooling/features on a much smaller scale with less complexity involved.

I.e. you can rent a few baremetal servers from hetzner, give them an internal lan interface and setup k3s on that internal network. Now you can add capabilities as you want and need them: I.e.

  * deploying longhorn for a distributed storage backend with easily configured backup.

  * Certmanager for cluster wide https certificate provisioning 

  * Promstack to get healthchecks etc

  * Loki for the just now mentioned logging aggregation
Each of these capabilities are usually just one deployment command (kubetcl apply/helm install) and will enable this capability for all existing and future deployments on the same cluster.

If you have a weekend to experiment and want to explore what it actually feels to use that kind of tooling I can only encourage you to maybe try it out. The whole experience has gotten pretty nice over the years, and while it is pretty much the same you already know... Just differently, it does solve it in a very reusable way, which makes it ultimately (at least now after about 1 year of usage) much more straightforward to figure out then just a bunch of shell scripts

After I got over that it's done wrong, (as in: not what I've come to know/expect) I kinda got to like it, especially for small scale stuff (a 4 node HA cluster with 64gb ram each costs slightly below 200€ per month on hetzner, and they're using nvme storage, so really fast IO)

Not everything you can SSH into that produces logs is necessarily a "server". Small embedded systems may not have the RAM or flash space to aggregate locally, and setting them up for remote logging may be problematic or at least inconvenient for one reason or another.
> Not everything you can SSH into that produces logs is necessarily a "server".

I never suggested that was the case. I just don’t consider then in scope for this topic.

I have managed numerous such devices over my career. In fact most of my home networking gear would fall into that category of “produces logs and SSH but not a server” but none of which runs anything UNIX-like (so you couldn’t install this tool even if you wanted to). However they do support a bunch of different tooling for pushing logs and metrics over the network. Which then brings me back to my earlier point about log collection.

Also if we take your example of embedded devices with low memory and storage, then installing additional utilities like this one when you can already do the same thing (albeit less conveniently) with busybox probably isn’t a good idea. So they’d be out of scope for this thread too.

I’m not going to say that there’s no such thing as a non-server platform that produces logs that this tool would be useful for (I don’t believe in absolutes!) however it is reasonable to assume that servers and desktops, ie more classical computing devices, are a vastly larger demographic for this tool. Hence why I focused my examples around that.

(comment deleted)
As an embedded developer I have different view. Linux is everywhere. I've been working within nothing but embedded Linux constantly since 2005.

I regularly install debugging tools onto target devices (strace, tcpdump, ...) even though those don't go into a production image.

Currently, I'm using pw on an embedded system, for capturing detailed traces from an application. It's very useful because it decouples the rate of the traces from the serial port, while still letting me get the info I need out of it.

The stripped executable is tiny: 20 kilobytes. It's so small I'm stashing it in a configuration flash partition which survives re-flashes of the kernel and filesystem, so I have the utility there no matter which image I load onto the target.

The alternatives these days is examining the combined logs in an aggregator like Splunk or ELK.
PSA: don't use linux.die.net. It's horribly outdated.

This particular man page is from 2012 or earlier.

That doesn't really matter in this instance, I just wanted some example of what pv was as it's a lesser known utility and that page happened to be the top non-spammy link in DDG.
> for real time log parsing

Or other things!

  tcdump -i <if> -l | pw    # -l for line buffering
Now you have an interactive network monitor that, in some ways, has better ergonomics than Wireshark. Want IPV6 only?

  :g IP6 
done. Okay, only packets from bob to alice:

  :g bob.*>.*alice
Only UDP:

  :g UDP
Forget UDP, how about http:

  :r
  :g http
Kind of thing.
Wanted to add websocketd [1] to your list. It's an amazing tool to stream debugging logs to another system where you can build your webapps that accumulate alerts.

Use it only for debugging builds and not for production builds (obviously).

[1] https://github.com/joewalnes/websocketd

I thought so too, but then I watched the demo video. Looks useful when you have a lot of data scrolling by but are interested in only a portion of it. Could be used to watch web server log files, strace output or anything else like that.

https://vimeo.com/705257869

Came here to say this. There's a very good reason that every man page ever written starts like so:

  NAME
         man - format and display the on-line manual pages

  SYNOPSIS
         man  [-acdfFhkKtwW] [--path] [-m system] [-p string] [-C config_file] [-M pathlist]
         [-P pager] [-B browser] [-H htmlpager] [-S section_list] [section] name ...
I agree. No idea what this was even for until I watched the demo video.
Description from man page:

       pw  stands for Pipe Watch. This is a utility which continuously reads textual input from a
       pipe or pipe-like source, and maintains a dynamic display of  the  most  recently  read  N
       lines.

       If  pw  is invoked such that its standard input is a TTY, it simply reads lines and prints
       them in its characteristic way, with control characters replaced  by  caret  codes,  until
       end-of-file is encountered. Long lines aren't clipped, and there is no interactive mode.

       The intended use of pw is that its standard input is a pipe, such as the output of another
       command, or a pipe-like device such as a socket or whatever.  In this  situation,  pw  ex-
       pects  to  be  executed in a TTY session in which a /dev/tty device can be opened, for the
       purposes of obtaining interactive input.  The remaining description pertains to  this  in-
       teractive mode.

       In  interactive mode, pw simultaneously monitors its standard input for the arrival of new
       data, as well as the TTY for interactive commands.  Lines from standard input  are  placed
       into  a  FIFO  buffer.  While the FIFO buffer is not yet full, lines are displayed immedi-
       ately.  After the FIFO buffer fills up with the specified number of lines  (controlled  by
       the  -n  option)  then  pw transitions into a mode in which, old lines are bumped from the
       tail of the FIFO as new ones are added to the head, and refresh operations are required in
       order to display the current FIFO contents.

       The display only refreshes with the latest FIFO data when

       1.     there is some keyboard activity from the terminal; or

       2.     when the interval period has expired without new input having been seen; or else

       3.     whenever the long period elapses.

       In  other words, while the pipe is spewing, and there is no keyboard input, the display is
       updated infrequently, only according to the long interval.

       The display is also updated when standard input indicates end-of-data.  In this situation,
       pw  terminates, unless the -d (do not quit) option has been specified, in which case it pw
       stays in interactive mode. The the end-of-data status is indicated by the string EOF being
       displayed in the status line after the data.
How is this different to tail -f ?
It tries to not flood your terminal when you're not interacting with it. Could be good on a slow SSH link, for instance.
The "triggers" seem to be the difference. The video shows it tailing strace, and only waking up the display and dumping data when it sees the poll() system call in the strace output.

Then they show a fancier example of a specific pattern of recvmsg(11,...) happening a specific amount of lines below poll(). That ends up showing the flow of data to a specific socket.

Or without triggers, it only shows you the latest when you tap the space bar or some other key.

pw has no tail -f functionality; it doesn't monitor whether a file that has reached EOF will start growing again. If you do pw < file, it will just read to the end and then exit. Or else stay running with pw -d, but without reading any more data. But it can make sense to do this:

  tail -f logfile | pw
the main behavior change that pw will introduce is that during periods when the logfile is rapidly growing, it will prevent spewage to the terminal. It will only update when there is a lull in in the output exceeding the short timeout (default: 1 s), or else once every long timeout (default: 10 s). (These values were hard coded at first, then they became command line options, then dynamically adjustable too).

The development of pw was inspired by a discussion in the GNU Coreutils mailing list by a user wanting behavior along these lines from tail itself.

Then the other feature ideas came.

For instance pausing: we can suspend the display indefinitely. With tail -f, that's easy: you can just kill it. You can't kill a program that's taking input from a pipe without breaking the pipe, though. You can pause the terminal output, but not indefinitely without causing a backlog that could block the producing application, and when you resume, all the backlogged spewage will show.

I like the tool a lot. I think people are still generally a bit confused about what it is, and the beginning of the man page is quite mechanistic, describing the implementation and leaving it up to the reader to interpret why that is useful.

The reasons I like it are that it's "an interactive streaming text viewer", along the lines of a readonly text editor designed for presenting inhuman amounts of data in a way that's useful for us humans. Think of the times you want to use your favorite text editor to interactively search and inspect streaming data, but can't because the stream is continuous.

- doesn't stream characters to the screen so fast that humans can't read it

- allows for interactively searching the stream and showing context

- operates on a large ring buffer, so all of this can be done without ever attempting to read a fixed subset or the entire stream

It reminds me of what Casey Muratori did for refterm: https://youtu.be/hNZF81VYfQo

Is this like pipe view?

This was used as an example on monitoring a MySQL restore. The command was:

  pv sqlfile.sql | mysql -uxxx -pxxxx dbname
It seems that pv copies the file to standard output, MySQL then consumes this. The normal command is

  mysql -u user -p pass < sqlfile.sql
But this shows no progress output.
No. pv shows a progress bar when data is moving through a pipe. This samples data as it moves through a pipe.

The difference being, pv is more like a fancy version of cat (and more, but that’s the primary use case). This samples data that is being piped around, so you can see the actual data, not just the progress.

It's pretty difficult to figure out what this even does or what the benefit is.
This is super handy. I've often found myself 'tail'-ing logs to 'grep' to filter out stuff; pw makes that process 10 times better. Very cool.
Can it do the following? I have a dev server which is spitting lots of logs and I want to interactively search the logs with something like `fzf`. No changes can be made to the dev server.
Tail the logs into fzf --no-sort --tac
I keep this alias in my zshrc for exactly this purpose

  alias lzf="fzf +s --tac --bind 'enter:select-all+accept' -m"
With the added bonus that you can press return to release the filtered lines to stdout. Really useful for interactively filtering logs.
No. pw works strictly with a small window of live data. It doesn't assume that the file descriptor being read is a seekable file. It doesn't hoard all the data for searching.

If there is some dynamic pattern in the logs that repeats (or that you can cause to repeat by executing some test steps on the software which produces the logs), you may be able to use pw to latch on to that pattern.

E.g. suppose there is detailed function enter/leave tracing in the log. You can discover that certain functions are traced when you do something, and then filter for that material to remove unrelated spewage, and perhaps set up a trigger to capture the scenario you are reproducing.

Once you know that, you can use other tools to find the same thing. You know what to look for in off-line analysis of a complete log.

This is not just for logs. If you pipe

  tcpdump -i <eth> -l | pw
you basically get an interactive network monitoring utility.

E.g. add:

  :g thishost.*>.*thathost
and now you're just seeing packets from thishost to thathost.
But suppose you're debugging some scenario (e.g. particular kind of request that generates some logs according to its access/interaction pattern). You can drill into this: interactively determine what things to filter for, trigger the display snapshot/update on particular multi-line patterns and such. You can have it so that whenever you trigger the use case, the new instance of the log appears in the display and freezes, and contains just the relevant information.
This seems like a less-interactive `lnav`, somewhere between `lnav` and `tail -f`, which is great for me to see, as I think this problem-space may be under-researched.
I'd love to have a tool to pipe into that removes low-entropy content automatically, perhaps with machine learning.

For example, last week I was grepping through some security logs looking for anomalous logins. There were a bunch of other security events in the log file. Eventually my command looked like `cat /var/log/auth.log | grep -v Disconnected | grep -v cron:session | grep -v 'Invalid user'` and so forth until I only had the "interesting" lines left.

Maybe the right approach to this tool would be to put the lines into a tree based on the longest substrings.

grep takes multiple arguments so if you want fewer processes then:

  grep -v -e Disconnected -e cron:session -e "invalid user" /var/log/auth.log
would also suffice (or you can use -E which would enable the usual "Disconnected|invalid|cron").
After watching the demo video for PW, I think it is better than what you were doing, but doesn't have the automation you want.
It may be a good way to discover what patterns are worth looking for, and then encoding that into a grep. You need grep if you want to keep everything that matches; no skipping. But the cycle of repeatedly killing "tail -f | grep ..." and then trying different patterns is a little less ergonomic than doing it inside one program. If the input is coming from a pipe directly, trying a different set of grep patterns means killing and restarting the program.
If I understand what it's doing correctly, this reminds me of the trigger function of an oscilloscope, which "enables repetitive waveforms to be displayed on the screen in steady fashion."

https://www.electronics-notes.com/articles/test-methods/osci...

Yes. I thought about changing the name to Pipe Scope after the trigger feature was added, but then people would bug me to make it dual trace, and extend the bandwidth to 100 MHz or more and such.
This looks same as the older "watch" command that I use once in a while.

From the man page. "watch - execute a program periodically, showing output fullscreen".

I use it as "watch -5 -d <command>". "-5" is to run the "command" every 5 seconds and "-d" is to highlight the difference from the previous output.

The "Watch" part of the name is inspired by the watch command. Something is being periodically redisplayed, namely the contents of a FIFO buffer of lines of text.
Next piece of development;

- pw currently misbehaves if you Ctrl-Z suspend it and then try "bg". It doesn't handle the SIGTTOU signal telling it that it's not allowed to send to the TTY due to being in the background. (It won't die, but you may have to suspend it again and do "fg").

- the ideal behavior would be: if we Ctrl-Z, bg, the program runs and just reads the pipe in the background.

With that functionality, we can take any program that spews logs onto standard output and just throw it into the background indefinitely without having it block.

   $ spewing-program | pw &
   [1] 123
   $ bg
   [1]+ spewing-program | pw &
To see the latest bit of logging from the program, we foreground it.

A program which produces detailed logging that cannot be turned off or adjusted dynamically will benefit from this.

I would recommend adding a license as well.
Thanks for pointing that out! I could have sworn I had put a BSD copyright block at the top, but now I see it's not there.

I did that to a different program and misremembered.

No, scratch that; I did that in the man page source code, but forgot the C source file.

[Issue fixed now.]

I'm tired of unix interfaces. Really. It's great to learn a few like vim and grep. But every single tool requiring you to remember a bunch of commands is just too much.

Great this tool looks cool. The UI turns me off. Just like almost every Unix UI. It's not about how good the UI is either. It's about the fact that every new tool requires reading MAN pages.

More tools need to be like nano because I simply don't have time to read the MAN for every single thing.

By the way, in the video nothing is initially happening after some traces are shown. Why is that? It's not that the strace isn't producing any new output, but rather because strace is spewing rapidly, and the default timeout is one second. Therefore, the pipe never times out, which means that the time-out-driven refresh isn't seen. Only the 10 second long interval refreshes the display automatically.

Before setting setting the trigger, I unpaused the display. Every input event from the TTY also refreshes the display in free-running mode. So as I'm typing /poll[Enter], there is display activity with each key.

Then when the trigger is set, things start going due to the frequent triggering.

Things might have been less confusing at the start of the video if I had shortened the long timeout.

that is a terrible page. it leads with a while bunch of stuff no one cares about, and then goes on ti completely fail to convey what the tool actually goes.