HN Explorer

Ask HN: Real-time log viewing

2 points by 87 ↗ HN
The powerful solutions seem to focus on post-hoc log analysis. The thing I'm working on produces a lot of logs. Logs that I like to look at when debugging. Viewing them is a bit overwhelming.

I'd like a viewer that: -filters by JSON properties

-formats JSON logs into human readable messages

-can fold large messages

-lets you save sets of filters

-lets you simultaneously view the same set of messages using different filters

Right now I settled for splitting logs into different files based on filters. Which is okayish but doesn't work well for teammates. And even at mediocre volumes Emacs will choke.

4 comments

[ 2.4 ms ] story [ 21.2 ms ] thread
java-man
- are these files local or remote?

- do you need tail?

- could you provide a link to a typical log file (with sensitive info removed of course)?

87
They're local, the logging logic is completely under my control, I'm fine with piping it to wherever as long as it does what I want.

For example the simplest JSON message would be:

  {"timestamp":"2018-06- 29T17:25:47.857Z",
  "module":"main",
  "msg":"[\"trying to reconnect\"]",
  "level":5}
while the human-readable representation for the same message is:

  global 17:06:067 trying to reconnect
So I'll have one main log file that gets everything in JSON, but that's only useful for post-hoc analysis. I can control what makes it to stdout at runtime but that's just a single view. So right now I've hard-coded the logging to spit out human-readable messages to different files based on the module and level properties. I open the files in auto-revert-tail-mode (emacs).

Not sure what you mean do I need tail? If you mean the tail program then yea that's what I effectively do at the moment. If you mean always seeing the latest logs as they come in - yes.

So the ideal software would just read that one file (or db) that has everything in JSON (or machine readable format) and let you have different "views" over the same data. Packaged in a nice (web?) GUI.

It shouldn't be too hard to implement but I'm hoping there's something out there already.

java-man
There is probably something out already -

https://softwarerecs.stackexchange.com/questions/37881/log-v...

https://github.com/gistia/json-log-viewer

https://github.com/brocode/fblog

I am currently writing a log viewer / terminal combo for internal use. Handling milti-line json with tailing is a bit of a challenge though.

87
Oh I put the newlines on HN for readability, the actual logs don't have that. There are numerous tools that handle json logs on the cli, for example there's jq so you could

  tail -F my.log | jq <filter>
This is okay, but changing filters isn't trivial, running the commands and displaying side-by-side isn't trivial etc etc.

So say you can develop a decent emacs workflow using that, but once you get a new dev you cannot really pass it on. And if they're not much of a cli person then yea...

Thanks for gistia/json-log-viewer I'll check it out, though it doesn't seem to solve that much it's still interesting.

There's some nice Kibana stuff out there, but it seems single-view-at-a-time focused too