HN Explorer

Ask HN: Log analysis tools for 1-2 MB logfiles?

4 points by victorhooi ↗ HN
I'm looking for a tool for small-scale log analysis - the log files are fairly small, most are around 1-2 MB.

For large-scale log analysis, I'm used to things like Graylog2, Kibana, Grafana etc.

However, these all seem like overkill for this scale.

A simple local tool, with either a interactive command-line interface, or that produced a self-contianed webpage output would be nice.

Lnav is the only one I can find so far, which kind of does it - but are there any others?

If we were to try to roll our own, are there any web UI toolkits that would make designing something like Kibana or Graylog2 interface-wise easier?

7 comments

[ 4.3 ms ] story [ 28.8 ms ] thread
viraptor
What do you expect from analysis exactly? Awk is always a good tool if you want to do some simple stream processing. Or do you mean something for looking at the actual stream?
victorhooi
They are pretty structured logfiles - so we want to be able to split on those (e.g. by timestamp, by log-level, by file).

Part of the application handles two-way syncing of files - so we want to say, trace the events as they come in from each direction, over the lifetime of a synced file.

epalmer
It depends on what you want to do and how structured your log files are. My web servers write json records for each log event and that makes analysis sweet and easy. I have other log files that are just text seached through the records.

Splunk is free for up to 500mb of import a day. It has a web interface and I use it to analyze web log files and other log files. I used the free one for a while and then went to paid because of my daily imports were occasionally >500MB.

I like it for my needs.

epalmer
There is also grep and awk which I use for small log files as well. I've written python scripts for very specific time order dependency analysis. Basically throw away python.
victorhooi
That's a good point - I could probably write some Python to parse the logfiles, and convert it into a more structured form. One major use case - the logfiles contains records about files and events associated with those files - we want to trace each file over the span of the logfile.

However, at the other end, I need a way to present it, and allow users to filter it, or select objects to trace - basically, if I could replicate something like Splunk or Kibana, either at the console or via a WebUI - I didn't know if there are UI frameworks/components out there that provided similar controls.

tstack
Where does lnav fall short in this case? As the author of it, I'd like to know so that I can give you some tips or try to fill in the gaps.