Ask HN: How do you analyze logs?

splunk

LogEntries.com

Grep, awk, sed for the easy stuff, logastash/elasticsearch/kibana for the harder stuff.

awk, grep, sed, ag, cut, sort,etc for most of the stuff. Elasticsearch, logstash, kibana for more complex queries

  since is a unix utility similar to tail.
  Unlike tail, since only shows the lines
  appended since the last time. It is useful
  to monitor growing log files.

logstash

Logstash/kibana (free, open source)

Splunk (good, but ridiculously expensive)

When I cared about logs on individual servers, I wrote a program to parse them to make it easy to find what I want, and I wrote it as a command line utility so that it could be used in combo with cat, grep, awk, sed, etc. It's on my github: https://github.com/jedberg/quickparse

logstash + elasticsearch + kibana

That's for personal projects, the CloudFlare setup is a little more complex and perhaps one of the data team would be best answering that... if you're interested then I can ping them to see if there's a volunteer for a blog post describing how we do logs at scale.

fluentd -> elasticsearch/kibana

Works pretty damn well.

www.sumologic.com

As others have said, the classics awk, grep, cut, sort, uniq, and scripting languages (perl/ruby for on the fly one-liners) are great for log analysis. One additional tool I've found particularly useful (three actually) is zgrep/zcat/zless. You'll often be searching through archived gzipped logs, so it's nice being able to work with the files without needing to pipe everything through tar.

Personally i use Emacs with occur-mode for filtering text also when it finds the pattern it is able to switch from instances quickly. Sometimes i also use regex with occur to find multiple items.

Also Notepad++ has a Analyze plugin which i recommend for complex stuff and if you dont like emacs.

Splunk/ELK. I have used Splunk since its inception (I was one of the first paying customers) and enjoy its many features and integrations (e.g. AWS Cloudtrail, Nagios, and anomaly detection). In the past few years, I've come to know and like ELK, ElasticSerch, Kibana, and Logstash. The open source approach has some nice properties as well. It's on you to get the logs ingested but generally that is remote syslog (rsyslog,syslog-ng, or equiv) and logstash with redis. The docs on this integration are not super strong but with a little hacking you can get it working. The feature set is less in this stack and the UI is not nearly as nice. The savings are huge though especially as log volume goes up. Splunk also has a free service targeted at developers called Spluk Storm. Good for proof of concept and easy to setup without any hardware requirements as it runs on AWS.

Used papertrailapp.com but as the number of servers grew, this became expensive.

Moved to EL + Kibana, but not liking the interface yet and it doesn't seem to have 'tail -f' kindof functionality.

I regularly search logs across a huge fleet of hosts (thousands). I have a script that will send an arbitrary command in parallel to each host and return the output in a JSON file.

Once I get the logs I'm interested in, it's usually a straightforward combination of jq, grep, sed, awk, cut, sort, uniq, xargs, etc. If I need to do some fancy queries on the data, I have another script that will parse the logs and load them into a SQLite db.

To the people suggesting ELK i just want to ask if you have actually used it in production? Like for real bughunting and investigating support requests?

As much as we absolutely love ElasticSearch for our other indexing needs, we find it quite hard to get the LK-part of the stack to deliver as promised. Kibana may serve up nice graphs and charts, but when you need to drill down into a large amount of log data, we often feel like loosing both overview _and_ detail.

It might very well be that we are to blame, and that we are just doing it wrong (tm) - but I would love to hear how other people are leveraging the ELK stack in production environments?

grep

I've only maintained a small number of servers, but I've found a good solution in what I'll call the GEL (Graylog2 - Elasticsearch - Logstash) stack. It's been some time since I last used Graylog2, and I can recall that it was somewhat lacking in the pretty charts and graphs department, though that may have improved recently, and the search functioned beautifully.

Did monitoring logs with Bayesian filters ever catch on? They are very good and finding things that are "off"

How many logs are we talking about, and produced at what rate? These are key questions. If you have tons of logs and you might have to make a full access pass over them, the last thing you want to do is to centralize them on some kind of logs host with lots of disks and few CPUs. But if you have little or moderate amounts of logs you may be able to get away with a single host and some xargs -P grep type of thing.

ELK is easy to setup but when you really need to analyse logs it can be painful, I think the real problem is Kibana which is just not good enough for that task.

- no logs coloration

- a lot of bugs / weird refresh behavior

- no auth

One thing we've done that's greatly improved the effectiveness of our API logs is creating a unique context identifier string for each api call and passing it back in the response headers of a call. This allows you to copy the string from the browser and grep the logs to immediately find the call in question, and if an error occurred.

At Mozilla Opsec, we use (and write) MozDef: https://github.com/jeffbryner/MozDef/