Ask HN: How do you log application events?

Checkout segment.io

Why do you feel the log is way too large?

If log entries take up too much disk space, switching to a different system will not help; you will have to do something with the data. You can either archive old years (export in some way, compress, put in cold storage) or throw them away, either partially or fully (do you need to keep debug logging around forever?). Using partitions can help here, as it makes it faster to drop older data (http://www.postgresql.org/docs/current/interactive/ddl-parti...)

You also may consider compressing some fields inside the database (did you normalize logType and userAgent or are they strings? Can your database compress descriptions?), but that may affect logging performance (that's a _may_. There's extra work to do, but less data to write)

If, on the other hand, indexes take up too much space or querying gets too slow, consider using a partial index (http://en.m.wikipedia.org/wiki/Partial_index). You won't be able to efficiently query older data, but if you do that only rarely, that may be sufficient.

Could you expand on how do you use the log data ? How often do you query it, what time periods do you query, have you considered building a data warehouse for your analytics?

We generally run them through central syslog servers or directly to a logstash tcp or udp input. One way or another all logs from around the world end up in an elasticsearch cluster where we either query for things manually or use kibana to interact with them. Works pretty well actually.

I use a logging library called Winston (https://github.com/winstonjs/winston). I have it hooked up to Pushbullet with Winston-Pushbullet (https://github.com/michaelmcmillan/winston-pushbullet) so that when an unhandled exception or error is thrown I get an instant notification on my Nexus 5 and MacBook.

Winston is a node/iojs library though, but I guess you could find something equivalent in any other stack. The Pushbullet part is really useful.

Edit: I run a pretty small site however (http://littlist.no). I don't think I would enable the Pushbullet part if I had several hundred thousand visitors per day.

We stage stuff out.

After a week, it goes out of cache. After a month, we no longer keep multiple copies around. After 3 months, we gather stats from it, and push it to some tar.xz files, which we store. So its out of the database.

We can still do processing runs over it, and do... but it is no longer indexed, so they take longer.

After 3 years, the files are deleted.

It depends on what type of data you are logging.

For performance metrics we use graphite/stats-d This allows us to log hits/access times for many things, all without state handling code inside the app.

This allows us to get rid of a lot of logs after only a few days. As we're not doing silly things like shipping verbose logs for processing.

However in your usercase this might not be appropriate. As other people have mentioned, truncing the tables and shipping out to cold storage is a good idea if you really need three years of full resolution data.

We push data into shared memory. Then we have clients that can read the memory and present it. This makes it possible to log millions of lines per second with a very limited cost.

This has the benefit of making logging more or less asynchronous. You still need to handle the logs coming out of this, of course.

I run services that log to plaintext files and I use logrotate to periodically gzip and rotate them out for archival.

Just use grep to query recent logs, zgrep if you have to dig a little.

    zgrep somepattern /var/log/messages*

It depends on the priority and importance of events.

For instance, we use a log file for HTTP access logs but I store all of errors and warnings from MongoDB. However, I clean the log storages every month.

We use NodeJS and MongoDB in www.floatalk.com

You could also take a look at Graylog (https://www.graylog.org/), it supports structured data in a variety of formats and can send alerts as well.

It's similar in spirit to elasticsearch + logstash + kibana, but more integrated.

Disclaimer: I work on it, so I'm not going say what's better, just giving another pointer.

I really like Sentry (https://github.com/getsentry/sentry) for exception tracking. It's easy to set up, supports different platforms, and looks great.

I am biased, but you should look into a logging system like splunk. You shouldn't be using an RDBMS for your logs. Your logs don't have a schema.

With splunk, you just output your logs in this format:

<timestamp> key1=value key2=value key3=value

install splunk agent on your machines, and splunk takes care of everything from there. You can search, filter, graph, create alerts etc...

Splunk indexer allows you to age your logs, and keeps the newer ones in hot buckets for fast access.

I just log to a file, rotating/deleting when/if needed

ElasticSearch + Logstash + Kibana.

Custom NLog renderer which implements SysLog protocol and NLog target which pushes logs to RabbitMQ.

Do you really need to debug events from 3 and half years ago? Full logs only really need to stick around as long as you're likely to want to debug them. Log rotation is a must (I've seen debug logs nobody reads sitting in the gigabytes ...) Past that, you can cherry pick and store metadata about the events (e.g. X hits from userAgent Y on this day) with enough information you'll need to do trend analysis, although it's generally a good idea to keep backups of old full logs in case you need to reload the logs to find out that one thing you forgot to add to your metadata ... If you do genuinely need all of the data back that far, you should look at partitioning the data so you're not indexing over millions of rows - how you do that depends how you're intending on using the data.

Rollbar has been pretty fantastic for us.

Also NewRelic if you want to spend the money (or get it throuhg Amazon/Rackspace for free)

I track them via the analytics event tracking API. It is really useful and full of surprises when you look at the stats:

https://news.ycombinator.com/item?id=9444862

We use elmah (https://code.google.com/p/elmah/) for logging our ASP.NET/MVC apps.

It works well for us, nice accessible UI if you need it and a solid database behind it. Also RSS/Email alerts if you need it. We've got thousands of entries in there and even on the old SQL2005 box we use, it seems to work just fine.

Elasticsearch is amazing. It lives up to the hype. It's perfect for rolling over logs, and they have lots of documentation on how to make it work just right.

Just as an example of how awesome Elasticsearch is, you can trivially segment your storage tiers (say, SSD versus HDD) and then easily move older data to other storage, with a single command.

They have a log-specific handler called Logstash, and a dashboard system called Kibana (which is sorta neat but the UI seems a big laggy in my brief experience). Apparently some folks use Logstash/Elasticsearch to record millions and millions of events per day and ES does a great job.

If you want hosted, check out Stackify. I'm totally blown away with the product (no affiliation other than being a new user). You can send log info to them and they'll sort it all out, similar to Splunk, but not ridiculously priced and no dealing with terrible sales teams. But it gets better - they offer all sorts of ways to define app-specific data and metrics, so you can get KPIs and dashboards just adding a line or two of code here and there. It's a lot easier than running your own system, and it looks like it can make ops a ton easier.

Another hosted service is SumoLogic. I only used them for logging, but it seemed to work well enough.

What kind of log data do you mean exactly? E.g. what's the granularity?

We have web server logs going 30 days back, on disk, managed by logrotate. Then we have error logging in Sentry. For user level events, we track in Analytics, but we also have our own database-backed event logging for certain events. Currently this is in the same db as everything else, but we have deliberately factored the tables such that there are no key constraints/joins across these tables and the rest of the schema, which means it should be trivial to shard it out in its own db in time.

Exactly that, but rotating after three years. If it's three years ago it probably doesn't matter any more.

Log analytics is a big topic so I'll hit the main points. The approach you take to logging depends on the analysis you want to do after the log event has been recorded. The value of the logs diminishes rapidly as the age of the events get older. Most places want to keep the logs hot for a period ranging from a day to week. After that,the logs are compressed using gzip or Google snappy compression. Even though they are in a compressed form they should still be searchable.

The most commont logging formats I've come across in production environments are:

1.log4j(java) or nlog(.NET)

2.json

3.syslog

Tools that I've used to search ,visualize and analyse log data have been:

1.Elasticsearch, Logstash and Kibana (ELK) stack

2.splunk (commercial)

3.Logscape (commercial)

Changes to the fields representing your data with the database approach is expensive because you are locked in by the schema. The database schema will never fully represent your full understanding of the data. With the tools I've mentioned above you have the option to extract ad-hoc fields at runtime.

Hope this helps.

    Log.Info("customerNum={customerNumber} transactionId={transactionId} state=completed", cN, tID)

    Log.Info(LogState.Add("state", "completed").Add("customerId", customerId).Add("transactionId", transactionId));

You didn't specify your location, but in some counties like the Netherlands, it's not legal to store PI (personally identifiable) data that long. There is no reason to keep access logs for 3+ years. What are you ever going to do with that data?

Like others here said, extract what you want to keep (unique visitors per day or so) and throw the rest out after a few weeks.

> It makes it trivial for us to debug any event by just querying the db. However, after three and a half years of continued use, the table is now way too large.

Why are you keeping all of the logs? Are you doing anything with it?

Are the old logs relevant at all? If your program structure has changed, then anything logged before that point isn't even applicable.

My advice: If what you is working, but only failed because of volume of data, apply a retention policy and delete data older than some point in time.

An example: Nuke all data older than 1 month for starters, and if you find that you really don't use even that much (perhaps you only need 7 days to provide customer support and debug new releases) then be more aggressive and store less.