Ask HN: Best way to store web traffic logs?
I am in the process of setting up a new website, and was hoping to get some advice on how others store web traffic log data.
At the moment I am thinking of just using Google Analytics and Mixpanel.
I have no intention of saving apache access log files - so Im looking for a web based service. I was considering rolling out a custom db table just to log hits, but it just feels wrong.
What are you using? If you are storing data in a db, what is the db (i.e. mongo, postgres, cassandra)?
3 comments
[ 4.5 ms ] story [ 20.7 ms ] threadI understand adhoc research is done via command line, but if you wanted to have a dashboard which shows stats, how would you handle that? I assume you would have to import into a db of some sort to run queries regularly?
Yes we've saved it mainly to look at it later. It's the lowest level of detail so I figure we can mine it later. To do anything useful with it it would have to be processed though I don't think it's that much.
> I understand adhoc research is done via command line, but if you wanted to have a dashboard which shows stats, how would you handle that?
We don't use those files for reporting. We have user stat reports (activity, actions, etc) generated and look at the numbers themselves but nothing I'd consider "pretty". All of those are generated from the audit trail data so it's already in a database. The ad-hoc reporting from the command line is if I want to trace something specific in detail. I've usually filtered it down to small enough set that grep/awk is more than enough and it only takes a couple seconds.