Ask HN: When do you index your logs?

4 points by redcircle ↗ HN
Do you write your log-worthy events to a file, and then parse them when you need to analyze them (e.g., Splunk); or do you write the structured event to a structured data store, such as a SQL database, which maintains indices in real time? Both approaches can be configured to perform the parsing and indexing on a remote node.

1 comment

[ 3.0 ms ] story [ 69.0 ms ] thread
I do it nightly to a SQL database. It's my morning routine to review the logs, add firewall rules based on various "things" I find, or order new/replace equipment based on what I see. I have a heavily partitioned SQL database and so my load times for one day are very fast (< 3 mins typically).