Parsing this way (in Graylog) is so very... manual. I'd rather go with
rsyslog's liblognorm, either in rsyslog itself or in a separate daemon, like
logdevourer I wrote. The rules then can be written in a regular text editor
and versioned.
Also: what happens if there is a typo in one of the rules and a log entry
doesn't get parsed? Does Graylog/grok have some indication that the log doesn't match? And where the mismatch starts? Because liblognorm does.
I'm not saying that's the best way to do it, just that it's how we did it back then. Of course there might be better ways to do it. I was in charge of exploring this way (which we didn't end up keeping, as I hint at at the end of the post), and thought it would be valuable to share what I learned with people.
Regarding your latter point, as I mentionned, a pattern fails silently if it doesn't match a log entry while parsing incoming entries. That's why Graylog requires you to load a log entry example when creating an extractor, and let you try it while defining the pattern (with the "Try" button next to the pattern input field). I didn't put enough emphasis on this feature in my post, though.
Yep. You might be able to configure Graylog to alert you when an extractor fails, but I didn't go this far (as I said, we switched to another solution since then).
4 comments
[ 2.5 ms ] story [ 18.2 ms ] threadAlso: what happens if there is a typo in one of the rules and a log entry doesn't get parsed? Does Graylog/grok have some indication that the log doesn't match? And where the mismatch starts? Because liblognorm does.
Regarding your latter point, as I mentionned, a pattern fails silently if it doesn't match a log entry while parsing incoming entries. That's why Graylog requires you to load a log entry example when creating an extractor, and let you try it while defining the pattern (with the "Try" button next to the pattern input field). I didn't put enough emphasis on this feature in my post, though.
So it still fails silently when the log format changes slightly, e.g. upstream fixes spelling or grammar in one of the log messages produced.