Protip: "sed -E 's/[0-9]+/N/g'" is, by itself, what I'd call 'unreasonably effective' at converting log lines into a canonicalized form, such that each line of printf or printf-equivalent source code corresponds to only one or a few distinct output lines. Then you can pipe it into "sort | uniq" (possibly "uniq -c" if you want counts, possibly for logspam-quantification purposes), and otherwise conceivably into a "profile" of a logfile, and then you can diff between a "normal" profile and something that may be less normal.
l = readline()
print l
while True:
n = readline()
if editdistance(n, l) > 30%:
print n
l = n
The simple tool doesn't appear to handle interleaving? Normally uniq handles that by sorting first, but this would need an edit distance aware sort first. I suppose that wouldn't be that hard to build either.
Perhaps it could be just handled with a short window in the tool.
What makes you think it doesn't handle interleaving? They demonstrate a mode where, using the previous day's logs as a baseline, uno can read the current days logs and not print anything that occurred the previous day. That's more sophisticated than you're giving them credit for.
That looked very interesting. But I had so search through the video and zoom in to learn how exactly to separate events that stand out from the baseline (instead of, you know, just reading that command from the webpage).
So, i did all that and downloaded the tool and run that command and found out that the feature is "experimental" and not included in the binaries. It's a bit strange: the tool got my attention, and i was ready to experiment, but i couldn't.
If the reason is that it's a paid or saas only feature, that's totally fine but i think authors should make it a bit more obvious from the web page. Otherwise, i'd suggest to put "experimental" binaries there, too.
Yes, i noticed this after downloading, and even quoted it it my comment.
The feature is called "baseline" everywhere in the text, not "preloading feature", and that's why i got confused.
For example, "As expected, when I start uno with the baseline and don’t do anything with my laptop, it is almost silent since everything that comes in has been seen before."
Open sourcing it is being worked on, so if you wait a bit you won't have to recreate it :). But there was a desire to already put it out there so people can toy with it while it gets ready.
Here's something I'd very much like: syslog(3) and anything like it that takes printf-style format strings and arguments to log the format string and the arguments in a structured way.
E.g.,
syslog(LOG_WARN, "this thing %s didn't go well", thing);
should log something like this, using JSON just for this example to show the actual structure:
[<timestamp>,"<hostname>",<pid>,"<ident>","<facility>",<priority>,"this thing %s didn't go well", "<thing>"]
This would make it much easier to recognize structure in log messages and extract details (syslog(3) format string arguments).
Granted, this would have to have happened 20 years ago for it to be widely available now, but still, gotta start somewhere.
Main issue with syslog (or rather CSVs/TSVs) -- quotes, commas, newlines, backslashes are a bit of a pain when parsing.
Apparently knowing how to use things like grep and awk aren't cool nowadays though. Too many people who grew up on windows. I attempted to install centos earlier, after 18 years of dealing with debian-installer, it amazed me how backwards the installer was. All about the clickity-click.
FWIW, you can get back to utopia by passing in the "text" parameter at the very first menu you see (where the default choice is something like "Check media and install CentOS").
You can configure network interfaces (manually or using DHCP) in "text" mode.
What you call "proper text mode" is referred to as "cmdline" mode. This mode, also known as "noninteractive" mode, is intended for 100% fully automated / "hands off" installations where everything has already has been defined and/or configured ahead of time. If you need or want to configure some configuration option or network interface or whatever manually at install-time, well, you'll want to pick either "graphical" or "text" instead.
(You'll notice that the "luxuries" of text mode -- tmux, ncurses, spacing and alignment of text -- are missing in cmdline mode. If you are able to use cmdline mode, it is only because your installation process has been totally and completely automated, tested extensively, and works flawlessly. If all of that is true, well, you almost certainly aren't staring at the console watching the installation as it takes place and, if no one is watching anyways, why bother spending the effort to make "cmdline" mode anything more than the minimum required -- especially if no one is going to even notice!?)
While CEE isn't developed any more, nothing precludes you from using it for pretty much exactly the purposes you described: http://cee.mitre.org/about/faqs.html#a1
We're doing so at work (using rsyslog forwarding as a transport for logs and metrics), and are very happy with what we have.
This tool shows a lot of promise to me and I've already installed it on our stage box. Thanks!
Some of our log lines contain large json blobs that uno only ignores when I set the difference ratio >50%. Maybe you could add some sort of json detection and parsing identify and ignore similar JSON? Seems tricky but just an idea. Also, the go routine seems to panic on the linux and mac binaries when I run "uno --help".
Tools are needed that turn log analysis into more science than art. We currently use the ELK stack to analyze our logs and it works pretty well but still feels like spelunking.
Do any of you HNers have any logging tools you'd recommend?
> Looking at a sample of the logs it’s obvious that there is both repetition and structure in this data – similarly structured messages repeat with different parameters, dates, numbers, file names, etc. This seems like a great candidate for some automation, let’s see what we can do!
While "uno" is, perhaps, "cooler", logcheck [0] has been around for ages and does just this -- go through your log files, weed out all the stuff that you don't care about, and mails you anything else (which, by definition, is all the stuff you do care about).
Logcheck doesn't use "fuzziness" or uniqueness or any intelligence or anything like that -- just regexes -- but I end up with one single mail per day that only takes me a minute or two to sift through.
└─$ git clone git://git.debian.org/git/logcheck/logcheck.git
Cloning into 'logcheck'...
fatal: unable to connect to git.debian.org:
moszumanska.debian.org[0: 5.153.231.21]: errno=No route to host
moszumanska.debian.org[1: 2001:41c8:1000:21::21:21]: errno=Network is unreachable
32 comments
[ 0.19 ms ] story [ 65.9 ms ] threadl = readline() print l while True: n = readline() if editdistance(n, l) > 30%: print n l = n
The simple tool doesn't appear to handle interleaving? Normally uniq handles that by sorting first, but this would need an edit distance aware sort first. I suppose that wouldn't be that hard to build either.
Perhaps it could be just handled with a short window in the tool.
So, i did all that and downloaded the tool and run that command and found out that the feature is "experimental" and not included in the binaries. It's a bit strange: the tool got my attention, and i was ready to experiment, but i couldn't.
If the reason is that it's a paid or saas only feature, that's totally fine but i think authors should make it a bit more obvious from the web page. Otherwise, i'd suggest to put "experimental" binaries there, too.
The feature is called "baseline" everywhere in the text, not "preloading feature", and that's why i got confused.
For example, "As expected, when I start uno with the baseline and don’t do anything with my laptop, it is almost silent since everything that comes in has been seen before."
Looking forward to the FOSS release.
That’s what Homebrew would do if there was a package anyway.
For a similar tool, I recommend GoAccess [2] and LNAV [3].
[1] https://s3-eu-west-1.amazonaws.com/unomaly/releases/uno_mac/...
[2] brew install goaccess — https://goaccess.io/
[3] brew install lnav — http://lnav.org/
E.g.,
should log something like this, using JSON just for this example to show the actual structure: This would make it much easier to recognize structure in log messages and extract details (syslog(3) format string arguments).Granted, this would have to have happened 20 years ago for it to be widely available now, but still, gotta start somewhere.
Apparently knowing how to use things like grep and awk aren't cool nowadays though. Too many people who grew up on windows. I attempted to install centos earlier, after 18 years of dealing with debian-installer, it amazed me how backwards the installer was. All about the clickity-click.
What you call "proper text mode" is referred to as "cmdline" mode. This mode, also known as "noninteractive" mode, is intended for 100% fully automated / "hands off" installations where everything has already has been defined and/or configured ahead of time. If you need or want to configure some configuration option or network interface or whatever manually at install-time, well, you'll want to pick either "graphical" or "text" instead.
(You'll notice that the "luxuries" of text mode -- tmux, ncurses, spacing and alignment of text -- are missing in cmdline mode. If you are able to use cmdline mode, it is only because your installation process has been totally and completely automated, tested extensively, and works flawlessly. If all of that is true, well, you almost certainly aren't staring at the console watching the installation as it takes place and, if no one is watching anyways, why bother spending the effort to make "cmdline" mode anything more than the minimum required -- especially if no one is going to even notice!?)
We're doing so at work (using rsyslog forwarding as a transport for logs and metrics), and are very happy with what we have.
https://blapid.github.io/cpp/2017/10/31/llcpp-a-quest-for-fa...
Although that writes binary, not JSON.
Some of our log lines contain large json blobs that uno only ignores when I set the difference ratio >50%. Maybe you could add some sort of json detection and parsing identify and ignore similar JSON? Seems tricky but just an idea. Also, the go routine seems to panic on the linux and mac binaries when I run "uno --help".
Tools are needed that turn log analysis into more science than art. We currently use the ELK stack to analyze our logs and it works pretty well but still feels like spelunking.
Do any of you HNers have any logging tools you'd recommend?
[1] http://lnav.org/
[2] https://goaccess.io/
While "uno" is, perhaps, "cooler", logcheck [0] has been around for ages and does just this -- go through your log files, weed out all the stuff that you don't care about, and mails you anything else (which, by definition, is all the stuff you do care about).
Logcheck doesn't use "fuzziness" or uniqueness or any intelligence or anything like that -- just regexes -- but I end up with one single mail per day that only takes me a minute or two to sift through.
Additionally, logcheck is open source.
[0]: http://logcheck.org
I haven't used logcheck, but I assume you can add your own regexes, but looks like you have something to start with.
[0]: https://salsa.debian.org/debian/logcheck/tree/master/rulefil...
[0]: https://salsa.debian.org/debian/logcheck