Author here. I wrote this to power a distributed tail of all of our logs on websolr.com. This lets us do more advanced monitoring, and we'll start exposing logs and insights from logs to users directly pretty soon. Golang was a bunch of fun for this project, and lets us ship cross-platform binaries with no dependencies, which makes our lives easier on the deploy side.
Most of the other log tailer/shippers are in java, so untenable for RAM-constrained environments. Dendrite also has an emphasis on structuring and re-emitting logs in e.g. json, rather than say, matching specific lines and sending email when you see them, or just forwarding all logs in their original format.
I notice you are using the Go Regexp library for extracting log data. Have you noticed any performance issues with the regexps, or any bump in performance between 1.0 and 1.1? It appears to be one of the parts of Go that doesn't perform very well yet, for example - http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t...
My understanding regarding this benchmark is that Go under-performs because the benchmarks uses a small input. For bigger inputs, Go should outperformed PCRE based Regexp, at least given this little article:
That article, at least in part, explains that the Go Regexp engine handles regexps that would cause pathological behavior in backtracking implementations. From the article:
"Some might argue that this test is unfair to the backtracking implementations, since it focuses on an uncommon corner case. This argument misses the point: given a choice between an implementation with a predictable, consistent, fast running time on all inputs or one that usually runs quickly but can take years of CPU time (or more) on some inputs, the decision should be easy. Also, while examples as dramatic as this one rarely occur in practice, less dramatic ones do occur. Examples include using (.) (.) (.) (.) (.*) to split five space-separated fields, or using alternations where the common cases are not listed first. As a result, programmers often learn which constructs are expensive and avoid them, or they turn to so-called optimizers. Using Thompson's NFA simulation does not require such adaptation: there are no expensive regular expressions."
Since the Go Regexp implementation trades off speed for predictability in performance, I'm wondering if in the real world the performance hit is noticeable.
It depends on if regex performance is a bottleneck in your app. I typically use the standard regexp package unless I noticed its a problem (which isn't often), then I switch to a package with the same API that uses PCRE [1].
Dendrite is actually a port of an unreleased library I wrote in C with pcre. For us, the slowdown (which at our log velocity of hundreds of lines per second per box, still looks like 0% cpu in top), was well worth the productivity we got from moving to Go.
I'm pretty interested, but most concerned about keeping cross-compilation intact for ease-of-install as the community grows. Sooner or later, we'll have the resources to do cross-compilation and pcre. If you want to jump on the google group or github and convince me otherwise, I'm happy to listen.
Golang is the colloquial name given to the language because Go is the most ridiculous and unsearchable name. Try searching HN for "Go" and you'll see the results are garbage.
> Why did you use yaml for configs? Yaml is/can be kind of fragile (whitespace, etc). Do you consider something more practical - nginx config formats?
Yaml is the most common reasonably suitable format for things that look like config files. It's familiar to more developers than nginx config, therefore we hope more people can contribute.
> How to set-up log collector? For example I need to collect logs from several instances. Could not find in cookbook.
> How do you separate logs for different hosts/applications?
Right now, the output map has a couple automatically inserted keys for hostname, shipping timestamp, and the name of the application. This isn't terribly well-documented.
> How do you ship logs? UDP? TCP? TLS?
Yes ;)
> How is this compared vs Logstash?
Dendrite isn't trying to be Logstash. I'd like dendrite to be the agent you use with logstash, greylog, papertrail, or whatever. I think the agnostic nature of dendrite will be a big win, as dev/ops people can persist their data to multiple stores, or swap out stores more easily. The application we wrote Dendrite for currently persists logs to three different stores.
I checked tutorial, nothing there. Since most [semi]serious projects have a least some servers, I suggest to add a separate example, smth like:
1. We have 5 servers. And we collect logs to server 6. 2. These are settings you need to use for 5 servers. 3. And these are settings you need to use for the collector [IP:Port:TCPwithTLS].
> Dendrite scrapes your existing logs
Does Dendrite tail? Can Dendrite consume/scrape some existing/old log files?
BTW, what was your actual max throughput? Did you use some internal queue in the design, like rsyslog is doing?
> Dendrite isn't trying to be Logstash. I'd like dendrite to be the agent you use with logstash, greylog, papertrail, or whatever.
I am confused here a bit. So Dendrite is like StatsD on steroids?
> Does Dendrite tail? Can Dendrite consume/scrape some existing/old log files?
Yes, dendrite tails/follows, with optional backfilling.
> Did you use some internal queue in the design, like rsyslog is doing?
Since you're using existing logs, all we do is maintain a pointer into the logs on disk. When we start consuming more transient data (e.g. perhaps we poll jmx for you), we'll have to add the queue.
> I am confused here a bit. So Dendrite is like StatsD on steroids?
Dendrite is "tail -f *.log | convert_to_json.pl | nc" on steroids.
22 comments
[ 3.3 ms ] story [ 70.0 ms ] threadMost of the other log tailer/shippers are in java, so untenable for RAM-constrained environments. Dendrite also has an emphasis on structuring and re-emitting logs in e.g. json, rather than say, matching specific lines and sending email when you see them, or just forwarding all logs in their original format.
http://swtch.com/~rsc/regexp/regexp1.html
"Some might argue that this test is unfair to the backtracking implementations, since it focuses on an uncommon corner case. This argument misses the point: given a choice between an implementation with a predictable, consistent, fast running time on all inputs or one that usually runs quickly but can take years of CPU time (or more) on some inputs, the decision should be easy. Also, while examples as dramatic as this one rarely occur in practice, less dramatic ones do occur. Examples include using (.) (.) (.) (.) (.*) to split five space-separated fields, or using alternations where the common cases are not listed first. As a result, programmers often learn which constructs are expensive and avoid them, or they turn to so-called optimizers. Using Thompson's NFA simulation does not require such adaptation: there are no expensive regular expressions."
Since the Go Regexp implementation trades off speed for predictability in performance, I'm wondering if in the real world the performance hit is noticeable.
1: https://github.com/glenn-brown/golang-pkg-pcre
Dendrite is actually a port of an unreleased library I wrote in C with pcre. For us, the slowdown (which at our log velocity of hundreds of lines per second per box, still looks like 0% cpu in top), was well worth the productivity we got from moving to Go.
Why did you use yaml for configs? Yaml is/can be kind of fragile (whitespace, etc). Do you consider something more practical - nginx config formats?
How to set-up log collector? For example I need to collect logs from several instances. Could not find in cookbook.
How do you separate logs for different hosts/applications?
How do you ship logs? UDP? TCP? TLS?
How is this compared vs Logstash?
Thank you!
Yaml is the most common reasonably suitable format for things that look like config files. It's familiar to more developers than nginx config, therefore we hope more people can contribute.
> How to set-up log collector? For example I need to collect logs from several instances. Could not find in cookbook.
Did you look at the tutorial? https://github.com/onemorecloud/dendrite/blob/master/tutoria...
> How do you separate logs for different hosts/applications?
Right now, the output map has a couple automatically inserted keys for hostname, shipping timestamp, and the name of the application. This isn't terribly well-documented.
> How do you ship logs? UDP? TCP? TLS?
Yes ;)
> How is this compared vs Logstash?
Dendrite isn't trying to be Logstash. I'd like dendrite to be the agent you use with logstash, greylog, papertrail, or whatever. I think the agnostic nature of dendrite will be a big win, as dev/ops people can persist their data to multiple stores, or swap out stores more easily. The application we wrote Dendrite for currently persists logs to three different stores.
> Yes ;)
ok, missed it, checked again. UDP, TCP - yes. TLS - not yet?
I checked tutorial, nothing there. Since most [semi]serious projects have a least some servers, I suggest to add a separate example, smth like:
1. We have 5 servers. And we collect logs to server 6. 2. These are settings you need to use for 5 servers. 3. And these are settings you need to use for the collector [IP:Port:TCPwithTLS].
> Dendrite scrapes your existing logs
Does Dendrite tail? Can Dendrite consume/scrape some existing/old log files?
BTW, what was your actual max throughput? Did you use some internal queue in the design, like rsyslog is doing?
> Dendrite isn't trying to be Logstash. I'd like dendrite to be the agent you use with logstash, greylog, papertrail, or whatever.
I am confused here a bit. So Dendrite is like StatsD on steroids?
tcp + tls is a one-liner I haven't added
> Does Dendrite tail? Can Dendrite consume/scrape some existing/old log files?
Yes, dendrite tails/follows, with optional backfilling.
> Did you use some internal queue in the design, like rsyslog is doing?
Since you're using existing logs, all we do is maintain a pointer into the logs on disk. When we start consuming more transient data (e.g. perhaps we poll jmx for you), we'll have to add the queue.
> I am confused here a bit. So Dendrite is like StatsD on steroids?
Dendrite is "tail -f *.log | convert_to_json.pl | nc" on steroids.