My first concern is what happens to daemons when they can't write to the FUSE mount because this program is down or misbehaving. I believe many daemons would misbehave, block, or die. That scares me but maybe I'm being paranoid.
I think all the arguments against log shippers are pretty weak, workarounds are simple, especially if the alternative introduces any instability.
It needs testing, like anything new, but I think the concept is sound enough to warrant at least a modicum of attention. It's something I've been wanting for a while actually.
To be clear, I do think it's a valid concern -- my point was that "what if log location isn't writable/disk is full" is pretty basic concern that any daemon should account for (what happens if somehow mount -oro /var/log?). And writes failing shouldn't be that much of an alien error condition.
In sum, if fuse/loggerfs together can guarantee that every possible state will result in sane (error) state on writes -- this shouldn't be worse than a disk dying/fs corruption/disk-full type failure.
It will obviously be another point of failure.
On a side note, mounting this under eg /var/log, and then having a strong guarantee that failure will result in an unmount, revealing a writable /var/log seems like the best of both worlds. Would probably have to HUP all writers though...
These are two different cases though. If a disk is full the write fails and software usually handles that well. Depending on the state of FUSE, it will just block indefinitely.
The failure more is very different. On log partition disk full log writes will fail, which you favourite logging library might decide to ignore. A misbehaving FUSE daemon can hang applications that write files and assume it's a local disk.
A smart logger can have a writer thread and a buffer and decide to drop log lines if the logger is too slow. But LoggerFS is meant to be a drop in solution for legacy code, so the concern is perfectly valid.
Yap one could put a watchdog. And then a status reporting and monitoring system to log the log system failures. At some point in that complexity just reverting to rsyslog would seem like a breath of fresh air. Even with its shortcomings.
I have been there, over-designed myself into a hole and then looking back at what was started as a simple 3 step idea now turning into exponentially growing number of branches and corner cases that have to be handled..
Sometimes it is easier to just say "ok this was not a good design" and just throw it away. I have done that and looking back it was a good decision.
It talks about buffering in memory. That could be bad.. I'm assuming/hoping they'd put some kind of upper bound on it, or you could easily take out a server.
Started in 2007, last update in 2013. Not sure if it's the same thing, but the description is similar:
> LoggerFS is a fuse-based virtual file system that allows you to store log files from apache, syslog and more directly in a database instead of a regular file.
This sounds really interesting, but I can't seem to ascertain what it actually does.
> The log data is buffered in-memory (potentially journaled for reliability) and sent over a configurable transport.
What are the options for this 'configurable transport'? What is/are the endpoint(s)? Does LoggerFS have facilities for storing and reading-back logs, or does it rely on other services for this?
The post only seems to /hint/ at answers to these questions.
Backend/Aggregator agnostic (includes multiple log transports)
Supports any Syslog-based log manager
Loggly, Splunk, Logstash, Rsyslog/Syslog-ng
ZeroMQ
NSQ transport – used internally at AutoRef.com
Generic UDP/TCP
And soon: AMQP and Redis (and later: Scribe? Fluentd?)
It deals with the specific problem of collecting the logs from the applications on your servers and shipping them to an aggregator (of which there are already many, e.x. Loggly, Splunk, Logstash)
This article was written in September 2013, just a few days before AutoRef - the company the author was working for at the time - announced they were shutting down [1]. Does anybody know if development continued on the project?
I've spoken with Eric (the engineer who spearheaded this), and while code exists it needs some work for release. It hasn't been worked on since the closure of AutoRef.
Eric and I will be working on getting a release ready within a week. If you'd like to be notified when a release is ready, please sign up for this mailchimp list and we'll email you: http://eepurl.com/WlqXz
A little light on detail. Can someone please explain exactly how this works?
The best I can figure is it's a shipper replacement. So apps write to a log file as usual, but it's actually buffered in memory and pushed to a central server via a FIFO queue.
Given it's all in-memory, it will be small and transient, so you're completely relying on the central server to store it reliably. (Not a criticism, just trying to understand it.)
My first impression is that this is a fantastic and novel development for centralized log analysis.
Basically, it's a virtual filesystem that pretends to create files, but actually intercepts writes to log-style files in this virtual location and buffers those in memory, eventually shipping them off to a place where you can run central analytics on them (ie: Splunk, Logstash, etc).
The application doesn't know that it's writing to a virtual file: it just opens a descriptor, dumps a line or two every few seconds and continues along its merry way. The logs never touch the disk, which means that they don't content for limited disk I/O bandwidth.
Take with a grain of salt, since it's anecdotal, but I have seen logger hang when syslog rotates. Logger is really only 100% reliable when you are piping it the output of a command that runs in a bounded amount of time. If you have a real daemon it's always better to use syslog directly, and an inability to use syslog is maybe an indication of the quality of the daemon in question.
Presuming your daemons rotate their logs, what's the benefit of this over mounting a plain-old tmpfs over your logs folder? Either way, the shipper reads virtual files that are actually stored in memory, and which eventually get purged. It's just the thresholds that are different.
You'd have to be rolling your logs fairly often to avoid filling up tmpfs with log lines that have already been shipped. The advantage of this approach is that log lines that are shipped are no longer on the box at all.
This isn't a way of adding logging to your own software; it's a way of taking any software's logging (yes, even printf) and sending it to a remote server without having to modify the software directly.
So, basically, it's like systemd-journal except not actually like systemd-journal, significantly easier to use, and can use printf directly instead.
Assuming one's software allows for that. Which a lot of software either doesn't do, doesn't do with sufficient configuration, does badly, or does but doesn't support your particular logging method.
As an example, nginx only recently gained the ability to log to syslog; Apache has a logging module but it's not exceptionally customizable if you wanted to log to, say, ActiveMQ, or to a custom service (unless you write a separate binary to accept logs on stdin).
Yup. Given that those logging mechanisms also expose all the problems with file based logging, you have a much more significant effect with a lot less complexity by working on fixing those deficiencies than this route.
As others have commented, in general, it seems simplest and best to use the remote features of rsyslog or journald rather than going through a filesystem layer.
But this looks like a fun project. Is there an advantage to this over using NFS for logs?
This article http://engineering.linkedin.com/distributed-systems/log-what... says that by the time you're done satisfying every requirement of a distributed logging system with multiple writers and readers and various reliability requirements, you've essentially rebuilt Apache Kafka.
Think about the different use cases for logs. I've spent an insane number of hours thinking about it, unfortunately. Live data is useful for graphing or alerting on things. Prehistoric data which is made to be searchable is useful for troubleshooting, or doing forensics work. These use cases require widely varying types of technology to make them scalable, reliable and useful, where 'useful' is defined as saving me time and money.
I LOVE the idea of turning the idea of logging on it's head!
46 comments
[ 2.4 ms ] story [ 107 ms ] threadLog-centric (LoggerFS) is a filesystem around managing log files.
Log-structured is a way of structuring data such that it's written to sequentially and is always appending (while dropping from the head).
Log files are most similar to log-structured, but not a filesystem dedicated to shipping log files for centralization.
I think all the arguments against log shippers are pretty weak, workarounds are simple, especially if the alternative introduces any instability.
In sum, if fuse/loggerfs together can guarantee that every possible state will result in sane (error) state on writes -- this shouldn't be worse than a disk dying/fs corruption/disk-full type failure.
It will obviously be another point of failure.
On a side note, mounting this under eg /var/log, and then having a strong guarantee that failure will result in an unmount, revealing a writable /var/log seems like the best of both worlds. Would probably have to HUP all writers though...
These are two different cases though. If a disk is full the write fails and software usually handles that well. Depending on the state of FUSE, it will just block indefinitely.
A smart logger can have a writer thread and a buffer and decide to drop log lines if the logger is too slow. But LoggerFS is meant to be a drop in solution for legacy code, so the concern is perfectly valid.
I have been there, over-designed myself into a hole and then looking back at what was started as a simple 3 step idea now turning into exponentially growing number of branches and corner cases that have to be handled..
Sometimes it is easier to just say "ok this was not a good design" and just throw it away. I have done that and looking back it was a good decision.
Started in 2007, last update in 2013. Not sure if it's the same thing, but the description is similar:
> LoggerFS is a fuse-based virtual file system that allows you to store log files from apache, syslog and more directly in a database instead of a regular file.
> I actually wrote exactly the same thing years ago (2007), even had the same name :)
[0] https://news.ycombinator.com/item?id=7855060
> The log data is buffered in-memory (potentially journaled for reliability) and sent over a configurable transport.
What are the options for this 'configurable transport'? What is/are the endpoint(s)? Does LoggerFS have facilities for storing and reading-back logs, or does it rely on other services for this?
The post only seems to /hint/ at answers to these questions.
[1] http://www.bizjournals.com/pittsburgh/blog/techflash/2013/09...
I've spoken with Eric (the engineer who spearheaded this), and while code exists it needs some work for release. It hasn't been worked on since the closure of AutoRef.
Eric and I will be working on getting a release ready within a week. If you'd like to be notified when a release is ready, please sign up for this mailchimp list and we'll email you: http://eepurl.com/WlqXz
The best I can figure is it's a shipper replacement. So apps write to a log file as usual, but it's actually buffered in memory and pushed to a central server via a FIFO queue.
Given it's all in-memory, it will be small and transient, so you're completely relying on the central server to store it reliably. (Not a criticism, just trying to understand it.)
Basically, it's a virtual filesystem that pretends to create files, but actually intercepts writes to log-style files in this virtual location and buffers those in memory, eventually shipping them off to a place where you can run central analytics on them (ie: Splunk, Logstash, etc).
The application doesn't know that it's writing to a virtual file: it just opens a descriptor, dumps a line or two every few seconds and continues along its merry way. The logs never touch the disk, which means that they don't content for limited disk I/O bandwidth.
http://sourceforge.net/projects/loggerfs/
I learned about FUSE while getting into Plan 9 and the "everything as a file" mentality. Lets make everything a file!
So, basically, it's like systemd-journal except not actually like systemd-journal, significantly easier to use, and can use printf directly instead.
As an example, nginx only recently gained the ability to log to syslog; Apache has a logging module but it's not exceptionally customizable if you wanted to log to, say, ActiveMQ, or to a custom service (unless you write a separate binary to accept logs on stdin).
But this looks like a fun project. Is there an advantage to this over using NFS for logs?
I LOVE the idea of turning the idea of logging on it's head!