Ask HN: How do you manage logs for your backend services?
I have been working on a few side projects that usually have Go backend services. Right now, they log straight to a file and it works just fine since I only have a couple of instances running.
If I were to scale this to more instances/servers, I would need a centralized logging service. I've considered ELK but it's fairly expensive to both self-host or buy a managed subscription.
Any advice?
141 comments
[ 4.2 ms ] story [ 197 ms ] threadIf you log everything to a single server, as you noticed yourself, it will become very expensive and difficult to filter out the stuff that you really need when looking for errors or when you try to hunt down a specific problem.
We work with a ton of log tools as company bringing graph analytics+vis to investogations largely log & event data, and I've increasingly shifted to that when I need easy basics.
I could not find an equivalent to Kibana though :-(
I've only been using it for my homelab, and haven't even moved everything to it yet - but I like it so far. I already use Grafana+influxdb for metrics so having logs in the same interface is nice.
https://grafana.com/oss/loki/
On a 1core 2gb vps it can do around 1200 logs/sec.
Compared to ELK, we're saving several hundred $ per month.
You can do 1200 logs/sec with a container limited to 1 core and 256mb of memory running elasticsearch.
This is extremely doable for some workloads, but not others. Really depends on what you're stuffing in.
Trying to do much customization was kind of painful, the config file structure is kind of confusing and the docs were sparse and differ depending on the plugin you're using, and there's no validation of the config so if you have any of the arguments slightly wrong it'll fail silently.
The modularity of routing and filtering logs seems like it would be great, but it turns out not to be all that flexible, you really have to follow the framework's preconceived idea of how the processing pipeline should work. I forget the details but we were trying to do some processing on the logs, like ensuring they're valid json and adding a few fields to them based on other metadata source, and it would have required writing our own plugin. In other ways it felt too modular for its own good, like it's up to individual plugins to implement anything they need and the core doesn't seem to provide you with much. Even things like an error handling or retry framework are not built in, so if a plugin throws an unexpected exception it's not handled gracefully. One plugin we were using would just stop sending logs until we restarted the fluentd process if it ever got an http failure from being temporarily unable to connect to the api, because they hadn't built in their own retry/recovering mechanism. Granted this was for a pretty new logging SAAS service we were trying out, if you're sticking with the basics like files and syslog it's probably more robust, but at that point you also might not need the flexibility promised by a tool like fluentd.
I would budget some time after setting it up to weed out uselessly verbose logging and rotating old logs out of RAM and onto cheap storage. You'll love it.
You can spin it up on a single machine with ES and start playing with it. I usually forward all of my logs to rsyslog, then that duplicates the logs out - they go to flat file storage, and to graylog for analysis.
Is it just ad-hoc text "debug" logs? Structured transaction or request logs to feed some sort of analytics? This impacts the backend trade-offs quite a bit.
Are you trying to do monitoring via logs? Don't. Export metrics via Prometheus (or something like it), much cheaper and dependable then extracting metrics via a log collection system.
But when it's not having an outage, it's lightning fast and easy to use. I especially like that you can do basically 100% of the configuration (aside from stuff like billing info) by uploading JSON files to them, so you can keep all your parsing and alerting configuration in your own version control system.
I do constantly wonder if there's a logging/searching solution small and lightweight enough to fit on the cheapest DO droplet. $7/gb/mo on Papertrail vs $5/mo with a 25GB SSD? That'd be a no-brainer, but ELK and Graylog won't run with that low of hardware.
I've been using Google App Engine for some ten years now and I'm still dumbfounded that this is still an ongoing struggle for other platforms. It collates logs from a variety of sources, presents requests as a single unit, has sophisticated searching capabilities, and the UI doesn't suck.
Best of all, it just works... there's zero configuration on GAE. Such a time saver.
The process would be: log data -> add index filters -> go to live tail and create a metric on a filtered log event -> create monitor on metric.
edit: also, you log 24 billion messages a month? I think that's what it would be to cost $30k for their platform per month
10k qps is certainly not a dev/test instance, but if you were to log all RPCs including database lookups, you could easily get there with even just 100-ish user-facing requests per second.
Also it's worth noting that not all of the logs are necessarily our application logs but could also be audit logs of third party services we use (Okta, GSuite, AWS, etc.) to detect anomalies or potential breach attempts. We have a pretty comprehensive alerting pipeline based on our logs so we're unfortunately unable to pay that much for logging. I understand that this doesn't apply to everyone but we're able to run a self-hosted logging pipeline for a fraction of that cost without a dedicated team running it (the Infrastructure team, the team I'm on, currently maintains this pipeline along our other responsibilities).
We used to run an elk stack but hit a bottleneck crunching logs with logstash. We found flume's morphline to be performant enough and the nice property of flume is that you can fanout and write to multiple datasources.
It's ironic, but because Athena is kind of a flaky product (lots of weird Hive Exceptions querying lots of data) and because it's really only good at searching if you know what you're looking for, we're considering having flume write to an elasticsearch cluster (but still persisting logs to long-term storage on s3).
Tweak your system so you can selectively turn on log collection, and collect useful metrics 24/7. Logs almost never matter until you're diagnosing an active problem, anyway, and then only a few logs will be necessary. Differentiating the type and quality of particular log messages is also very useful.
- Exist a tool that allow to navigate structured logs easily, without bring a heavy machinery like ELF stack? and work in terminal?
- That also filter only ERRORS and few lines above/below?
What's the volume of logs?
What's the expected usage? (debugging? audit?)
Are you using text, or structured logs? (or can you switch to structured easily?)
How far back do you want to keep the logs? How far back do you want to query them easily?
Are you in an environment with an existing simple solution? (GCP, AWS)
Running your own log infrastructure is the absolute worst. As in, we had to put a staff devops engineer on just that for 2 months the last time the company I was at needed an upgrade.
If it's big, Graylog is great.
If it's too big, /dev/null, best logs gathering since 1971.