27 comments

[ 0.24 ms ] story [ 28.6 ms ] thread
"Splunk that runs locally with a worse UX, but on balance doesn't eat all the resources you throw at it and doesn't charge you a kidney."
There's far cheaper alternatives to Splunk that are still a step up from traditional syslog.

KubeCon over the last couple of years was showing the market was a glut with Observability vendors which is just time series and log management. (traces are logs with a span id).

We moved to splunk now and mostly happy with it, mix of windows/linux/etc logs.

But with AI, I can see opensource alternatives getting better.

I've never been in the sales room. I've heard from multiple corps that Splunk is quite expensive. I've been to one place that ran its own Elastic stack, which was a bit of cost in its own right because they had to run all the infrastructure and storage themselves, and had two FTEs whose main job was keeping the thing running. I don't know what Splunk offers from a UI/features perspective that Kibana couldn't do.
I agree (as the owner of an elastic stack at work). A second FTE sounds like a luxury though..
That's indeed pretty rare IME :)
> A syslog server is a foundational tool for centralized log management in modern IT environments

I'd very much recommend a more modern log stack than a traditional syslog server.

There are many articles covering the limitations of Syslog. Better to replace that component by utilities like OpenTelemetry and JSON structured logging.

You can run a single binary version of Loki https://grafana.com/docs/loki/latest/get-started/deployment-...

Or use something like the Otel collector to send your logs to a remote host.

I have done my fair share of rsyslog and syslog-ng.

I would not say a "Syslog" server belongs in a modern stack.

Yeah, for a modern large scale distributed system both the client api and implementations are pretty bad.
The problem with modern stuff is it doesn't do the very basics. Sometimes I really do want UDP dumping out into a file on another part of the network. The modern setups forget how to do this.
I saw one place that had the logs going into a database. On the same connection as the app-data. So, when the transaction failed, the logs also didn't get written. LMAO. I made them do syslog in their code, which for some of the devs was a mind-blowing. They were amazed at that we could just barf text quick&lightweight over UDP.
Disagree. JSON logging is fine, but make it line based and just log to a syslog server. Then I can route it wherever I want (or to multiple places) including to a simple file on a disk that I can actually inspect, rather than having to use an API.
Yes, but if you want pus truly wide events (e.g. >64KiB), then it migth push syslog implementation from happy/well-throded path.
I've been down this road many times over the years, and each new promised land of logging always fails to replace syslog for me. Nothing else is as widely-compatible and performant. Fortunately, there are some great tools out there to modernize the network-admin experience: syslog-ng is a favorite of mine.
Same boat here. Go as simple as possible (syslog) as the base, and you can bolt on whatever the hell other tools you’d like to perform advanced functionality if you really want it.
> I'd very much recommend a more modern log stack than a traditional syslog server.

Once everything is sent to the syslog server it can be bounced to whatever 'non-traditional' stack you want.

My firewalls, PDUs, rear-door heat exchangers, etc, do not talk (nor can they run) "Loki". Just about everything in existence can (a) talk syslog and/or (b) send SNMP traps.

Why have the additional hop? You can send Syslog to an Otel collector.
There will often be a hop in there anyways because you may want to bounce/forward to many locations: ELK, SIEM, offsite, etc. I would argue you want your first hop to be the simplest (IMHO, rsyslogd saving to text files, which can be easily rsync'd or zfs-send/recv).
That recommendation is hard to understand. Unless you have some completely trivial environment you will have syslog in it. It is not something you choose.

What you can do is have syslog ingestion for your Loki/Elastic/Splunk/whatever you use. But it will not magically structure your logs for you beyond the standard syslog date, host, severity, system, line of text format.

Both syslog products you mention are solid and mature, and absolutely has an important place in any modern environment. They manage queues, throttle, guarantee delivery and support any backend you can wish for.

Their respective rule engines are fast. They do log ingestion, and they slice data into fields and can enrich it (for example by resolving dns). This means rules are set in advance, and traps on data can be triggered immediately (not subject to races).

(comment deleted)
An intro article on syslog servers in the year of our Lord 2026?

Did you know you can replace your noSQL db with VSAM too (honestly better than mongo).

Mongo stopped having major data loss issues after a decade - my understanding is it's pretty stable now. You might hate the company for beta testing on your data but that's a separate issue.
I was mostly just being cheeky :P

I don't have anything against mongo in particular, the real nosql problem imo is people overusing it.

I'd reckon that even in this modern age there are on the order of 10^9 people who don't know what a syslog server is.
Syslog is still everywhere on a lot of hardware and appliance style devices.

For those bringing that into modern telemetry stack I can't recommend Grafana Alloy enough (https://grafana.com/docs/alloy/latest/). There's a loki.source.syslog component that lets you ingest RFC5424 or RFC3164 format messages over UDP, TCP, and with mTLS. These can then be parsed, filtered, transformed, and routed to anywhere else you require.

If you're stuck with a device that's UDP only it can deploy to a nearby host (e.g. container runtime on a switch, or local ingest node), perform log collection, then queue and forward via OTel to central or cloud infra with encryption, delivery and integrity guarantees from there. It's a really neat and powerful tool.

The article is not bad, but a little outdated. Syslog is a database that will not go anywhere, because a lot of equipment and software still send logs this way. But in 2026, building the entire infrastructure around it is already strange.
Database?