As someone who has made the mistake of using kafka in a non enterprise space - it really seems like the etcd problem where you need more time to run etcd than to run whatever service you're providing.
Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.
If you have this stance to technology ("feels bloated", meme-level knowledge on Java), then I really hope that you are not responsible for technical decisions.
My only complaint with this article is that it seems to be implying kafka that linkedIn's problem couldn't have been solved with a bunch of off-the-shelf tools.
I worked on this while I was at LI and I think the major selling point back then was Replayability of messages but it was something similar to what you would get with Pub/Sub. We could also have multiple clients listening and processing same messages for their own purposes so you could use the same queue and have different clients process them as they wanted.
I wanted to write a comment on this topic, but after several tries this thread is where I ended up because it describes my sentiment as well.
The arguments in the article are very compelling. But as soon as you choose Kafka you realize the things you hate.
Many of the reasons are stupid things - like it uncovers otherwise unimportant bugs in your client code. Or that it just makes experimenting a hassle because it enforces poking around in lots of different places to do something. Or that writing and maintaining the compulsory integration test takes weeks of your time.
Sure - you can replay your data - but not until you have fixed all the issues for that special case in your receiving service.
I think maybe my main gripe (for us) was that it was a difficult to get an understanding what is actually inside your pipe. Much easier to have that in a solid state in s3?
At the end of they day you get annoyed because it slows you down. In particular when you are a small localized team.
I read it and looked at the block diagrams. I still don't get it. You have "data integration problems". Many software share data. Use database. Problems solved.
Kafka's ability to ingest the firehose and present it as a throttle-able consumable to many different applications is great. If you're thinking "just use a database", it's worth noting that SQL databases are _not well suited_ to drinking from a firehose of writes, and that distributed SQL in 2012 was not a thing. Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was a big step forward for web applications at scale. If you bristle at that, know that using read-replicas of your postgres database present the same correctness problems.
These days though, unless I was at Fortune 100 scale, I'd absolutely turn to Redis Cluster Streams instead. So much simpler to manage and so much cheaper to run.
Also I like Kafka because I met two pretty Russian girls in San Francisco a decade back and the group we were in played a game where we described what the company we worked for did in the abstract, and then tried to guess the startup. They said "we write distributed streaming software", I guessed "confluent" immediately. At the time confluent was quite new and small. Fun night. Fun era.
> Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was a big step forward for web applications at scale.
Could you expand on this - when does it drop C? Are you referring to cases where you write to Kafka without waiting for all replicas to acknowledge the write? (acks=1)
And why was it a big step - what other systems didn't embrace dropping the C?
> that SQL databases are _not well suited_ to drinking from a firehose of writes
Now I’m wondering if we’re all overthinking this when we could just use rendezvous hashing and a bunch of database servers with a heap table called “eventlog” and be done with it…
For a long time I've wondered if we could just invent an extension for Postgres that allow physically ordered, append-only tables.
The main two things that makes Postgres less suitable for Kafka-type logs is that tables aren't very efficient for sequentially ordered data, and that deletion incurs bloat until vacuumed. You could solve both by providing a new table engine (table access method), although I'm not sure you can control heap storage placement to the degree desired for a physically ordered table. But you could also do a lot of tricks to make it delete faster (append only means no updates; just prune from the head without MVCC when provably safe against concurrent reads?) and make filtering faster.
Kafka is of course more than that, but I bet you can get quite far with this.
LinkedIn recently announced that it transitioned from Kafka to Northguard.
Introducing Northguard and Xinfra: scalable log storage at LinkedIn [1] & LinkedIn: Stream Processing 4.16.25 [2]
This might be hyperbolic, but I think Kafka (or at least the concept of event driven architecture for sharing data across many systems) is one of the most under-rated technologies. It's used at a lot of big corps but is never talked about.
> According to LinkedIn's engineers, Kafka had become increasingly difficult to manage at LinkedIn's scale (32T records/day, 17 PB/day, 400K topics, 150 clusters).
Wtf is LinkedIn doing that they create 17 PB/day?????
The biggest strength of Kafka in my opinion is consumer groups.I have been using it since 2016 in at least 3 projects and it never failed, not that big workloads though (~100 messages/sec max). However it is a bit difficult to monitor and manage using only the out of the box applications.
In my experience, Apache Kafka must be understood not as an isolated messaging tool, but as a comprehensive data streaming platform. Its successful implementation demands a holistic approach that encompasses performance, governance, and lifecycle management. I have consistently found that simply adopting the technology without a robust supporting architecture is an ineffective practice that leads to operational challenges.
Based on my work managing large-scale Kafka environments across critical sectors, I have identified that their stability and efficiency are upheld by a set of essential practices and tools. These are the non-negotiable pillars for success:
Health Checks & Observability: Proactive cluster health monitoring and complete visibility into the data flow are paramount.
Failure Management: Implementing dedicated portals and processes for handling Dead-Letter Queues (DLQs) ensures that no critical information is lost during failures.
Automation & DevOps: I leverage Strimzi for Kubernetes-native cluster management, orchestrating it through ArgoCD and GitOps practices. This ensures consistent, secure, and repeatable deployments.
The correct application of these engineering principles allows for remarkable results. For instance, at a large fashion retail group, I successfully scaled an environment to handle a peak traffic of 480,000 TPS. This high-availability system is efficiently maintained by a lean operational team of just two junior-to-mid-level professionals.
From my perspective, success in adopting Kafka is determined by the business context and the maturity of the applied software engineering. The investment in a well-planned architecture and a robust support ecosystem has a clear return, paying for itself through a significant reduction in operational costs (OPEX) within an estimated two-year period.
Taming Kafka isn't about new, complex secrets. It's about applying the same robust software engineering and architecture fundamentals we've relied on for +50 years (Software Engineering). The platform is new (2011), the principles are not.
26 comments
[ 2.7 ms ] story [ 45.1 ms ] threadThe arguments in the article are very compelling. But as soon as you choose Kafka you realize the things you hate.
Many of the reasons are stupid things - like it uncovers otherwise unimportant bugs in your client code. Or that it just makes experimenting a hassle because it enforces poking around in lots of different places to do something. Or that writing and maintaining the compulsory integration test takes weeks of your time.
Sure - you can replay your data - but not until you have fixed all the issues for that special case in your receiving service.
I think maybe my main gripe (for us) was that it was a difficult to get an understanding what is actually inside your pipe. Much easier to have that in a solid state in s3?
At the end of they day you get annoyed because it slows you down. In particular when you are a small localized team.
These days though, unless I was at Fortune 100 scale, I'd absolutely turn to Redis Cluster Streams instead. So much simpler to manage and so much cheaper to run.
Also I like Kafka because I met two pretty Russian girls in San Francisco a decade back and the group we were in played a game where we described what the company we worked for did in the abstract, and then tried to guess the startup. They said "we write distributed streaming software", I guessed "confluent" immediately. At the time confluent was quite new and small. Fun night. Fun era.
> Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was a big step forward for web applications at scale.
Could you expand on this - when does it drop C? Are you referring to cases where you write to Kafka without waiting for all replicas to acknowledge the write? (acks=1)
And why was it a big step - what other systems didn't embrace dropping the C?
Now I’m wondering if we’re all overthinking this when we could just use rendezvous hashing and a bunch of database servers with a heap table called “eventlog” and be done with it…
The main two things that makes Postgres less suitable for Kafka-type logs is that tables aren't very efficient for sequentially ordered data, and that deletion incurs bloat until vacuumed. You could solve both by providing a new table engine (table access method), although I'm not sure you can control heap storage placement to the degree desired for a physically ordered table. But you could also do a lot of tricks to make it delete faster (append only means no updates; just prune from the head without MVCC when provably safe against concurrent reads?) and make filtering faster.
Kafka is of course more than that, but I bet you can get quite far with this.
[1]: https://www.linkedin.com/blog/engineering/infrastructure/int...
[2]: https://www.youtube.com/watch?v=RDV6-MUVEbQ
Are they really generating data at this scale?
I cant even imagine a system which creates and stores this much data this fast.
https://www.infoq.com/news/2025/06/linkedin-northguard-xinfr...
> According to LinkedIn's engineers, Kafka had become increasingly difficult to manage at LinkedIn's scale (32T records/day, 17 PB/day, 400K topics, 150 clusters).
Wtf is LinkedIn doing that they create 17 PB/day?????
It was later open-sourced in 2011 and became a top-level project under the Apache Software Foundation in 2012.
The creators went on to co-found Confluent, a company that provides commercial support and enterprise features around Kafka.
Based on my work managing large-scale Kafka environments across critical sectors, I have identified that their stability and efficiency are upheld by a set of essential practices and tools. These are the non-negotiable pillars for success:
Health Checks & Observability: Proactive cluster health monitoring and complete visibility into the data flow are paramount.
Failure Management: Implementing dedicated portals and processes for handling Dead-Letter Queues (DLQs) ensures that no critical information is lost during failures.
Automation & DevOps: I leverage Strimzi for Kubernetes-native cluster management, orchestrating it through ArgoCD and GitOps practices. This ensures consistent, secure, and repeatable deployments.
The correct application of these engineering principles allows for remarkable results. For instance, at a large fashion retail group, I successfully scaled an environment to handle a peak traffic of 480,000 TPS. This high-availability system is efficiently maintained by a lean operational team of just two junior-to-mid-level professionals.
From my perspective, success in adopting Kafka is determined by the business context and the maturity of the applied software engineering. The investment in a well-planned architecture and a robust support ecosystem has a clear return, paying for itself through a significant reduction in operational costs (OPEX) within an estimated two-year period.
Taming Kafka isn't about new, complex secrets. It's about applying the same robust software engineering and architecture fundamentals we've relied on for +50 years (Software Engineering). The platform is new (2011), the principles are not.