I've been eyeing Kafka for a long time now - tried to set it up on a smaller DO box and had memory issues. Is it worth using this for smaller operations (<100k requests per day)? The stream processing seems extremely useful for my use case. Is there something else that may be a better fit?
If they're evenly distributed, 100K requests per day are barely 1/sec. It's peak throughput for which you should be planning. At 1/sec you can use just about anything.
That aside, if you're not sure if you should be using it I'm going to guess the answer is, "No you shouldn't". It's very cool tech but most likely overkill for whatever you're doing. Also, until recently securing it was a particular pain due to lack of TLS and authentication (I think this may have been resolved now).
All right, that makes sense. It is likely that we'll see a peak in the 10s of thousands at a time, if that. Probably more as we move into the future.
I want to avoid deploying a piece of software now that we'll need to swap out in a year since it does not provide what we need anymore, though I suppose that is a natural part of scaling.
What would you recommend as an alternative to Kafka that does adequate real-time processing? The real-time-ness of it is important for our use-case. Maybe even some kind of time-series database system could work.
If you really need "real-time-ness" TSDB will probably not work.
You could go with sparkstreaming or flink but for that amount that will be overkill. Kafka scales quite nicely and is relativly easy to operate (compared to spark etc). But thats only if you really need to scale, for your current usage even kafka is a bit overkill.
It really depends on what you need to do. That's around 1 request per second. I think Kafka might be overkill, but if you plan on needing to increase capacity then it might be worth it. It might just be easier to use RabbitMQ or even Redis at that scale though.
Okay, thank you. I've been looking into Rabbit quite a bit. I like the stream processing aspect of Kafka, where you can introspectively peek into things as they happen rather than a straight message queue.
Some of our stuff is on the GCP, so have also been investigating Pub/Sub and DataFlow, which seem more viable since we don't have the up-front cost of hardware and maintenance but pay for resources used. If we don't use much, we should not pay much.
One thing to note about Pub/Sub vs Kafka. Pub/Sub does not offer as strict of guarantees around ordering as Kafka. If you don't need that strictness I'd definitely recommend Pub/Sub over Kafka but if you do I don't believe any of the Google Cloud Offerings will provide you that (Kinesis on AWS will).
Kafka is pretty unique in what it is and how it does things and the way it is designed means you can build all kinds of solutions on it. If what you want is a more traditional messaging system, Kafka is probably overkill for your scale.
If on the other hand, you want an event store, it might be worth using. You can build event stores on top of lots of different data stores but you'll have to manage the edge cases your self usually. Kafka has them built in.
That said, Kafka is pretty complex to operate. I'd only start using it when it became obvious it was the solution to your problem or there was some external factor pushing you to use it.
Kafka's pretty neat and 'everyone' uses it. But for small scale stuff, this is easy enough to do with a queue, a timer (maybe even cron) and a small SQL DB (SQLite, H2, Etc.). Msgs flow into queue. Repeating timer wakes up, runs script/program to: create temp table in DB, dump queue to temp table, query temp table, write query results wherever, drop temp table. You can add further features/complexity by adding sliding/tumbling window, coalescing queue or other parts into the mix, per your specific requirements.
It had 512MB of RAM, which I realise is a bit rediculous to run anything like this. I don't have a whole lot of budget for R&D so I try to keep things cheaper where possible.
It actually runs well in such scenarios, it's just the default settings assume you're going to be running at large scale because that's what it's designed to do.
As other people say, Kafka is overkill for smaller use cases, but using it sets you up to scale if you need it and it gets you thinking in a good way architecturally.
Confluent also provide commercial support for it and employ Magnus who developed it.
I've been working closely with librdkafka for a few months now and find it fast and reliable. Many companies rely on it in production. In addition to the Confluent clients, Blizzard has also developed a client that wraps librdkafka for nodejs (which they use in production).
26 comments
[ 3.3 ms ] story [ 66.1 ms ] threadThat aside, if you're not sure if you should be using it I'm going to guess the answer is, "No you shouldn't". It's very cool tech but most likely overkill for whatever you're doing. Also, until recently securing it was a particular pain due to lack of TLS and authentication (I think this may have been resolved now).
I want to avoid deploying a piece of software now that we'll need to swap out in a year since it does not provide what we need anymore, though I suppose that is a natural part of scaling.
What would you recommend as an alternative to Kafka that does adequate real-time processing? The real-time-ness of it is important for our use-case. Maybe even some kind of time-series database system could work.
Some of our stuff is on the GCP, so have also been investigating Pub/Sub and DataFlow, which seem more viable since we don't have the up-front cost of hardware and maintenance but pay for resources used. If we don't use much, we should not pay much.
If on the other hand, you want an event store, it might be worth using. You can build event stores on top of lots of different data stores but you'll have to manage the edge cases your self usually. Kafka has them built in.
That said, Kafka is pretty complex to operate. I'd only start using it when it became obvious it was the solution to your problem or there was some external factor pushing you to use it.
https://kafka.blog/posts/scaling-down-apache-kafka/
It actually runs well in such scenarios, it's just the default settings assume you're going to be running at large scale because that's what it's designed to do.
As other people say, Kafka is overkill for smaller use cases, but using it sets you up to scale if you need it and it gets you thinking in a good way architecturally.
I've been working closely with librdkafka for a few months now and find it fast and reliable. Many companies rely on it in production. In addition to the Confluent clients, Blizzard has also developed a client that wraps librdkafka for nodejs (which they use in production).
Disclaimer: I work at Confluent.
And even from the same stable as Kafka, there's Samza[2]
[1] - http://www.espertech.com/esper/ [2] - https://speakerdeck.com/ept/turning-the-database-inside-out-...