While basically almost no real world application needs more performance than what kafka delivers, i love to see research on this topic.
hopefully at some point a more easy to use, non java, "cloud-native" distributed log with decent client apis shows up. (if anyone wants to help building it, please ping me. i am 10 years into building startups and companies with architectures ontop of kafka and pulsar)
Your dream has arrived! Redpanda [1] is a non-Java, ZooKeeper-free streaming platform. It's Kafka compatible, so you can use any Kafka client, of which there are many.
Finally, and perhaps most importantly, it is substantially faster than Kafka.
The problem with Kafka is not Java nor ZooKeeper - it is that non-native (non-Java) clients are buggy as hell (last time I touched them), because everybody have to re-implement all the complex logic for each language (often multiple times).
Do you have any plans for Redpanda-native protocol/client library?
7 comments
[ 3.0 ms ] story [ 27.8 ms ] thread[1] https://blog.acolyer.org/2020/11/09/delos/
hopefully at some point a more easy to use, non java, "cloud-native" distributed log with decent client apis shows up. (if anyone wants to help building it, please ping me. i am 10 years into building startups and companies with architectures ontop of kafka and pulsar)
Can't think of a more well behaved java app. Heap usage generally low, never seen GC issues and fast as needed
Finally, and perhaps most importantly, it is substantially faster than Kafka.
Disclosure: I work there.
---
[1] https://github.com/vectorizedio/redpanda