Show HN: Walrus – a Kafka alternative written in Rust (github.com)

As someone who myself worked on a hobby-level Rust based Kafka alternative that used Raft for metadata coordination for ~8 months: nice work!

Wasn't immediately clear to me if the data-plane level replication also happens through Raft or something home-rolled? Getting consistency and reliability right with something home-rolled is challenging.

Notes:

- Would love to see it in an S3-backed mode, either entirely diskless like WarpStream or as tiered storage.

- Love the simplified API. If possible, adding a Kafka compatible API interface is probably worth it to connect to the broader ecosystem.

Best of luck!

There's also Iggy https://github.com/apache/iggy

Never tried it, but looks promising

Nice! How does it compare to Redpanda, NATS, etc?

For Kafka alternative written in C++ there's Redpanda [1],[2].

Redpanda claim of better performance but benchmarks showed no clear winner [3].

It will be interesting to test them together on the performance benchmarks.

I've got the feeling it's not due to programming language implementation of Scala/Java (Kafka), C++ (Redpanda) and Rust (Walrus).

It's the very architecture of Kafka itself due to the notorious head of line problem (check the top most comments [4].

[1] Redpanda – A Kafka-compatible streaming platform for mission-critical workloads (120 comments):

https://news.ycombinator.com/item?id=25075739

[2] Redpanda website:

https://www.redpanda.com/

[3] Kafka vs. Redpanda performance – do the claims add up? (141 comments):

https://news.ycombinator.com/item?id=35949771

[4] What If We Could Rebuild Kafka from Scratch? (220 comments):

https://news.ycombinator.com/item?id=43790420

Fun anecdote; a couple years ago I started writing a Kafka alternative in C++ with a friend. I got pretty far, but abandoned the project.

We called it `tuberculosis`, or `tube` for short; of course, that is what killed Kafka.

[flagged]

[dead]

I never understood the popularity of Kafka. It's just a queue with persistent storage(ie. not in-memory queu with ram-size limited capacity) after all.

[flagged]

Walrus isn’t trying to replace Kafka, but it does beat Kafka in a few narrow areas. It’s a lightweight Rust-based distributed log with a fast WAL engine and modern I/O (io_uring), so the operational overhead is much lower than running a full Kafka stack. If you just want a simple, fast log without JVM tuning, controllers, or the entire Kafka ecosystem, Walrus is a lot easier to run. Kafka still wins on ecosystem, connectors, and massive scale, but Walrus is appealing for teams that want the core idea without the complexity. Really impressed by the direction here, great work!!.

[dead]

We need Rust alternative not written in Rust

For Rust-based Kafka alternatives, I like Tansu[1]. It at least provides Kafka API parity, and critically also gives users a pluggable backend (embedded SQLite, S3 for low cost diskless type workloads and Postgres because just use Postgres)

It’s nice to try and out innovate Kafka, but I fear the network effect can’t be beaten unless the alternative is 10x better.

Something like Warpstream’s architecture[2] had a shot at dethroning Kafka, but critically even they adopted the Kafka API. Sure enough, Apache Kafka introduced a competing feature[3] within two years of warpstreams launch too.

[1] - https://github.com/tansu-io/tansu [2] - https://www.warpstream.com/ [3] - https://topicpartition.io/blog/kip-1150-diskless-topics-in-a...

[flagged]

Are those dips in the write throughput comparison for Kafka GC pauses?

[flagged]

i would love to see a simple HTTP API built on top of this

like how postgrest works with postgres

What value does this provide other than being written in Rust? I'm pretty sure all of the five "key features" listed are either present in alternatives or already in Kafka itself.

[flagged]

That's funny.

- Let's create a new Kafka in rust. Yeah!

- Let's create a Kafka client that's ready to use. Blah.

I see no mention of recovering redundancy of stored data after a node failure.

It seems that the data is written to only a single node. But if that's the case, how is fault-tolerant ensured? In terms of data synchronization, how is this different from setting the replication factor to one in Kafka or Pulsar?