Launch HN: Batch (YC S20) – Replays for event-driven systems
We are Ustin and Daniel, co-founders of Batch (https://batch.sh) - an event replay platform. You can think of us as version control for data passing through your messaging systems. With Batch, a company is able to go back in time, see what data looked like at a certain point and if it makes sense, replay that piece of data back into the company's systems.
This idea was born out of getting annoyed by what an unwieldy blackbox Kafka is. While many folks use Kafka for streaming, there is an equal number of Kafka users that use it as a traditional messaging system. Historically, these systems have offered very poor visibility into what's going on inside them and offer (at best) a poor replay experience. This problem is prevalent pretty much across every messaging system. Especially if the messages on the bus are serialized, it is almost guaranteed that you will have to write custom, one-off scripts when working with these systems.
This "visibility" pain point is exacerbated tenfold if you are working with event driven architectures and/or event sourcing - you must have a way to search and replay events as you will need to rebuild state in order to bring up new data stores and services. That may sound straightforward, but it's actually really involved. You have to figure out how and where to store your events, how to serialize them, search them, play them back, and how/when/if to prune, delete or archive them.
Rather than spending a ton of money on building such a replay platform in-house, we decided to build a generic one and hopefully save everyone a bunch of time and money. We are 100% believers in "buy" (vs "build") - companies should focus on building their core product and not waste time on sidequests. We've worked on these systems before at our previous gigs and decided to put our combined experience into building Batch.
A friend of mine shared this bit of insight with me (that he heard from Dave Cheney, I think?) - "Is this what you want to spend your innovation tokens on?" (referring to building something in-house) - and the answer is probably... no. So this is how we got here!
In practical terms, we give you a "connector" (in the form of a Docker image) that hooks into your messaging system as a consumer and begins copying all data that it sees on a topic/exchange to Batch. Alternatively, you can pump data into our platform via a generic HTTP or gRPC API. Once the messages reach Batch, we index them and write them to a long-term store (we use https://www.elassandra.io). At that point, you can use either our UI or HTTP API to search and replay a subset of the messages to an HTTP destination or into another messaging system.
Right now, our platform is able to ingest data from Kafka, RabbitMQ and GCP PubSub, and we've got SQS on the roadmap. Really, we're cool with adding support for whatever messaging system you need as long as it solves a problem for you.
One super cool thing is that if you are encoding your events in protobuf, we are able to decode them upon arrival on our platform, so that we can index them and let you search for data within them. In fact, we think this functionality is so cool that we really wanted to share it - surely there are other folks that need to quickly read/write encoded data to various messaging systems. We wrote https://github.com/batchcorp/plumber for that purpose. It's like curl for messaging systems and currently supports Kafka, RabbitMQ and GCP PubSub. It's a port from an internal tool we used when interacting with our own Kafka and RabbitMQ instances.
In closing, we would love for you to check out https://batch.sh and tell us what ...
62 comments
[ 6.3 ms ] story [ 135 ms ] threadRe: lowering the bar - we hope so. What we've noticed is that the papers that talk about event sourcing mention replays but don't talk at all about the implementation (or give any pointers). We're hoping that if at least that part is done for you, you've got one less thing to worry about.
As for the CLI tool - thanks! We found it super useful ourselves and figured others would too. I like to think of it as a sort of intelligent `netcat` for messaging systems :D
It's definitely on our roadmap to fix. Thank you for the feedback!
Re: ragtag in-house solution that is complicated
^ That's exactly what we're talking about. These systems get complex pretty quick and you end up with duct tape in more than a few places.
As for demo - yeah, our plan is to open up registrations for accounts soon which will allow you to pump data into us for free with a low retention period.
We've still got some pieces to tighten before we can open the gates fully but we'll try to make it happen soon (within next few weeks?). In the meantime, if you want a demo, ping us and we'll make it happen.
Two questions:
- If I have some data in Kafka, why would I want to pump it into your platform instead of spawning an Elasticsearch instance and using something like Kafka Connect to write to it and gain visibility?
- If I use Kafka as a permanent data store (with infinite retention), I can easily replay all events with existing clients (or with plumber). What additional functionality does the "replay" feature offer compared to that?
> - If I have some data in Kafka, why would I want to pump it into your platform instead of spawning an Elasticsearch instance and using something like Kafka Connect to write to it and gain visibility?
To avoid having to build, own and maintain the infra you just mentioned. As the number of events on your system increase, you will have to scale ES and other pieces of the system as well.
Our point is just that - if you know what's involved in collecting and indexing the events - that is awesome but maybe you shouldn't have to spend time building the infra around that stuff.
> If I use Kafka as a permanent data store (with infinite retention), I can easily replay all events with existing clients (or with plumber). What additional functionality does the "replay" feature offer compared to that?
I think it depends on your definition of "easily replay" - a kafka replay for a topic that's being consumed by a consumer group would require you to disconnect that consumer group and then run a shell script to move the offsets. You also would not have any way to replay any specific messages - your only point of reference would be an offset (and keyname, if you use it) - not terribly flexible.
With Batch, you get to drill in and replay the _exact_ messages you want (and avoid having to pump and dump potentially millions of messages your consumer doesn't care about).
That's a totally legit point - we've talked about offering it all in-house before but it would require us to split our efforts into two - operating a PaaS (for a bunch of different messaging tech) and running the event collection platform.
Operating the PaaS part would be a full-time effort and there's a lot of competition out there. We've decided to focus on the observability/replay part first (since there is a lot less competition) and then later maybe explore the hosted bus option.
LMK if that's not what you meant :)
The other way around. If I'm not storing my messages today it's probably because it is too expensive in terms of storage or compute to do so. But, presumably, you can't do that any cheaper than I can. And now we are duplicating the work so even more resources are being consumed making it that much more expensive than just doing it myself.
It seems like your service is something I'd want to run pointed towards my Kafka/RabbitMQ/whatever servers. I don't see how duplicating that stream is cost effective.
For example: if you are sending serialized data on your bus - you will need to write something that will deserialize it before inserting it into your elastic search cluster - and now you're managing even more infra (message systems, decoders, document storage).
There is definitely a price attached to the luxury - but we're betting that it'll be significantly less than doing it yourself.
I think maybe we should just list out the messaging systems we support on the front page, so you don't have to dig through stuff... Good point. Let me know if you've got any other suggestions.
Congrats on the launch. The pain-point makes sense to me. I'm just curious - what's the big picture for you all? I imagine it must be larger than just replay.
- How do you handle events with side effects (sending emails, for example), and ensuring they aren't triggered on replay when they shouldn't be?
- How do you handle randomness, like uuid generation?
Someone else already addressed this, but to paraphrase: your application should be able to deal with duplicate events (and gracefully handle side-effects).
> How do you handle randomness, like uuid generation?
Are you referring to id generation and tagging in events (ie. aggregate id's)? If so, that'd be an application responsibility - you'd have to determine how to properly attach id's.
Hmm. But that does bring up an interesting idea - what if we provided a way to "group" events and generate aggregate id's on your behalf. Maybe that's what you meant - it's an interesting idea.
We currently don't do anything "extra" in regards to grouping events - we tag each individual event but that's about it.
There are many use cases from piping website activity tracking, metrics, log aggregation and stream processing. For us, it's a communication layer utilized by our microservices. An event goes into the stream and any services that cares about that data will consume it. In other words, it's like an ultra-resilient, scalable, redis pub-sub with history that runs on the JVM. You can read more about the use cases here: https://kafka.apache.org/uses
edit: Sidenote, Kafka is often waaaaaay overkill - if you need messaging, use something simpler like Rabbit or NATS or Redis and only use Kafka if you know why you need it.
Something like Batch could be helpful in this situation. For example, let's say a dev makes a deploy that breaks only the FedEx delivery notification or the FedEx API breaks in a way your were not expecting. Once the issue is fixed on the dev side or FEDEX side you could use Batch to search for all FEDEX delivers that were handled improperly during the time frame of the issue. This way you are not randomly resending messages to all your delivery companies for an issue that was only related to one vendor.
> How does bookmarking work/How do I keep track of how far I've read while replaying from Batch?
We do not have any bookmarking functionality built (yet) as we currently expect folks to just tweak their search query. Each one of the events has a new id attached to it that you can query and reference during search.
> Will you also index by date?
We do! Every event has a microsecond timestamp attached to it.
> It can take a long time to replay a lot of data; do you have any numbers on the read rates you support per topic?
We've done some initial replay throughput tests and have been able to reach ~10k/s outbound via HTTP - of course, this is all _highly_ dependent on where you're located. We expect that for folks who need super high throughput, we'll probably need to be closer to them - we fully expect to have to peer with some of our customers and optimize for throughput by doing gRPC and ... batching :)
So far, we've done most of our testing on inbound and we are currently able to sustain ~50k/s (with ~5KB event size). Our inbound is able to scale horizontally and so can go waaaaay beyond 50k/s if needed.
We have a ton of service instrumentation so we've got good visibility around throughput (and thus should know well in advance as to when we're starting to hit limits).
Will this work with Celery (python) configured with RabbitMQ as the broker?
Pretty cool idea though. Hope it pans out for you guys.
Will try to figure out a way to better communicate what we do.
Two-sentence pitches are much harder than two-minute pitches.
I'm curious how long it took you to build this initial product given the complexity.
YC has a bias for shipping quickly, but my gut instinct is that it would have taken you a while to build this initial version.
Did it only take a few months, or closer to 8-12+?
Yes and no :)
We've been exceptionally lucky to have several of our close friends help us out with building an MVP (also helps that our friends have serious experience!). There's a total of 6 of us - 3 people focusing on infra, frontend and Java connector bits, which allowed myself, Ustin and another dev to put 100% of our attention on backend services + arch.
That enabled us to knock this out in a few months. Without the assist, it would probably be closer to your estimate.
Something that may be of interest to some folks: we saved a significant amount of time by not having to run our own k8s - we use EKS, it's very nice. Also, MSK - not having to run/manage ZK clusters and kafka nodes is a (costly) privilege haha
Good luck building out the rest of the product!
;)