Hi, OP. Any thoughts on why to prefer using your OLTP for queueing as opposed to Kafka? In my mind it would be a drop in ease of observability (though there is still KSQL and a number of UI wrappers) in exchange for much better performance. I'm interested if you had other reasons.
Observability is a big advantage, another advantage (in the context of DBOS specifically) is integration with durable workflows, so you can write a large workflow that enqueues and manages many smaller tasks.
I was architecture team lead at Skype 2005-2011 and persistent queues were the only ones around. Basically, because they knew how to scale databases, our DBA team (Hannu Krosing comes to mind) built queues into and on top of Postgres. It happened not just at Skype, too: eBay guys (Dan Prichett in particular) had built a very cool Oracle-based queue solution. Scaling persistent queues seems to be something that needs to be reinvented periodically. Maybe it’s too nuanced of a problem?
From what I recall, Reddit uses AWS extensively. Could they not have replaced RabbitMQ with SQS? You get the near unlimited horizontal scalability, extremely good uptime, guaranteed at least once message delivery and for the case of a worker crash, the messages will become visible again after the visibility timeout (since they wouldn’t have been deleted by the worker).
7 comments
[ 2.8 ms ] story [ 29.9 ms ] thread> What we really needed to make distributed task queueing robust are durable queues ...
> Durable queues were rare when I was at Reddit, but they’re more and more popular now.
It sounds like the answer was known at the time but there wasn't the resources to solve it ?