Trade-offs vs Redis:
- Not for multi-region distributed systems
- Best for single server or small clusters
Happy to answer any questions about the architecture!
We are using a service that abstracts redis from us and requires to be treated like a critical dependency, think RDS, Aurora, Postgres, if they are down the whole site is down. Every job push is a call to this service. Upgrading the service = downtime.
For us this is resulted in a big weak point on our architecture because when the service reboots both job pushing and job pulling stops, with the pushing being on the API side bringing the API down. With containers we could have multiple of them running at the same time, but the shared reading/writing of the abstract Redis locks itself.
We are considering BullMQ, because the architecture is sane:
* job push: API writes to Redis
* job pull: Worker reads from Redis then writes the completion.
How do you see this issue for Bunqueue? What happens when it goes down for 5 minutes, can the jobs be enqueued? Can you run multiple instances of it, failover?
Our throughput (jobs/sec) is small we do have 100k+ scheduled jobs anywhere from minutes to months from now.
At Apple, on iTunes, we used an Oracle database to do job queue stuff. Initially I will admit I made fun of it because I wanted to use a "real" queue like RabbitMQ or ActiveMQ or something, but I have to admit that it worked fine and to be fair it did predate both of those.
Anyway, it made me realize that there's really no reason you can't use a SQL database as a backing store for queue stuff. I should try building my own at some point.
4 comments
[ 3.1 ms ] story [ 19.9 ms ] threadThe idea: for single-server deployments, SQLite can handle 100k+ ops/sec with WAL mode, so why add infrastructure?
Features: priorities, delays, retries, cron jobs, DLQ, job dependencies, BullMQ-compatible API.For us this is resulted in a big weak point on our architecture because when the service reboots both job pushing and job pulling stops, with the pushing being on the API side bringing the API down. With containers we could have multiple of them running at the same time, but the shared reading/writing of the abstract Redis locks itself.
We are considering BullMQ, because the architecture is sane: * job push: API writes to Redis * job pull: Worker reads from Redis then writes the completion.
How do you see this issue for Bunqueue? What happens when it goes down for 5 minutes, can the jobs be enqueued? Can you run multiple instances of it, failover?
Our throughput (jobs/sec) is small we do have 100k+ scheduled jobs anywhere from minutes to months from now.
How would this work for anything _but_ a single server?
Anyway, it made me realize that there's really no reason you can't use a SQL database as a backing store for queue stuff. I should try building my own at some point.