I don't understand why we're rebuilding Resqueue/Sidekiq in Elixir. This seems like a problem which is solved largely by OTP.
Yes there are issues with back-pressure and over-filling mailboxes, but I can't help but wonder why you'd want to bring Redis into the picture when asynchronous processing is built-in for you.
Similarly if you want more robust queueing semantics, isn't RabbitMQ there?
OTP won't out of the box persist jobs that you want to guarantee that it will run.
It won't also retry and it can't schedule jobs to happen in the future.
Unless you are willing to lose the jobs that are just in memory you can simply use Task and send messages.
RabbitMQ also does not handle retries with backoff, scheduled jobs, etc.
Disclaimer: maintainer here.
Edit: I added the "with backoff" to not mix with automatic reenqueue if you reject a message from a Rabbitmq consumer.
That's a great question. I chose Redis for 2 reasons:
* Simple to use
* Compatibility with other systems like Sidekiq/Resque. Out of the box it will understand their clients.
My experience with Mnesia is close to none but I know it's not trivial to deal with split brain scenarios for example. It's also complicated to connect multiple nodes for my use case (using docker) and also to deal with moving instances inside a cluster. I would need to care about the locality of the data.
One can use any Redis service (AWS Elasticache, Redis Labs) and not care about maintaining a database. The Verk user just needs to have a working Redis database.
I'm pretty sure that having the database in the same space as the job processing system would speed up things. I would love to see a project that tried this approach. Probably not using JSON but simply Erlang terms.
Honest question, what is redis then? I've used it for all sorts of reasons, it seams like a swiss army knife, caching, queing, pub/sub, persistence, non-persistence all-in-one utensil. You can do a lot with it, and if it's already in the stack, it's a lot easier for others to reason with.
10 comments
[ 4.8 ms ] story [ 17.6 ms ] threadYes there are issues with back-pressure and over-filling mailboxes, but I can't help but wonder why you'd want to bring Redis into the picture when asynchronous processing is built-in for you.
Similarly if you want more robust queueing semantics, isn't RabbitMQ there?
Am I missing something?
Unless you are willing to lose the jobs that are just in memory you can simply use Task and send messages.
RabbitMQ also does not handle retries with backoff, scheduled jobs, etc.
Disclaimer: maintainer here.
Edit: I added the "with backoff" to not mix with automatic reenqueue if you reject a message from a Rabbitmq consumer.
Is there a compelling reason to add an additional piece of tech in my stack?
* Simple to use
* Compatibility with other systems like Sidekiq/Resque. Out of the box it will understand their clients.
My experience with Mnesia is close to none but I know it's not trivial to deal with split brain scenarios for example. It's also complicated to connect multiple nodes for my use case (using docker) and also to deal with moving instances inside a cluster. I would need to care about the locality of the data.
One can use any Redis service (AWS Elasticache, Redis Labs) and not care about maintaining a database. The Verk user just needs to have a working Redis database.
I'm pretty sure that having the database in the same space as the job processing system would speed up things. I would love to see a project that tried this approach. Probably not using JSON but simply Erlang terms.
Have you seen ferd's pobox library [1]?
[1]: https://github.com/ferd/pobox