I have found Celery to be a major pain. It's very complicated (Are your Celery tasks guaranteed at-most-once or at-least-once delivery? Are you sure?). There are bugs with the Redis broker that cause it to drop tasks: https://github.com/celery/kombu/issues/305
I heartily recommend: https://dramatiq.io/motivation.html - "if you’ve ever had to use Celery in anger, Dramatiq could be the tool for you".
The main problem with Celery is that it supports many brokers and many features. This creates many subtle edge cases, specifically for brokers not built with AMQP, for instance the "visibility_timeout" attribute of the redis broker which feels like a hack.
Personally I prefer meesee[1], because setting up Celery or other big worker systems is always a lot of work, and they have more components than I usually need.
Meesee is an elegantly simple worker system (written in python) using redis as a backend, and it works really well if you don't need all those extra features that stuff like Celery give you.
You just need a redis instance and then you're good to go :)
I haven't heard of RQ before, but on first glance they look pretty similar.
The main difference I see is that meesee workers are always setup as python programs, whereas RQ workers are separate programs. (calling a `rq` program insead of `python3 my_worker.py`)
Though looking at the docs tells me that RQ can do the latter as well.
10 comments
[ 2.6 ms ] story [ 38.3 ms ] threadhttps://github.com/samuelcolvin/arq
I find celery's api kinda painful.
I have no financial interest in Prefect, but I have used Airflow and their criticisms ring true.
I heartily recommend: https://dramatiq.io/motivation.html - "if you’ve ever had to use Celery in anger, Dramatiq could be the tool for you".
This is why I created yet another task framework based on redis with robustness as a the main goal: https://github.com/NicolasLM/spinach
Meesee is an elegantly simple worker system (written in python) using redis as a backend, and it works really well if you don't need all those extra features that stuff like Celery give you.
You just need a redis instance and then you're good to go :)
1: https://github.com/Attumm/meesee
The main difference I see is that meesee workers are always setup as python programs, whereas RQ workers are separate programs. (calling a `rq` program insead of `python3 my_worker.py`) Though looking at the docs tells me that RQ can do the latter as well.