I like idea of standardising a way for jobs to be defined and then shipped off to a background server. The adapter registration looks neat.
But by the looks of it the library bundles in the adapters to talk directly to these worker frameworks (currently it only is Resque). Are we going to see an active merging in of adapters or will at some point will the response be 'we won't be allowing any more adapters to bundled'.
You don't really need the adapters to be bundled. The queue libraries can easily bundle their adapters in their gem or in 3rd party gems like sidekiq-rails etc
Generally one runs defined cron jobs on a schedule set in advance. This allows you to queue up arbitrary methods to fire at arbitrary times, generally "a short while after this HTTP request has been dealt with."
This is an extraordinarily common pattern in web applications. My Rails apps do it hundreds of thousands of times per day, often to avoid hitting an external API during a request/response cycle. (I use Delayed::Job. This would not be my #1 recommendation for this pattern in 2014. Take a look at Sidekiq, Resque, or this new thing instead.)
I see, cron is absolute time events (every 5 minutes say or 2 minutes past the hour or once a week) and Active Job standardizes an interface to queue events relatively (5 minutes from this HTTP request, ten minutes from now). Is there a way cron could be extended to handle the relative case?
The main value of these systems is normally not "relative queuing" (although that is sometimes done).
Often, there are certain activities you might want to do in a way that doesn't block the request/response cycle, but still have them performed as soon as possible. For example, a user buys a widget on your site - send a receipt email. You don't want the "Thanks for buying" success page response to wait for your servers to compose & send the email, so you offload the email to your queue system.
To give an example let's say you have a mailing list with 1000 people and you want to send out notification. You push "notify " button, but you don't want to wait for your app to go through 1000 people and send them email until it responds again. You also don't want to wait until midnight for crontab to start processing all requests.
Background jobs takes that 1000 deliveries job and puts it into the background queue, then a worker starts sending those emails right away. Meanwhile your app returns from your request instantly so you can continue doing what you were doing while the job is executed in the background.
Another thing to bear in mind is that cron won't have access to your Rails environment per default. If for example you used cron to call a rake script which needed the Rails environment, you would be loading the Rails env every time, which is pretty heavy.
Rails 4.x was originally supposed to come with a Queueing API, but it was delayed/removed for reasons. It's nice to see this being picked back up and implemented. There's a ton of fragmentation out there in terms of background job libraries, there's also a lot of apps still using DelayedJob and others that "can't" upgrade to something better such as Sidekiq due to the sheer amount of work required. Standardizing on an API for this will help a lot and enable easily moving to the Latest and Greatest™ queueing library.
Sigh. This is like that timecop thing -- dumping more stuff in core that lives better as part of a gem.
Resque, DelayedJob, Sidekiq... All have their drawbacks/benefits. I know this can just work as a 'wrapper', but in an agency environment, i've used queueing on about 20% of sites.
Absolutely no reason to want to put this in core imo.
It being included will have no performance overhead since it's only loaded if it's used.
I'm personally for this idea, I've used DJ, Resque and Sidekiq, all in the same project (increasing load), and in that order; It would have been much nicer to have to only change the Gemfile when I made the switch.
Overcomplicating the rails 'core' is my concern. I'd much rather start from a lightweight 'base' which solves most of the common cases and pull in extras as required for what i'm working on.
Rails should be a framework that allows you to layer extra functionality ontop of it.
I don't really see the need for this. A unified interface could be nice, but this will also limit the scope that background processors will operate in. Besides that the amount of rails projects that need background jobs aren't that large of a percentage comparatively.
Having recently migrated from Resque to Sidekiq, I can attest that this solves a problem, albeit not terribly a big one. As with any abstraction, implementation-specific features make leaks. So far, the README doesn't say how I will control features like retries and queue weights, but maybe that's coming soon.
23 comments
[ 3.1 ms ] story [ 60.2 ms ] threadBut by the looks of it the library bundles in the adapters to talk directly to these worker frameworks (currently it only is Resque). Are we going to see an active merging in of adapters or will at some point will the response be 'we won't be allowing any more adapters to bundled'.
https://github.com/dwbutler/multi_worker
Haven't tried it yet but looks promising.
Cron is a POSIX task scheduler, this is a Ruby normalized API for background job systems.
This is an extraordinarily common pattern in web applications. My Rails apps do it hundreds of thousands of times per day, often to avoid hitting an external API during a request/response cycle. (I use Delayed::Job. This would not be my #1 recommendation for this pattern in 2014. Take a look at Sidekiq, Resque, or this new thing instead.)
If you ever wanted to upgrade form delayed job, this could come in handy for you.
Often, there are certain activities you might want to do in a way that doesn't block the request/response cycle, but still have them performed as soon as possible. For example, a user buys a widget on your site - send a receipt email. You don't want the "Thanks for buying" success page response to wait for your servers to compose & send the email, so you offload the email to your queue system.
Background jobs takes that 1000 deliveries job and puts it into the background queue, then a worker starts sending those emails right away. Meanwhile your app returns from your request instantly so you can continue doing what you were doing while the job is executed in the background.
For instance:
* Use cron to dump a database backup nightly
* Use some job system to send a user an email after they sign up
There's a lot of overlap, and if used cleverly (which may not be smart), they can both achieve the same goals.
Resque, DelayedJob, Sidekiq... All have their drawbacks/benefits. I know this can just work as a 'wrapper', but in an agency environment, i've used queueing on about 20% of sites.
Absolutely no reason to want to put this in core imo.
I'm personally for this idea, I've used DJ, Resque and Sidekiq, all in the same project (increasing load), and in that order; It would have been much nicer to have to only change the Gemfile when I made the switch.
Your comment, whilst a perfectly valid concern, reminds me of an article DHH posted: http://david.heinemeierhansson.com/2012/rails-is-omakase.htm...
Overcomplicating the rails 'core' is my concern. I'd much rather start from a lightweight 'base' which solves most of the common cases and pull in extras as required for what i'm working on.
Rails should be a framework that allows you to layer extra functionality ontop of it.
Background jobs are a common case. How many web applications never send an e-mail?
Example, you'd prefer A to B?
A) $ rails new foo_app # add 'active_job' if/when you want it
B) $ rails new foo_app --skip-active-job # forced to rip it out
EDIT: I think you answered my question in another sub-comment. No reply requested unless you have something new to add
I'd rather see some more interesting things going into core, such as https://github.com/kickstarter/rack-attack, that should be used by nearly every app.