I haven't modeled it, but I wonder how far you'd get on randomizing the policy choice for concurrency limit 1. Maybe weighted by past results, but bounded to allow it to shift instead of falling permanently into a basin.
Or, you could be a government agency that implemented a method for citizens to upload compliance data - so they have to upload broken CSVs into a broken queue system. Get the worst of both worlds, and deployed as a green-field project in 2021.
Major lesson from when I worked on Google Search indexing is that queues have a lot of hidden complexity and can make your outages much longer than they need to be. We had a big project to get rid of a bunch of queues by just scaling up our synchronous backends and making them faster.
I think the the second part was kinda obvious? The moment I read this:
> If you’re anything like me, you would probably have said Parallel Spawn, Prefer New, and Wait are perhaps defensible, whereas Prefer Old feels weird/backward.
it was pretty obvious I was not anything like him. My intuitive answers are pretty different.
- Parallel Spawn is useful, but it's orthogonal to the rest. Even if you have 4 workers, you'll still have to worry about hitting concurrency limit once you have enough jobs. What is it even doing in this list?
- Wait is very useful for non-scheduled tasks: if user uploaded 100 files to process, you better process them all. Sometimes you need limit, sometimes you do not (let them queue for a while until devops notices and either allocate more workers or clear them and has some harsh words with consumer).
For scheduled tasks, "Wait" seems much less useful. I can come up with a reason but they are all somewhat convoluted - perhaps you are hitting 3rd-party service, and it has a quota, so you've decided to use scheduler to avoid hitting ratelimits?
- "Prefer Old" is normally the best way. You repack takes 3-8 hours, so you set your timer to "every 1 hours, skip if running already" and you can be sure that your job finishes and the next one will start.
- "Prefer New" seems almost useless. You've already spent all this effort doing the job, why are you cancelling it and throwing away the results? If you want to add a timeout, add a timeout, preferably to specific operation. For example, if there job starts by fetching data, and this fetch can be super slow, use 'Prefer Old' and put a timeout on the fetch part. This way your job won't be interrupted if the fetch just finally succeed minutes before next scheduled interval hit.
Oh, and re "If the Prefer Old semantics are not offered, you can’t really emulate them using the two primitives of regular scheduling and limiting concurrency." - you totally can. Set concurrency to 2, and add as a first thing: "fetch the list of jobs running; if there is anyone except me, exit right away".
At Google we actually had 'prefer new' (ie a stack instead of a queue) for certain jobs, that were likely no longer useful after some time had passed; and where less and less useful the longer you waited until you started.
One example was running certain ad auctions when rendering websites, or something like that. You don't want to delay serving the side, if the ads are delayed.
So you have a certain wall clock time budget until the rest of the page is assembled to be sent to the user, and if you can fit your ad-serving in there, that's good.
If you have more work than you can currently handle, then it makes sense to continue with the newest open request after you handled the previous request.
The actual system was a lot more complicated, and combined a short, bounded queue on the inside with a large stack on the outside or something like that.
Similar considerations can apply, when you are one of many competing market makers for some financial assets on an exchange.
Basically, if you have a situation where serving quick is a lot more important than the distinction between late and very late (or even dropping the request).
> - "Prefer New" seems almost useless. You've already spent all this effort doing the job, why are you cancelling it and throwing away the results?
The obvious use-case is if the result depends on a state and the state has changed since the job has started. Examples would be updating the landing page when a new article has come in (you don't need the outdated landing page w/o the new article) or if you did a code change while compiling (assuming it's not a commit).
For external events, like code update or new publish, sure. But the author talks about scheduled events - it's harder to think of good examples here.
(because with scheduled events, if you missed deadline once, there is a good chance that restarting with exactly the same deadline will miss again, and again, and you will never finish. Better do "prefer old" and at least have _some_ finished data)
Tangential, but when dealing with queues, the first thing you want to do is have a basic grounding of queuing theory, and know whether you're optimising for throughput or worker utilisation (i.e. what are your SLAs and efficiency targets?). IME each goal involves fairly different metrics and scaling rules, so you'll want to know what you're prioritising.
The most popular resource manager for job submission and queueing system is Slurm. It's being used in majority of TOP500 supercomputers, and overwhelming majority of the world HPCs [1].
SchedMD the leading developer of Slurm has recently being acquired by Nvidia, while Slurm remaining free and open source, but somehow it's Wikipedia entry is not yet updated accordingly.
The UNIX pipe really is an incredible concurrency invention which is not well understood, and attempts to work around its features turn into bugs.
A buffer to accumulate data that blocks when it’s full allows you to handle bursty loads. It solves the back pressure problem of readers and writers operating at different speeds. It doesn’t over consume resources.
It also solves the architecture problem of when to trigger work. Both consumer and producer act on the pipe imperatively, rather than one end being imperative and the other being a declarative graph of callbacks (all those “reactive” libraries).
Even using a term like “back pressure” is a tell to me that someone is confused snd doing something architecturally wrong.
The schedule interval vs. hard timeout distinction is the useful bit here. Treating them as one setting makes backlog behavior much harder to reason about.
There is 4th option that looks like a combination of how all three are described here: If J1 is running and J2 gets queued, then when J3 gets queued you cancel J2 but not J1. Prefer the oldest running (well, any running, just don't kill them) and newest queued, which waits on the running one. This is how we had long-running test suites configured on Jenkins/Hudson/buildbot ages ago (though it wasn't exactly a queue, more just a flag that the job needed to be run and the job pulled in the latest state).
26 comments
[ 0.19 ms ] story [ 610 ms ] threadAre you talking about trying to interpret malformed data?
> If you’re anything like me, you would probably have said Parallel Spawn, Prefer New, and Wait are perhaps defensible, whereas Prefer Old feels weird/backward.
it was pretty obvious I was not anything like him. My intuitive answers are pretty different.
- Parallel Spawn is useful, but it's orthogonal to the rest. Even if you have 4 workers, you'll still have to worry about hitting concurrency limit once you have enough jobs. What is it even doing in this list?
- Wait is very useful for non-scheduled tasks: if user uploaded 100 files to process, you better process them all. Sometimes you need limit, sometimes you do not (let them queue for a while until devops notices and either allocate more workers or clear them and has some harsh words with consumer).
For scheduled tasks, "Wait" seems much less useful. I can come up with a reason but they are all somewhat convoluted - perhaps you are hitting 3rd-party service, and it has a quota, so you've decided to use scheduler to avoid hitting ratelimits?
- "Prefer Old" is normally the best way. You repack takes 3-8 hours, so you set your timer to "every 1 hours, skip if running already" and you can be sure that your job finishes and the next one will start.
- "Prefer New" seems almost useless. You've already spent all this effort doing the job, why are you cancelling it and throwing away the results? If you want to add a timeout, add a timeout, preferably to specific operation. For example, if there job starts by fetching data, and this fetch can be super slow, use 'Prefer Old' and put a timeout on the fetch part. This way your job won't be interrupted if the fetch just finally succeed minutes before next scheduled interval hit.
Oh, and re "If the Prefer Old semantics are not offered, you can’t really emulate them using the two primitives of regular scheduling and limiting concurrency." - you totally can. Set concurrency to 2, and add as a first thing: "fetch the list of jobs running; if there is anyone except me, exit right away".
Really, not that tricky at all.
One example was running certain ad auctions when rendering websites, or something like that. You don't want to delay serving the side, if the ads are delayed.
So you have a certain wall clock time budget until the rest of the page is assembled to be sent to the user, and if you can fit your ad-serving in there, that's good.
If you have more work than you can currently handle, then it makes sense to continue with the newest open request after you handled the previous request.
The actual system was a lot more complicated, and combined a short, bounded queue on the inside with a large stack on the outside or something like that.
Similar considerations can apply, when you are one of many competing market makers for some financial assets on an exchange.
Basically, if you have a situation where serving quick is a lot more important than the distinction between late and very late (or even dropping the request).
The obvious use-case is if the result depends on a state and the state has changed since the job has started. Examples would be updating the landing page when a new article has come in (you don't need the outdated landing page w/o the new article) or if you did a code change while compiling (assuming it's not a commit).
(because with scheduled events, if you missed deadline once, there is a good chance that restarting with exactly the same deadline will miss again, and again, and you will never finish. Better do "prefer old" and at least have _some_ finished data)
SchedMD the leading developer of Slurm has recently being acquired by Nvidia, while Slurm remaining free and open source, but somehow it's Wikipedia entry is not yet updated accordingly.
[1] Slurm Workload Manager:
https://en.wikipedia.org/wiki/Slurm_Workload_Manager
[2] Nvidia Acquires SchedMD (7 comments):
https://news.ycombinator.com/item?id=46277190
A buffer to accumulate data that blocks when it’s full allows you to handle bursty loads. It solves the back pressure problem of readers and writers operating at different speeds. It doesn’t over consume resources.
It also solves the architecture problem of when to trigger work. Both consumer and producer act on the pipe imperatively, rather than one end being imperative and the other being a declarative graph of callbacks (all those “reactive” libraries).
Even using a term like “back pressure” is a tell to me that someone is confused snd doing something architecturally wrong.