Even though I first created images for EC2 in 2008, I've still never used SQS. I have a lot of experience with Google PubSub and this blog post is making me wonder if I should try leveraging SQS in production. I'm sure it's very mature, but I'm curious about how it feels from a developer perspective with a heavy production load.
IMO, SQS is one of the best products that AWS offers. Airbnb built a queueing + scheduling system on SQS to great success [0] (disclaimer, I wrote this post). There are many others, e.g., Slack, that are building on top of SQS.
> SQS is one of the best products that AWS offers.
I would never trust anyone who says this, regardless of who did what with it.
SQS Standard (which was the only for a long time) has historically been a nightmare for almost any use case including job queueing. I love creating a whole separate cache to ensure when SQS delivers message multiple times, that I don't act on them. That's just ONE issue that I don't need to have. When you get into FIFO to avoid that, now your costing is ridiculous at scale AND the (relatively small) number of issues with FIFO that are still outstanding .
Sharing an SQS queue between services is ok then? Even the most basic example for SQS queues from the AWS SA training course recommends using multiple queues for different classes of video transcoding. All of those benefits evaporate once you share a single queue; if you're just reading that queue to create a job in a database, then the queue provides no benefit.
I'm not quite clear what you mean by "sharing a single queue between services". If you mean having multiple different services reading from the same queue then no, I wouldn't do that. If you mean having a sending service and a receiving service share the same queue (either directly or indirectly) in the sense that the sender sends a message and the receiver reads it then I can't see any alternative to doing that. If I've misunderstood then would you be able to share what you were thinking of?
I interpret arduinomancer's post above to mean that if the receiving service must action a single message once and only once (which is sometimes but not always necessary) then you need to give each SQS message a unique ID and that ID needs to be stored in as DB by the receiving service. I've used that pattern a few times and it works well (and doesn't result in unnecessary coupling between sender and receiver). In the past, I've used messaging systems that gave stronger transactional guarantees, but those systems were things like JMS or MQSeries and I don't really want to go back to those days.
> I'm not quite clear what you mean by "sharing a single queue between services".
I was referring to your (seemingly snarky, and I apologize if I mistook your intent) comment that you hoped I wasn't suggesting to a shared database between services.
I understand and agree that this is what OP was stating. What I was saying was that having the consumers and producers sharing a queue is not that much different from having them share a database, but a database gives you other guarantees as well. (I am a big fan of queues and have been using SQS since 2007.)
In other words, the database is still going to be your bottleneck/SPOF and a queue just introduces needless complexity; you might as well just write to the DB directly and have a table in the DB (or something like a Redis set) to track the jobs.
A queue with indeterminate ordering is better aimed at idempotent jobs than jobs that will eventually need to be serialized by necessity or design.
> I was referring to your (seemingly snarky, and I apologize if I mistook your intent) comment that you hoped I wasn't suggesting to a shared database between services.
Apologies, I did write a snarky comment and I'm very sorry that I did that, rather than asking for clarification and/or explaining my position a bit more clearly.
I take the view that sharing an SQS queue (or a topic, or a stream...) between two services is OK. It creates a contract between those two services, of course, and part of that contract might be that there is a uniqueId field that the receiving service is going to have to persist in its own database in order to only action each request once, but it doesn't add an excessive amount of complexity or coupling between the services, not for me at least.
I'm still not OK with having two services share the same database, though, even if you're using it for a narrow purpose. For me, the danger is that you end up introducing more coupling than just agreeing on a message format and messaging semantics. I've been burned by that once or twice and I don't want to get burned by it again and I'm pretty happy that it's widely seen as an anti-pattern these days and personally I dogmatically avoid it. I do take your point that the database is going to end up being a SPOF/bottleneck, though, and I don't really have a great answer for that in the general case.
I’m no longer with Airbnb / the project, but we did in fact use SQS Standard exclusively for job queueing (I don’t think we ever did explicitly add any support for FIFO). Our SLA was at-least-once (as most job queues are), so this didn’t matter as much. In practice it also doesn’t happen that often.
It's really not hard to pick a natural key to use for idempotency in your persistence layer. If you don't want at least once delivery, why are you picking SQS instead of some random ephemeral queue like NATS?
I second the parent post. I've billions of messages through SQS at a previous job and I can remember having issues with SQS availability exactly once (due to a systemic failure involving DyanmoDB where cascading failures took down almost everything (and all of our integrations) that would have hosed us completely if not for our durable-queue/s3-journal[0] at the edge). SQS is simple to use, does what it says, and has very good SDK support. A++++ Would build a business on top of again!
This is a surprising take. Accounting for duplicate events is such a common pattern in distributed systems that it never occurred to me to think of it as an issue. What is your preferred approach?
The one legitimate "problem" in that list that is #3, the hard limit on FIFO messages (20,000). The rest are expected behavior for a 'deliver exactly once and in order' queue. The use cases offered (client fails to process and therefore consume the next message, blocking subsequent messages) and somehow (?) expecting the queue to facilitate reprocessing previous consumed messages are both misuses of such a queue.
SQS (standard and FIFO) has another problem that precludes using it in one case I have; the 256KB message size limit. Amazon has a Java-only workaround (Amazon SQS Extended Client Library for Java) that will spool large messages to/from S3 -- so clearly my case isn't unreasonable -- but there is nothing for the general case.
Most consistently excellent AWS product: it's a simple thing that does a simple job with minimal hassle. I suspect it will save me the trouble of ever having to run messaging infrastructure again.
Like you can only send certain character sets. I had a very angry customer, and in the end it was some character (0x07?) that couldn't be sent via SQS.
I had to either be certain that contents would never contain an SQS-unallowed character, or base64-encode all my contents.
As a side note - For someone not totally an expert in AWS it’s sometimes difficult to choose between SQS, SNS, EventBridge and the list of other related technologies.
I interned on the SQS team back in 2006 and subsequently joined the team full time for a few years including briefly managing it. I have some great memories of that time. Keeping things 'simple' while scaling as AWS grew was a great introduction to the industry.
The last time I used SQS it was kinda expensive because I had to poll every minute or so it ate up the free tier. If I wanted to poll faster (like every 5 seconds) it was even more expensive. Maybe not the best use for my use case.
30 comments
[ 2.8 ms ] story [ 61.2 ms ] thread[0]: https://medium.com/airbnb-engineering/dynein-building-a-dist...
I would never trust anyone who says this, regardless of who did what with it.
SQS Standard (which was the only for a long time) has historically been a nightmare for almost any use case including job queueing. I love creating a whole separate cache to ensure when SQS delivers message multiple times, that I don't act on them. That's just ONE issue that I don't need to have. When you get into FIFO to avoid that, now your costing is ridiculous at scale AND the (relatively small) number of issues with FIFO that are still outstanding .
eg https://tomgregory.com/3-surprising-facts-about-aws-sqs-fifo...
I've found its trivial to just have a unique ID in the message and just check if the ID already exists in a jobs table to avoid double processing.
I pretty much always have some metadata table needed for the job anyway, so its not like I'm building something extra.
I interpret arduinomancer's post above to mean that if the receiving service must action a single message once and only once (which is sometimes but not always necessary) then you need to give each SQS message a unique ID and that ID needs to be stored in as DB by the receiving service. I've used that pattern a few times and it works well (and doesn't result in unnecessary coupling between sender and receiver). In the past, I've used messaging systems that gave stronger transactional guarantees, but those systems were things like JMS or MQSeries and I don't really want to go back to those days.
I was referring to your (seemingly snarky, and I apologize if I mistook your intent) comment that you hoped I wasn't suggesting to a shared database between services.
I understand and agree that this is what OP was stating. What I was saying was that having the consumers and producers sharing a queue is not that much different from having them share a database, but a database gives you other guarantees as well. (I am a big fan of queues and have been using SQS since 2007.)
In other words, the database is still going to be your bottleneck/SPOF and a queue just introduces needless complexity; you might as well just write to the DB directly and have a table in the DB (or something like a Redis set) to track the jobs.
A queue with indeterminate ordering is better aimed at idempotent jobs than jobs that will eventually need to be serialized by necessity or design.
Apologies, I did write a snarky comment and I'm very sorry that I did that, rather than asking for clarification and/or explaining my position a bit more clearly.
I take the view that sharing an SQS queue (or a topic, or a stream...) between two services is OK. It creates a contract between those two services, of course, and part of that contract might be that there is a uniqueId field that the receiving service is going to have to persist in its own database in order to only action each request once, but it doesn't add an excessive amount of complexity or coupling between the services, not for me at least.
I'm still not OK with having two services share the same database, though, even if you're using it for a narrow purpose. For me, the danger is that you end up introducing more coupling than just agreeing on a message format and messaging semantics. I've been burned by that once or twice and I don't want to get burned by it again and I'm pretty happy that it's widely seen as an anti-pattern these days and personally I dogmatically avoid it. I do take your point that the database is going to end up being a SPOF/bottleneck, though, and I don't really have a great answer for that in the general case.
For the most expensive tier: 40 cents per million API requests for plain SQS vs 50 cents per million API requests for FIFO SQS.
Like many AWS Services, it started off pretty basic or limited, but gradually and incrementally improved over time.
I started off pretty skeptical, but over time I've grown to be impressed with SQS and SQS FIFO.
I second the parent post. I've billions of messages through SQS at a previous job and I can remember having issues with SQS availability exactly once (due to a systemic failure involving DyanmoDB where cascading failures took down almost everything (and all of our integrations) that would have hosed us completely if not for our durable-queue/s3-journal[0] at the edge). SQS is simple to use, does what it says, and has very good SDK support. A++++ Would build a business on top of again!
[0] https://github.com/Factual/s3-journal
SQS (standard and FIFO) has another problem that precludes using it in one case I have; the 256KB message size limit. Amazon has a Java-only workaround (Amazon SQS Extended Client Library for Java) that will spool large messages to/from S3 -- so clearly my case isn't unreasonable -- but there is nothing for the general case.
Like you can only send certain character sets. I had a very angry customer, and in the end it was some character (0x07?) that couldn't be sent via SQS.
I had to either be certain that contents would never contain an SQS-unallowed character, or base64-encode all my contents.
It's just polite to write the meaning of acronyms.
Sure you can mix and match and get all confused. But I would stick to that.
SNS and EventBridge is push and 1:many.
Kinesis and MSK is pull and 1:many.