Sometimes I wonder if it’s better to subscribe to a queue.
As a person who uses webhooks from a variety of api’s there are upsides and downsides to them.
I suppose the same could be said for queues - but it strikes me there is a certain amount of “flimsiness” (if I can use that word) in just having a web hook fling packets at your endpoint until it finally dies.
I read about a strategy of having Webhooks, and supplementing with polling an "/events" endpoint that has a historical record of events for occasion backfilling.
Yup, that's one way of doing it (which we support at Svix). Though another great way is "thin webhooks", where you just use the webhook to notify your customers something happened, and then they can use your API to get the exact data that changed.
I'm annoyed by these. Just give me the payload, authenticated of course. I can always just ignore your payload/signature and treat it as a notification to poll you anyway.
Yeah, so give em to me in the event endpoint cause for sure you have them. I will iterate it and do what I need to do.
The key is these things need to have IDs and be idempotent always. Do that, and getting out of a lot of big screw ups needs no further effort/thinking after a reboot or rollback.
The WebSub spec recommends this approach, and I've implemented it, and I really would recommend against doing this.
It adds a tremendous amount of complexity to all of your services, where now all of your services will need to maintain the ability to answer the question 'what was your state at this point in time.'
It makes it extremely difficult to support new webhooks, and honestly in my experience nobody actually ever queries for the historical data anyway. My opinion is that webhooks just aren't the correct approach to take for anything absolutely critical. They're a great addition for stuff outside the critical path where delivery is not essential, and as long as you stick to that paradigm you can dodge these issues entirely.
Re 'thin webhooks' which require hydration:
- subject to either complexity to support historical state (e.g. what was the state at the point in time when the webhook was issued) or race conditions (two things happened, but your API only gets current state)
- honestly none of your consumers want this - they're gonna be like 'why do i have to also query an API why can't you just hydrate the payload in the webhook body'. Guaranteed.
- still probably the correct solution if the data is super private and you want to be extremely cautious about issuing webhooks for it
source: worked exclusively on webhooks and similar eventing stuffs at place you've heard of for a few years
IMO webhooks are best used when your SLA for delivery is "best effort." Anytime you need to guarantee delivery (including guaranteeing the delivered data was correctly processed by the receiver), or guarantee anything about ordering or replayability, they just aren't that great.
> My opinion is that webhooks just aren't the correct approach to take for anything absolutely critical.
What is your recommendation for situations that are absolutely critical? Is the recommendation to just have an /events endpoint?
I'm honestly curious because I have also implemented several wehbook-based systems over the past few years, and definitely seen the downsides, and so I'm trying to figure out the best way to manage asynchronous processing in distributed systems.
> It adds a tremendous amount of complexity to all of your services
But it makes my life so much easier as a user of your service. I admit that it's not worth it for every type of service, but for financial ones it's 100% worth it. I'm trying to keep my ledger balances in sync with yours. I hate having to make a manual adjustment because something got missed.
Having my system heal itself even in the case when we really fucked up, pushed that bad update where we crashed and didn't persist what you said, or cloudflare fucked us and blocked you, or something... there's always something... I really really appreciate that.
I mean, if you don't because it's too complex for you.. okay, I'll have have to add the complexity on my side to compensate. I'm really thankful when I don't have to, and when I'm the event/truth source for others I try my best to do that courtesy for the event consumers.
I feel very strongly that first you should support event polling, and then add webhooks. So your basic method of getting events is to poll for them, and acknowledge each one. This must be idempotent in both directions.
Then you can add webhooks. Then you make the polling frequency slow as its job is really just to heal missed events while the webhooks provide real time notification.
As the service user you can use one or the other or both as you see fit. It's easy to get excellent performance and reliability.
Webhooks only is easier for the event provider but makes it hard on the consumer to recover from errors or outages. And pretty much everyone who offers webhooks fails to fire them or retry them sometimes.
This is the way. It's the direction a number of projects I follow / help have gone, and it's improved a number of things dramatically. Robustness is the biggest win, but you can also turn off a bunch of noisy webhooks (looking at you, Shopify) and not worry that you'll end up missing things.
You technically don't need any acknowledgements for polling. Just make it so you can provide a time range when polling and have each event have a GUID.
I'm an API guy for a large Residential IoT company... we have some platforms that support Webhooks and some that support Azure Event Hub.
I personally prefer the Event Hub (replayability, no HTTP request resiliency required, etc)
The problem with Event Hub is that it requires an SDK and if you've never used it it can be a pain. Also if you don't have an Azure account it's a problem (we typically provide our trusted partners an Event Hub on our subscription).
I also agree with some of the Webhook feedback here (better to also have some endpoint to poll for data/event verification and whatnot). At millions of homes, with a Thermostat being a REALLY chatty device, that's a lot of HTTP requests.
So, long story short, I prefer an event queue model over webhooks.
I think you're right. The answer that works nicely is: place the event in a queue and process the queue with retries idempotently. For robustness, you probably also want good logging and, if possible, an event store for things like replay.
Shameless plug: One of the use cases of our product at https://www.inngest.com is reliably handling webhooks.
We accept a JSON payload, can optionally transform it using ES6 on ingestion, then do a few things:
- Fully type (and typecheck) the payload
- Store the payload in an event store
- Run N functions triggered by the payload, backed by a full idempotent queue (eg. your function can forward the event to your own API, and retry on failure)
This lets you do a lot: you get reliability because we'll continue to retry your API endpoint if it fails. You also get local testing, replay, and historical retries for free because we store the payload. For example, you can `inngest run --replay` from the CLI to replay historic events from your webhooks.
It's complex to build, but good event-driven architectures help with this.
Shameless plug: if you like the development experience of webhooks at Stripe we offer the same experience on our platform at https://hookdeck.com/ which you can drop in to help you add reliability and a better developer experience to receiving webhooks from anywhere.
Edit to add: If you are interested in webhooks I am dusting off https://www.reddit.com/r/webhooks/ and would love to continue the conversation there as well!
19 comments
[ 2.8 ms ] story [ 61.2 ms ] threadAs a person who uses webhooks from a variety of api’s there are upsides and downsides to them.
I suppose the same could be said for queues - but it strikes me there is a certain amount of “flimsiness” (if I can use that word) in just having a web hook fling packets at your endpoint until it finally dies.
There has to be a better way?
1. Keep a live connection to the queue, which is not great operationally (a lot of open connections just idling most of the time).
2. Poll the queue - at this point you're just polling.
3. Send an HTTP ping when something is added to the queue - essentially webhooks.
That's why a queue is not a silver bullet.
The key is these things need to have IDs and be idempotent always. Do that, and getting out of a lot of big screw ups needs no further effort/thinking after a reboot or rollback.
It adds a tremendous amount of complexity to all of your services, where now all of your services will need to maintain the ability to answer the question 'what was your state at this point in time.'
It makes it extremely difficult to support new webhooks, and honestly in my experience nobody actually ever queries for the historical data anyway. My opinion is that webhooks just aren't the correct approach to take for anything absolutely critical. They're a great addition for stuff outside the critical path where delivery is not essential, and as long as you stick to that paradigm you can dodge these issues entirely.
Re 'thin webhooks' which require hydration:
- subject to either complexity to support historical state (e.g. what was the state at the point in time when the webhook was issued) or race conditions (two things happened, but your API only gets current state)
- honestly none of your consumers want this - they're gonna be like 'why do i have to also query an API why can't you just hydrate the payload in the webhook body'. Guaranteed.
- still probably the correct solution if the data is super private and you want to be extremely cautious about issuing webhooks for it
source: worked exclusively on webhooks and similar eventing stuffs at place you've heard of for a few years
IMO webhooks are best used when your SLA for delivery is "best effort." Anytime you need to guarantee delivery (including guaranteeing the delivered data was correctly processed by the receiver), or guarantee anything about ordering or replayability, they just aren't that great.
What is your recommendation for situations that are absolutely critical? Is the recommendation to just have an /events endpoint?
I'm honestly curious because I have also implemented several wehbook-based systems over the past few years, and definitely seen the downsides, and so I'm trying to figure out the best way to manage asynchronous processing in distributed systems.
But it makes my life so much easier as a user of your service. I admit that it's not worth it for every type of service, but for financial ones it's 100% worth it. I'm trying to keep my ledger balances in sync with yours. I hate having to make a manual adjustment because something got missed.
Having my system heal itself even in the case when we really fucked up, pushed that bad update where we crashed and didn't persist what you said, or cloudflare fucked us and blocked you, or something... there's always something... I really really appreciate that.
I mean, if you don't because it's too complex for you.. okay, I'll have have to add the complexity on my side to compensate. I'm really thankful when I don't have to, and when I'm the event/truth source for others I try my best to do that courtesy for the event consumers.
Then you can add webhooks. Then you make the polling frequency slow as its job is really just to heal missed events while the webhooks provide real time notification.
As the service user you can use one or the other or both as you see fit. It's easy to get excellent performance and reliability.
Webhooks only is easier for the event provider but makes it hard on the consumer to recover from errors or outages. And pretty much everyone who offers webhooks fails to fire them or retry them sometimes.
I personally prefer the Event Hub (replayability, no HTTP request resiliency required, etc)
The problem with Event Hub is that it requires an SDK and if you've never used it it can be a pain. Also if you don't have an Azure account it's a problem (we typically provide our trusted partners an Event Hub on our subscription).
I also agree with some of the Webhook feedback here (better to also have some endpoint to poll for data/event verification and whatnot). At millions of homes, with a Thermostat being a REALLY chatty device, that's a lot of HTTP requests.
So, long story short, I prefer an event queue model over webhooks.
Shameless plug: One of the use cases of our product at https://www.inngest.com is reliably handling webhooks.
We accept a JSON payload, can optionally transform it using ES6 on ingestion, then do a few things:
- Fully type (and typecheck) the payload
- Store the payload in an event store
- Run N functions triggered by the payload, backed by a full idempotent queue (eg. your function can forward the event to your own API, and retry on failure)
This lets you do a lot: you get reliability because we'll continue to retry your API endpoint if it fails. You also get local testing, replay, and historical retries for free because we store the payload. For example, you can `inngest run --replay` from the CLI to replay historic events from your webhooks.
It's complex to build, but good event-driven architectures help with this.
Disclaimer: I built inngest :)
https://news.ycombinator.com/item?id=27823109
Edit to add: If you are interested in webhooks I am dusting off https://www.reddit.com/r/webhooks/ and would love to continue the conversation there as well!