> Building polling infrastructure is substantially more complex
That really depends on the setup you're using. There are plenty of server platforms out there where setting up a cron is no more complicated than making a GET route handler.
I'm looking at it from the perspective of a web dev. If you're already building a web application, it is almost always more work to do the polling setup.
In Python, it's either a cron script or Celery beat. In PHP, usually a cron script.
That all means more processes running outside of your web framework, more stuff to deal with, more complexity. Now you have to manage e.g. running a celery process, managing crontabs wherever you deploy...
Even in languages like Elixir where long-running scheduled processes are cheap and native, you still have to write a GenServer in comparison to just using your web framework.
I'm fine with the approach in the article. I actually like that they're giving the option for multiple ways of getting those events. Just please, please don't make /events the only option...
I actually have the opposite opinion. I'm an experienced web developer. Setting up polling is easier for me than setting up an HTTP endpoint.
I typically use a language with an event loop (async/await), so there is always something like `setInterval(poll, 500)`. All that code needs is a connection to the internet. If the server is down for 24 hours I can start it up and it'll read the missed events. I can set up 10 dev environments with just an API key difference in the config. I can batch apply events in a single database transaction, ensuring consistency.
But with a webhook, I need to ensure that my server can accept incoming connections, the external API knows that location. Each dev env needs to replicate this public HTTP server set up. I need to monitor the uptime of the server closely as missing events or erroring on a subset of events could leave my database in an inconsistent state.
> If you need strict consistency guarantees, sure. Otherwise, don't piss off your API consumers, webhooks work just fine.
Ahh, but all problems are queues. What happens if your webhook destination is down and your source system sending queue backs up? What happens if your destination endpoint is up, providing 200s to requests, but throwing away the data quietly due to a mistake during a rollout? Or your webhook source quickly ramps to a volume that is effectively a DoS attack? (These are all problems I encountered in a role at a low code/no code product).
I strongly endorse others in this thread who indicate long polling events is a suitable pattern. You want the best worlds of data durability and consistency through polling functionality, but also as close to real time event firing as possible.
Of course, some APIs you have no control over, and are stuck building robust, chatty polling infra to support because your (paying) users demand access to those APIs. Such is the schlep.
Why do I need two different ways of retrieving events? Wouldn't the data also be available via the usual rest api? If events fail on the consumer end it's their responsibility to resync. Presumably code was written for an initial sync anyway no?
> One idea for Stripe and other API platforms: support long-polling!
It’s great that we’ve went full circle. But make no mistake that this only means one thing: that servers are cheaper than ever. We can now afford to entertain previously extravagant ideas.
Long polling is a lot easier to support now than it was a few years ago, thanks to the wide availability of async server frameworks - Node.js, Python ASGI etc - which make supporting thousands of simultaneous long-polling connections with a single server much less expensive.
also the OP seems to ignore all the issues with webhooks that longpolls suffers or is worse at. eg: you'll lose all those tcp connections if your service is down -- one of the main complaints about webhooks...
Which, if the parts discussed before long-polling was mentioned were implemented, wouldn't be an issue since you'd be starting from the first event following your last-recieved cursor anyway.
I've have had a couple issues implementing Long Polling in the past. At times I've had the firewall or reverse proxy drop client connections if it detects no data transfer. Which meant I had to timeout all the requests every 30 secons or so. Make a new request before the other one ends, and it all becomes messy.
At this point, its honestly just easier to just have a websocket + events endpoint with a cusor both.
I wouldn't recommend long polling for this reason. There are also some security products that have trouble with what appears to be a long file download.
Websockets or server sent events at least signal to intermediaries that a longer term connection will be open.
> It’s great that we’ve went full circle. But make no mistake that this only means one thing: that servers are cheaper than ever. We can now afford to entertain previously extravagant ideas.
We've not come full circle. This is just one blog saying "how about long-polling". While also ignoring that since then we've gained web sockets, HTTP/2 and HTTP/3, each of which make long-polling pointless in three different ways.
But Long-polling doesn't scale in certain situations where you can have infinite webhooks.
I'm sure many people reading this have many idle GitHub repos set up with webhooks into some kind of build server. The repo might see no more than one commit a week.
It makes absolutely no sense for GitHub to long-poll (or websocket) all of these build servers.
(Now what would make sense is for /events to support a way to flip over to a webhook when it's idle. IE, long-poll for a minute, then the next request sends a URL for a 1-time webhook called on the next event.)
It's an incentives issue. Webhooks or for when the service provider is aiming for the bare-minimum functionality with the least overhead. If they have the incentive to maximize your consumption of the data, then they should offer both, and may even overnight a physical copy.
There are lots of reasons to want to immediately respond to an external event besides building an eventually consistent data syncing system. Polling an API endpoint works fine for the latter case, but not much else.
A good platform should offer both of these and more (for example Slack does webhooks, REST endpoint, websocket-based streaming and bulk exports), and let the client pick what they want based on their use case.
Long-polling is the way to immediately retrieve events. It's more efficient and lower latency than waiting for a sender to initiate a TCP and TLS handshake.
A persistent connection has a cost. Your statement may be true in some circumstances but definitely not all. Namely, for infrequent events it is much more efficient to be notified than to be asking nonstop. Sure, the latency is lowest if the connection is already established, but for efficiency the answer is not cut and dry but is rather a tradeoff decision based on the expected patterns.
yeah. a(n improperly configured) firewall is going to start dropping packets if it thinks a connection is idle for too long, so the system never sees an RST and think the connection’s been terminated.
Because what usually happens is the connection is just forgotten from the NAT table. Both sides still see it as connected but the middle box will no longer forward any packets.
It doesn’t just “fall off” the NAT table. Some process in the firewall chose that entry in the NAT table to drop at that moment. It could use the entries from that NAT table to construct RST packets to both sides of the connection. This should be easy and obvious.
What’s the issue with that? This will be discovered as soon as the endpoint tries to send an event, right? At which point the client will see that the connection has been closed, reconnect, and receive the event.
No, the server will try to send an event, and the server will notice the connection has dropped. The client will still have no idea until some sort of timeout is reached, as the client will usually not be sending any data over the connection, as the connection's sole purpose is for the server to send events to the client.
A way to fix this is to use an application-level keepalive (TCP keepalives are generally useless), but then that increases the load on the server and adds a scaling burden.
Meanwhile, unless the event stream is stateful (more overhead!), the client has lost all events since the connection has dropped, and the client can't even be sure when the connection actually dropped.
With webhooks, assuming the callback sending service has a generous retry policy, and the customer's receiving service does not return 200 unless the webhook has been completely processed, or persisted to storage, you won't lose events.
I've been at Twilio for the past 10 years. We recently started offering an event stream service (that customers had been requesting for some time), but it's complicated to get right (on both the server and client side) and difficult to scale, and, frankly, webhooks have worked fine for most customers for a very long time.
> No, the server will try to send an event, and the server will notice the connection has dropped. The client will still have no idea until some sort of timeout is reached, as the client will usually not be sending any data over the connection, as the connection's sole purpose is for the server to send events to the client.
Exactly why mqtt has the ping packet for the client.
Long-polling is usually configured to reset at the both sides after a timeout preferred by the client-side (/events?t=30), long before any network effects kick in, e.g. 10-30 seconds. A client then simply spams requests in a loop, backing off only at http errors. If you have some crazy firewall in between, just set “t” appropriately.
are you sure? specifically, are you sure a persistent connection has _more_ of a cost than repeatedly re-establishing a connection & TLS, etc.?
in terms of energy costs alone, DNS resolution, establishing routes, generating cryptographic session keys, etc. it's definitely not as cheap.
in terms of today's computation power, the "memory" costs of maintaining a connection are minuscule, and the performance "penalties" are negligible.
example: lets say you have 50k event subscribers. if nothing happens, then, aside from a few TCP keepalives (which are not strictly speaking required, and can happen very infrequently), no traffic moves. if instead you have polling once every second, then that's nearly ~13-14 connections a second, each one with at least 4 round trips of traffic. that's a measurable amount of load.
If the webhook events are coming at some sort of a brisk pace, the sender well may be able to reuse an already-open connection. And if they're rather infrequent, is the efficiency or latency likely to be a significant concern?
Websockets can cause issues especially if you're not closing sockets properly, or have too much activity on a small server etc... Livewire for instance accounts for this by just polling every 2 seconds for changes, this is much more performant than keeping 10000 sockets open if people leave open the page/app but don't actually do anything...
Straight long-polling should be avoided, but intermittent polling is a good solution for performance when you don't want to use all your socket bandwidth.
My understanding is that long polling is the thing that will reliably work at scale. Perhaps this changed in the past few years, but I’ve asked various companies like PubNub why they only use long polling and the answer was that there are too many incompatibilities out there in the wild for anything but that.
Server-Sent Events are very reliable. What you might be thinking of is the fact that you probably shouldn't rely just on server push. But that doesn't mean you should use long polling.
You should use normal short polling and Server-Sent Events.
Also it makes no sense to say long polling is more reliable than SSE, because SSE is essentially a non-hacky implementation of long polling.
Someone has to maintain an always-running listener for `/events`. If a server does that, and triggers client calls, we call that webhooks. If a client does that, and triggers internal functions, it's what the op describes. I think that for APIs, `/events` should indeed be the fundamental feature, and "webhooks" should be a nice-to-have service on top of `/events`, for those who don't want to maintain a local subscriber.
One nice benefit of long polling is the built in catch-up-after-a-break functionality: When the client initiates the poll, it tells the server the state it knows about (timestamp, sequence number, hash, whatever), and the server either replies right away if it's different, or waits and replies once it's different.
With webhooks, as in the article, you only get state changes; you need some separate mechanism to achieve (or recover) the initial state.
That's true, although it's also true of any `/events` endpoint that doesn't go back to the beginning of time. Stripe's endpoint only goes back 30 days, so you still need to solve for the initial state unless you have launch all of your desired functionality at the very beginning of your Stripe account!
Hopefully if it's a system like payments where you not only need to know state, you also need to know the time and nature of all transitions, there's a way to query all of that information.
I'm thinking of simpler situations like my source host's CI spinner that seems to get stuck all the time due to missing the ping back from Jenkins about build statuses. In that case it really would be fine to always just say "I think the state is X, please answer me now or in the future whenever the state is other than X." I don't care about anything other than an up to date sync.
Why wouldn't service providers want to offer long polling? It seems a lot easier to build (no webhook registration backend) and no retry logic/SLAs outside of your control. SSE seems so much simpler.
Keeping those connections open probably isn’t super cheap and complicates deploying rolling updates - you’d need to kill all connections when you update and that would require some sort of RPC (so that you only kill those existing connections after they’re done sending in-flight data).
With polling you have to make sure you finish sending data for the current in-flight event. If you don’t then the client doesn’t receive that event, unless you also send them events from the past 30 seconds on first poll.
Quite the opposite. HTTP servers and clients are essentially a solved problem. Massive scale-out, load balancing, retries, authentication, authorization, rolling deploys etc. can all be done out of the box by a hundred different providers. Anything to do with maintaining a large number of open TCP connections is still a massive pain on the server side.
Yep. Consider: you can build a damn reliable & resilient Webhook handler out of a few lines of PHP or Lua, ready to accept a fairly heavy load, zero dependencies, and only default-available packages for most any distro or BSD (anything where nginx or Apache2 with standard modules is available by default) and without tweaking the config at all. You can be live in hours, or even inside a single hour if you want to cowboy it up pretty hard, and despite not taking a lot of care, the webhook-handling part of your service probably won't get you woken up at night with a everything's-on-fire support call (what it does with the data might, of course). Logging? Trivial and standard. Service management? It's the OS' default service definition for a web server daemon, and that's it. Config and deployment? So tiny it'd be nearly no work to document it in a run-once shell script, if you don't have anything fancier at hand. Operationally, it doesn't get much simpler. "Is it working?" checks? You can test it with curl, from any address that's able & allowed to talk to it.
With long-polling, now you're managing a custom daemon, basically. That's a big step down in reliability-by-default, and a bunch more work to do it right.
In either case, you'll be looking at more work if you want to check any kind of log on the other end for missed messages, but that looks pretty similar for either, and not all systems need that level of accuracy (and if they do, they probably need even more and this whole thing is Doing It Wrong)
I think this is also getting at the difference between pub/sub and state synchronization. While one might think they want the former, what they really want is the latter. Get some state and receive updates continuously rather than deal with unreliable stream of updates
On HTTP, pub/sub is eventually guaranteed to drop some messages because TCP/IP itself is not a guaranteed networking protocol (it just makes some promises about failures being uncommon and probably detectable).
If what you want is guaranteed state synchronization, pub/sub alone can't give it to you.
> To mitigate both of these issues, many developers end up buffering webhooks onto a message bus system like Kafka, which feels like a cumbersome compromise.
Kafka solves exactly the issue that the author is complaining about. This is a safeguard to ensure that data isn't dropped in the event of an issue, and provides mechanisms to replay events.
The tradeoff between pushing and polling have been argued since forever.
In other news, mechanics who work with bolts often do so with ratchets. This is a cumbersome compromise, just give me Torx fasteners!
It would if the source was pushing into the Kafka stream directly. It doesn't solve the problem of going out of sync if my code to push to the Kafka stream is entirely down and I miss POSTs.
(And, of course, I don't want Kafka. I want Google PubSub. No, wait, I mean SQS. No, wait, I mean I want zeroMQ. No, I mean....)
The question is: who maintains the queue of events, and pays for it?
Certainly the event producer is in a better position to maintain a queue without missing events, but it also means they need to buffer more data in their queue system to accommodate for your receiver's downtime
Not disagreeing with your point, and I'm sure you already know this, I just wanted to point out (for the benefit of people that don't have other options) that it is possible to build "webhooks" in such a way that you're confident nothing is dropped and nothing goes (permanently) out of sync. (At least, AFAIK -- correct me if this sounds wrong!)
Conceptually, the important thing is each stage waits to "ACK" the message until it's durably persisted. And when the message is sent to the next stage, the previous stage _waits for an ACK_ before assuming the handoff was successful.
In the case that your application code is down, the other party should detect that ("Oh, my webhook request returned a 502") and handle it appropriately -- e.g. by pausing their webhook queue and retrying the message until it succeeds, or putting it on a dead-letter queue, etc. Your app will be "out of sync" until it comes back online and the retries succeed, but it will eventually end up "in sync."
Of course, the issue with this approach is most webhook providers... don't do that (IME). It seems like webhooks are often viewed as a "best-effort" thing, where they send the HTTP request and if it doesn't work, then whatever. I'd be inclined to agree that kind of "throw it over the fence" webhook is not great and risks permanent desync. But there are situations where an async messaging flow is the right decision and believe it or not, it can work! :)
We have a system that pushes loads of messages (as in thousands a minute) and some consumer insists on using there http backend to push the messages to.
There system is down every once in a while for quite some time.
We're using an async queueing solution, but you can't keep those messages forever.
We sometimes have milions of messages for them in there queue's, which take up space...
If all of our consumers had those problems we would have to buy loads of storage..
We're simply dropping messages older than x, and have an endpoint that they can call to retreive the 'latest state of things'.
This way when they come back from a failure, they simply get the latest state, and then continue with updates from our end..
It's far from perfect, but it works really well.
I know the goal for most systems is just to be 'up to date'
Not to get the entire history.
So in most cases you don't need to stash all the messages, you just need to be able to retreive the latest state of stuff...
This misses the problem explained in the article, which is that there are scenarios where events are "acked" but things still go wrong because of bugs.
For example, you rolled out code on the receiver side that did the wrong thing with each message. Now there's no way to replay the old webhooks events in order to reinstate the right behaviour; there's no way to ask the producer to send them again.
The only way around this is to store a record of every received message on the receiver side, too, which the article author thinks is an unnecessary burden compared to polling.
Personally, I think push is an antipattern in situations where data needs to be kept in sync. The state about where the consumer is in the stream should be kept at the consumer side precisely so it can go back and forth.
> "Of course, the issue with this approach is most webhook providers... don't do that "
Embedded systems don't do that for webhooks because they can't (very little RAM or non-volatile storage) but customers clamor for webhooks anyway because it's what their web developers know how to use. So inevitably they're going to lose data but they're only getting what they asked for.
If you want to be 100% sure that you get all the webhooks, the sender could implement an incrementing "webhook ID". If the receiver knows the last webhook ID was 53 and the sender sends one for 55, you can tell one has been dropped. There are some other concerns around that like if 54 has been sent but they arrived out of order, or if they arrive almost simultaneously. Nothing that isn't solvable afaict though.
Of course, then you need a way for the receiver to retrigger or view the webhook if one gets missed, which starts to look like you have to have a polling endpoint anyways, though.
As long as you guarantee delivery to your message queue before acknowledging receipt, you should be golden.
Also, swapping out one messaging system for another is trivial. Pick the one best suited to the environment you're working in, and if that environment changes, changing messaging queues is going to be one the easiest transitions you'll make.
Yeah I was scratching my head reading this article; they're bending so far backwards to avoid the obvious solution that I thought they were gearing up to pitch some competing tech.
> If the sender's queue starts to experience back-pressure, webhook events will be delayed, and it may be very difficult for you to know that this slippage is occurring
I've never before seen anyone try to argue that properly dealing with backpressure is a bad thing. The author's proposed model makes this situation even worse. With kafka, consumers can continue processing the event stream and you can continue to serve reads from your primary datastore. With the author's model the event stream lives in your primary datastore, so if that starts to lock up the blast radius is much larger.
It's a common writing style as of late, set down a premise and solve that premise decisively.
Now, if that premise isn't based in reality, or if it's already been solved some other way, discredit it without giving it too much air time.
A one liner about kafka being cumbersome and then building your own solution, warts and all, doesn't need to exist in the same thought if you've made the reader mentally disregard it as a possible solution.
Are you going to expose your Kafka brokers directly to your integration partners? Are they going to use the Kafka client library and wire protocol to send you data? That’s the thing about webhooks, HTTP is universal and if you’re comfortable exposing anything externally, it’s going to be a web service.
That's a pretty straight-forward design that's widely used, robust, and easy to put together. I've probably done that same workflow 100s of times without issue.
As long as you guarantee the message was pushed to the queue before acknowledging, that will be fabulously reliable. You need to make contingencies for duplicate messages, but that's not usually difficult.
Totally, things can get very reliable if you start processing webhooks asynchronously. Personally I've found it pretty cumbersome and complicated to build the necessary infrastructure in the past. I've been building https://hookdeck.com as a simpler alternative specifically to ingesting incoming webhooks.
It seems to me that you could build a protocol where normal syncs happen through webhooks where each webhook event refers to the event id of an immediately previous event. If the system receiving notifications doesn’t have that event, it makes an API request for all events between the latest event it has and the one it just received.
That is pretty much what TCP does, except that it doesn't "request" missing packets: the client just acknowledges the ID of the last packet it was happy with.
It requires a sequence though, so that you know that the client can know that packet it just received isn't the one it expected, but you could build that with a chain of IDs like you're proposing.
It the system is moving fast this is still somewhat complicated to implement robustly because by the time the "catchup" request to t0 returns, more time has passed and more events have happened, so you still can't resume consuming the webhooks.
To be correct with such a system you have to be prepared to queue the incoming webhook events, do the catchup query, then replay the queued events.
Yeah, if the events can be handled idempotently and this handling properly accounts for time, you don’t have to stop processing webhooks while filling in the gaps.
OK but, it usually matters. If the events represent changes to some object, those changes almost always have to be handled in order if you are to arrive at the same end state as the source.
edit: Also, idempotent is not the right term here. Idempotent just would mean the event could be handled by the receiver multiple times w/o changing the meaning. If you need the events to be applicable out of order, then you need them to be commutative. This is a much more difficult property to ensure and in practice, I am guessing, almost non-existent in deployed webhook APIs.
I think that’s right re. idempotency. However, I think if each webhook is a statement of the state of an object at time t (using some monotonic definition of time, like a Lamport clock), commutativity is trivial: compare the time in the webhook event to the time in your local database, and only update your database if the webhook event is newer.
long polling is not good for serverless consumers. webhooks are great for compute on demand so we should work towards that (it's cheaper coz it's more effecient).
You do not need storage in the producer AND in the consumer, you just need a queue in the producer. Yes, even that is annoying, but the suggested architecture will still lose data if the long poller is down unless there is storage in the producer... so nothing significant has really been solved
I think with a right a strategy for retry then webhooks are great. On my email forwarding app https://hanami.run we offer both method to give user access to their email:
1. webhook with retry up to 7 days.
2. a REST api to fetch all data
Sending webhook out properly require lots of effort, especially idempotent key concept to avoid duplicate data. And control concurency to avoid swarming the webhook endpoint.
So at the end of day, both are require same amount of resources, either on the sender side or the receiver side.
Are events and webhooks mutually exclusive? How about a combination of both: events for consuming at leisure, webhooks for notification of new events. This allows instant notification of new events but allows for the benefits outlined in the article.
Yeah... I'd go so far as to argue that this is the only architecture that should even ever be considered, as only having one half of the solution is clearly wrong.
What about supporting fast lookup of the event endpoint, so it can be queried more frequently?
I think that a combo of webhooks / events is nice, but "what scope do we cut?" is an important question. Unfortunately, it feels like the events part is cut, when I'd argue that events is significantly more important.
Webhooks are flashier from a PM perspective because they are perceived as more real-time, but polling is just as good in practice.
Polling is also completely in your control, you will get an event within X seconds of it going live. That isn't true for webhooks, where a vendor may have delays on their outbound pipeline.
Yea, you're right. I am reading the advocacy as "if you need real-time, then support long-polling."
I see the value in this, but I actually disagree with the article in terms of that being the best solution. Long-polling is significantly different than polling with a cursor offset and returning data, so you wouldn't shoe-horn that into an existing endpoint.
Couldn't keeping a request open indefinitely open the system up to the potential of DoS attacks though? Correct me if I'm wrong, but isn't it kind of expensive to keep HTTP requests open for an indeterminate amount of time, especially if the system in question is servicing many of these requests concurrently?
I don't think the original comment meant long polling (i.e. keeping the connection alive), they meant periodically call the endpoint to check for events.
> Webhooks allows for zero resource usage until a message needs to be delivered.
Doesn't that only work in the case where the server treats each webhook delivery as ephemeral? If you're keeping a queue to allow reliable / repeatable delivery, that's definitely not "zero resource usage", right?
I think that's what the author was getting at, after reading through the whole article. The idea isn't to get rid of webhooks, but provide an endpoint that can be used when webhooks won't necessarily work.
Very similar to how I built my previous application.
1) /events for the source of truth (I.e. cursor-based logs)
2) websockets for "nice to have" real-time updates as a way to hint the clients to refetch what's new
Yes to the combination of both. I worked on architecture and was responsible for large-scale systems at Google. Reliable giant-scale systems do both event subscription and polling, often at the same time, with idempotency guarantees.
Sorry if I'm daft, could you/someone explain why one would want to use both at the same time for the same system?
One thing that makes sense: if you go down use polling so you can work at your own pace. But this isn't really at the same time. When/why does it make sense to do both simultaneously?
There is an inherent speed / reliability tradeoff that is extremely difficult to solve inside one message bus. When you get to truly large systems with a lot of nines of reliability, it starts to make sense to use two systems:
1. Fast system that delivers messages very quickly but is not always partition-tolerant or available
2. Slower, partition tolerant system with high availability but also higher latency (i.e. a database)
The author goes through this in the very first section. Webhook events will eventually start getting lost often enough for the developer to think about a backup mechanism.
Long-polling works if you have a lot of memory on your database frontend. Most shared databases want none of your long-running requests to occupy their memory which is better used for caches.
Even if your message bus has the ability to store and re-deliver events, you might want to limit this ability (by assigning a low TTL). Consider that the consumer microservice enters and recovers from an outage. In the meantime, the producer's events will accummulate in the message service. At the same time, the consumer often doesn't need to consume each individual event but rather some "end state" of some entity or a document. If all lost events were to get re-delivered, the consumers wouldn't be able to handle them, and would enter an outage again. This is where deliberately decreasing the reliability of the message bus and rely on polling would automatically recover the service.
There are other reasons, of course. The author is absolutely correct in their statement, though: whenever a system is implemented using hooks / messages, its developers always end up supplementing it with polling.
This is the way to go and I'd love to see more API's with robust events endpoint for polling & reconciliation. Deletes are especially hard to reconcile with many APIs since they aren't queryable and you need to instance check if every ID still exist. Shopify I'm looking at you.
I definitely don't want to see long polling. In that case I'd prefer a combination of /events and websockets, where websockets can push (or pass via a GET param) the last read event from /events to notify the server which is the last known event.
TFA addresses this by suggesting long-polling as an option rather than the only way to request `/events`
> In our integration with Stripe, it would be neat if we could request /events with a parameter indicating we wanted to long-poll. Given the cursor we send, if there were new events Stripe would return those immediately. But if there wasn't, Stripe could hold the request open until new events were created. When the request completes, we simply re-open it and repeat the cycle. This would not only mean we could get events as fast as possible, but would also reduce overall network traffic.
Webhooks are great for producers of events, and I'd argue that it's too cumbersome for them to provide an '/events' endpoint primary because of scaling. With webhooks, they can offload events at their own pace.
For consumers, I agree with most here that Kafka is certainly overkill. We've gotten away with a very simple architecture to have reliable event consumption. We point all webhooks to an (AWS) API Gateway backed by Lambdas. The Lambdas push the events to an SQS queue (FIFO-queue, if it needs some sort of sequence), and we take our time consuming the events through a very generic poll.
> Webhooks are great for producers of events, and I'd argue that it's too cumbersome for them to provide an '/events' endpoint primary because of scaling. With webhooks, they can offload events at their own pace.
TBF they could do something similar with `/events`, instead of pushing events to a webhooks-sending queue just push them to the events buffer, which could even be a circular buffer just to point out that the essay is completely wrong. TFA is not asking for /events, they're asking for a very specific kind of /events with a large non-drained buffer. Something which would only ever work for low number of events: $dayjob's github integration takes in several events per second.
A proper event stream would be nice though, github's webhooks delivery system is not exactly reliable.
> With webhooks, they can offload events at their own pace.
They can't offload webhooks at their own pace if the two parties want reliable delivery. The server providing the webhook might be experiencing a prolonged outage, in which case the sender needs the ability to buffer the events anyway.
This is just my opinion of course, but I've consumed a great number of APIs and I love when there is. You don't even need /events as long as there's a good index endpoint for the object in question.
For a great number of applications it's not that big of a deal to miss a webhook, and the extreme simplicity that it gives the developer is worth a great deal. With how enormously complex a lot of systems have gotten, I really favor simplicity whenever possible.
Makes me think of how the OpenStack project watches for events to Gerrit. They open an SSH connection to the Gerrit server with the stream-event command. It then stays connected to the Gerrit server and all the events that occur show up in the stream and the program can take actions based on the events it sees.
Gerrit stream-events doesn't solve the issue if the connection is dropped and events occur while disconnected.
I personally (for my hobbyist use case) would prefer the article's /event system over webhooks as webhooks require you have a system that is available on the Internet to receive the webhook. Where having this /event system would not require that.
The article is interesting but is misleading. What the article really says, is that if in average you receive a webhook every second or faster, it is better to poll every second an /events endpoint.
Well the best advantage of Webhooks over polling is that you receive events straightaway no matter the volume of events. If you already know the volume and the volume is high, of course polling is going to be better for everyone.
I like having a one-off command-line app triggered by a webhook. If you miss an event, invoke the app manually. If you pass it the same event twice it won't matter (idempotency!)
IMO the real solution is give me a better transport that RESTful HTTP! As many others have pointed out things like Kafka are built for these kinds of usecases. So often I see people trying to design around the flaws in REST while ignoring that we've had some pretty good progress in the ensuing 20yrs.
Ah, I hear that a lot from customers I work with. A true event-driven system requires both events and webhooks. You will always have apps that only interact via REST so you can't really use streaming architecture here but you can make them more real-time via webhooks.
The article talks about issues with webhooks such as not being reliable if the service goes down and messages are lost. It also talks about developers daisy-chaining multiple services together to put forward a solution which is not robust.
That's why you need a broker that does event distribution, supports multi-protocols (REST, AMQP, MQTT, WebSockets...) natively without any proxies and supports Webhooks. You can push messages to your REST clients and if they disconnect, the messages will pile up in a queue, ready to be consumed when the client reconnects.
Solace PubSub+ Broker does all of this. Disclaimer: I work at Solace.
I tried skimming through the article to attempt and derive the missing elevator pitch, but I saw them reimplement precisely the system they maligned at the start (push notifications with polling).
Did anyone here understand what their value proposition even is?
Did you check the website? The value prop is incredibly clear: they give you a Postgres database you can query with SQL rather than having to either (a) learn and code to a custom API or (b) setup your own sync system to get that Postgres db.
I'm talking about the elevator pitch of their blog post, not of their company. Which are actually two distinct things (they're not describing their service in this blog post).
326 comments
[ 3.3 ms ] story [ 296 ms ] threadIf you need strict consistency guarantees, sure. Otherwise, don't piss off your API consumers, webhooks work just fine.
That really depends on the setup you're using. There are plenty of server platforms out there where setting up a cron is no more complicated than making a GET route handler.
In Python, it's either a cron script or Celery beat. In PHP, usually a cron script.
That all means more processes running outside of your web framework, more stuff to deal with, more complexity. Now you have to manage e.g. running a celery process, managing crontabs wherever you deploy...
Even in languages like Elixir where long-running scheduled processes are cheap and native, you still have to write a GenServer in comparison to just using your web framework.
I'm fine with the approach in the article. I actually like that they're giving the option for multiple ways of getting those events. Just please, please don't make /events the only option...
I typically use a language with an event loop (async/await), so there is always something like `setInterval(poll, 500)`. All that code needs is a connection to the internet. If the server is down for 24 hours I can start it up and it'll read the missed events. I can set up 10 dev environments with just an API key difference in the config. I can batch apply events in a single database transaction, ensuring consistency.
But with a webhook, I need to ensure that my server can accept incoming connections, the external API knows that location. Each dev env needs to replicate this public HTTP server set up. I need to monitor the uptime of the server closely as missing events or erroring on a subset of events could leave my database in an inconsistent state.
Or way simpler if you're not building a web application in the first place.
Polling is chatty when updates are infrequent.
Ahh, but all problems are queues. What happens if your webhook destination is down and your source system sending queue backs up? What happens if your destination endpoint is up, providing 200s to requests, but throwing away the data quietly due to a mistake during a rollout? Or your webhook source quickly ramps to a volume that is effectively a DoS attack? (These are all problems I encountered in a role at a low code/no code product).
I strongly endorse others in this thread who indicate long polling events is a suitable pattern. You want the best worlds of data durability and consistency through polling functionality, but also as close to real time event firing as possible.
Of course, some APIs you have no control over, and are stuck building robust, chatty polling infra to support because your (paying) users demand access to those APIs. Such is the schlep.
It’s great that we’ve went full circle. But make no mistake that this only means one thing: that servers are cheaper than ever. We can now afford to entertain previously extravagant ideas.
At this point, its honestly just easier to just have a websocket + events endpoint with a cusor both.
Websockets or server sent events at least signal to intermediaries that a longer term connection will be open.
We've not come full circle. This is just one blog saying "how about long-polling". While also ignoring that since then we've gained web sockets, HTTP/2 and HTTP/3, each of which make long-polling pointless in three different ways.
None of those have strong support across language frameworks for using them as a client from a server context. Especially http/3.
I'm sure many people reading this have many idle GitHub repos set up with webhooks into some kind of build server. The repo might see no more than one commit a week.
It makes absolutely no sense for GitHub to long-poll (or websocket) all of these build servers.
(Now what would make sense is for /events to support a way to flip over to a webhook when it's idle. IE, long-poll for a minute, then the next request sends a URL for a 1-time webhook called on the next event.)
A good platform should offer both of these and more (for example Slack does webhooks, REST endpoint, websocket-based streaming and bulk exports), and let the client pick what they want based on their use case.
A way to fix this is to use an application-level keepalive (TCP keepalives are generally useless), but then that increases the load on the server and adds a scaling burden.
Meanwhile, unless the event stream is stateful (more overhead!), the client has lost all events since the connection has dropped, and the client can't even be sure when the connection actually dropped.
With webhooks, assuming the callback sending service has a generous retry policy, and the customer's receiving service does not return 200 unless the webhook has been completely processed, or persisted to storage, you won't lose events.
I've been at Twilio for the past 10 years. We recently started offering an event stream service (that customers had been requesting for some time), but it's complicated to get right (on both the server and client side) and difficult to scale, and, frankly, webhooks have worked fine for most customers for a very long time.
Exactly why mqtt has the ping packet for the client.
are you sure? specifically, are you sure a persistent connection has _more_ of a cost than repeatedly re-establishing a connection & TLS, etc.?
in terms of energy costs alone, DNS resolution, establishing routes, generating cryptographic session keys, etc. it's definitely not as cheap.
in terms of today's computation power, the "memory" costs of maintaining a connection are minuscule, and the performance "penalties" are negligible.
example: lets say you have 50k event subscribers. if nothing happens, then, aside from a few TCP keepalives (which are not strictly speaking required, and can happen very infrequently), no traffic moves. if instead you have polling once every second, then that's nearly ~13-14 connections a second, each one with at least 4 round trips of traffic. that's a measurable amount of load.
Straight long-polling should be avoided, but intermittent polling is a good solution for performance when you don't want to use all your socket bandwidth.
You should use normal short polling and Server-Sent Events.
Also it makes no sense to say long polling is more reliable than SSE, because SSE is essentially a non-hacky implementation of long polling.
With webhooks, as in the article, you only get state changes; you need some separate mechanism to achieve (or recover) the initial state.
I'm thinking of simpler situations like my source host's CI spinner that seems to get stuck all the time due to missing the ping back from Jenkins about build statuses. In that case it really would be fine to always just say "I think the state is X, please answer me now or in the future whenever the state is other than X." I don't care about anything other than an up to date sync.
Quite the opposite. HTTP servers and clients are essentially a solved problem. Massive scale-out, load balancing, retries, authentication, authorization, rolling deploys etc. can all be done out of the box by a hundred different providers. Anything to do with maintaining a large number of open TCP connections is still a massive pain on the server side.
With long-polling, now you're managing a custom daemon, basically. That's a big step down in reliability-by-default, and a bunch more work to do it right.
In either case, you'll be looking at more work if you want to check any kind of log on the other end for missed messages, but that looks pretty similar for either, and not all systems need that level of accuracy (and if they do, they probably need even more and this whole thing is Doing It Wrong)
If what you want is guaranteed state synchronization, pub/sub alone can't give it to you.
Kafka solves exactly the issue that the author is complaining about. This is a safeguard to ensure that data isn't dropped in the event of an issue, and provides mechanisms to replay events.
The tradeoff between pushing and polling have been argued since forever.
In other news, mechanics who work with bolts often do so with ratchets. This is a cumbersome compromise, just give me Torx fasteners!
(And, of course, I don't want Kafka. I want Google PubSub. No, wait, I mean SQS. No, wait, I mean I want zeroMQ. No, I mean....)
Certainly the event producer is in a better position to maintain a queue without missing events, but it also means they need to buffer more data in their queue system to accommodate for your receiver's downtime
Conceptually, the important thing is each stage waits to "ACK" the message until it's durably persisted. And when the message is sent to the next stage, the previous stage _waits for an ACK_ before assuming the handoff was successful.
In the case that your application code is down, the other party should detect that ("Oh, my webhook request returned a 502") and handle it appropriately -- e.g. by pausing their webhook queue and retrying the message until it succeeds, or putting it on a dead-letter queue, etc. Your app will be "out of sync" until it comes back online and the retries succeed, but it will eventually end up "in sync."
Of course, the issue with this approach is most webhook providers... don't do that (IME). It seems like webhooks are often viewed as a "best-effort" thing, where they send the HTTP request and if it doesn't work, then whatever. I'd be inclined to agree that kind of "throw it over the fence" webhook is not great and risks permanent desync. But there are situations where an async messaging flow is the right decision and believe it or not, it can work! :)
I know the goal for most systems is just to be 'up to date' Not to get the entire history. So in most cases you don't need to stash all the messages, you just need to be able to retreive the latest state of stuff...
For example, you rolled out code on the receiver side that did the wrong thing with each message. Now there's no way to replay the old webhooks events in order to reinstate the right behaviour; there's no way to ask the producer to send them again.
The only way around this is to store a record of every received message on the receiver side, too, which the article author thinks is an unnecessary burden compared to polling.
Personally, I think push is an antipattern in situations where data needs to be kept in sync. The state about where the consumer is in the stream should be kept at the consumer side precisely so it can go back and forth.
Embedded systems don't do that for webhooks because they can't (very little RAM or non-volatile storage) but customers clamor for webhooks anyway because it's what their web developers know how to use. So inevitably they're going to lose data but they're only getting what they asked for.
Of course, then you need a way for the receiver to retrigger or view the webhook if one gets missed, which starts to look like you have to have a polling endpoint anyways, though.
Also, swapping out one messaging system for another is trivial. Pick the one best suited to the environment you're working in, and if that environment changes, changing messaging queues is going to be one the easiest transitions you'll make.
> If the sender's queue starts to experience back-pressure, webhook events will be delayed, and it may be very difficult for you to know that this slippage is occurring
I've never before seen anyone try to argue that properly dealing with backpressure is a bad thing. The author's proposed model makes this situation even worse. With kafka, consumers can continue processing the event stream and you can continue to serve reads from your primary datastore. With the author's model the event stream lives in your primary datastore, so if that starts to lock up the blast radius is much larger.
Now, if that premise isn't based in reality, or if it's already been solved some other way, discredit it without giving it too much air time.
A one liner about kafka being cumbersome and then building your own solution, warts and all, doesn't need to exist in the same thought if you've made the reader mentally disregard it as a possible solution.
HTTP Endpoint -> Push to message queue (kafka, SQS, etc) -> Acknowledge receipt
That's a pretty straight-forward design that's widely used, robust, and easy to put together. I've probably done that same workflow 100s of times without issue.
As long as you guarantee the message was pushed to the queue before acknowledging, that will be fabulously reliable. You need to make contingencies for duplicate messages, but that's not usually difficult.
It requires a sequence though, so that you know that the client can know that packet it just received isn't the one it expected, but you could build that with a chain of IDs like you're proposing.
To be correct with such a system you have to be prepared to queue the incoming webhook events, do the catchup query, then replay the queued events.
edit: Also, idempotent is not the right term here. Idempotent just would mean the event could be handled by the receiver multiple times w/o changing the meaning. If you need the events to be applicable out of order, then you need them to be commutative. This is a much more difficult property to ensure and in practice, I am guessing, almost non-existent in deployed webhook APIs.
You do not need storage in the producer AND in the consumer, you just need a queue in the producer. Yes, even that is annoying, but the suggested architecture will still lose data if the long poller is down unless there is storage in the producer... so nothing significant has really been solved
1. webhook with retry up to 7 days. 2. a REST api to fetch all data
Sending webhook out properly require lots of effort, especially idempotent key concept to avoid duplicate data. And control concurency to avoid swarming the webhook endpoint.
So at the end of day, both are require same amount of resources, either on the sender side or the receiver side.
I think that a combo of webhooks / events is nice, but "what scope do we cut?" is an important question. Unfortunately, it feels like the events part is cut, when I'd argue that events is significantly more important.
Webhooks are flashier from a PM perspective because they are perceived as more real-time, but polling is just as good in practice.
Polling is also completely in your control, you will get an event within X seconds of it going live. That isn't true for webhooks, where a vendor may have delays on their outbound pipeline.
I see the value in this, but I actually disagree with the article in terms of that being the best solution. Long-polling is significantly different than polling with a cursor offset and returning data, so you wouldn't shoe-horn that into an existing endpoint.
Doesn't that only work in the case where the server treats each webhook delivery as ephemeral? If you're keeping a queue to allow reliable / repeatable delivery, that's definitely not "zero resource usage", right?
1) /events for the source of truth (I.e. cursor-based logs) 2) websockets for "nice to have" real-time updates as a way to hint the clients to refetch what's new
One thing that makes sense: if you go down use polling so you can work at your own pace. But this isn't really at the same time. When/why does it make sense to do both simultaneously?
1. Fast system that delivers messages very quickly but is not always partition-tolerant or available 2. Slower, partition tolerant system with high availability but also higher latency (i.e. a database)
The author goes through this in the very first section. Webhook events will eventually start getting lost often enough for the developer to think about a backup mechanism.
Long-polling works if you have a lot of memory on your database frontend. Most shared databases want none of your long-running requests to occupy their memory which is better used for caches.
Even if your message bus has the ability to store and re-deliver events, you might want to limit this ability (by assigning a low TTL). Consider that the consumer microservice enters and recovers from an outage. In the meantime, the producer's events will accummulate in the message service. At the same time, the consumer often doesn't need to consume each individual event but rather some "end state" of some entity or a document. If all lost events were to get re-delivered, the consumers wouldn't be able to handle them, and would enter an outage again. This is where deliberately decreasing the reliability of the message bus and rely on polling would automatically recover the service.
There are other reasons, of course. The author is absolutely correct in their statement, though: whenever a system is implemented using hooks / messages, its developers always end up supplementing it with polling.
> In our integration with Stripe, it would be neat if we could request /events with a parameter indicating we wanted to long-poll. Given the cursor we send, if there were new events Stripe would return those immediately. But if there wasn't, Stripe could hold the request open until new events were created. When the request completes, we simply re-open it and repeat the cycle. This would not only mean we could get events as fast as possible, but would also reduce overall network traffic.
For consumers, I agree with most here that Kafka is certainly overkill. We've gotten away with a very simple architecture to have reliable event consumption. We point all webhooks to an (AWS) API Gateway backed by Lambdas. The Lambdas push the events to an SQS queue (FIFO-queue, if it needs some sort of sequence), and we take our time consuming the events through a very generic poll.
TBF they could do something similar with `/events`, instead of pushing events to a webhooks-sending queue just push them to the events buffer, which could even be a circular buffer just to point out that the essay is completely wrong. TFA is not asking for /events, they're asking for a very specific kind of /events with a large non-drained buffer. Something which would only ever work for low number of events: $dayjob's github integration takes in several events per second.
A proper event stream would be nice though, github's webhooks delivery system is not exactly reliable.
They can't offload webhooks at their own pace if the two parties want reliable delivery. The server providing the webhook might be experiencing a prolonged outage, in which case the sender needs the ability to buffer the events anyway.
That being said, I've also lobbied for a similar endpoint so I don't get support tickets for "missing data".
Both? Both are good.
For a great number of applications it's not that big of a deal to miss a webhook, and the extreme simplicity that it gives the developer is worth a great deal. With how enormously complex a lot of systems have gotten, I really favor simplicity whenever possible.
Gerrit stream-events doesn't solve the issue if the connection is dropped and events occur while disconnected.
I personally (for my hobbyist use case) would prefer the article's /event system over webhooks as webhooks require you have a system that is available on the Internet to receive the webhook. Where having this /event system would not require that.
Well the best advantage of Webhooks over polling is that you receive events straightaway no matter the volume of events. If you already know the volume and the volume is high, of course polling is going to be better for everyone.
The article talks about issues with webhooks such as not being reliable if the service goes down and messages are lost. It also talks about developers daisy-chaining multiple services together to put forward a solution which is not robust.
That's why you need a broker that does event distribution, supports multi-protocols (REST, AMQP, MQTT, WebSockets...) natively without any proxies and supports Webhooks. You can push messages to your REST clients and if they disconnect, the messages will pile up in a queue, ready to be consumed when the client reconnects.
Solace PubSub+ Broker does all of this. Disclaimer: I work at Solace.
Did anyone here understand what their value proposition even is?