The usual nomenclature is that regular callbacks and async API calls are made and registered in the same process space as the caller. (E.g. passing a function pointer in C, or using “await” in modern languages.)
Webhooks are callbacks received over HTTP from a public API. They may not be programmatically registered at all, e.g. the callback URL may be configured manually in an admin dashboard. So the application has less control over webhooks vs. regular callbacks or async functions.
There are some interesting attack vectors to be aware of if you run a service where users can define webhooks, and your service will will call the user-defined webhooks to notify about certain system events. In my case, a monitoring service which can send notifications by calling user-defined webhook.
* Timeouts: the user can set up a webhook receiver that takes very long to generate a response. Your service must be able to deal with that.
* Timeouts (slowloris): the webhook target could be sending back one byte at a time, with 1 second pauses inbetween. If you are using, say, the "requests" python library for making HTTP requests, the "timeout" parameter will not help here
* Private IPs and reserved IPs: you probably don't want users defining webhooks to http://127.0.0.1:<some-port> and probing your internal network. Remember about private IPv6 ranges too
* Domains that resolve to private IPs: attacker could set up foo.com which resolves to a private IP. It is not enough to just validate webhook URLs when users set them up.
* HTTP redirects to private IPs. If your HTTP client library follows HTTP redirects, the attacker can set up a webhook endpoint that redirects to a private IP. Again, it is not enough to validate the user-supplied URL.
* Excessive HTTP redirects. The attacker can set up a redirect loop - make sure this does not circumvent your timeout setting.
It seems like webhooks have enough corner cases for the sender to require a specialized tool to protect itself from malicious users and to stay performant.
Does anyone have suggestion for such tools/services that they might have used in production?
This is the path I've seen be fairly robust at a few tech companies I've helped sort out this defense for. I've helped write libraries too but the proxy is the easiest approach when targeting many languages.
Those are all great points! At Svix (we do webhooks as a service), we disallow redirects because of what you mentioned above, and because it's also just bad for performance (for both sides) and is most likely a configuration error anyway.
Resolution and timeouts: the aiohttp library for Python is slightly better in terms of letting you configure these things, though it's better to just use a sending proxy that does it all for you and is also located in an isolated VPC to make sure that you're protected.
Disallowing redirects altogether is probably too big of a hammer. There are legit reasons to use redirects (like migration to new versions). A limit to the number of redirects seems ideal -- that's what Twilio does, for example.
Migrating to a new service: you can just update the webhook URL. On the other hand, there are at least a couple of problems with allowing even one redirect: it opens a pandora's box of security implications, and it's a performance penalty that is paid both by the sender and the receiver on every webhook sent. Realistically, 3xx are most likely to be a mis-configurations (e.g. including a trailing slash where one shouldn't) so I think being noisy about it is a great idea.
Not saying that there aren't valid use-cases (e.g. maybe some sort of dynamic webhook receiving), I'm just saying that we made this choice given the above, and we would be willing to change it if it's ever a barrier for someone.
Fair enough, but often consumers that use multiple vendors receive webhooks in the same service; think about going from /v1/webhooks -> /v2/webhooks, they'd have to change the URL for every vendor. Easier to redirect first then update the URLs later. I think it's a reasonable expectation that a HTTP client would honor redirects as long as the usage isn't malicious (like loops etc)
>Domains that resolve to private IPs: attacker could set up foo.com which resolves to a private IP
There's a clever extension to this attack; a naive way to mitigate it is to do a DNS resolution first to verify it's not a private IP and then do the actual request. An attacker can simply return a public IP on the first DNS resolution (with a 0 TTY) and then return a private IP on the second. This is called a "TOCTOU" (time-of-check time-of-use) vulnerability. I've written about this and other security best practices on my blog here - https://www.ameyalokare.com/technology/webhooks/2021/05/03/s...
Yeah, resolving twice is a really bad idea. A good rule of thumb for security: if you think you have a clever hack (e.g. checking DNS twice as a workaround to not being able to patch DNS resolution), it probably isn't so clever as you think.
Great list. Closing layer 3* access (inbound firewall rule of deny-all) helps mitigate against the common webhook attack vectors as well (should still do all the L7 mitigation and best practices alluded to above for a multi-layered approach).
As this page makes very clear, it's actually pretty hard to make a robust webhooks implementation!
What alternatives are there? I've looked at:
* Publishing AWS EventBridge events to other accounts.
* /events instead of webhooks https://blog.sequin.io/events-not-webhooks/
* ???
We built Svix[1] exactly for this reason. We make it super easy for companies to send webhooks reliably. There are only three companies with a "perfect score" on webhooks.fyi's list, we power one of them. ;)
That's awesome and glad to see others making it easier and better to do this right.
Obviously you should add some of your other customers to make sure they get the attention they deserve.
Also, there are other patterns or practices that should be in the list, let us know. PRs are welcome but feel free to email me directly: danger@ngrok.com
Great resource! I’ve encountered most of these problems at some point.
It would be great to see advice about implementations. Things like when to process in a queue and when not to, tying idempotency tokens to database transactions, etc. These are subtle issues, but when done well can help make very robust systems.
Hello all! I'm one of the creators of webhooks.fyi over at https://ngrok.com/
Happy to answer any questions!
First, we started this project when we launched in-product webhook verification. We realized that we were collecting great information, had uncovered some clear and obvious patterns (both good & bad), and knew we could publish the results to help the ecosystem as a whole.
Next, yes it's ridiculously hard to build webhooks correctly. There are so many shortcuts that feel "okay" and you don't really think about until you realize that it didn't quite work as you had planned and now you have a gap. We're hoping the site can help people move more of that thinking earlier and make better decisions.
Finally, we're missing a bunch of webhooks! While we looked at 100+ in our own research, we only had time to add ~50 to this initial pass. If you'd like to add your favorites, pull requests welcome: https://github.com/ngrok/webhooks.fyi
And yes, seriously, you can use ngrok to verify webhooks pretty easily:
Oh, I didn't know about the new --verify-webhook addition to ngrok, that's cool!
Who can I chat with about adding Svix style webhooks[1] to ngrok? We do webhook sending as a service and power the webhooks for many companies, including one of the only three that got a "perfect score" on your list.
Adding Svix will make ngrok automatically support many webhooks services. I'd be happy to chat further my email is my name at the domain of the Svix service.
If only there was a single standard for webhook subscription verification. Cloud services invent their own authentication protocols. Webhook verification in Dropbox is different from that in Trello. The lack of a single standard makes it hard to design a universal incoming webhook service (we faced this problem).
Amen. FWIW, we are open source, and we publish libraries that let anyone sign and verify webhooks, hoping to save people the work and have a unified standard.
The docs mention `Svix-Signature`, but we (and the libs) also support `Webhooks-Signature` which are useful for the generic standard.
The fact that we don't have REST for webhooks is both silly and annoying. Especially since (as you can see on the list in the original post) many companies who implement webhooks end up doing it poorly.
One option is to use a platform like we created at https://hookdeck.com/ where we handle verification with multiple providers and then offer downstream destinations a verifiable request (the Hookdeck signature is a SHA-256 based HMAC digest).
Drop Hookdeck in as a proxy between the origin service and your destination and you get seamless integrations that help with reliability, observability, and recovery (request replays).
This is a fantastic resource! Thank you to the folks at ngrok for putting this together! As this site makes clear: webhooks are harder than they appear. Even just consuming webhooks it's easy to get bogged down dealing with issues around rate limits or recovering from bugs that cause missed events! Missed events being particularly painful with platforms that don't offer replay / retry.
Disclaimer: I work at https://hookdeck.com/ & I shamelessly plug our tool for giving you an awesome developer experience working with webhooks and helping deal with some of the concerns brought up on webhooks.fyi.
And if you are interested in webhooks at large a couple more resources worth checking out is the awesome-webhooks[1] list and the r/webhooks[2] subreddit (I just got ownership of the sub and started dusting it off this week after being neglected for the past few years! Please, come join!)
41 comments
[ 3.0 ms ] story [ 99.8 ms ] threadWebhooks are callbacks received over HTTP from a public API. They may not be programmatically registered at all, e.g. the callback URL may be configured manually in an admin dashboard. So the application has less control over webhooks vs. regular callbacks or async functions.
* Timeouts: the user can set up a webhook receiver that takes very long to generate a response. Your service must be able to deal with that.
* Timeouts (slowloris): the webhook target could be sending back one byte at a time, with 1 second pauses inbetween. If you are using, say, the "requests" python library for making HTTP requests, the "timeout" parameter will not help here
* Private IPs and reserved IPs: you probably don't want users defining webhooks to http://127.0.0.1:<some-port> and probing your internal network. Remember about private IPv6 ranges too
* Domains that resolve to private IPs: attacker could set up foo.com which resolves to a private IP. It is not enough to just validate webhook URLs when users set them up.
* HTTP redirects to private IPs. If your HTTP client library follows HTTP redirects, the attacker can set up a webhook endpoint that redirects to a private IP. Again, it is not enough to validate the user-supplied URL.
* Excessive HTTP redirects. The attacker can set up a redirect loop - make sure this does not circumvent your timeout setting.
My current solution for all of the above is to use libcurl via pycurl. I wrote a wrapper that mimics requests API: https://github.com/healthchecks/healthchecks/blob/master/hc/... (may contain bugs, use at your own risk :-)
Does anyone have suggestion for such tools/services that they might have used in production?
Timeout attacks: instead of failing you'll now be just be paying through the nose. And also with enough scale you can hit lambda limits and fail.
SSRF: you are still as vulnerable unless the lambda is outside your VPC (but then it's the VPC that solved it, not the lambda).
https://github.com/stripe/smokescreen
Resolution and timeouts: the aiohttp library for Python is slightly better in terms of letting you configure these things, though it's better to just use a sending proxy that does it all for you and is also located in an isolated VPC to make sure that you're protected.
Not saying that there aren't valid use-cases (e.g. maybe some sort of dynamic webhook receiving), I'm just saying that we made this choice given the above, and we would be willing to change it if it's ever a barrier for someone.
Fair enough, but often consumers that use multiple vendors receive webhooks in the same service; think about going from /v1/webhooks -> /v2/webhooks, they'd have to change the URL for every vendor. Easier to redirect first then update the URLs later. I think it's a reasonable expectation that a HTTP client would honor redirects as long as the usage isn't malicious (like loops etc)
https://webhooks.fyi/security/one-time-verification-challeng... https://webhooks.fyi/ops-experience/resiliency
Some also don't follow 301s or 302s and will process only specific http responses.
There's a clever extension to this attack; a naive way to mitigate it is to do a DNS resolution first to verify it's not a private IP and then do the actual request. An attacker can simply return a public IP on the first DNS resolution (with a 0 TTY) and then return a private IP on the second. This is called a "TOCTOU" (time-of-check time-of-use) vulnerability. I've written about this and other security best practices on my blog here - https://www.ameyalokare.com/technology/webhooks/2021/05/03/s...
I've also built an egress proxy that prevents such attacks here - https://github.com/juggernaut/webhook-sentry
Same caveat applies, use at your own risk :-)
* examples:
https://netfoundry.io/zero-trust-webhook-security/ (blog)
https://youtu.be/TWohosldLP4 (video specifically for a Lambda function)
What alternatives are there? I've looked at: * Publishing AWS EventBridge events to other accounts. * /events instead of webhooks https://blog.sequin.io/events-not-webhooks/ * ???
[1] https://www.svix.com
Obviously you should add some of your other customers to make sure they get the attention they deserve.
Also, there are other patterns or practices that should be in the list, let us know. PRs are welcome but feel free to email me directly: danger@ngrok.com
At the top of my head: Short Polling, long polling, websockets, response streaming, SSE, SNS, paging API, comet...
There are only 3 companies on webhooks.fyi's list with 3 green checks, and we're powering webhooks for one of those. :)
Looks promising!
It would be great to see advice about implementations. Things like when to process in a queue and when not to, tying idempotency tokens to database transactions, etc. These are subtle issues, but when done well can help make very robust systems.
It's high level, so it doesn't touch everything you mentioned, but it's a good primer.
Happy to answer any questions!
First, we started this project when we launched in-product webhook verification. We realized that we were collecting great information, had uncovered some clear and obvious patterns (both good & bad), and knew we could publish the results to help the ecosystem as a whole.
Next, yes it's ridiculously hard to build webhooks correctly. There are so many shortcuts that feel "okay" and you don't really think about until you realize that it didn't quite work as you had planned and now you have a gap. We're hoping the site can help people move more of that thinking earlier and make better decisions.
Finally, we're missing a bunch of webhooks! While we looked at 100+ in our own research, we only had time to add ~50 to this initial pass. If you'd like to add your favorites, pull requests welcome: https://github.com/ngrok/webhooks.fyi
And yes, seriously, you can use ngrok to verify webhooks pretty easily:
ngrok http 80 --verify-webhook=slack --verify-webhook-secret=[secret]
Who can I chat with about adding Svix style webhooks[1] to ngrok? We do webhook sending as a service and power the webhooks for many companies, including one of the only three that got a "perfect score" on your list.
Adding Svix will make ngrok automatically support many webhooks services. I'd be happy to chat further my email is my name at the domain of the Svix service.
[1] https://docs.svix.com/receiving/verifying-payloads/how-manua...
Libs: https://github.com/svix/svix-webhooks
Docs: https://docs.svix.com/receiving/verifying-payloads/how-manua...
The docs mention `Svix-Signature`, but we (and the libs) also support `Webhooks-Signature` which are useful for the generic standard.
The fact that we don't have REST for webhooks is both silly and annoying. Especially since (as you can see on the list in the original post) many companies who implement webhooks end up doing it poorly.
On the signature front, there's this but it's not enough: https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-mes...
Drop Hookdeck in as a proxy between the origin service and your destination and you get seamless integrations that help with reliability, observability, and recovery (request replays).
Disclaimer: I work at https://hookdeck.com/ & I shamelessly plug our tool for giving you an awesome developer experience working with webhooks and helping deal with some of the concerns brought up on webhooks.fyi.
And if you are interested in webhooks at large a couple more resources worth checking out is the awesome-webhooks[1] list and the r/webhooks[2] subreddit (I just got ownership of the sub and started dusting it off this week after being neglected for the past few years! Please, come join!)
[1] https://github.com/realadeel/awesome-webhooks
[2] https://www.reddit.com/r/webhooks/