Yeah, this is point 3 under the mitigation section. The only gotcha with this one is that you want to be sure there aren't any vulnerable local services on the proxy server itself. But this should be manageable.
It also may be possible to bind the connect port on the proxy and then use an iptables rule to prevent any connections from that source port from connecting to localhost.
The local port for outgoing TCP connections is chosen randomly in the normal case - and you can't have two TCP connections between the same pair of ports and hosts, so it would be hard to write a good proxy server that let you pick the local outgoing port.
You could use ipt_owner with iptables to block by source user matching the proxy server, however. Or just configure your proxy to block by outgoing IP if it supports this.
Most proxy servers can be configured bind to a specific IP. So if your proxy server has an external IP and an internal IP, configure the proxy server to bind to the external IP. And then use IPTables to block any connections from the external IP to the internal network.
We run all our webhook invocations through our own product (a proxy that allows you to inspect any API call) which gives us this protection and gives us full logging to boot.
This is a good start, but misses two potential problems:
- The 172.16.x.x - 172.31.x.x address space, which is another internal-only one. Chances are that you'll know what your internal addresses look like, but things can change after something is built.
- What about IPv6?
It's really hard to get something right when you only have a blacklist to work with. It's much better to whitelist a few things that are allowed, but then that would limit the utility of Webhooks.
Other internal ranges: Agreed. We should probably put a formal list somewhere (maybe on webhooks.org) that all developers can refer to. Of course, there may be times when you do want internal targets to work as well. I want to say the problem should be something that people in charge of operations need to be aware of, and webhook-sending apps need configuration options to specify ranges. Then it's up to the expertise of the operator to make the right decisions. This doesn't feel very satisfactory though.
IPv6: Good question. I don't know enough about how it handles address spaces.
I have a feeling that IPv6 is going to be a lot more complex than simply blocking FC00::/7. I seem to recall that one of the benefits of IPv6 was that NAT-ing would be a thing of the past and every system would have a public address.
For Stripe's webhooks, we currently use the mechanism described in this post (resolve the IP address, munge the URL we're connecting to, and manually add a Host header), but we've run into a lot of problems.
If your client library supports SNI, you're likely going to send the wrong SNI hostname (we had to disable Ruby's SNI feature because sending an IP address in the SNI field broke a handful of sites), and cert verification generally becomes quite tricky.
We're in the process now of switching to a simple go-based HTTP proxy, which tries to pass through connections as faithfully as possible, but rejects connections to internal IP addresses. Here's the current implementation (though I'm still playing with it): https://gist.github.com/ebroder/ae9299e0078094211bde
This turns out to be way simpler - all HTTP client libraries have good support for proxies, and it doesn't interfere with anything about the SSL handshake (since HTTPS connections go through a CONNECT-based proxy).
We also looked at using Squid or Apache, but concluded that they were much heavier weight than we needed and it was difficult to make them behave as transparently as we wanted (e.g. they always want to add headers and such).
17 comments
[ 3.3 ms ] story [ 47.8 ms ] threadIt also may be possible to bind the connect port on the proxy and then use an iptables rule to prevent any connections from that source port from connecting to localhost.
You could use ipt_owner with iptables to block by source user matching the proxy server, however. Or just configure your proxy to block by outgoing IP if it supports this.
Most proxy servers can be configured bind to a specific IP. So if your proxy server has an external IP and an internal IP, configure the proxy server to bind to the external IP. And then use IPTables to block any connections from the external IP to the internal network.
edit: ok, it doesn't pass the socket.gethostbyname check
Fortunately, links in a browser will always perform a GET, which hopefully is unlikely to do anything terrible on your own computer.
- The 172.16.x.x - 172.31.x.x address space, which is another internal-only one. Chances are that you'll know what your internal addresses look like, but things can change after something is built.
- What about IPv6?
It's really hard to get something right when you only have a blacklist to work with. It's much better to whitelist a few things that are allowed, but then that would limit the utility of Webhooks.
IPv6: Good question. I don't know enough about how it handles address spaces.
http://www.ietf.org/rfc/rfc1918.txt http://www.ietf.org/rfc/rfc4193.txt
I have a feeling that IPv6 is going to be a lot more complex than simply blocking FC00::/7. I seem to recall that one of the benefits of IPv6 was that NAT-ing would be a thing of the past and every system would have a public address.
If your client library supports SNI, you're likely going to send the wrong SNI hostname (we had to disable Ruby's SNI feature because sending an IP address in the SNI field broke a handful of sites), and cert verification generally becomes quite tricky.
We're in the process now of switching to a simple go-based HTTP proxy, which tries to pass through connections as faithfully as possible, but rejects connections to internal IP addresses. Here's the current implementation (though I'm still playing with it): https://gist.github.com/ebroder/ae9299e0078094211bde
This turns out to be way simpler - all HTTP client libraries have good support for proxies, and it doesn't interfere with anything about the SSL handshake (since HTTPS connections go through a CONNECT-based proxy).
We also looked at using Squid or Apache, but concluded that they were much heavier weight than we needed and it was difficult to make them behave as transparently as we wanted (e.g. they always want to add headers and such).
The effect would be similar to what we do with Zurl (also described in the article), but more generalized.