HN Explorer

Ask HN: How can I tell if a site is being rate limited?

48 points by caseyslaught ↗ HN
I live in a country with less than ideal Internet freedom standards and I have a suspicion that a popular publishing site is being rate limited. How can I know for sure? Is it possible to show a smoking gun?

28 comments

[ 2.6 ms ] story [ 61.1 ms ] thread
chefkoch
You could test from vps in another country?
Raed667
My country used to heavily censor internet pages, do finishing attacks on Facebook, Gmail etc..

The easiest way to prove it is by using a VPN or a proxy. Check with different ISPs, also try mobile vs home.

caseyslaught
I can definitely notice an improvement in performance when I use a VPN, however it would be cool to see where things are being delayed. Is traceroute or curl something that could help here?
Retric
Yea, if it’s faster via VPN that’s a strong hint. If you want proof you need to examine the internet between you and the site. If a VPN connection that uses a specific router to to the site is fast, your connection through that same router is slow, and you can connect to some other site through that same router and get a fast connection, then that’s a smoking gun that your connection is deliberately slowed.

That said, it’s not easy to prove it’s your country/ISP doing the slow down vs that website slowing down connections to your country.

rdbell
My company has a residential proxy network that lets you test endpoints with requests from residential broadband ISPs all over the world. Some of our customers are using our network for exactly this use case: https://packetstream.io
sairamkunala
its possible that your ISP has a bad peering connection. Checking with your ISP usually helps understand what is going on.

It could be the bandwidth limitation of your ISP itself.

Sites like Github are slow when accessed from few countries. It's just that their servers are located mainly in the US(afaik).

caseyslaught
I ran a curl test and noticed that the TCP connect time is where things are getting hung up, taking over 80 seconds! Does anyone know if this is indicative of deliberate rate limiting or just a bad peering connection?
toast0
If it takes 80 seconds to TCP connect, something is pretty wrong.

tcpdump / wireshark would likely show lots of retransmits. Traceroute/mtr may give you an idea of where the loss starts. If you can find servers on similar hosting, you can get some idea of if it's targetted or not.

bbulkow
Possibly neither, that is, there are many possible reasons.

for example, stateful firewalls can only track a certain number of connections. If you don't have a public IP, or if they are trying to protect you, it may be necessary to wait for a connection resource to come available. Connections are generally expensive, different from data transfer, for these and other reasons, such as port starvation, syn flood defenses inappropriately targeting you, so many others.

leowoo91
If implemented with respect to the http code, it should say 429 in the developer console.
sfkdjf9j3j
State internet restrictions aren't implemented in the application layer.
bmm6o
That's not the kind of rate limiting he's asking about. He is talking about the data stream being throttled.
crazygringo
Just curious -- if a country didn't want a site to be seen, why would they rate-limit it instead of just blocking connections completely?

Is that a thing countries do, like if they want to "punish" a site while retaining plausible deniability?

Liru
People's attention spans are ridiculously short. If the person is actively looking for the content, sure, that obviously won't work. However, if it's a more casual browser, then going to a site that takes a long time to load between different pages is definitely a way to get them to not visit it any more. This even works on tech-savvy people, as opposed to just blocking a site which will just get them to use a proxy.

> Is that a thing countries do, like if they want to "punish" a site while retaining plausible deniability?

China and Russia do this with a lot of western sites, from what I've heard.

jedberg
It would be too obvious. If you can't get to the site you know it's blocked. If it's just super slow all the time, you won't know why and may just get frustrated and leave.
[deleted]
(comment deleted)
willcipriano
One possible method: Write a script to request the site continuously and see if it fails to respond with any regularity. From there you can start dialing it in and determine what the rate limit actually is. A rate limit should be consistently reproducible unless it's implemented in a clever way.
W4phle_Stomp
Have some fun at the same time: 1) get a vpn account (proton, other) 2) hack your worst enemy's wifi with your favourite tools 3) while outside your enemy's home, boot a linux live distro 4) change your computer's name to your enemy's name 5) connect to the enemy's wifi 6) try to visit all the sites banned by your country 7) insult your country's leader in a public forum using your enemy's name 8) test the speed of the suspected rate-limited site 9) set-up your vpn account settings 10) using your vpn, re-test the rate-limited site 11) subtracting a fair percentage slow down because of the vpn, gauge if there's a significant difference between the two tests 12) turn off computer, go home, enjoy tasty beverage 13) grab some popcorn, go watch your enemy's house from a reasonable distance 14) post video on youtube of your enemy getting busted, using your favourite heading

If that's all too much fun for you, then skip most of it and only do #s 8 to 11 from your own network or favourite hot-spot.

DeathArrow
He should also modify the said enemy's browser cache for this to work well.
caseyslaught
This is very insightful. Tip of the hat to you!
meesterdude
The thing about rate limits is, there are often hard limits. Try querying the site a bunch and see what the data transfer rate is. if it's consistently incapable of going above say, 128K/sec, you've got your rate limit.

Other times, they'll allow for bursts but sustained get rate limited - these can be detected just the same depending on their approach; if it's per connection (and not IP) you may have to find a big file to download.

It also helps to establish that traffic to other sites is not rate limited, and to use 3rd party checkers that check for connection rate across the globe. Or DIY and spin up a bunch of EC2 machines that are geographically diverse.

rozab
I don't know about this, I've heard that limiting is often intentionally inconsistent because this makes people less likely to visit. I think Harlow did some similar stuff with his monkeys.
[dead] bitcrazy
If the government entity really is throttling that site, assuming they are competent, you're unlikely to find conclusive proof that they're actually throttling. Compare this with a full on blackout (i.e. Great Firewall of China, where that's obvious proof of government intervention); the fact that they're allowing some requests to go thru means that the meddling is likely engineered to not be detectable. Best you can find is maybe the speeds are slower when accessed from within the country, but they can always plausibly deny it.