Interesting. If you click on one of the blue circles representing a data center, it shows latencies to the other data centers.
This took me a second to figure out — maybe consider adding a note along the lines of “click to select a data center” on the site?
These aren't even data centers, but aggregates. They're regions, composed of many different bits of networking and compute in various levels of abstraction - dc, edge installation, whatever.
Within these regions there's a lot of variation from zone to zone, so the methodology matters.
I appreciate the effort to collect the data, but I think the rotating globe is an idea that looks cool, but makes the visualization harder to use. If I click on us-east-1, there's a 229ms line to...somewhere that I can't see. Meanwhile, I can't see the latency between us-east-1 and us-east-2.
Perhaps if you selected a datacenter, and it switched to a 2-d projection with that datacenter at the center of the map, it would be better?
Or perhaps augment the visualization with a table?
Author here - You can see the raw data as a table here: https://www.cloudping.co. Sometimes visualizations like this are a careful act of balancing practicality with cool-factor.
The data is really useful, and the globe is visually impressive, but it feels like it'd be more practically useful to have a flat world map that shows all the data centers at once and makes it easier to read the lines without them getting excessively close to each other.
newbie here, you basically loaded d3.js to draw that globe. Is there a tutorial you followed to create those lines dynamically on the globe? Mind sharing some info on how you made this?
Is this just fiber distance between each datacenter? The coloring makes it seem significant, but from the distances it kinda just looked like ever < _km (100ms) was green, everything between _km(100ms) and _km(200ms) was orange, and everything over was red.
I'm not sure what you're wondering here. Of course physical distance is going to be a dominating factor, but this is measuring packet transit times. The speed of light over half a great-circle is only 67ms or so, ot maybe 100ms considering velocity factor in fiber, so clearly there's more to it than just distance. We can talk about what those other things are, but we both know they exist, right?
How did you choose which datacenters to include? For example, eu-south-2 (Spain) is missing.
The reason I know is because I worked on a project that required latency to be under 30ms between datacenters, and we had to use eu-west-1 (Ireland) and eu-south-2.
Turns out that latency is closer to 42ms, mainly because there are no undersea cables between Ireland and the continent (they only go to England, then they have to route across England to get to a cable to the content).
> How did you choose which datacenters to include? For example, eu-south-2 (Spain) is missing.
At the bottom of the page it says: «Data scraped from CloudPing», with the CloudPing dataset linked through. If you click through to CloudPing, you won't find «eu-south-2» in the dataset.
Author here - I just used what was available on https://www.cloudping.co, which is certainly missing a few. The CloudPing GitHub repo has not had a code change in 4 years. Maybe a few new regions have popped up since it was last actively worked on.
I believe it's a combination of a lack of customers and lack of infrastructure. It's a big continent to cover with the necessary fibre capacity, and the market is much smaller for nearby services.
Also what you don't see in things like this, or even a list of datacenter locations, is the relative sizes of the datacenters. After US east/west coasts and Europe, datacenter capacity rapidly tails off. Parts of Asia have plenty but not on the same scale I believe (although I don't know about the Chinese market). The difference in size can be quite a few orders of magnitude between different regions.
Smaller market, less reliable power grids, more challenging heat management, less political stability in many african countries. Also, given their pricing, big cloud vendors AWS are a luxury many local businesses would probably not even consider.
This jives with measurements I've done before. I ended up running a ping setup for a few months from every region to every other region to get these timings. I was using it to calculate what our GQL latencies would look like if the backing servers were in another region or in the same region as a way to start regionalization work. Sadly we had to depend on those latencies so much that it was deemed a non-starter of an approach. Even us-west-2 (home) to us-east-2 took us from p99 300ms to p99 2.4 seconds. That sweet sweet latency reduction.
> Close to the network centrality of the of the internet
Most of the Internet is fractured even though technically publically routable.
E.g., for someone living in China the US isn't anywhere near "network centrality".
If an internet centrality exists, it is somewhere in France or the Netherlands - usually cross-continent traffic goes through there, they have dedicated interchanges for that.
Agreed the network is much denser in Europe than the USA. This is obvious if you've really tried looking for network infrastructure services. The USA is just where a lot more high-level services are, like social media, due to the peculiarities of capitalism. There's no shortage of infrastructure there either, of course.
Data caps are apparently illegal here. This is good for the quality of infrastructure.
And if you have customers somewhere else you want to be in that place, or close to it network-wise.
Although in my experience all the traffic from East Asia and Oceania (Australia, Japan, Korea, Hong Kong, Singapore, etc) to Europe goes through the US. So network-wise, the US is more central.
You are right, the best region that optimises median latency against all internet users over the world is `us-west-3`, which is Paris - I believe. Likely because it has much better latency towards Asia where the majority of internet users are.
I also investigated which two regions to choose for a multi-region setup, which ends up being London and Japan.
My knee jerk reaction was to comment that this is an America-centric thinking (I live in eastern Europe, us-east is not that great), but... After consulting the map, it really looks better than the other options (of course assuming you care mostly about Europe, Americas, and don't want to piss off Asia too much)
Random fact: I did some planning around this for a client a while ago. While measuring the AWS latencies I found I could get approximate latencies (within 10%) by measuring the rough undersea cable length (km) and dividing by 150.
While not overly surprising, it was very consistent.
I read this yeaaaars ago. I'm about to re-read this, but before I do, I think this was the article that installed a little goblin in my brain that screams "TTS" in instances like this. I will edit this if the article confirms/denies this goblin.
What an embarrassing typo! I was thinking of 0.66, and somehow I thought 0.66 = 1/3 (must've been distracted by the "2" in 1/2). I should've written 0.66 or 2/3.
AWS provides latency numbers between regions, AZ's and within an AZ in network manager. Useful to have as a latency baseline and to see if they have any issue.
> AWS provides latency numbers between regions, AZ's and within an AZ in network manager.
AWS also provides dashboards that shows what regions/services are down, and history tells us those are not to be trusted for precisely the same reasons.
Afaik it also requires someone to manually set it to be down on that page. Pretty sure that nobody is entering latency numbers manually every second, but maybe they have a team for that.
Additional - I see the app here just shows good/moderate/bad latency. An actual data table would be useful to many people but not as pretty. Maybe there should be a distributed latency measurement network project.
Its a bit sad, we have one of the Swedish, actual physical buildings for AWS in my town. But of course the traffic does not exit here, but is instead aggregated between the different sites spread around cities regionally. So no sub-ms latencies for me towards that center. I think the traffic basically went a couple hundred kms before turning back here.
I presume your town is too small for it since you called it a town, but AWS also does these things called "wavelength zones" which are as close as possible to certain cellular networks (designed to ride th 5G M2M hype train - self-driving vehicles, etc). Not sure if they do something similar for fixed networks.
Of course, it doesn't actually matter since friends don't let friends use AWS.
Even if they did it's very unlikely they would peer in your town.
Geographic proximity isn't the main factor for peering and even if they did your session may still get terminated at a border network gateway in a big city and your traffic has to travel the same way.
Do you have some kind of an accessibility tool for this? Maybe a whole screen filter that changes colors in a specific way so you can distinguish them?
No I don't. It's actually not a big deal in day-to-day life. People often go "But how the hell can you drive if you can't distinguish red from green at the stoplight?"... in reality it's more nuanced. As another comment already mentioned, perception varies across even among colorblind people. I find it hard to distinguish R/G if the colors are not fully saturated or in low-light situation. Also the brain knows that "red is on top" and "green is at the bottom" at the stoplight and thereby improves the contrast for me. ;-)
My comment was meant to raise awareness of this issue with the author of the tool. Many video games, especially the ones with some kind of HUD, minimap, etc. these days have a color-blind mode.
Yeah I’ve seen colorblind modes in a lot of apps. It is great for those affected. And probably not a huge hassle to implement anyway.
But I was curious if one needs to rely on the application developers to deliver a solution or if there was a generalized filter or whatever that would work always. Maybe like screen readers, those work fine if the app does not do something horrible. But with some help from apps, they perform much better.
Color-blind man here. While I think it’s important to consider color blindness when choosing colors, it’s not actually 8% of men who would have trouble distinguishing the two colors. That number is somewhat lower. Perception of color varies even across colorblind people so just because someone says it works for them doesn’t mean it will work for someone else, and vice versa.
As a quick workaround, you can set a CSS filter on the whole page: Either use dev tools to put a rule `filter: hue-rotate(60deg);` on the `body` element, or simply run `javascript:void(document.body.style.filter='hue-rotate(60deg)')` from the url bar.
In case you are not aware, you can put this sort of thing in a bookmark on the bookmark bar (both FF and Chrom{e|ium}, I assume other browsers too) for easy access. If you don't have the bookmark bar visible hit [ctrl][shift][B] to flip it on (and the same to flip it back off later if you don't want to keep it).
It's sad that this is the top comment for the post. Many people have stopped posting their crappy work online due to harsh comments like yours. There's no easy reply to your comment.
Maybe we should be less critical specially with "Make it fit for my workflow" type comment, and more so if it is built by some random guy in their free time, and not say a project which is asking money.
I think this an uncharitable take – the parent comment is just proposing an improvement that would really help them given their colour-blindness (they also say they like the visualisation). Personally I find part of the reason for putting things on the internet is to allow other people to use them and obtain their feedback.
Every product has flaws which are outside of design scope. Pointing that is unnecessary. If I want feedback on my quick and dirty project, I want it on within the scope of design, not the missing features, bad accessibility etc.
Specially HN crowd is very susceptible to feeling for accessibility comment. Return of "think of poor kids in Africa".
Sorry it came across that way, that was not my intent at all… it was meant as a simple suggestion for a potential low-hanging fruit improvement that would benefit people like me.
Clearly you did not perceive it that way.
I totally understand being frustrated about people demanding workflow changes or huge accessibility features, but this is literally just a color swap that can be done with a touch of CSS it's really not a big deal.
This was not a harsh criticism. Accessibility on the web is important, especially if you want people to actually engage with what you have published.
Color blindness is nothing new, there are freely available color blind friendly color plates. Pointing out to the author that they could make a small tweak to make their work more accessible is good feedback and should continue to be given.
Author here - Thanks for the suggestion Alex. From your perspective, what are some of the best ways you've seen people solve for this in the past? If you have links, please share.
Hi! Thanks for getting back to me, appreciate it. To be honest, I‘m not an expert at all in this topic. I‘d imagine choosing a colorblind-friendly palette (see: https://davidmathlogic.com/colorblind/ ) would be an easy fix. Alternatively, or in addition, you could use dotted/dashed/straight lines to visualize the latency buckets. Might make for an interesting effect?
Also it‘s common to hide this „colorblind mode“ behind a checkbox somewhere. So you don’t have to uglify your product. :-)
the "globe" visualization is good ... but we cna only see half the world at a time ... can we have an option of a flat projection as well ... so I can see all latencies for a region at a glance?
ap-south-2 (Asia Pacific - Hyderabad,India) opened in Nov 2022 seems to be missing from the list?
Azure has Ohio and Texas, but you'll probably not be able to provision all the machines you want so it doesn't really matter.
I think a lot of people are sleeping on the benefits of hosting workloads in these regions. Many finance, banking & insurance companies have already taken advantage. Most of your credit card transactions are handled by data centers that live ~barycentric to the continental US. Much of small US banking tech happens in places like Missouri.
Obviously the biggest contribution to latency is distance. But there's also some close-ish regions with poor latency because there's not fiber running directly between them (for example, over the poles)
Are there an examples of regions which dramatically violate the triangle equality? (That is, where the A--C latency is much worse than the best A--B + B--C latencies)?
Just as a curiosity, could you use that idea to "infer" which data-centers are most likely directly connected by fiber, and show only the likely fiber connections?
Going back decades when a billion US was real money the original NSA (No Such Agency) that essentially no one had ever heard of, including most of the US houses and much of the defence committees that had clearance but not that clearance, had a 4 Billion+ budget for "off-book" satellites.
Black cables are a damn sight cheaper than black satellites.
Yes, although I think it would also clean up the visualization, since you wouldn't have nearly so many lines connecting data centers which actually aren't connected; and it would therefore also be explanatory
> Are there an examples of regions which dramatically violate the triangle equality? (That is, where the A--C latency is much worse than the best A--B + B--C latencies)?
I don't think this would happen at a significant scale, due to how routing works. If taking the "detour" through B is how the ICMP packets get there cheapest, that's the path they will go.
If anything, we could look at where A–C is nearly equal to A–B + B–C and find where such a thing has happened. I suppose it could happen for reasons other than lack of fiber: financially better peering agreements, etc?
I can't speak for AWS specifically, but in my PhD thesis [1] I found a bunch of such examples by using RIPE Atlas probes. Essentially looking for pairs of probes where the RTT between probe A and probe C is larger than probe A-B + B-C.
Now there are some issues with this methodology (all common issues with ICMP/RTT measurements + traffic was not really routed through the "relay" probe), but such pairs do exist.
There’s very few and poor cables in the areas between Russia/Mongolia/India.
AFAIK, the latency from Mumbai to southern Russia (not that far in distance) is surprisingly high. Much higher than from e.g. Frankfurt to Moscow. Don’t know if it’s enough to violate the triangle equality between Frankfurt-Moscow-Mumbai.
222 comments
[ 71.3 ms ] story [ 4411 ms ] threadWithin these regions there's a lot of variation from zone to zone, so the methodology matters.
Bonus: select the nearest data center based on the user’s IP :)
Perhaps if you selected a datacenter, and it switched to a 2-d projection with that datacenter at the center of the map, it would be better?
Or perhaps augment the visualization with a table?
(One of several options, though the best IMO.)
<https://en.wikipedia.org/wiki/Winkel_tripel_projection>
https://en.wikipedia.org/wiki/Azimuthal_equidistant_projecti...
The reason I know is because I worked on a project that required latency to be under 30ms between datacenters, and we had to use eu-west-1 (Ireland) and eu-south-2.
Turns out that latency is closer to 42ms, mainly because there are no undersea cables between Ireland and the continent (they only go to England, then they have to route across England to get to a cable to the content).
At the bottom of the page it says: «Data scraped from CloudPing», with the CloudPing dataset linked through. If you click through to CloudPing, you won't find «eu-south-2» in the dataset.
https://www.submarinecablemap.com
This will show you everything connected to Ireland:
https://www.submarinecablemap.com/country/ireland
Also what you don't see in things like this, or even a list of datacenter locations, is the relative sizes of the datacenters. After US east/west coasts and Europe, datacenter capacity rapidly tails off. Parts of Asia have plenty but not on the same scale I believe (although I don't know about the Chinese market). The difference in size can be quite a few orders of magnitude between different regions.
My best guess is that it is a combination lower demand (vs rest of world), and infrastructure availability (connectivity + power).
I can imagine a bunch of secondary factors too, but this to me sounds like the key broad reasons.
Close to the network centrality of the of the internet, low latency to both the west coast and Europe.
Most of the Internet is fractured even though technically publically routable. E.g., for someone living in China the US isn't anywhere near "network centrality".
If an internet centrality exists, it is somewhere in France or the Netherlands - usually cross-continent traffic goes through there, they have dedicated interchanges for that.
Data caps are apparently illegal here. This is good for the quality of infrastructure.
And if you have customers somewhere else you want to be in that place, or close to it network-wise.
You are right, the best region that optimises median latency against all internet users over the world is `us-west-3`, which is Paris - I believe. Likely because it has much better latency towards Asia where the majority of internet users are.
I also investigated which two regions to choose for a multi-region setup, which ends up being London and Japan.
While not overly surprising, it was very consistent.
Edit: I think it was actually 155
EDIT: mostly, probably, sort of.
Is the math-planation of your random fact basically
(thanks to https://news.ycombinator.com/user?id=Hikikomori for correcting the lightspeed in fibre medium from 3e5 to 2e5 !)
- lightspeed is 2e5 km/s ~ 2e2 km/ms, so/
- length (km) / 200 (km)/ms ~ K length (km) / 200 (km)/ms, so
- latency (ms) ~ K' length (km)
Where K is approximately 1.3 (K' is 1/155) and factors in things like:
- non straight line distance
- networking overhead / switching
- both ways / measurement error
Basically?
1/2 c ~ 150 km/ms in circuit board.
1/3 c ~ 100 km/ms in cable. And...
2/3 c ~ 200 km/ms in fiber?
I'm a bit confused about difference between cable and fiber heh :)
"Through LabVIEW the speed of light in the optical fiber is calculated to be ~ 2.054 x 108 m/s corresponding to a refractive index of n ≈ 1.4606 which is a typical value" https://web.phys.ksu.edu/posters/2009/juma-Adv-Lab-S09.pdf
https://docs.aws.amazon.com/network-manager/latest/infrastru...
AWS also provides dashboards that shows what regions/services are down, and history tells us those are not to be trusted for precisely the same reasons.
Late edit: it would also be cool to see inter-cloud latency.
https://gcping.com/
Of course, it doesn't actually matter since friends don't let friends use AWS.
Geographic proximity isn't the main factor for peering and even if they did your session may still get terminated at a border network gateway in a big city and your traffic has to travel the same way.
This affects about 8% of male population btw, maybe you can add a color-blind mode, very nice visualization otherwise!
My comment was meant to raise awareness of this issue with the author of the tool. Many video games, especially the ones with some kind of HUD, minimap, etc. these days have a color-blind mode.
But I was curious if one needs to rely on the application developers to deliver a solution or if there was a generalized filter or whatever that would work always. Maybe like screen readers, those work fine if the app does not do something horrible. But with some help from apps, they perform much better.
I started putting myself in the shoes of a family member who is in the 8% and now i spend more time trying to pick better color schemes
https://gist.github.com/aclarknexient/c39c83f2f97c3c6b1c307c...
Tested with uBlock Origin on Firefox Mobile.
Disco mode!
(better to use requestAnimationFrame but I'm lazy atm)
Maybe we should be less critical specially with "Make it fit for my workflow" type comment, and more so if it is built by some random guy in their free time, and not say a project which is asking money.
Specially HN crowd is very susceptible to feeling for accessibility comment. Return of "think of poor kids in Africa".
This wasn't posted with that directive.
And if your "design is great" but your implementation sucks then maybe the design sucks too.
Color blindness is nothing new, there are freely available color blind friendly color plates. Pointing out to the author that they could make a small tweak to make their work more accessible is good feedback and should continue to be given.
Those are becoming somewhat more prevalent these days.
Also it‘s common to hide this „colorblind mode“ behind a checkbox somewhere. So you don’t have to uglify your product. :-)
https://venngage.com/tools/accessible-color-palette-generato... also seems nifty
the "globe" visualization is good ... but we cna only see half the world at a time ... can we have an option of a flat projection as well ... so I can see all latencies for a region at a glance?
ap-south-2 (Asia Pacific - Hyderabad,India) opened in Nov 2022 seems to be missing from the list?
I think a lot of people are sleeping on the benefits of hosting workloads in these regions. Many finance, banking & insurance companies have already taken advantage. Most of your credit card transactions are handled by data centers that live ~barycentric to the continental US. Much of small US banking tech happens in places like Missouri.
I found myself going to CloudPing often enough to make a CLI for it
Are there an examples of regions which dramatically violate the triangle equality? (That is, where the A--C latency is much worse than the best A--B + B--C latencies)?
Just as a curiosity, could you use that idea to "infer" which data-centers are most likely directly connected by fiber, and show only the likely fiber connections?
It's highly unlikely there are any non-disclosed undersea ones since they cost rather a lot to lay down.
Going back decades when a billion US was real money the original NSA (No Such Agency) that essentially no one had ever heard of, including most of the US houses and much of the defence committees that had clearance but not that clearance, had a 4 Billion+ budget for "off-book" satellites.
Black cables are a damn sight cheaper than black satellites.
I don't think this would happen at a significant scale, due to how routing works. If taking the "detour" through B is how the ICMP packets get there cheapest, that's the path they will go.
If anything, we could look at where A–C is nearly equal to A–B + B–C and find where such a thing has happened. I suppose it could happen for reasons other than lack of fiber: financially better peering agreements, etc?
Now there are some issues with this methodology (all common issues with ICMP/RTT measurements + traffic was not really routed through the "relay" probe), but such pairs do exist.
[1] https://theses.hal.science/tel-03666771/document (see page 84 for an example; if you can read French :-))
AFAIK, the latency from Mumbai to southern Russia (not that far in distance) is surprisingly high. Much higher than from e.g. Frankfurt to Moscow. Don’t know if it’s enough to violate the triangle equality between Frankfurt-Moscow-Mumbai.