I can only imagine malicious uses for this. "Sorry, you're no longer allowed to access Google, Facebook, Twitter, or Wikipedia." Not that that is entirely a bad thing.
This is by the same guys as the million short search engine (Google minus the top million). Probably good to use this in combination with the dns to find things that are not just broken links.
Yes, it's not very useful without the search engine. Which I have tried just now and the experience was... frustrating.
Why? Cause it did not return any results for any of my queries (e.g. "hello"). I thought, no, it can't be broken - must be something on my end. I opened it in Chrome incognito window and it worked! Aha, "location based", I thought. And I was right - they use IP location by default to localize results.
I know it is a common practice now, sadly, popularized by Google, et al - but it sucks! I deal with this each time I travel. Can you, please, prioritize this and first look into my actual request header, which explicitly says that I prefer response in English? Thank you very much.
If you use these servers you may see a lot less advertising. That's because more than a few of the top 100/1000/10000/1000000 sites are actually just ad servers, assuming Million Short is using Alexa as the source. And because they appear in the top Alexa list one might guess those particular ad servers serve a significant share of the internet's advertising.
Another thought is you could potentially use these as general purpose DNS servers; e.g. they are all Amazon EC2 I believe so with respect to the DNS-based geolocation efforts of many websites, you'd be treated as if coming from the location of whatever region the datacenter is in. Just add the top 100/1000/10000/1000000 sites to your HOSTS file.
so with respect to the DNS-based geolocation efforts of many websites, you'd be treated as if coming from the location of whatever region the datacenter is in
wut? how does dns based geolocation work? you seem to be saying that sites assume you share the physical location of your dns server?
No. What sites assume is that you are located (at least in a regional sense) near the (recursive) DNS servers you use. For example, that's how many CDN's work.
Note: It's certainly possible to share the exact same location (or interface, to be more precise) as your DNS server. I run my own personal DNS cache on localhost. It's not unheard of. I'd guess there would be a few other readers of HN who do this as well.
I think that's wrong: there's no way for a site to know what DNS servers I use. Instead, they use a reverse lookup from the apparent IP I'm connecting from... that is available to them, and is unrelated to my DNS servers.
Or can you supply a reference/explanation for how'd they'd know my DNS servers?
what he (i assume?) is saying is that when a cdn wants to supply data to you, they want to do so from a server as close as possible.
now, typically, dns is configured so that your dns requests go to servers that are "near" you on the network.
so, say you're looking for google.com. the dns server near you will be configured to say that google.com is a server near you (and near the dns server). effectively they are inferring location from dns lookups (and then providing you with a nearby source).
this is completely different to looking up the requesting ip in a database which is what i originally assumed was being discussed (hence my confusion and perhaps yours). but it (this process for choosing cdn providers) does seem to be called geolocation by cdn people (just google "cdn geolocation").
Run 'dig +short whoami.ultradns.net' in your terminal. You'll get back the IP of the DNS server you are using.
Your ISPs recursive DNS servers send off a query to the sites authoritative servers, which in turn look at the source IP address. That's how they know. (Source: I've built a few CDNs)
Sure, but that only applies to the CDNs who have been careful to send diferent answers to different places, for sites relying heavily on such CDNs.
A standalone (single-IP) site not using a CDN, or even a site that uses a CDN solely for bulky static assets, has no direct way to query what DNS servers a client used, other that the fact that those servers resolved the request Host to the listening IP. (Perhaps it could probe by attempting a number of resource loads from hostnames that resolve differently based on different major DNS sources, but that's be obtrusive and require constant maintenance.)
Especially in the 'long tail' (of not-top-1-million-sites), I'd expect the non-CDN or CDN-only-for-big-assets setup to predominate, and so any geographic adaptation would be more likely based on IP lookups (via a database like from MaxMind), rather than CDN inference.
Or is there some other way even static-asset CDNs somehow communicate back their geography-sensing back to primary sites?
A "standalone" site can get the IP address of the users DNS server by doing an AJAX request to http://[random].ip.yourdomain.com/. Your DNS server responds to requests for *.ip.yourdomain.com with the IP of your webserver and stores the requesting IP address in a database using [random] as its key. Finally a script on your website fetches the IP from the database when it gets the request and prints it out wrapped in a cute little JSON wrapper.
You can see an example of this at http://entropy.dns-oarc.net/test/
Clever, but it seems to me that might only coarsely reveal some global service my DNS server falls back to, NOT the server my local machine consults first.
Is this technique, including running your own authoritative DNS server and remembering every unique lookup, commonly used to geolocalize individual web visitors? Or do servers more often just look up the originating IP? My conjecture is that the latter dominates.
OK, a typical website acting alone can't know what DNS servers my local machine is configured to contact, and furthermore doesn't use such DNS sensing to geo-localize its content (the claim I was responding to).
But, with the technique you've described, a website coordinating with a DNS server can probe to learn one of the DNS servers that gets consulted (directly or indirectly) by my machine. Got it. Neat and useful trick.
You can get into the top million on Alexa with a minuscule amount of traffic so you'd be extremely limited. Losing the top 1000 would probably be a more interesting experiment for mid/long term purposes.
How did they do their ranking? Is it based on a web crawl, dns stats, other? Is their list of the top million domains public? I would love to see the data.
A great refinement would be: on the error page, suggest alternate sites with similar content that are still reachable.
Or even: for the exact URL visited, suggest the one page in the remaining long tail that's most like (by some text/semantic measure) the originally-requested page. (Or even: redirect automatically to that page.)
38 comments
[ 4.4 ms ] story [ 69.2 ms ] threadIs changing DNS easier or more difficult than editing a HOSTS file?
nameserver 1.2.3.4
http://www.millionshort.com/
The search engine was discussed on HN before: http://news.ycombinator.com/item?id=3910304
Why? Cause it did not return any results for any of my queries (e.g. "hello"). I thought, no, it can't be broken - must be something on my end. I opened it in Chrome incognito window and it worked! Aha, "location based", I thought. And I was right - they use IP location by default to localize results.
I know it is a common practice now, sadly, popularized by Google, et al - but it sucks! I deal with this each time I travel. Can you, please, prioritize this and first look into my actual request header, which explicitly says that I prefer response in English? Thank you very much.
Another thought is you could potentially use these as general purpose DNS servers; e.g. they are all Amazon EC2 I believe so with respect to the DNS-based geolocation efforts of many websites, you'd be treated as if coming from the location of whatever region the datacenter is in. Just add the top 100/1000/10000/1000000 sites to your HOSTS file.
wut? how does dns based geolocation work? you seem to be saying that sites assume you share the physical location of your dns server?
Note: It's certainly possible to share the exact same location (or interface, to be more precise) as your DNS server. I run my own personal DNS cache on localhost. It's not unheard of. I'd guess there would be a few other readers of HN who do this as well.
Or can you supply a reference/explanation for how'd they'd know my DNS servers?
now, typically, dns is configured so that your dns requests go to servers that are "near" you on the network.
so, say you're looking for google.com. the dns server near you will be configured to say that google.com is a server near you (and near the dns server). effectively they are inferring location from dns lookups (and then providing you with a nearby source).
this is completely different to looking up the requesting ip in a database which is what i originally assumed was being discussed (hence my confusion and perhaps yours). but it (this process for choosing cdn providers) does seem to be called geolocation by cdn people (just google "cdn geolocation").
Your ISPs recursive DNS servers send off a query to the sites authoritative servers, which in turn look at the source IP address. That's how they know. (Source: I've built a few CDNs)
A standalone (single-IP) site not using a CDN, or even a site that uses a CDN solely for bulky static assets, has no direct way to query what DNS servers a client used, other that the fact that those servers resolved the request Host to the listening IP. (Perhaps it could probe by attempting a number of resource loads from hostnames that resolve differently based on different major DNS sources, but that's be obtrusive and require constant maintenance.)
Especially in the 'long tail' (of not-top-1-million-sites), I'd expect the non-CDN or CDN-only-for-big-assets setup to predominate, and so any geographic adaptation would be more likely based on IP lookups (via a database like from MaxMind), rather than CDN inference.
Or is there some other way even static-asset CDNs somehow communicate back their geography-sensing back to primary sites?
A "standalone" site can get the IP address of the users DNS server by doing an AJAX request to http://[random].ip.yourdomain.com/. Your DNS server responds to requests for *.ip.yourdomain.com with the IP of your webserver and stores the requesting IP address in a database using [random] as its key. Finally a script on your website fetches the IP from the database when it gets the request and prints it out wrapped in a cute little JSON wrapper. You can see an example of this at http://entropy.dns-oarc.net/test/
Is this technique, including running your own authoritative DNS server and remembering every unique lookup, commonly used to geolocalize individual web visitors? Or do servers more often just look up the originating IP? My conjecture is that the latter dominates.
Is it used to geolocate users? No. Is it used to route traffic in most major CDNs? Yes. The two are completely different use cases.
I think this is getting way out of scope for HN. If you are still curious how this stuff works I can email you directly if you'd like.
But, with the technique you've described, a website coordinating with a DNS server can probe to learn one of the DNS servers that gets consulted (directly or indirectly) by my machine. Got it. Neat and useful trick.
http://s3.amazonaws.com/alexa-static/top-1m.csv.zip
Or even: for the exact URL visited, suggest the one page in the remaining long tail that's most like (by some text/semantic measure) the originally-requested page. (Or even: redirect automatically to that page.)
I might be wrong, but it might be an easy way to keep users on the "bright streets" of the Internet instead of wandering down malware-ridden alleys.