Extracing such a list from the generated index only takes a small script and a few hours to download the 200+ GB index file. Which is a lot less than the slightly bigger script and the months/years to download and process 80+ TB of arc files that it previously would have taken to extract all domains.
Anyways, if you want a copy of domains from the index file just send me a mail to the address in my profile.
Usually as a starting point for a crawler when checking statistics about the web.
Crawling is much faster if you don't have to "spider" all the links and check if you've already visited them or not.
With a big enough list, you can just iterate over those domains. (average number of links on a website, how often does javascript framework x vs y get used, how many sites have an HTML5 doctype yet, ...)
You can get the top million sites from Quantcast in a download for free. Alexa used to also offer something similar, but I don't see it on their site any longer (they try to sell their data through aws, so that's probably why).
A number of the sponsoring orgs will grant you access to their zonefiles if you're willing to do a little paperwork (HelloFax is awesome) and have a valid reason for access. Granted it's not the complete list of registered domains (those without nameservers on file won't show up), but it's pretty close.
19 comments
[ 5.4 ms ] story [ 46.7 ms ] thread*Edit: The problem seems to be fixed now.
Anyways, if you want a copy of domains from the index file just send me a mail to the address in my profile.
Crawling is much faster if you don't have to "spider" all the links and check if you've already visited them or not.
With a big enough list, you can just iterate over those domains. (average number of links on a website, how often does javascript framework x vs y get used, how many sites have an HTML5 doctype yet, ...)
The list of sponsoring orgs for each TLD is at: http://www.iana.org/domains/root/db