By only profiling the top 1M sites, I wonder if this may be sampling from a set not normally distributed? I suspect the frequency of use of WordPress might go up the further down the list you go — some random blog is less likely to be on the top 1M list yet be more likely to use WordPress.
I suspect this is so. The top million (specifically a subset of that) is where the sites fall whose owners can afford to hire people to build their own custom stuff. Small independent sites seem more likely to rely on something like WordPress.
Not every site is static. Sometimes you need forms, processing, or other back-end business logic. Some people advocating using saas widgets for stuff like cc payments, but you're just farming out the dynamic parts to someone else. That gets hard to maintain with scale and with increased complexity.
I see a lot of people on this site debating the pure merits of a technology, but what others know and can work with is just as important as the technology itself. People know wordpress.
I agree with your remarks. There certainly are sites out there in need of dynamic behaviour. But that long tail has a lot of restaurant websites and such where nothing is really needed.
That is definitely not true. I think that statement is correct roughly for the top 50,000. The fall off in economic viability is fairly abrupt.
There are some websites in certain markets where the earn a disproportionately high amount of revenue per visitor, and these are the exception. But, given the nature of web and the difficulty in how the user actually found that site, a good chunk of their margins may have accrued to another party such as Google.
I said "a subset of that." The top 50,000 is absolutely a subset of the top million. But you don't need to conjecture about that number 50,000. Again I'm saying the top million most-visited sites is where all the profitable sites likely fall. (Not that they're all profitable.)
"Afirman que usan una base de datos de 10 millones de dominios sacadas directamente de Alexa, pero cuando vas a la fuente no existe. Alexa solo proporcionan 1 millón."
Translation:
They [W3techs] say they use a database of 10 million domains taken directly from Alexa, but when you inspect the source, it doesn't exist. Alexa provides only 1 million.
Alexa provides a list of the top 1m... but you can also page through the top sites (100 at a time) using their 'top sites' api... and I don't see a limit on that.
Some other notes:
1) you're not checking subdomains like blog.company.com or paths like company.com/blog
2) if you use something like zgrab you can do 1M site crawl in a couple of hours. Consider checking it out.
I suspect (hope) that means none of the WordPress sites I maintain will be in the tally.
Customizing the metadata, stripping out unnecessary cruft, and moving the user and admin login pages behind IP-restricted firewalls (when possible) are among the first things I do.
I don't think there's an unquestionable way to know if a site is made in WordPress, although you can intuit by some clues: header, meta tag, robots.txt, cookie login, sitemap ...
This is a pretty amazing feat. The top 1 million sites includes many who have the money to afford custom sites and yet Wordpress is still almost 1/5 of sites.
WordPress is the software HN loves to hate, but while it certainly has plenty of warts it's also a very flexible, pliable system for building the kinds of web sites that most people want to build. It'll never win any architectural beauty contests, but market share is driven by utility, not beauty. And WordPress can be very useful software.
The Readme says "Warning that it can take a long time: between 20 to 30 days."
How in the world can it take so long time? The csv file seems to be 24mb in size and the computation performed can't be that advanced. Did the author do something seriously wrong?
I saw that it's actually going through all the website urls and makes a request for each. To be optimisitic, each request takes 1 second, that's 1m seconds. That's ~10 days, so doesn't seem unrealistic it would actually take ton of time.
What? I guess this is a toy program used to learn clojure or sth - it even uses sed for line parsing. A 10-line php script could do the same with a few MB of RAM
I agree, even a Bash script would be more efficient. Everything was born as an exercise to learn Clojure. I can tell you that it actually uses 1.1Gb, but for safety I recommend twice as much. Why does it use so much RAM? Ask Java :) (I am the author of the script)
It'd be interesting to do all that you said (and more) and then determine what's the combined amount, as well as what % of sites do some sort of obfuscation... and why?
I’m wondering why this takes 20-30 days to run all up? Seems crazy for 1M requests. Could one make this a concurrent task and get much greater efficiency?
47 comments
[ 2.7 ms ] story [ 120 ms ] threadI never understood this. I've also been suggested Wordpress by a web guy.
The only time I had Wordpress on a pretty basic website I got a notification it had been hacked, while on holiday!
I've been using static websites ever since.
I suspect a lot of the current small Wordpress sites could be just a well and more securely served by some static thing.
I see a lot of people on this site debating the pure merits of a technology, but what others know and can work with is just as important as the technology itself. People know wordpress.
There are some websites in certain markets where the earn a disproportionately high amount of revenue per visitor, and these are the exception. But, given the nature of web and the difficulty in how the user actually found that site, a good chunk of their margins may have accrued to another party such as Google.
Wordpress isn't there(but 34% is huge).
Why is that ?
[1] https://w3techs.com/technologies/overview/content_management...
"Afirman que usan una base de datos de 10 millones de dominios sacadas directamente de Alexa, pero cuando vas a la fuente no existe. Alexa solo proporcionan 1 millón."
Translation:
They [W3techs] say they use a database of 10 million domains taken directly from Alexa, but when you inspect the source, it doesn't exist. Alexa provides only 1 million.
Edit: confirmed by aws: https://forums.aws.amazon.com/thread.jspa?threadID=167457
How is that a random sample?
Custom themes etc. might choose to omit that so it's not a 100% reliable check
Customizing the metadata, stripping out unnecessary cruft, and moving the user and admin login pages behind IP-restricted firewalls (when possible) are among the first things I do.
How in the world can it take so long time? The csv file seems to be 24mb in size and the computation performed can't be that advanced. Did the author do something seriously wrong?
What? I guess this is a toy program used to learn clojure or sth - it even uses sed for line parsing. A 10-line php script could do the same with a few MB of RAM
I do have to agree this sounds relatively inefficient, especially considering you could just immediately flush the result out to disk.
However, this is an [embarassingly parallel](https://en.wikipedia.org/wiki/Embarrassingly_parallel) problem, and renting out some machines would speed it up.
The script seems to detect a WordPress site by looking for a meta generator tag containing WordPress:
https://github.com/tanrax/calculate-wordpress-usage/blob/5aa...
It's pretty common to remove that meta tag — popular WordPress theme frameworks like Genesis do it by default.
A more reliable test would be to look for additional strings in the source that point to the use of WordPress, such as “wp-content” and “wp-includes”.
A faster way that avoids string searches would be to send an HTTP head request to `/wp-login.php` and check for:
Set-Cookie: wordpress_test_cookie
(/wp-login.php doesn't always appear in the root directory and it's not always accessible to all IPs, but that setup is most common).
https://translate.google.com/translate?sl=auto&tl=en&u=https...