Crawlers impact the operations of the Wikimedia projects (diff.wikimedia.org) 13 points by manuelfcreis 1y ago ↗ HN
[–] anotherpaul 1y ago ↗ Seems like a serious issue many sites have nowadays. I really don't want this solved by Captchas everywhere. [–] [deleted] 1y ago ↗ (comment deleted) [–] [deleted] 1y ago ↗ (comment deleted) [–] Handy-Man 1y ago ↗ If it comes to it, just block all known IP blocks of cloud providers. [–] thcipriani 1y ago ↗ There are likely a number of beneficial bots[0] running from cloud providers. And residential proxies are also widely available for very determined scrapers.[0]: <https://stuartgeiger.com/talks/2013-08-03-wikisym-levee-brea...>
[–] Handy-Man 1y ago ↗ If it comes to it, just block all known IP blocks of cloud providers. [–] thcipriani 1y ago ↗ There are likely a number of beneficial bots[0] running from cloud providers. And residential proxies are also widely available for very determined scrapers.[0]: <https://stuartgeiger.com/talks/2013-08-03-wikisym-levee-brea...>
[–] thcipriani 1y ago ↗ There are likely a number of beneficial bots[0] running from cloud providers. And residential proxies are also widely available for very determined scrapers.[0]: <https://stuartgeiger.com/talks/2013-08-03-wikisym-levee-brea...>
6 comments
[ 0.23 ms ] story [ 22.5 ms ] thread[0]: <https://stuartgeiger.com/talks/2013-08-03-wikisym-levee-brea...>