still have the ability to disable the crawler via robots.txt flags
Never rely on robots.txt. Configure it, but do not rely on it. A bot will do what a bot can do. Their bots are trivial to spot so close the connection, bonus if done silently closing the socket and inhibiting RST.
Returning some text files from the server fs is _wayyy_ lighter than letting the bot go through the search, middleware, and db while crawling your stuff.
Angel banning is better, 100% agreed on giving it false data that the scraper builders can't easily detect. Give it seemingly correct and useful but totally wrong answers
I would assume it's exactly what they described, banning something by pretending to still helpfully assist them while silently undermining them instead.
I'm not convinced that it's possible to poison AI. Real data all correlates, fake data is all unique in its own way. I suspect feeding AI fake data is a waste of time.
This implies the solution is every actor in the world willfully complying which doesn’t sound very realistic if you ask me.
Similar to saying world hunger would be solved in everyone who could donated $100 to those in need. Sure it may be true but foot stomping about why isn’t it organically happening is missing the point.
I have been writing bots and scrapers for almost 20 years.
I've enjoyed the arms race with scrapers as they have taught me a lot over the years. Some of them leave me funny messages in my logs and seem to enjoy working around my crude mitigations almost as if it were a puzzle. Which of your scrapers can parse this post without the help of a real browser or headless chrome? [1] I am curious how far the tools have evolved and how someone with your experience instruments them if you are willing to share.
The name Suckerberg I mean Zuckerberg is so strange. That's just raw german:
Zucker = Sugar
Berg = Mountain (or a big hill, but typically it refers to a mountain)
So, a "mountain of sugar" (lots of sugar).
Suckerberg kind of fits better, since "sucker" is not german (berg still is of course; I think scandinavian countries also have berg, though that could be burg, aka castle). Plus, Suckerberg describes the evil behaviour of Facebook better. The USA really has a problem with its billionaires.
I read somewhere once that Sam Altman sounds like a name for a fictional villain made up by Hideo Kojima for Metal Gear Solid, and it will never not make sense.
[resource]+[mountain/stone/field/valley/tree] in German is a very common pattern for Jewish surnames. A quirk of how surnames were adopted in German-speaking regions in the 18th century
Sugar mountain at least makes sense, unlike e.g. wine stone
Blocking by UA is pointless now, meta-externalagent rotates and a chunk comes off non-Meta ASNs. We matched on ASN plus TLS JA4 and that actually stuck.
If that were true, you wouldn’t have people shouting into the wind demanding that billions of people stop using Instagram, WhatsApp, Muse, Facebook, Quest, etc.
Other people don’t share your values. You’ll have to live with it.
You can but you'll probably want to do it with a browser extension or an undetectable webdriver.
I suggest doing it, actually. Could be a fun project. I suggest not making the script make any extra requests on your main account just in case - anything already on the page is free for scraping.
I'm a bit confused why people are freaking out about these term changes - Meta scrapes my website already on a fairly regular basis (even with robots.txt). Is the point of this change to justify more intense scraping?
Color me shocked... one of the most profitable companies in the world, that doesn't contribute anything positive to society, is unethical. Just another day in America that ends in Y.
My suggestion is get your business the f*ck off Facebook.
I would never in a thousand years accept such a one-sided, despicable business relationship as their proposed terms seek to establish.
It's already bad enough they hoover up your customer lists and cross-link to their own - for purposes of gaining insight on individuals who never consented to it.
Now they're trying to use their monopoly over a form of social networking to bully you into arbitrary contract terms for a largely unrelated product.
You don't need Meta as much as you think. And in aggregate, they need their customers far more than those customers need them.
There are much friendlier ecosystems in which to engage with your users.
Regarding the customer lists, I have personal experience.
My bank advertises with Meta and included my account in some kind of upload list (probably Meta Ads Manager). I noticed their logo while poking around in Facebook's settings, under Audience-based Advertising. I was livid.
As far as I'm concerned they had no business telling a creepy company like Meta who I bank with. I certainly never consented to that.
I dug up my original banking agreement - a 5-page document on aging, yellowed parchment that I signed over a decade ago. Nothing in there about them being able to unilaterally change terms without notice. I also printed out their latest, 30+ page standard banking agreement which (after making inquiries) they seemed to think applied to my account.
I brought both in to my bank manager and put them on the table. At a high level, I pointed how ridiculous the size had grown to, and he was sympathetic.
In terms of details, I'd highlighted several portions that pertain to privacy, and we walked through the differences together. I underscored how none of them were needed to provide me banking services, how the benefits they conveyed were solely to TD, and how they were harmful to me as a customer.
In the end, he agreed to formally acknowledge on file that I did not agree to the new terms.
You may disagree over how much effect that will have in practice, but the acknowledgment was not just a bureaucratic exercise. It established very firm evidence that I never agreed to the new contract, or any terms the bank might rely on as a basis for the objectionable practices. It would become material if those practices cause me damage in the future, significant enough to litigate. i.e. They knew they lacked agreement, and continued to provide me banking services.
He also worked with internal departments to toggle all available opt-out flags.
I was fortunate in this case the manager comprehended my concerns and seemed to want to keep me as a customer. I recognize not everyone would necessarily get as far.
It really disgusts me that some of the most degenerate practices of silicon valley are spreading to other industries. With lawyers more broadly adopting some of the most viciously one-sided legal language big tech pioneered.
I for one am not on board, and refuse to be bullied into providing some kind of imaginary consent. I say 'imaginary', because I don't think anyone in their right mind who reads and understands such terms would ever agree to them willingly.
64 comments
[ 0.26 ms ] story [ 12.1 ms ] threadNever rely on robots.txt. Configure it, but do not rely on it. A bot will do what a bot can do. Their bots are trivial to spot so close the connection, bonus if done silently closing the socket and inhibiting RST.
Make of that what you will.
flat files, a tarpit, maybe zip bombs as the sibling suggested. Nothing calculated. I'd go with zipped Brothers Grimm
Not once have I had someone I work for in this capacity even ask about robots.txt
Similar to saying world hunger would be solved in everyone who could donated $100 to those in need. Sure it may be true but foot stomping about why isn’t it organically happening is missing the point.
The internet is built on scrapers though - Google is a scraper, and an opt-out one at that.
I've enjoyed the arms race with scrapers as they have taught me a lot over the years. Some of them leave me funny messages in my logs and seem to enjoy working around my crude mitigations almost as if it were a puzzle. Which of your scrapers can parse this post without the help of a real browser or headless chrome? [1] I am curious how far the tools have evolved and how someone with your experience instruments them if you are willing to share.
[1] - https://altchan.org/ot/20
News at never.
Zucker = Sugar Berg = Mountain (or a big hill, but typically it refers to a mountain)
So, a "mountain of sugar" (lots of sugar).
Suckerberg kind of fits better, since "sucker" is not german (berg still is of course; I think scandinavian countries also have berg, though that could be burg, aka castle). Plus, Suckerberg describes the evil behaviour of Facebook better. The USA really has a problem with its billionaires.
Sugar mountain at least makes sense, unlike e.g. wine stone
https://en.wikipedia.org/wiki/Potassium_bitartrate
Other people don’t share your values. You’ll have to live with it.
I suggest doing it, actually. Could be a fun project. I suggest not making the script make any extra requests on your main account just in case - anything already on the page is free for scraping.
Whilst I assume you meant to say “scrape”, I can’t help but appreciate the poetry of that typo.
I would never in a thousand years accept such a one-sided, despicable business relationship as their proposed terms seek to establish.
It's already bad enough they hoover up your customer lists and cross-link to their own - for purposes of gaining insight on individuals who never consented to it.
Now they're trying to use their monopoly over a form of social networking to bully you into arbitrary contract terms for a largely unrelated product.
You don't need Meta as much as you think. And in aggregate, they need their customers far more than those customers need them.
There are much friendlier ecosystems in which to engage with your users.
My bank advertises with Meta and included my account in some kind of upload list (probably Meta Ads Manager). I noticed their logo while poking around in Facebook's settings, under Audience-based Advertising. I was livid.
As far as I'm concerned they had no business telling a creepy company like Meta who I bank with. I certainly never consented to that.
I dug up my original banking agreement - a 5-page document on aging, yellowed parchment that I signed over a decade ago. Nothing in there about them being able to unilaterally change terms without notice. I also printed out their latest, 30+ page standard banking agreement which (after making inquiries) they seemed to think applied to my account.
I brought both in to my bank manager and put them on the table. At a high level, I pointed how ridiculous the size had grown to, and he was sympathetic.
In terms of details, I'd highlighted several portions that pertain to privacy, and we walked through the differences together. I underscored how none of them were needed to provide me banking services, how the benefits they conveyed were solely to TD, and how they were harmful to me as a customer.
In the end, he agreed to formally acknowledge on file that I did not agree to the new terms.
You may disagree over how much effect that will have in practice, but the acknowledgment was not just a bureaucratic exercise. It established very firm evidence that I never agreed to the new contract, or any terms the bank might rely on as a basis for the objectionable practices. It would become material if those practices cause me damage in the future, significant enough to litigate. i.e. They knew they lacked agreement, and continued to provide me banking services.
He also worked with internal departments to toggle all available opt-out flags.
I was fortunate in this case the manager comprehended my concerns and seemed to want to keep me as a customer. I recognize not everyone would necessarily get as far.
It really disgusts me that some of the most degenerate practices of silicon valley are spreading to other industries. With lawyers more broadly adopting some of the most viciously one-sided legal language big tech pioneered.
I for one am not on board, and refuse to be bullied into providing some kind of imaginary consent. I say 'imaginary', because I don't think anyone in their right mind who reads and understands such terms would ever agree to them willingly.
1792169267 | Meta may use new commercial TOS to avoid millions in AI licensing fees | https://arstechnica.com/tech-policy/2026/10/meta-may-use-swe... | https://news.ycombinator.com/item?id=50120842 | 1 comment
1792224846 | Meta to scrape content from business customers for model training | https://www.theregister.com/ai-and-ml/2026/10/16/meta-grants... | https://news.ycombinator.com/item?id=50129015 | 2 comments
1792233707 | Meta to Every Business User: Nice Website You Got There | https://gizmodo.com/meta-to-literally-every-business-user-ni... | https://news.ycombinator.com/item?id=50129833 | 1 comment
1792255399 | Meta removes language that may have allowed it to scrape news sites | https://www.theverge.com/tech/1013409/meta-removes-language-... | https://news.ycombinator.com/item?id=50132627 | 0 comments
1792317472 | Meta updates data scraping policy following criticism | https://www.mercurynews.com/2026/10/16/meta-updates-data-scr... | https://news.ycombinator.com/item?id=50139568 | 0 comments