Maybe they should allow the Internet Archive access to their article after a week or 2.
But I think this will hurt them as time goes on more then help. IIRC, one news org blocked free access and their revenue fell. I think that was in Australia.
But seems they are using AI as the reason. So allowing after a week will not avoid AI access.
But, what happens of an AI Company subscribes to the news site using a person's name (or a fake name) ? They will still get the article and avoid hassles.
There really should be a micropayments setup on the internet that's not advertising based. Let these models pay a nickel to read the article, covered by the multi trillion dollar AI blank check.
For as long as there's free data to be downloaded, there will be an AI to ruin a public good. Why pay when you can just have your scraper generate a million trial accounts?
A pay wall at the news site would just bankrupt the internet archive, and a pay wall at the internet archive will kill most public interest in the service.
Apologies for the self-promo. Downvote and I'll know not to do it again.
This trend of outright banning the Internet Archive has me extremely worried. I fear a future where news articles are memoryholed, and no one can remember exactly what was reported and how sensational it all seemed.
I've been working on this project [0] for a while. Originally, I started with a tool that would allow people to snapshot webpages in their own browser, and they could selectively share their snapshots. Then by consensus, everyone could understand what exactly had changed, and they could draw their own conclusion about why.
While working on it, I realized that an authoritative answer to "what did it look like on $DATE" can't be produced by a no-name company. It's gotta be a non-commercial entity that's got a track record of integrity. The dream would be to allow MemoryHole customers to submit their snapshots to the Internet Archive (or other non-commercial entity). It's definitely a copyright nightmare - so no clue how this could work.
Perhaps I imagined this, however some months ago on X someone pointed out a historical article on dailymail.co.uk related to Prince Phillip and Epstein had been scrubbed, which likely would be intelligence or through D-Notices, but where instead of showing a 404 page would redirect to an article that was similar but benign. I checked the URL on the Wayback Machine and it turned up zero results, but not even the redirected article, however the user on X had screen grabbed the original, which everyone was reading and commenting on. As of 21st May I can't find this discussion on X and Grok denies it ever existed. This is a "maximally truth-finding" AI, so I must be mistaken. Perhaps the Internet Archive cannot be trusted, so this is why 340 local news outlets need to limit access.
I think its bound to happen and in some ways it a good thing to happen too. The current state of AI affairs is a lot about outrightly selling some one else's intellectual property. The short term incentives are eroding the trust and goodwill among the natural knowledge actors.
The next natural thing to happen would be privatization or consolidation of the internet itself. Its already happening in the form of grabbing and consolidating IPv4 addresses.
That's a real shame. I am involved with some history-related projects and the number of websites which go offline is huge, and the wayback machine is incredibly helpful for unearthing these dead sites.
It is not hard to imagine a future in 50 years time where a huge percentage of this content is lost forever, or at best incredibly hard to find.
I wonder what’s the motivation though. Subscriptions? At a minimum they could limit based on recency. It’s “news” after all so having a number of days of delay for archival would solve most issues, I assume.
Some countries have national archives that all published material must by law be submitted to, including material published online. I know at least Sweden and the UK has that. This will be available for researchers, though usually you have to physically travel to the archive to access the data, so not as convenient as IA.
(It is worth noting that at least in Sweden "published" here has a very specific meaning, that doesn't include personal websites etc, but it does include news outlets.)
Unfortunately the IA itself has become way less usable as their aggressive anti-bot protection means that actually doing any kind of (manual) explorative research, as opposed to pulling an individual website that you already know the exact URL of, is more than likely to get you temp banned.
I gave a talk about this when I worked for The Archive. There was an article in Scientific American about how the average lifetime of a page on the net before it 404s is about 100 days. That article is offline now and we accessed it through the wayback machine.
My own last project before I left was to ingest records from crawl dumps from the defunct cuil.com website. About 200 TB of stuff that brought back 60 billion URLs.
The nature of the internet has changed and it's become an ephemeral place for many people where you just through things in and others mine it as "data".
Ugh - our local paper used to have a wonderful archive, that got limited and locked down after the pandemic. IDK if they got bought out, but it's a real shame, I think some of the problem is things that used to be public information (birthdates, families, names) in hospital admissions (I found old entries of my friends parents and my own for being "in the hospital" in the newspaper for example).
I'm sure that plays a role, but still... This obviously is about cost and money making, not security as a whole (ime)
I see archive.is / archive.today used a lot more for that.
First of all it's a lot faster than archive.org, second of all they are much better at bypassing the paywalls. In some cases the operator even seems to have taken up a subscription or at least a user account (in some cases you could see there is a user logged in). They're really clever at this.
For this site the main purpose is indeed the bypass. For the internet archive it's not.
And yeah it could easily be fixed by just publishing the pages a month later or so.
Of course they are, because they are not primarily concerned with the reporting of noteworthy events. They are most worried about profit with the secondary goal of reporting but only insofar as it serves the first goal. This is a wider trend across many industries.
Obviously, a business needs to have an income but it's becoming more common for businesses to function first and foremast as revenue generators and the thing that enables that is only seen as a means to an end. When the quality of the product/service and it's function as a revenue generator diverge, the product/service will always take 2nd chair.
Maybe we could argue that the primary product is the revenue, especially when there are investors involved who are looking for big returns.
> "as profit margins for news thin, it’s only become more important to news publishers to protect their intellectual property."
So their argument is that people who would be paying money at their paywalls, are going to IA to get their news for free? And if they can thwart those people, they'll show up and become monthly subscribers?
I am vaguely sympathetic to newspapers as a concept, though the actually owners of approximately all of them are just PE companies looking to extract maximum profit from this dying industry, not really trying to prolong their existence.
But I think everyone who is interested in subscribing to their newspapers' paywalls already has subscribed. Those of us who bypass paywalls with that archive.whatever site, or apparently IA (I have never tried it for this purpose) are doing so because there is zero chance we're going to (recurringly!) pay the asking price for some random out-of-town newspaper, The Verge, Bloomberg, whatever. It's fair game to call us immoral for that decision, but if (and it's a big if) this move prevents more people from being able to bypass a paywall, I predict zero incremental dollars will go to the news publishers.
It's interesting how much we lost with the end of the advertising model (though likely its death would arrive with agentic access anyway). An unsurprising reaction to that was the advent of the widespread paywall. And in a world where every paywalled article on social media, including HN, is on an archived paywall-bypass site there was going to be a natural cat-and-mouse game. The distributed payment model of online advertising was surprisingly effective. No single person was worth very much but the aggregate of attention had a probabilistic conversion that enabled a sufficient ecosystem of news.
Now most of those who spend money get access to relatively good news in comparison to those who don't. The interesting thing is that if you model the utility of a customer base as trifactorial (subscriptions, ad-supported, influence-ability) and you set ad-support to near zero you're left with this situation where those with no ability to pay are now overwhelmingly useful to the website provider only as an influenceable base.
"If you're not paying, you're not the customer, you're the product", we used to say[0]. It turns out that's true, but if you can't pay by looking at ads, you will pay by the actions you take when you believe what the actual customer wants you to believe.
0: Though sometimes you do pay and you're still "the product" haha!
It's clear that people place some non-zero value on archival content. It should be unsurprising that news outlets also place some non-zero value on it. Given that they place some non-zero value on it, it is unsurprising that they do not give it away for zero. Disagreeing with their estimation of the value is understandable, but surely it's easy to see why most news outlets do what they do.
Do these major publications charge per article? They should, but they don't. So their whole sell is that in aggregate (so access to all, including old articles) they are worth paying monthly for.
Not trying to be paranoid, but losing recorded history raw as it was originally reported could lead to quick AI-assisted rewrites in the archives of news outlets to fit whatever narrative of the "jour" is in fashion/that powerful of those times want. We are already seeing it in new editions of some old books that suddenly miss some currently controversial topics. History is written by the victors could change to history is rewritten by the (current) victors, as they see fit.
why not just agree on a release date?
while i enjoy circumventing moneyfences, i understand the wallmakers do not. i think this would be an easy deal, if someone just laid it on the table.
46 comments
[ 2.7 ms ] story [ 65.9 ms ] threadBut I think this will hurt them as time goes on more then help. IIRC, one news org blocked free access and their revenue fell. I think that was in Australia.
But seems they are using AI as the reason. So allowing after a week will not avoid AI access.
But, what happens of an AI Company subscribes to the news site using a person's name (or a fake name) ? They will still get the article and avoid hassles.
A pay wall at the news site would just bankrupt the internet archive, and a pay wall at the internet archive will kill most public interest in the service.
This trend of outright banning the Internet Archive has me extremely worried. I fear a future where news articles are memoryholed, and no one can remember exactly what was reported and how sensational it all seemed.
I've been working on this project [0] for a while. Originally, I started with a tool that would allow people to snapshot webpages in their own browser, and they could selectively share their snapshots. Then by consensus, everyone could understand what exactly had changed, and they could draw their own conclusion about why.
While working on it, I realized that an authoritative answer to "what did it look like on $DATE" can't be produced by a no-name company. It's gotta be a non-commercial entity that's got a track record of integrity. The dream would be to allow MemoryHole customers to submit their snapshots to the Internet Archive (or other non-commercial entity). It's definitely a copyright nightmare - so no clue how this could work.
[0] - https://memoryhole.app
The next natural thing to happen would be privatization or consolidation of the internet itself. Its already happening in the form of grabbing and consolidating IPv4 addresses.
It is not hard to imagine a future in 50 years time where a huge percentage of this content is lost forever, or at best incredibly hard to find.
(It is worth noting that at least in Sweden "published" here has a very specific meaning, that doesn't include personal websites etc, but it does include news outlets.)
My own last project before I left was to ingest records from crawl dumps from the defunct cuil.com website. About 200 TB of stuff that brought back 60 billion URLs.
The nature of the internet has changed and it's become an ephemeral place for many people where you just through things in and others mine it as "data".
I'm sure that plays a role, but still... This obviously is about cost and money making, not security as a whole (ime)
Redditors then had the gall to pretend like it wasn’t their number one use case.
First of all it's a lot faster than archive.org, second of all they are much better at bypassing the paywalls. In some cases the operator even seems to have taken up a subscription or at least a user account (in some cases you could see there is a user logged in). They're really clever at this.
For this site the main purpose is indeed the bypass. For the internet archive it's not.
And yeah it could easily be fixed by just publishing the pages a month later or so.
Obviously, a business needs to have an income but it's becoming more common for businesses to function first and foremast as revenue generators and the thing that enables that is only seen as a means to an end. When the quality of the product/service and it's function as a revenue generator diverge, the product/service will always take 2nd chair.
Maybe we could argue that the primary product is the revenue, especially when there are investors involved who are looking for big returns.
So their argument is that people who would be paying money at their paywalls, are going to IA to get their news for free? And if they can thwart those people, they'll show up and become monthly subscribers?
I am vaguely sympathetic to newspapers as a concept, though the actually owners of approximately all of them are just PE companies looking to extract maximum profit from this dying industry, not really trying to prolong their existence.
But I think everyone who is interested in subscribing to their newspapers' paywalls already has subscribed. Those of us who bypass paywalls with that archive.whatever site, or apparently IA (I have never tried it for this purpose) are doing so because there is zero chance we're going to (recurringly!) pay the asking price for some random out-of-town newspaper, The Verge, Bloomberg, whatever. It's fair game to call us immoral for that decision, but if (and it's a big if) this move prevents more people from being able to bypass a paywall, I predict zero incremental dollars will go to the news publishers.
Now most of those who spend money get access to relatively good news in comparison to those who don't. The interesting thing is that if you model the utility of a customer base as trifactorial (subscriptions, ad-supported, influence-ability) and you set ad-support to near zero you're left with this situation where those with no ability to pay are now overwhelmingly useful to the website provider only as an influenceable base.
"If you're not paying, you're not the customer, you're the product", we used to say[0]. It turns out that's true, but if you can't pay by looking at ads, you will pay by the actions you take when you believe what the actual customer wants you to believe.
0: Though sometimes you do pay and you're still "the product" haha!
For the later, archive could just limit access to stuff that's less than 7 days old.
I don't see why every news outlet doesn't just do this.
In which case archive is a major revenue slumper