Something I wish we could have is some kind of peer mirror of archive.org. The main IA web application gets angry pretty quickly if you're trying to click through a few different dates. If there were some kind of way to slowly mirror (torrent-style) and offer pages as a peer from archive.org that would be neat. It would be cool to show up as an alternative source for the data and the archive.org app could fetch it out of there on a user's choice and validate the checksum if required.
In the end, I've ended up just keeping my own ArchiveBox and it's an all right experience. In the end, it's only useful for things I know I wanted to archive. For almost everything I go to the IA - which has so much.
I do wonder why IA does not maintain a IPFS instance, or if they do, why they're not more popular? There's tons of IPFS mirror services out there that operate at reasonable speeds. One issue I've run into with IA is old enough websites that there's JS or CSS that just wont render, what I'm not sure about is, can we retroactively fix such things? Would be nice to be able to un-ruin the code somehow if they exported everything possible at the time.
Edit:
Would be really neat if you could click on a domain while on IA, and a desktop client downloads as many WAR files in a slower priority download queue, as many as you're interested in, with higher priority pages first, and then you can view it fully offline.
I have a design for a system where you can "donate" your disk space to a provider. Basically, you run the client, you say you want to make 1TB available to archive.org, and their server can push the rarest content to your computer.
It's based on torrents, and you can easily make a content delivery system on top of this (so people can fetch data from this network).
I emailed a few archiving teams but nobody seemed interested, so I never made it.
Yeah, I did a scraping project a while back where I wanted to look back at historical snapshots. Getting the info out of Internet Archive was surprisingly difficult. I ended up using https://pypi.org/project/pywaybackup/, which helped quite a bit.
Presumably there needs to be some human to decide something is worth archiving to stop someone just using it as a free way to store all their holiday snaps?
Hi, I run the datacenter/infrastructure team at the Internet Archive! We would love to see you at our various events this fall but if paying for the ticket is difficult for you, please email me (in bio) and we'll get you in (if possible).
Yeah but their view and download metrics are flat out wrong all the time. If they weren’t a nonprofit they’d be sued for that. But still great company a place for obsolete AWS equipment to retire.
It wasn't shut down but definitely hobbled after they lost the lawsuit and were forced to pull copyrighted content from their site that they used to allow signed-in users to check out an hour at a time. My visits to the site dropped 10x after this.
Would be nice to have visit statistics per domain. So people who host their live sites could determine who visits and what on archive.org under their domain vs their live site :).
1 trillion web pages archived is quite an achievement. But...there's no way to search them? You have to know what url your want to pull from the archive, which reduces the usefulness of the service. I'd like to search through all those trillion pages for, say, the name of an artist, or for a filename, or for image content.
The internet archive should be striking deals with AI companies....
We'll load a truck with a copy of our complete archive if you give us a substantial donation to keep the archive going for a few more years.
If you don't agree to this deal, you can still access the archive, but it's gonna be at sluggish download speeds and take you years to get all the content.
kinda unrelated and stupid question: if we archived the version of every page on the internet every second for 10 years, would there be 1 decillion pages at the end of a decade?
35 comments
[ 4.9 ms ] story [ 64.2 ms ] threadhttps://blog.archive.org/2025/09/23/celebrating-1-trillion-w...
In the end, I've ended up just keeping my own ArchiveBox and it's an all right experience. In the end, it's only useful for things I know I wanted to archive. For almost everything I go to the IA - which has so much.
It's been dormant / on hiatus for a few years now.
Edit:
Would be really neat if you could click on a domain while on IA, and a desktop client downloads as many WAR files in a slower priority download queue, as many as you're interested in, with higher priority pages first, and then you can view it fully offline.
It's based on torrents, and you can easily make a content delivery system on top of this (so people can fetch data from this network).
I emailed a few archiving teams but nobody seemed interested, so I never made it.
https://ipfs.tech/
https://archiveteam.org/
And for single page archives I tend to use archive.is nowadays. For as long as I can remember, IA has been unusably slow.
But still kudos to them for the effort.
Do you hash them in some sort of block chain?
The inability to rewrite history will be a fantastic gift to the world.
How does their scope or infrastructure compare?
I know they serve different purposes, but both are essentially doing similar things.
We'll load a truck with a copy of our complete archive if you give us a substantial donation to keep the archive going for a few more years.
If you don't agree to this deal, you can still access the archive, but it's gonna be at sluggish download speeds and take you years to get all the content.