Surely those who are long-time users of the Internet Archive must have known about the parameter "noindex" in items which makes it hidden from the internal search engine, although remains on their servers. This is essential cases where which the uploader had uploaded content that is controversial in certain jurisdictions but doesn't violate US law in any way and they fear that making it publicly in search engines will invite unnecessary hassles and liabilities to the Internet Archive. Modi BBC documentary might be one tenuous example.
Besides, some people might be fine that their data will be studied by archivists in the far future, say a hundred years or more, however paradoxically in terms of near future and present they are not really okay for privacy/security reasons and prefer to see it noindexed, as opposed to the dichotomy that either an item is fully public, or doesn't exist at all, which in one way or more defeats the purpose of an "archive".
Here's one of the plausible example. A political activist or so want their memories to be archived for posterity becuase they think it will be helpful for far future archivists/researchers who study their career et al. However they prefer to see it hidden but archived because right now bad actors could use that info to doxx or harass them.
There was even an informal policy by Internet Archive to noindex some YouTube videos out of copyright-related reasons, per this.
However, they had remove the function from casual usage about May or June, perhaps after the mass-scraping DDOS incident, and unhide most of the items that are used to be noindexed. Therefore I had emailed the Internet Archive to ask for an explanation and to get them reverse the decision which I think is hare-brained.
After a few going forth and backs, they finally come with this reply which I feel is totally ludicrous:
There is no bug or mistake in removing no-index settings for many Internet Archive items in the Community collection.
At no point was the Archive contacted to arrange a situation of no-indexing (or Darking) items with an intention of later release; the no-index setting was not documented for this use, and represented a security hole that was closed. Tens of thousands of items were found, being used for encrypted files hidden from the search engine, and represented a major problem, so many items have been removed or set noindex quickly.
A number of people have contacted us explaining situations where items might need to be made no-indexed, in a collection for later or timed release for example, but they've done it with communication and discussing their needs, not just uploading files under disposable accounts and then assuming the archive would keep them un-accessible in perpetuity. In some cases their requests have gotten arrangements so that community items that were noindex are noindex again, in separate collections.
A situation can theoretically exist where the original uploader can e-mail us from their e-mail address and discuss arrangements, but you've indicated you intentionally obfuscated your location and have disposed your addresses. If you're able to gain access again, you can mail through those addresses.
An additional situation is you can e-mail info@archive.org if you want to report items at the archive (by identifier) that you believe might need to be removed from the archive; we receive a number of these requests throughout the months and respond according to policy.
It is as if enshittification which had plagued Reddit just not long ago, has now made its landfall to the Internet Archive. The removal of "noindex" had in my two cents, destroyed the chance to attain delicate balance between preservation and privacy, hence meaning increased vulnerability to privacy and copyright related risks.
Perhaps, if the enshittification proves irreversible, there is Texas-based permanent.org which may one day become a successor of the Internet Archive.
Like one of the commenters in that thread said, this sounds like they were using the noidex feature to use the IA as a personal private backup, and thus abusing it, and ruined it for everyone else. The IA is great as a personal public backup. (For example, I've deliberately submitted copies of certain OSS projects I've worked on to the Wayback Machine.)
The IA can always defer to putting a census of noindexed items as a json item though if they want to stay true to their motto of "universal access to all knowledge". For every invalid reasons there are always a valid reason for retaining such features. Some people might not want to see their works easily scraped by AI companies and so on which then profit off their backs, among all other sensitive use cases. If they do away with the self noindex option, the search results will one day become very clogged if you want to look for a refined result.
Permanent.org which is more geared toward the personal private backup use cases were only formally released in 2020. Before it there was only IA which could really provide digital preservation for all swathes of people. Even then I occasionally saw that Permanent.org cross-pollinates their datasets into the IA.
Contrary to the claims that I'm not looping in with them about noindex, in fact they had already been keenly aware of my (and probably others) use cases for a while. Before then I'd sometimes even contacted their staff members for help if I for some reason cannot no-index the items alone. Their tone changed slightly to say that "IA is a library" around the time of the bookseller's lawsuit but coincidentally because all the items I wanted to be archived, had been so, I was able to promise them to not upload any private items, at least I could discuss with Mr. Kahle about the issue. I did sent an email to the latter afterwards, but perhaps because he's heavily distracted by things like the lawsuits, it was eventually let slide and forgotten, until now.
I actually conversed with the founder somewhere before this year where I floated the possibility of the IA becoming an all-out archive and digital vault, more than just a library, in which he seemed to be welcoming with.
Edit: At one point IA even endorsed/recommended the use of noindex as an option to hide bulk-uploaded files from search results to avoid clogging it.
3 comments
[ 2.7 ms ] story [ 12.7 ms ] threadSurely those who are long-time users of the Internet Archive must have known about the parameter "noindex" in items which makes it hidden from the internal search engine, although remains on their servers. This is essential cases where which the uploader had uploaded content that is controversial in certain jurisdictions but doesn't violate US law in any way and they fear that making it publicly in search engines will invite unnecessary hassles and liabilities to the Internet Archive. Modi BBC documentary might be one tenuous example.
Besides, some people might be fine that their data will be studied by archivists in the far future, say a hundred years or more, however paradoxically in terms of near future and present they are not really okay for privacy/security reasons and prefer to see it noindexed, as opposed to the dichotomy that either an item is fully public, or doesn't exist at all, which in one way or more defeats the purpose of an "archive".
Here's one of the plausible example. A political activist or so want their memories to be archived for posterity becuase they think it will be helpful for far future archivists/researchers who study their career et al. However they prefer to see it hidden but archived because right now bad actors could use that info to doxx or harass them.
There was even an informal policy by Internet Archive to noindex some YouTube videos out of copyright-related reasons, per this.
However, they had remove the function from casual usage about May or June, perhaps after the mass-scraping DDOS incident, and unhide most of the items that are used to be noindexed. Therefore I had emailed the Internet Archive to ask for an explanation and to get them reverse the decision which I think is hare-brained.
After a few going forth and backs, they finally come with this reply which I feel is totally ludicrous:
There is no bug or mistake in removing no-index settings for many Internet Archive items in the Community collection.
At no point was the Archive contacted to arrange a situation of no-indexing (or Darking) items with an intention of later release; the no-index setting was not documented for this use, and represented a security hole that was closed. Tens of thousands of items were found, being used for encrypted files hidden from the search engine, and represented a major problem, so many items have been removed or set noindex quickly.
A number of people have contacted us explaining situations where items might need to be made no-indexed, in a collection for later or timed release for example, but they've done it with communication and discussing their needs, not just uploading files under disposable accounts and then assuming the archive would keep them un-accessible in perpetuity. In some cases their requests have gotten arrangements so that community items that were noindex are noindex again, in separate collections.
A situation can theoretically exist where the original uploader can e-mail us from their e-mail address and discuss arrangements, but you've indicated you intentionally obfuscated your location and have disposed your addresses. If you're able to gain access again, you can mail through those addresses.
An additional situation is you can e-mail info@archive.org if you want to report items at the archive (by identifier) that you believe might need to be removed from the archive; we receive a number of these requests throughout the months and respond according to policy.
It is as if enshittification which had plagued Reddit just not long ago, has now made its landfall to the Internet Archive. The removal of "noindex" had in my two cents, destroyed the chance to attain delicate balance between preservation and privacy, hence meaning increased vulnerability to privacy and copyright related risks.
Perhaps, if the enshittification proves irreversible, there is Texas-based permanent.org which may one day become a successor of the Internet Archive.
Here's a pseudocode for...
Permanent.org which is more geared toward the personal private backup use cases were only formally released in 2020. Before it there was only IA which could really provide digital preservation for all swathes of people. Even then I occasionally saw that Permanent.org cross-pollinates their datasets into the IA.
Contrary to the claims that I'm not looping in with them about noindex, in fact they had already been keenly aware of my (and probably others) use cases for a while. Before then I'd sometimes even contacted their staff members for help if I for some reason cannot no-index the items alone. Their tone changed slightly to say that "IA is a library" around the time of the bookseller's lawsuit but coincidentally because all the items I wanted to be archived, had been so, I was able to promise them to not upload any private items, at least I could discuss with Mr. Kahle about the issue. I did sent an email to the latter afterwards, but perhaps because he's heavily distracted by things like the lawsuits, it was eventually let slide and forgotten, until now.
I actually conversed with the founder somewhere before this year where I floated the possibility of the IA becoming an all-out archive and digital vault, more than just a library, in which he seemed to be welcoming with.
Edit: At one point IA even endorsed/recommended the use of noindex as an option to hide bulk-uploaded files from search results to avoid clogging it.
https://archive.org/post/1086430/should-i-archive-this-ever-...