647 comments

[ 3.2 ms ] story [ 352 ms ] thread
Just noticed the site now alerts this:

> Have you ever felt like the Internet Archive runs on sticks and is constantly on the verge of suffering a catastrophic security breach? It just happened. See 31 million of you on HIBP!

Is it a genuine alert, or hacking artifact?

Sometimes with friendly / attempt-at-humorous error messages it’s difficult to tell

It's a literal window.alert()
But was that code placed there by IA or by the malicious party?
Sounds snarky to me. I'll bet it was the malicious party.
Verge reports someone has taken credit for an ongoing DDOS against IA. "An account on X called SN_Blackmeta said it was behind the attack and implied that another attack was planned for tomorrow" https://www.theverge.com/2024/10/9/24266419/internet-archive...
This bad actor has videos of them supposedly “ddosing” Spotify by pinging 1.1.1.1 in two terminal windows on their Twitter.

Is there any link between them and the real attack or are they just unrelated people claiming credit for it?

it wouldn’t be a window.alert if it were IA
I feel like it's safe to assume the official Internet Archive would not write a "friendly"/attempt-at-humurous/unprofessional/confusing/delivered-by-popup message advertising a devastating security breach. Oh also while announcing that nowhere else.

Obv an attackers ability to insert a message does imply a breach beyond a DoS. But I am pretty confident that message was not from the IA.

I assume that if this is a bad actor, then account email/name will be leaked?
Jokes on them... I'm already on HIBP countless of times...
It's all good, as long as you're not in that recent AI Girlfriend breach which exposed a ton of users who were trying to coax it into generating CSAM images.

https://x.com/troyhunt/status/1843788319785939422

“I went to the site to jerk off (to an adult scenario, to be clear) and noticed that it looked like it [the Muah.ai website] was put together pretty poorly,” the hacker told 404 Media. “It's basically a handful of open-source projects duct-taped together. I started poking around and found some vulnerabilities relatively quickly. At the start it was mostly just curiosity but I decided to contact you once I saw what was in the database.”

What a nice guy.

Not sure if you're being sarcastic or not, but pentesting is not a particularly evil activity — and you often have to look at data to see if you actually found something.

What is evil is the way that he's ensured that the predators in the dataset will never face any consequences by making the data available to HaveIBeenPwned, making it trivial for predators to protect themselves (the method through which this is possible intentionally left as an exercise for the reader), and making the data available to a news website for...some reason, but it's bound to ensure that the vulnerability will be patched out quickly and no one else will be able to access the data.

I find it much more likely that this hacker who sought out a website for uncensored AI erotica isn't actually a good guy, and might even have something to hide within the dataset. Hopefully, I'm wrong and we'll see more of this.

How would that protect predators?
Did you miss the joke? Parent poster means penetration as in penetrative sex
True penetration testing.
Well, only success with one kind.
I'm also on HIBP over 10x. What are we supposed to do? Create a new email address for every service we sign up for?

I don't know what the best practice is for keeping our personal data safe anymore.

Password manager + unique password per site + 2FA for anything of value.
Using unique email addresses makes phishing attempts extremely obvious…

(No, this official looking email from my bank is fake since it was sent to Grocery@my.domain …)

I get a ton of "This is your email administrator -- your email password needs to be reset" to github@mydomain
Hey at least after they fill your account up with spam they also send you warnings that you are running out of space.
> Create a new email address for every service we sign up for?

Yes! Just get a domain and have every email it go to you. Mine is something like “@super-secure-no-viruses.email”

There are probably people that would sign up for such a mail. Like urlify.io and other similar URL "shorteners".
Truly unique email addresses and passwords per service is the strongest approach, but there may be alternatives. For instance, Gmail allows address+tag@gmail.com, which will save you from the lowest hanging fruit (block the +tag when it’s compromised to prevent the laziest spam from reaching you). iCloud also allows automatically generating a new email address that forwards to your inbox for a new account when using iCloud Keychain (possibly when using other password managers too, but I haven’t tried).
Gmail's +tag (and the .) is nice in theory, but terrible in practice. It's super easy for malicious actors to just drop them and there are a few services out there that simply are not able to work with the +tag, potentially getting you locked you out of your own account. Not gmail's fault, but I would recommend against using it.
> Create a new email address for every service we sign up for?

Exactly that, yes! Various services like icloud or proton offer "hide-my-email" addresses, or you can use any email service and just leverage a dedicated email aliasing service like SimpleLogin (paid but cheaper).

This way your email addresses are always random, and since these are shared services, the fact that it's random doesn't identify you either. In proton's / simplelogin's case, you can even set the display name used and email first, so from the outside it's not going to appear as strange, or have any real limitations.

If you think about it, modern email services don't really allow for easily testing if an email address is valid or not, so pretty much the only way your email is ever found out is if you share it on. So never share it on. Always share an alias instead. With automated systems, you may even want to rotate it every so often, so that if there's a leak, you can identify not just who leaked, but also roughly when.

Fixed identifiers, like an email address, are terrible, as their lifetime is always significantly longer than whatever context they're being used in for.

Yep. ~300 addresses on my domain, 0 breaches across all of them on HIBP domain search over >6 years.

I guess internet security is not as bad these days. :)

And my SSN's probably available for purchase with 9 types of crypto, too.
Damn I get the notice too
Strange I just received this message when going to the archive.org website I thought I might have misspelled the url
This thread is looking like it'll be one of the first places this incident will be documented (seems to be on the top of Google).

Already there are two new users just for this.

Yeah, I was looking around, but saw no mention of it anywhere until I realized it just happened.
[flagged]
now internet archive is offline. uh-oh?
Bet it’s just a stored XSS alert from a poisoned cache.
Troy Hunt received the leak, tested it and confirmed it. You can find emails on HIBP now
They reported a DDOS attack yesterday, wonder if this is their alert as they manage the fallout?
After this error 504 Gateway Time-out Now 503 Service Unavailable No server is available to handle this request. Not looking good
This is why humanity can't have nice things.
It looks like someone has compromised one of their subdomains for Polyfill

Update: Subdomain seems to be returning normal responses again now.

You mean the IA included some JS polyfill from a subdomain and that's what's compromised / where the alert is coming from?
Correct. The source subdomain of the popup seems to be hxxps[:]//polyfill[.]archive[.]org
yes, "https://polyfill.archive.org/v3/polyfill.min.js?features=fet..." is the URL with the malicious code
It looks like it is running the service that was part of the supply chain attacker earlier this year. https://github.com/polyfillpolyfill/polyfill-service/issues/...
That was a DNS hack of polyfill.io though right? This looks like it was/is self hosted.
The service was fine, it was the "official" hosted instance of the service which was compromised. IA appears to be running their own instance.
That would perhaps explain how they managed to inject the JS alert popup, right?
Yeah, but the leak has been confirmed by HIBP, I found my address in there.
"You are all cooked" vibes from that message hahaha
Now it shows a 'Temporarily Offline' message
Should we be linking to the site that is very likely to be breached? Could start to host any type of malware until the access can be definitively revoked
This - dang/mods is there a policy for this?
(comment deleted)
That's just about article quality though. Is there a policy about linking to known compromised sites? Should one flag the submission for moderator attention?
We don't have policies really, but the way to get moderator attention is hn@ycombinator.com.
As a first step, it might be a good idea to add an indication to https://news.ycombinator.com/submit or https://news.ycombinator.com/newsguidelines.html to not submit links to sites that are known to be hacked/compromised, and to use a text post instead if making a public service announcement!

Even if we assume folks are using up-to-date browsers (and many aren't!), a compromised site could deliver payloads to browsers ranging from zero-days to phishing content to browser extension compromises (esp. for crypto wallets etc.), that might be delivered differently to different viewers. We don't want to amplify the spread of an attack, especially to our community!

There are too many things to add if we start adding things like that. Each one is important in its own context, of course—like here—but once you start making lists of important things, you end up in a whole-is-less-than-the-sum-of-its-parts situation. I don't think such lists are likely to be effective in the long run.

That's also why the site guidelines (https://news.ycombinator.com/newsguidelines.html) are nowhere near as long as they would be if we tried to include all the important things. Better a shorter list that people can actually read.

I hope that doesn't come across as dismissive—I do see your point!

Why go for the Internet Archive go for something else not the fucking archive!
We all need our easily accessible decentralized archive of some sort...
Archive.org is now down. Could anyone explain what it used to show?
A pop-up that said,

"Have you ever felt like the Internet Archive runs on sticks and is constantly on the verge of suffering a catastrophic security breach? It just happened. See 31 million of you on HIBP!"

I just got a Discord "breaking news" notification about this from a server I am, said it may not show on Have I Been Pwned as it is so new.
That's a shame.

We need not one but many internet archives. Just one and we will repeat the outcome of the Library of Alexandria.

(comment deleted)
The Library of Alexandria wasn't that significant and likely wasn't destroyed in one cataclysmic event, but rather centuries of neglect.
The metaphor takes precedence over the fact.
If an attractive story takes precedence over fact, then we will repeat the story of a James Bond film. Maybe the one with that bikini scene, bikinis are attractive after all.
(comment deleted)
More like the library of Baghdad.
Then you have to write legislation in multiple countries to do so, including large carveouts in DMCA and copyright law.

"Goodwill and donations" will never be robust against an entire industry that makes profit off of artificial digital scarcity.

Well this should be fun.

Now I'll have to dig through my IA account and remember if I donated to them directly via credit card (and if they stored it), or if it was through PayPal.

Good point and thank you for the reminder. Time to go check my email archives...
they use Stripe
If you're a blackhat and you want to be annoying, you can use Stripe tokens to charge your target's customers. The target is the payee, so you won't make any money, but it'll add to the chaos.
If Stripe hasn't already, it won't be long until they revoke all of IA's tokens in the event they start using them.
HaveIbeenpwnd says it was just passwords/usernames/emails, so seemingly not. (My company just got an email from them about the breach and I confirmed I'm in there with a quick search on their website.)
That's what Troy got sent. It's not necessarily all the attacker took.
Even if you paid by credit card, there's zero chance they processed the payment themselves.
If they stored your email from your donation the IA would have already used it to spam you themselves, no attackers needed.
Maybe this will make Google reconsider relying on them for cached versions of webpages.
What an asshole, honestly this is a good public service they offer.
Yeah, I can't understand why anyone would attack IA. The service is a gift to the whole internet.
Because in the main, people are vicious, blind, narcissistic brutes.
https://www.reddit.com/r/DataHoarder/comments/h02jl4/lets_sa...

I found this reddit thread from /r/DataHoarder about backing up the internet archive particularly interesting, given the circumstances

50 PB * $0.014/GB = $0.7M. $0.014/GB is from[1], bare drive cost without chassis, power, or redundancy.

1: https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/

How long does an average hard drive last? You'd have to spend that 700k every that many years (plus the extra bits you mentioned). Quite an operation actually
I actually find that fairly tame. For a point of comparison, Wikipedia gets ~$150M in revenue a year, an "asset rise" (I presume this is what non-profits call profit?) of ~$15M a year, and is sitting on about a quarter billion in the bank.

Not that they want to, but I think Wikipedia could fund this using their current donations if they wanted. Hell, I almost wonder if one of the big storage providers would do it for free if they could do it in their staging environment so they get real traffic. It would be less good than real backups, but extra copies are still extra copies even if they're unreliable.

You're right, I guess it is tame and achievable so far as organisations go. I was imagining trying to get some friends together to have a decent percentage of the IA backed up, but that seems out of reach based on this napkin math. Not that that is necessarily demotivating, but it's going to depend on a lot of people intuitively seeing the value and keeping up their share
Yeah, as a sort of pet project I don’t think backing up the whole thing is possible.

You might be able to back up a significant portion of the unique data in IA if you limited it to text files. I think they probably have the highest information to file size ratio.

It’s also probably the most likely to already be back up, though. Interesting issue; you might also get somewhere by cutting the 50TB up into 10GB torrents (or 100GB or whatever, something reasonable for a consumer hard drive) and maybe adding a script that checks the torrent swarm stats to recommend a torrent to download.

Something where I run it, tell it I want to let it use 600GB, and it hands me torrent files for the least seeded 600GB. Maybe a super basic web UI so people can see how well backed up it is?

Unsure if people would sign on or not; I probably would. I’ve got 10 or so TB of NFS I’m not using I could chuck at it. I would guess there are other data hoarders out there who would do the same, but only if it were somewhat easy. I’m probably not going to volunteer to do an hour of rtorrent cleanup a week to make sure I’m backing up the right things.

They should probably consider it, really.

A good portion of the text on Wikipedia relies on Wayback Machine links to remain verifiable. If they lose that, I guess the editors might have to comb every page for information which would need to be either resourced or deleted.

> How long does an average hard drive last?

This is a great question, and a state of the art kind of thing.

HDDs are sold with a lifetime drive read/write amount and power cycle warranty, along with usually some environmental operating envelope. read/write relates to the quality/space of the platter, power cycle is usually the actuator & read/write head being reseated/wearing out. Environment is the same as all other devices in a DC.

Most folks replace drives when they die (reads/writes stall or return garbage), or when the warranty runs out. Some will pay for a warranty exception, and some will just use the drive outside of warranty. Depending on how you use the drive, what environment it's in, etc changes how much you can push things.

I'd say anywhere from 4-8 years, depending on how it's used. In many cases it can be cheaper to have a worse environment for your fleet (thus using less power on hvac) and replace devices more frequently.

There's also the fact that hard drive capacities keep increasing and increasing significantly faster that the power required, and sooner or later for very long term storage it'd become cheaper to migrate all your data from those 5 year old 4TB drives to more modern 16TB ones. That's assuming you want hot access to the data and don't plan on spinning them down as soon as you've written to them, like you'd do for a cold backup of the whole IA.

I remember for a long time (I'm talking 20-ish years back here), every hard drive I bought had double or more the capacity of every drive I'd ever bought previously combined. My first ever 40MB (yes, megabyte) drive got upgraded to an 80MB one, that got updated to a 250MB one, then a 750MB, and then a whopping 2GB drive (how would I _ever_ fill that up???) - and so on. That's slowed down some, but I'm currently starting to think about upgrading my 8TB drives (Raid1 pair) with 20TB drives when the prices start to drop a bit more.

Just don't forget that IO matters.

Drives do 140-220MB/s depending on the LBA distance of the readhead, and that's not really changing. 160MB/s is very common.

So your 8TB drives, assuming 1MiB writes with a 20ms latency and 160MB/s, you can rewrite the drive ~155 times/year. At 20T this drops to ~62 times/yr.

> ...or when the warranty runs out.

Do people really replace their drives when the warranty runs out? Hard drive manufacturers won't provide data recovery on drives that fail under warranty[1]. It makes more economical sense to just run a drive until it dies. You'll end up paying the price for a new drive either way, but less often if you ignore the warranty expiring.

1: I discovered this myself when a Seagate drive containing some important data failed under warranty. If you're foolish enough to send them a failed drive with data you need recovered (like I was), all they'll do is throw it in the bin and send you a replacement drive.

I'm a new user, is this a good time to plug my project that hopes to put 200 GB on a piece of paper? https://sourceforge.net/u/acaiblue44/blog/2024/09/gigapaper0...
> I learned that I can't use file i/o in a function outside the main, which is an unspoken rule that no tutorial elucidated.

is for sure not true, that would be crazypants

I tried for 6 weeks. Eventually, it just stops functioning. The same program and arguments spits out "segmentation fault" 33% of the time I run it, with the other 67% working perfectly. The only way I could explain it was that it was in a function outside the main, because when I put the exact same code in the main, compiled and ran, it worked.

I have no other explanation. At some point, having too many nested loops and variables causes segmentation faults, whereas less complex code functioned without error. I needed to have certain things performed, and it only functioned in the main.

Why would you try to do this in C of all languages? It's one of the worst choices, especially for a self-learner and a beginner like you. Consider: choosing another language could, on its own, 100% eliminate any possibility of getting a segfault! With just that, you'd be spared from having to produce an abomination of many thousands of loc inside a single function, which is never (unless you're Donald Knuth) a good programing practice.
Python is slower but easier, and less likely to segfault out of blue! You don't even have to have a main() loop. If you just have an idea worth demoing quick, I'd recommend switching to Python 3.
If this is a backup, you don't need it to be powered up and available 24x7.

So the question becomes more like "how long does an average hard drive last while powered down and still reliably be able to power back up and be read?".

I'm fairly sure that is a lot longer than the single digit years that'd be the probably answer to your question.

I wonder if there are useful guidelines for long term storage of powered down hard drives? My gut feel is the major failure modes would be electrolytic capacitor failure, bearings sticking as the lubrication ages, and obseleting of the interfaces. I wonder how hard it'd be to find hardware that'd read my Mac SCSI hard drives from 25 years ago?

> I wonder how hard it'd be to find hardware that'd read my Mac SCSI hard drives from 25 years ago?

Easy… that original Mac is sitting in my basement and it worked like a charm last time it was powered on 4 years ago.

> How long does an average hard drive last? You'd have to spend that 700k every that many years (plus the extra bits you mentioned). Quite an operation actually

You'd have to spend a lot more, because with that many drives, you need redundancy now.

True, that would be an up front cost. At the same time, the IA is still live. This initial expense can be softened by building up redundancy over some years rather than trying to do everything at once
> True, that would be an up front cost. At the same time, the IA is still live. This initial expense can be softened by building up redundancy over some years rather than trying to do everything at once

I think with that many drives, you'd be losing them constantly, and I suppose you wouldn't know which ones until later (assuming you're doing an offline backup, if you aren't you have to factor in power costs).

For archival, if you use tape, it comes out cheaper (~225k) and ought to last longer (~30 years).
Backblaze keeps good stats. https://www.backblaze.com/cloud-storage/resources/hard-drive...

1.71% a year failure rate if you care for the hardware as much as they do.

But that number would increase year on year, a 10 year old drive is far more likely to fail than a 1 year old drive
Internet archive is going for long enough that I'd expect it to stabilise by now. If you replace enough of the drivers, you get a good mix.
For that purpose you might want to use magnetic tape like they use in long term archival services

They are cheaper per Gio, and last significantly longer

Interesting to compare their stated drive $/GB to their B2 offering: $6/TB/mo for "pay-as-you-go",

hard-drive price: $0.014/GB

B2 price (12*6/1024): $0.070/GB/year

Electricity, bandwidth, and generally running a business is not free. Also for these pay-as-you-go setups you'd need a considerable amount of free space available on demand. That said, it's not an especially cheap option. Hetzner has storage boxes for EUR 2.5/TB/mo (in fixed 5 and 10TB boxes though)
Yeah, I wasn't trying to point out that it's a bad price. I think it's pretty good: same price for two years with all the maintenance.
IA stores lots of redundant stuff in 5 file formats and none of them are particularly well-compressed, I think. There are (big) savings to be had, but maybe figuring that out (software dev and compute time) isn't worth it?
It's been tried several times, but it's hard because it's such a massive quantity of data. The IPFS backup never really got off the ground.

They have their own backups which I think is good enough for now unless someone plans on donating a few hundred million.

Oh no! I didn't know their IPFS initiative didn't pan out. What happened to it? I am surprised how hard it is to google. I remember interviewing for a role on that team at the archive to help move it to filecoin. Was so happy to hear that the effort was underway to decentralize their datastore. We need this more than ever.
There are people still working on trying to make it happen but it's just a collosal amount of data and filesystems are notoriously hard, so it's very slow going.

From my own personal experience doing distributed archiving with no relation to Archive.org, Filecoin/IPFS's UX isn't quite there yet. They still don't let you serve data to the network from a normal filesystem, you have to let their system ingest all of your stuff so you end up double-storing data or you have to give into everything being stored as inscrutable binary blobs.

That's why I still haven't integrated ArchiveBox with IPFS/Filecoin/Storj, let my data live in a normal filesystem dammit!

> They still don't let you serve data to the network from a normal filesystem, you have to let their system ingest all of your stuff so you end up double-storing data or you have to give into everything being stored as inscrutable binary blobs.

I don't understand this part. What data would you have to give them? Why can't it just live next to your stuff on your OS' filesystem?

They're saying that the client software (the servers that speak the IPFS protocols) has to load the files to be served into their own local storage database, it can't just keep a "metadata file" and read the existing files off disk. Presumably somebody could write a client that spoke the IPFS protocol and did this, or fork the main Go or JS one, but until someone does that they're stuck with the software that's already been written
(comment deleted)
IPFS is all content-hash-addressed, so my guess is the IPFS service spirits the files away to a (hopefully) immutable store for the sake of sanity.
For IPFS, I'm fairly sure you can now serve from your normal filesystem, rather than load it into their blockstorage -- or at least the blockstorage has pointers to real data blocks that are part of your existing files (it's the nocopy option[1]; it's marked as experimental, so there may be some sharp edges.)

For Filecoin, if you want fast access, you do need to keep a second hot plaintext copy, as well as the sealed Filecoin copy. But that works for the backup case for IA, because the hot copy would be served from the archive's existing infrastructure (and/or a distributed IPFS hot cache) -- you'd just use Filecoin for the proven safe backup.

The project to back up IA to Filecoin is still ongoing. The IA dashboard that shows the current state is (perhaps predictably) down at the moment, but it crossed the 1PiB line last year[2], and they've been optimising the onboarding flow recently.

[1] https://docs.ipfs.tech/reference/kubo/cli/#ipfs-add

[2] https://blog.archive.org/2023/10/20/celebrating-1-petabyte-o...

(Disclosure: I work at the Filecoin Foundation/Filecoin Foundation for the Decentralized Web, which partners with the Archive on this project, as well as supporting other Internet Archive backup projects.)

Needing to keep a separate hot copy at 220PiB is already ~$7M/yr, and multiples much more than that if you factor in labor and redundancy. The --nocopy option looks great though, I didn't see it last time I was looking around for an MFS/FUSE solution, I'll try it.

I appreciate your effort and I hope the project continues.

Perhaps you can persuade Elon that it owns the libs?
I don't want Elon anywhere near Archive.org, please don't give him any ideas. There are plenty of other people in the world with money.
"Based on historical records from the first half of the last century, Mr Musk (inventor of the car and the rocket) and President Xi were the most respected and popular individuals on earth."
History is written by the winners...
Maybe in the immediate aftermath, but not long after. King Leopold "won" but we now all think he was terrible.
Yes please, we need this lunatic out of our life, not the other way around
Backup / duplication is not an easy project for sure. But meanwhile for now IA is a single organization operating under one legal system. And one technical setup, would be relevant today. That's a major weakness.
Suppose we each backed up sites we cared about rather than trying to mirror the whole thing...
They're hiring, if you're looking for a job.

https://www.indeed.com/viewjob?jk=3bb8222ccd9a88ea

> Software Engineer, Archiving & Data Services (Remote) [...] Preliminary duties of the role will primarily focus on developing Archive-It

That is. Paying over 100k at the lower end of the range for 3y experience as software engineer

Not even in the 10th % for the area per https://www.levels.fyi/heatmap/
It says it's remote though, so doesn't seem too bad?
Right in most southern states in rural areas that would be pretty good and you could enjoy fresh air and nature while working from your back porch and scanning a few acres of land and wildlife, sipping on sweet tea.
It's a non profit. You're probably not choosing to work for the IA for high compensation.
The undertone was intended to be: that's an insane amount of money, something one with quadruple that amount of experience would maybe earn in a for-profit organisation, but I guess your reaction further proves it's different where you're from
It's not high for bay area software jobs; there are new grads who were paid more than that 10 years ago and I assume new grad wages have gone up since. Of course cost of living (particularly rent) and taxes are high there too, but if you don't blow it all on renting a higher-end place or luxuries you can still save a lot.

For context someone making less than $105k is classified as "low income" in San Francisco. https://www.sfgate.com/local/article/under-100k-low-income-s...

The way you worded it was confusing to read, I thought it was a complaint about "only 100k".

Thanks for clarifying your intent.

The IA is located in the Inner Richmond, which is a ~ medium income area of SF. Rent alone is ~ $4K, or ~ $60K of your income before taxes.
They might be there, but the position was remote-friendly.
To put the reaction into context, the individual low income threshold in that area is 105k USD [0].

[0] https://www.hcd.ca.gov/sites/default/files/docs/grants-and-f...

What area, precisely, is '(Remote)'? Why does the Californian government track income information on Remotistan?
FTR that page just says 403 forbidden

Does this mean you get benefits (like free housing, healthcare, and money to buy food with) if you earn less than 105k/year? Or what does low income threshold mean here

It's one criteria for eligibility for social benefits that can include being able to live in certain kinds of public housing. Usually there's a lot more criteria that go into it, but income is a fairly major one.