> So: the matrix.org database secondary lost its FS due to a RAID failure earlier today (11:17 UTC). Then, we lost the primary at 17:26. We're trying to restore the primary DB FS (which could be fastish), while also doing a point-in-time backup restore from last night (which takes >10h). We believe the incremental DB traffic since last night is intact however. Apologies for the downtime; folks on their own homeserver are of course not impacted.
Love y'all and love matrix. Thanks for the free matrix.org server. But maybe now is the time to research setting up a home server for mission critical stuff?
I don't immediately see an official doc on this; is it right under my nose?
Looks like it's going to take a while to come back up.
> Sorry, but it's bad news: we haven't been able to restore the DB primary filesystem to a state we're confident in running as a primary (especially given our experiences with slow-burning postgres db corruption). So we're having to do a full 55TB DB snapshot restore from last night, which will take >10h to recover the data, and then >4h to actually restore, and then >3h to catch up on missing traffic.
9 comments
[ 2.9 ms ] story [ 27.8 ms ] threadwhoops
The stuff of absolute nightmares...
https://mastodon.matrix.org/@matrix/115136245785561439
I don't immediately see an official doc on this; is it right under my nose?
Is this doc good? https://www.redpill-linpro.com/techblog/2025/04/08/matrix-ba...
> Sorry, but it's bad news: we haven't been able to restore the DB primary filesystem to a state we're confident in running as a primary (especially given our experiences with slow-burning postgres db corruption). So we're having to do a full 55TB DB snapshot restore from last night, which will take >10h to recover the data, and then >4h to actually restore, and then >3h to catch up on missing traffic.
https://mastodon.matrix.org/@matrix/115136866878237078
See also
https://www.theregister.com/2025/09/03/matrixorg_raid_failur... https://www.heise.de/en/news/Matrix-main-server-down-million...