Tell HN: Recent SUSE update caused my first real dataloss in 20 years

2 points by 2ton_jeff 5y ago ↗ HN

I have used openSUSE as my daily driver for many years, and today took my Leap 15.2 to 15.3. All of that worked fine and as expected, UNTIL I LOGGED OUT.

I cried actual tears of furious rage today at whoever thought this: https://github.com/systemd/systemd/blob/main/src/shared/clean-ipc.c was a good idea.

I routinely treat /dev/shm as a "that will persist until this machine loses power or is rebooted." Apparently, on April 13th, one or more twits dealing with systemd decided that I no longer get to treat /dev/shm that way.

I am trying my best to remain calm as I write this, but the aforementioned code deleted not only my primary operating directories, but ALL Of my locally attached backups as well.

In all, 24TB of BADLY deleted "rm -rf" but without deleting the directories was done, in about 21 minutes before I realised what was happening.

I now have empty directories of all of my client working directories, all of my multimedia drives that our family uses at night, all of the original writes of the births of my children... all because someone decided that removing POSIX shared memory directories when I logout belongs to systemd and not me.

FAR OUT man.

Fortunately, I have a nightly that does backups to an external machine, but it is REMOTE, and now has to do a FULL COPY of the entire thing. I am typing this from my missus' lappy while it presumably will take many days to restore.

Case end point: I only really lost today, tomorrow and maybe the next day worth of billable productivity as a result, but ONLY because I have an external nightly backup running. Local backups (which mirror all local content) were all compromised because I mount them the same way inside /dev/shm.

openSUSE/SLES/SUSE itself is not to blame, the change is a systemd change and all of the major systemd distros will do the same thing to you.

Caveat emptor.

5 comments

[ 2.8 ms ] story [ 18.2 ms ] thread

2ton_jeff 5y ago ↗

In fairness, that history seems to indicate that maybe it really was SUSE who decided to enable it. Sure seems like "RemoveIPC" option should have been advertised as a 15.2 -> 15.3 update repercussion. :-( :-( Leap 15.2 didn't do that, 15.3 does. Unknown what the other systemd distros do with "POSIX shared memory cleanup"

Vogtinator 5y ago ↗

15.2 -> 15.3 included a systemd version update.

kstenerud 5y ago ↗

I don't understand... Why would you put 24TB of irreplaceable data and backups in a temporary place like /dev/shm instead of regular mounts to /mnt/my-precious-stuff and /mnt/my-backups?

I routinely don't want persistent mount points. Imagine my: cryptsetup luksOpen /dev/sdXY name Followed by: mkdir -pv /dev/shm/name mount -o noatime /dev/mapper/name /dev/shm/name

What this "RemoveIPC" did for me was rm -rf all of that, even if I mounted them as root.

That a single "RemoveIPC" difference between openSUSE 15.2 and 15.3, undocumented, caused me to lose all that data is one thing.

Another thing entirely to question WHY "RemoveIPC" is even a thing to begin with. I have never logged into any machine and said "wow look at all of this stale data taking up RAM in /dev/shm".

That looks very much like a bug in systemd, that the deletion crosses filesystem boundaries. You should really report that.