I have experienced this very thing with a SunOS box at the remote end of a satellite link. It was in the US and we were in NZ. Let's just say that you can do a lot with bash and a single root shell.
While true, you can boot from a removable media and restore the snapshot fairly quickly. The original article author seems to have had a problem where he wasn't sure what exactly did the system/boot backup contain.
The choice of what snapshot to mount is part of the boot process, as chosen by grub. Nexenta's apt-clone creates a new snapshot for every system upgrade and adds a new boot entry for rolling back in case it doesn't work.
Perhaps just use shells with more builtin commands? Something like busybox. So if you ever do rm -rf * and don't close your shell you still have all the basic utilities available.
Busybox is a binary that inspects argv[0] to determine how to behave. This is very different from shell builtins that do not launch a new executable, and will thus function correctly after the executable is remove from the file system. As far as I know, you cannot get busybox to perform it's other functions as "builtins". But if you have a functional debugger, it should be possible, though computing offsets is likely to be hard without another machine (chances are that the busybox binary has been stripped).
I had a similar thing happen (having to recover from a very very stupid mistake), I hadn't deleted everything, but I had moved the entire tree into /var/www (oops)
This is an profanity-laden blog I wrote immediately after I figured out how to fix it and got everything back up and running:
(I'll be the first to admit that I sound like an obnoxious teenager in this blog, and that some of the things that I tried to fix it were very naive...it was a few years ago that it happened and as stupid as it is to admit, trying to get everything back up and running made my adrenaline pump pretty hard, so...I apologize in advance...)
To be fair: you had this happen on a piece of consumer hardware which can boot off of any of several dozen live CDs or USB drives, available off the internet at megabit speeds using one of the (no doubt) several other pieces of consumer hardware you have in arms reach.
The VAX 11/780 had no portable boot device, and BSD unix had no "installation media". You bootstrapped to a system monitor, loaded a second stage off of tape header into memory manually, and hand-build your root filesystem. And that's assuming you still have your original tapes from Berkeley in storage somewhere.
I actually tried doing this on the simh simulator a while back, out of sheer curiosity. I gave up before managing to make it work.
To be fair: you had this happen on a piece of consumer hardware which can boot off of any of several dozen live CDs or USB drives, available off the internet at megabit speeds using one of the (no doubt) several other pieces of consumer hardware you have in arms reach.
I wish! This was done on a VPS located god knows where and run by a company that was owned by people who didn't seem to speak any English. I remember emailing them and asking them if they could just move my dirs back using a chroot, but they had no clue what I was talking about. If I had allowed my SSH session to time out, I would have most-certainly lost the server and everything on it.
I have since switched to a combination of slicehost and linode.
Absolutely, and I didn't say that switching VPS providers was a replacement for backups (the linode and slicehost boxes now trade rsyncs every night which include mysqldumps). My point was that I think if a similar thing happened now, that the fine folks at either provider would be more able to help me out.
Even still, I would much rather be able to just mv the dirs back to their proper location from a chroot than having to do a restore from scratch over the network.
It's one thing to work under root sometimes, and it's another thing entirely to work in the root directory. I'm still trying to understand how the file upload sent a bunch of images into the root directory. I would undoubtedly have scp'd or ftp'd the files to either my home directory or the standard ftp directory with user permissions, and then I would have moved the images to where they belong (as root if necessary).
Still, you had some good points. The information on explicitly indicating the library directory is good. Most important of all is the lesson to double check any mv, rm, and splat-completing command that is run as root (or under sudo, for you sudo junkies out there). Personally when I'm using a * in a root command, I first add "echo " to the beginning of the line so that I am sure of what is going to happen before I execute it.
You know that if you just left the slash off, the files go into root's home directory (typically /root/)?
On a separate note, I tend to disable root SSH access on my servers. I SSH as a user and then when necessary, I'll su to root. Not only does that kill the temptation to use root as the default login, it also makes it doubly-hard to get to root as far as password-stealing goes.
This should only happen to you once in a lifetime.
and this is why my "rm" is replaced by this script... just moving stuff to another directory (trash) and I clean my trash manually (with at least 2 confirmations)...
With all these precautions, if I delete something important, I don't deserve admin rights :-)
randstring=`date "+%Y-%m-%d_%H-%M-%S"_bak`
while [ ! -z "$1" ]; do
mv -b -S "$randstring" "$1" ~/.local/share/Trash/files
shift
done
Perhaps they need to alias rm to 'rm -I' (-I : prompt once before removal)? I remember one of the MandrakeLinux versions had that enabled by default many years ago. It saved me a couple of times when I was still a Unix noobie.
Just because you are root doesn't mean you should be able to do whatever you like to the filesystem at any time. Most modern unix variants (including linux) support file attributes such as immutability, see the man page for chattr(1) if you want more. In BSD land there is additionally the concept of Secure Levels which limit the ability of root to change file attributes under different circumstances; generally you have to reboot into single user mode to overwrite files or directories marked as immutable or delete ones marked as append_only.
AFAIK a single cd command will set current working directory to /root, and rm -rf will only remove everything under /root. Or does cd command had different behavior in Unix at that time, comparing to shells today?
Did that version of BSD have secure levels? Updating systems where the system binaries are marked immutable can sometimes be a pain; but the comfort level it provides is remarkable.
I got into this situation once. I got pretty far into my recovery before being stymied by the discovery that tar is only distributed via tar archives: http://www.gnu.org/software/tar/tar.html
What if you were to symlink your entire tree to say /z - as far as I'm aware, this would not use much more disk space, and as it would be the last to go, could you not call /z/bin/whatever ?
I may be way wrong here, but I thought the symlink would still work with the original being deleted, because the file system won't actually mark the file as gone until all links are gone.
Would (or does) this work? Haven't got access to a unix box right now, so can't try myself, but thought it might be a naive failsafe (at least until you actually wait for rm -rf * to finish...)
Symlinks won't save you but you could use hardlinks to make two pointers to the same file. You would have to do it for each file — was it cp -l which would do that.
That's called a link (a hard link). It's just another reference to the inode on disk, indistinguishable from the original. All regular files are hard links, most just have a link count of 1.
Symlinks are symbolic links (soft links), potentially-relative references to fungible user-facing filesystem paths, that work across filesystems and respect the permissions of the original. They're a special kind of file.
It would work with hard links, but you'd have to make and maintain a whole separate real directory structure with an individual link to every file. People used to do this to efficiently maintain branches of the Linux kernel in the days before bitkeeper, depending on their editor to replace files by rename+copy instead of rewriting in-place.
Hard links to directories were removed in an early unix for being hostile to reality and fsck, but were added again in Mac OS 10.5's HFS+ implementation to implement Time Machine's storage.
More fun: libc is hosed on a remote box and you can't find any working statically linked binaries with network support, but still have a few live shells accessible.
I was starting to write a UUDECODE function in pure bash (I'd saved my ass before by pasting uuencoded binaries into terminal emulators) — but then I remembered that bash has built-in tcp support!
At least when called from the exec builtin to spawn a file descriptor, you can open /dev/tcp/$host/$port psuedo-devices. It's existence would have saved me countless more times, if the spoilsports at Debian/Ubuntu hadn't disabled it at compile-time.
> but then I remembered that bash has built-in tcp support!
You have to be careful making that assumption anymore, though; the version of bash that's included with Debian (and downstream variants, e.g. Ubuntu) has /dev/tcp and /dev/udp support disabled.
So it's not something you can count on having available in any random environment you log into.
I disagree with Debian's decision to disable it, but they seem pretty firmly set on it -- people have been arguing with them to reverse it for the better part of a decade, to no avail, so I doubt it's going to change. It's a declining minority of users who probably care. But you could do some neat 'party tricks' with it.
Reminds me of my first job straight out of University (1988) - there was a client who had a sysadmin who had a bad habit of logging in as root and running rm *. This didn't cause any immediate problems - but all hell broke loose when tried to reboot their system.
They had about 25 users on a 386-25 - quite a decent system in its day.
I wish there were an idiot mode for unix that would tell you exactly what the command you just entered is going to do. I'd be turning it on all the time when using commands I rarely use.
It would be nice to know I'm about to do something that I don't intend. As long as the system doesn't ask me "are you sure you're sure you're sure?"
Command line mysql client has a i-am-a-dummy mode where you cannot run a query like "DELETE FROM table". You could of course run "DELETE FROM table WHERE 1=1" but at least it saves you sometimes.
This is awesome, I never knew about this; I usually type "delete from where" and then fill out the query, just to avoid accidentally hitting enter after typing "delete from users".
For me, this is sort of like standing at the edge of a big cliff; you just have this weird feeling that you might accidentally, compulsively jump.
Well, my point is that there are ways around it. Like WHERE id IS NOT NULL, or WHERE id=1 OR 1=1. It's not full-proof, it just protects you from writing out a query that is not complete.
It sort of sucks that in SQL you first say what action you want to perform and then say what you want to perform it on. I end up writing the WHERE part of every query first anyways.
(we actually have a huge pile of old old machines in a store room at the office here - Im not sure if there is a VAX in there or not.. there could well be)
I stopped $ rm -rf ./things -and instead do- $ mv ./things /tmp and just forget about it. Yeah yeah, it won't work on a server, but for a home system that reboots occasionally, it should work fine.
I've learned that nobody and nothing should use /tmp, on account of it being 777. The idea is that Mallory can set up a symlink from an anticipated filename ("things" in your example) to an important file or dir. So writing to /tmp/things destroys the symlinked file/dir.
with tcsh, "set rmstar" will give you a confirmation prompt when doing something like "rm (star)".
and while i'm at it, "set noclobber" has saved me a few times from doing something like "grep file > someotherfile" when i meant to do "grep file >> someotherfile".
while a newbe i asked on irc what would happen if i move /usr in my sgi indy to a new scsi drive i just installed. sice no one replied i opened a terminal and `sudo mv /usr /newusr; sudo ln -sf /newusr /usr` or something dumb like that.
While i waited i opened netscape and went to read something. Then while it was coping irc was flooded with bad omens and curses about what would happen if i did that on a live system.
The only thing i noticed was that the icon state (the desktop shortcuts changed picture if there was any process open for their target files) would some time blink. But that was it.
Not a thing happened. Even the bookmarks i saved on the netscape bookmark file opened before the move ended up on the right place.
My favorite recovery (though it was not fun at the time) involved a headless machine that wasn't booting, and all I had available that would boot on the machine was a floppy disk that ran a rescue shell. I modified the floppy to start linux with console=/dev/fd0, and so was able to type blind and then remove the floppy to read the output. After a few painful cycles of this, I noticed I could often guess at what was happening by listening to the write patterns of the floppy drive, and completed the rest of the recovery using essentially morse code.
85 comments
[ 4.1 ms ] story [ 165 ms ] threadThis is an profanity-laden blog I wrote immediately after I figured out how to fix it and got everything back up and running:
http://newslily.com/blogs/94
(I'll be the first to admit that I sound like an obnoxious teenager in this blog, and that some of the things that I tried to fix it were very naive...it was a few years ago that it happened and as stupid as it is to admit, trying to get everything back up and running made my adrenaline pump pretty hard, so...I apologize in advance...)
The VAX 11/780 had no portable boot device, and BSD unix had no "installation media". You bootstrapped to a system monitor, loaded a second stage off of tape header into memory manually, and hand-build your root filesystem. And that's assuming you still have your original tapes from Berkeley in storage somewhere.
I actually tried doing this on the simh simulator a while back, out of sheer curiosity. I gave up before managing to make it work.
I wish! This was done on a VPS located god knows where and run by a company that was owned by people who didn't seem to speak any English. I remember emailing them and asking them if they could just move my dirs back using a chroot, but they had no clue what I was talking about. If I had allowed my SSH session to time out, I would have most-certainly lost the server and everything on it.
I have since switched to a combination of slicehost and linode.
Switching to a different VPS provider is not the right way to prevent data loss. Doing backups is the right way to prevent data loss. :-)
Even still, I would much rather be able to just mv the dirs back to their proper location from a chroot than having to do a restore from scratch over the network.
Still, you had some good points. The information on explicitly indicating the library directory is good. Most important of all is the lesson to double check any mv, rm, and splat-completing command that is run as root (or under sudo, for you sudo junkies out there). Personally when I'm using a * in a root command, I first add "echo " to the beginning of the line so that I am sure of what is going to happen before I execute it.
On a separate note, I tend to disable root SSH access on my servers. I SSH as a user and then when necessary, I'll su to root. Not only does that kill the temptation to use root as the default login, it also makes it doubly-hard to get to root as far as password-stealing goes.
and this is why my "rm" is replaced by this script... just moving stuff to another directory (trash) and I clean my trash manually (with at least 2 confirmations)...
With all these precautions, if I delete something important, I don't deserve admin rights :-)
randstring=`date "+%Y-%m-%d_%H-%M-%S"_bak`
while [ ! -z "$1" ]; do mv -b -S "$randstring" "$1" ~/.local/share/Trash/files shift done
-hbt
Perhaps they need to alias rm to 'rm -I' (-I : prompt once before removal)? I remember one of the MandrakeLinux versions had that enabled by default many years ago. It saved me a couple of times when I was still a Unix noobie.
Just because you are root doesn't mean you should be able to do whatever you like to the filesystem at any time. Most modern unix variants (including linux) support file attributes such as immutability, see the man page for chattr(1) if you want more. In BSD land there is additionally the concept of Secure Levels which limit the ability of root to change file attributes under different circumstances; generally you have to reboot into single user mode to overwrite files or directories marked as immutable or delete ones marked as append_only.
I may be way wrong here, but I thought the symlink would still work with the original being deleted, because the file system won't actually mark the file as gone until all links are gone.
Would (or does) this work? Haven't got access to a unix box right now, so can't try myself, but thought it might be a naive failsafe (at least until you actually wait for rm -rf * to finish...)
Symlinks are symbolic links (soft links), potentially-relative references to fungible user-facing filesystem paths, that work across filesystems and respect the permissions of the original. They're a special kind of file.
It would work with hard links, but you'd have to make and maintain a whole separate real directory structure with an individual link to every file. People used to do this to efficiently maintain branches of the Linux kernel in the days before bitkeeper, depending on their editor to replace files by rename+copy instead of rewriting in-place.
Hard links to directories were removed in an early unix for being hostile to reality and fsck, but were added again in Mac OS 10.5's HFS+ implementation to implement Time Machine's storage.
I was starting to write a UUDECODE function in pure bash (I'd saved my ass before by pasting uuencoded binaries into terminal emulators) — but then I remembered that bash has built-in tcp support!
At least when called from the exec builtin to spawn a file descriptor, you can open /dev/tcp/$host/$port psuedo-devices. It's existence would have saved me countless more times, if the spoilsports at Debian/Ubuntu hadn't disabled it at compile-time.
You have to be careful making that assumption anymore, though; the version of bash that's included with Debian (and downstream variants, e.g. Ubuntu) has /dev/tcp and /dev/udp support disabled.
So it's not something you can count on having available in any random environment you log into.
I disagree with Debian's decision to disable it, but they seem pretty firmly set on it -- people have been arguing with them to reverse it for the better part of a decade, to no avail, so I doubt it's going to change. It's a declining minority of users who probably care. But you could do some neat 'party tricks' with it.
They had about 25 users on a 386-25 - quite a decent system in its day.
It would be nice to know I'm about to do something that I don't intend. As long as the system doesn't ask me "are you sure you're sure you're sure?"
What we really need are transactions or a versioned filesystem. rm -rf *? rollback!
[1] Command Line User Environment
For me, this is sort of like standing at the edge of a big cliff; you just have this weird feeling that you might accidentally, compulsively jump.
It sort of sucks that in SQL you first say what action you want to perform and then say what you want to perform it on. I end up writing the WHERE part of every query first anyways.
BEGIN;^M<potentially dangerous query>;^M<look at results>[COMMIT|ROLLBACK];^M
Bookmarked! :)
(we actually have a huge pile of old old machines in a store room at the office here - Im not sure if there is a VAX in there or not.. there could well be)
I wasn't being too serious though :)
I never just
First, I do inspect the output, and then use CLI editing to recall the command, and delete the "echo"I do this for damn near everything that involves a wildcard. It's a lifesaver.
alias rm='rm -i'
to my .bashrc. Sure its a pain when i have to delete whole directories but saves my ass.
and while i'm at it, "set noclobber" has saved me a few times from doing something like "grep file > someotherfile" when i meant to do "grep file >> someotherfile".
While i waited i opened netscape and went to read something. Then while it was coping irc was flooded with bad omens and curses about what would happen if i did that on a live system.
The only thing i noticed was that the icon state (the desktop shortcuts changed picture if there was any process open for their target files) would some time blink. But that was it.
Not a thing happened. Even the bookmarks i saved on the netscape bookmark file opened before the move ended up on the right place.
You unix kiddies finally got a versioning filesystem now I hear ?