Just by the by, this is a terrible idea to do online. The best case is a crash-consistent replica. The more likely case is an inconsistent one that may or may not be repairable, because of the length of time it takes to do the dump.
If you want a bit-for-bit dump of an online filesystem, use Linux LVM to create a snapshot, then make an image of that (100% known crash-consistent) copy.
If you want to backup with this kind of technique, use partimage instead, and do so from a rescue CD. You need a quiescent filesystem for imaging, so it can't be under use by the running OS.
Partimag development has been dead for years; take a look at partclone instead. Another program with a similar goal but a different approach is FSArchiver.
Hmm, I'm a bit wary of this kind of backup, because it seems like it's nontrivial to recover part of your data if half of the backup goes bad. While the old fashioned rsync/copy all your files may be less efficient, you'll still have whatever it managed to copy if something goes wrong partway through.
This reminds be of doing a tar archive of all your files--if the archive is corrupted during that process, what then?
I use clonezilla to image the whole hard drive including OS once I have a 'clean' setup with an empty home and all my software installed.
Then I use rsync to copy all my 'stuff' back off a large external hard drive to the 'clean' installation. I try not to change OS to often :-)
Subsequently I use rsync with --delete set to sync the home drive only to an external drive of similar capacity to the computer drive for daily changes. This drive is ex4 and preserves ownership and permissions, and includes dotfiles.
Once a week there is rsync without --delete to a large external hard drive as 2nd backup. This drive is NTFS and does not preserve permissions and excludes dotfiles.
I have a free dropbox account for really crucial 'working' files. That gets rsync'd to a FAT32 stick each day using the 'time window' command so it does not write everything each time.
Basically the clonzilla whole drive backup is just to save time/bandwidth with re-installing and setting up. The rsync backup to the large drive is file readable on work PCs that use Windows should disaster strike. The 'sync' to the small hard drive is readable at file level on anything I can boot from a live CD.
Now, a bootable whole disc image would be of interest now and again.
Well actually, from a recovery point of view, you get some options. You can mount the image as a block device and perform operations on it. Say the image was corrupted, you could always try to mount the specific partition and run a fsck utility on it.
Consider using Clonezilla [1] instead. While dd will produce a sector-by-sector image of your medium (which can be terribly inefficient if you're using considerably less than 100% of its available space) Clonezilla is file system-aware for a number of common file systems and has built-in support for compression as well. It resembles the recently discontinued Norton Ghost this way but is fully FOSS.
I normally run it off a live CD/USB of Parted Magic [2], another tool I can wholeheartedly recommend.
I created a SystemRescueCd screencast a while ago if anyone is interested [1]. Although, it does not look like SysRescueCD includes Clonezilla, there are tools like rsync, dd, and scp for moving files around in a pinch.
I second the Clonezilla recommendation. Clonezilla is a monumental set of bash scripts that act as a "wizard" to drive a number of other programs. The most important one is probably partclone. You don't have to use Clonezilla to use partclone (it's packaged in linux distributions like Debian), but Clonezilla makes it easier.
I would be concerned about how dd handles a bad sector that has not been reallocated by SMART. Many drives have 1 or 2 of these. Clonezilla is a good option because it also includes GNU ddrescue (so does PartedMagic) which can cope with most file system damage like bad sectors.
It stops being eerie after a while. Several months ago, my brother was putting together a computer and wanted to test it (ie. the motherboard) before attaching any peripherals (other than the monitor). Obviously, installing the OS on the HD to do this was out of the question, so I just copied my computer onto a thumb-drive and used that. I didn't dd it, but just copied my filesystem over and did a grub-install.
Scenario 1:
I boot my PC up from a live CD do dd clone the main drive, since it's a bad idea to try to do it online. I create a giant, 500 gigabyte file containing every bit on my main hard drive. This takes 20 hours.
A month later my hard drive crashes and I want to restore it. I put in new hard drive, boot from live CD, and write my giant 500 gigabyte file onto the new drive and reboot. This takes 18 hours. Voila, I have my setup... from a month ago.
Scenario 2:
I have a cron job to run rsync on a few folders that I care about. Rsync copies them over the network to my NAS. It runs every 12 hours and only copies the diffs. My main drive crashes. I put in a new one, re-install Linux (good opportunity to upgrade distributions), and sync my data back over. The whole process takes 2 hours.
If you want to do something less tricky and low level than dd just remount the drive in question read-only and use tar. (be root to keep permissions, --use-compress-program=pigz)
On the restore side, you can just use pv like cat, and don't have to provide a size param:
In an ideal world a tool like pv could just look at /proc/fd/fdinfo/ instead of shuffling all the data through itself but I haven't found a tool that clever yet.
Remember that this cannot be run on a live system, since you'll get a corrupt image. Thus it's more useful for infrequent maintenance tasks than as part of a regular regime.
I used dd booting from a USB stick to transfer a Windows install from hard drive to SSD this weekend - there are a number of steps that can be done beforehand to tighten up the size of the final output. These steps are also useful for virtual machines, but they tend to take quite a while...
(5) After transfer, expand volume to fit target disk
Pretty much any utility automating these steps is acquired by the various virtualization vendors, or heads directly to the enterprise market (kind of like how inexpensive screencast creation software always disappears!).
I've done something completely silly. I originally wrote this site on github, but then moved it over to a new host on its own domain and totally forgot about the old one.
If you like this article, please check it out here at:
The author mentions that removed files stick around on disk after being removed, and that this can increase the size of a compressed bit-for-bit copy.
They then suggest you can mitigate this by writing zeros to a file before rm'ing it. However, this is slow in the general case, and doesn't help if you've already rm'd the file.
As an alternative, the 'sfree' utility, available in the debian package 'secure-delete', can be used to fill the unallocated portions of a disk with zeroes (or random data).
sfree -llz <disk device>
will write zeroes to the free areas of a disk. -ll limits it to only one pass over the disk, and -z makes that pass write zeroes, instead of random data.
33 comments
[ 3.0 ms ] story [ 71.7 ms ] threadIf you want a bit-for-bit dump of an online filesystem, use Linux LVM to create a snapshot, then make an image of that (100% known crash-consistent) copy.
http://www.partclone.org
http://www.fsarchiver.org/Main_Page
This reminds be of doing a tar archive of all your files--if the archive is corrupted during that process, what then?
Then I use rsync to copy all my 'stuff' back off a large external hard drive to the 'clean' installation. I try not to change OS to often :-)
Subsequently I use rsync with --delete set to sync the home drive only to an external drive of similar capacity to the computer drive for daily changes. This drive is ex4 and preserves ownership and permissions, and includes dotfiles.
Once a week there is rsync without --delete to a large external hard drive as 2nd backup. This drive is NTFS and does not preserve permissions and excludes dotfiles.
I have a free dropbox account for really crucial 'working' files. That gets rsync'd to a FAT32 stick each day using the 'time window' command so it does not write everything each time.
Basically the clonzilla whole drive backup is just to save time/bandwidth with re-installing and setting up. The rsync backup to the large drive is file readable on work PCs that use Windows should disaster strike. The 'sync' to the small hard drive is readable at file level on anything I can boot from a live CD.
Now, a bootable whole disc image would be of interest now and again.
http://rachelbythebay.com/w/2011/12/11/cloning/
I normally run it off a live CD/USB of Parted Magic [2], another tool I can wholeheartedly recommend.
[1] http://clonezilla.org/
[2] http://partedmagic.com/doku.php
http://www.sysresccd.org/System-tools
I've used the imaging tools on the boot cd a few times to clone/restore hosts a few times without any issues. Very handy.
[1] http://sysadmincasts.com/episodes/3-systemrescuecd
http://partclone.org/
I use rsync to back up certain folders. I'm OK with re-installing Linux if my hard drive crashes.
I've actually been able to restore to a drive, and when it boots its as though nothing ever happened. Plus its fast.
Its eerie to turn the computer on after knowing you formatted the drive for whatever reason, and your desktop pops up again.
Scenario 1: I boot my PC up from a live CD do dd clone the main drive, since it's a bad idea to try to do it online. I create a giant, 500 gigabyte file containing every bit on my main hard drive. This takes 20 hours.
A month later my hard drive crashes and I want to restore it. I put in new hard drive, boot from live CD, and write my giant 500 gigabyte file onto the new drive and reboot. This takes 18 hours. Voila, I have my setup... from a month ago.
Scenario 2: I have a cron job to run rsync on a few folders that I care about. Rsync copies them over the network to my NAS. It runs every 12 hours and only copies the diffs. My main drive crashes. I put in a new one, re-install Linux (good opportunity to upgrade distributions), and sync my data back over. The whole process takes 2 hours.
So yeah. Incredibly naive. Sorry.
On the restore side, you can just use pv like cat, and don't have to provide a size param:
In an ideal world a tool like pv could just look at /proc/fd/fdinfo/ instead of shuffling all the data through itself but I haven't found a tool that clever yet.I just use BackInTime though, like TimeMachine I guess. It's really, really useful.
http://backintime.le-web.org/screenshots/
(1) Defrag - built-in / maybe SysInternals contig http://technet.microsoft.com/en-us/sysinternals/bb897428
(2) Defrag the page file - SysInternals PageDefrag http://technet.microsoft.com/en-us/sysinternals/bb897426
(3) Zero unused space - SysInternals SDelete http://technet.microsoft.com/en-us/sysinternals/bb897443.asp...
(4) Shrink volume (if backup is filesystem-aware, and disk is one large volume) - http://technet.microsoft.com/en-us/magazine/gg309169.aspx
(5) After transfer, expand volume to fit target disk
Pretty much any utility automating these steps is acquired by the various virtualization vendors, or heads directly to the enterprise market (kind of like how inexpensive screencast creation software always disappears!).
Actually with the current RAM prices you don't need one at all.
dd if=/dev/da0 | ssh user@rsync.net "dd of=backup.dd"
or maybe:
pg_dump -U postgres db | ssh user@rsync.net "dd of=db_dump"
pv /dev/da0 | ssh user@rsync.net "dd of=backup.dd"
I've done something completely silly. I originally wrote this site on github, but then moved it over to a new host on its own domain and totally forgot about the old one.
If you like this article, please check it out here at:
http://allgood38.io/a-snapshot-of-your-computer-with-dd-pv-a...
They then suggest you can mitigate this by writing zeros to a file before rm'ing it. However, this is slow in the general case, and doesn't help if you've already rm'd the file.
As an alternative, the 'sfree' utility, available in the debian package 'secure-delete', can be used to fill the unallocated portions of a disk with zeroes (or random data).
sfree -llz <disk device>
will write zeroes to the free areas of a disk. -ll limits it to only one pass over the disk, and -z makes that pass write zeroes, instead of random data.
man page: http://manpages.ubuntu.com/manpages/lucid/man1/sfill.1.html
I'll update the article.