The real lesson from this is that you pretty much never actually want a full disk backup.
Why would you copy over all the operating system files and partition scheme when you can instead do a clean install and clone over your home directory? It in nearly all cases takes much less time and is much cleaner.
If I had to, though, the first tool I would reach for would be `dd`...which the author never does, because it's too archaic, unlike those fancy GUI programs that don't work?
To keep all the applications, OS modifications, hardening, config, etc. When I used Windows, I used one partition for OS/apps and one for data. I backed up the data in a straight-forward, file-level way. Did the other partition as a whole with Acronis. Periodically restored from clean state, did updates, backed that up. So whole system was always one restore away with critical data available with secondary box until that finished.
"If I had to, though, the first tool I would reach for would be `dd`...which the author never does, because it's too archaic, unlike those fancy GUI programs that don't work?"
dd, Gparted... not sure what old stuff like Acronis or Ghost do on modern hardware. I use non-SSD disks on older laptops and servers so I'm not sure. On Linux past few years, I just backup key files and clean-slate install when I have problems because a reinstall/re-config is quicker than a differential backup on a Windows box. :)
I was planning to dust it off and try it on a bunch of disks with different configurations. This thread was a nice reminder. Windows might even benefit from a port of DD to its boot disks with a GUI.
Because it's reinstalling all the apps that actually takes time.
Yes, hopefully you put all the Really Important Stuff in your home or somewhere else where it can be trivially copied (and backed up...) - including your favorite editor, all your configuration files, and any other tools you can't live without.
But then you remember that you recently added a new tool to your workflow, a really shiny package, but one which also required installing approximately 1300 dependencies and for some reason a kernel driver. It took you a solid week to get running properly, and you've long forgotten all the hoops you had to jump through to do it.
This is why I thank my lucky stars that I use a rolling release distro. Configuration is in my home folder, and my system is no more than a list of packages or PKGBUILDs.
Which is actually very similar to what Microsoft are doing with Windows 10. The OS and "apps" are all "rolling" in that sense with the list of packages stored on some MS server. Not exactly the same as something like Arch and it only works for store apps but it is a step in the right way I guess.
"Well, there’s always booting into Linux and doing the old DD if=something of=something bs=somethingbig and then maybe hex-edit the partition tables after it runs off the end of the source drive. This is my wife’s computer and she’s worth it. For kicks, I look at what partition tables have mutated into. They were a miserable hack back in the early 80s, and time has not improved them. I wince and close that web page."
> you pretty much never actually want a full disk backup
I pretty much want a full disk backup all the time:
- when I buy a new laptop that comes with a non-SSD hard disk that I immediately upgrade to an SSD, saving hundred of $$ compared to buying a laptop with an SSD
- when I upgrade / change the drive of my home PC
- when I backup the boot drives of said laptops and home PC in order to have an instant swap-in clone in case of hard disk failure.
I think the author is incompetent. Not only does Clonezilla work well, but I've also had good success with Macrium.
A windows partition that is non-bootable but properly copied can be fixed by booting to Windows install media (USB or CD), entering the recovery console and running FIXBOOT and FIXMBR in sequence.
I've even used the latter technique to remove Linux from a dual-boot drive without affecting the Windows partition. Blow the distro, recovery, fix the boot record, boot into Windows, resize the partition. And this is on an SSD that was originally cloned from the spinning HDD that came with the laptop.
I'm not sure what filesystem he was copying from (I'd guess NTFS from the comment about not being shut down correctly, as I recall NTFS has that particular flag, which is why you will see the disk-scan on some reboots), but NTFS / Windows is a massive PITA to clone. Especially if you have a newer motherboard and have Windows installed with UEFI. You're better off not even trying.
Note that if your windows was installed to boot via UEFI, and your bios is set for BIOS boot (maybe you changed it to run a copy tool and forgot), fixboot and/or fixmbr will instead give you cryptic errors instead of fixing things.
I've cloned hundreds of drives while I worked for two years in the IT department at a large university (30k+ students), and I never had an issue with using Clonezilla or Norton Ghost (which is no longer even supported, but still works perfectly).
If you're moving to a larger drive, just do the above process and then use Gparted to resize the partition. As many warnings as it throws about trashing copies of data, I've never actually seen it happen.
Author was a competent computer programmer when you were still in your diapers. What works for you may not work for him due to a variety of circumstances.
I went from a spinny disk to a larger SSD on my MacBook Pro. The process was roughly:
1) Hook SSD up to an external adapter.
2) Plug it in.
3) Format it.
4) Use CarbonCopyCloner to clone it.
5) Shut down, swap drives, restart.
At this point I had spent a couple of hours and everything worked except for TimeMachine backups. There was some commandline thing I found to fix that, but that wasn't a big deal.
On a Windows machine I would have cloned the disk with dd (from a LiveCD of some Linux), then used the Disk Management thing to expand the partition once I had booted from the new drive.
On Windows all I do is download Acronis Easy Migrate which has a free trial version. No messing with dd (often the SSD is smaller than the spinning disk anyway) or disk management. Swap drives and boot up.
I do all of my Windows stuff in VMs these days, so I don't really know what tools there are anymore. It doesn't surprise me that there's an easy-to-use answer to this problem, especially since the problem isn't all that exotic.
I used to work with Medical equipment from ~2000. All kinds of proprietary file-system formats, funky disk-layouts, etc. Clonezilla was able to handle 90% of them with some tweaking. Great tool
How are you using Time Machine? Over the network I've found it incredibly slow; and of course my network is more or less unusable during that time.
It is simple when it works. However, it is not very good at communicating its status and since I've had it fail in the past, I'm never quite sure its working when it takes long pauses in updating its status.
On Windows, just use dism (formerly imagex). /capture-image the old drive, and /apply-image the new drive. It uses volume shadow copy and can run on an online system.
Not sure what tools were tried in the article, but using the official one from the OS vendor seems like it should have been the first choice.
Coincidentally, I did it this weekend to upgrade my SSD, the exact situation described in the article. Worked fine with no issues.
I am honestly baffled why he had so much trouble getting a cleanly unmounted filesystem.
From either Windows or Linux there are trivial tools for setting the 'check this filesystem on next boot' (if it doesn't do it then and there for you) flag. Let windows do it's check, then actually shut it down (don't suspend it). That should have been the end of that. If /that/ never works then something else is seriously messed up (at best some deeply integrated 'anti virus' software) and a re-install is warranted anyway.
My typical procedure after that is to dd copy the start of the disk (up to a few 10s of MB in to the largest partition) and also the end (if GPT), re-read the partition table of the new disk, then use ntfsclone (linux command) on the big partition. If I'm feeling fancy I'll also pre-copy the end of the old NTFS partition to the same offset in the new one. AFTER booting Windows once you can grow the partition to take on the new space.
1) Plug both disks into a host machine (or boot up off of a WinPE or WinRE disk).
2) Mount the source them as D:\ in this example
3) Format the destination disk in diskpart and mount as E:\ (be sure to mark active and format as a primary partition)
4) Run robocopy D:\ E:\ /x /e /COPYALL /zb /r:1 /w:0 (or something similar
5) wait...
6) verify that the copy didn't miss anything
7) Run bootsect e:\ /nt60
8) Try to boot from the new drive, if it doesn't work then try 'bootmgr /rebuildbcd' and fixboot
it's not sector level, but i don't think it really needs to be, just mirror at the file level and fix up the boot sector. I'm not sure if it is dangerous but it works for me and it doesn't need 3rd party tools.
If you put this on a blog, you could probably become the top Google result for how to copy a bootable Windows partition, and make tons of money from advertising.
The last time I was able to make head or tail of the Windows bootloader was in XP, before they decided that plain text configuration files were too easy for users and they needed to obfuscate the interface by using binary blobs that you could only work with using a rather confusing, poorly documented tool.
As a MS note, I'll just point that copy begat xcopy begat robocopy (and now apparently Copy-Item in powershell). This sequence doesn't fill me with Windows cli tool confidence.
I'm pretty sure that using this method will actually break the WinSxS component cache - it won't preserve hard links. It may appear to work, but you'll possibly have issues applying updates and service packs later on.
You know, I know the author didn't actually name Tools A - E, but how much you want to bet that there is at least a bit of overlap between Tools A-E and the set of tools that people here have sworn you "just" have to use and have never failed them?
Since no one mentioned it yet -- and for what it's worth, and YMMV and all that -- I use Casper to clone drives on all versions of Windows, without any problem whatsoever (used on XP, 7, 8.1 and 10).
This is just sad. It's not a difficult problem, especially for a technophile. It became clear to me when he got to the end and mentioned how dd would require booting into Linux - this blog post is the result of using the disaster that is Windows. At this point it should be obvious to any technophile that Windows is a piece of crap and I'm not surprised that a Windows user struggled with such a simple task. I know this comment is going to be grey soon, but I seriously think that this is true - technophiles who use Windows are worse at even basic tasks. It's a pattern I've seen over and over again.
What good can come of posting something like this, even if you've noticed a pattern or think you have?
Sam once asked me if HN should have a guideline about not attacking entire classes of people (in this case Windows users). He wasn't arguing for it, just wondering, and I thought it sounded too restrictive. Yet every time I notice a comment doing it, it's such a low-value comment that I get a little keener on the idea.
> What good can come of posting something like this, even if you've noticed a pattern or think you have?
Drawing attention to the idea that maybe Windows really isn't the right choice.
>Sam once asked me if HN should have a guideline about not attacking entire classes of people (in this case Windows users). He wasn't arguing for it, just wondering, and I thought it sounded too restrictive. Yet every time I notice a comment doing it, it's such a low-value comment that I get a little keener on the idea.
Consider this - there's several kinds of cars. Most of them have four wheels and doors and can drive at highway speeds. But a substantial portion of the population drives three wheeled cars with doors on all but the driver's side that max out at 40 MPH, and they are ignorant of the fact that their cars are objectively worse. This is the state of Windows in hacker communities.
This isn't really a "class of people". It's a bunch of hackers like us that are ignorant of the fact that they're driving crappy cars, and struggle with things that most of us have no trouble with because we drive proper cars.
Anecdote but I actually did a backup of a 98% full 250GB SSD (Samsung 850 EVO) over the weekend and it took less than 10 minutes. The read speed is ~540MB/s though.
This was done using robocopy on a running Windows 10 machine although not of the system drive, just a second SSD that was used for log files that was full.
The author doesn't seem to be very clear on why dd didn't work. Perhaps he was trying t partition the new drive and then use dd on the individual partitions?
I'm guessing the one thing he didn't try was using dd on the raw devices themselves, rather than the paritions. This would throw away the additional space of the larger drive, but should serve well as a last-ditch "there's no way this can fail" measure.
if that fails(NTFS can be an arse), dd of=/dev/source of=/dev/destination bs=8M(or m) (use the device not the partition, that way pain lies) which will bit copy all the things.
then use windows to resize the partition. (but only if gparted fails.)
52 comments
[ 1.4 ms ] story [ 114 ms ] threadWhy would you copy over all the operating system files and partition scheme when you can instead do a clean install and clone over your home directory? It in nearly all cases takes much less time and is much cleaner.
If I had to, though, the first tool I would reach for would be `dd`...which the author never does, because it's too archaic, unlike those fancy GUI programs that don't work?
"If I had to, though, the first tool I would reach for would be `dd`...which the author never does, because it's too archaic, unlike those fancy GUI programs that don't work?"
dd, Gparted... not sure what old stuff like Acronis or Ghost do on modern hardware. I use non-SSD disks on older laptops and servers so I'm not sure. On Linux past few years, I just backup key files and clean-slate install when I have problems because a reinstall/re-config is quicker than a differential backup on a Windows box. :)
Yes, hopefully you put all the Really Important Stuff in your home or somewhere else where it can be trivially copied (and backed up...) - including your favorite editor, all your configuration files, and any other tools you can't live without.
But then you remember that you recently added a new tool to your workflow, a really shiny package, but one which also required installing approximately 1300 dependencies and for some reason a kernel driver. It took you a solid week to get running properly, and you've long forgotten all the hoops you had to jump through to do it.
I pretty much want a full disk backup all the time:
- when I buy a new laptop that comes with a non-SSD hard disk that I immediately upgrade to an SSD, saving hundred of $$ compared to buying a laptop with an SSD
- when I upgrade / change the drive of my home PC
- when I backup the boot drives of said laptops and home PC in order to have an instant swap-in clone in case of hard disk failure.
A windows partition that is non-bootable but properly copied can be fixed by booting to Windows install media (USB or CD), entering the recovery console and running FIXBOOT and FIXMBR in sequence.
If you're moving to a larger drive, just do the above process and then use Gparted to resize the partition. As many warnings as it throws about trashing copies of data, I've never actually seen it happen.
1) Hook SSD up to an external adapter.
2) Plug it in.
3) Format it.
4) Use CarbonCopyCloner to clone it.
5) Shut down, swap drives, restart.
At this point I had spent a couple of hours and everything worked except for TimeMachine backups. There was some commandline thing I found to fix that, but that wasn't a big deal.
On a Windows machine I would have cloned the disk with dd (from a LiveCD of some Linux), then used the Disk Management thing to expand the partition once I had booted from the new drive.
I do all of my Windows stuff in VMs these days, so I don't really know what tools there are anymore. It doesn't surprise me that there's an easy-to-use answer to this problem, especially since the problem isn't all that exotic.
1) Boot into Parted Magic
2) Copy disk using Clonezilla
3) Expand NTFS partition with gparted
4) There was no step 4.
The whole process was about as painless as possible. I kept waiting for something to blow up on me, but nope!
1) Backup current drive to Time Machine
2) Swap drives
3) Boot with new drive
4) Restore Time Machine backup
It is simple when it works. However, it is not very good at communicating its status and since I've had it fail in the past, I'm never quite sure its working when it takes long pauses in updating its status.
Not sure what tools were tried in the article, but using the official one from the OS vendor seems like it should have been the first choice.
Coincidentally, I did it this weekend to upgrade my SSD, the exact situation described in the article. Worked fine with no issues.
https://technet.microsoft.com/en-us/library/hh825258.aspx
All of these tools have been around in some form (not necessarily fully baked) since Windows Vista and part of the TechNet deployment documentation.
From either Windows or Linux there are trivial tools for setting the 'check this filesystem on next boot' (if it doesn't do it then and there for you) flag. Let windows do it's check, then actually shut it down (don't suspend it). That should have been the end of that. If /that/ never works then something else is seriously messed up (at best some deeply integrated 'anti virus' software) and a re-install is warranted anyway.
My typical procedure after that is to dd copy the start of the disk (up to a few 10s of MB in to the largest partition) and also the end (if GPT), re-read the partition table of the new disk, then use ntfsclone (linux command) on the big partition. If I'm feeling fancy I'll also pre-copy the end of the old NTFS partition to the same offset in the new one. AFTER booting Windows once you can grow the partition to take on the new space.
Here's some info on Hybrid Boot, which is on by default in Windows 8 or later: http://www.eightforums.com/tutorials/6320-fast-startup-turn-...
I have used "shutdown /s /t 0" typed into an elevated Command Prompt, but there are other methods given in the article I linked.
1) Plug both disks into a host machine (or boot up off of a WinPE or WinRE disk).
2) Mount the source them as D:\ in this example
3) Format the destination disk in diskpart and mount as E:\ (be sure to mark active and format as a primary partition)
4) Run robocopy D:\ E:\ /x /e /COPYALL /zb /r:1 /w:0 (or something similar
5) wait...
6) verify that the copy didn't miss anything
7) Run bootsect e:\ /nt60
8) Try to boot from the new drive, if it doesn't work then try 'bootmgr /rebuildbcd' and fixboot
it's not sector level, but i don't think it really needs to be, just mirror at the file level and fix up the boot sector. I'm not sure if it is dangerous but it works for me and it doesn't need 3rd party tools.
The last time I was able to make head or tail of the Windows bootloader was in XP, before they decided that plain text configuration files were too easy for users and they needed to obfuscate the interface by using binary blobs that you could only work with using a rather confusing, poorly documented tool.
For Vista, imagex (part of the Windows Automated Installation Kit, what you use to make official WinPE images), is really what you want. It'll be loads faster too: https://technet.microsoft.com/en-us/library/cc749447(v=ws.10...
Most of the tools I tried are mentioned on this HN thread.
My dismay was driven by the question: "This is a simple problem, how can so many tools fail so miserably?" I still do not have an answer.
What's really sad, he never does backups, so there will be a sequel.
https://www.grc.com/sr/spinrite.htm
https://www.mail-archive.com/gnhlug-discuss@mail.gnhlug.org/...
> SpinRite may have had some relevance back in the days of MFM
...which is pretty much what I meant. I should have said 1987.
Sam once asked me if HN should have a guideline about not attacking entire classes of people (in this case Windows users). He wasn't arguing for it, just wondering, and I thought it sounded too restrictive. Yet every time I notice a comment doing it, it's such a low-value comment that I get a little keener on the idea.
Drawing attention to the idea that maybe Windows really isn't the right choice.
>Sam once asked me if HN should have a guideline about not attacking entire classes of people (in this case Windows users). He wasn't arguing for it, just wondering, and I thought it sounded too restrictive. Yet every time I notice a comment doing it, it's such a low-value comment that I get a little keener on the idea.
Consider this - there's several kinds of cars. Most of them have four wheels and doors and can drive at highway speeds. But a substantial portion of the population drives three wheeled cars with doors on all but the driver's side that max out at 40 MPH, and they are ignorant of the fact that their cars are objectively worse. This is the state of Windows in hacker communities.
This isn't really a "class of people". It's a bunch of hackers like us that are ignorant of the fact that they're driving crappy cars, and struggle with things that most of us have no trouble with because we drive proper cars.
Comments like that don't draw attention, they draw retaliation. There's no way this ends in anything but an angry morass.
I used to be able to copy a hard disk in an hour, these days it can take a full day. A floppy disk takes a few minutes.
This was done using robocopy on a running Windows 10 machine although not of the system drive, just a second SSD that was used for log files that was full.
I'm guessing the one thing he didn't try was using dd on the raw devices themselves, rather than the paritions. This would throw away the additional space of the larger drive, but should serve well as a last-ditch "there's no way this can fail" measure.
It has a really nice gui.
if that fails(NTFS can be an arse), dd of=/dev/source of=/dev/destination bs=8M(or m) (use the device not the partition, that way pain lies) which will bit copy all the things.
then use windows to resize the partition. (but only if gparted fails.)