huh, interesting.
Is this a HW or SW problem?
3TB drives are a bit of a sticky issue with older controllers so I wouldn't be surprised if this was the issue or not.
I recently discovered that my Sharkoon USB to SATA dock (old version, USB2) corrupts data. It happens under both Linux and Windows and seems to only depend on the disk used. All the newer HDDs I tried are affected. For all test files larger than 256 MiB (or less, don't remember exactly) the checksums were off.
Disk corruption can be hard to detect under Linux because of the disk cache. Remember to drop it before testing backups.
Before I figured out it was garbage, I bought one of the USB<->SATA+IDE dongle things I used at ex-work for use at home:
USB to ATA/ATAPI Bridge:
Product ID: 0x2338
Vendor ID: 0x152d (JMicron Technology Corp.)
Version: 1.00
Serial Number: 152D203380B6
Speed: Up to 480 Mb/sec
Manufacturer: JMicron
Location ID: 0x24112000 / 6
Current Available (mA): 500
Current Required (mA): 2
I don't doubt it. At ex-work, I had two different USB<->SATA dongle things with two different brands of USB<->SATA controller IC inside, and both would occasionally corrupt data read from or written to the drive.
Yeah, in the old days this was all done in hardware and you got hardware bugs like the infamous ones in the CMD640 IDE controller. (I think Apple had their fair share of data corruption bugs in that era too, actually.)
On CMD640/RZ1000, if OS/2 2.x actually replaced DOS/Windows instead of turning into an entire fiasco, these hardware would not likely have shipped with the problems.
I suspect this is closer to the truth, defects in firmware being more likely given more firmware. But it is also true that less experienced people are being asked to build bigger and more complex systems and there is a cognitive limit.
Stuff is a lot cheaper than it used to be; while some of that is due to economy of scale, I wouldn't be surprised if corners are being cut. Firmware never got much respect to begin with.
Its not getting worse, its just more pervasive. Quality tends to be lower than hardwired implementations ("because we can fix it at manufacture time!"), primarily due to the rush to first to market.
I don't see how zfs can detect the problem any sooner than any other filesystem.
If the problem is caused by the controller wrapping at 2tb then I wouldn't expect zfs to figure it out until later when it tries to read files back and finds damage.
Lets say zfs writes a file at 2tb but due to wrapping its actually written to 0 tb. Then zfs reads the file at 2tb to verify that is good. But the file appears to be fine because its really reading it again at 0tb. At some point zfs will detect a problem but I don't see how it can catch the problem instantly.
Oh, I see what your are saying ... that zfs can detect corrupted files, true.
Whats likely to happen in this case is this:
The first blocks of the disk contain the superblock, labels-descriptors and other filesystem metadata. Most of which will be sitting in cache. The damaged overwritten area at 0mb won't be noticed for some time - like next time the volume is mounted. The filesystem will eventually notice the damage and go into some recovery mode or halt to protect itself. Zfs has lots of redundancy so the beginning volume labels could be rebuilt.
One benefit of zfs will be that you can tell which recovered files are good or bad.
I am currently working on a data recovery caused by 2tb limited bridge controller.
Customer had an external usb/firewire box with a 3tb drive.
At 2tb the writes 'wrapped' back to block 0. Its a mac filesystem. Everything worked great until he wrote past 2tb. At that point it overwrote the beginning of the mac volume. The next time he connected the drive the mac thought that it was a new raw drive and told him that he needed to initialize the drive.
This is the second time he did this - the first time he didn't reinitialize it and we got it all back.
Unfortunately this time they initialized the drive. This zeroed out all the metadata, catalogs, maps etc - much bigger mess.
So some bridge controllers cannot handle drives larger than 2tb. 2tb is the last sector that you can address with a 4 byte disk address.
So for sectors 0-0xffffffff .....works fine.
The next sector is 0x100000000 - 5 bytes,
it only sees the lower 4 bytes 0x00000000 and starts overwriting the beginning of the drive.
NTFS filesystems are more recoverable in this situation because their master file table starts 6 million sectors out on the drive so you would have to write alot more data before you start losing all your filenames and folder structures and pointers to the data.
25 comments
[ 3.4 ms ] story [ 57.5 ms ] threadDisk corruption can be hard to detect under Linux because of the disk cache. Remember to drop it before testing backups.
Sharkoon isn't the only one that uses those controllers (Astone might also) and I keep some of them around for testing purposes.
(Actually, the real problem is the race to the bottom because most buyers buy on price alone.)
If the problem is caused by the controller wrapping at 2tb then I wouldn't expect zfs to figure it out until later when it tries to read files back and finds damage.
Lets say zfs writes a file at 2tb but due to wrapping its actually written to 0 tb. Then zfs reads the file at 2tb to verify that is good. But the file appears to be fine because its really reading it again at 0tb. At some point zfs will detect a problem but I don't see how it can catch the problem instantly.
Whats likely to happen in this case is this: The first blocks of the disk contain the superblock, labels-descriptors and other filesystem metadata. Most of which will be sitting in cache. The damaged overwritten area at 0mb won't be noticed for some time - like next time the volume is mounted. The filesystem will eventually notice the damage and go into some recovery mode or halt to protect itself. Zfs has lots of redundancy so the beginning volume labels could be rebuilt.
One benefit of zfs will be that you can tell which recovered files are good or bad.
Customer had an external usb/firewire box with a 3tb drive. At 2tb the writes 'wrapped' back to block 0. Its a mac filesystem. Everything worked great until he wrote past 2tb. At that point it overwrote the beginning of the mac volume. The next time he connected the drive the mac thought that it was a new raw drive and told him that he needed to initialize the drive.
This is the second time he did this - the first time he didn't reinitialize it and we got it all back.
Unfortunately this time they initialized the drive. This zeroed out all the metadata, catalogs, maps etc - much bigger mess.
So some bridge controllers cannot handle drives larger than 2tb. 2tb is the last sector that you can address with a 4 byte disk address. So for sectors 0-0xffffffff .....works fine. The next sector is 0x100000000 - 5 bytes, it only sees the lower 4 bytes 0x00000000 and starts overwriting the beginning of the drive.
NTFS filesystems are more recoverable in this situation because their master file table starts 6 million sectors out on the drive so you would have to write alot more data before you start losing all your filenames and folder structures and pointers to the data.