When your engines stop, you trim for best glide ratio, point yourself at an open field, and fly the plane all the way to the ground. Failure to do so is a great way to get yourself killed.
Nice to see a fellow pilot -- well, I'm still in the student phase, but I've been doing lots of "glide, grass, gas!" lately, as my instructor likes to say.
Sadly, the number of open fields to aim for in the SF Bay Area is pretty small. It's either hillsides, bay, or city. Oh, and a couple of thin dikes you could land on. ("That's why nailing the centerline is so important. Someday you might need to land on something like that dike wall -- and if you nail it, you walk away fine.")
Either those drives are all from the same bad batch, or this isn't a drive problem. I would be thinking either a cooling problem or flaky controllers, and a drive swap in that case is not the solution.
Doubt those drives were from bad batch. As these were SSDs, most likely write wear. Two SSDs should never be in RAID 1 pair or replace at least one SSD in RAID 1 at predetermined schedule instead of replacing on failure only. They are going to wear off at same pace and will fail about the same time.
In my experience, all SSDs are a bad batch unless they're Intel 520 or Samsung's high end range.
A big mistake to make is to fill an SSD drive completely up with data. This is a huge no-no unless it's an enterprise drive which usually use a totally different design.
BTW I should add that I'm still a fan of TOR and have no plans to switch away. In fact, their upfront explanation of what went wrong and how has improved my opinion of the project.
Although the fail characteristics and sizes of SSDs are not the same as conventional harddrives so you can't really make much out of that article in an SSD context.
Honestly, this doesn't sound like a bad batch of drives or the like -- sounds like they weren't doing scrubbing on their RAID.
In case it's helpful, and for general knowledge dissemination:
What likely happened is that a drive "failed". This is usually when the RAID card decides that a drive has had enough command errors that it fails the drive. It may actually be fine, and just had a spate of bad responses. You might try to online the drive again and let it rebuild, but that's debatable.
At any rate, so they replaced the drive. That's fine. But then, to rebuild the RAID back to optimal state, it has to read all of the data off of the other drives. Here's where a bad scrubbing policy bites you -- because if those drives have any sectors that have gone bad or problems with the hardware, those drives might fail as soon as the rebuild runs.
Scrubbing should be done regularly (weekly?). What it does is, in essence, test every sector on all of the disks in the array to make sure that all of them are still fully functional so that -- if there is a failure -- you're pretty sure you can rebuild.
The downside of scrubbing is that, for better or worse, it does exercise your disks fairly heavily. Also, if you don't have a suitable trough period then you might even find it difficult to have the available I/O bandwidth to do it.
A backup would be a filesystem level event, so at best it would only access 1/N of the sectors (where N is the duplication level). It isn't guaranteed to access all of the sectors the data resides in -- and also, it may have hit the kernel's filesystem cache or the on-disk cache or something like that, too, bypassing the physical media entirely.
Also keep in mind that a RAID rebuild is a media level event, it doesn't just copy files. It will copy over the "empty" sectors, too, because it just has no idea what's what on the disk. It faithfully recreates the raw bytes on the media.
The disk might have failed in a part of the disk you aren't actually using and never noticed because all of your actual data is intact -- the the RAID can't really rebuild without them.
I'd recommend ZFS for working with these kind of arrays; scrub is a command that's easy to regularly schedule, and will only use idle I/O bandwidth (helps that the RAID functionality is integrated with the whole filesystem).
If you have an infrequent scrub policy and hit bad sectors on rebuild, it can detect checksum failures and mark specific files as corrupted, rather than declaring all your disks defective. Traditional linux md raid behaviour is particularly bad in this regard: if you have a raid6 configuration and haven't been scrubbing, and then have a single disk failure, all your disks will have a few random isolated bad sectors (i.e. sectors that will URE when you attempt to read them) on, but since you still have one disk's worth of parity it's possible to recover all your data with no downtime (and with ZFS raidz2 this is what would happen). But with md raid as soon as you hit those bad sectors during the rebuild it will consider those drives as failing and kick them out of the array, and since all your drives have at least one bad sector on that means it's impossible to recover the array.
That said, ZFS sounds great, and I've been meaning to learn it sometime. The big thing holding me off has been just -- well, how much time I've put into learning md and such, and how I would find it hard to justify going back to not knowing much (since this is my livelihood).
If you have any tips on "read this, it's a good intro" stuff (slanted to Linux, as that's my cuppa), I'd welcome them.
I'm no expert, just a satisfied user; I got into it from the FreeBSD side, and followed their tutorials, but I don't know how well they'd apply to Linux. In general I like the Gentoo and Arch wikis, particularly for slightly lower-level things like this - even if one is using a different distribution, they tend to take the time and explain the concepts in a way that's mostly distribution-independent.
I will say that ZFS felt very coherent and even - dare I say it - easy. I was worried that merging layers that I'm used to thinking of as separate would mean a loss of control, but I was able to put together all the layouts I wanted. The various commands with a single interface felt a lot like using lvm2 - different tools but under a unifying structure with the same parameters - and a lot of what it does is lvm-like. So my best tip is to approach it more as an LVM that can also do raid-like functionality and be mounted directly, rather than as a raid system that includes volume management.
Sorry if that's not terribly helpful - as I said, I'm not an expert, but I wanted to at least give you some kind of reply.
+1 for the Arch wiki. Despite being a distro wiki they have very nice in depth and distro-independent pages. I use Debian but regularly read through the Arch wiki when trying to do new stuff.
You should also be doing a nightly SMART disk test, and that too will also only use idle I/O bandwidth.
All my servers do a short test nightly, and a long (i.e. read the entire disk) weekly (during the lowest usage period for each site).
On linux install smartmontools then add
-s (S/../.././08|L/../../6/00)
to /etc/smartd.conf (adjust the hour and dow for your needs). This will both schedule the tests and keep an eye on them to make sure if there are any errors you will be told about it. (Use the test email command at least once and check!) (Make sure the hour when you schedule the two tests doesn't overlap - if one test is in progress the other won't run, so you may end up never running the long test if you do it wrong - although smartd will tell you about it if that happens.)
That blog post seems remarkably positive to me: "We thrashed this filesystem and pushed some little used features really hard & found a few showstopper bugs that crop up for our use case. We lost no data & discovered a RAID array error that hadn't been picked up before. In summary, btrfs is great, but we can't use it yet."
They didn't mention RAID, and they talk about using SSDs. The failure mode you describe (timeouts) is one typical for spinning rust drives, and not for SSDs.
Typical for a killer backplane / killer power supply.
Feeding 15 or so volts out the 12 volt line usually has negative results after a couple hours
I went thru something like this circa 1996. It was not fun at all.
Sometime similar happened with some networking gear at a previous employer. "I know we have pri/sec control cpu boards, but they're burning out as fast as I can swap a new one in!"
everyone seems to be using this to push their favourite file system, which is fine and all, but if you're using software raid on linux this is the kind of thing you need:
#!/bin/bash
#
# This script checks all RAID devices on the system
# http://en.gentoo-wiki.com/wiki/Software_RAID_Install#Data_Scrubbing
for raid in /sys/block/md*/md/sync_action; do
echo "check" >> ${raid}
done
(the link referred to seems to be down for me at the moment - i just took this from my main machine).
then add a crontab entry to run it once a week or so:
0 0 * * 0 /root/bin/scrub-raid.sh &
you can check the script by running it by hand and then doing
I don't think this applies here. They just moved over to their new RAID setup last Saturday, so they had a major failure within about 72 hours of that transition.
They use a managed hosting provider, which means that there is no guarantee that the disks are new. When one customer finishes with disks, they get wiped (hopefully) and passed on to the next customer.
Even if they are new, though, you never know if it's completely good until you've tested every block on the device. These days most format operations don't actually wipe out the whole disk (takes ages!) so even formatting on a brand new disk won't tell you if it's valid.
That said, the odds of having multiple disk failures on a brand new RAID in 72 hours are pretty low. If that's the case, I'd start to suspect a bad batch of disks and/or bad backplane or other hardware (bad power?) that is causing the hardware to fail.
If your service goes down, be sure to at least have some notion of what your service does on the homepage. Right now, it's only an error report and I had to skim through the blog for a while to figure out that it's a kind of Google Reader replacement.
When your engines stop and you're getting lots of eyeballs, don't assume they all know you.
SSDs still seem to have a bunch of nasty failure cases. Right now, for production use I'm not sure I'd trust the things for reliable storage. As a fast cache for spinning rust? Definitely. As my only live copy, even duplicated in a RAID? Hmm.
Not even Intel SSDs are immune: One of the Debian developers has reported that the SSDs shipped in the latest Thinkpads die if you try and construct an encrypted filesystem on them. Somehow they corrupt themselves during the initial write of random data to the disk.
(Interesting that these SSDs died whilst under high write load too: is this a particular weak point for some reason?)
So, a glide to an emergency landing means that the flight is not over?
Similarly just because a rebuild fails doesn't mean that the data is lost. Just pop in the drives individually and fetch whatever you can from all of them, it is seldom the case that the exact same parts of them become irretrievable at the same time or that the drives stop function completely. It's just some more work than a regular rebuild.
Sympathies. We had a similar thing happen at a previous job in the days of the "deathstar" drives. Lost a drive, no biggie. Tell the DC guy to replace it. Lost a second drive, told him to start running towards our cage. Lost a third drive, uhh - how current are our backups?
Different job - had a developer accidentally run a where-clause-less delete in production. Same net result. RAID and SANs are definitely not backup solutions.
Very much so. For us, raids are mostly time we can use to move all the important data off of that raid. It might be drastic, but it's safe and our data isn't too big.
48 comments
[ 4.2 ms ] story [ 100 ms ] thread“When all your engines stop, the flight is just starting"
Sadly, the number of open fields to aim for in the SF Bay Area is pretty small. It's either hillsides, bay, or city. Oh, and a couple of thin dikes you could land on. ("That's why nailing the centerline is so important. Someday you might need to land on something like that dike wall -- and if you nail it, you walk away fine.")
A big mistake to make is to fill an SSD drive completely up with data. This is a huge no-no unless it's an enterprise drive which usually use a totally different design.
It's still unclear how the storage failure was related to the migration. Was the new engine/fs disruptive to the SSD?
Edit re migration connection: Anything that stresses hardware already close to the edge can trigger a failure.
"But even today a 7 drive RAID 5 with 1 TB disks has a 50% chance of a rebuild failure. RAID 5 is reaching the end of its useful life. "
http://www.zdnet.com/blog/storage/why-raid-5-stops-working-i...
In case it's helpful, and for general knowledge dissemination:
What likely happened is that a drive "failed". This is usually when the RAID card decides that a drive has had enough command errors that it fails the drive. It may actually be fine, and just had a spate of bad responses. You might try to online the drive again and let it rebuild, but that's debatable.
At any rate, so they replaced the drive. That's fine. But then, to rebuild the RAID back to optimal state, it has to read all of the data off of the other drives. Here's where a bad scrubbing policy bites you -- because if those drives have any sectors that have gone bad or problems with the hardware, those drives might fail as soon as the rebuild runs.
Scrubbing should be done regularly (weekly?). What it does is, in essence, test every sector on all of the disks in the array to make sure that all of them are still fully functional so that -- if there is a failure -- you're pretty sure you can rebuild.
The downside of scrubbing is that, for better or worse, it does exercise your disks fairly heavily. Also, if you don't have a suitable trough period then you might even find it difficult to have the available I/O bandwidth to do it.
That said, you should do it if you're not.
A backup would be a filesystem level event, so at best it would only access 1/N of the sectors (where N is the duplication level). It isn't guaranteed to access all of the sectors the data resides in -- and also, it may have hit the kernel's filesystem cache or the on-disk cache or something like that, too, bypassing the physical media entirely.
Also keep in mind that a RAID rebuild is a media level event, it doesn't just copy files. It will copy over the "empty" sectors, too, because it just has no idea what's what on the disk. It faithfully recreates the raw bytes on the media.
The disk might have failed in a part of the disk you aren't actually using and never noticed because all of your actual data is intact -- the the RAID can't really rebuild without them.
If you have an infrequent scrub policy and hit bad sectors on rebuild, it can detect checksum failures and mark specific files as corrupted, rather than declaring all your disks defective. Traditional linux md raid behaviour is particularly bad in this regard: if you have a raid6 configuration and haven't been scrubbing, and then have a single disk failure, all your disks will have a few random isolated bad sectors (i.e. sectors that will URE when you attempt to read them) on, but since you still have one disk's worth of parity it's possible to recover all your data with no downtime (and with ZFS raidz2 this is what would happen). But with md raid as soon as you hit those bad sectors during the rebuild it will consider those drives as failing and kick them out of the array, and since all your drives have at least one bad sector on that means it's impossible to recover the array.
That said, ZFS sounds great, and I've been meaning to learn it sometime. The big thing holding me off has been just -- well, how much time I've put into learning md and such, and how I would find it hard to justify going back to not knowing much (since this is my livelihood).
If you have any tips on "read this, it's a good intro" stuff (slanted to Linux, as that's my cuppa), I'd welcome them.
I will say that ZFS felt very coherent and even - dare I say it - easy. I was worried that merging layers that I'm used to thinking of as separate would mean a loss of control, but I was able to put together all the layouts I wanted. The various commands with a single interface felt a lot like using lvm2 - different tools but under a unifying structure with the same parameters - and a lot of what it does is lvm-like. So my best tip is to approach it more as an LVM that can also do raid-like functionality and be mounted directly, rather than as a raid system that includes volume management.
Sorry if that's not terribly helpful - as I said, I'm not an expert, but I wanted to at least give you some kind of reply.
You should also be doing a nightly SMART disk test, and that too will also only use idle I/O bandwidth.
All my servers do a short test nightly, and a long (i.e. read the entire disk) weekly (during the lowest usage period for each site).
On linux install smartmontools then add
to /etc/smartd.conf (adjust the hour and dow for your needs). This will both schedule the tests and keep an eye on them to make sure if there are any errors you will be told about it. (Use the test email command at least once and check!) (Make sure the hour when you schedule the two tests doesn't overlap - if one test is in progress the other won't run, so you may end up never running the long test if you do it wrong - although smartd will tell you about it if that happens.)Although, unlike btrfs, ZFS and a lot of other RAID solutions are considered safe for production use.
(while I'm a huge fan of btrfs I've already been waiting many years for it to become stable and the wait is far from over...)
http://www.anchor.com.au/blog/2013/04/the-btrfs-backup-exper...
Just don't use btrfs where a failure is going to hurt.
But it completely baffles me that anyone saying that that article seems remarkably positive at the same time as saying btrfs is stable...
http://forums.opensuse.org/english/get-technical-help-here/i...
Feeding 15 or so volts out the 12 volt line usually has negative results after a couple hours
I went thru something like this circa 1996. It was not fun at all.
Sometime similar happened with some networking gear at a previous employer. "I know we have pri/sec control cpu boards, but they're burning out as fast as I can swap a new one in!"
They then asked the provider to swap the disk. They had a RAID1; it's a common enough setup.
SSDs fail in many of the the same ways as spinning disks, they just add a few more failure cases.
then add a crontab entry to run it once a week or so:
you can check the script by running it by hand and then doing where you'll see something like: finally, my notes on this - http://www.acooke.org/cute/ScrubbingR0.htmlI don't think this applies here. They just moved over to their new RAID setup last Saturday, so they had a major failure within about 72 hours of that transition.
Even if they are new, though, you never know if it's completely good until you've tested every block on the device. These days most format operations don't actually wipe out the whole disk (takes ages!) so even formatting on a brand new disk won't tell you if it's valid.
That said, the odds of having multiple disk failures on a brand new RAID in 72 hours are pretty low. If that's the case, I'd start to suspect a bad batch of disks and/or bad backplane or other hardware (bad power?) that is causing the hardware to fail.
When your engines stop and you're getting lots of eyeballs, don't assume they all know you.
They are too relatively slow for home computer use but for servers, much more reliable.
That said, this chart concerns me:
http://www.ssdaddict.com/ss/Endurance_cr_20130122.png
25nm Vs 34nm http://google.com/search?q=cache%3Ahttp%3A%2F%2Fwww.xtremesy...
My first personal computer SSD is going to be the Samsung 830 from different batches in Raid1
Also, when someone else builds your servers, you should query the smart info from the drive to make sure they aren't used SSD.
Not even Intel SSDs are immune: One of the Debian developers has reported that the SSDs shipped in the latest Thinkpads die if you try and construct an encrypted filesystem on them. Somehow they corrupt themselves during the initial write of random data to the disk.
(Interesting that these SSDs died whilst under high write load too: is this a particular weak point for some reason?)
You should not trust anything for "reliable storage". That's what backups and redundant drives are for.
Pedantic and off-topic as it is, this is incorrect. At least for airplanes.
When all your engines stop you continue to glide, and you can even manage to land succesfully with a little skill and luck.
Similarly just because a rebuild fails doesn't mean that the data is lost. Just pop in the drives individually and fetch whatever you can from all of them, it is seldom the case that the exact same parts of them become irretrievable at the same time or that the drives stop function completely. It's just some more work than a regular rebuild.
By the very definition of the world "flight", yes.
Different job - had a developer accidentally run a where-clause-less delete in production. Same net result. RAID and SANs are definitely not backup solutions.