Great write-up. Personally I'd have liked a more explicit warning for using raid 5/6 as the bigger they get, the more likely you're to lose the whole array on failures because the remaining disks will be under heavy load for recovery and are likely about to fail as well.
He did mention it as a side note, but it deserves more attention I think.
I remember standing with a few people discussing a RAID failure of this kind and asking each person how it could have happened. All of them could explain the benefits of RAID 5 for the specific application but not one of them could explain its failure modes.
That's just one of many. Another good one is the violent failure mode. We had a RAID 6 go bad because a head crashed hard, hard enough that two other disks in the array also head crashed. Since the enclosure puts them pretty much lines them up, this is kinda inevitable.
Wow, was that an IBM Deathstar? Two additional reasons for raid 6: 1. Fan failure on a hard disk pod causes damage to multiple drives simultaneously. 2. Many adjacent drives are connected via the same power cable which always has the potential to destroy drives in groups via over-voltage from a failing power supply.
Yes, there's also the topic of unrecoverable read errors (URE) and their effect on successful RAID rebuilds. [1][2]
Most consumer drives are still sold rated as <10^14 bits read per error. That's 12.5 terabytes, so in the worst case you could end up in situations where — on average — you are unable read a full 16TB drive without an error. Needless to say this is less than optimal for rebuilding a failed RAID array.
Anecdotal evidence (i.e. very low error rates during ZFS scrubs) suggests that manufacturers underrate their drives and they are much more reliable than that, but it is something to keep in mind.
Fortunately drive capacity completely outpaced my needs for personal data storage in the recent years, so I am happy with JBOD or RAID1, with backups of course.
> Anecdotal evidence (i.e. very low error rates during ZFS scrubs) suggests that manufacturers underrate their drives and they are much more reliable than that, but it is something to keep in mind.
I've seen plenty of drives returning invalid data with correct CRC over the years. On reliable server-grade Xeon + ECC hardware. Of course the vast majority of drives never do it, there's just no way to know which ones do until it happens.
Firmware bugs in weird corner cases? Cosmic rays? Perhaps, but I think it's more reasonable just consider it one of those weird things that occasionally Just Happen (TM) and just need to be protected against at a higher level.
All drives produced in the last 30 years or so are running ever more complicated software stacks. For example they all have features that move data at risk to safer locations without the host system requesting or even knowing about it. Their physical (like bits on spinning rust or NAND flash block) and logical (what the host sees) data representations can be completely different.
Plain old CRC errors, though, are way more frequent. I feel much more comfortable about those errors, at least the drive knows the data is corrupted.
The other issue is that RAID (even RAID 6 with no dead disks) fundamentally cannot deal with data corruption or any of other fun bugs hardware likes to throw at you like phantom reads and writes. This is because it has no checksumming and most implementations will overwrite the parity if a mismatch is found. There's also the RAID "write hole" problem, but most implementations have worked around that since the mid-2000s.
I switched to ZFS a few years ago and it's really quite amazing. In fairness, raidz does suffer from the same "resilvering takes a long time and increases the chance of subsequent failures" problem, which is one of the reasons why mirrors are more commonly used. Though unlike RAID, ZFS knows which data on the disk is actually used so can reduce resilver times pretty drastically.
This is really well written and I learned a bunch. I wonder how much money vendors make by selling hardware RAID when customers could get away with kernel RAID. Also, thank you for this bit! "All tests performed here are random access, because nearly any real-world storage workload is random access."
Question for you smart people... is there a reason to use such an old Linux kernel version? They're testing on 4.15, which was released over 20 months ago. Would there be any appreciable difference on 5.6.4 or something in the 5.x range? Perhaps not since hard drives can't really improve much, but I'm curious.
LSI controllers (adapter cards really no better) for VMS Alpha used to go for a couple thousand list for a retail sub 500 dollar part lavishly upgraded with some firmware that ensured the margins remained as high as the system clock on our ES45s. Fifteen years elapsed before we got the same bandwidth inside a Xeon.... I've still got the above mentioned dual 3.GHz boxen in archive because after that failure we copied all the internal data and mothballed them in case. Furloughed a office lot of twenty thousand dollars workstations because of a shoddy poxy consumer grade aic we were probably missold in the first place has we not had the full pr moves made on us to forget that... NDA expired years ago and only said we wouldn't be beastly to Dell who have been transformed since then the better imo though I am still not buying
> _One final warning about hardware RAID controllers: It's difficult to predict whether a hardware RAID array created under one controller will import successfully to a different model of controller later._
> _We find that with hardware RAID, it's frequently difficult to tell whether you're nuking your array or importing it safely. So in the event of a controller failure and replacement, you may end up sweating bullets, YOLOing, and hoping._
This was the main reason I decided to switch to Linux software RAID from a 3ware card years ago.
A thunderstorm caused a power trip and after powering the system back on, the 3ware controller lost its EEPROM contents that held the PCI vid/pid. Fortunately, it was just a RAID 1 mirror. We used dd to cut past the metadata header and managed to mount the ext2 filesystem.
Using software RAID, I’m confident that we can take the drives to another machine in a pinch and still be able to mount it without much hassle.
Somehow storage creates the most intriguing narratives - maybe we collectively can most everyone appreciate losing or having data at risk, while otter arcana can't quite find the same empathy. Anyway...
In 05 Dell sold me a workstation and internal RAID array that suffered a drive fault requiring replacement and rebuild. Unfortunately the engineer who was handholding me through the steps just to make the changes under our support conditions of the job being theirs and otherwise we left alone, told me it was a LSI card and not a Adaptek installed. This was a very entry level card and the only such card I've experience of. (Accustomed to the luxury of the high end) So I had to take him on his assertion it really was a LSI OEM PERC controller. Erm, not exactly. I learned this when a second connector dislocated and I blithely yanked and reseated U32PSCSI ribbon neatly and promptly as I imagine anyone would. Only the PERC was Adaptek and not LSI and for reasons my recall may be protecting me from* failed the second drive in the R10 array.
I vowed to never run any hardware RAID again and I have maintained my resolution until last month, if specifying a rack of dedicated storage counts for proprietary hardware in equivalece of black box opacity.
I should have written my experiences and blogged them because I took this loss of the subsystem so badly I became obsessive about learning first how to reconstruct the borked array from raw disk, a pleasure indescribable in polite company, and carried on until I was holding forth in charge of a budget for storage space in my dreams I never imagined presiding over. Every which way I believe only in software control of stores. I'm a true disciple to the cause. This r as rack of proprietary storage hardware only got its PO after the methods of code escrow had been thrashed to death in the final thrills of fall and last winter. (Incidentally the outside legal lead wrote me privately afterwards and pondered out loud whether the escrow didn't create a backdoor kind of vendor lockup because of the potential for contaminating employees with IP during emergency contingencies operating. Thanks Buddy! I'm not sounding like it but I'm actually grateful for that consideration if only because it sounds good for avoiding any repetition of the process just endured. It's a dual licensed codebase but being on the commercial code and the open code trailing in develop... I'd love to see what eg DDN routines are like I'm absolutely sure eager audience awaits publications like that.
Surely the lock down must have awoken the industry to the needs of self sufficient repair and diagnosis.
People still use hardware RAID? Windows exempted, as there's unfortunately nothing better available for Windows. (ReFS SW RAID might be ok, haven't checked its state in a while. I know Microsoft is definitely working hard to fix this.)
Hardware RAID is just data corruption waiting to happen, simply because there are more vulnerable components and interconnects along the data path.
It's much better to checksum data while it's still protected by ECC RAM and CPU ECC caches – which is something only software RAID can do. Although even that is not a total guarantee.
There's no longer reason to assume a slow SoC on a RAID adapter (even with XOR acceleration) would perform better than an Intel or AMD server CPU, as long as I/O path is not bandwidth limited.
Hardware RAID controllers always use ECC. Hardware RAID controllers haven't been using battery backed-up cache for the past decade, but supercapacitors and flash memory. So they provide better cache safety than any software RAID stack.
Also hardware RAID allows for a more consistent performance under heavy system load in my experience.
This is not what I was talking about. Of course they do. The problem is that data needs to somehow be transferred to the adapter first, and this alone adds a point of failure. Controller can (and too often do) have hardware and firmware bugs. The less components the lower chances for corruption.
> battery backed-up cache for the past decade, but supercapacitors and flash memory
No difference to software RAID there. For example ZeusRAM and Optane are used for this. Of course high endurance SLC flash with supercaps is also an option.
> Also hardware RAID allows for a more consistent performance under heavy system load in my experience.
I'd tend to count this subjective, but I'll grant you this one, it's at least plausible. I haven't encountered this issue, though – today's hardware has insane bandwidth. SW RAID can saturate 100 Gbps ethernet, how much more performance you really need?
Even if performance was slightly degraded, data integrity is more important.
Notoriously embedded CPUs have a much worse price/performance ratio than mainstream CPUs so almost inevitably they use a much weaker CPU than is needed.
This is one reason why WD is involved in RISC-V, so they can escape this, and another reason why Intel's storage strategy (ex. optane) has worked out, because the main CPU has power to RAID NVMe SSDs and off-brand CPUs don't.
This comparison seems a little disingenuous in this context. I read GP's post as saying simply that the current speed of SW RAID is enough to saturate current network speeds. If in the future we make some great breakthroughs in network technology without making the same progress in compute performance, then perhaps hardware RAID will become more relevant again, but right now software strikes a better balance.
Not my best post. I can't argue about bandwidth saturation aobviously but "how much more performance you really need?" is a silly question. It dismisses the possibility of it being not enough.
You fell into a classic trap: premature optimization is the root of all evil.
It makes no sense to seg out [superior and more agile] software functionality to a one-and-done ASIC or fancy FPGA that has a magnitude order or more of overhead/management that you must immediately invest in.
When I can't feed Nx10GbE or 100GbE with ZFS, I'll consider this an issue. Until then, it is merely an exercise in logical masturbation.
It wasn't a silly question, and it didn't dismiss the possibility of it being not enough. It's correct and reasonable to say that it's extremely unusual to need disk transfer speeds in excess of 100 Gbps in any standard deployment.
I wanted to love Storage Spaces, but I had huge issue with a leak in the driver that caused the non-paged pool to grow on my machines. It got fixed, but I'm struggling with the idea of going back to it after the experience and stuck with dynamic disks.
Create a mirrored volume out of two disks, both with ReFS (or ZFS) and dynamic disk software RAID.
Remember to enable ReFS integrity streams! ZFS protects full integrity by default. NTFS supports only protecting metadata, file data integrity is not protected.
Fill the volumes with known data. Randomly corrupt some blocks out of disks (but not from both disks at same offsets!).
Observe which solution recovers and returns correct file data and which does not.
Hint: you might want to stop using dynamic disk RAID after this experiment...
Storage Spaces operates on a higher level, building on lower level volumes. I'd feel way more comfortable using it on ReFS, at least once it can be considered sufficiently mature in the future. See: https://docs.microsoft.com/en-us/windows-server/storage/refs....
No qualms about trusting ZFS, it's very solid on good hardware. Of course running ZFS on Windows in production might be a Very Bad Idea, so do it on FreBSD, Solaris derivatives or Linux instead!
I'm using storage spaces on a home machine. It's very flexible, my machine is using 5 disks arranged in 2 separate logical volumes (one parity, one mirror). The write speeds are terrible, around 25MB/s. Read speeds are >100 MB/s.
Pretty great example of how it is it to spend other people’s money. The cost difference in two servers with 10-20x SAS drives vs a performance comparable SAN is not very close.
> People still use hardware RAID? Windows exempted, as there's unfortunately nothing better available for Windows.
I've always wondered whether anyone has gotten fed up with this, and decided to just virtualize Windows Server under Linux in order to feed the Windows Server VM a virtual disk that sits atop all the Linux storage-layer tech (but where otherwise the Windows VM gets all the rest of the computer's resources passed through.)
(Come to think of it, I've considered doing the same for a Hackintosh at some point. It got too contrived, but I'd like to come back to it one day.)
Intel also have solutions like ICH10 vs ICH10R. Today they also support it on some chipsets.
But you're 100% correct. I have friend who have few RAID controllers in his office just in case that one of it's servers fail. And all of them are purchased in same order to be from same batch.
Isn't software RAID just as likely to cause additional failures in remaining disks during rebuilds? Rebuilding a degraded array causes a lot of I/O. Storage capacities have increased tremendously but improvements in I/O speed and read error rates are orders of magnitude smaller. So we ended up with rebuilds that take over a day to complete and are likely to fail despite the redundancy.
Using this RAID calculator gives me worrying numbers for high capacity arrays, even with enterprise hardware:
Following the link to libeatmydata that some hardware controllers use for asynchronous writes:
libeatmydata is a small LD_PRELOAD library designed to (transparently) disable fsync (and friends, like open(O_SYNC)). This has two side-effects: making software that writes data safely to disk a lot quicker and making this software no longer crash safe.
DO NOT use libeatmydata on software where you care about what it stores. It's called libEAT-MY-DATA for a reason.
Also just some latin pedantry, the article mentions the phrase: Caveat imperator a couple of times. I assume to mean Let the buyer beware, which as far as I know is Caveat emptor.
The main issue with big drives is that you are sitting in a degraded state for longer while you await the rebuild. That allows more time for another disk to fail while your rebuild is incomplete.
If you have a set of N drives all purchased at the same time, they are going to all have the same approximate bathtub curve and could start failing at the same time.
I had five Seagate 7200.11s (terrible, shitty drives in many ways) in a RAIDZ1 pool. One failed, I started a resilver. Another started failing during that time. The rebuild slowed significantly due to the second failing drive. ZFS resilvered eventually then the second drive failed completely. That was at only 26k hours or so. I think a third started acting up shortly after but I was already replacing them all.
These were only 500GB drives, and the chances aligned on two of them....
I had as well some Seagate drives failing... . (using now only HGST and WD drives - no experience with Toshiba)
Question about ZFS and its RAIDZ: do you have any recommendation (personal experience, links, books, ...) concerning parameters/setup to be used when setting up a RAIDZ(1 and 2)?
I'm new to ZFS and I already had to do a lot of tests when using ZFS only on 1 HDD until I finally managed to get good performance out of it (used by a "Clickhouse" database which itself writes data in a CoW-style => I had to raise the "recordsize" to 2MiB) and I imagine that with RAIDZ it can get more complicated?
I would like to set up a RAIDZ for the database (again, "Clickhouse", which generates multi-GB files) and two more to be used as simple NAS (a main one and a backup, storing files of all sizes).
Servethehome.com has some subforums with a lot of info, particularly the Napp-it one which is a management UI for ZFS like FreeNAS is
Reddit.com/r/ZFS
Reddit.com/r/datahoarder
I'll try to think of some others.
Honestly the best way is to start tinkering and see how it works for you as it can depend heavily on what you're using it for.
But yes there is a lot of FUD out there.
One thing to beware of, RAIDZ1/2 is limited to the IOPS of one drive per VDEV... Can be quite limiting for some things.
Read up on IOPS vs VDEVs and ashift, ashift has some important considerations for space consumption and performance and future drive replacement. You almost always want ashift=12.
I have avoided RAID for the past 20 years and will avoid it in future. The whole technology and its benefits is about picking pennies before the steamroller, with explosive complexity of failures.
I prefer much more simple and understandable replication and backup solutions - there are dozens of them.
Have you tried out ZFS? It's quite good at resolving failures and data corruption and can be configured in all sorts of combinations, depending on your performance and disk failure tolerance requirements.
It's very important to understand that RAID is not a backup solution. You still need to backup even if you have RAID, no matter what level.
What it offers (in parity mode) is improved reliability and more chance that your server will be online if your drive dies. In majority of cases, it saves you from restoring from backup. Just replace the bad drive, and away you go! However, never assume that it will protect you from data loss. Backup. Backup. Backup!
Exactly. One doesn't have to be superhuman to set up modern DB replication and manage its failure modes.
Even if you need to salvage broken HDD, it's a pretty straightforward task with high rate of success.
In the sense that your backups are a surviving copy of your data, sure.
That said, an argument such as "system is down while we restore from backup; some recent data may be lost" will not go over as well as "the system remains available, but performance will be negatively impacted while the array rebuilds; no data has been lost at this time".
In the former, you're in crisis mode. In the latter, you'll probably be under some stress, but you're still in the clear.
Edit: But we're all on the same page when it comes to backups. They're a necessity.
Trouble is that the industry has not kept up with people's backup needs.
A tape drive costs ten times what a hard drive with the same capacity costs, and then you have to buy the tapes.
I've been able to read data written to a hard drive 99.99% of the time but I can't say that for tapes. Often it has been human error, but the fact that it's a strange thing you don't do everyday and isn't part of your workflow is a reliability problem that is hard to overcome. (e.g. why a Toyota Corolla is more reliable than a Ferrarri Testarosa.)
In theory you could do offline backup to "the cloud" but probably your net isn't fast enough and the other people at home will be wondering why Zoom isn't working for them.
Technically, disconnected RAID1 (or RAID10 multiple) is a backup.
Recovery is a bit funny with you marking all remaining drives as faulty then plugging backup for resilvering. (Or you can use it directly, less safely, as a normal separate array.)
Must be disconnected and unpowered, yet regularly checked. Not super useful for extreme long term storage but neither are tapes.
Not familiar with the "disconnected" technique - does this mean you disconnect & reconnect your drive periodically?
Sounds like a misuse of RAID features. What could possibly go wong? Eg. Reconnect the drive accidentally forgetting to mark others as bad and bye bye "backup"?
Also, RAID controllers die sometimes. It's not pretty when that happens, which means you really need a backup of the whole array anyways. So technically RAID is not a backup because you are still not protected from that type of failure.
Sorry, not very experienced in RAID. But when you say RAID do you mean RAID 10 or something? Isn't "replication" (i.e. having a mirror as a backup) literally just RAID 1?
Great writing. however a question for geeks. what is wrong with using RAID0 for performance along with external backup/deduplication solution like borg backup/duplicati etc... ?
It depends only on whether you can afford to lose the data.
I'm using two SATA SSDs in md RAID0 for VMs on my desktop (where each VM gets its own LV). Dangerous thing, but if one of them fails, nothing happens, I will shrug and reinstall.
You can take as an example the Recovery Point Objective and Recovery Time Objective terms used on disaster recovery. Depending on: a) how often you do backups, b) how many data are you backing up, and c) how fast are your network and transfer medium, you can calculate how much data you may lose in case of failure and how much time you'll use on recovery.
As long as your choice of RAID0 is an informed one (i.e., you are fully aware of the risks and failure modes as well as the benefits -- and it sounds like you are), there's absolutely nothing wrong with using it.
There are three drawbacks to the setup you describe:
- If a drive fails, the system is offline until you restore from backup. Fine for personal use, but not okay if customers are paying you for a service.
- If a drive fails, any changes since the last backup are lost. Fine for long-term archival, not so great for a bank ledger.
- No capability to detect data corruption. RAID with parity can run a scrub to find and repair bit rot. If data is corrupted on your RAID 0, you probably won't notice, and you'll back up the corrupted. If your filesystem has checksums you will be able to detect the corruption but not repair it. Your filesystem probably doesn't have checksums.
If you frequently run I/O intensive workloads, the extra performance might be worth the tradeoffs. If you're looking for your PC to feel slightly faster it seems foolish.
- I've got 1 HDD failure on RAID10 4 x 10TB HDDs, I had to take the system down to do RAID10 rebuild. because running server with huge I/O slowdown the rebuild and I had my fears of second HDD failure. (I had to take the system down anyway).
RAID didn't help in my usage scenario.
- I guess deduplication fixes this already.
- Any advice on ways fixing this and preserving my RAID0 usage scenario?
Broadly speaking, you have two options. One is, make the server faster so that it can keep operating during a rebuild. You could do this by adding more drives. The other option is, add a second server to handle the load while you fix the first.
It seems every post here saying negative things about "hardware RAID" should prefix the term with "inexpensive consumer" or at least "single-system integrated".
Which, to be fair, makes sense with the subject of the article limiting us to a discussion of 8 drives.
Right. I struggle to see how anyone in this thread is seriously comparing their PC with ZFS to my server with 12x 3.84TB SAS SSD and a PERC 740 controller. Yet they are with absolutes like “Hardware RAID is dead”
Plus, you’re talking about RAID as a backup, it is not. RAID is redundancy, not disaster recovery. If your backup solution is bas d on your RAID controller series, you’ve already lost.
I manage a server fleet with local storage ranging from single-disk systems, to servers with 12+ internal disks and multiple attached JBODs, with disk tech including SATA and SAS HDDs of various RPMs, as well as SATA SSDs. Many of these machines are equipped with hardware RAID controllers of various brands. Hopefully that helps to demonstrate that I do actually have experience with enterprise hardware, and I'm not just some kid with a home lab parroting what I've read online.
So, here goes...
The management tooling for hardware RAID is a pain to deal with. Dell's OpenManage tools aren't that bad (other than being stupidly bloated), but every time I have to use LSI's MegaCLI utility, I end up wanting to shoot myself. (Seriously, do a Google search for "MegaCLI cheat sheet". It's _bad_.) For the cheaper LSI FusionMPT-based cards, I've completely given up trying to manage them from the OS, because I've never been able to get it to work reliably. And because hardware RAID controller manufacturers have changed ownership so many times over the past decade, even _obtaining_ the software can be a challenge. And that's assuming that it supports your OS, which isn't guaranteed.
Monitoring the health of hardware RAID arrays is challenging. Not only do you generally need to use a proprietary tool of varying quality to do so (with all the issues stated above), the information you get is typically not that helpful. Most RAID controllers will at least tell you if the array is degraded or not (i.e., one or more disks are missing), but the quantity and quality of that information can vary. Additionally, monitoring the health of the underlying physical disks that make up the array is also important, but getting this information from a hardware RAID controller is tedious (and sometimes impossible), which can leave you blind to the actual health of your storage. The disk health typically reported by RAID controllers (the greem/amber blinky light that indicates a failed or failing disk) is based entirely on the disks reported SMART health check status, which is so prone to false negatives that it may as well not even exist. Oh, and if you're using an OEM RAID controller with third-party disks, you often have to deal with a nag message informing you that the disks aren't "official," which at best is annoying, and at worst can hide legitimate disk issues.
Hardware RAID can have annoying limitations. Want to mix a SAS and SATA disk in the same array? Most controllers won't support that. Want to mix RAID levels on the same disks, like in the case of a distributed database where you might want some redundancy for the OS volume, but the actual database volume can be RAID 0 for performance/space efficiency? Most controllers don't support that, either. Want to create multiple distinct logical volumes from a single array (e.g., a 20 GB volume for OS, and a 10+ TB volume for data)? That's usually possible, but in the case of many LSI MegaRAID controllers (including their Dell PERC rebrands), that can prevent you from expanding the space on the array without a rebuild. Want things like RAID 6, more advanced nested RAID levels like RAID 50 or 60, or tiered storage? If the controller supports it at all, it's probably a premium-feature that costs extra to unlock.
Hardware RAID is slow. The onboard processors on most controllers have no problem keeping up with mechanical disks, but struggle with the performance potential of SSDs. The centralized cache on the controller can be useful in some situations, but the typical configuration is to use this cache while disabling the cache located on the disks themselves, which runs into scalability issues for obvious reasons. And lastly, you're ultimately limited by the throughput of the bus to which the controller's attached, which means that a hardware PCIe RAID controller can't possibly compete with NVMe disks that are directly attached to the processor via the PCIe bus.
I'll agree with all of this, I've had many of the same pain points over, whatever, 15 years of doing the same. The UI on these controllers is terrible and often impenetrable. I remember whatever HP used for 'hardware raid' on DL380g6/g7s was prone to locking up+crashing under heavy load.
But, again, this is very low end to middle end 'hardware raid'. It's single-digit disk stuff. It's a cheapo PCI card in a cheapo server.
I'm struggling to think of high end software RAID solutions. I suppose backblaze pods are all software though, but I haven't personally worked with one; I'm not sure if they're primarily designed for capacity or performance. Through enough spindles at a problem, though....
Ultimately disks don't really care whether it's software or hardware that sends them the commands.
Even hardware RAID is somewhat of a misnomer. Just like the disks themselves, it's just an embedded CPU, varying degrees of hardware acceleration, possibly battery/ultracap backup and a big software stack.
Software RAID can achieve high capacity and/or performance. No difference in principle to hardware RAID there. You can use multiplexing for huge SAS backplanes in either case.
I wish articles like this could afford production-grade hardware to test on. When you’ve run large Fibre or SAS arrays in the past on enterprise hardware it’s hard not to see gaps in the testing methods/setup. Sequential writes metrics ARE important when doing database backups. SATA (even nearline) and SAS disks have vastly different performance profiles so I would expect optimal configuration would be different. What about SSD vs. spinning? What about different rotational speeds? Were the disks aligned properly? How does performance scale with increasing the number of threads requesting I/O? The setup in the article really only applies to certain prosumer/small business requirements. I guess I’m not the right audience for the article.
This is a great article on the basics of RAID. Outside of consumer grade RAID systems it's a bit out of date. There are some perceptible performance differences between enterprise and consumer storage systems.
I manage a small (<100TB) storage system for our office. We have servers using RAID10 SSDs and we have other servers using RAID10 10k SAS HDDs. The read performance on the SATA SSDs is great but the writes are atrocious.
> The read performance on the SATA SSDs is great but the writes are atrocious.
If a 10K SAS HDD array is outperforming even a single SATA SSD, either your I/O access pattern is extremely sequential, or there's something wrong with how your storage is configured.
Our environment is almost entirely virtualized, so yes, extremely sequential. Each host has 2 or more 10Gbps NICs. So starting, suspending, moving, snapshoting and cloning a VM shows the limits in both. Our SAN is 4 years old and has tiered storage (10k SAS mainline HDDs and high-write SAS SSDs). We use cheaper SSDs in some instances. But there is a performance difference of 15%-25% between SAS and SATA and that can't be ignored.
> In our experience, administrators are overwhelmingly likely not to notice when a hardware controller's cache batteries fail. Frequently, those administrators will still be operating their systems at reduced performance and reliability levels for years afterward.
Can anyone else weigh in on this claim? In the two jobs where I've been responsible for such equipment, monitoring both the performance and battery status was pretty critical and done with due care and focus.
105 comments
[ 5.5 ms ] story [ 170 ms ] threadHe did mention it as a side note, but it deserves more attention I think.
Most consumer drives are still sold rated as <10^14 bits read per error. That's 12.5 terabytes, so in the worst case you could end up in situations where — on average — you are unable read a full 16TB drive without an error. Needless to say this is less than optimal for rebuilding a failed RAID array.
Anecdotal evidence (i.e. very low error rates during ZFS scrubs) suggests that manufacturers underrate their drives and they are much more reliable than that, but it is something to keep in mind.
Fortunately drive capacity completely outpaced my needs for personal data storage in the recent years, so I am happy with JBOD or RAID1, with backups of course.
[1]http://www.raidtips.com/raid5-ure.aspx [2]https://magj.github.io/raid-failure/
I've seen plenty of drives returning invalid data with correct CRC over the years. On reliable server-grade Xeon + ECC hardware. Of course the vast majority of drives never do it, there's just no way to know which ones do until it happens.
Firmware bugs in weird corner cases? Cosmic rays? Perhaps, but I think it's more reasonable just consider it one of those weird things that occasionally Just Happen (TM) and just need to be protected against at a higher level.
All drives produced in the last 30 years or so are running ever more complicated software stacks. For example they all have features that move data at risk to safer locations without the host system requesting or even knowing about it. Their physical (like bits on spinning rust or NAND flash block) and logical (what the host sees) data representations can be completely different.
Plain old CRC errors, though, are way more frequent. I feel much more comfortable about those errors, at least the drive knows the data is corrupted.
I switched to ZFS a few years ago and it's really quite amazing. In fairness, raidz does suffer from the same "resilvering takes a long time and increases the chance of subsequent failures" problem, which is one of the reasons why mirrors are more commonly used. Though unlike RAID, ZFS knows which data on the disk is actually used so can reduce resilver times pretty drastically.
Question for you smart people... is there a reason to use such an old Linux kernel version? They're testing on 4.15, which was released over 20 months ago. Would there be any appreciable difference on 5.6.4 or something in the 5.x range? Perhaps not since hard drives can't really improve much, but I'm curious.
The block layer has been able to deliver very close to 100% of the drives' stated bandwidth for many years now.
> _We find that with hardware RAID, it's frequently difficult to tell whether you're nuking your array or importing it safely. So in the event of a controller failure and replacement, you may end up sweating bullets, YOLOing, and hoping._
This was the main reason I decided to switch to Linux software RAID from a 3ware card years ago.
A thunderstorm caused a power trip and after powering the system back on, the 3ware controller lost its EEPROM contents that held the PCI vid/pid. Fortunately, it was just a RAID 1 mirror. We used dd to cut past the metadata header and managed to mount the ext2 filesystem.
Using software RAID, I’m confident that we can take the drives to another machine in a pinch and still be able to mount it without much hassle.
In 05 Dell sold me a workstation and internal RAID array that suffered a drive fault requiring replacement and rebuild. Unfortunately the engineer who was handholding me through the steps just to make the changes under our support conditions of the job being theirs and otherwise we left alone, told me it was a LSI card and not a Adaptek installed. This was a very entry level card and the only such card I've experience of. (Accustomed to the luxury of the high end) So I had to take him on his assertion it really was a LSI OEM PERC controller. Erm, not exactly. I learned this when a second connector dislocated and I blithely yanked and reseated U32PSCSI ribbon neatly and promptly as I imagine anyone would. Only the PERC was Adaptek and not LSI and for reasons my recall may be protecting me from* failed the second drive in the R10 array.
I vowed to never run any hardware RAID again and I have maintained my resolution until last month, if specifying a rack of dedicated storage counts for proprietary hardware in equivalece of black box opacity.
I should have written my experiences and blogged them because I took this loss of the subsystem so badly I became obsessive about learning first how to reconstruct the borked array from raw disk, a pleasure indescribable in polite company, and carried on until I was holding forth in charge of a budget for storage space in my dreams I never imagined presiding over. Every which way I believe only in software control of stores. I'm a true disciple to the cause. This r as rack of proprietary storage hardware only got its PO after the methods of code escrow had been thrashed to death in the final thrills of fall and last winter. (Incidentally the outside legal lead wrote me privately afterwards and pondered out loud whether the escrow didn't create a backdoor kind of vendor lockup because of the potential for contaminating employees with IP during emergency contingencies operating. Thanks Buddy! I'm not sounding like it but I'm actually grateful for that consideration if only because it sounds good for avoiding any repetition of the process just endured. It's a dual licensed codebase but being on the commercial code and the open code trailing in develop... I'd love to see what eg DDN routines are like I'm absolutely sure eager audience awaits publications like that.
Surely the lock down must have awoken the industry to the needs of self sufficient repair and diagnosis.
Hardware RAID is just data corruption waiting to happen, simply because there are more vulnerable components and interconnects along the data path.
It's much better to checksum data while it's still protected by ECC RAM and CPU ECC caches – which is something only software RAID can do. Although even that is not a total guarantee.
There's no longer reason to assume a slow SoC on a RAID adapter (even with XOR acceleration) would perform better than an Intel or AMD server CPU, as long as I/O path is not bandwidth limited.
Also hardware RAID allows for a more consistent performance under heavy system load in my experience.
This is not what I was talking about. Of course they do. The problem is that data needs to somehow be transferred to the adapter first, and this alone adds a point of failure. Controller can (and too often do) have hardware and firmware bugs. The less components the lower chances for corruption.
> battery backed-up cache for the past decade, but supercapacitors and flash memory
No difference to software RAID there. For example ZeusRAM and Optane are used for this. Of course high endurance SLC flash with supercaps is also an option.
> Also hardware RAID allows for a more consistent performance under heavy system load in my experience.
I'd tend to count this subjective, but I'll grant you this one, it's at least plausible. I haven't encountered this issue, though – today's hardware has insane bandwidth. SW RAID can saturate 100 Gbps ethernet, how much more performance you really need?
Even if performance was slightly degraded, data integrity is more important.
"640K ought to be enough for anyone". To answer more fully, you need as much as you need. You may not, but others will need more.
Notoriously embedded CPUs have a much worse price/performance ratio than mainstream CPUs so almost inevitably they use a much weaker CPU than is needed.
This is one reason why WD is involved in RISC-V, so they can escape this, and another reason why Intel's storage strategy (ex. optane) has worked out, because the main CPU has power to RAID NVMe SSDs and off-brand CPUs don't.
But yes, not a good post.
It makes no sense to seg out [superior and more agile] software functionality to a one-and-done ASIC or fancy FPGA that has a magnitude order or more of overhead/management that you must immediately invest in.
When I can't feed Nx10GbE or 100GbE with ZFS, I'll consider this an issue. Until then, it is merely an exercise in logical masturbation.
Nobody said everyone everywhere.
Nobody said forever.
Something wrong with dynamic disks for software raid for the os in windows?
https://docs.microsoft.com/en-us/windows/win32/fileio/basic-...
Or storage spaces? (the latter i have not used)
https://support.microsoft.com/en-gb/help/12438/windows-10-st...
Remember to enable ReFS integrity streams! ZFS protects full integrity by default. NTFS supports only protecting metadata, file data integrity is not protected.
Fill the volumes with known data. Randomly corrupt some blocks out of disks (but not from both disks at same offsets!).
Observe which solution recovers and returns correct file data and which does not.
Hint: you might want to stop using dynamic disk RAID after this experiment...
Storage Spaces operates on a higher level, building on lower level volumes. I'd feel way more comfortable using it on ReFS, at least once it can be considered sufficiently mature in the future. See: https://docs.microsoft.com/en-us/windows-server/storage/refs....
No qualms about trusting ZFS, it's very solid on good hardware. Of course running ZFS on Windows in production might be a Very Bad Idea, so do it on FreBSD, Solaris derivatives or Linux instead!
Last I checked... VMWare VXSI does not support a software RAID at all anywhere. I think that’s still fairly common in server environments.
See for example https://www.vmware.com/content/dam/digitalmarketing/vmware/e...
You get what you pay for.
I've always wondered whether anyone has gotten fed up with this, and decided to just virtualize Windows Server under Linux in order to feed the Windows Server VM a virtual disk that sits atop all the Linux storage-layer tech (but where otherwise the Windows VM gets all the rest of the computer's resources passed through.)
(Come to think of it, I've considered doing the same for a Hackintosh at some point. It got too contrived, but I'd like to come back to it one day.)
But you're 100% correct. I have friend who have few RAID controllers in his office just in case that one of it's servers fail. And all of them are purchased in same order to be from same batch.
Using this RAID calculator gives me worrying numbers for high capacity arrays, even with enterprise hardware:
https://wintelguy.com/raidmttdl.pl
Also, expanding the capacity of an array is a needlessly complicated process:
https://raid.wiki.kernel.org/index.php/Growing
The best solution seems to be something like Ceph... But that isn't practical for a home lab.
Yes? But that's unrelated to the comment you were replying to.
libeatmydata is a small LD_PRELOAD library designed to (transparently) disable fsync (and friends, like open(O_SYNC)). This has two side-effects: making software that writes data safely to disk a lot quicker and making this software no longer crash safe.
DO NOT use libeatmydata on software where you care about what it stores. It's called libEAT-MY-DATA for a reason.
Also just some latin pedantry, the article mentions the phrase: Caveat imperator a couple of times. I assume to mean Let the buyer beware, which as far as I know is Caveat emptor.
Google latin translating: https://translate.google.com/#view=home&op=translate&sl=la&t...
Does caveat imperator have another meaning (perhaps the emperor is infallible)?
https://en.wikipedia.org/wiki/Imperator
https://arstechnica.com/information-technology/2020/04/under...
Please note that the Author Jim Salter also has a nice podcast: https://techsnap.systems
Also interesting: https://arstechnica.com/gadgets/2020/02/how-fast-are-your-di...
My main gripe with the article is that the topic of latency seems underdeveloped.
The testing I've done using some of the parameters discussed on the article shows extreme, probably unrealistic latencies on the storage.
This causes RAID rebuilds to take longer. But I am not aware of any hard evidence that this means that this increases the risk of a failed rebuild.
There are more sectors that could be bad but you need to do a patrol read / scrub at least one month to detect them in advance.
I think there are a lot of scares about RAID of which I really wonder how much of it is rooted in anekdote and folklore.
I had five Seagate 7200.11s (terrible, shitty drives in many ways) in a RAIDZ1 pool. One failed, I started a resilver. Another started failing during that time. The rebuild slowed significantly due to the second failing drive. ZFS resilvered eventually then the second drive failed completely. That was at only 26k hours or so. I think a third started acting up shortly after but I was already replacing them all.
These were only 500GB drives, and the chances aligned on two of them....
I only use RAIDZ2 now (and I don't use Seagates).
Question about ZFS and its RAIDZ: do you have any recommendation (personal experience, links, books, ...) concerning parameters/setup to be used when setting up a RAIDZ(1 and 2)?
I'm new to ZFS and I already had to do a lot of tests when using ZFS only on 1 HDD until I finally managed to get good performance out of it (used by a "Clickhouse" database which itself writes data in a CoW-style => I had to raise the "recordsize" to 2MiB) and I imagine that with RAIDZ it can get more complicated?
I would like to set up a RAIDZ for the database (again, "Clickhouse", which generates multi-GB files) and two more to be used as simple NAS (a main one and a backup, storing files of all sizes).
I searched a lot and found some websites which were "ok", and bought as well 2 tiny books ( https://www.amazon.com/Introducing-ZFS-Linux-Understand-Stor... and https://www.amazon.com/ZFS-Linux-Administration-William-Spei... ) but the books were mediocre and the informations I found in the web were sparse and a bit oldish.
Cheers
I'll try to think of some others.
Honestly the best way is to start tinkering and see how it works for you as it can depend heavily on what you're using it for.
But yes there is a lot of FUD out there.
One thing to beware of, RAIDZ1/2 is limited to the IOPS of one drive per VDEV... Can be quite limiting for some things.
Didn't know that website - it has a lot of interesting stuff.
Ok, I'll then do some tests and will see how the RAIDZ behaves.
What it offers (in parity mode) is improved reliability and more chance that your server will be online if your drive dies. In majority of cases, it saves you from restoring from backup. Just replace the bad drive, and away you go! However, never assume that it will protect you from data loss. Backup. Backup. Backup!
That said, an argument such as "system is down while we restore from backup; some recent data may be lost" will not go over as well as "the system remains available, but performance will be negatively impacted while the array rebuilds; no data has been lost at this time".
In the former, you're in crisis mode. In the latter, you'll probably be under some stress, but you're still in the clear.
Edit: But we're all on the same page when it comes to backups. They're a necessity.
I.e. ceph with pods distributed over a bunch of disks without a raid
Not if you need speed it doesn’t.
A tape drive costs ten times what a hard drive with the same capacity costs, and then you have to buy the tapes.
I've been able to read data written to a hard drive 99.99% of the time but I can't say that for tapes. Often it has been human error, but the fact that it's a strange thing you don't do everyday and isn't part of your workflow is a reliability problem that is hard to overcome. (e.g. why a Toyota Corolla is more reliable than a Ferrarri Testarosa.)
In theory you could do offline backup to "the cloud" but probably your net isn't fast enough and the other people at home will be wondering why Zoom isn't working for them.
Recovery is a bit funny with you marking all remaining drives as faulty then plugging backup for resilvering. (Or you can use it directly, less safely, as a normal separate array.)
Must be disconnected and unpowered, yet regularly checked. Not super useful for extreme long term storage but neither are tapes.
Mmhh, sounds weird - what do you mean with "disconnected RAID1"?
Sounds like a misuse of RAID features. What could possibly go wong? Eg. Reconnect the drive accidentally forgetting to mark others as bad and bye bye "backup"?
Also, RAID controllers die sometimes. It's not pretty when that happens, which means you really need a backup of the whole array anyways. So technically RAID is not a backup because you are still not protected from that type of failure.
This seems a bit like someone saying "my blog never needed Kubernetes". Of course not, that's not what it's for.
Obviously those things can be worked around with complete re-architecting of, well, everything, but I'm considering that out-of-scope.
10 is 1+0
I'm using two SATA SSDs in md RAID0 for VMs on my desktop (where each VM gets its own LV). Dangerous thing, but if one of them fails, nothing happens, I will shrug and reinstall.
You can take as an example the Recovery Point Objective and Recovery Time Objective terms used on disaster recovery. Depending on: a) how often you do backups, b) how many data are you backing up, and c) how fast are your network and transfer medium, you can calculate how much data you may lose in case of failure and how much time you'll use on recovery.
- If a drive fails, the system is offline until you restore from backup. Fine for personal use, but not okay if customers are paying you for a service.
- If a drive fails, any changes since the last backup are lost. Fine for long-term archival, not so great for a bank ledger.
- No capability to detect data corruption. RAID with parity can run a scrub to find and repair bit rot. If data is corrupted on your RAID 0, you probably won't notice, and you'll back up the corrupted. If your filesystem has checksums you will be able to detect the corruption but not repair it. Your filesystem probably doesn't have checksums.
If you frequently run I/O intensive workloads, the extra performance might be worth the tradeoffs. If you're looking for your PC to feel slightly faster it seems foolish.
- I've got 1 HDD failure on RAID10 4 x 10TB HDDs, I had to take the system down to do RAID10 rebuild. because running server with huge I/O slowdown the rebuild and I had my fears of second HDD failure. (I had to take the system down anyway). RAID didn't help in my usage scenario.
- I guess deduplication fixes this already.
- Any advice on ways fixing this and preserving my RAID0 usage scenario?
Which, to be fair, makes sense with the subject of the article limiting us to a discussion of 8 drives.
Plus, you’re talking about RAID as a backup, it is not. RAID is redundancy, not disaster recovery. If your backup solution is bas d on your RAID controller series, you’ve already lost.
So, here goes...
The management tooling for hardware RAID is a pain to deal with. Dell's OpenManage tools aren't that bad (other than being stupidly bloated), but every time I have to use LSI's MegaCLI utility, I end up wanting to shoot myself. (Seriously, do a Google search for "MegaCLI cheat sheet". It's _bad_.) For the cheaper LSI FusionMPT-based cards, I've completely given up trying to manage them from the OS, because I've never been able to get it to work reliably. And because hardware RAID controller manufacturers have changed ownership so many times over the past decade, even _obtaining_ the software can be a challenge. And that's assuming that it supports your OS, which isn't guaranteed.
Monitoring the health of hardware RAID arrays is challenging. Not only do you generally need to use a proprietary tool of varying quality to do so (with all the issues stated above), the information you get is typically not that helpful. Most RAID controllers will at least tell you if the array is degraded or not (i.e., one or more disks are missing), but the quantity and quality of that information can vary. Additionally, monitoring the health of the underlying physical disks that make up the array is also important, but getting this information from a hardware RAID controller is tedious (and sometimes impossible), which can leave you blind to the actual health of your storage. The disk health typically reported by RAID controllers (the greem/amber blinky light that indicates a failed or failing disk) is based entirely on the disks reported SMART health check status, which is so prone to false negatives that it may as well not even exist. Oh, and if you're using an OEM RAID controller with third-party disks, you often have to deal with a nag message informing you that the disks aren't "official," which at best is annoying, and at worst can hide legitimate disk issues.
Hardware RAID can have annoying limitations. Want to mix a SAS and SATA disk in the same array? Most controllers won't support that. Want to mix RAID levels on the same disks, like in the case of a distributed database where you might want some redundancy for the OS volume, but the actual database volume can be RAID 0 for performance/space efficiency? Most controllers don't support that, either. Want to create multiple distinct logical volumes from a single array (e.g., a 20 GB volume for OS, and a 10+ TB volume for data)? That's usually possible, but in the case of many LSI MegaRAID controllers (including their Dell PERC rebrands), that can prevent you from expanding the space on the array without a rebuild. Want things like RAID 6, more advanced nested RAID levels like RAID 50 or 60, or tiered storage? If the controller supports it at all, it's probably a premium-feature that costs extra to unlock.
Hardware RAID is slow. The onboard processors on most controllers have no problem keeping up with mechanical disks, but struggle with the performance potential of SSDs. The centralized cache on the controller can be useful in some situations, but the typical configuration is to use this cache while disabling the cache located on the disks themselves, which runs into scalability issues for obvious reasons. And lastly, you're ultimately limited by the throughput of the bus to which the controller's attached, which means that a hardware PCIe RAID controller can't possibly compete with NVMe disks that are directly attached to the processor via the PCIe bus.
Softw...
But, again, this is very low end to middle end 'hardware raid'. It's single-digit disk stuff. It's a cheapo PCI card in a cheapo server.
You can multiplex hundreds of disks across multiple SAS backplanes. And, yeah, those software RAID [0] systems are really cheap. :-)
Software RAID is deployed in low end systems, sure. But also in very high end systems where compromises can't be made. And anywhere in between.
[0]: For example, see: https://www.oracle.com/storage/nas/zs7-2/
Even hardware RAID is somewhat of a misnomer. Just like the disks themselves, it's just an embedded CPU, varying degrees of hardware acceleration, possibly battery/ultracap backup and a big software stack.
Software RAID can achieve high capacity and/or performance. No difference in principle to hardware RAID there. You can use multiplexing for huge SAS backplanes in either case.
I manage a small (<100TB) storage system for our office. We have servers using RAID10 SSDs and we have other servers using RAID10 10k SAS HDDs. The read performance on the SATA SSDs is great but the writes are atrocious.
If a 10K SAS HDD array is outperforming even a single SATA SSD, either your I/O access pattern is extremely sequential, or there's something wrong with how your storage is configured.
Can anyone else weigh in on this claim? In the two jobs where I've been responsible for such equipment, monitoring both the performance and battery status was pretty critical and done with due care and focus.