As always the YMMV of caching is access patterns, but the more consistent cacheable pattern has been the ext4 journals for me.
They are tiny and often hit with a huge number of IOPS.
Ext4 supported external journals and moving it to a single SSD for a large number of otherwise slow SMR disks has worked great in the past.
However, when you hit a failure that SSD becomes a single root cause of data loss from several disks when losing that SSD (unlike a read cache).
Where I was working that didn't matter as I was mostly working with HDFS which both likes a JBOD layout of several disks instead of RAID (no battery backed write caches), tolerant to a single node failing completely and having a ton more metadata operations thanks to writing a single large file as many fixed-size files named blk_<something> with a lot of directories containing thousands of files.
SSDs were expensive then, but it's been a decade of getting cheaper from that.
> However, when you hit a failure that SSD becomes a single root cause of data loss from several disks when losing that SSD (unlike a read cache).
In theory you can massively reduce this risk by keeping a copy of the journal in memory so it only corrupts if you have a disk loss and a power outage within a few seconds of each other. But I don't know if the tools available would let you do that properly.
Yeah I really didn’t understand why ZFS or something like a TrueNAS dedicated storage solution setup was ruled out so quickly.
But before we even go there, we have to consider an all-SSD solution. SSDs are so cheap nowadays. It’s a trivial cost to buy a 2-4TB SSD. Heck, even an 8TB SSD is reasonably affordable, only about $600.
So first step is justifying a complicated caching setup. Why cache at all when you can just use all SSD storage and bring the complexity down a whole lot?
And if you’re using more than 2-8TB of storage and you’re in the spinning disk territory, why are you not using a dedicated NAS solution like TrueNAS?
This solution just seems like one of the worst possible choices.
something that people forget with raid1 is that this only protect from catastrophic disk failure.
this means your your drive need to be dead for raid to do it's protection and this is usually the case.
the problem is when starts corrupting data it reads of writes. in that case raid have no way to know that and can even corrupt data on the healthy drive. (data is read corrupted and then written to both drives)
the issue is that there are 2 copies of the data and raid have no way of telling with one is correct so it's basically flips a coin and select one of them, even if filesystem knows that content makes no sense.
that's basically biggest advantage of filesystems like zfs or btrfs that manage raid themselves, they have checksums and that know with copy is valid and are able to recover and say that one drive appears healthy but it's corrupting data so you probably want to replace it
I was surprised when I found out my fancy RAID5 card was using DMA for all of its disk accesses instead of having its own memory. Just the dumbest design. I had a power issue fry the last memory stick in my machine so it would come up clean and then under load report disk corruption. The disk was fine. Memory was broken. I rebuilt that fucking array three times before I ran a memory analyzer on the box.
Does someone know what the technology behind the tiering on QNAP NAS Systems is? I use an SSD RAID 1 in front of an RAID 10, which seems to work great.
IMHO flexible tiering rather than caching would be very nice for many Systems as it is rather difficult to teach users to separate rather stale data from changing data. Often does not have to be perfect.
A reminder that zfs recently (past ~5 years) implemented dedicated metadata cache devices ... which allows you to cache either filesystem metadata or even small files to a blazing fast SSD mirror:
This is a quick and easy way to add thousands of iops to even something very slow like a raidz3 zpool.
As always:
"Let's repeat, and emphasize: unlike an SLOG or L2ARC which are merely inconvenient to lose, if you lose your metadata vdev (your "special" vdev) you will lose your entire zpool just as surely as if you lost one of the other full vdevs ..."
RAID is great but without monitoring and alerting you can still have a problem. Better still is the automatic creation of incident records and escalation.
mdadm can be scripted using any ordinary system health monitoring system. On some servers I was responsible for, I had automatic monitoring setup configured to send e-mail the instant any drive failure was noticed.
This is one of the nice things with open software. I can't imagine the trouble it would be to do the same for a proprietary raid card.
When using LVM there is no need to use separate mdadm (MD) based RAID - just use LVM's own RAID support.
I have a workstation with four storage devices; two 512GB SSDs, one 1GB SSD, and one 3TB HDD. I use LUKS/dm_crypt for Full Disk Encryption (FDE) of the OS and most data volumes but two of the SSDs and the volumes they hold are unencrypted. These are for caching or public and ephemeral data that can easily be replaced: source-code of public projects, build products, experimental and temporary OS/VM images, and the like.
dmsetup ls | wc -l
reports 100 device-mapper Logical Volumes (LV). However only 30 are volumes exposing file-systems or OS images according to:
ls -1 /dev/mapper/${VG}-* | grep -E "${VG}-[^_]+$" | wc -l
The other 70 are LVM raid1 mirrors, writecache, crypt or other target-type volumes.
This arrangement allows me to choose caching, raid, and any other device-mapper target combinations on a per-LV basis. I divide the file-system hierarchy into multiple mounted LVs and each is tailored to its usage, so I can choose both device-mapper options and file-system type. For example, /var/lib/machines/ is a LV with BTRFS to work with systemd-nspawn/machined so I have a base OS sub-volume and then various per-application snapshots based on it, whereas /home/ is RAID 1 mirror over multiple devices and /etc/ is also a RAID 1 mirror.
The RAID 1 mirrors can be easily backed-up to remote hosts using iSCSI block devices. Simply add the iSCSI volume to the mirror as an additional member, allow it to sync 100%, and then remove it from the mirror (one just needs to be aware of and minimising open files when doing so - syncing on start-up or shutdown when users are logged out is a useful strategy or from the startup or shutdown initrd).
Doing it this way rather than as file backups means in the event of disaster I can recover immediately on another PC simply by creating an LV RAID 1 with the iSCSI volume, adding local member volumes, letting the local volumes sync, then removing the iSCSI volume.
I initially allocate a minimum of space to each volume. If a volume gets close to capacity - or runs out - I simply do a live resize using e.g:
lvextend --resizefs --size +32G ${VG}/${LV}
or, if I want to direct it to use a specific Physical Volume (PV) for the new space:
lvextend --resizefs --size +32G ${VG}/${LV} ${PV}
One has to be aware that --resizefs uses 'fsadmn' and only supports a limited set of file-systems (ext*, ReiserFS and XFS) so if using BTRFS or others their own resize operations are required, e.g:
One thing I’ve been doing lately is storing data on SSDs or raid0s of SSDs, then running continuous backups of changed files to a slow HDD. I don’t have single-transaction data safety requirements and the window in which I can lose data is very small, given that the backups run continuously.
This is so much faster than these hyper-vigilant HA setups, too, while being quite a bit cheaper (the HDDs can be in a different box or building, or be Glacier, or whatever).
make a zfs temp snapshot, cd to snapshot dir, rsync it somewhere (usually rsync over ssh), cd out of snapshot dir, delete snapshot, do it again. it’s fairly easy to script. i usually run it via runit in /etc/service/realtimesync or such.
LVM is a very cool thing that I feel is a bit underappreciated.
I've always built my systems like this:
two hard disks => RAID1 => LVM VG1.
nvme => LVM VG2
Because at some point I decided LVM is just like flexible partitions and didn't pay much attention to the huge amounts of other stuff in the manpages.
Turns out that's a pretty crappy way of doing things.
LVM can do RAID, can deal with moving things across devices, and has many other fancy features. So instead of the above, stick it all into the same VG without RAID on top.
You can RAID individual LVs. Some RAID1, some RAID5, some RAID0, some nothing, and pick on which disks stuff goes. You can add integrity checking which is not something MD RAID does. You can do caching. You can do writable snapshots. You can do thin provisioning. It's extremely flexible tech.
The only thing I'd recommend staying away from is thin provisioning unless it's necessary. The performance impact is significant, virt-manager for some reason doesn't see thin LVs as legitimate disks (can be worked around easily though) and thin provisioning seems to be able to develop faults of its own. I've had one old setup where one block was somehow not readable and it seems to be a thin provisioning thing -- it doesn't say the underlying disk is bad, it's something more along the lines of the metadata being broken.
To upgrade you can do things like moving LVs from one disk to another to free it up and remove it. Disks of different sizes are much less of a problem than with standard RAID, too.
~20 years ago I was helping someone with their storage server for their mail server, it had 8x 10K discs in a RAID-1. It was always struggling to keep up and often couldn't keep up. Intel had just come out with their SSDs, and I had a friend with access to one that was larger in size than the array. I shipped this little laptop drive off to get installed in the storage server.
Once the SSD was on the server, I did a "pvmove" to move the data from spinning array to SSD, while the system was up and running. Over the next several hours the load on the server dropped. It was comical that these 8 hard drives could be replaced by something smaller than my wallet.
LVM isn't underrated, it's obsolete. ZFS in particular has none of the many, maaany problems you run into trying to make LVM work reliably in practice (no self-corrupting snapshots, e.g.).
AFAIK LVM RAID is mdraid under the covers, I guess the advantage of using LVM to configure and manage RAID is doing it all with one set of tools, assuming that the LVM utilities offer everything you need. I've always used mdraid for that, and (sometimes) LVM on top of the mdraid devices.
It's a bit pain in three neck for me to rebuild my NAS using LVM or ZFS (to handle RAID in place of mdadm) but if I was starting over, I'd use either of them to avoid bit rot.
As it is though, I'm happy with RAID 10 on SATA HDDs a and RAID 1 on NVME SDDs, using bcache to create a single volume using both, and then ext4 on top. With sufficient SMART monitoring and backups, it'll do.
> Because at some point I decided LVM is just like flexible partitions and didn't pay much attention to the huge amounts of other stuff in the manpages.
I remember when I was first learning to use Linux, I was mystified by the convention of LVM on top of md. I was reading the LVM man pages and it was clear I could just do all that stuff with LVM without having to involve another abstraction layer.
But I had a hardware raid controller and just used that instead.
I should try and do something like this. I've got lots of media stored on several large spinning rust drives and combine them using mergerfs (a union filesystem). However, I think I'd need something that caches at the file level rather than block devices, as I've got separate filesystems on the different disks.
Seems like there’s a very old trick we may have forgotten: put your WAL file on the fastest storage you have to speed up your database. I’d like to see benchmarks comparing both on the same hardware.
This is a good writeup, but it's also missing the testing methodology (fio, a rsync job, etc) which allows one to understand the performance improvement quantitatively.
Been using lvm cache since about 2017 in high performance storage environment. Didn't like it much as it was too much of a blask box back then and didn't have any knobs to tune to adjust its behavior to the workload and we had plenty that didn't fit its logic. Ended up going full flash.
I have a special and burning hatred for mdadm and to a lesser extent LVM.
I suspect that nowadays they are much easier to look after, but when I first was introduced, the quality of the documentation was utterly appalling. WE had special classes with old sysadmins who passed on the secret knowledge for how to not fuck everything with mdadm.
LVM is slightly better, apart from the early false promise that was LVM snapshots which were essentially just a placebo(this might have changed now).
I've only ever seen LVM used once successfully in production. That was on a 60 bad LFF raid draw to create a single namespace over 4 hardware raid-7 disk array things.
Every other time it turn out to be a clusterfuck.
ZFS if youre poor, GPFS if youre rich.
I'm sure there are newer systems but they almost always fail ever the resilience, latency or performance test. (Cough gluster, ceph, anything that uses HTTP as a transport system)
I've got an LVM setup where the EFI boot stuff is triple RAID1 and the OS is on LVM with RAID5, but all other LVM volumes for data are selectively RAID0 or RAID5. It was not a fun experience to set up and it seems like certain actions could break it. But as long as it's respected it keeps going really well.
32 comments
[ 0.26 ms ] story [ 64.4 ms ] threadThey are tiny and often hit with a huge number of IOPS.
Ext4 supported external journals and moving it to a single SSD for a large number of otherwise slow SMR disks has worked great in the past.
However, when you hit a failure that SSD becomes a single root cause of data loss from several disks when losing that SSD (unlike a read cache).
Where I was working that didn't matter as I was mostly working with HDFS which both likes a JBOD layout of several disks instead of RAID (no battery backed write caches), tolerant to a single node failing completely and having a ton more metadata operations thanks to writing a single large file as many fixed-size files named blk_<something> with a lot of directories containing thousands of files.
SSDs were expensive then, but it's been a decade of getting cheaper from that.
In theory you can massively reduce this risk by keeping a copy of the journal in memory so it only corrupts if you have a disk loss and a power outage within a few seconds of each other. But I don't know if the tools available would let you do that properly.
The pro case for BTRFS is being able to do JBOD with a bit of additional comfort around mirror state over drives.
But before we even go there, we have to consider an all-SSD solution. SSDs are so cheap nowadays. It’s a trivial cost to buy a 2-4TB SSD. Heck, even an 8TB SSD is reasonably affordable, only about $600.
So first step is justifying a complicated caching setup. Why cache at all when you can just use all SSD storage and bring the complexity down a whole lot?
And if you’re using more than 2-8TB of storage and you’re in the spinning disk territory, why are you not using a dedicated NAS solution like TrueNAS?
This solution just seems like one of the worst possible choices.
this means your your drive need to be dead for raid to do it's protection and this is usually the case.
the problem is when starts corrupting data it reads of writes. in that case raid have no way to know that and can even corrupt data on the healthy drive. (data is read corrupted and then written to both drives)
the issue is that there are 2 copies of the data and raid have no way of telling with one is correct so it's basically flips a coin and select one of them, even if filesystem knows that content makes no sense.
that's basically biggest advantage of filesystems like zfs or btrfs that manage raid themselves, they have checksums and that know with copy is valid and are able to recover and say that one drive appears healthy but it's corrupting data so you probably want to replace it
IMHO flexible tiering rather than caching would be very nice for many Systems as it is rather difficult to teach users to separate rather stale data from changing data. Often does not have to be perfect.
https://www.rsync.net/resources/notes/2021-q3-rsync.net_tech...
This is a quick and easy way to add thousands of iops to even something very slow like a raidz3 zpool.
As always:
"Let's repeat, and emphasize: unlike an SLOG or L2ARC which are merely inconvenient to lose, if you lose your metadata vdev (your "special" vdev) you will lose your entire zpool just as surely as if you lost one of the other full vdevs ..."
This is one of the nice things with open software. I can't imagine the trouble it would be to do the same for a proprietary raid card.
I have a workstation with four storage devices; two 512GB SSDs, one 1GB SSD, and one 3TB HDD. I use LUKS/dm_crypt for Full Disk Encryption (FDE) of the OS and most data volumes but two of the SSDs and the volumes they hold are unencrypted. These are for caching or public and ephemeral data that can easily be replaced: source-code of public projects, build products, experimental and temporary OS/VM images, and the like.
reports 100 device-mapper Logical Volumes (LV). However only 30 are volumes exposing file-systems or OS images according to: The other 70 are LVM raid1 mirrors, writecache, crypt or other target-type volumes.This arrangement allows me to choose caching, raid, and any other device-mapper target combinations on a per-LV basis. I divide the file-system hierarchy into multiple mounted LVs and each is tailored to its usage, so I can choose both device-mapper options and file-system type. For example, /var/lib/machines/ is a LV with BTRFS to work with systemd-nspawn/machined so I have a base OS sub-volume and then various per-application snapshots based on it, whereas /home/ is RAID 1 mirror over multiple devices and /etc/ is also a RAID 1 mirror.
The RAID 1 mirrors can be easily backed-up to remote hosts using iSCSI block devices. Simply add the iSCSI volume to the mirror as an additional member, allow it to sync 100%, and then remove it from the mirror (one just needs to be aware of and minimising open files when doing so - syncing on start-up or shutdown when users are logged out is a useful strategy or from the startup or shutdown initrd).
Doing it this way rather than as file backups means in the event of disaster I can recover immediately on another PC simply by creating an LV RAID 1 with the iSCSI volume, adding local member volumes, letting the local volumes sync, then removing the iSCSI volume.
I initially allocate a minimum of space to each volume. If a volume gets close to capacity - or runs out - I simply do a live resize using e.g:
or, if I want to direct it to use a specific Physical Volume (PV) for the new space: One has to be aware that --resizefs uses 'fsadmn' and only supports a limited set of file-systems (ext*, ReiserFS and XFS) so if using BTRFS or others their own resize operations are required, e.g:This is so much faster than these hyper-vigilant HA setups, too, while being quite a bit cheaper (the HDDs can be in a different box or building, or be Glacier, or whatever).
I've always built my systems like this:
two hard disks => RAID1 => LVM VG1. nvme => LVM VG2
Because at some point I decided LVM is just like flexible partitions and didn't pay much attention to the huge amounts of other stuff in the manpages.
Turns out that's a pretty crappy way of doing things.
LVM can do RAID, can deal with moving things across devices, and has many other fancy features. So instead of the above, stick it all into the same VG without RAID on top.
You can RAID individual LVs. Some RAID1, some RAID5, some RAID0, some nothing, and pick on which disks stuff goes. You can add integrity checking which is not something MD RAID does. You can do caching. You can do writable snapshots. You can do thin provisioning. It's extremely flexible tech.
The only thing I'd recommend staying away from is thin provisioning unless it's necessary. The performance impact is significant, virt-manager for some reason doesn't see thin LVs as legitimate disks (can be worked around easily though) and thin provisioning seems to be able to develop faults of its own. I've had one old setup where one block was somehow not readable and it seems to be a thin provisioning thing -- it doesn't say the underlying disk is bad, it's something more along the lines of the metadata being broken.
To upgrade you can do things like moving LVs from one disk to another to free it up and remove it. Disks of different sizes are much less of a problem than with standard RAID, too.
Once the SSD was on the server, I did a "pvmove" to move the data from spinning array to SSD, while the system was up and running. Over the next several hours the load on the server dropped. It was comical that these 8 hard drives could be replaced by something smaller than my wallet.
But pvmove was the other star of this show.
As it is though, I'm happy with RAID 10 on SATA HDDs a and RAID 1 on NVME SDDs, using bcache to create a single volume using both, and then ext4 on top. With sufficient SMART monitoring and backups, it'll do.
I remember when I was first learning to use Linux, I was mystified by the convention of LVM on top of md. I was reading the LVM man pages and it was clear I could just do all that stuff with LVM without having to involve another abstraction layer.
But I had a hardware raid controller and just used that instead.
I got the idea of using mergerfs from this: https://perfectmediaserver.com/02-tech-stack/mergerfs/
I suspect that nowadays they are much easier to look after, but when I first was introduced, the quality of the documentation was utterly appalling. WE had special classes with old sysadmins who passed on the secret knowledge for how to not fuck everything with mdadm.
LVM is slightly better, apart from the early false promise that was LVM snapshots which were essentially just a placebo(this might have changed now).
I've only ever seen LVM used once successfully in production. That was on a 60 bad LFF raid draw to create a single namespace over 4 hardware raid-7 disk array things.
Every other time it turn out to be a clusterfuck.
ZFS if youre poor, GPFS if youre rich.
I'm sure there are newer systems but they almost always fail ever the resilience, latency or performance test. (Cough gluster, ceph, anything that uses HTTP as a transport system)