This is what lot numbers are for, the tracking (and in this case: leveraging) of variability between production runs. Spread multiple lots over more than one storage array and you're good to go.
Neither will brand diversification, where they can not only independently implement their own vision of what a data eating binary blob should look like - they can also converge onto a single controller source. Wanna run down that potential failure mode on a dozen different drives, or just one?
Sometimes that's a good idea, sometimes it might make things worse. A previous server that had a RAID-1 of SSDs was limited by SSD write speed. As the SSDs got old the write speed dropped significantly. If I had 2 servers that each had the same brand of SSDs then one server would need both SSDs replaced while if I had 2 servers with different brands then each server would need a single SSD replaced which would have been more hassle.
I'm not sure it makes a lot of difference unless you have hundreds of snapshots. The way changes work to BTRFS is that every change without snapshots does most of the same stuff that a change with snapshots does. If however you have many snapshots (more than 500 according to rumours, I've seen it with over 1000) performance can get really bad. If your script to make snapshot backups works and the script to remove old ones doesn't work properly then the system runs fine for months and suddenly goes ridiculously slow.
A VM with a BTRFS filesystem running on a host using BTRFS gives serious write amplification. But if the real storage is SSD then it's not too bad.
What's nice is that as of Linux 5.16 BTRFS now supports ZNS (https://zonedstorage.io/) to drastically reduce write amplification compared to traditional SSDs by bypassing a lot of the FTL and moving to an interface that's similar to host managed SMR drives.
As a ZFS user: quite useful. I’ve had numerous of cases of corruption repaired by routine scrubs. Almost every other filesystem (btrfs being an exception afaik) would have been none the wiser.
In addition to cables and backplanes that have been mentioned, the entire path from RAM to PCIe to disk controller has non zero risks of errors. That said I'm pretty sure that the BTRFS errors I've seen from SSDs have all been the fault of the SSDs in question.
Also note that hard drives have had internal error detection and correction codes for decades, so SSDs doesn't change things much. Probably most checksum errors are misplaced writes and missing writes.
HDDs also use ECC hence why read errors occur on failing drives instead of just silently returning corrupt data. That's not an ironclad guarantee and plenty of places in the data path can let an errant cosmic ray flip bits. BTRFS doesn't even eliminate this, it just reduces the surface area and most importantly gives you a way to durably detect or correct this from when the checksum is generated in the kernel onwards.
+1, having the visibility (and the ability to correct it) is also often a leading indicator of drive failure.
I would argue though that at this point most other filesystems have this, but they're not as ergonomic/obviously built in. LVM has integrity checking at the volume level (if you give drives to the pool instead of giving the pool an already-RAIDed device), mdraid has dm-integrity. The options are there but not quite as discoverable.
BTRFS RAID1 has numerous advantages compared to regular RAID1. It does not mirrors whole disks, but chunks of the disk, usually 1GB chunks. This has numerous implications:
- drive size can be mixed without usable space loss (kind of, with some combinations you end up losing a bit)
- you don't need an even number of drives
- you don't need a raid0 atop several raid1 to use e.g. 6 drives, it's all one big raid
- a drive can be swapped for a bigger or smaller drive without issue, meaning spare drives don't need to be identical
This is such a great feature, it makes it really easy and convenient to expand btrfs arrays. It's such a pity that the parity support is still unstable. Now that ZFS has RAID-Z expansion I don't need to wait anymore.
How would a Btrfs RAID1 array of (equally sized) spinning rust and SSDs work? Will it slow down to the speed of the HDD, or will I get SSD speeds and will the HDD eventually catch up? If not by default, then by option?
In the post I wrote 'between the 2 SSDs SMART reports 18 uncorrectable errors and “btrfs device stats” reports 55 errors on one of them'. That's 37 errors that BTRFS found that mdadm wouldn't have found. 37 errors out of 500G of data isn't a lot, but it depends where it occurs, if it's in the root directory then it will mess up your day (annoyingly BTRFS doesn't track where the errors were so you can't easily determine what the implications of not having BTRFS might be). That is not unusual, I have had similar things on almost all my systems that run BTRFS or ZFS.
I've got 73 vms running on a production hypervisor (lvm-backed storage) and using their methods it's averaging 433.1 GiB/day over the disks for the VM storage. I don't know how they're managing double that hosting a blog.
The "server which runs my blog (among many other things)" probably has most of it's writes from a mail server for dozens of domains and has some commercial web sites that are very interactive and database backed.
or throw money at RAM rather than have any swap space at all, I have seen used dual xeon servers (2 socket, 16-core per socket) with 512GB of RAM for sale very slightly under $2000 now, with no storage in them. By the time you add a few nice fast SSDs and a 10GbE NIC the all-in cost is looking under $2850.
Instead of using swap (on disk) at all, try deleting the swap partition and just use zram [1]. It essentially puts swap in-memory but compressed. This is likely faster than disk anyway, and avoids worrying about endurance at all. I don’t bother with swap partitions anymore.
zram is a good thing. However in my investigations in 2014 "my conclusion was that swap wouldn’t be a problem as no normally operating systems that I run had swap using any significant fraction of total disk writes". During the last 8 years the amount of RAM in all my systems has increased significantly so swap is even less of an issue.
Sure but setting up swap is still recommended. I guess some internal linux memory algorithms prefer to have some safety net? Setting up a 1GB zram swap was effective for me, it‘s not much wasted memory as servers have so much memory these days and because of compression it can fit more than 1GB.
What I've seen is that under memory pressure kernel tasks trying to evict or free pages to satisfy an allocation request can race with other tasks dirtying and filling pages even faster, especially via the buffer cache. This can induce patterns of lock contention on low-level VM data structures and flushing procedures that effectively behave like a deadlock. Eventually various loop limits and lock timeouts will help unstick things, but in the worst cases the system gets caught in higher order loops and I've seen systems lock up for minutes. The systems I've seen this on never had any swap.
Some of the heuristics designed to minimize pathological contention latency seem to implicitly assume that the swapping subsystem--both in its ability to help free space, and the latency it introduces when loading and evicting pages--will help mitigate the chance tasks will get caught in a tight contention loop. IOW, the I/O latency of swap effectively induces back pressure on load, helping operations freeing pages to progress faster than operations consuming pages. (Predictably, the faster your swap, the less well this works. When people began putting swap on SSDs, heuristics had to be retuned.)
Arguably the root of the problem is the legacy of overcommit. Even though it can be nominally disabled, many aspects of the kernel were designed with the notion that the only direction to move under memory pressure is forward, relying on the promise of the OOM killer eventually freeing up enough memory to maintain forward progress on the current operation, rather than unwinding and returning a failure condition. The dynamic seems similar to buffer bloat.
Overcommit on POSIX systems is a consequence of fork/exec architecture of starting new processes. Fork must double the memory accounting if one disables overcommit and for a memory hog that starts a helper process that may lead to OOM when when in fact there are enough memory.
Windows, that does not have fork, does not suffer from this.
POSIX is not restricted to a fork/exec model, there's also posix_spawn(). Additionally it may be possible to somehow rely on COW to avoid double accounting after fork, provided that the new process execs "soon after". This would cover cases where posix_spawn() cannot be directly used, because some fixup needs to occur before the new process exec's.
Without double-accounting after the fork it is in general impossible to guarantee that no running program needs to be killed on OOM as the kernel does not know what the process is going to do after the fork. And fork still has legitimate users that does not involve exec. With multi-process architecture that many programs like browsers use for security it is a common optimization to have a seed process with sandbox initialized that is forked as necessary into a worker process.
A fast swap like zram or modern SSD is very beneficial on a development machine as one allows to keep active file cache for much longer so grep and git on huge trees works instantly even if one have a lot of memory hogs like VMs or language servers around.
Modern SSD are much faster at read than lz4, the fastest compression algorithm, at decompression and just 2-3 times slower in practice when writing than lz4 at compression. And they do not tax CPU while doing that, which is important since heavy memory usage may come from CPU-intensive tasks.
I did some benchmarks in my laptop and for heavy compilation tasks that may occasionally spike the memory usage over the amount of RAM memory compression was not beneficial compared with using SSD for swap.
The picture changes if one needs encrypted storage and cannot use hardware encryption on SSD. The software encryption will tax CPU on roughly same scale as lz4 and lz4 is still faster overall for read/write combination.
When I was using a Thinkpad X1 Carbon Gen6 with Linux it would hit a thermal limit and throttle the CPU speed if I read from NVMe at maximum speed for more than about 30 seconds. That was partly due to an issue with the way Linux and the BIOS interacted which gave a lower thermal limit than expected.
But it really depends on what NVMe devices you use, I've seen cheap ones bottleneck at 30MB/s when doing continuous writes with no downtime.
RAM is getting so cheap and easy to get that in recent years I only really use swap on laptops. Using a CPU core for compression isn't a big deal in a laptop because you usually have 4 cores and only 1 of them in use most of the time.
On my laptop (i7-1165G7, DDR4 3200 MHz) lz4 decompression is ~3000 MB/s, compression is ~485 MB/s (on my 3960X desktop it’s ~1.5x that) (numbers from lzbench). Zram by default uses one stream per CPU, so theoretical max is (I think) 8x those for the laptop and, well, a lot more for the Threadripper. Non-Optane SSDs only really have a potential advantage in sequential writing. Max throughput aside, Zram will generally have better latency (and thus likely IOPS as well), and better consistency since most SSDs get their max performance from DRAM caches.
In the end, though, quibbling over synthetic numbers doesn’t matter much if you have benchmarked your actual use case. I don’t know what “not beneficial” means (the same? 10x worse?), but part of my original point was that
With modern big SSD endurance is no longer the issue. You would need to write at max speed for many month before hitting the limit. I had a laptop with 512GB SSD that I used heavily for 3 years for development work and then for 2 years as a backup machine with light usage. Its write counter reports only 7% of the max usage. So on a new laptop with 2TB SSD after benchmarking things I just disabled zram/zswap. But then again, it was critical to enable hardware encryption on SSD rather than using LUKS on Linux. The latter is so slow [1] that zram is really a must.
Well it all depends on how much writing one does. OP estimates they have 1 year before they use too much of their drive’s endurance.
Be aware that on-disk hardware encryption should generally not be trusted. It is not auditable or trivially verifiable. Some years back there were some revelations that some drive firmware encryption was woefully deficient. So much so that whereas Windows previously would treat on-disk encryption as sufficient, Microsoft stopped relying on it entirely.
I am aware of this. Plus there are constant power attacks when one disconnects the disk while the system is suspended when the key is unlocked on the disk.
But this is a development machine and I figured out that surprisingly non-trivial overhead of LUKS encryption in Linux is not worth it. On the plus side wake up from hibernation is very fast and to mitigate the risk of the above I configured the laptop to hibernate shortly after suspend.
However, I did benchmark using stress-ng my laptop with Samsung 980 PRO SSD and when the stressors had to use swap, using zram or zswap was like 5%-15% slower than the real swap according to bogo ops counters that the tool reported. And the more CPU was taxed during the tests, the slower zram became while CPU utilization has not affected the real swap.
Another potential offender is the hibernation process. It's probably compressed but still writing several GB to the SSD every time the machine goes into hibernation certainly doesn't help.
I am disappointed that in Linux the hibernation process is so dumb.
For example, if you hibernate a Linux machine, then boot it up and immediately re-hibernate, there should be barely any memory to write out (since very little will have changed since the last hibernation). Yet the written data is the whole image with no regards for the fact most of the data is already on disk and doesn't need to be rewritten.
The code and feature duplication between "zram" and "zswap/zcache" seems quite unfortunate. Is there no prospect of unifying these kernel-side tools in the future, so that a single configuration choice can be used to set up memory compression regardless of it being "backed" by a physical swap device?
When I set up Debian on this machine (an old Proliant ML330G6 with 96Gb of memory) I did think "what do you do with swap?".
In the before times, when I was a SunOS/Solaris admin in the 1990s we used to have a rule of thumb handed down from the boss that swap should be 2x physical memory but in the intervening years this seems like massive overkill. The Debian installer gave it 1Gb of swap as a default. Probably not enough to get you into trouble if something decided to go truly rogue but you can always hit the off switch.
I did toy with the idea of giving it 200Gb of swap but watching the process monitor with the dev environments running and a few VMs I struggle to break 64Gb of memory usage. It is running hybrid disks and not SSDs though.
If I ever bother to put SSDs in it (unlikely) I guess I'll think about it again but swap seems almost irrelevent on a workstation now. On a server...would probably have to get a new rule of thumb.
Don't you still need at least as much swap as RAM if you want to use hibernate? It used to be that hibernate mode simply dumped the RAM into the swap. Does it still work that way?
It still works this way on Linux. It's better to overprovision the swap though, if you actually intend to use it as swap at runtime. Otherwise there may not be enough free space for the hibernation.
I'm typing this on a Windows system with 32GB of memory and an idle swap file. What I currently have open would probably fit into 16GB. But at the same time, I have over 30GB swapped out right now. 2x physical memory is about right for my desktop, when accounting for web browsers and electron apps and other programs the soak up gobs of memory and leave that data alone for days or weeks on end. Then there's the committed but unused memory on top of that, representing another 20GB of page file I need to have just to make it happy.
And then I have linux servers with 1-4 gigabytes of swap that use either none at all or a couple hundred megabytes of it.
There is also an architectural difference in how Linux and Solaris (at least historically) handle swap. Linux adds swap as additional memory, Solaris backs RAM with swap. So 32MB RAM plus 64MB swap give you 96MB of allocatable storage on Linux, but 64MB on Solaris. That is the origin of the "twice as much swap as RAM" rule.
On my MacBook Pro 2015 I am averaging about 100GB+ per day simply because every time I trip the MagSafe Cable macOS will start paging or doing lots of write on Kernel_Task.
Generally speaking I dont think macOS as a whole dont put much thought into SSD Endurance. That is a problem because you cant swap your SSD on Mac.
Are there many reports of SSD failures on M1 Mac's?
I suspect Apple might have tweaked SSD firmware to behave better in various ways and increase endurance or performance near the end of the ssd's life, because if they hadn't I think we'd be seeing a flurry of people complaining about SSD failures by now.
Most things last longer than their warranty, if even 5% of an item fail to reach the warranty then it's a disaster for the manufacturer. Planning for replacement is probably a good idea when you are 5* the warranty.
I intentionally use spinning hard drive on any data that are transient or have heavy write operations: swap, logs (ie /var/log/), etc. The spinning hard drive should last a long time and they are fairly cheap, while the performant and main data stays on NVMe or SSD.
If you use the - option in syslogd configuration for non-synchronous writes then /var/log will be only a tiny portion of writes. If you have a mail server then the many log messages about a message delivery will have hardly any write load compared to all the synchronous writes to the mail queue and mail store.
The only time I've seen a significant part of write load coming from /var/log is when the systemd-journald log database or a database server replication log or rollback log is there.
73 comments
[ 2.9 ms ] story [ 140 ms ] threadJust back sure they all use the same block size.
Funny enough, this plays very well with RAID10: with each leg made of 2 different brands, a firmware failure will not affect you.
A VM with a BTRFS filesystem running on a host using BTRFS gives serious write amplification. But if the real storage is SSD then it's not too bad.
Not sure how useful it is, but this is what btrfs/zfs people say
Also note that hard drives have had internal error detection and correction codes for decades, so SSDs doesn't change things much. Probably most checksum errors are misplaced writes and missing writes.
I would argue though that at this point most other filesystems have this, but they're not as ergonomic/obviously built in. LVM has integrity checking at the volume level (if you give drives to the pool instead of giving the pool an already-RAIDed device), mdraid has dm-integrity. The options are there but not quite as discoverable.
etc.
For writing, you have to wait for the slow one.
Also I use snapshots too.
[1]: https://www.kernel.org/doc/html/latest/admin-guide/blockdev/...
Some of the heuristics designed to minimize pathological contention latency seem to implicitly assume that the swapping subsystem--both in its ability to help free space, and the latency it introduces when loading and evicting pages--will help mitigate the chance tasks will get caught in a tight contention loop. IOW, the I/O latency of swap effectively induces back pressure on load, helping operations freeing pages to progress faster than operations consuming pages. (Predictably, the faster your swap, the less well this works. When people began putting swap on SSDs, heuristics had to be retuned.)
Arguably the root of the problem is the legacy of overcommit. Even though it can be nominally disabled, many aspects of the kernel were designed with the notion that the only direction to move under memory pressure is forward, relying on the promise of the OOM killer eventually freeing up enough memory to maintain forward progress on the current operation, rather than unwinding and returning a failure condition. The dynamic seems similar to buffer bloat.
Windows, that does not have fork, does not suffer from this.
I did some benchmarks in my laptop and for heavy compilation tasks that may occasionally spike the memory usage over the amount of RAM memory compression was not beneficial compared with using SSD for swap.
The picture changes if one needs encrypted storage and cannot use hardware encryption on SSD. The software encryption will tax CPU on roughly same scale as lz4 and lz4 is still faster overall for read/write combination.
But it really depends on what NVMe devices you use, I've seen cheap ones bottleneck at 30MB/s when doing continuous writes with no downtime.
RAM is getting so cheap and easy to get that in recent years I only really use swap on laptops. Using a CPU core for compression isn't a big deal in a laptop because you usually have 4 cores and only 1 of them in use most of the time.
On my laptop (i7-1165G7, DDR4 3200 MHz) lz4 decompression is ~3000 MB/s, compression is ~485 MB/s (on my 3960X desktop it’s ~1.5x that) (numbers from lzbench). Zram by default uses one stream per CPU, so theoretical max is (I think) 8x those for the laptop and, well, a lot more for the Threadripper. Non-Optane SSDs only really have a potential advantage in sequential writing. Max throughput aside, Zram will generally have better latency (and thus likely IOPS as well), and better consistency since most SSDs get their max performance from DRAM caches.
In the end, though, quibbling over synthetic numbers doesn’t matter much if you have benchmarked your actual use case. I don’t know what “not beneficial” means (the same? 10x worse?), but part of my original point was that
…part of my original point was that Zram will resolve the SSD endurance problems.
[1] - https://www.reddit.com/r/Fedora/comments/rzvhyg/default_luks...
Be aware that on-disk hardware encryption should generally not be trusted. It is not auditable or trivially verifiable. Some years back there were some revelations that some drive firmware encryption was woefully deficient. So much so that whereas Windows previously would treat on-disk encryption as sufficient, Microsoft stopped relying on it entirely.
But this is a development machine and I figured out that surprisingly non-trivial overhead of LUKS encryption in Linux is not worth it. On the plus side wake up from hibernation is very fast and to mitigate the risk of the above I configured the laptop to hibernate shortly after suspend.
However, I did benchmark using stress-ng my laptop with Samsung 980 PRO SSD and when the stressors had to use swap, using zram or zswap was like 5%-15% slower than the real swap according to bogo ops counters that the tool reported. And the more CPU was taxed during the tests, the slower zram became while CPU utilization has not affected the real swap.
For example, if you hibernate a Linux machine, then boot it up and immediately re-hibernate, there should be barely any memory to write out (since very little will have changed since the last hibernation). Yet the written data is the whole image with no regards for the fact most of the data is already on disk and doesn't need to be rewritten.
In the before times, when I was a SunOS/Solaris admin in the 1990s we used to have a rule of thumb handed down from the boss that swap should be 2x physical memory but in the intervening years this seems like massive overkill. The Debian installer gave it 1Gb of swap as a default. Probably not enough to get you into trouble if something decided to go truly rogue but you can always hit the off switch.
I did toy with the idea of giving it 200Gb of swap but watching the process monitor with the dev environments running and a few VMs I struggle to break 64Gb of memory usage. It is running hybrid disks and not SSDs though.
If I ever bother to put SSDs in it (unlikely) I guess I'll think about it again but swap seems almost irrelevent on a workstation now. On a server...would probably have to get a new rule of thumb.
Laptops would be a different story I guess.
I'm typing this on a Windows system with 32GB of memory and an idle swap file. What I currently have open would probably fit into 16GB. But at the same time, I have over 30GB swapped out right now. 2x physical memory is about right for my desktop, when accounting for web browsers and electron apps and other programs the soak up gobs of memory and leave that data alone for days or weeks on end. Then there's the committed but unused memory on top of that, representing another 20GB of page file I need to have just to make it happy.
And then I have linux servers with 1-4 gigabytes of swap that use either none at all or a couple hundred megabytes of it.
Generally speaking I dont think macOS as a whole dont put much thought into SSD Endurance. That is a problem because you cant swap your SSD on Mac.
I suspect Apple might have tweaked SSD firmware to behave better in various ways and increase endurance or performance near the end of the ssd's life, because if they hadn't I think we'd be seeing a flurry of people complaining about SSD failures by now.
My 980 Pro 1TB has about 3 PBW (about 5 times its rated warranty) and it still operates great.
The only time I've seen a significant part of write load coming from /var/log is when the systemd-journald log database or a database server replication log or rollback log is there.