.. and that's how people end up with ZFS. My story was cheap SATA cables (or was it cheap power supplies? I never found out) introducing silent corruption and breaking decades of archived files.
MDADM RAID1 does not seem to have the ability to recover from silent data corruption. In the standard two-disk RAID1, it's actually impossible to recover, since it doesn't know what copy is the uncorrupted one. With three disk RAID1 it would theoretically be possible to use a majority vote, but I don't believe MDADM has such code.
Btrfs raid1, on the other hand, uses the checksums of the data stored in its metadata trees to validate which copy of the data is correct and repair the corrupted copy. 3-disk makes this even more robust, as you get an exponential growth of data-metadata pairs to pull from. If any one pair matches, that data can be assumed to be correct.
Not sure how well this works out in practice, but if one drive returns a read error for a sector and the other returns the corresponding sector without an error, it should be a simple matter to trust the latter. Of course that assumes that storage corruption occurred after uncorrupted data was written to disk.
Modern drives have rather large per physical sector checksums to detect the occurrence of these errors and presumably retry a couple times before returning an I/O error if configured correctly.
In addition a good I/O bus should have error detection checksums for data in transit, and good hardware should have ECC RAM as well.
Indeed. That's why I specified /silent/ corruption. If the stack returns an explicit error, that is correctable (assuming the other disks don't error and return the correct data).
Sadly, a large number of SSDs (and nearly all cheap USB thumb drives / SD cards / eMMC devices) will happily return silently corrupted data.
The command told it to reattach the drive and they expected it to rebuild but instead md saw part of the mirror and simply added the drive as a good drive.
And the drive would return bad data instead of just erroring out - this is sometimes a difference between “home use” drives vs enterprise - enterprise assume you’re in a multi redundant raid setup and instead of retrying will just fail out fast and let the raid take care of it. Home drives will continue retrying and perhaps even eventually give you what they got - the idea being you’d rather have one byte bad of a word doc than lose a 4K chunk of it.
ZFS catches these tricks because the checksums will fail. Multi-redundancy RAID can do it - but since most are enterprise they are built with the assumption the drive will return good data - or none.
This can be the worst scenario as corruption can be silent and infect all backups after a time.
> The command told it to reattach the drive and they expected it to rebuild but instead md saw part of the mirror and simply added the drive as a good drive.
> ZFS catches these tricks because the checksums will fail.
Yea, I think this is the right explanation. I guess it simply doesn’t keep any consistency information that it can check, whereas ZFS keeps a checksum of every block. Having used ZFS, it is difficult for me to understand how anyone could trust anything else. I don’t think that there is anything a ZFS user could do to accidentally destroy their data (aside from obvious things such as deleting the wrong filesystem or pool, which could happen in any case).
While switching from Hardware Raid1 to ZFS recently, I had 1) sata cables failing on me and 2) a 8088-8087 PCI-Bracket corrupting half of the SATA connections through it. I would have never detected this without ZFS (scrub). All was fine for weeks. I did `fio` speedtests, `smart --long`, `dmesg` monitoring. Nothing. Until I ran the first scrub and 22 CRC read errors appeared in the 6x8TB RaidZ2. Much worse after switching drives and cables around. I spend like $400 for different cable-replacements, but finally identified the failing ones and all is good since then.
Now, I am more observant. Scrub once a month. Monitoring of SSD TBW, UDMA-CRC, Temperature, Watt, Fan-Speed in InfluxDB 2.0 with alerts set. Besides, ZFS mails all the important status info.
ZFS is really good for peace of mind, resilience and predictability, not necessarily for speed (but not exclusively, either), or your wallet.
> Nonetheless, my plans are to rebuild treefort with ZFS and SSDs from another vendor. Whatever happened with the Samsung SSDs has made me anxious enough that I don’t want to trust them for continued production use.
Not sure if having one bad experience with mdraid is worth switching SSD vendors as well. You can mix and match different vendors, but they may have different performance characteristics as well.
Anyway, ZFS is a great choice either way. Definitely implement snapshots and a regular backup habit with plain zfs send/recv or syncoid.
I agree with getting SSDs from different vendors, for the following reason: I once had a FreeNAS system fail. It was using two (mirrored) SSDs to hold the OS. One SSD failed; while I was procuring a replacement, the second failed.
I later found that both SSDs were from the same batch, which (I think) is why both failures happened so close together. Purchasing same-size SSDs from different vendors (with similar product lines) helps ensure you're using SSDs that should hopefully fail at different times.
This was common knowledge on the internet before the "modern web" era. Never use in a RAID disks from the same vendor. When you have to use disks from the same vendor try hard to get different type/ batches. And backup.
I very intentionally bought two Samsung ssd from different countries, so I would get different batches. They show up on my door. Same batch, only a few hundreds apart in their serial number :(
If you had tried to get two that close in production, I think you’d have struggled. I didn’t bother and got drives off the stack and bought both, and they are a long way apart.
I first heard of ZFS back around 2009, and I don't regret my decision to go with it. But it isn't a panacea either. I nearly wiped out my storage pool one time when I moved it to a different machine without exporting it first. The pool came off an OpenIndiana system and the drives were not in their expected locations on the new box. Whoooops, the whole pool showed up FAULTED. After my panic settled down, I decided to put them back where they were, and the pool came up fine. whew
I've loved seeing the progress of ZoL. Whether I use Ubuntu's prebuilt modules or Debian's DKMS-based setup, it Just Works. But I'm still leery of running Linux with a ZFS root filesystem. I've compromised and use ext4 for root and put my home directory on ZFS.
> I've loved seeing the progress of ZoL. Whether I use Ubuntu's prebuilt modules or Debian's DKMS-based setup, it Just Works. But I'm still leery of running Linux with a ZFS root filesystem. I've compromised and use ext4 for root and put my home directory on ZFS.
If you want to use ZFS for root, I can heartily recommend NixOS. I've never had a problem with it there. Of course there are other downsides, but there are plenty of upsides as well.
The only thing exporting a pool does is declare "I am not being mounted by someone else". The FAULTED state would be because the search paths for the devices didn't exist. Clearing /etc/zfs/zpool.cache and doing `zpool import -f` would fix it entirely (ZFS identifies the disks by UUIDs in the headers AFAIK).
Mostly true, but I've found this doesn't work if one of the drives is missing. It doesn't handle renames simultaneously with drive failure, or at least it didn't four years ago.
Can also use cachefile=none and the name does not matter at all. Every disk vdev has a unique guid in the ZFS label area. Without the pool cachefile the guid is used to reliably assemble the pool, instead of the kernel device path.
My home fileserver has drives from four different manufacturers after I 20 years ago had bad experiences with closely correlated drive failures (the infamous IBM Death Star batch)
I've never deployed anything without at a minimum mixing batches, and usually vendors since.
We avoided data loss, but had a tense two weeks while cycling drives out as fast as we could (we had backups, but having to shut everything down to replace drives and restore would have been massively costly)
Those were actually some of my favourite drives cost wise ... -after- the problems happened.
Turned out at least on the ones I had that it was always the outer edges of the platter that failed, so a (from 20 year old memory, may be wrong) 60Gb Death Star if carefully partitioned out to a maximum of about 58Gb ran absolutely beautifully and you could get them dirt cheap because (understandably if you weren't a crazy person like me) almost nobody wanted to touch them at that point.
Slight segue but you sound like you know about ZFS (and if not, Cunningham's Law abides :D )
Does a VM on one physical host, backed up via nightly disk image on a geographically separate physical host, with the latter storing the backups on a 2-disk mirrored ZFS pool, count as a reasonable 3-2-1 backup regime? It's not storing mission critical up-to-the-minute user data, 'just' company files, so a one-day rollback would be annoying but acceptable.
Disclaimer: I'm a bumbling idiot who does this type of thing as a hobby.
Add nightly or weekly offline backups and you should be good to go. And by offline, I mean physically connecting a storage device, making the backup (preferrably via an automated script), and then disconnecting the device and storing it elsewhere in a safe place.
In your scenario, I understand that the original physical host where you're making a backup of a VM has no redundancy or data integrity guarantees? I'd just like to point out that backups are good, but if all you have are backups of the same corrupted data, then they are pretty much useless.
My approach is to have redundancy and data integrity guarantees (ZFS mirror) at the source and have offline backups on external hard drives. The original data is likely to be OK, and well, if I need to recover from a backup, I'll be happy enough with what I have there anyway.
Caveat that you probably already know but others reading may not: 'zfs send' while absolutely awesome when used for things it's designed for will stream the file system byte for byte so if there's corruption brewing you will absolutely transfer the corruption along with everything else.
(taking a snapshot, mounting the snapshot, and then backing up from -that- via something 'cp -a' like is probably the safest option, IMO, though it's always possible I've missed something here in which case I hope somebody will reply correcting me)
there's so many layers that could have caught the issue, even if you don't use ZFS... What's the point of splurging on EPYC with 256 GB of RAM if you don't want to spend a penny on resilience
> While I think Oracle and the Illumos and OpenZFS contributors should come together to relicense the ZFS codebase under MPLv2 to solve the GPL compatibility problem
For starters Oracle would need to relicense their work. Which is unlikely to happen any time soon. If they wanted to relicense it they would have done it by now.
All OpenZFS contributers would also need to relicense their code. If I recall correctly the OpenZFS codebase consists for close to 50% out of non-Oracle code. A lot of this code might be contributed by companies where the people who did the work no longer work there. Meaning there is no-one in those companies to get legal to sign-off on it.
All Oracle needs to do is release CDDL 2.0 that says "At your option, you may choose to relicense CDDL 1.0 licensed code under any of the MPLv2, LGPLv2+, GPLv2+, AGPLv3+ licenses" and then anyone can relicense any CDDL 1.0 code from any copyright holder under those licenses, since CDDL 1.0 contains a license upgrade clause. That would solve the situation for OpenZFS contributor's code too and also Oracles code and also dtrace/Solaris/etc code in general.
Its a pretty simple way to achieve compatibility between CDDL and other license spheres, which can only be a good thing really, personally I consider that incompatibility to have been a mistake that didn't really help anyway. Achieving compatibility is unlikely something Oracle cares about though, so the license situation isn't going to change anyway. BTW, they have relicensed things under the GPL before (IIRC ZFS for grub and ISTR something around dtrace), so they aren't adverse to it.
PS: I didn't think CDDL was used much outside of Solaris (now proprietary again and the forks of it aren't active I thought) and OpenZFS. Are there other significant CDDL licensed codebases?
> Solaris (now proprietary again and the forks of it aren't active I thought)
The Solaris forks (based on Illumos[1]) such as SmartOS[2] are pretty active!
> Are there other significant CDDL licensed codebases?
A lot of the Java stuff was also CDDL licensed. Though some of that (larger projects such as OpenJDK) were already dual licensed with "GPLv2 + classpath exception".
I also doubt it was used much outside of Sun. But even it's just the Sun stuff I still think no corporation would ever dare to make such a license update.
You'll often find angry greybeards on here griping about how you shouldn't do X or Y. Don't run your own mail, don't write your own crypto, don't run K8s. And you'll also find greenhorns talking about how they've been doing X for 10 years and never had a problem.
The thing is, both perspectives are valid. It's true that you can sometimes do really complicated things yourself and it can work out fine. And you can also get burned really easily. So the reader needs to figure out for themselves what the risk is and if it's worth it.
But there are things we should all do that are just never easy to do right, like.... backups. There's just no cheap and easy and good way. If you want real off-site backups, you're going to have to either pay a small fortune for a managed service (either in subscription fees or network egress fees or both), or physically carry a bunch of drives or tapes somewhere else weekly/monthly. And of course you also have to restore regularly or the whole thing could be a waste. It's just a pain.
You seem to conflate backups with off-site backups, and concluding that off-site backups are hard. However, nothing in this story would require off-site backups. A separate backup server on-site would have been fine here.
Of course, off-site backups are good, but would not have been actually needed in this case.
If you have a lot of data then a cheaper choice would be to put a raspberry pi at your mom's house, add a few external drives, forward a port, and send backups with restic.
One thing I like the idea of, that ZFS doesn't have, is redundancy broken down to a smaller size than the the whole disk so I don't need matched disk sizes and can easily expand a pool. Btrfs does this.
Difficulty aside, I'm wondering if it would be possible for ZFS to implement that as an option without giving up other features.
You can do it yourself at the cost of (a lot of) performance. Make say 4TB partitions on each drive, and make vdevs across the partitions.
Then you can upgrade by putting the partitions on different drives etc. Of course must then be careful not to put two partitions of the same vdev on the same disk.
The major downside is that ZFS isn't aware that the underlying devices are the same, so IO which looks nicely scheduled as far as ZFS is concerned might cause massive random IO on the underlying disks.
That said, if one could tell ZFS about which zfs-devices share physical hardware, so the IO scheduler would be aware, then that might be a viable strategy. Might be handy in case you have multiple iSCSIs sharing a 10G link as well.
Huh. I hadn't even considered that it would be possible to assign partitions to the vdevs instead of the entirety of the drive. Though I guess it does make sense that you can, otherwise you wouldn't be able to use a drive participating in a vdev as a boot device.
I don't need the feature that badly though. It's just something that's interested me about Unraid (and most consumer NAS devices actually), especially since I'm running 4 2TB drives and 2 3TB drives as a single RAIDZ2 vdev in my home TrueNAS setup.
> I hadn't even considered that it would be possible to assign partitions to the vdevs instead of the entirety of the drive.
You can even use files[1]! Mostly available for testing though.
It would be nice with a bit more flexibility with regards to the underlying storage, hence this concept. I tested it in a VM and it works fine, just slow when the underlying disks gets trashed. Mostly noticeable on heavy writes and scrubs.
Should work fine on SSDs though, and might possibly even be a boon to very fast NVMe drives (increasing concurrency).
There are a lot of comments here about how the author should've used ZFS, or how well now they know about ZFS, but the post addresses this:
> The other big elephant in the room, of course, is “why weren’t you using ZFS in the first place”. While it is true that Alpine has supported ZFS for years, I’ve been hesitant to use it due to the CDDL licensing. In other words, I chose the mantra instilled in me about GPL compatibility since the days when I was using GNU/Linux over pragmatism.
It's not as if the author wasn't aware of ZFS, but had issues with the license.
TLDR: They got an error from the RAID1 array, but they re-added the broken drive to the array.
Also, they did’t have backups.
Then, they did fsck -y on the filesystems on the array incorporating the broken drive. What they probably should have done first is remove the broken drive from the array again.
One part I noticed: The backup strategy. If ariadne fucks up this bad, mere mortals have no chance.
I understand her very well, btw, as it took a minor data loss to teach me it will happen to all of us. I just got lucky.
But consider the situation as an end user: zillions of options, but no clear strategy. Simply choosing what to back up and how many times is expert level IT. Choosing a software package and storage location is headache inducing for expert level IT. Validating long term all the effort will bear fruit on the worst day of your computers life brings enterprises to their knees.
Non-tech end users are expected to backup everything to the Cloud. Not just backup but even store working copies. In fact non-technical users can, at most, plug in an external HDD/SSD and backup their files to it, but unless they take pains to keep a 2nd copy (and both copies in sync), it is no better than having your files on the system drive. Anything beyond this is too technical at the moment for laymen to use. I don't know why that is so. Many programs are more difficult/technical to use than they have to be. It's like some kind of unconscious gatekeeping.
I'd rock ESXI as hypervisor, and Truenas as the Storage environment.
Passthrough the SSD's and any Disk controller to TrueNas. I always recommend HDD Array to backup your primary SSD array. This could have all been avoided with ZFS, along with future worry of bitrot.
Then you can run your Primary Linux VM for Docker, and another other standalone Linux VM's you may need. You'll want to keep Truenas on a datastore connected to ESXI, but the rest of the storage back be kept on Truenas and fed back into esxi.
Not only keeps your system portable for future upgrade paths, but keeps your system reliable.
55 comments
[ 3.3 ms ] story [ 114 ms ] threadBtrfs raid1, on the other hand, uses the checksums of the data stored in its metadata trees to validate which copy of the data is correct and repair the corrupted copy. 3-disk makes this even more robust, as you get an exponential growth of data-metadata pairs to pull from. If any one pair matches, that data can be assumed to be correct.
Modern drives have rather large per physical sector checksums to detect the occurrence of these errors and presumably retry a couple times before returning an I/O error if configured correctly.
In addition a good I/O bus should have error detection checksums for data in transit, and good hardware should have ECC RAM as well.
Sadly, a large number of SSDs (and nearly all cheap USB thumb drives / SD cards / eMMC devices) will happily return silently corrupted data.
And the drive would return bad data instead of just erroring out - this is sometimes a difference between “home use” drives vs enterprise - enterprise assume you’re in a multi redundant raid setup and instead of retrying will just fail out fast and let the raid take care of it. Home drives will continue retrying and perhaps even eventually give you what they got - the idea being you’d rather have one byte bad of a word doc than lose a 4K chunk of it.
ZFS catches these tricks because the checksums will fail. Multi-redundancy RAID can do it - but since most are enterprise they are built with the assumption the drive will return good data - or none.
This can be the worst scenario as corruption can be silent and infect all backups after a time.
> ZFS catches these tricks because the checksums will fail.
Yea, I think this is the right explanation. I guess it simply doesn’t keep any consistency information that it can check, whereas ZFS keeps a checksum of every block. Having used ZFS, it is difficult for me to understand how anyone could trust anything else. I don’t think that there is anything a ZFS user could do to accidentally destroy their data (aside from obvious things such as deleting the wrong filesystem or pool, which could happen in any case).
Now, I am more observant. Scrub once a month. Monitoring of SSD TBW, UDMA-CRC, Temperature, Watt, Fan-Speed in InfluxDB 2.0 with alerts set. Besides, ZFS mails all the important status info.
ZFS is really good for peace of mind, resilience and predictability, not necessarily for speed (but not exclusively, either), or your wallet.
Currently away, will update later.
Not sure if having one bad experience with mdraid is worth switching SSD vendors as well. You can mix and match different vendors, but they may have different performance characteristics as well.
Anyway, ZFS is a great choice either way. Definitely implement snapshots and a regular backup habit with plain zfs send/recv or syncoid.
I later found that both SSDs were from the same batch, which (I think) is why both failures happened so close together. Purchasing same-size SSDs from different vendors (with similar product lines) helps ensure you're using SSDs that should hopefully fail at different times.
Wrong. The advice is to use the same vendor/model but from a different production run (batch).
I've loved seeing the progress of ZoL. Whether I use Ubuntu's prebuilt modules or Debian's DKMS-based setup, it Just Works. But I'm still leery of running Linux with a ZFS root filesystem. I've compromised and use ext4 for root and put my home directory on ZFS.
If you want to use ZFS for root, I can heartily recommend NixOS. I've never had a problem with it there. Of course there are other downsides, but there are plenty of upsides as well.
The only thing exporting a pool does is declare "I am not being mounted by someone else". The FAULTED state would be because the search paths for the devices didn't exist. Clearing /etc/zfs/zpool.cache and doing `zpool import -f` would fix it entirely (ZFS identifies the disks by UUIDs in the headers AFAIK).
Using /dev/disk/by-uuid is always safe, though.
This is why one should not use sda/sdb or vda equivalent but use /dev/disk/by-uuid/
I've never deployed anything without at a minimum mixing batches, and usually vendors since.
We avoided data loss, but had a tense two weeks while cycling drives out as fast as we could (we had backups, but having to shut everything down to replace drives and restore would have been massively costly)
Turned out at least on the ones I had that it was always the outer edges of the platter that failed, so a (from 20 year old memory, may be wrong) 60Gb Death Star if carefully partitioned out to a maximum of about 58Gb ran absolutely beautifully and you could get them dirt cheap because (understandably if you weren't a crazy person like me) almost nobody wanted to touch them at that point.
Does a VM on one physical host, backed up via nightly disk image on a geographically separate physical host, with the latter storing the backups on a 2-disk mirrored ZFS pool, count as a reasonable 3-2-1 backup regime? It's not storing mission critical up-to-the-minute user data, 'just' company files, so a one-day rollback would be annoying but acceptable.
Add nightly or weekly offline backups and you should be good to go. And by offline, I mean physically connecting a storage device, making the backup (preferrably via an automated script), and then disconnecting the device and storing it elsewhere in a safe place.
In your scenario, I understand that the original physical host where you're making a backup of a VM has no redundancy or data integrity guarantees? I'd just like to point out that backups are good, but if all you have are backups of the same corrupted data, then they are pretty much useless.
My approach is to have redundancy and data integrity guarantees (ZFS mirror) at the source and have offline backups on external hard drives. The original data is likely to be OK, and well, if I need to recover from a backup, I'll be happy enough with what I have there anyway.
(taking a snapshot, mounting the snapshot, and then backing up from -that- via something 'cp -a' like is probably the safest option, IMO, though it's always possible I've missed something here in which case I hope somebody will reply correcting me)
It works now, it'll be fine.
Sure a Docker or VM can be easy to rebuild from scratch. But a Database is not.
I learned years ago when a failed HDD took 2TB of data I will never get back.
For starters Oracle would need to relicense their work. Which is unlikely to happen any time soon. If they wanted to relicense it they would have done it by now.
All OpenZFS contributers would also need to relicense their code. If I recall correctly the OpenZFS codebase consists for close to 50% out of non-Oracle code. A lot of this code might be contributed by companies where the people who did the work no longer work there. Meaning there is no-one in those companies to get legal to sign-off on it.
Short version: Don't hold your breath.
I would therefor say it is extremely unlikely any company (let alone Oracle) would ever sign-off on such a thing.
The Solaris forks (based on Illumos[1]) such as SmartOS[2] are pretty active!
> Are there other significant CDDL licensed codebases?
A lot of the Java stuff was also CDDL licensed. Though some of that (larger projects such as OpenJDK) were already dual licensed with "GPLv2 + classpath exception".
I also doubt it was used much outside of Sun. But even it's just the Sun stuff I still think no corporation would ever dare to make such a license update.
[1] https://www.illumos.org/ [2] https://wiki.smartos.org/
If I have an amd CPU in my dev server that would also run with ECC.
But I also run zfs with snapshots.
The thing is, both perspectives are valid. It's true that you can sometimes do really complicated things yourself and it can work out fine. And you can also get burned really easily. So the reader needs to figure out for themselves what the risk is and if it's worth it.
But there are things we should all do that are just never easy to do right, like.... backups. There's just no cheap and easy and good way. If you want real off-site backups, you're going to have to either pay a small fortune for a managed service (either in subscription fees or network egress fees or both), or physically carry a bunch of drives or tapes somewhere else weekly/monthly. And of course you also have to restore regularly or the whole thing could be a waste. It's just a pain.
Of course, off-site backups are good, but would not have been actually needed in this case.
I'm no greybeard, but this one IS important.
Difficulty aside, I'm wondering if it would be possible for ZFS to implement that as an option without giving up other features.
Then you can upgrade by putting the partitions on different drives etc. Of course must then be careful not to put two partitions of the same vdev on the same disk.
The major downside is that ZFS isn't aware that the underlying devices are the same, so IO which looks nicely scheduled as far as ZFS is concerned might cause massive random IO on the underlying disks.
That said, if one could tell ZFS about which zfs-devices share physical hardware, so the IO scheduler would be aware, then that might be a viable strategy. Might be handy in case you have multiple iSCSIs sharing a 10G link as well.
I don't need the feature that badly though. It's just something that's interested me about Unraid (and most consumer NAS devices actually), especially since I'm running 4 2TB drives and 2 3TB drives as a single RAIDZ2 vdev in my home TrueNAS setup.
You can even use files[1]! Mostly available for testing though.
It would be nice with a bit more flexibility with regards to the underlying storage, hence this concept. I tested it in a VM and it works fine, just slow when the underlying disks gets trashed. Mostly noticeable on heavy writes and scrubs.
Should work fine on SSDs though, and might possibly even be a boon to very fast NVMe drives (increasing concurrency).
[1]: https://openzfs.github.io/openzfs-docs/man/7/zpoolconcepts.7...
> The other big elephant in the room, of course, is “why weren’t you using ZFS in the first place”. While it is true that Alpine has supported ZFS for years, I’ve been hesitant to use it due to the CDDL licensing. In other words, I chose the mantra instilled in me about GPL compatibility since the days when I was using GNU/Linux over pragmatism.
It's not as if the author wasn't aware of ZFS, but had issues with the license.
Also, they did’t have backups.
Then, they did fsck -y on the filesystems on the array incorporating the broken drive. What they probably should have done first is remove the broken drive from the array again.
I understand her very well, btw, as it took a minor data loss to teach me it will happen to all of us. I just got lucky.
But consider the situation as an end user: zillions of options, but no clear strategy. Simply choosing what to back up and how many times is expert level IT. Choosing a software package and storage location is headache inducing for expert level IT. Validating long term all the effort will bear fruit on the worst day of your computers life brings enterprises to their knees.
I'd rock ESXI as hypervisor, and Truenas as the Storage environment.
Passthrough the SSD's and any Disk controller to TrueNas. I always recommend HDD Array to backup your primary SSD array. This could have all been avoided with ZFS, along with future worry of bitrot.
Then you can run your Primary Linux VM for Docker, and another other standalone Linux VM's you may need. You'll want to keep Truenas on a datastore connected to ESXI, but the rest of the storage back be kept on Truenas and fed back into esxi.
Not only keeps your system portable for future upgrade paths, but keeps your system reliable.
Opensource is not always the best route to take.