At a conference, I once met somebody who said his team had the joy of dismantling broken SD cards and figuring out why they broke.
One of the anecdotes I distinctly recall was about a batch of cards that (for obvious cost saving reasons) used the actual data flash to store its own firmware as well.
IIRC due to a bug in the wear leveling accounting, once the card got sufficiently full, the wear leveling code ended up partially overwriting the firmware itself.
I recall once having an SD card with a mysterious file in the root folder with system and hidden attributes on it. I decided to delete the file and some time later (months), the card suddenly stopped working.
It maybe stopped working once the OS decided the bytes were available for use (which it would only have done without the file marking that space as unavailable) and overwrote that section. I suspect defragmenting or otherwise moving the file about for whatever reason would have had the same effect.
Wouldn't a regular filesystem format of the card have the same effect - wipe the file? Wouldn't such SD cards be fatally flawed for most applications? My camera, drone, etc. all format the cards.
I vaguely recall thinking the file was just some sort of serial number or stock tracking metadata when I deleted it. It might have instead failed for quite mundane reasons such as low quality manufacture. I don't know.
Exposing the firmware in this fashion would be a bad idea for all of these reasons. It is a good thing that SD card manufacturers don't do this any more, if they ever did.
That doesn't at all mean that the firmware can't be exposed through the filesystem. Is that a bad idea? Almost certainly. Is it possible? Absolutely yes.
Man, I'd love to work in that team for a year. I've had so many SD cards fail on me (most recently a 1TB SanDisk Extreme which was used in my Steam Deck) and it feels like most of them are junk.
IIRC the denser the memory on the cards, the more likely problems are to arise. I forget the specifics but it has to do with the physics involving electronics on such a small scale. That's why industrial grade cards tend to be in much smaller capacities, but they are much longer lasting.
I wish we'd move away from microSD cards to a larger format if it meant more durable storage. I hate these things. I just end up keeping them in an SDcard adapter when they're not in active use so I don't lose them.
USB sticks seem to have the same issue with durability though. I've also had multiple of those fail on me. I'm currently eyeing those SSD USB sticks and I wonder how much more durable they might be.
The regular SD card form factor seems great when it comes to more durable storage, but obviously would not be ideal for smaller devices.
I've heard that many USB sticks use lower quality binned chips so that may be why they usually fail more often despite their size, but this is just anecdotal from what I've heard.
I read elsewhere here that if you fdisk/gdisk a partition that is one third of the drive and limit yourself to that space, then the drive will exhibit the reliability of SLC media, and last far longer than rated.
Have an SD card that was copying file on to and in the middle of transfer the card went kaput and hasnt seen since. No machine I try it on even acknowledges it existence.
The card is almost a decade old at this point but is there a way to recover data ?
Best to consult a data recovery specialist, will cost you an arm and a leg though.
10 years without power makes me think most of it wont be recoverable even if they can get to the data on it, but then again it depends on a lot of factors when it comes to flash memory.
IIRC SD cards and most SSDs don't go through error check routine when powered, it has to be specified. Some do use the power-on time and run a routine if left on for too long.
You usually you have to read the entire drive to maintain, in my case S.M.A.R.T. reported changes to reserve blocks after doing that on an old drive.
I have a 16 year old 4GB SDcard with all data still intact.
Sometimes you can recover data over the SPI interface instead of the proprietary SD interface. Essentially, plug it into an arduino with one of the many SD card libraries and slowly dump the data over the serial link.
I'm unsure if this is just urban legend or not, though. I've never tried myself.
I am really disappointed by SD card reliability. It seems even a little wear and the whole thing slows down massively and eventually throws read/write errors.
I would like to see a card design which, instead of failing when there is flash wear, instead just gets smaller.
For compatibility with existing OS's, that would take the form of a self-partition-resizing sd card. It would understand fat32, ext3, NTFS, afs etc, and when the card is next powered up the partition would be slightly smaller and files physically at the 'end' would be moved inwards.
For newer OS's, a new API could be introduced which tells the OS 'this card is smaller now, please give me some blocks to mark as unavailable'.
The now-smaller card can use the removed-and-worn-out blocks to store error correction data for the remaining blocks. Ie. Additional error correction data on top of the data already stored within each page.
That effectively dramatically increases the lifespan of each page, at the cost of reduced IO performance.
Unfortunately because of the way the SD card - computer interface works, storing everything twice on the same card doesn't actually get you much more reliability.
The card is unaware of which bits of data are duplicates, and therefore can easily end up storing both copies of some data on areas of the physical flash that are weak - and this is even more likely considering they will probably be written around the same timestamp.
i've heard some cards do duplicate data detection? that way they can avoid storing multiple copies of the same data, so for example copying a large file is fast
Possible, but I haven't heard the same. Sometimes copying a large file feels fast because the OS has a huge cache, and it'll just take a long time in the background later when you unmount the disk.
mdraid? Create two partitions and build a raid on top of them. Or any of advanced filesystems with built-in raid capabilities like bcachefs, btrfs or zfs.
ZFS can do this. Just set copies=2 on a dataset, and it'll always keep 2 copies of everything written thenceforth (with ~twice the storage usage).
And since it's a dataset setting instead of a pool setting, it can be applied selectively on a storage device.
"Oh, those photos are also backed up to Google Photos. It's maybe not ideal, but it's a lot of data and not worth keeping 2 copies those locally, so I'll leave those at the default of copies=1. But the this collection of code and short stories and I've written? That's pretty important to me, and it's actually kind of small -- I'll set that as copies=2. Actually, maybe copies=3 would be even better..."
The setting will stick with it between machines/sessions/whatevers without further user input. Things with copies=2 get mounted the same way as anything else is with ZFS in a given environment.
ZFS already detects bitrot very well even with copies=1, but it can't fix bitrot with chechsums alone. With copies=2, it has a non-zero chance at actually fixing it (or at least being completely readable so the data can be transitioned to the next storage device, completely intact).
Is there even that much value to keeping multiple copies on the same device? I fail to see what this adds over backups which can be done with any filesystem.
If you have a pool consisting of a single device (such as a drive in a laptop) the second copy could protect from loss of a file that includes storage that can not be read.
In a multiple device pool (with some redundancy) it would IMO be a extra and unneeded layer of protection.
RAID is good (if one pays for it and makes it work).
Backups are better (if they actually get done, and if they are actually usable for recovery).
Backups + RAID is better yet.
But OP asked for a particular set of things that do not have any direct relationship to the traditional concepts of either backups or RAID, and ZFS can provide that set of things using copies=2.
It isn't for us to decide the value of this for them.
The fact that they're used in dash cams, exposed to high temperature thermal cycles, and recording tens of terabytes of data which can be more than the typical consumer HDD/SSD in it's lifetime is quite impressive.
Recently I found a long lost go-pro style camera using metal detector, after it had fallen off an RC plane, which crashed into a tree in dense forest. Over nearly a decade the camera had been buried several centimeters into the ground, and the card inside the camera wasn't particularly well protected (just an open slot, not waterproof or anything), which made it exposed to lots of water and freezing winters for nearly a decade. After some cleaning the SD card worked perfectly, and all the data was intact. It even had the crash recorded in it. I was impressed by that. The camera didn't survive though.
I recommend everyone take a picture of their contact information with their digital cameras, that way if it gets lost and found, people know who to at least send the pictures to if the camera is destroyed.
I imagine the RC plane's owner would get a real kick if years later the camera was found and they got the footage back.
(I used to work at Philmont Scout Ranch and would do a lot of detective work to get people their cameras back.)
SD Cards are just so damn cheap. Less than $20 USD for a 128GB microsd is unbelievable (cost and size). Only the most premium of the premium have a chance of surviving significant random or small writes, it seems. "Ultra" isn't quite enough, we need "Extreme Pro"? (Needing to figure out the ranking of superlatives is silly.) There's "Max Endurance" or "Pro Endurance" now, sounds promising ... but those are barely any more expensive, so I'm not sure how much different they could be ...
You can get Swissbit pSLC (pseudo-SLC) SD cards from component distributers like Mouser and Digikey, and they are much more expensive, but should be "industrial grade". I've recently purchased a couple of their 8GB microsd cards for about $30 each (but haven't put them through a lot of real usage yet ...)
> For example, lets consider that we wrote to the sector 1024 on the SDCARD, and this address got initially mapped to physical sector 1024 itself on flash. Later we overwrite the contents, but this time write cannot go to the same physical address because it needs to be erased before programmed again.
Maybe someone here will know the answer to something I've been wondering for a while: When appending to a file with writes smaller than a sector, does this remapping process trigger every time? Or is the controller smart enough that it can append without remapping, even though to overwrite would need remapping?
Obviously for an SD card used for saving photos or videos you could write a sector at a time - but for an SD card in a temperature logger or similar, your appends might be a lot smaller.
You cannot do single byte writes on NAND. You have erase blocks (typically several 100k to M range) that are divided into pages (typically a few to several dozen k), some devices have sub-pages but that's about it, that's the smallest unit of data you can write.
If you attempt to clear bits from 1 to 0 on an already written page, it will generally not work.
I say generally, because I have actually tried this using raw NAND flash on an embedded Linux device. I found a chip, where it did work, but not very reliably.
I also had an MLC chip, where it did not work, but instead caused random bit-flips on seemingly unrelated pages. I used this trick to systematically hammer down on the pages and figure out the pairings (only document in the NDA version of the datasheet), but failed to reproduce this on a different, more expensive MLC NAND.
But even if appending in this fashion did somehow work, you would still need to update a data structure somewhere else anyway, telling you the exact amount of data written (including cases where you actually append 0xff bytes).
thank you so much! this is something i've been wondering about for years, and figuring it out obviously took you a huge amount of work
normally with raw nand you use ecc, right? was there any ecc happening in your tests, either on the nand chip or in the linux driver? it seems like if you started with a page with valid ecc data and then tried to bash an arbitrary payload 1 bit in it to 0, you'd have a real challenge updating the ecc data to be consistent with the updated payload
yes, reasonably priced ones have an internal ECC engine, very cheap ones don't. The Linux NAND framwork has software ECC engines for those (see also: previous discussion on HN[1]).
> was there any ecc happening in your tests?
I deliberately turned it off. Linux has an ioctl for that. If the chip supports it, you can read/write without ECC or even directly into the OOB area where ECC data would normally be stored. The problem with the more expensive MLC NAND was that it had some smarts built in. It could discover overwrite attempts, as well as certain write patterns and apply a scrambling mask to the bits, so that the stored bits would (hopefully) not diffuse over to their neighbors.
> if you started with a page with valid ecc data and then tried to bash an arbitrary payload 1 bit in it to 0, you'd have a real challenge updating the ecc data to be consistent with the updated payload
If you are implementing a raw flash driver, there is a "nandbiterrs" test that does roughly what you describe: it bashes an increasing number of bit errors into a page and reads it back with ECC re-enabled, checking how many errors are successfully corrected during read, before the ECC engine gives up.
unfortunately the daunting task in front of me is to do this without linux or gcc, because what i want is a self-hosted operating system on a microcontroller that happens not to have an mmu or enough ram for gcc. my experience with sd cards in the past has been depressingly terrible reliability so i bought some slc nand chips with a public datasheet and without built-in ecc logic
>> Couldn't you perform a file append without the need for a block erase?
> You cannot do single byte writes on NAND.
This isn't true - that the OP wrote here certainly does work (at least for many places I've tested and used it). I've used it to do all sorts of low level logging in small devices (and I've thoroughly tested many, many chips to complete destruction to see how things degrade). I have not yet seen a place it fails and I've done it on dozens or projects. I guess there may be places wear leveling fights with this, but if you dig into the particular device you have you can usually find low level details to make it work. The places I've pushed it are all commercial projects for which we characterize long term failure by running the flash parts fast enough to destroy the flash, to ensure that once in the field the expected lifetimes meet any guarantees we need to provide.
Also, if you've never ran a flash chip to death, try it - it's interesting. Get a small SD card and hammer it withRead/write cycles until you start getting errors, then log and watch how those errors propagate. Good stuff :)
Every chip I've tested does indeed let you erase a block (which sets all bits to 0 or 1 depending on chip), then append bits to that block by changing the bits you need.
To append, you don't need to update some other data structure, just append a single bit of the correct type, then a bit scan of the page tells you where the data ends. Or, if you have another page telling how many "slots" of yourData are used, again append s single bit. Then small appends only cost a bit.
On top of this implement wear leveling (unless the chip does - most bigger ones do, but some low level devices don't).
Other projects I've worked on professionally are products for desoldering flash from captured and destroyed enemy devices which are then read raw and reconstructed, fault tolerant and error shielding for NASA space ops with flash, reverse engineering proprietary flash protocols for Secret Service (weird, right? but they pay) uses. I somewhat often get called for consulting on precisely these types of issues. So I do know a bit about low level flash abuse :)
I was talking about bare NAND flash, not eMMC or anything otherwise "fancy" with builtin FTL. What you say is certainly true for those.
None of the projects you describe sound "cheap" or like they involve devices first-and-foremost optimized for cost, and are unlikely to involve such dirt-cheap, raw NAND devices that are becoming increasingly rare even in the super-cheap embedded space.
The original context of the discussion (see: a few comments up) is on-die NAND flash, inside SD cards, which TFA is about.
The flash chips I brought up for comparison were from an extremely cost-optimized, embedded device that I worked on ~2016. The more expensive one already being on the upper end, regarding smarts & cost, but still exposing the flash directly (plus ECC & bad block management, but no FTL).
I mostly work with industrial devices, and also maintain the Linux user space for raw flash (mtd-utils). I have seen my fair share of broken flash devices over the years (often in-field) and also have a my own little pile of broken industrial SD cards on my desk :)
> None of the projects you describe sound "cheap" or like they involve devices first-and-foremost optimized for cost
A lot of them are low cost PIC microcontrollers, which are super cheap, and the lowest cost MCUs were chosen precisely because at large commercial sale scale saving pennies on HW is very important. The reason we went to such lengths to reuse bits at the lowest level is because of needing the lowest cost devices on those projects.
Some were higher end, but I think it's more useful on low end chips.
ahhh, I think there's the source of our confusion: I suspect you are actually talking about NOR flash?
NOR flash is physically organized different from NAND. It typically does allow single word/byte or sometimes even bit level accesses, like you described. NOR flash is also much more reliable, often not needing any wear leveling at all. The downside is that NOR flash has orders of magnitudes lower data density.
Microcontrollers would use NOR flash internally for program memory, precisely because it allows direct, single-cycle access to individual opcodes. With NAND flash, program memory would have to be read page-wise and require some sort of instruction cache in between. Other EEPROM-like devices (e.g. AT28Fxxx) also use NOR flash for pretty much the same reasons.
On some embedded Linux devices, NOR flash is used for config/boot loader/kernel, while the rootfs lives on NAND.
> ahhh, I think there's the source of our confusion: I suspect you are actually talking about NOR flash?
Nope. PIC has on device NOR (which I've tested), and often has external NAND flash (depending on design choices), such as using SDCard interfaces.
Here's [1] a Micron tech doc on their NAND chips stating quite clearly they allow exactly what I described doing above: "A NAND Flash block is 128KB. Erasing a block sets all bits to 1 (and all bytes to FFh). Programming is necessary to change erased bits from 1 to 0. The smallest entity that can
be programmed is a byte."
There's plenty of manufacturers and chip types that guarantee one can do this. It is tech notes like that led me to do some testing to see if devices fall into spec on R/W cycles before death when evaluating which chips to ship in consumer products that need this.
AFAIK, it's not clear how much difference in wear a series of writes cause when compared to writes and erases.
I do know that very old flash firmwares applied that idea, and that new flash evolved in a direction where the wear caused by an erase is closer to the one caused by a write. If modern firmwares do not use it, it's probably because there isn't a lot to gain.
Probably not for a variety of reasons. The SSD itself does not know about files or appends; only block reads and writes. If the OS fills the unused space with zeros then it's going to write a block with trailing zeros, not ones. Most SSDs also scramble or encrypt data before writing to flash.
That could explain why my dashcam is working fine after ten years whereas my previous job's IoT devices regularly ate SD cards. (Then again, the IoT devices were also overheating badly whereas the dashcam was designed for its purpose, kept cool while running, and runs on a low duty cycle)
What are you supposed to do, then? Buffer up writes and accept that you'll lose logs a minute or so before a power outage?
As much hell as we had with those SD cards I'd rather put tiny iPod spinners in them...
Losing the logs on a power outage may be a good compromise. Alternatively, you can buffer on a battery-powered RAM unit, accept that your cards will be short-lived, immediately send the logs through some network, not log...
When you are designing the hardware, you have lots and lots of choices.
Yeah. This company wasn't big enough to entertain battery backups, and the product was not expensive enough to add hardware that didn't bring in more sales.
And we logged too much to pull all the logs over WAN all the time, resulting in a clever hack for remotely browsing logs with (relative) security
There are industrial microSD's with pSLC memory, e.g. Kingston's SDCIT2 line. Those are significantly more durable - in a few years of using them I haven't managed to kill a single one. Much pricier per-GB, though.
I've seen some flash drives that prepare a pool of free AU blocks which is common with Phison controllers
They prepare and erase a new pool only when it's full causing bursts in speeds and long periods of unresponsiveness (10 sec) while writing, reading is blocked as well. You can see it as a zigzag in file transfer speed graphs. This is not to be confused with OS buffer flushing.
While I couldn't find any official info for the reasons behind it, it's assumed that the erase speed is slow on low quality flash and hence the controller has to pool in order to erase pages in parallel, increasing speed when copying small files between computers on plug and unplug.
In my case it was about 500MB before before freezing on a 128GB flash drive.
It sort of depends, but in general flash cannot write a page without erasing it first. Sometimes you can set, but not un-set bits (can be a 0 or 1 depending, usually 0 is 'set') without erasing a page.
At the lowest level this means to update a page you have to first read it all back out, apply your modifications, then write back in a single operation. You must always write the full page size.
In high level contexts like SD cards or SSDs being driven by an operating system, all of this is handled transparently in the background. If you're using raw flash in an embedded context, you usually have to manage this yourself (or let a library do it for you).
Some A2 SD cards have command queueing for random read speed improvement and TRIM enabled. It requires kernel support and hardware acceleration being optional.
Even more strangely, proprietary SxS/XQD cards also don't support discard/trim, neither for devices nor from a PC. Only the more recent CFX cards support trim (most of them, but not with every reader, and formatting in a camera doesn't seem to issue them, perhaps to aid with data recovery in case of accidental formats).
Are CFExpress cards similar/comparable at all to SD cards? Afaik CFExpress uses nvme, which will provide more features.
Also, interesting thing one can do if you have a need for a CFExpress card (like a camera) and want to save some money. You can buy an adapter online such as the linked one, and then buy an m2.2230 ssd and put it in there. That’s how I made myself a cheaper 1TB CFExpress B card for my camera, and it’s doing just fine, even with 8k30fps at ~3.5Gb/s.
Does anyone have access to the full specs of some SD cards? To get access to the full specifications it seems like you're forced to join the SD Association: 2500$ p/year
The SD Association is a non-profit organization, but that doesn't mean that they don't have expenses to pay and mouths to feed.
It's $6.84 per day for a membership.
If that's an outrageous amount to you, then: Please remember that you are absolutely free to direct your rage into creating your own competing portable flash storage systems, and to license them in any way that you wish.
You may even include a free pony, if you choose. It's your world.
71 comments
[ 2.6 ms ] story [ 82.8 ms ] threadOne of the anecdotes I distinctly recall was about a batch of cards that (for obvious cost saving reasons) used the actual data flash to store its own firmware as well.
IIRC due to a bug in the wear leveling accounting, once the card got sufficiently full, the wear leveling code ended up partially overwriting the firmware itself.
It maybe stopped working once the OS decided the bytes were available for use (which it would only have done without the file marking that space as unavailable) and overwrote that section. I suspect defragmenting or otherwise moving the file about for whatever reason would have had the same effect.
Exposing the firmware in this fashion would be a bad idea for all of these reasons. It is a good thing that SD card manufacturers don't do this any more, if they ever did.
exfatprogs can pack bitmap with table for SDHC and lower: https://manpages.debian.org/experimental/exfatprogs/mkfs.exf...
USB sticks seem to have the same issue with durability though. I've also had multiple of those fail on me. I'm currently eyeing those SSD USB sticks and I wonder how much more durable they might be.
I've heard that many USB sticks use lower quality binned chips so that may be why they usually fail more often despite their size, but this is just anecdotal from what I've heard.
The card is almost a decade old at this point but is there a way to recover data ?
10 years without power makes me think most of it wont be recoverable even if they can get to the data on it, but then again it depends on a lot of factors when it comes to flash memory.
You usually you have to read the entire drive to maintain, in my case S.M.A.R.T. reported changes to reserve blocks after doing that on an old drive.
I have a 16 year old 4GB SDcard with all data still intact.
I'm unsure if this is just urban legend or not, though. I've never tried myself.
I would like to see a card design which, instead of failing when there is flash wear, instead just gets smaller.
For compatibility with existing OS's, that would take the form of a self-partition-resizing sd card. It would understand fat32, ext3, NTFS, afs etc, and when the card is next powered up the partition would be slightly smaller and files physically at the 'end' would be moved inwards.
For newer OS's, a new API could be introduced which tells the OS 'this card is smaller now, please give me some blocks to mark as unavailable'.
The now-smaller card can use the removed-and-worn-out blocks to store error correction data for the remaining blocks. Ie. Additional error correction data on top of the data already stored within each page.
That effectively dramatically increases the lifespan of each page, at the cost of reduced IO performance.
So that for instance a 64 gigabyte would have a filesystem with 32 gigabyte space but with a lot of redundancy?
I know there are many ways to achieve similar outcomes but all I know of relies on knowing beforehand how to mount the thing.
The card is unaware of which bits of data are duplicates, and therefore can easily end up storing both copies of some data on areas of the physical flash that are weak - and this is even more likely considering they will probably be written around the same timestamp.
And since it's a dataset setting instead of a pool setting, it can be applied selectively on a storage device.
"Oh, those photos are also backed up to Google Photos. It's maybe not ideal, but it's a lot of data and not worth keeping 2 copies those locally, so I'll leave those at the default of copies=1. But the this collection of code and short stories and I've written? That's pretty important to me, and it's actually kind of small -- I'll set that as copies=2. Actually, maybe copies=3 would be even better..."
The setting will stick with it between machines/sessions/whatevers without further user input. Things with copies=2 get mounted the same way as anything else is with ZFS in a given environment.
ZFS already detects bitrot very well even with copies=1, but it can't fix bitrot with chechsums alone. With copies=2, it has a non-zero chance at actually fixing it (or at least being completely readable so the data can be transitioned to the next storage device, completely intact).
https://docs.oracle.com/cd/E19253-01/819-5461/gevpg/index.ht...
In a multiple device pool (with some redundancy) it would IMO be a extra and unneeded layer of protection.
Backups are better (if they actually get done, and if they are actually usable for recovery).
Backups + RAID is better yet.
But OP asked for a particular set of things that do not have any direct relationship to the traditional concepts of either backups or RAID, and ZFS can provide that set of things using copies=2.
It isn't for us to decide the value of this for them.
The fact that they're used in dash cams, exposed to high temperature thermal cycles, and recording tens of terabytes of data which can be more than the typical consumer HDD/SSD in it's lifetime is quite impressive.
I imagine the RC plane's owner would get a real kick if years later the camera was found and they got the footage back.
(I used to work at Philmont Scout Ranch and would do a lot of detective work to get people their cameras back.)
You can get Swissbit pSLC (pseudo-SLC) SD cards from component distributers like Mouser and Digikey, and they are much more expensive, but should be "industrial grade". I've recently purchased a couple of their 8GB microsd cards for about $30 each (but haven't put them through a lot of real usage yet ...)
Develop it once, and you can make as many SD cards as you like reliable for no extra money...
Maybe someone here will know the answer to something I've been wondering for a while: When appending to a file with writes smaller than a sector, does this remapping process trigger every time? Or is the controller smart enough that it can append without remapping, even though to overwrite would need remapping?
Obviously for an SD card used for saving photos or videos you could write a sector at a time - but for an SD card in a temperature logger or similar, your appends might be a lot smaller.
Directly logging to an SD card (i.e. lots of small writes) is the fastest way to destroy it.
Couldn't you perform a file append without the need for a block erase? By keeping the unwritten area as all ones until written?
If you attempt to clear bits from 1 to 0 on an already written page, it will generally not work.
I say generally, because I have actually tried this using raw NAND flash on an embedded Linux device. I found a chip, where it did work, but not very reliably.
I also had an MLC chip, where it did not work, but instead caused random bit-flips on seemingly unrelated pages. I used this trick to systematically hammer down on the pages and figure out the pairings (only document in the NDA version of the datasheet), but failed to reproduce this on a different, more expensive MLC NAND.
But even if appending in this fashion did somehow work, you would still need to update a data structure somewhere else anyway, telling you the exact amount of data written (including cases where you actually append 0xff bytes).
normally with raw nand you use ecc, right? was there any ecc happening in your tests, either on the nand chip or in the linux driver? it seems like if you started with a page with valid ecc data and then tried to bash an arbitrary payload 1 bit in it to 0, you'd have a real challenge updating the ecc data to be consistent with the updated payload
yes, reasonably priced ones have an internal ECC engine, very cheap ones don't. The Linux NAND framwork has software ECC engines for those (see also: previous discussion on HN[1]).
> was there any ecc happening in your tests?
I deliberately turned it off. Linux has an ioctl for that. If the chip supports it, you can read/write without ECC or even directly into the OOB area where ECC data would normally be stored. The problem with the more expensive MLC NAND was that it had some smarts built in. It could discover overwrite attempts, as well as certain write patterns and apply a scrambling mask to the bits, so that the stored bits would (hopefully) not diffuse over to their neighbors.
> if you started with a page with valid ecc data and then tried to bash an arbitrary payload 1 bit in it to 0, you'd have a real challenge updating the ecc data to be consistent with the updated payload
If you are implementing a raw flash driver, there is a "nandbiterrs" test that does roughly what you describe: it bashes an increasing number of bit errors into a page and reads it back with ECC re-enabled, checking how many errors are successfully corrected during read, before the ECC engine gives up.
[1] https://news.ycombinator.com/item?id=38361139
unfortunately the daunting task in front of me is to do this without linux or gcc, because what i want is a self-hosted operating system on a microcontroller that happens not to have an mmu or enough ram for gcc. my experience with sd cards in the past has been depressingly terrible reliability so i bought some slc nand chips with a public datasheet and without built-in ecc logic
> You cannot do single byte writes on NAND.
This isn't true - that the OP wrote here certainly does work (at least for many places I've tested and used it). I've used it to do all sorts of low level logging in small devices (and I've thoroughly tested many, many chips to complete destruction to see how things degrade). I have not yet seen a place it fails and I've done it on dozens or projects. I guess there may be places wear leveling fights with this, but if you dig into the particular device you have you can usually find low level details to make it work. The places I've pushed it are all commercial projects for which we characterize long term failure by running the flash parts fast enough to destroy the flash, to ensure that once in the field the expected lifetimes meet any guarantees we need to provide.
Also, if you've never ran a flash chip to death, try it - it's interesting. Get a small SD card and hammer it withRead/write cycles until you start getting errors, then log and watch how those errors propagate. Good stuff :)
Every chip I've tested does indeed let you erase a block (which sets all bits to 0 or 1 depending on chip), then append bits to that block by changing the bits you need.
To append, you don't need to update some other data structure, just append a single bit of the correct type, then a bit scan of the page tells you where the data ends. Or, if you have another page telling how many "slots" of yourData are used, again append s single bit. Then small appends only cost a bit.
On top of this implement wear leveling (unless the chip does - most bigger ones do, but some low level devices don't).
Other projects I've worked on professionally are products for desoldering flash from captured and destroyed enemy devices which are then read raw and reconstructed, fault tolerant and error shielding for NASA space ops with flash, reverse engineering proprietary flash protocols for Secret Service (weird, right? but they pay) uses. I somewhat often get called for consulting on precisely these types of issues. So I do know a bit about low level flash abuse :)
None of the projects you describe sound "cheap" or like they involve devices first-and-foremost optimized for cost, and are unlikely to involve such dirt-cheap, raw NAND devices that are becoming increasingly rare even in the super-cheap embedded space.
The original context of the discussion (see: a few comments up) is on-die NAND flash, inside SD cards, which TFA is about.
The flash chips I brought up for comparison were from an extremely cost-optimized, embedded device that I worked on ~2016. The more expensive one already being on the upper end, regarding smarts & cost, but still exposing the flash directly (plus ECC & bad block management, but no FTL).
I mostly work with industrial devices, and also maintain the Linux user space for raw flash (mtd-utils). I have seen my fair share of broken flash devices over the years (often in-field) and also have a my own little pile of broken industrial SD cards on my desk :)
Same here.
> None of the projects you describe sound "cheap" or like they involve devices first-and-foremost optimized for cost
A lot of them are low cost PIC microcontrollers, which are super cheap, and the lowest cost MCUs were chosen precisely because at large commercial sale scale saving pennies on HW is very important. The reason we went to such lengths to reuse bits at the lowest level is because of needing the lowest cost devices on those projects.
Some were higher end, but I think it's more useful on low end chips.
ahhh, I think there's the source of our confusion: I suspect you are actually talking about NOR flash?
NOR flash is physically organized different from NAND. It typically does allow single word/byte or sometimes even bit level accesses, like you described. NOR flash is also much more reliable, often not needing any wear leveling at all. The downside is that NOR flash has orders of magnitudes lower data density.
Microcontrollers would use NOR flash internally for program memory, precisely because it allows direct, single-cycle access to individual opcodes. With NAND flash, program memory would have to be read page-wise and require some sort of instruction cache in between. Other EEPROM-like devices (e.g. AT28Fxxx) also use NOR flash for pretty much the same reasons.
On some embedded Linux devices, NOR flash is used for config/boot loader/kernel, while the rootfs lives on NAND.
Nope. PIC has on device NOR (which I've tested), and often has external NAND flash (depending on design choices), such as using SDCard interfaces.
Here's [1] a Micron tech doc on their NAND chips stating quite clearly they allow exactly what I described doing above: "A NAND Flash block is 128KB. Erasing a block sets all bits to 1 (and all bytes to FFh). Programming is necessary to change erased bits from 1 to 0. The smallest entity that can be programmed is a byte."
There's plenty of manufacturers and chip types that guarantee one can do this. It is tech notes like that led me to do some testing to see if devices fall into spec on R/W cycles before death when evaluating which chips to ship in consumer products that need this.
[1] https://user.eng.umd.edu/~blj/CS-590.26/micron-tn2919.pdf
I do know that very old flash firmwares applied that idea, and that new flash evolved in a direction where the wear caused by an erase is closer to the one caused by a write. If modern firmwares do not use it, it's probably because there isn't a lot to gain.
What are you supposed to do, then? Buffer up writes and accept that you'll lose logs a minute or so before a power outage?
As much hell as we had with those SD cards I'd rather put tiny iPod spinners in them...
When you are designing the hardware, you have lots and lots of choices.
And we logged too much to pull all the logs over WAN all the time, resulting in a clever hack for remotely browsing logs with (relative) security
Honestly we could have batched up the logs.
They prepare and erase a new pool only when it's full causing bursts in speeds and long periods of unresponsiveness (10 sec) while writing, reading is blocked as well. You can see it as a zigzag in file transfer speed graphs. This is not to be confused with OS buffer flushing.
While I couldn't find any official info for the reasons behind it, it's assumed that the erase speed is slow on low quality flash and hence the controller has to pool in order to erase pages in parallel, increasing speed when copying small files between computers on plug and unplug.
In my case it was about 500MB before before freezing on a 128GB flash drive.
At the lowest level this means to update a page you have to first read it all back out, apply your modifications, then write back in a single operation. You must always write the full page size.
In high level contexts like SD cards or SSDs being driven by an operating system, all of this is handled transparently in the background. If you're using raw flash in an embedded context, you usually have to manage this yourself (or let a library do it for you).
https://forums.raspberrypi.com/viewtopic.php?t=367459
Also, interesting thing one can do if you have a need for a CFExpress card (like a camera) and want to save some money. You can buy an adapter online such as the linked one, and then buy an m2.2230 ssd and put it in there. That’s how I made myself a cheaper 1TB CFExpress B card for my camera, and it’s doing just fine, even with 8k30fps at ~3.5Gb/s.
https://www.bhphotovideo.com/c/product/1685487-REG/monster_a...
It's $6.84 per day for a membership.
If that's an outrageous amount to you, then: Please remember that you are absolutely free to direct your rage into creating your own competing portable flash storage systems, and to license them in any way that you wish.
You may even include a free pony, if you choose. It's your world.