I mean, I love the insane GB/sec figures, but does anyone else mostly care about IOPS? These state 1.8M read and 1M write, which sounds quite impressive.
Related: anyone know of a good video or diagram for helping CS students get an intuition regarding the interplay of bandwidth and latency? Including how saturating bandwidth increases latency by causing queueing bottlenecks?
I'm looking for something a bit more visual and dynamic than the old "station wagon full of tapes going down the highway" imagery.
[EDIT: Just for clarification, I feel like I already have a pretty good grasp on these concepts. I'm looking for good ways to help others at the ~ undergrad level.]
Those graphs cut off the essentially vertical latency spike that results from enqueuing requests faster than the sustained rate at which the drive can serve them. For a different view in terms of queue depth rather than throughput, there are some relevant graphs from an older review that predates io_uring: https://www.anandtech.com/show/11930/intel-optane-ssd-dc-p48...
Generally speaking, latency starts increasing long before you've reached a drive's throughput limit. Some of this is inevitable, because you have a relatively small number of channels (eg. 8) and dies to access in parallel. Once you're up to the throughput range where you have dozens of requests in flight at a time, you'll have constant collisions where multiple requests want to read from the same plane/die/channel at once, and some of those requests have to be delayed. But that's mostly about contention and link utilization between the SSD controller and the NAND flash itself. The PCIe link is pretty good about handling transactions with consistently low latency even when on average it's mostly busy.
There was another post I saw recently comparing the increase in disk SIZE over the last 30 years vs the increase in disk SPEED vs LATENCY (so size in GB, speed in GB/s, latency in IOPS) - and how size increases far outstripped speed which outstripped latency, though all had improved.
IOPS helps, but for the average user, hundreds of thousands is already functionally infinite. What matters at this point is latency. Where you really feel that is Queue Depth 1. Where you read a file that points you to other files, that point you to other files, etc. That is the exact case where the computer is still making you wait.
This happens when you start your operating system, it starts services, you launch apps, etc. Driving that latency down is the biggest improvement you'll ever see past where we are today in terms of IOPS and throughput.
This is where the latest Optane actually shines. Optane doesn't win on IOPS or throughput, but where it shines is its crazy latency (delivered at relatively low power levels). Where latencies are 10% of that of even the highest end PCIe 4.0 NVMe SSDs. Do something like launch 20 applications at once, and it'll be done in a fraction of a time compared to even something like a Samsung 980 Pro because of latency being more around 10 μs instead of 100 μs.
PCIe 5.0 SSDs will cut latencies down to where Optane is today, but driving latency under 1 μs is where we'll get into a new level of crazy.
Technically, it would comparatively excel at launching 64K+ applications at once, but yes - flash is not quite amazing at launching 20 applications serially.
Related: Notice how the public cloud marketing material tends to focus on scalability over other metrics? That's because scaling horizontally for them is easy: They just plop down more "stamps" -- a set of clusters and controllers that is their unit of scale. Need 1,000 more servers in US East? Plop down 10 more stamps of 100 servers. Easy!
Except of course this involves an awful lot of networking, with long cable runs and many hops and virtualisation layers. The end result is that you can't get anywhere near the underlying storage latency.
Azure's Premium SSD has "write flush" latencies of about 4 milliseconds according to my measurements, which is easily 100x slower than what my laptop can do with a now very outdated Samsung NVMe SSD.
Cloud customers are also living in the land of millisecond latency for most things. Their customers are at the other end of a multi-ms round trip. Sharding reduces the need to push as many IOPS through a single database machine, leaving slack for higher storage latency.
For people who really need low latency in the cloud there is EC2 instance store or GCE local SSD.
It doesn't matter if the user running Google Chrome is 50ms away from the web server, if that server in turn does 10,000 round-trips to the database server. Suddenly, microseconds matter.
Why 10K round-trips you ask? Because of ORMs like Entity Framework. They have a habit of producing wildly inefficient query code, but hey... it saves on developer time!
I literally just optimised a large web site that actually had fairly decent SQL queries. I reduced the latency from 1,500 microseconds to about 50 microseconds and the performance of most pages improved by about 150-300 milliseconds!
Let me reiterate: A latency difference that can't even be benchmarked easily because it's so small made a very noticeable difference to page load times. The site feels snappier. Navigating it is now a joy instead of a drag.
Hey, thank you, you have just corrected a massive misconception I had about SSD performance.
My logic always worked like this: if IOPS is input / output per second, surely that means 10,000 IOPS of, say, the usual random 4kb file write at QD1, then that means I can get 10,000 of those in one second. So, a higher number must be better.
I had no idea that IOPS needs to take latency into account. After having done some reading I am still somewhat confused as to how this really works, but for future reference I will use latency as a measurement of performance again, as that is indeed what I thought IOPS was.
Is there a theoretical "practical" limit for how fast these things can physically get (in this particular form factor)
My understanding is that NVMe interface is pretty much as close as you can get to the CPU without being integrated. Is there a world where these things can operate as fast as RAM?
Flash chips themselves can't operate as fast as RAM. You can get as much bandwidth as RAM, but you can not get it as physically fast as RAM i.e. faster than 200ns
I actually was one of the people who did performance benchmarking of 3D XPoint before it came out. In app direct mode, you can maybe eek out 70% of the throughout and about 3x worse latency. Also, not all apps support app direct mode.
Many customers try to use 3D XPoint as another part of the cache hierarchy in-between regular ssd and ram. It's actually pretty neat for faas workloads which want containers to be "warm" rather than "hot" or "cold"...
To clarify for readers who aren't current on all the lingo: app direct mode refers to one of the modes for using the Optane persistent memory modules that connect to the CPU's DRAM controller. It doesn't apply to the Optane NVMe SSDs that use PCI Express and require the software overhead of a traditional IO stack. In a few years, something like CXL may allow for something close to the app direct mode to be usable on persistent memory devices with an SSD-like form factor, but we're not there yet.
Optane comes in two flavors. As NVMe storage and as NVDIMM. The former sits a bit below flash in terms of latency. The latter sits between a bit above DRAM in terms of latency and is byte-addressable.
NVMe is built on PCIe, so latency wise, they will be limited by the PCIe latency (roughly an order of magnitude slower than the memory bus though pcie 5 may be smaller than that) + media latency. Throughput wise, they are limited by the number of PCIe lanes that the controller supports, and the system it plugs into has been sized to allocate.
> My understanding is that NVMe interface is pretty much as close as you can get to the CPU without being integrated.
The best for consumer technology, yes. But future I/O protocols continue to improve and get better.
NVidia / IBM's collaboration on OpenCAPI (which is deployed in Summit as a CPU/GPU interface) has 300GBps I/O between the CPU and GPU, far faster than NVMe speeds (and even DDR4 RAM bandwidth).
And future chips may go even faster. I/O is probably one of the fastest growing aspects of a modern computer. PCIe 5.0, CXL, OpenCAPI, etc. etc. Lots and lots of new technology coming into play here.
There are even some products on making Flash-RAM work on the DDR4 interface. Non-volatile memory is what that's called. Intel's Optane works pretty well on that. Its not very mainstream but I hear reports that its impressive (slower than real RAM of course, but storage that has the bandwidth of RAM is still cool).
> Is there a world where these things can operate as fast as RAM?
Well... yesish. Flash is a kind of RAM (random access memory).
To answer your fundamental question though: No. Flash is fundamentally higher latency than DRAM (aka: DDR4). But with enough parallelism / big enough flash arrays (or DRAM arrays), you can continue to get higher and higher bandwidth.
At the top-of-the-line is SRAM, the RAM used to make L1 cache and registers inside of CPUs. This is very expensive and rarely used.
--------
Then you've got various non-mainstream RAMs: FRAM, MRAM, Optane, etc. etc.
Just put a battery on ram and you have an sdd as fast as ram, in principle.
The big thing about SSDs is that while sequential reads and writes have steadily improved, many workloads have not. From a cheap SSD to the best, there is very minimal difference in things like "how fast does my computer boot", "how fast does my game or game level load" and "how fast does visual studio load", or "how fast does gcc compile"
Is your second paragraph really true? These exact worlds are the reasons many have upgraded their ssd, the PS5 will only let you start new games from they built in fast hard drive for example.
The PS5s operating system and even how they package game files had to be developed from the ground up to realize these gains. There’s a good technical talk about it from one of Sony’s engineers.
They might still be using BSD stuff, but they developed new IO and compression offload functionality that doesn't match any off the shelf capabilities I'm aware of.
You can get most of the gains just by changing how your game is coded, in a way that will perform very badly if installed on a hard drive. The extra compression and better DMA are useful but not necessary. And I'm not sure what you mean about packaging. The change there is mostly just undoing optimization for slow drives.
The PS5 and Xbox Series X were designed to offer high-performance SSDs as the least common denominator that game developers could rely on, so that game devs could start doing things that aren't possible if you still have to allow for the possibility of running off a hard drive. That's still largely forward-looking; most games currently available still aren't designed to do that much IO—but the consoles will treat all new games as if they really rely on that expectation of high performance IO, and that means running them only from the NVMe storage.
Partially, on a PC. The difference, on a PC, between a high end NVMe SSD and a SATA SSD is minimal for most use cases; small enough that an average user won't perceive much difference. The workloads in question (booting, loading a program, compiling code, etc.) involve a lot of operations that aren't bound by IO performance (network exchanges, decompressing textures, etc.) and haven't been optimized to maximize the benefit of high performance storage so the throughput and IOPS difference of the storage device don't entirely dominate.
I second that. I have M.2 SSD on laptop and SATA SSD on desktop. There's no perceived difference on disk operations for me outside of corner cases like copying huge file. But there's very noticeable difference between prices.
+1, I installed an M.2 SSD in my desktop looking at the quoted 10x difference in speed but I was taken aback at the lack of any perceivable improvement. Money down the drain I guess.
That may be due to the fact that your software (like your OS) was built for a world of slow spinning disks, or maybe semi-slow SSDs at best. Not a lot of code is written with the assumption that disks are actually fast, and it's not too common to see people organizing sequential read/writes in their programs (except for dbs).
Do you have an NVMe M.2 device specifically? The M.2 form factor supports both NVMe and SATA (though any given device or slot might not support one or the other).
I've got a workstation that has both SATA 3 and M.2 NVMe SSDs installed.
The SATA 3 device can do sustained reads of about 550 MB/sec, fairly close to the 600 MB/sec line rate of SATA 3.
The NVMe device can do about 1.3 GB/sec, faster than physically possible for SATA 3.
As the platter density increased, the head could glide over more data at the same spindle speed and you would get increasingly higher sequential speed. But you would not get much better latency - physically moving the head to a different track could not be done significantly faster. With each new generation, you would get only marginally better speed for many workloads that users actually care about. The latest HDDs could saturate SATA bus with sequential reads, but would still dip into KB territory (not MB, let alone GB) for sufficiently random-access workloads.
SSDs are similar in a sense that they can be massively parallelized for the increased throughput, but the latency of an individual cell is much harder to improve. Benchmarks will saturate the I/O queue and reach astronomical numbers by reading many cells in parallel, but for most desktop users, the queue depth of 1 (and the individual cell latency) is probably more relevant. That's why a lowly SATA SSD will be only marginally slower than the newest NVMe SSD for booting an OS or loading a game.
Yeah, there's absolutely no reason game loading should be serial. It's straightforward for a game to figure out everything it needs and grab dozens in parallel.
That said, it's still true that you typically won't see a very big difference. Any SSD can support loading a level in a few seconds. The difference between normal flash and ultra-fast flash won't show up much unless you're aggressively streaming level data off the drive in real time.
> Yeah, there's absolutely no reason game loading should be serial. It's straightforward for a game to figure out everything it needs and grab dozens in parallel.
And that would likely tank the performance if the game is running from an HDD.
I suspect you could design the storage layer of your game/app/whatever to take into account the performance characteristics of the underlying drive, and adjust your I/O pattern to be more parallel or more serial accordingly.
I also suspect that would not be easy/economical to do within the time/budget constraints of most projects.
Loading on a hard drive means you want to arrange the data so that you can do bigger chunks. But by serial I mean start a load, wait for it to finish, start next load, etc, and you don't want to do that on any drive.
There is serial as in reading byte 1 2 3 4. This is what you're talking about. This is good on any drive and great on hard drives.
There is serial as in read a byte, do a calculation, read a byte, do a calculation. This is what I was talking about. This is bad on any drive and even more noticeable on SSDs.
If you can have all your data in one single 500MB read, that's great. But I wasn't talking about that, because any drive can do a single giant read with good performance.
If you have to do 800 different reads, it's much better to ask for them all at once. You don't want to trickle them out one at a time. This is what my comparison was.
> From a cheap SSD to the best, there is very minimal difference.
I've got a cheap SSD that will change your mind. It has amazingly poor performance though, I agree with the general concept that while quantitative differences can be measured, there's not a qualitative difference between competent SSDs.
> Just put a battery on ram and you have an sdd as fast as ram, in principle.
That reminds me of non-volatile SRAM modules you can get by Microchip. How it works is that its just normal SRAM, but each cell is connected to a matching flash cell and when the mdoules voltage drops below a threshold, it writes the SRAM contents to flash in parallel. When the voltage is restored, it does the reverse. The user-observable effect is that you have normal SRAM (speed and it doesn't wear out) but its non-volatile. In reality, its a clever trick, but since the flash is only written when the module is powered off (not per-write to the module), you get a lot more use before it wears out.
Of course these modules are tiny (a few KB), but I still thought it was pretty interesting!
1) is the actual NAND flash faster, or are we talking about going from "awesome" to "abysmal" as soon as you run out of DRAM/SLC caches or what have you? Which, given this kind of bandwidth, is going to be sooner rather than later.
2) this plus QLC cells, which, durability-wise, make TLC look good, makes me anticipate headlines like "This new PCIe 5.0 SSD ran out of its rated TBW in a week!"
These controllers are for enterprise drives where SLC caching is almost unheard of and all the performance specs are for sustained performance on a full drive. But the best performance may only be attainable on a drive with higher capacity than you can afford.
> Which, given this kind of bandwidth, is going to be sooner rather than later.
I don't see why — like NAND up until now, it keeps up by adding more separately-addressable chips to the board and striping writes across them. A 512GB SSD with this chip would hit the wall pretty soon, but they wouldn't waste this controller on a 512GB SSD. They'd use it for 16TB+ SSDs.
1 TB of QLC can't get anywhere close to 9 GB/s write speed. The best is currently about 40 MB/s for a 1 Tbit die, so 320 MB/s for 1 TByte. The slow write speed of QLC generally prevents you from burning out a drive in less than a few weeks.
~10W Controller. Doesn't matter on a Desktop or Server. But we sort of hit the limit what could be done on a Laptop.
Having said that it doesn't mention what node it was fabbed on. I assume there are more energy efficiency could be squeezed out.
<6us Latency. Doesn't say what percentile it is and under what sort of condition. But Marvell claim this is 30% better than previous Marvell SSD Controller.
I think ServeTheHome article [1] is probably better. ( Cant change it now >< )
We also have PCI-E 6.0 [1] finalised by the end of this year which we cant expect to be in 2023/2024. SSD Controller with 28GB/s.
I am also wondering if we are approaching the end of S-Cruve.
SSD bandwidth is not really approaching modern memory bandwidth - a computer with one of these will probably have 4 to 8 channels of DDR4 at 25GB/s each with DDR5 released about a year ago. That is 100GB/s to 200GB/s currently and more by the time PCIe 5 becomes a reality.
I am sure if you go back a few years you can find systems that have the same or less memory bandwidth than this has now, but they have both been moving forward enough that SSDs are still not close. That being said, between bandwidth and random access times, swapping out to disk backed virtual memory is very pragmatic and isn't the death of performance it used to be.
I think a fairer comparison would be 1 DDR channel to one SSD, or comparing the 8 channels to a multi-SSD striped RAID array. SSDs are ~7GB/sec, so I would say that SSD bandwidth is roughly within one or two orders of magnitude slower than RAM. I certainly wouldn't want to replace an app optimized to use all the RAM on a machine with one running on a lower-RAM SSD machine, though.
Maybe I'm old. I never used to think that an application that is able to use my whole ram is optimized. It evokes exactly the opposite impression, indeed.
I think I understand what you mean, but the gut reaction is that one.
Yeah, I am really old, after all.
Edit: let me explain. Something that is able to use the whole space of a flat memory model is way less sophisticated than something that is able to deal with a complex memory hierarchy. Our machines are indeed a complex pyramid of different subsystems with varying bandwidth and latency characteristics. A program that is able to embrace the inherent hierarchical nature of our machines (or multi-node systems) is way more "optimized", according to my sensibility.
I'm talking about situations like search, where you hold the entire index in ram. Total # of machines = size of index / indexserver ram. Usually the apps that run on these have, say, 96 cores and they're using about 80, and the idle time is mostly instructions waiting for memory fetches.
Typically that index fronts a disk repository which wouldn't fit in RAM, although over time, what fit in RAM, what fit in disk, what lived in RAM, what got cached in RAM, etc, have changed over time.
BTW, I'm probably of the same generation as you and the single most important lesson I ever learned for computing performance was "add more RAM"; in the days when I first started using Linux with 4MB of RAM, it wasn't enough to do X11, g++ and emacs all at the same time without swapping, so I spent my hard-earned money to max out the RAM, at which point it didn't swap and I could actually do software development quickly.
It really depends on the type of memory usage though. On current Intel multi-socket the bandwidth a single process can have for core meditated memory accesses, even for node local memory, is seriously disappointing. Often < 10GB/s. Yes, it scales reasonably nicely, but it's very painful to tune for that, given that client and low core count, single socket, SKUs end up with > 35GB/s. And that this affects you even from within VMs that are on a single socket.
I heard that ice lake sp improves a bit in the area, but haven't gotten access to one yet.
10W definitely matters on servers. You can easily have a dozen of these in 1U. 120W isn’t nothing to dissipate.
I wonder what node this controller is made on. If it’s made on TSMC N7 then they could cut power consumption roughly in half by going to N5P. The package size makes me wonder if it’s an even older node however.
It’s not an issue in servers really, look at how much power high end NIC consume heck 10GBE SFP+ modules can consumer 5W> each and you can easily have 48> of those in a switch...
But a switch doesn't include a hairdrier, er, GPU generating heat within the enclosure. Server cases have potentially multiple GPUs, CPUs, plus now these 10W controller chips
Some switches could do with much better cooling. A POE switch or one that is putting 10gb down a cat 6 cable is very toasty though. I’ve got one that is borderline too hot to touch. Thanks Ubiquiti.
Most switches don't allow 5W pull in every port, those levels are usually only found in 10G copper SFPs which can't reach full distance due to power requirements so typically pull max allowed (or higher) levels. Typical 10G SFP+ SR or Twinax will consume about a watt per module. The ASIC may be a couple hundred watts under load.
Servers typically have much higher power density unless you're talking 400G switches compared to low end servers.
It's absolutely an issue in servers. Within the past month, I have worked with two massive global banks that are backpedaling on adoption of NVMe at scale due to power draw concerns. They ran the numbers, and the performance benefits of NVMe in most use cases don't justify the costs of retrofitting their data centers to handle the increased power draw associated with NVMe (2-3x per disk compared to SAS SSD).
Granted, this is a somewhat self inflicted problem: the banks' data centers were built on the assumption that power-hungry storage platforms would always be isolated to a relatively small footprint, but they are adopting hyperconverged storage architecture as their new operating standard for the future. Newer data centers sized for the increased power draw associated with hyperconverged/converged infrastructure don't tend to have this problem.
Curious - what's the general rationale for hyperconverged storage at a bank across the entire footprint? I'd imagine that data storage systems would need to have highly restricted APIs and strong persistence requirements, so you'd naturally tier your services as you would between an application/service layer and a database layer, as would be common for web applications. Certainly the database layer would benefit from converged infra, but wouldn't it be overkill to provide that to nodes running application & service code that shouldn't be doing much, if any, persistence to local drives?
There's a few factors at play. The first is that the % of apps at a global bank that follow a sensible modern design pattern is often surprisingly small; these are gargantuan, risk-averse (and hence change-averse) organizations that are just now barely wrapping their head around the idea of modern application infrastructure for core business needs. So each bank tends to have absolutely, stupidly enormous legacy app footprints.
Next, there's the VDI factor - these banks are nearly entirely reliant on VDI for most employees. Most work cannot be done without connecting to a Citrix or Horizon session. As a random I/O-heavy workload, VDI benefits greatly from the hyperconverged infrastructure model in terms of cost efficacy.
Also, these banks never pay anywhere close to list price to license any of their software, so the incremental cost of spinning up a cluster to support apps with relatively meagre storage requirements is often close to negligible, which is a strong advantage over the operational costs associated with setting up dedicated storage appliances.
Makes a ton of sense, thanks! Though the idea that critical banking information is just chilling on cluster-local drives rather than in replicated and continuously backed-up database systems is... a wee bit unnerving!
Nothing unnerving about it; every hyperconverged storage platform out there uses some sort of RAIN type architecture with all data replicated across 2 or more mirrors within a cluster, or spread out in stripes using an erasure coding algorithm with parity bits. Real-time storage-layer replication to another cluster is an optional feature for extremely critical use a cases, but most orgs just rely on their backup and replication measures to handle availability concerns where a cluster ends up completely dead. Cases where hyperconverged storage is used without making allowances for availability are generally limited to purely ephemeral workloads (e.g. 'disposable' virtual desktops that re-compose on logout) or workloads with app-level availability through ring topology, sharding, etc.
> ~10W Controller. Doesn't matter on a Desktop or Server.
Of course it does. For one, that's heat you have to efficiently remove or face SSD throttling back on you or degrade over time. It does not make sense to buy expensive hardware if it is going to show impaired performance due to thermal throttling.
This makes the business of putting together your PC this much more complicated, because up until recently you only had to take care about CPU and GPU and everything else was mostly afterthought.
We are already facing motherboards with their own fans, now I suppose it is time for SSDs.
Weren't 10,000 or 15,000 RPM hard drives like 15W or 20W or something?
These 2U or 4U cases, or tower desktop cases, were designed to efficiently remove heat from storage devices, as well as the 3000W++ that the dual socket CPU and 8way GPUs will pull.
10W is tiny for desktop and server. Barely a factor in overall cooling plans.
They were, but 3,5" drives are large hunks of aluminium with many times the surface. Meaning that as long as you have some airflow around them and the air is not too hot they are fine.
Also the heat was mostly generated by motor and an actuator and not the controller.
High performance m.2 drives come with removable finned heatsinks for this reason. Without them they rely on throttling during sustained heavy workload. Dissipating 10 Watts in a desktop isn't the concern.
The heatsink doesn't really work, though, and is marketing for the most part [1].
I have a crucial P1 NVME SSD and I can make it overheat pretty reliably. Pretty much any synthetic workload makes it overload if the SSD is empty (it reaches 70° pretty quickly and even starts throttling until it reaches 80° and the whole system starts shuttering because of extreme throttling do it doesn't damage itself. Although I have not properly tested it, it seems that not using any heatsinks from my motherboard makes the temps actually better but it still overheats.
The main reason it can overheat quickly is probably because its sitting in a really bad position where it gets close to zero airflow despite being in an airflow focused case. Most motherboards place the nvme slot directly under the GPU. The main problem seems to be that the controller is overheating when it's writing at close to 2000 MB/s. It's also important to note that only the controller (an actual relatively powerful ARM processor), not the flash memory, seems to overheat.
Fortunately, this is mostly not an issue because it's a QLC drive and the workload is unrealistic in the real world. When writing to an empty drive at 2000MB/s (Queue depth 4, 128k sequential writes), it takes 2 minutes until the cache is full. The way its currently used, it takes 30 secs for the cache to become full and for write speeds to drop to 150MB/s. The only way it has every overheated in the real world was during the loading screen of a gameplay when it reached 78C quickly (and I only noticed it in the hardware monitor). If the GPU hadn't heated up the nvme drive before (it was sitting at 65C mostly idle), and starved it for air, I doubt it would have hit 60C.
So until motherboards start placing nvme where it can get some actual cooling, or they make actual functioning heatsinks, their power usage can make a difference.
The Crucial P1 is a budget SSD that doesn't come with a proper heatsink, similar in quality to that god awful "heat shield" in that linked review. When I say "High performance m.2 drives come with removable finned heatsinks" I mean an actual high performance drives that come with a finned heatsinks like https://www.amazon.com/Corsair-Force-MP600-Gen4-PCIe/dp/B07S... not examples of budget drives paired with flat pieces of metal.
Also your high performance SSD should be going in the direct-to-CPU slot to the right of the GPU, not under it.
I had a 10k RPM SCSI 3.5" HDD in my desktop 20 years ago. I don't remember the power draw, but it got hot enough to the touch to burn flesh. I ended up rigging a case fan to blow directly on it to keep it reasonably cool. Neither case nor motherboard were intended for that drive (mid-tower Dell home case of the era). I even had to solder the fan directly to the missing pins on the motherboard.
Would be cool to have a general liquid cooling solution for all components. We'd install a radiator outside our homes just like an air conditioning unit and then connect the computers to it.
Probably not a small node, remember flash gets more fragile the smaller the process.
SSD makers are layering on larger nodes, and focusing on multibit (they are basically at PLC/5bit flash for consumer or non-heavy wear, which is frankly a bit nuts)
The 9.8 W is peak power. Probably the rest of the system is going to need a couple of watts to deliver and process application data at 14 GB/s as well.
You can always wish for more efficiency, but it's important to understand that SSDs have actually been getting steadily more efficient in the Joules per Byte transferred sense. We're just seeing a simultaneous concentration and consolidation of performance that is moving a bit faster, hence the need for enterprise SSDs to abandon the M.2 form factor in favor of the somewhat larger and easier to cool EDSFF family.
One way of looking at things is to realize that until now, a storage controller handling 14 GB/s of traffic would have been a full-height RAID card demanding 300 LFM of airflow, and now we're putting that much throughput into a SSD with a heatsink that's roughly 1U by 1in in cross section.
Pretty impressive. And if it can really do 4 GB/s[1] of random write performance is super helpful in database applications.
I am wondering where the "Cloud SSD" stuff takes it relative to general purpose use. Does anyone have any insights on that?
[1] They quote 1 million IOPS and assuming a 4K block size, which is a good compromise between storage efficiency and throughput, gives the 4GB/s number.
It covers a lot of ground, but as far as I'm aware nothing in there really makes drives less suitable for general server use. It just tightens up requirements that aren't addressed elsewhere.
The first wave of consumer gen4 SSDs that all used the Phison E16 controller were only good for about 5 GB/s out of the ~7 GB/s possible (and 3.5 GB/s on PCIe gen3). But the newer gen4 consumer drives that started hitting the market last fall come a lot closer, and this summer a lot of those are getting refreshed with newer, faster NAND that will have PCIe 4.0 as thoroughly saturated as PCIe 3.0 has been for the past few years. Phison has already clearly stated that their next high-end controller after the recently-launched E18 will be a PCIe gen5 chip, and the E18 is fast enough to finish out the gen4 era.
Try to imagine with all these developments how much performance you can get from vertical scaling.
I bet that many applications don’t need to care about horizontal scaling with all the burdens involved before you outgrow the performance of a ‘single’ box (aka stack overflow)
Given a theoretical either/or choice between a PCIE 5.0 SSD, or more PCI-E lanes using the tech we have now, I would rather have a greater number of PCI-E 4.0 lanes in single socket consumer/workstation grade motherboards.
Leaving open the possibility for dual NVME SSD in a workstation along with a x16 video card, and 10Gbps NICs.
It means you can configure the drive to appear as 16 or 32 separate SSDs (each with lower capacity) and assign each of those devices directly to a separate VM with almost no overhead. Google SR-IOV for the details.
123 comments
[ 2.4 ms ] story [ 165 ms ] threadI'm looking for something a bit more visual and dynamic than the old "station wagon full of tapes going down the highway" imagery.
[EDIT: Just for clarification, I feel like I already have a pretty good grasp on these concepts. I'm looking for good ways to help others at the ~ undergrad level.]
Those graphs cut off the essentially vertical latency spike that results from enqueuing requests faster than the sustained rate at which the drive can serve them. For a different view in terms of queue depth rather than throughput, there are some relevant graphs from an older review that predates io_uring: https://www.anandtech.com/show/11930/intel-optane-ssd-dc-p48...
Generally speaking, latency starts increasing long before you've reached a drive's throughput limit. Some of this is inevitable, because you have a relatively small number of channels (eg. 8) and dies to access in parallel. Once you're up to the throughput range where you have dozens of requests in flight at a time, you'll have constant collisions where multiple requests want to read from the same plane/die/channel at once, and some of those requests have to be delayed. But that's mostly about contention and link utilization between the SSD controller and the NAND flash itself. The PCIe link is pretty good about handling transactions with consistently low latency even when on average it's mostly busy.
There was another post I saw recently comparing the increase in disk SIZE over the last 30 years vs the increase in disk SPEED vs LATENCY (so size in GB, speed in GB/s, latency in IOPS) - and how size increases far outstripped speed which outstripped latency, though all had improved.
Found it! The key is IOPS/GB as a metric.
https://brooker.co.za/blog/2021/03/25/latency-bandwidth.html
https://louwrentius.com/understanding-storage-performance-io...
https://louwrentius.com/understanding-iops-latency-and-stora...
IOPS helps, but for the average user, hundreds of thousands is already functionally infinite. What matters at this point is latency. Where you really feel that is Queue Depth 1. Where you read a file that points you to other files, that point you to other files, etc. That is the exact case where the computer is still making you wait.
This happens when you start your operating system, it starts services, you launch apps, etc. Driving that latency down is the biggest improvement you'll ever see past where we are today in terms of IOPS and throughput.
This is where the latest Optane actually shines. Optane doesn't win on IOPS or throughput, but where it shines is its crazy latency (delivered at relatively low power levels). Where latencies are 10% of that of even the highest end PCIe 4.0 NVMe SSDs. Do something like launch 20 applications at once, and it'll be done in a fraction of a time compared to even something like a Samsung 980 Pro because of latency being more around 10 μs instead of 100 μs.
PCIe 5.0 SSDs will cut latencies down to where Optane is today, but driving latency under 1 μs is where we'll get into a new level of crazy.
Related: Notice how the public cloud marketing material tends to focus on scalability over other metrics? That's because scaling horizontally for them is easy: They just plop down more "stamps" -- a set of clusters and controllers that is their unit of scale. Need 1,000 more servers in US East? Plop down 10 more stamps of 100 servers. Easy!
Except of course this involves an awful lot of networking, with long cable runs and many hops and virtualisation layers. The end result is that you can't get anywhere near the underlying storage latency.
Azure's Premium SSD has "write flush" latencies of about 4 milliseconds according to my measurements, which is easily 100x slower than what my laptop can do with a now very outdated Samsung NVMe SSD.
Notice that if you go to their marketing page, they talk about "low latency" and "lowest latency", but they have no numbers? Meanwhile the MB/s and IOPS is stated with numbers: https://azure.microsoft.com/en-us/pricing/details/managed-di...
For people who really need low latency in the cloud there is EC2 instance store or GCE local SSD.
It doesn't matter if the user running Google Chrome is 50ms away from the web server, if that server in turn does 10,000 round-trips to the database server. Suddenly, microseconds matter.
Why 10K round-trips you ask? Because of ORMs like Entity Framework. They have a habit of producing wildly inefficient query code, but hey... it saves on developer time!
I literally just optimised a large web site that actually had fairly decent SQL queries. I reduced the latency from 1,500 microseconds to about 50 microseconds and the performance of most pages improved by about 150-300 milliseconds!
Let me reiterate: A latency difference that can't even be benchmarked easily because it's so small made a very noticeable difference to page load times. The site feels snappier. Navigating it is now a joy instead of a drag.
My logic always worked like this: if IOPS is input / output per second, surely that means 10,000 IOPS of, say, the usual random 4kb file write at QD1, then that means I can get 10,000 of those in one second. So, a higher number must be better.
I had no idea that IOPS needs to take latency into account. After having done some reading I am still somewhat confused as to how this really works, but for future reference I will use latency as a measurement of performance again, as that is indeed what I thought IOPS was.
My understanding is that NVMe interface is pretty much as close as you can get to the CPU without being integrated. Is there a world where these things can operate as fast as RAM?
It seems to be kinda in between RAM and flash spec wise.
It wasn't really that fast, or wear resistant to replace flash.
Many customers try to use 3D XPoint as another part of the cache hierarchy in-between regular ssd and ram. It's actually pretty neat for faas workloads which want containers to be "warm" rather than "hot" or "cold"...
The best for consumer technology, yes. But future I/O protocols continue to improve and get better.
NVidia / IBM's collaboration on OpenCAPI (which is deployed in Summit as a CPU/GPU interface) has 300GBps I/O between the CPU and GPU, far faster than NVMe speeds (and even DDR4 RAM bandwidth).
And future chips may go even faster. I/O is probably one of the fastest growing aspects of a modern computer. PCIe 5.0, CXL, OpenCAPI, etc. etc. Lots and lots of new technology coming into play here.
There are even some products on making Flash-RAM work on the DDR4 interface. Non-volatile memory is what that's called. Intel's Optane works pretty well on that. Its not very mainstream but I hear reports that its impressive (slower than real RAM of course, but storage that has the bandwidth of RAM is still cool).
> Is there a world where these things can operate as fast as RAM?
Well... yesish. Flash is a kind of RAM (random access memory).
To answer your fundamental question though: No. Flash is fundamentally higher latency than DRAM (aka: DDR4). But with enough parallelism / big enough flash arrays (or DRAM arrays), you can continue to get higher and higher bandwidth.
At the top-of-the-line is SRAM, the RAM used to make L1 cache and registers inside of CPUs. This is very expensive and rarely used.
--------
Then you've got various non-mainstream RAMs: FRAM, MRAM, Optane, etc. etc.
The big thing about SSDs is that while sequential reads and writes have steadily improved, many workloads have not. From a cheap SSD to the best, there is very minimal difference in things like "how fast does my computer boot", "how fast does my game or game level load" and "how fast does visual studio load", or "how fast does gcc compile"
https://m.youtube.com/watch?v=ph8LyNIT9sg
Partially, on a PC. The difference, on a PC, between a high end NVMe SSD and a SATA SSD is minimal for most use cases; small enough that an average user won't perceive much difference. The workloads in question (booting, loading a program, compiling code, etc.) involve a lot of operations that aren't bound by IO performance (network exchanges, decompressing textures, etc.) and haven't been optimized to maximize the benefit of high performance storage so the throughput and IOPS difference of the storage device don't entirely dominate.
I've got a workstation that has both SATA 3 and M.2 NVMe SSDs installed.
The SATA 3 device can do sustained reads of about 550 MB/sec, fairly close to the 600 MB/sec line rate of SATA 3.
The NVMe device can do about 1.3 GB/sec, faster than physically possible for SATA 3.
As the platter density increased, the head could glide over more data at the same spindle speed and you would get increasingly higher sequential speed. But you would not get much better latency - physically moving the head to a different track could not be done significantly faster. With each new generation, you would get only marginally better speed for many workloads that users actually care about. The latest HDDs could saturate SATA bus with sequential reads, but would still dip into KB territory (not MB, let alone GB) for sufficiently random-access workloads.
SSDs are similar in a sense that they can be massively parallelized for the increased throughput, but the latency of an individual cell is much harder to improve. Benchmarks will saturate the I/O queue and reach astronomical numbers by reading many cells in parallel, but for most desktop users, the queue depth of 1 (and the individual cell latency) is probably more relevant. That's why a lowly SATA SSD will be only marginally slower than the newest NVMe SSD for booting an OS or loading a game.
That said, it's still true that you typically won't see a very big difference. Any SSD can support loading a level in a few seconds. The difference between normal flash and ultra-fast flash won't show up much unless you're aggressively streaming level data off the drive in real time.
And that would likely tank the performance if the game is running from an HDD.
I suspect you could design the storage layer of your game/app/whatever to take into account the performance characteristics of the underlying drive, and adjust your I/O pattern to be more parallel or more serial accordingly.
I also suspect that would not be easy/economical to do within the time/budget constraints of most projects.
The name of the game is minimizing the head movements.
Try the following experiment: find a couple of multi-GB files on an HDD in order to copy them to an SSD.
First measurement: copy the first file then the second (then the third etc...) and sum-up the time.
Second measurement: just drag'n'drop all the files together and measure the total time.
Reboot before each measurement to purge the file cache.
There is serial as in reading byte 1 2 3 4. This is what you're talking about. This is good on any drive and great on hard drives.
There is serial as in read a byte, do a calculation, read a byte, do a calculation. This is what I was talking about. This is bad on any drive and even more noticeable on SSDs.
If you can have all your data in one single 500MB read, that's great. But I wasn't talking about that, because any drive can do a single giant read with good performance.
If you have to do 800 different reads, it's much better to ask for them all at once. You don't want to trickle them out one at a time. This is what my comparison was.
I've got a cheap SSD that will change your mind. It has amazingly poor performance though, I agree with the general concept that while quantitative differences can be measured, there's not a qualitative difference between competent SSDs.
That reminds me of non-volatile SRAM modules you can get by Microchip. How it works is that its just normal SRAM, but each cell is connected to a matching flash cell and when the mdoules voltage drops below a threshold, it writes the SRAM contents to flash in parallel. When the voltage is restored, it does the reverse. The user-observable effect is that you have normal SRAM (speed and it doesn't wear out) but its non-volatile. In reality, its a clever trick, but since the flash is only written when the module is powered off (not per-write to the module), you get a lot more use before it wears out.
Of course these modules are tiny (a few KB), but I still thought it was pretty interesting!
2) this plus QLC cells, which, durability-wise, make TLC look good, makes me anticipate headlines like "This new PCIe 5.0 SSD ran out of its rated TBW in a week!"
I don't see why — like NAND up until now, it keeps up by adding more separately-addressable chips to the board and striping writes across them. A 512GB SSD with this chip would hit the wall pretty soon, but they wouldn't waste this controller on a 512GB SSD. They'd use it for 16TB+ SSDs.
If this drive supports claimed 9 GB/s write speed, you can write 324 TB at 10 hours. Samsung QLC warranty for 1 TB drive is 360 TBW.
~10W Controller. Doesn't matter on a Desktop or Server. But we sort of hit the limit what could be done on a Laptop.
Having said that it doesn't mention what node it was fabbed on. I assume there are more energy efficiency could be squeezed out.
<6us Latency. Doesn't say what percentile it is and under what sort of condition. But Marvell claim this is 30% better than previous Marvell SSD Controller.
I think ServeTheHome article [1] is probably better. ( Cant change it now >< )
We also have PCI-E 6.0 [1] finalised by the end of this year which we cant expect to be in 2023/2024. SSD Controller with 28GB/s.
I am also wondering if we are approaching the end of S-Cruve.
[1] https://www.servethehome.com/marvell-bravera-sc5-offers-2m-i...
[2] https://www.anandtech.com/show/16704/pci-express-60-status-u...
I do like the fact that SSD bandwidth may be approaching memory bandwidth.
ROFL. Thanks. Keep thinking about the good old days.
I am sure if you go back a few years you can find systems that have the same or less memory bandwidth than this has now, but they have both been moving forward enough that SSDs are still not close. That being said, between bandwidth and random access times, swapping out to disk backed virtual memory is very pragmatic and isn't the death of performance it used to be.
I think I understand what you mean, but the gut reaction is that one.
Yeah, I am really old, after all.
Edit: let me explain. Something that is able to use the whole space of a flat memory model is way less sophisticated than something that is able to deal with a complex memory hierarchy. Our machines are indeed a complex pyramid of different subsystems with varying bandwidth and latency characteristics. A program that is able to embrace the inherent hierarchical nature of our machines (or multi-node systems) is way more "optimized", according to my sensibility.
Typically that index fronts a disk repository which wouldn't fit in RAM, although over time, what fit in RAM, what fit in disk, what lived in RAM, what got cached in RAM, etc, have changed over time.
BTW, I'm probably of the same generation as you and the single most important lesson I ever learned for computing performance was "add more RAM"; in the days when I first started using Linux with 4MB of RAM, it wasn't enough to do X11, g++ and emacs all at the same time without swapping, so I spent my hard-earned money to max out the RAM, at which point it didn't swap and I could actually do software development quickly.
I heard that ice lake sp improves a bit in the area, but haven't gotten access to one yet.
I wonder what node this controller is made on. If it’s made on TSMC N7 then they could cut power consumption roughly in half by going to N5P. The package size makes me wonder if it’s an even older node however.
Some switches could do with much better cooling. A POE switch or one that is putting 10gb down a cat 6 cable is very toasty though. I’ve got one that is borderline too hot to touch. Thanks Ubiquiti.
Servers typically have much higher power density unless you're talking 400G switches compared to low end servers.
Granted, this is a somewhat self inflicted problem: the banks' data centers were built on the assumption that power-hungry storage platforms would always be isolated to a relatively small footprint, but they are adopting hyperconverged storage architecture as their new operating standard for the future. Newer data centers sized for the increased power draw associated with hyperconverged/converged infrastructure don't tend to have this problem.
Next, there's the VDI factor - these banks are nearly entirely reliant on VDI for most employees. Most work cannot be done without connecting to a Citrix or Horizon session. As a random I/O-heavy workload, VDI benefits greatly from the hyperconverged infrastructure model in terms of cost efficacy.
Also, these banks never pay anywhere close to list price to license any of their software, so the incremental cost of spinning up a cluster to support apps with relatively meagre storage requirements is often close to negligible, which is a strong advantage over the operational costs associated with setting up dedicated storage appliances.
Of course it does. For one, that's heat you have to efficiently remove or face SSD throttling back on you or degrade over time. It does not make sense to buy expensive hardware if it is going to show impaired performance due to thermal throttling.
This makes the business of putting together your PC this much more complicated, because up until recently you only had to take care about CPU and GPU and everything else was mostly afterthought.
We are already facing motherboards with their own fans, now I suppose it is time for SSDs.
These 2U or 4U cases, or tower desktop cases, were designed to efficiently remove heat from storage devices, as well as the 3000W++ that the dual socket CPU and 8way GPUs will pull.
10W is tiny for desktop and server. Barely a factor in overall cooling plans.
Also the heat was mostly generated by motor and an actuator and not the controller.
It's the controller that gets most of the heat from running PCIE at top speed.
I have a crucial P1 NVME SSD and I can make it overheat pretty reliably. Pretty much any synthetic workload makes it overload if the SSD is empty (it reaches 70° pretty quickly and even starts throttling until it reaches 80° and the whole system starts shuttering because of extreme throttling do it doesn't damage itself. Although I have not properly tested it, it seems that not using any heatsinks from my motherboard makes the temps actually better but it still overheats.
The main reason it can overheat quickly is probably because its sitting in a really bad position where it gets close to zero airflow despite being in an airflow focused case. Most motherboards place the nvme slot directly under the GPU. The main problem seems to be that the controller is overheating when it's writing at close to 2000 MB/s. It's also important to note that only the controller (an actual relatively powerful ARM processor), not the flash memory, seems to overheat.
Fortunately, this is mostly not an issue because it's a QLC drive and the workload is unrealistic in the real world. When writing to an empty drive at 2000MB/s (Queue depth 4, 128k sequential writes), it takes 2 minutes until the cache is full. The way its currently used, it takes 30 secs for the cache to become full and for write speeds to drop to 150MB/s. The only way it has every overheated in the real world was during the loading screen of a gameplay when it reached 78C quickly (and I only noticed it in the hardware monitor). If the GPU hadn't heated up the nvme drive before (it was sitting at 65C mostly idle), and starved it for air, I doubt it would have hit 60C.
So until motherboards start placing nvme where it can get some actual cooling, or they make actual functioning heatsinks, their power usage can make a difference.
[1]: https://www.gamersnexus.net/guides/2781-msi-m2-heat-shield-i... but there are many more articles/forum posts with similar issues.
Also your high performance SSD should be going in the direct-to-CPU slot to the right of the GPU, not under it.
SSD makers are layering on larger nodes, and focusing on multibit (they are basically at PLC/5bit flash for consumer or non-heavy wear, which is frankly a bit nuts)
I think right now the real money is in somebody who can implement PCI-E 5 efficiently, or soon we'll see every SSD with a mini fan on it.
( It's not all PCI-E 5's fault, these controllers have been more and more power hungry )
One way of looking at things is to realize that until now, a storage controller handling 14 GB/s of traffic would have been a full-height RAID card demanding 300 LFM of airflow, and now we're putting that much throughput into a SSD with a heatsink that's roughly 1U by 1in in cross section.
I am wondering where the "Cloud SSD" stuff takes it relative to general purpose use. Does anyone have any insights on that?
[1] They quote 1 million IOPS and assuming a 4K block size, which is a good compromise between storage efficiency and throughput, gives the 4GB/s number.
It covers a lot of ground, but as far as I'm aware nothing in there really makes drives less suitable for general server use. It just tightens up requirements that aren't addressed elsewhere.
I bet that many applications don’t need to care about horizontal scaling with all the burdens involved before you outgrow the performance of a ‘single’ box (aka stack overflow)
Leaving open the possibility for dual NVME SSD in a workstation along with a x16 video card, and 10Gbps NICs.
Does anyone know what this means in the specs?