This article does not appear to do what the title claims. It mostly rounds up news from other sources.
Has anyone done the actual analysis of what the dedicated logic cores on the M1 are? I'm always curious what kinds of work Apple dedicates hardware to.
I agree, the article contains nothing new, most regurgitates or links to other articles. Despite the large chip image at the head of the article there is no discussion or description of the silicon.
Is it really possible to get such analysis if you're not a macos developer? From a layman point of view, I would take a list of instructions executed during common case OS usage and then extract the most common patterns (basically compression) and put them into silicon. These wouldn't be easy to understand if you don't know exactly what the OS is doing and how. I mean of course you need some cleanup and well thought-out design for future software development, but I wouldn't be surprised if in the future, to get the full experience of a new OS you will also need to update the hardware.
Looking at the die shot, the CPU, GPU, and NPU jumps out, if you've seen a bunch of other die shots before, and combine that with the core counts that Apple announced. For less obvious blocks, engineers with the right specialties might be able to make a guess. Otherwise thermal imaging would allow some guesses when running targeted algorithms.
Not really (that i have seen). I am not an analyst though, so not a guarantee.
Die real estate is extremely expensive. Plus it wouldn’t really throw us off the scent due to the techniques we employ.
They actually did this in the M1 for retain/release for Objective-C and Swift objects. It's an extremely common operation on the platform and is 4x faster on M1 natively, and still 2x faster than native Intel under Rosetta.
This is effectively what happens anyway because the CPU manufacturers beat each other over the head with various benchmarks, but it's worth saying that very little area of a modern CPU is actually taken up by functional units like adders
Are there any home made methods to take picture of the chips like this? What tools would you use to sand the top of the chip? What type of microscope would work?
For regular ICs I put them in a test tube with nitric acid and heat it to extract the die. Then to image the die you'll need (in order of preference) an electron microscope, or a metallurgy microscope, or a regular bio microscope.
To get the die off the flip chip carrier I use heat and scraping. Then I use a razor blade to scrape off most of the metal layers (which obscure the functional blocks), then a cheap Chinese inspection microscope ($100) attached to a DSLR with a 3D printed adapter (a DSLR with a lens attached backwards also works in a pinch). Then take a bunch of individual photos and stitch them with Hugin.
Edit: sorry, the link was previously the (lower res!) Chipworks shot I'd annotated previously. I got confused. Replaced with mine. It's less clean but higher resolution than what they released.
At the highest zoom I could get I could just about make out individual bits in the eFuse array. This is a Wii U GPU (Latte).
I do happen to have access to a SEM too, and obviously that gets a lot more fun. Here's the same eFuse area (again this is still the mechanically deprocessed chip!):
I think the mechanical delayering usually gets me to M1 or so, though it's not completely consistent depending on the specific process. Some chips work better than others.
I had no idea this could be done using relatively inexpensive tools and (to use your word) mechanical technique. It reminds me of discovering the backyard astrophotography community after long assuming photos of stellar objects could only be captured by huge telescopes owned by institutions. Thank you for sharing!
This works on modern processes because modern chips are made using CMP (chemical-mechanical polishing) to planarize the layers. On older processes the layers were just stacked on each other without being flat in the depth dimension, so you couldn't just cut cleanly in one plane like you can now.
Then again, on older chips you can usually just take a top layer shot and tell apart most of the layers anyway.
I'm not sure what to make of that Neural Engine. I was under the impression that most machine learning was using GPUs. The M1 already has a large GPU, so I'm surprised that the chip developers decided that a dedicated neural processor was worthwhile, especially in the middle of those CPUs.
Also, what kinds of jobs is that neural engine for? Presumably voice and face recognition to start with, but that doesn't strike me as enough to justify that die space. Good for research of course, but Apple seems to focus on what's practical for everyday users. I've seen stories about CPU designers wanting to use machine learning to manage system cache. I wonder if the NPU will be involved in managing the workload of the M1 chip itself.
It’s an interesting question. For one, I know that the Photos app on Macs and iPhones tends to spend loads of time going through your photos and recognizing everyone’s faces. Due to Apple’s stance on privacy, they’ve emphasized that this task should be performed on device, not in the cloud.
Without the neural engine, the processing of tens of thousands of photos can be a real burden on a typical MacBook. Apple has tried to throttle the process so that it doesn’t peg the CPU, but then it takes a lot longer to complete. The neural engine lets the process run full out without consuming a ton of power.
Most machine learning is using _NVIDIA_ GPUs, which themselves have a neural engine (tensor cores) for the last two generations. An NVIDIA A100 has around 19 Teraflops but 156 "tensor flops" (312 if you use sparse matrices).
In addition to being useful for training and inference, the consumer cards use tensor cores for things like mic filtering (RTX Voice) and neural upscaling (DLSS) in games.
General purpose GPU hardware is way more wasteful for matrix math, like maybe >10x waste on power and equally worse performance, than tensor cores.
> General purpose GPU hardware is way more wasteful for matrix math, like maybe >10x waste on power and equally worse performance, than tensor cores.
I didn't realize there was that distinction; I thought GPU's were just optimized for vector arithmetic across the board. What is the difference between general purpose GPU hardware and tensor cores? What does general purpose GPU hardware do that tensor cores do not?
A GPU's compute core, to my limited understanding, is sort of like a CPU with a lot of threads that execute the same instruction at the same time (aka SIMT). It has a decode frontend, local memory cache, addressing, registers, and a lot of instructions available to dispatch - each cycle can do something wildly different from the previous cycle. Each distinct instruction requires some dedicated hardware.
To my knowledge, "tensor cores" and neural accelerators are modeled on something like a coprocessor with a very fast memory bus, and a bajillion of the same small execution unit that can do a single operation in parallel (like a 3x3 matrix multiply) on behalf of the main processor. Like if AVX512 was actually AVX 1,000,000 and only had one instruction, and that instruction did some kind of 3x3 matrix math.
Imagine if you had a very large specialized house made entirely of kitchens (instead of bedrooms, bathrooms, etc), and your roommates were all cooks, you could cook significantly more food at once than in the typical one kitchen small household. You also save power per meal cooked because all of the lights and electricity go to kitchens instead of other rooms.
So on a pipelined CPU, the processor has different pipeline stages that execute in order. An instruction may for example move from fetch, to decode, to load, to execute, to store. The execute step may be executed on a different part of the CPU depending on what kind of instruction it is. Basic arithmetic, floating point, and vector math (such as AVX) can be dispatched through different execution ports and run on different parts of the processor. So a processor may (per core) have a pipeline, an integer math unit, a floating point math unit, and two vector units. Operations running on the execution unit also take some variable amount of time to complete. Having ports to two execution units of the same type available makes it so the processor pipeline can dispatch a second long instruction of the same type before stalling.
I don't actually know how the hardware of a tensor accelerator works, but what I would imagine a "tensor core" to be, is thousands of identical execution units that can only do basic matrix math, and a basic pipeline that is much simpler than a typical CPU.
CPUs and GPUs have highly variable workloads and need a lot of specialized hardware on chip that may not always be in use. This wastes power and means you can't have any one task as densely or efficiently as a dedicated chip. If you're designing a dedicated chip, which has direct access to the main cpu memory (as the apple neural engine has), you can design the chip to directly (streaming) read a large matrix from memory, perform an operation on it, and store the result back into memory.
Normal CPUs and GPUs don't have this capability. They approximate matrix math with lots of individual instructions that just go through a pipeline, stall, cache miss, etc just to do a lot of floating point vector math and store the result back to memory. A dedicated chip can skip all the overhead and just tile efficient matrix math in thousands of execution units. That's why an NVIDIA A100 is 19 teraflops when doing normal floating point vector math, and 150 teraflops when doing fp16 matrix math. It has a section of chip dedicated to efficiently doing the required floating point operations en masse without overhead or extra cycles and cache for fetching instructions.
The hardware requirements for training are much higher than for inference. Roughly:
- Training: Requires massive throughput and compute, so is done on high-end GPUs or TPUs in datacenters (or on researchers' beefy workstations)
- Inference: Compute requirement depends on the size and complexity of the model, but many models can be run on low-end smartphone hardware.
Generally speaking, any time you see "ML hardware" in a consumer device, that hardware is for inference, not training. Apple's Neural Engine is an example, of this. Other current examples include the ARM Ethos-N series and Qualcomm AI Engine, among others. Often this hardware is designed to balance inference performance with power consumption, so that inference can be done efficiently on battery-powered devices.
26 comments
[ 0.25 ms ] story [ 57.6 ms ] threadHas anyone done the actual analysis of what the dedicated logic cores on the M1 are? I'm always curious what kinds of work Apple dedicates hardware to.
A determined enough team of people with sulfuric acid and electron microscopes could work out exactly what is going on inside the chip.
(not that anyone would actually do this - maybe Intel? But at least in principle, the information is accessible to non-Apple folks)
Results: https://marcan.st/transf/latte_stitched.jpg
Edit: sorry, the link was previously the (lower res!) Chipworks shot I'd annotated previously. I got confused. Replaced with mine. It's less clean but higher resolution than what they released.
At the highest zoom I could get I could just about make out individual bits in the eFuse array. This is a Wii U GPU (Latte).
https://marcan.st/transf/latte_otp_slr.jpg
No chemicals needed. I'm probably one of very few people crazy enough to do this mechanically, but it clearly works!
Here's a single shot done with a reversed cheap kit Canon lens of a PS3 GPU (RSX), again with the scraping method.
https://marcan.st/transf/rsx.jpg
I do happen to have access to a SEM too, and obviously that gets a lot more fun. Here's the same eFuse area (again this is still the mechanically deprocessed chip!):
https://marcan.st/transf/sem/latte/20161210_201044.png
I think the mechanical delayering usually gets me to M1 or so, though it's not completely consistent depending on the specific process. Some chips work better than others.
Then again, on older chips you can usually just take a top layer shot and tell apart most of the layers anyway.
Also, what kinds of jobs is that neural engine for? Presumably voice and face recognition to start with, but that doesn't strike me as enough to justify that die space. Good for research of course, but Apple seems to focus on what's practical for everyday users. I've seen stories about CPU designers wanting to use machine learning to manage system cache. I wonder if the NPU will be involved in managing the workload of the M1 chip itself.
Without the neural engine, the processing of tens of thousands of photos can be a real burden on a typical MacBook. Apple has tried to throttle the process so that it doesn’t peg the CPU, but then it takes a lot longer to complete. The neural engine lets the process run full out without consuming a ton of power.
I think the NPU is not as power hungry for inference than the GPU is for the same job.
In addition to being useful for training and inference, the consumer cards use tensor cores for things like mic filtering (RTX Voice) and neural upscaling (DLSS) in games.
General purpose GPU hardware is way more wasteful for matrix math, like maybe >10x waste on power and equally worse performance, than tensor cores.
I didn't realize there was that distinction; I thought GPU's were just optimized for vector arithmetic across the board. What is the difference between general purpose GPU hardware and tensor cores? What does general purpose GPU hardware do that tensor cores do not?
To my knowledge, "tensor cores" and neural accelerators are modeled on something like a coprocessor with a very fast memory bus, and a bajillion of the same small execution unit that can do a single operation in parallel (like a 3x3 matrix multiply) on behalf of the main processor. Like if AVX512 was actually AVX 1,000,000 and only had one instruction, and that instruction did some kind of 3x3 matrix math.
Imagine if you had a very large specialized house made entirely of kitchens (instead of bedrooms, bathrooms, etc), and your roommates were all cooks, you could cook significantly more food at once than in the typical one kitchen small household. You also save power per meal cooked because all of the lights and electricity go to kitchens instead of other rooms.
So on a pipelined CPU, the processor has different pipeline stages that execute in order. An instruction may for example move from fetch, to decode, to load, to execute, to store. The execute step may be executed on a different part of the CPU depending on what kind of instruction it is. Basic arithmetic, floating point, and vector math (such as AVX) can be dispatched through different execution ports and run on different parts of the processor. So a processor may (per core) have a pipeline, an integer math unit, a floating point math unit, and two vector units. Operations running on the execution unit also take some variable amount of time to complete. Having ports to two execution units of the same type available makes it so the processor pipeline can dispatch a second long instruction of the same type before stalling.
I don't actually know how the hardware of a tensor accelerator works, but what I would imagine a "tensor core" to be, is thousands of identical execution units that can only do basic matrix math, and a basic pipeline that is much simpler than a typical CPU.
CPUs and GPUs have highly variable workloads and need a lot of specialized hardware on chip that may not always be in use. This wastes power and means you can't have any one task as densely or efficiently as a dedicated chip. If you're designing a dedicated chip, which has direct access to the main cpu memory (as the apple neural engine has), you can design the chip to directly (streaming) read a large matrix from memory, perform an operation on it, and store the result back into memory.
Normal CPUs and GPUs don't have this capability. They approximate matrix math with lots of individual instructions that just go through a pipeline, stall, cache miss, etc just to do a lot of floating point vector math and store the result back to memory. A dedicated chip can skip all the overhead and just tile efficient matrix math in thousands of execution units. That's why an NVIDIA A100 is 19 teraflops when doing normal floating point vector math, and 150 teraflops when doing fp16 matrix math. It has a section of chip dedicated to efficiently doing the required floating point operations en masse without overhead or extra cycles and cache for fetching instructions.
The hardware requirements for training are much higher than for inference. Roughly:
- Training: Requires massive throughput and compute, so is done on high-end GPUs or TPUs in datacenters (or on researchers' beefy workstations)
- Inference: Compute requirement depends on the size and complexity of the model, but many models can be run on low-end smartphone hardware.
Generally speaking, any time you see "ML hardware" in a consumer device, that hardware is for inference, not training. Apple's Neural Engine is an example, of this. Other current examples include the ARM Ethos-N series and Qualcomm AI Engine, among others. Often this hardware is designed to balance inference performance with power consumption, so that inference can be done efficiently on battery-powered devices.
Training workloads on the M1 use the GPU, not the Neural Engine: https://blog.tensorflow.org/2020/11/accelerating-tensorflow-...