Ask HN: Is Apple's R1 a Discrete GPU?
The confirmed details of the R1 chip are scant:
- It does a lot of image processing and sensor data integration: "the brand-new R1 chip processes input from 12 cameras, five sensors, and six microphones to ensure that content feels like it is appearing right in front of the user's eyes, in real time. R1 streams new images to the displays within 12 milliseconds"[0]
- It has substantial memory bandwidth: "256GB/s memory bandwidth"[1]
- It uses special, high bandwidth memory: "To support R1’s high-speed processing, SK hynix has developed the custom 1-gigabit DRAM. The new DRAM is known to have increased the number of input and output pins by eightfold to minimize delays. Such chips are also called Low Latency Wide IO. According to experts, the new chip also appears to have been designed using a special packaging method – Fan-Out Wafer Level Packaging – to be attached to the R1 chipset as a single unit […]"[2]
- This is subjective, but: Apple shows it as being roughly the same size as the M2 processor in the Vision Pro marketing[3], indicating it's a peer to the M2. Now, that may mean nothing, but based on my experience with Apple kremlinology they are not arbitrary about stuff like that.
So I see all this and get major GPU vibes. But I'm just some guy on a the internet, so what do I know?
0: https://www.macrumors.com/2023/06/05/apple-reveals-vision-pro-headset/
1: https://www.apple.com/apple-vision-pro/specs/
2: https://9to5mac.com/2023/07/11/vision-pro-performance/
3: https://www.apple.com/apple-vision-pro/
57 comments
[ 3.2 ms ] story [ 27.0 ms ] threadEdit: many are saying that is just an Image Signal Processor (https://en.wikipedia.org/wiki/Image_processor). I don't think that's quite the case because 1) The M-series chips are already known to have ISP's packaged into them. and 2) My understanding is that the R1's job is to provide continuity of passthrough even in the event of a kernel panic by the M-series chip. To my thinking, this means the R1 chip must have a level of independence beyond that of a traditional coprocessor. I think it is an entire SOC.
1) There are plenty of pro users they can’t serve without discrete GPUs, since they’ll never have the power or transistor budget in an integrated offering that would allow them to compete with AMD and Nvidia.
2) Why do a Mac Pro that doesn’t support discrete GPUs when the Mac Studio exists?
3) They’re doing hardware stuff in the M3 GPU that indicates a fairly serious GPU effort.
4) If they’re willing to put this much effort into a co-processor for a low-volume experimental product, putting similar effort into a co-processor (discrete GPU) for high volume sure-thing products (Pro laptops and desktops) seems at possible.
If Apple ever makes an "M3 Extreme" by gluing 4x Max dies together, they could have up to 512GB of VRAM. You'd need to 7x H100 GPUs to match total RAM size which will cost you $210k.
https://en.wikipedia.org/wiki/Image_processor
Errr, that’s not how this works. The size of them being similar means absolutely nothing.
The R1 is a coprocessor, I mean it’s basically a big signal processor with a decent amount of RAM to handle all the camera inputs, and does some GPU like stuff, but isn’t a GPU
I know discrete GPU comparable memory bandwidth doesn't necessarily mean it's discrete GPU or a peer to the M2, but there's clearly a lot of something going on. Plus isn't it kinda weird to specifically mention a spec that is applicable to programmable devices like CPUs or GPUs on a signal processor? Like, who cares?
0: https://www.tomshardware.com/news/apple-m2-gpu-analysis
M2 was 100GB/s.
https://en.wikipedia.org/wiki/Digital_signal_processor
(Of which https://en.wikipedia.org/wiki/Image_processor are a subset of, which someone else in this forum mentions.)
DSPs are similar to GPUs in many respects but they are also similar to CPUs in many respects, but it doesn't make them the same.
1. GPUs are good at matrix multiplication!
2. GPUs are programmable, so you can do software updates, which seems like it would be useful if they're also using the R1 to do object recognition and other machine vision tasks.
3. GPUs would be useful in other parts of their product lineup.
I guess the unknown is what exactly happens on this chip vs the M2. Clearly some computer vision is being done to do object/people recognition and to separate out objects in order to place the content in 3D space. And then there’s stuff like deciding when to allow the external environment to “break through” immersive content. And of course placing the content in the scene.
It seems like those would make the most sense on the R1, since it avoids contention with “userland” processes. But I don’t know if those tasks are more GPU tasks or DSP tasks. If more GPU, then I guess this thing is a SoC with a lot of die space dedicated to DSP.
https://techcrunch.com/2023/06/05/apple-r1-chip-apple-vision...: "The specialized chip was designed specifically for the challenging task of real-time sensor processing, taking the input from 12 cameras, five sensors (including a lidar sensor!) and six microphones. The company claims it can process the sensor data within 12 milliseconds — eight times faster than the blink of an eye — and says this will dramatically reduce the motion sickness plaguing many other AR/VR systems."
Any lag from the position of your head/body and the eye display is going to mess with your proprioception. The worse that lag the more likely you are to get motion sickness.
I think it would actually make more sense for the M1 to treat the R1 as a display that it writes final composited frames to, then the R1 integrates the output from the M1 into the rest of the scene it’s rendering from the other sensors. IE, the output of the M1 is essentially another camera input to the R1 (well, camera plus multi-channel audio).
imo the illusion was rock solid, extremely challenging given that the display was transparent so you had to keep up with the real backdrop moving - vision pro and all passthrough devices get to fake it but at the cost of proprioception as you said
I have four of them at work. I have thousands of hours on them. They’re amazing.
That's why I assume the R1 is trying to provide the render pipeline with "current" positional state and then tries to finish the drawing in the remaining 4ms (for 60fps) then the display is only going to lag the wearer's perception by 16ms which is less likely to cause discomfort.
This could be mitigated more if the objects in the VR scene are tagged with motion vectors. If the R1 state update doesn't land in time the renderer can interpolate the "current" position by applying those motion vectors to objects in the scene.
Not even the most expensive high-end gaming setups can finish the entire input-to-screen processing within just one frame, and yet they can easily render some games at 500Hz or more.
Every time you move a byte one stage away from the sensor you (general) incur an order of magnitude of power cost. There are many many caveats, and its not a hard and fast rule, but its a useful illustration. Don't get up about precise numbers, just understand that moving data between devices, even inside a processor has a power cost.
e.g. reading out a pixel from a camera costs 1, moving it to a UART/interface costs 10, reading it into a register costs 100, l1 1000 etc etc etc.
The closer you can do the processing to the sensor the greater the (potential) power saving, and in theory the lower the latency.
For example the eye cameras you only really want the direction the eyes are looking. So you don't want to ship the entire image at 120fps to a processor, do some maths to then get that vector. Ideally you want the sensor to do it for you and just ship four floats every 1/120th of a second.
I'm also not sure where the main processor is for the goggles, so data compression and coordination of the sensors also becomes critical. The images sensors almost certainly don't run at a high enough framerate to be "magical" so you need IMU to generate a fudge factor. They need to run at a known sample rate, one that doesn't deviate. Using the CPU to do that is bad, because that'll require a lot of interrupts, leaving little room for other stuff that's important (like other sensors)
now, I've hinted that the "SPU" does processing, which is almost certainly does. This means that it might actually look like a GPU after all, as its doing a lot of calculations that are very "graphics-y" All that computer vision: SLAM for head orientation, HAnd tracking for, well, hands, Image stitching/warping and correction to make the pass through work, plus the audio shit, it all very computer vision. So its highly likely that it looks a bit like a GPU, because its doing similar tasks
And that's apparently from single chip, the RTX 4060 needs 6 chips get equivalent bandwidth.
Though for that exact reason, I'm really not sure I trust the "single chip" part of the rumour. That's a lot of bandwidth per chip and it seems more likely to have four memory chips, which would result in 2 GB of capacity.
> it seems more likely to have four memory chips, which would result in 2 GB of capacity.
Four times 1 Gigabit is 4 Gigabit, which equals half a Gigabyte: 512 MB. Which was less than the high-end for a desktop GPU 15 years ago. I'm still not seeing any way that this memory configuration is remotely plausible for a GPU today even though it is comparable on one metric.
Here's an obviously-incomplete list of source files part of the R1 firmware, probably referenced from asserts or other logging messages, thus present as strings in the firmware binary:
https://transfer.archivete.am/inline/Ydfxb/bora.txt
It seems it's handling data from cameras (CImageSensor*), LIDAR (SensorMgr/Tof = time of flight?), and display (DCP). I also see mentions of accel, gyro, bmi284 (IMU from Bosch?).
The H13 codename suggests it's actually M1 derived.
And no sign of the GPU, which goes by the name AGX, seems to be more IPU than anything else.
You can do a lot with not much if it is all specialised hardware. Some of the wider features of the chip are due to the huge data bandwidth. But general purpose processing is a little slow for this task, certainly at this power envelope
And it's entirely possible that one of the components Apple took from their grap-bag of SoC component is the GPU compute cores. Properly not the rasterizer or texture samplers, but I could see the compute cores being useful for running tracking algorithms. The massive amount of memory bandwidth does kind of suggest GPU compute cores.
But even if it does, that doesn't make it a discrete GPU. Just a dedicated SoC with some GPU components, and a bunch of other things.
But it simply can't be used as in indicator of discrete GPUs in Macs, because all those other SoC components would be a waste of silicon and it's unlikely Apple would reuse the R1 die as a dedicated GPU.
Yeah, this is where I'm at after reading all the comments on this thread.