What are the variables that prefer local GPUs vs cloud inference? Is connectivity the dividing line or are there other variables that influence the choice?
Anduril submersibles probably need local processing, but does my laundry/dishes robot need local processing? Or machines in factories? Or delivery drones?
The CPU cores are very different (big Cortex-X925 + small Cortex-A725 vs. medium-size Cortex-X4 = Neoverse V3AE).
The CPU of DGX Spark has better single-threaded performance, while that of Thor has better multi-threaded performance per die area and per power consumption.
So the single place that we can buy this is showing no stock (already) and not clear if this will even ship given all the customs and tariffs stuff. I must say after waiting for months on the 'almost ready to ship' DGX Spark (with multiple partners no less), getting strong announce-ware vibes from this already.
My naive first reaction was that a unit like that would consume a way too much power to be practical on a robot, but then I remembered how many calories our own brains need vs the rest of our body (Google says 20% of total body needs).
Looks like power consumption for the Thor T5000 is between 30W-140W. The Unitree G1 (https://www.unitree.com/g1) has a 9Ah battery that lasts 2hrs under normal operation. Assuming an operating voltage of 48V (13s battery), that implies the robot's actuator and sensor power usage is ~216W.
Assuming average power usage is somewhere down the middle (85W), a thor unit would consume 28% of the robot's total power needs. This doesn't account for the fact that the robot would have to carry around the additional weight of the compute unit though. Can't say if that's good or bad, just interesting to see that it's in the same ballpark.
If I were Jensen Huang, the first thing I'd do...well, the _first_ thing I'd do is ditch the silly leather jacket and dress like an adult. But the first thing I'd do with Nvidia is make sure the company's product line is well diversified for the coming AI winter.
Memory bandwidth is less than 300 GB/sec, looking at the data sheet. So it won't really be any faster at local inference than a Mac Pro or a decent x86 box.
It appears to be an embedded DGX Spark, at the end of the day.
Wow: notably a more advanced CPU than DGX GB200! 14 Neoverse V3AE cores, where-as Grace Hopper is 72x Neoverse V2. Comparing versus big GB100: 2560/96 CUDA/Tensor cores here vs big Blackwell's 18432/576 cores.
> Compared to NVIDIA Jetson AGX Orin, it provides up to 7.5x higher AI compute and 3.5x better energy efficiency.
I could really use a table of all the various options Nvidia has! Jetson AGX Orin (2023) seems to start at ~$1700 for a 32GB system, with 204GB/s bandwidth, 1792 Ampere, 56 Tensor, & 8 A78AE ARM Cores, 200 TOPS "AI Performance", 15-45W. Slightly bigger model of 2048/64/12 cores/275 TOPS, 15-60W available. https://en.wikipedia.org/wiki/Nvidia_Jetson#Performance
Now Jetson T5000 is 2070 TFLOPS (but FP4 - Sparse! Still ~double-ish). 2560 Core Blackwell, 96 Tensor cores, 14 Neoverse V3AE cores. 273GB/s 128GB. 4x25Gbe is a neat new addition. 40-130W. There's also a lower spec T4000.
Seems like a pretty in line leap at 2x the price!
Looks like a physically pretty big unit. Big enough to scratch my head in the intro video of robots opening up the package & wonder: where are they going to fit their new brain? But man, the breakdown diagram: it's- unsurprisingly- half heatsink.
It should be noted that Neoverse V3AE and Neoverse V3 are the automotive/server versions of the Cortex-X4 core, which is well known from many smartphones (and which is similar in performance to the Skymont E-cores of the Intel Lunar Lake, Arrow Lake S and Arrow Lake H CPUs).
While the Cortex-X925, the successor of Cortex-X4, has better absolute performance, it has much worse performance per die area. Therefore, for a CPU where the best multi-threaded performance is desired, Neoverse V3AE/Neoverse V3/Cortex-X4 remains the best CPU core designed by the Arm company.
This year's Arm core announcements have been delayed and it is not clear how the future Cortex-A930 and Cortex-X930 will compare with the currently existing Cortex-X4, Cortex-A725 and Cortex-X925.
I was reading Xiaomi YU7 marketing page[0] yesterday and the NVIDIA AGX Thor stood out (says: NVIDIA DRIVE AGX Thor). I was wondering what it was and this showed up! Looks like it is (or a Drive variant of it) is already being used in newer cars for self-drive and such.
[0] https://www.mi.com/global/discover/article?id=5174
28 comments
[ 2.6 ms ] story [ 57.7 ms ] threadAnduril submersibles probably need local processing, but does my laundry/dishes robot need local processing? Or machines in factories? Or delivery drones?
The CPU of DGX Spark has better single-threaded performance, while that of Thor has better multi-threaded performance per die area and per power consumption.
NVIDIA Jetson Thor Unlocks Real-Time Reasoning for General Robotics and Physical AI
https://blogs.nvidia.com/blog/jetson-thor-physical-ai-edge/
> The Jetson Thor chips are equipped with 128GB of memory, which is essential for big AI models.
Just put it into a robot and run some unhinged model on it, that should be fun.
https://www.arrow.com/en/products/945-14070-0080-000/nvidia?...
Looks like power consumption for the Thor T5000 is between 30W-140W. The Unitree G1 (https://www.unitree.com/g1) has a 9Ah battery that lasts 2hrs under normal operation. Assuming an operating voltage of 48V (13s battery), that implies the robot's actuator and sensor power usage is ~216W.
Assuming average power usage is somewhere down the middle (85W), a thor unit would consume 28% of the robot's total power needs. This doesn't account for the fact that the robot would have to carry around the additional weight of the compute unit though. Can't say if that's good or bad, just interesting to see that it's in the same ballpark.
It appears to be an embedded DGX Spark, at the end of the day.
> Compared to NVIDIA Jetson AGX Orin, it provides up to 7.5x higher AI compute and 3.5x better energy efficiency.
I could really use a table of all the various options Nvidia has! Jetson AGX Orin (2023) seems to start at ~$1700 for a 32GB system, with 204GB/s bandwidth, 1792 Ampere, 56 Tensor, & 8 A78AE ARM Cores, 200 TOPS "AI Performance", 15-45W. Slightly bigger model of 2048/64/12 cores/275 TOPS, 15-60W available. https://en.wikipedia.org/wiki/Nvidia_Jetson#Performance
Now Jetson T5000 is 2070 TFLOPS (but FP4 - Sparse! Still ~double-ish). 2560 Core Blackwell, 96 Tensor cores, 14 Neoverse V3AE cores. 273GB/s 128GB. 4x25Gbe is a neat new addition. 40-130W. There's also a lower spec T4000.
Seems like a pretty in line leap at 2x the price!
Looks like a physically pretty big unit. Big enough to scratch my head in the intro video of robots opening up the package & wonder: where are they going to fit their new brain? But man, the breakdown diagram: it's- unsurprisingly- half heatsink.
While the Cortex-X925, the successor of Cortex-X4, has better absolute performance, it has much worse performance per die area. Therefore, for a CPU where the best multi-threaded performance is desired, Neoverse V3AE/Neoverse V3/Cortex-X4 remains the best CPU core designed by the Arm company.
This year's Arm core announcements have been delayed and it is not clear how the future Cortex-A930 and Cortex-X930 will compare with the currently existing Cortex-X4, Cortex-A725 and Cortex-X925.
Would be interested to see head to head benchmarks including power usage between those mini PCs and the Nvidia Thor.