It's hard to square & compare the numbers versus other offerings. The XDNA engine on AMD's Strix Point APU can do 50 TFlOps INT8, so the 67tflops here isn't vastly higher. But XDNA is a pretty limited setup, for inference only really.
The iGPU on Strix Point can do 9tflops of good old fp32. Unsure what the TFlOps drops to on the new Jetson here as it goes from INT8->FP32. But probably still has a very healthy lead.
It does make the ~$120 ARM SBC out there look a little silly. Six A78 cores is way better than we can get elsewhere. And then with colossal memory bandwidth and a huge GPU. It lacks the CPU (and power efficiency) to compete with Qualcomm's mobile oriented Snapdragon X Elite fully, but would be a very interesting possible daily driver desktop.
Pretty middling io options. Single DisplayPort 1.2. 6x lanes of PCIe. 1Gbe. 4k60 decode but no video encode. Very curious if 7-25W power implies it idles at 7W; that feels like a lot to me.
> The iGPU on Strix Point can do 9tflops of good old fp32. Unsure what the TFlOps drops to on the new Jetson here as it goes from INT8->FP32. But probably still has a very healthy lead.
Not as much as you think. Per their developer blog [1], it only has 17.1 FP16 Tensor TFLOPS, so about neck and neck w/ the even the last gen (7840/8840) Radeon 780Ms.
The Orin Nano Super looks like a neat little board at a great price ($250) and wattage (25W), but basically if you are doing FP16 inference, it's more incremental than revolutionary - vs AMD it's a bit cheaper (7840 miniPCs are as low as $400 atm, there's an upcoming SBC that's launching at $330), a bit more efficient (you'll probably need 30-45W to get the max perf on the AMD board), and with better INT8 (note the 67 TOPS is sparse, it's only 33 TOPS of dense INT8), but w/ the AMD boards you can get a lot more memory (you'll need to use UMA to access it). Strix Point's NPU can match on INT8 perf, but those minipcs are selling for like $1000 and I haven't seen any SBCs so not very good price/perf there yet.
As another point of reference btw, since the Orin Nano Super is using (32) Ampere Tensor cores @ 1020 MHz. A $170 RTX 3050 has 72 Ampere Tensor Cores @ 1470 MHz w/ 168 GB/s of MBW (6GB VRAM) @ 70W. You'll expect 1/3 the compute and half the inference perf. Again, not bad for the power envelope, and a better form factor, but you aren't really getting much for "free" so to speak.
As a performance target (more MBW, more FLOPS), I'm more excited for Strix Halo and Orin Thor as a better fit for inferencing bigger models, but it seems like both AMD and Nvidia are going to continue charging an arm and a leg for those new products.
6 comments
[ 2.9 ms ] story [ 20.9 ms ] threadIt's hard to square & compare the numbers versus other offerings. The XDNA engine on AMD's Strix Point APU can do 50 TFlOps INT8, so the 67tflops here isn't vastly higher. But XDNA is a pretty limited setup, for inference only really.
The iGPU on Strix Point can do 9tflops of good old fp32. Unsure what the TFlOps drops to on the new Jetson here as it goes from INT8->FP32. But probably still has a very healthy lead.
It does make the ~$120 ARM SBC out there look a little silly. Six A78 cores is way better than we can get elsewhere. And then with colossal memory bandwidth and a huge GPU. It lacks the CPU (and power efficiency) to compete with Qualcomm's mobile oriented Snapdragon X Elite fully, but would be a very interesting possible daily driver desktop.
Pretty middling io options. Single DisplayPort 1.2. 6x lanes of PCIe. 1Gbe. 4k60 decode but no video encode. Very curious if 7-25W power implies it idles at 7W; that feels like a lot to me.
( NVIDIA Quadro RTX A500 GPU, NVIDIA Ampere GA107, 4GB GDDR6, )
https://www.adlinktech.com/en/pocket-ai-with-nvidia-rtx-a500...
https://www.wdlsystems.com/adlink-egx-tbt-a500 ( $520.00 )
Not as much as you think. Per their developer blog [1], it only has 17.1 FP16 Tensor TFLOPS, so about neck and neck w/ the even the last gen (7840/8840) Radeon 780Ms.
The Orin Nano Super looks like a neat little board at a great price ($250) and wattage (25W), but basically if you are doing FP16 inference, it's more incremental than revolutionary - vs AMD it's a bit cheaper (7840 miniPCs are as low as $400 atm, there's an upcoming SBC that's launching at $330), a bit more efficient (you'll probably need 30-45W to get the max perf on the AMD board), and with better INT8 (note the 67 TOPS is sparse, it's only 33 TOPS of dense INT8), but w/ the AMD boards you can get a lot more memory (you'll need to use UMA to access it). Strix Point's NPU can match on INT8 perf, but those minipcs are selling for like $1000 and I haven't seen any SBCs so not very good price/perf there yet.
As another point of reference btw, since the Orin Nano Super is using (32) Ampere Tensor cores @ 1020 MHz. A $170 RTX 3050 has 72 Ampere Tensor Cores @ 1470 MHz w/ 168 GB/s of MBW (6GB VRAM) @ 70W. You'll expect 1/3 the compute and half the inference perf. Again, not bad for the power envelope, and a better form factor, but you aren't really getting much for "free" so to speak.
As a performance target (more MBW, more FLOPS), I'm more excited for Strix Halo and Orin Thor as a better fit for inferencing bigger models, but it seems like both AMD and Nvidia are going to continue charging an arm and a leg for those new products.
[1] https://developer.nvidia.com/blog/nvidia-jetson-orin-nano-de...
And I think it would be cooler if this board could be made compatible with the Framework laptop https://www.youtube.com/watch?v=l6khGznGeyY