HN Explorer

18 comments

[ 0.22 ms ] story [ 41.8 ms ] thread
tosh
> in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS

wild

noosphr
>For example, getting good performance on a dataset with deep learning also involves a lot of guesswork. But, if your training loss is way lower than your test loss, you're in the "overfitting" regime, and you're wasting your time if you try to increase the capacity of your model.

https://arxiv.org/abs/1912.02292

jdw64
Right now, all I know how to do is pull models from Hugging Face, but someday I want to build my own small LLM from scratch
big-chungus4
How does x.cos().cos() work faster than doing two cos calls separately? Like the first cos call returns a tensor either way, the only difference is that it's not assigned to a variable. But how is it even possible know that difference in python?
teo_zero
The author forgot to add "fused" here, like they did in other parts of the same section.

Non-fused:

  foreach i
    y[i] = cos(x[i])
  foreach i
    z[i] = cos(y[i])
Fused, no intermediate variable:

  foreach i
    t = cos(x[i])
    z[i] = cos(t)
The temporary "t" doesn't leave the GPU. Sweeping the array twice makes you twice as dependent on memory bandwidth.
marketingan
Deep learning is just glorified linear algebra. Master the progression: Feed-forward CNN RNN LSTM Attention. You don't even need a GPU to understand the climax; Karpathy’s llama2.c implements a full transformer inference engine in just ~300 lines of C using SIMD pragmas for CPU execution.
fc417fc802
So you created a new account to blatantly plagiarize another comment from this same page? What's even going on here?
ThouYS
I feel like there is no portable advice for performance. A torch model exported as onnx is a different model.

That onnx model run using onnxruntime with cuda ep is a different model than the one run with TRT ep.

And even among the same runtime, depending on the target hardware and the memory available during tuning, the model behaves differently. It is a humongous mess

mycall
That's interesting as I was considering GGUF --> ONNX conversions (via Olive), but if this creates unknown distortions in the effectiveness and stability, it might be a dead-end idea.
ThouYS
Just to clarify: I mean VRAM, RAM and runtime performance, not the numerical outputs (even though those also vary to some degree, haha)
[deleted]
(comment deleted)
xiaod
I'd want to see more about the failure modes. Production systems need graceful degradation more than optimal performance.
liuliu
One thing people seems not to acknowledge, and this post made it super clear is that NVIDIA kept their lead extremely well in a few years of very high growth. The TFLOPs, the bandwidth, the interconnect mentioned in this post continues to grow at exponential rate with no sign of stopping yet. This is a 30-year-old incumbent reminding you. The willingness to compete from NVIDIA is just simply remarkable.
cold_harbor
the real lesson: GPUs win on memory bandwidth not just FLOPs. batching ops keeps VRAM fed at 2TB/s instead of tripping to RAM at 50GB/s for every operation