Train CIFAR10 to 94% in under 10 seconds on a single A100 (github.com)

The hyperparameters are fiddly but the region of good performance is pretty flat, so I'd expect it to do well. I might take it for a spin on something else at some point, let me know if you do!

One benefit of having to go extremely fast is that it sort of makes you have to make your architectures as simple and short as possible. This helps with generalization to other datasets, I'd reckon from my experience here and other professional experience.

Thanks for your question -- much appreciated -- and there are indeed a lot of potentials for this little research project, hope this helps!

I browsed your Twitter, this might be the shortlist that you find the most interesting:

> fp16 -> fp8

>>>>> loss(dtype=fp16).sum() -> loss(dtype=fp8).mean() # the summing is for regularization's sake

> nn.MaxPool2d's backward is hellaciously slow due to what appears to be indexing. right now we're under 10 seconds without _any_ custom kernels or JITting (!!!, as far as I know!!!). this may need to change, though, because I think max_pooling is now taking longer than some of the convolutions themselves. it's also annoyingly necessary for performance, very much so

> BatchNorm2d(bias=True, weight=False) + Conv2d(bias=False) is paradoxically faster than BatchNorm2d(affine=False) + Conv2d(bias=True). This shouldn't be the case, as having a fused kernel like that should be faster.

> Information flow engineering. Knowing which layers are more information-stressed during training could help with micro-adjustments of layer depths to an uncomfortable level, but right now we're just sort of guessing in the dark. Being able to visualize and track this would be nice.

> Waiting for Pytorch 2.0! That will open up a lot of neat tricks that would otherwise slow down training with tons of small kernel launches. However, that likely will mean we end up losing our JIT card. I'm hoping that's not the Faustian bargain we have to make.

And a few others, however, those are under wraps as I think we'd like to publish those if/when they work out and once they're completed. We always have other improvements in the pipeline even as we're finalizing and publishing a release. :D

Hope that helps, the library you're working on looks super cool! Thanks for sharing the pictures/notes of that on your Twitter, I truly enjoyed reading through that.

Keep in mind that the last entry was from 2020 *(94.39% in 10 seconds)*, so it may not be a fair comparison to newer hardware like the A100.

As far as speed goes, I've filled out a few good comments about a "what's next" here with Bram in my comment to them. There's also a corresponding Reddit thread that I've been frantically corresponding on, but most of that is more historical-and-hyperparameter focused! (https://www.reddit.com/r/MachineLearning/comments/10op6va/r_...)

I'm that enby https://www.youtube.com/watch?v=jPl_1RpBK_4! :D

Where I've been for work before, I have a bit of a reputation for making things fast...unreasonably so. Reasonably unreasonably so? I don't know, I have no idea!

First, I think we implemented it basically from scratch so we could do some serious hacking and speed up the development cycles, then it's been just seriously off to the races to keep improving things with the basics -- and the basics done as well as can be! We do have some very nefarious things up our sleeves for the goal of getting below the (hopefully eventual) ludicrous bar of a 2s training time.

Very observant of you to think about and check on that, I never would have guessed that someone would have looked for that! The other posts really did not get a lot of traction, but I think now that we're really getting into the thick of it, it's starting to get a lot more spicy for different people who are interested in this sort of thing.

Alright, I'm too wired from all the nonstop commenting, I need to go to bed. 'Night! :D :) <3 <3 <3 <3 :DDDD

I think I'll certainly have to try that, the closest I have in the meantime to what you might be interested in is a more detailed profiling I did of the kernel launches in Pytorch -- it handily comes with a _very_ lovely profiler that is such a departure of the rather overly intricate and fiddly ones of yore.

Anywho -- here's what I ran into and all of the requisite debugging with it: https://github.com/tysam-code/hlb-CIFAR10/issues/2#issuecomm...

Currently, MaxPooling is one of the slowest operations per-byte of memory handled, I think due to the very slow indexed backprop. Unfortunately, it is surprisingly/shockingly necessary for good performance.

Thank you for the suggestion, I had a friend that I learned this from, and totally forgot to do that basic-identity-overhead-testing trick. D'oh'! XD If I get any good results from it, I'll be sure to do my best to tag/credit you in the GitHub release notes for the next release. :D :D :))))) <3 <3 <3 <3 :D :D :))))

I am so freaking exhausted from all the commenting here and on Reddit (self-inflicted!), so I'll just give you a gem that has saved my bacon multiple times. You are correct that profiling in this world for GPU stuff is basically absolute crap. There is one exception that I have found (finally) at long last:

with torch.profiler.profile( schedule=torch.profiler.schedule(wait=10, warmup=15, active=5), on_trace_ready=torch.profiler.tensorboard_trace_handler('./log/resnet18'), record_shapes=True ) as profiler:

    # training loop

    # do a step

    profiler.step()
    if steps > [number of wait+warmup+active steps]:
        break

    # oh, and don't forget to set your epoch count to 0 for this so you don't loop over and over again

Hope that helps. Much love and 'Night! ::) :D :D :D :D :))))

Upper limit with (I think very large scale?) training on different data, then transferring to CIFAR10 I believe is 99.5%, but that is extremely unreasonable for commodity algorithms and hardware, unfortunately, I believe D:

Hope this helps, thank you for your comment, I really appreciate it! :D

This is helpful as we don't need a BatchNorm in front of our first layer that way (and I'm sure for other reasons too).

Many thanks for asking, that's a very great question! I was super scared for a minute that I somehow had a naked hyperparameter untagged, but these certainly are very ~mysterious~ numbers indeedy!~ :D <3 <3 <3 <3 :D :D :D :)))

For deep learning, out-of-core instance loading by means of batching is the standard approach. Sure you sometimes have to tune batch size and learning rate for optimal learning, but it's a solved issue.

Opening that up, should compile times not take as long, should _~hopefully~_ allow for a lot more cool things that are slow under an eager/interpreted mode -- like, perhaps, better parallel branches of execution, or some ops which technically are much faster due to being in-place, but were on-the-whole much slower due to requiring a lot of teeny tiny individual kernel launches. I'd estimate the network will improve maybe 1-2+ seconds with it (and I really hope the compile overhead isn't that long! D: D: D: D: That would/could be another nightmare in and of itself!)