Neural Networks: Zero to Hero (karpathy.ai)

Let's build GPT: from scratch, in code, spelled out: https://www.youtube.com/watch?v=kCc8FmEb1nY

https://deepai.org/machine-learning-glossary-and-terms/logit....

For example in many classification problems you get a 1D vector of logits from the final layer, you apply softmax to normalise, then argmax to extract the predicted class. It extends to other tasks like semantic segmentation (predict pixel classes) where the "logit" output is the same size as the image with a channel for each class and you apply the same process to get a single channel image with class-per-pixel.

Here's a nice explanation: https://stackoverflow.com/a/66804099/395457

Before going through the softmax layer, the logits will be small numbers around 0, probably. Something like: [2.89, -4.53, 0.24, -1.556, 0.57]. Logits like this are natural outputs of a neural network, because they can be any real number and everything will still work.

The logits become percentage as follows:

    julia> x = [2.89, -4.53, 0.24, -1.556, 0.57]
    5-element Vector{Float64}:
      2.89
     -4.53
      0.24
     -1.556
      0.57
    
    julia> x = e.^x
    5-element Vector{Float64}:
     17.993309601550315
      0.010780676072743085
      1.2712491503214047
      0.2109782988178321
      1.768267051433735
    
    julia> x / sum(x)
    5-element Vector{Float64}:
     0.8465613320288766
     0.0005072164987105474
     0.05981058503789324
     0.009926248902037537
     0.08319461753248213

Generally I'm a huge fan of not getting too caught up in theory before diving into practice, but I'm seeing multiple responses to this comment without a single mention of "log odds".

The logit function transforms probabilities into the log of the odds (ln P(X)/(1-P(X)), which is important because it makes probabilities linear, which they are not in their standard [0,1] form. It's the foundation of logistic regression, which is, despite much misinformation, quite literally linear regression with a transformed target.

The logistic function is the inverse of the logit: it turns log odds values into probabilities once again. Logistic regression actually transforms the model not the target (most of the time) because the labels 0 and 1 are negative and positive infininity which can't be handled by linear regression (so we transform the model using the inverse instead).

I don't think I can stress enough how important is its to really understand logistic regression (which is also the basic perceptron) before diving into neural networks (which are really just an extension of logistic regression).

Watching only is much nicer for entertainment.

There is simulated environment in which all teams of the cohort receive millions of requests per day (and hundreds of thousands of users and items) and you have to build out your infrastructure on an EC2 instance, build a basic model, and then iteratively improve on it. Imagine a simulated facebook/youtube/tiktok-style system where you aim for the best uptime and the best recommendations!