ActivatedAI
- Karma
- 10
- Created
- August 2, 2024 (1y ago)
- Submissions
- 0
LLM focused research lab in SF.
Cofounded by a ML eng at Google with 16 years of experience, and an LLM obsessed brilliant hacker who was using the GPT3 API on the first day of its release.
Page 9 on the Llama tech report has an interesting graph that predicts task level performance from the cross-entropy loss. The sigmoidal model fits well, and at the steepest part of the S, a .01 change in NLL is worth…
There is some good published research about doing multiple passes over the training data, and how quickly learning saturates. The TL:DR is that diminishing returns kicks in after about 4 epochs.…