Ask HN: Training my own chat gpt

ChatGPT uses a modified version of GPT3 (details are sparse, but here are some: https://beta.openai.com/docs/model-index-for-researchers). There’s a logbook associated with the training of OPT (similar size to GPT-3): https://github.com/facebookresearch/metaseq/blob/main/projec...

This article estimated the cost of training GPT3 to be over $4M: https://heits.digital/articles/gpt3-overview

I think there are different ways to get involved:

1) Creative applications of existing models

Start playing with https://huggingface.co/

2) MLOps/data engineering - Deploying, automating, uptime, cost control

Classic SRE / SysAdmin skills

3) Creating new ML models (4 - 8 months investment)

You probably want to understand some fundamentals. https://www.deeplearning.ai/

4) ML Research - Like #3 but 4 - 8 years of investment, heavy Sciences