I have experience working with Large Language Models (LLMs) during my tenure at my previous company. However, one aspect I didn't get the opportunity to explore there was fine-tuning my own LLM, so I decided to experiment.
While there are many pieces of documentation available on this topic scattered across the Internet, putting them all together is not obvious. Therefore, in this repository, I present the full training code, from scrapping, to generating the dataset to the finetuning itself.
The whole thing takes a few days to complete, and it does work on my AMD GPU!
1 comment
[ 3.9 ms ] story [ 14.3 ms ] threadWhile there are many pieces of documentation available on this topic scattered across the Internet, putting them all together is not obvious. Therefore, in this repository, I present the full training code, from scrapping, to generating the dataset to the finetuning itself.
The whole thing takes a few days to complete, and it does work on my AMD GPU!