HN Explorer

Fine Tuning Language Models

3 points by amohajerani1 ↗ HN
I want to fine tune a language model to summarize the action items from meeting transcriptions. Due to privacy concerns, I need to run on the model on my local machine. I also need to be able to fine tune the model for our specific need. How do I pick a model to start with? There are so many options, and I feel lost.

3 comments

[ 3.6 ms ] story [ 16.5 ms ] thread
brucethemoose2
Most AI runners just ignore licensing and (illegally?) run LLaMA finetunes. There are tons of good 33B/65B instruct models you can run on any decent PC.

But if you want to avoid the non commercial LLaMA license, you have 3 good options for a base model.

- OpenLlama 13B

- MPT 30B

- Falcon 40B

Of these, Falcon 40B is very difficult to run (slow in 4 bit, basically requires a professional GPU, no good cpu offloading yet).

OpenLLaMA 13B only supports a context size of 2048 as of today... But that could change soon.

So you probably want MPT instruct 30B, specifically this one:

https://huggingface.co/TheBloke/mpt-30B-instruct-GGML

As the page says, you can try it out on a decent PC of your own with the OpenCL build of KoboldCPP. Change it to "instruct" mode, use the template on the page, offload as many layers as you can to your PC's dGPU, and run it in instruct mode. It may already work for your summarization needs.

If not, you can finetune it with MPT's code and summarization data.

https://github.com/mosaicml/llm-foundry

Or train OpenLLaMA 13B with SuperHOT + summarization data using QLORA.

matrix2596
Is there a good summary of the trade offs of the various available models (open source) and their comparison to other models.