Notes on Building LLMs in 2024

Here's what I'm observing about developing LLMs. What are others seeing?

1. Prompting a base model - very fast for prototyping - eventually hits an accuracy limit, e.g. after 100 iterations of prompts and around 4 hours of dev time

2. RAG - fast to set up, e.g. can build a pipeline within 1 day. Sometimes provides a decent boost in accuracy, e.g. 10-20% relative. It can also hurt accuracy, e.g. if the search misses and the model answers confidently. You can put in optimizations like query rewriting, MMR, reranking, etc. Once you do, if the model still makes a mistake you are somewhat stuck unless you are willing to edit the model or the search, which leads us into.

3. Fine Tuning & Memory Tuning & Embedding Tuning - 1-2 weeks to set up a data and tuning pipeline and iteratively improve it to go beyond the base model. However, at this point the accuracy uplift is unlimited but requires effort. You can keep improving the training data to focus on the classes/concepts that the model is missing to keep pushing accuracy higher and giving the model new abilities. If there is a fact that you want the model to get exactly right, memory tuning lets you brainwash the model, e.g. even if the index lookup fails, back prop into the embeddings will eventually fix the lookup.

Prompts and in context learning makes prototyping rapid. So you can quickly get a sense if a new application you are building is going to work and has value.

LLMs are getting more capable, but there are still some tasks that are currently impossible. E.g. LLMs can easily write basic code like 'reverse a list in python'. They can also build simple apps like a mortgage calculator. However, writing 'windows' or 'chrome' or 'git' from scratch is beyond them, regardless of how you prompt them. This same effect appears in other domains like finance, healthcare, biotech, materials, etc.

I usually start with prompting, and if it looks promising I add RAG to get the accuracy boost, and then do FT & MT for additional accuracy improvements until I hit a target.

If I start with prompting and it doesn't even get close, I would downscope and pick a simpler problem.

One very helpful trend is that LLMs can automate most data pipeline and labeling tasks. E.g. before LLMs if I had a task like, "read through 10,000 PDFs to find and label all of the entities", I would have to build a labeling pipeline with humans. Today, an LLM script can handle it in a few minutes if I have a big enough GPU cluster.

I think this is leading to a world where training/tuning LLMs can arbitrarily boost their accuracy, and the process of labeling data is becoming automated. If you put these together, we are headed towards fully autonomous learning AIs.

The limits will be access to data/feedback for learning, choice of objectives (alignment), capital for energy, and access to technology for compute (GPUs, network, etc).

I'm personally curious to see if there is an inflection point where the LLM becomes better at technology than us. Could it break the industry and national monopolies on compute?