Ask HN: Do LLMs get "better" with more processing power and or time per request?
Do they make more (recursive) queuries into their training data for breadth and depth? Or does the code limit the algorithms by design and or by constraints other than the incompleteness of the encoded semantics?
78 comments
[ 3.3 ms ] story [ 115 ms ] threadIt takes time to train them. More = better. Usually about 6 months or so. More processing power can allow the model to cram more power in
What makes models "good" is if the dataset "fits" the model architecture properly and you have given it enough time (epochs) to have a semi accurate prediction ratio (lets say 90% accurate). For image classification models I've done around ~100 epochs for 10,000 items seems to be the best certain data sets will ever get. There will at some point come a time when the continued training of the model is either underfitting/overfitting and no amount of continued training/processing power would help improve it.
More tokens = more useful compute towards making a prediction. A query with more tokens before the question is literally giving the LLM more "thinking time"
https://arxiv.org/abs/2310.02226
I mean, i can imagine you wouldn't always need the extra compute.
The Impact of Reasoning Step Length on Large Language Models - https://arxiv.org/abs/2401.04925
>They discovered that appending dummy tokens (ignored during both training and inference) improves performance somehow. Don’t confuse their guess as to why this might be happening with actual understanding.
More tokens is more compute time for the model to utilize, that is completely true.
What they guess is that the model can utilize the extra compute for better predictions even if there's no extra information to accompany this extra "thinking time".
This is completely orthogonal to CoT, which is simply a better prompt - it probably causes some sort of better pattern matching (again very poorly understood).
I've linked 2 papers now that show very clearly the extra compute helps. I honestly don't understand what else it is you're looking for.
>This is completely orthogonal to CoT, which is simply a better prompt - it probably causes some sort of better pattern matching (again very poorly understood).
That paper specifically dives in on the effect of the length of the CoT prompt. It makes little sense to say - "oh it's just the better prompt" when Cot prompts with more tokens perform better than the shorter ones even when the shorter ones contain the same information. There is also the clear correlation with task difficulty and length.
Though I still don’t quite understand what is going on in the dummy tokens paper - what is “computation width” and why would it provide any benefit?
Models can be sensitive to the location of a needle in the haystack of its input block.
It's why there are models which are great at single turn conversation but can't hold a conversation past that without multi-turn training.
You can even corrupt the outputs by pushing past the number of turns / show the model data in a form it hasn't really seen before.
But only if we use some sort of attention optimization. For the quadratic attention algo it shouldn’t matter where the needle is, right?
It's why "RAG" techniques work, the models learn during training to make use of information in context.
At the core of self-attention is dot product measurement which causes the model to act like a search engine.
It's helpful to think about it in terms of search: the shape of the outputs look like conversation but were actually prompting the model to surface information from the QKV matrices internally.
Does it feel familiar? When we brainstorm we usually chart graphs of related concepts e.g. blueberry -> pie -> apple.
I'm not saying this isn't part of it but even if it's just dummy tokens without any new information, it works.
https://arxiv.org/abs/2310.02226
> if the [resulting] dataset "fits" the model architecture properly,
right?
I have too many questions. It seems unreasonable to ask away and I should instead read the studies and some books.
Most models, however, don't, so no special benefit from better processing other than speed
However the truth is that inference platforms do take shortcuts that affect accuracy. E.g. LLama.cpp will down convert fp32 intermediates to 8-bit quantized so it can do the work using 8-bit integers. This is degrading the computation's accuracy for performance.
[nodding repeatedly with a serious face and lot of resolve]
The same model will give the same result, and more processing power will simply enable you to get the inference done faster.
On the other hand, more resources may enable (or be required for) a different, better model.
So if you want it to spend more "time" in a useful manner without changing the architecture, you have to get it to write down the temporary information in the tokens, as "think step by step" does or alternatively iterative prompts "write a draft for the rough structure" "now rewrite it better with more detail".
It also feels like a multiplication of required processing power but I have no clue yet how one could use the previous generation of weights of and the tokens themselves to improve, elaborate on, widen the range of predicted potential results.
Is it wrong to think of this as misleading? Don't the results for exactly the same request differ because there are multiple output strings with the same computed weights?
Or do you include "multiple ways to phrase the same" in "same results" and I'm being a noob?
[No] If you mean "during inference", then the answer is mostly no in my opinion, but it depends on what you are calling a "LLM" and "processing power", haha. This is the interpretation I think you are asking for though.
[Yes] If you mean everything behind an endpoint is an LLM, eg. that includes a RAG system, specialized prompting, special search algorithms for decoding logits into tokens, then actually the answer is obviously a yes, those added things can increase skill/better-ness by using more processing power and increasing latency.
If you mean the raw model itself, and purely inference, then there's sorta 2 classes of answers.
[No] 1. On one side you have the standard LLM (just a gigantic transformer), and these run the same "flop" of compute to predict logits for 1 token's output (at fixed size input), and don't really have a tunable parameter for "think harder" -> this is the "no" that I think your question is mostly asking.
[Yes] 2. For mixture of experts, though they don't do advanced adaptive model techniques, they do sometimes have a "top-K" parameter (eg. top-1 top-2 experts) which "enable" more blocks of weights to be used during inference, in which case you could make the argument that they're gaining skill by running more compute. That said, afaik, everyone seems to run inference with the same N number of experts once set up and don't do dynamic scaling selection.
[Yes] Another interpretation: broadly there's the question of "what factors matter the most" for LLM skill, if you include training compute as part of compute (amortize it or whatever) --> then, per the scaling law papers: it seems like the 3 key things to keep in mind are: [FLOPs, Parameters, Tokens of training data], and in these parameters there is seemingly power-law scaling of behavior, showing that if you can "increase these" then the resulting skill also will keep "improving" (hence an interpretation of "more processing power" (training) and "time per request" (bigger model / inference latency) is correlated to "better" LLMs.
[No] You mention this idea of "more recursive queries into their training data", and its worth noting a trained model no longer has access to the training data. And in fact, the training data that gets sent to the model during training (eg. when gradients are being computed and weights are being updated) is sent on some "schedule" usually (or sampling strategy), and isn't really something that is being adaptively controlled or dynamically "sampled" even during training. So it doesn't have the ability to "look back" (unless a retrieval style architecture or a RAG inference setup)
[Yes] Another thing is the prompting strategy / decoding strategy, hinted at above. eg. you can decode with just taking 1 output, or you can take 10 outputs in parallel, rank them somehow (consensus ranking, or otherwise), and then yes, that can also improve (eg. this was contentious when gemini ultra was released, because their benchmarks used slightly different prompting strategies than GPT-4 prompting strategies, which made it even more opaque to determine "better" score per cost (as some meta-metric)) (some terms are chain/tree/graph of thought, etc.)
[Yes (weak)] Next, there's another "concept" of your question about "more processing power leading to better results", which you could argue "in-context learning" is itself more compute (takes flops to run the context tokens through the model (N^2 scaling, though with caches)) - and purely by "g...
If you skip to the graph here that shows the attention + feed forward displacements tending to align (after a 2d projection), is this something known/understood? Are the attention and feed forward displacement vectors highly correlated and mostly pointing in the same direction.
https://shyam.blog/posts/beyond-self-attention/
Skip to the graph above this paragraph: "Again, the red arrow represents the input vector, each green arrow represents one block’s self-attention output, each blue arrow represents one block’s feed-forward network output. Arranged tip to tail, their endpoint represents the final output from the stack of 6 blocks, depicted by the gray arrow."
I can copy paste some of my raw notes / outputs from poking around with a small model (Phi-1.5) into a gist though: https://gist.github.com/bluecoconut/6a080bd6dce57046a810787f...
quickly scanning the blog led to this notebook which shows how they're computed and shows other examples too with similar behavior. https://github.com/spather/transformer-experiments/blob/mast...
Pretty much every Yes and No apply. I had to understand bits of the gaps I was trying to close myself, so thanks for taking the time to interpret into my question.
Larger models generally perform better than smaller models (this is a generalization, but a good enough one for now). The problem is that larger models are also slower.
This ends up being a balancing act for model developers. They could get better results but it may end up being a worse user experience. Models size can also limit where the model can be deployed.
Building an LLM model consists of defining its "architecture" (an enormous mathematical function that defines the model's shape) and then using a lot of trial and error to guess which "parameters" (constants that we plug in to the function, like 'm' and 'b' in y=mx+b) will be most likely to produce text that resembles the training data.
So, to your question: LLMs tend to perform better the more parameters they have, so larger models will tend to beat smaller models. Larger models also require a lot of processing power and/or time per inferred token, so we do tend to see that better models take more processing power. But this is because larger models tend to be better, not because throwing more compute at an existing model helps it produce better results.
The 75B param model simply has more complexity to work with than the 5B model.
In the same sense that: `y = mx + b` is just not as expressive as `y = ax^2 + bx + c`.
It's viable if you have tools or humans in the loop to comment on them and add new insights.
But the speed isn't really a factor here, and seeing 1000 new apps isn't obviously going to make it better if the model is already at the limits of what it can represent with its parameter count and compression so to speak.
There's a caveat here - allowing the model to produce more tokens (i.e. giving it more compute time to "think") can produce better results. E.g. asking a model to reason before producing an answer, leads to better answers. And the extra tokens = more compute.
Different prompting techniques like what you're describing are one way, and RAG [0] and ART [1] are also in a similar category.
[0] https://stackoverflow.blog/2023/10/18/retrieval-augmented-ge...
[1] https://www.promptingguide.ai/techniques/art
The concern people might feel when they realise an ai might have private thoughts is another issue entirely.
Larger and smaller, in my beginner mind, was a difference of much recursiveness the design of the model allowed.
- User request implies knowledge about X. - PULLING in weights for X. - Probability of user knowing about Xm and Xz is low (because the training data says Xm and Xz are PhD-level knowledge or something). - Pulling in weights for an ELI5-level explanation of Xm and Xz ...
I thought, an LLM would do this recursive pulling of weights based on the semantics of the user request, which it does, but it doesn't do that "dynamically" based on "recalculated" weights and regenerated combos of tokens, which could happen if the training data wasn't "frozen" and accessible, which I learned further down in the comments, isn't.
That's why I wondered whether more processing power and or time would benefit this recursive generation and pulling.
I thought the LLM was "getting to know the user" but it had it a short memory span (the context) and thus "forgot" already calculated weights that it would use to (re)generate new weights.
Further down I learned it freaking forgets all the previous weights in general (I think that's what I learned, I'm getting there)
I think it has already been implied that we are not talking about increasing the quantity of parameters in this context but the possibily of applying additional compute to a model with a given number of parameters
* LLMs are lossy compression functions on their training data.
* The size of the model dictates how lossy the compression is.
* You can't spend compute to get more detail out of a model once it's been compressed/trained, anymore than you can spend compute to get an incredibly lossily-compressed movie to go from 240p back to the original 1080p source.
Similarly, LLMs can produce better answers if you teach them thinking strategies that remind them to put the available evidence and intermediate steps in their context window. Otherwise they'll tend to hallucinate an answer out of vaguely correct words.
Upscaling, technically, is a thing without limits, no?
I highly recommended watching Andrej Karpathy's Intro to LLMs talk, particularly the section on System 1 vs System 2 thinking. Long story short, what you are describing, using more processing to prepare a better response, is something that is an area of interest, but is not currently part of ChatGPT (or any other LLM that I am aware of).
See: https://youtu.be/zjkBMFhNj_g?t=2100&si=jaImuf3UCn6ReTp4
1) At runtime, when you feed a "request" (prompt) into the model, the model will use a fixed amount of compute/time to generate each word of output. There is no looping going on internally - just a fixed number of steps to generate each word. Giving it more or less processing power at runtime will not change the output, just how fast that output is generated.
If you, as a user, are willing to take more time (and spend more money) to get a better answer, then a trick that often works is to take the LLM's output and feed it back in as a request, just asking the LLM to refine/reword it. You can do this multiple times.
2) At training time, for a given size of model and given set of training data, there is essentially an optimal amount of time to train for (= amount of computing power and time taken to train). Train for too short a time and the model won't have learnt all that it could. Train for too long a time (repeating the training data), and the model will start to memorize the training set rather than generalize from it, meaning that the model is getting worse.
My thoughts after this sentence filled a huge gap I was wondering about, thanks.
So in that case, more models could give a better response, which costs more compute.
You can definitely stream and choose the highest scoring values amongst a few shots at generating the best next token candidate.
- If you have a fixed time budget and increase the GPU memory+compute available, you can directly query a bigger model. Raw models are basically giant lookup functions, and without the extra memory+compute, they'll spill to slower layers of your memory hierarchy, e.g., GPU RAM -> CPU RAM -> disk. Likewise, with MoE models, there are multiple concurrent models being queried.
- Most 'good' LLM systems are not just direct model calls, but code-based agent frameworks on top that call code tools, analyze the results, and decide to edit+retry things. For example, if doing code generation, they may decide to run lint analysis & type checking on a generated output, and if issues, ask the LLM to try again. In Louie.AI, we will even generate database queries and run GPU analytics & visualizations in on-the-fly Python sandboxes. These systems will do backtracking etc retries, and > 50% of the quality can easily come from these layers: LLM leaderboards like HumanEval increasingly report both the raw model + what agent framework on top. All this adds up and can quickly become more expensive than the LLM. So better systems can enable more here too.
So MoE models are a bit like thinking tools running concurrently, right(?), sieving through training data on paths that are the same contextually, but different in terms of specificity and sensitivity.
If the agents/experts/ architectures - the code - don't have the minimum required amount of memory & processing power, they might even miss entire bunches of tokens that are or might be relevant within the given (the prompt) and predicted/requested context. So more processing power and or time is relevant only to the extent, here: size, of the to-be-queried-at-inference-time training data (tokens and weights).
Now here's where I find myself exactly within the realm that I was in when I phrased my question: analysing the result of a request and evaluating different sets of tokens, which, I now understand, makes much more sense within the subject of code generation than with the recitation of facts or bits of narratives.
Generated code has functions (things to do with other things). Functions can be done more or less efficient, while even the least efficient code works "more than good and fast enough". There is no value in looping through versions of fact and fiction when the answer fits the expectation. And if it doesn't fit, users can have an actual conversation, which is where I get another part of my answer, which is that more processing power only becomes relevant in relation to the amount of concurrent requests in relation to the parts of the training data that are queried at inference time.
No single request will ever query so much data at the same time, that memory and compute become a bottleneck.
It definitely can become a bottleneck when a long/large/broad( but specific) request gets processed by MoEs simultaneously or when versions of results of engineering tasks are being evaluated. But that is simply not within the task or design of current LLMs and is instead added on top (or as a wrapper, for example, which I still fail to find a non-replaceable usecase for while also still being certain that I will find one once I get to LLMs and AIs).
Again, thanks!
An LLM can only give probabilities of the next token of output. The time to improve an LLM is during design, training, or fine tuning. Once you've got the final weights, the function is "locked in" and doesn't change.
However part of the process of learning to predict human output from the internet, literature, etc. causes some deeper learning to occur, potentially even more than in humans, certainly of a different nature. The LLM is communicating through a lossy process, and there is some randomness imposed on its outputs, so results may vary.
The nature of the prompt used can trigger some of this deeper learning, and yield better results than you might otherwise get. These weren't put in by design, they are emergent properties of the LLM. For instance "train of thought" prompting has been show to result in better output.
Prompt "engineering" is an empirical process of discovering the quirks and hidden strengths in the model. It is entirely possible that there is a super-human set of cognitive skills embedded inside GPT4, Mistral, or even LLAMA. Given sufficient time, there might be some prompting that could expose it and make it usable.
Because LLMs aren't "programs" in the traditional sense, you should treat them as if they were an alien intelligence, because that is effectively what they are. They don't understand humans, no matter how well they act like it at times. They are wild beasts, and we haven't figured out how to domesticate them yet.