LoRA? The parameter-efficient fine-tuning method published 2 years before Llama and already actively used by researchers? RoPE? The position encoding method published 2 years before Llama and already in models such as…
If your starting position is already that Sam Altman lies about everything that doesn't fit your preconceived positions, that doesn't seem like a very useful meaningful position to update.
>that they need to rig their elections against themselves to get dissenting voices I don't believe this is true. If you're talking about Non-Constituency Members of Parliament, they are consolation prizes given to best…
GPT-1 wasn't used as a zero-shot text generator; that wasn't why it was impressive. The way GPT-1 was used was as a base model to be fine-tuned on downstream tasks. It was the first case of a (fine-tuned) base…
I believe the above post was highlighting that as a misconception young people may have, not saying it is the case.
Two points to consider, one against and one for. 1) It's a small island, but it's also a major trading port. Which means its whole economy is already geared towards importing food from neighboring countries. 2) On the…
The long story short is you are technically correct but in practice things are a little different. There are 2 factors to consider here: 1. Model Capability You are right that mechanically, input and output tokens in a…
Is it stated somewhere that Radford was inspired by that blog post?
It is no coincidence that EleutherAI named their pretraining dataset "the Pile"
This arguments feels like it's trying to be an inch too smart. Consider the following: Amazon isn't really an online retail company; it doesn't really sell goods to the consumer. What it does is use "goods" that it…
> the RoPE embeddings in Code Llama were designed for this. The RoPE embeddings were not "designed" for that. The original RoPE was not designed with length extrapolation in mind. Subsequent tweaks to extrapolate RoPE…
BERT was on arXiv before being peer reviewed. As were T5, BART, LLaMA, OPT and GPT-NeoX-20B. The Pile and FLAN were also on arXiv before being peer reviewed. Of course, the original Transformer paper was also on arXiv…
Makes sense! But expensive...
But what would they be calling out? If industry groups want to run a training run based on the configurations of a well-performing model, I don't see anything wrong with that. Now, if they were to claim that what they…
Yep I understood that you were using it informally, just trying to keep things informative for other folks reading too.
I want to jump in and correct your usage of "LLaMA Laws" (even you are using it informally, but I just want to clarify). There is no "LLaMA scaling law". There are a set of LLaMA training configurations. Scaling laws…
If you want a speedrun explanation for how we get to "2": In the limit of model scaling, context size doesn't matter (yes, forget about the quadratic attention), most of the compute is in the linear layers, which boil…
It's actually even less remarkable than that. It was an experiment in having a limited release, to shift the field toward a different release convention. > Nearly a year ago we wrote in the OpenAI Charter: “we expect…
While MoE-LoRAs are exciting in themselves, they are a very different pitch from full on MoEs. If the idea behind MoEs is that you want completely separate layers to handle different parts of the input/computation, then…
Another example that I read about once and have never been able to verify (or it may be completely made up) is that the because the Chinese invented porcelain first (which was more sturdy than glass or something) they…
As a researcher in the field, I agree with this characterization. I think it's more accurate the say that GPT and then BERT massively popularized and simplified the idea/approach. Prior to ULMFiT/GPT/BERT, fine-tuning…
This really should be it. Open up but slowly deteriorate the experience. Here's a concrete proposal: Set automod to delete any post/comment longer than 20 characters. This makes discussion virtually impossible, and…
Right, I hate to be that guy, but it essentially boils down to "take a random step and see if it's good, if so update" (with some fanciness). It should go without saying that this is horribly inefficient in a very high…
As someone who is in the field: papers proposing to solve the context length problem come out every month. Almost none of the solutions stick or work as well as a dense or mostly dense model. You'll know when the…
1. It is available via API 2. Likely not a conspiracy theory. Newer models don't have logits available and that's almost certainly because they didn't want other labs distilling from them.
LoRA? The parameter-efficient fine-tuning method published 2 years before Llama and already actively used by researchers? RoPE? The position encoding method published 2 years before Llama and already in models such as…
If your starting position is already that Sam Altman lies about everything that doesn't fit your preconceived positions, that doesn't seem like a very useful meaningful position to update.
>that they need to rig their elections against themselves to get dissenting voices I don't believe this is true. If you're talking about Non-Constituency Members of Parliament, they are consolation prizes given to best…
GPT-1 wasn't used as a zero-shot text generator; that wasn't why it was impressive. The way GPT-1 was used was as a base model to be fine-tuned on downstream tasks. It was the first case of a (fine-tuned) base…
I believe the above post was highlighting that as a misconception young people may have, not saying it is the case.
Two points to consider, one against and one for. 1) It's a small island, but it's also a major trading port. Which means its whole economy is already geared towards importing food from neighboring countries. 2) On the…
The long story short is you are technically correct but in practice things are a little different. There are 2 factors to consider here: 1. Model Capability You are right that mechanically, input and output tokens in a…
Is it stated somewhere that Radford was inspired by that blog post?
It is no coincidence that EleutherAI named their pretraining dataset "the Pile"
This arguments feels like it's trying to be an inch too smart. Consider the following: Amazon isn't really an online retail company; it doesn't really sell goods to the consumer. What it does is use "goods" that it…
> the RoPE embeddings in Code Llama were designed for this. The RoPE embeddings were not "designed" for that. The original RoPE was not designed with length extrapolation in mind. Subsequent tweaks to extrapolate RoPE…
BERT was on arXiv before being peer reviewed. As were T5, BART, LLaMA, OPT and GPT-NeoX-20B. The Pile and FLAN were also on arXiv before being peer reviewed. Of course, the original Transformer paper was also on arXiv…
Makes sense! But expensive...
But what would they be calling out? If industry groups want to run a training run based on the configurations of a well-performing model, I don't see anything wrong with that. Now, if they were to claim that what they…
Yep I understood that you were using it informally, just trying to keep things informative for other folks reading too.
I want to jump in and correct your usage of "LLaMA Laws" (even you are using it informally, but I just want to clarify). There is no "LLaMA scaling law". There are a set of LLaMA training configurations. Scaling laws…
If you want a speedrun explanation for how we get to "2": In the limit of model scaling, context size doesn't matter (yes, forget about the quadratic attention), most of the compute is in the linear layers, which boil…
It's actually even less remarkable than that. It was an experiment in having a limited release, to shift the field toward a different release convention. > Nearly a year ago we wrote in the OpenAI Charter: “we expect…
While MoE-LoRAs are exciting in themselves, they are a very different pitch from full on MoEs. If the idea behind MoEs is that you want completely separate layers to handle different parts of the input/computation, then…
Another example that I read about once and have never been able to verify (or it may be completely made up) is that the because the Chinese invented porcelain first (which was more sturdy than glass or something) they…
As a researcher in the field, I agree with this characterization. I think it's more accurate the say that GPT and then BERT massively popularized and simplified the idea/approach. Prior to ULMFiT/GPT/BERT, fine-tuning…
This really should be it. Open up but slowly deteriorate the experience. Here's a concrete proposal: Set automod to delete any post/comment longer than 20 characters. This makes discussion virtually impossible, and…
Right, I hate to be that guy, but it essentially boils down to "take a random step and see if it's good, if so update" (with some fanciness). It should go without saying that this is horribly inefficient in a very high…
As someone who is in the field: papers proposing to solve the context length problem come out every month. Almost none of the solutions stick or work as well as a dense or mostly dense model. You'll know when the…
1. It is available via API 2. Likely not a conspiracy theory. Newer models don't have logits available and that's almost certainly because they didn't want other labs distilling from them.