So does the title. And the abstract, if you have enough of an understanding of the field to make it fruitful to be reading prepubs in the first place...
There were certain language choices in the paper that led me to think that just maybe they weren't doing what they were doing, but the longer I read, the more certain I became that they were doing exactly what they shouldn't, wasting my time.
well yes, so said the discussion. I was just irritated that i read theough it that far.
but not being a machine learning pro, I'm happy that I knew enough to be highly suspicious.
It's a 3 pages paper with absurd ideas from the title on.
Imagining it was not clear enough, the author decided to name the model something 'pronounced “fictional”', and shows how it can answer questions that weren't even asked yet with perfect accuracy right on the abstract.
Accept that as an exercise of familiarizing yourself with the area's language. If you are interested on it, it's not lost time.
It's a two-and-a-half-page paper with about a page and a half of total text with an obvious joke title, absurd and obviously joking claims in the abstract, and names the model "fictional". This is all before you even click through to the PDF!
If it takes you reading the whole paper up to the point where the author explains the joke to realize that the paper tagged satire is satire, I mean, I'm not sure paper-reading is for you. There are unmarked papers out there that aren't even satirical, just wrong!
Contamination seems to be a very serious problem for LLM evaluation. These models typically seek to learn from, essentially, The Entire Internet.
What's left, after you use, say, "half" of The Entire Internet for training? What's left that is definitely not the same as the half you used? How can you be sure, when you're working with terabytes or petabytes?
Credible benchmarking is going to have to become a field in itself.
There are 'canary tokens' that you can add to your dataset which are supposed to be respected by self-regulating scrapers & responsible ML practitioners to grep them out before doing any training. How well it works is unknown, of course...
> Disclaimer: if you haven’t figured out by now that this manuscript is satire, this manuscript is
satire
I find this annoying, and not funny, honestly.
Why are satire papers mixed in with real ones on arxiv.org? Obviously, I'm not a research scholar and don't know how the whole process works, but as someone who likes to know whether they are dealing with fact or fiction from the start, this was a waste of time.
If I were you, my take away would be that if you can't evaluate a paper critically enough to immediately tell this is satire, maybe stick to sources that have been vetted by those who can (published papers, books, etc) until you're familiar enough with the field. Arxiv is for researchers communicating with other researchers, and the tendency of AI/crypto groups to use it for marketing has been a disservice to the public.
There aren't any special critical evaluation skills required. The arxiv page says:
> Comments: 3 pages, satire
Tangentially, it's worth pointing out that you are gatekeeping read access to a resource because of a write access problem. If you don't want open access then go back to Elsevier.
I was giving advice for how to learn a subject. The text is obviously satire even without the warnings if you understand the fundamentals of ML. I'm not saying they need a PhD or something.
I have no problem with open access, and would rather journals were required to be open access. But reading brand new results that haven't had any peer review without familiarity with the field is putting the cart before the horse. I certainly don't learn a new subject by reading the hottest papers on arxiv.
Because it's not just trying to be funny for the sake of it, it's a genuine and biting criticism of the way LLMs are commonly evaluated. Reductio ad absurdum as a paper.
This is funny because everyone uses the same tests to evaluate different models. One example is the Hugging Face OpenLLM Leaderboard. [1]
The problem that this "paper" is making fun of is that any model can score very well on these tests when the content of the tests themselves is used in the training of the models. Here is one example of this happening with a 13B model that scored way too high. [2]
In reality, when reviewing test scores, it's difficult to discern whether a model has been pre-trained on some specific content that is related to the test, or if the model has been trained in a way that produces novel emergent abilities not seen in other models. [3] This means that LeaderBoards and test scores for LLMs provide a low-confidence metric for what the propose to be testing.
The paper's title is itself a continuation of the phenomenon where papers are named after the landmark paper "Attention Is All You Need". [4] A good example of this is the "Textbooks Are All You Need" [5] paper that propsed pre-training in an incremental way, similar to how humans learn, produces novel results.
The implementation is just enough to run very small workloads:
* Full GPT2 small (124M parameters) model including byte pair encoding, embeddings, multi-headed attention, and multi-layer perceptron stages
* Inference/forward pass only (no training)
* Context is limited to 10 tokens in length
* 10 characters per word limit
* Zero temperature output only
This sheet is very big. Unfortunately, it is not unusual for Excel to lock up while using this spreadsheet. It is highly recommended to use the manual calculation mode in Excel.
29 comments
[ 4.5 ms ] story [ 73.3 ms ] threadI guess there are cases where training on a tests set is reasonable, but why use a LLM in that case?
(apparently, a lot of things are "all you need" !)
(or, in other words, "Is All You Need" Considered Harmful)
Imagining it was not clear enough, the author decided to name the model something 'pronounced “fictional”', and shows how it can answer questions that weren't even asked yet with perfect accuracy right on the abstract.
Accept that as an exercise of familiarizing yourself with the area's language. If you are interested on it, it's not lost time.
If it takes you reading the whole paper up to the point where the author explains the joke to realize that the paper tagged satire is satire, I mean, I'm not sure paper-reading is for you. There are unmarked papers out there that aren't even satirical, just wrong!
What's left, after you use, say, "half" of The Entire Internet for training? What's left that is definitely not the same as the half you used? How can you be sure, when you're working with terabytes or petabytes?
Credible benchmarking is going to have to become a field in itself.
Maybe it needs to be obfuscated, or all the questions should be obscene so they don't pass the content filtering.
I find this annoying, and not funny, honestly.
Why are satire papers mixed in with real ones on arxiv.org? Obviously, I'm not a research scholar and don't know how the whole process works, but as someone who likes to know whether they are dealing with fact or fiction from the start, this was a waste of time.
> Comments: 3 pages, satire
Tangentially, it's worth pointing out that you are gatekeeping read access to a resource because of a write access problem. If you don't want open access then go back to Elsevier.
I have no problem with open access, and would rather journals were required to be open access. But reading brand new results that haven't had any peer review without familiarity with the field is putting the cart before the horse. I certainly don't learn a new subject by reading the hottest papers on arxiv.
The problem that this "paper" is making fun of is that any model can score very well on these tests when the content of the tests themselves is used in the training of the models. Here is one example of this happening with a 13B model that scored way too high. [2]
In reality, when reviewing test scores, it's difficult to discern whether a model has been pre-trained on some specific content that is related to the test, or if the model has been trained in a way that produces novel emergent abilities not seen in other models. [3] This means that LeaderBoards and test scores for LLMs provide a low-confidence metric for what the propose to be testing.
The paper's title is itself a continuation of the phenomenon where papers are named after the landmark paper "Attention Is All You Need". [4] A good example of this is the "Textbooks Are All You Need" [5] paper that propsed pre-training in an incremental way, similar to how humans learn, produces novel results.
1. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
2. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
3. https://arxiv.org/abs/2206.07682
4. https://arxiv.org/abs/1706.03762
5. https://arxiv.org/abs/2306.11644
Full disclosure: I have an AI side project that also uses “is all you need” in the title.
The implementation is just enough to run very small workloads:
* Full GPT2 small (124M parameters) model including byte pair encoding, embeddings, multi-headed attention, and multi-layer perceptron stages
* Inference/forward pass only (no training)
* Context is limited to 10 tokens in length
* 10 characters per word limit
* Zero temperature output only
This sheet is very big. Unfortunately, it is not unusual for Excel to lock up while using this spreadsheet. It is highly recommended to use the manual calculation mode in Excel.
from: https://spreadsheets-are-all-you-need.ai/