The way human IQ testing developed is that researchers noticed people who excel in one cognitive task tend to do well in others - the “positive manifold.”
They then hypothesized a general factor, “g,” to explain this pattern. Early tests (e.g., Binet–Simon; later Stanford–Binet and Wechsler) sampled a wide range of tasks, and researchers used correlations and factor analysis to extract the common component, then norm it around 100 with a SD of 15 and call it IQ.
IQ tend to meaningfully predicts performance across some domains especially education and work, and shows high test–retest stability from late adolescence through adulthood. It is also tend to be consistent between high quality tests, despite a wide variety of testing methods.
It looks like this site just uses human rated public IQ tests. But it would have been more interesting if an IQ test was developed specifically for AI. I.e. a test that would aim to Factor out the strength of a model general cognitive ability across a wide variety of tasks. It is probably doable by doing principal component analysis on a large set of benchmarks available today.
IQ is a discovery about how intelligence occurs in humans. As you mentioned, a single factor explains most of the performance of a human on an IQ test, and that model is better than theories of multiple orthogonal intelligences. To contrast, 5 orthogonal factors are the best model we have for human personality.
The first question to ask is "do LLMs also have a general factor?". How much of an LLMs performance on an IQ test can be explained by a single positive correlation between all questions? I would expect LLMs to perform much better on memory tasks than anything else, and I wouldn't be surprised if that was holding up their scores. Is there a multi factor model that better explains LLM performance on these tests?
> The way human IQ testing developed is that researchers noticed people who excel in one cognitive task tend to do well in others
My son took an IQ test and it wouldn't score him because he breaks this assumption. He was getting 98% in some tasks and 2% in others. The psychologist giving him the test said it was unlikely enough pattern that they couldn't get an IQ result for him. He's been diagnosed with non-verbal learning disability, and this is apparently common for nvld folks.
There is probably a correlation between how fast a human can do math problems and how intelligent they are in general.
But a very trivial python program running on a normal computer will beat the fastest human at math problems in terms of speed. Even though it does nothing else useful
> The way human IQ testing developed is that researchers noticed people who excel in one cognitive task tend to do well in others - the “positive manifold.”
I'm pretty sure that this is not true, and that the tests were developed to measure children's intellectual development, and whether they were behind or ahead for their age. A bunch of people saw them and decided that it was far better than the primitive tests they had devised in an attempt to limit immigration from southern Europe, or to justify legal discrimination against black people, and wished a universal intelligence scalar into existence.
They justify this by saying that the results on this year's test correlate with the results of last years test. They are not laughed at. The thing it most correlates with is the value of your parent's car or cars.
If a model can get an IQ of 120, but can't draw clocks at a precise requested time, or properly count the b's in blueberry, can we then agree IQ tests don't measure intelligence?
One potential issue with that approach is the factors wouldn't stay very constant across generations of AI models.
While a lot of people have used various methods to try to gauge the strength of various AI models, one of my favorites is this time horizon analysis [1] which took coding tasks of various lengths and looked at how long it takes to humans to complete those tasks and compared that to chance that the AI would successfully complete the task. Then they looked at various threshholds to see how long of tasks an AI could generally complete with a certain percent threshold. They found the length of a task that AI is able to complete with a various threshholds is doubling about every 7 months.
The reason I found this to be an interesting approach is both because AI seems to struggling with coding tasks as the problem grows in complexity and also because being able to give it more complex tasks is an important metric both for coding tasks or more generally just asking AIs to act as independent agents. In my experience increasing the complexity of a problem has a much larger performance falloff for AI than in humans where the task would just take longer, so this approach makes a lot of intuitive sense to me.
I believe the ARC-AGI benchmark fits that description, it's sort of an IQ test for LLMs, though I would caution against using the word "Intelligence" for LLMs.
My take is that it’s easier to train a model to ace short, low-context tasks like IQ tests. That doesn’t necessarily transfer to more complex reasoning. While on the Mensa Norway test GPT-5 gets over 140, on an offline test it goes down to ~120.
It is interesting to look at the political spectrum as well (https://www.trackingai.org/political-test) - ar are liberals, even Grok 4.
The political leaning isn’t surprising either. Mainstream models need to be broadly acceptable, which in practice means being respectful of all groups. An authoritarian right-wing model might work for one country, group, or religion, but would almost certainly be offensive elsewhere.
This website's method doesn't work at all for humans the way it works for LLMs. For humans, there is a strict time limit on these IQ tests (at least in officially recognised settings like Mensa). This kind of sequence completion is mostly a question of how fast your brain can iterate on problems. Being able to solve more questions within the time limit means you get a higher score because your brain essentially switches faster. But for LLMs, they just give them all the time in the world in parallel and see how many questions they can solve at all. If you look at the examples, you'll see some high end models struggling with some the first questions, that most humans would normally get easily. Only the later ones get hard where you really have to think through multiple options. So a 100 IQ LLM in here is not technically more intelligent in IQ test questions than 50% of humans.
If anything, this shows that some LLMs might win against humans because they can spend more time thinking per wall clock time interval thanks to the underlying hardware. Not because they are fundamentally smarter.
The point of this is not so much to compare humans with AI. But to compare AI with other traditional software development approaches to solve this domain (IQ test, in this case). I believe, and I could be wrong, it will be nearly impossible, or too expensive, to develop deterministic software to beat AI in IQ test.
Mensa really needs to be left out of these discussions. It’s not scientific, it is just a money grab for people who need intellectual validation. You can be admitted with a top 10% SAT score and no in-person testing at all. The in-person testing is in three parts, one part is a memory test, the second part is a Mensa test, the third part is the Weschler test. Source: I joined in 1995 because I needed intellectual validation. :)
AI has a 140 “IQ” but understands nothing. That’s because AI does not understand anything: it just predicts the next token based on previous tokens and statistics. AI can give me five synonyms for any Latin word, because that’s just statistics, and it can regurgitate rules about metrical length of syllables, but it can’t give me synonyms matching a particular metrical pattern, because that would involve applying knowledge. If I challenge its wrong answer, it will apologize and give me further wrong answers that are wrong in the same way, because it cannot learn.
That critique is half-right: large language models don’t “understand” in the human sense, but they do apply learned patterns across vast data in ways that often look like knowledge, even if it’s statistical pattern-matching. The real frontier is that these statistical engines can already combine rules, constraints, and creativity in ways their critics dismiss too quickly—making the line between “mere prediction” and “applied knowledge” fuzzier than it seems.
> AI can give me five synonyms for any Latin word, because that’s just statistics, and it can regurgitate rules about metrical length of syllables, but it can’t give me synonyms matching a particular metrical pattern,
I love when people say that AI can't do things that it can do.
me: give me a synonym for overconfident that is a dactyl
GPT5: A good dactylic synonym for overconfident is arrogant.
It scans as Ár-ro-gant (stressed, unstressed, unstressed), which makes it a perfect dactyl.
Other possible dactyls, though a bit less direct, include:
Ignorant (if you want to imply careless overconfidence)
Insolent (if you want a sharper, defiant edge)
Want me to give you a list of dactylic synonyms with slightly different shades of meaning (e.g., boastful, careless, reckless)?
Even assuming that companies prune out authoritarianism from their models for whatever reason, surely we'd expect at least one of them to drift over into mild economic right-wing territory. It'd be interesting to know what is causing that bias.
> It'd be interesting to know what is causing that bias.
I'd imagine it's the same kind of "bias" against hitler that you get by reading any history book. Or that makes the overwhelming majority of scientists liberal or leftist. Just look at what happened when Musk made Grok promote the south african white genocide nonsense. It was evidently not the "make_model_liberal.parquet“ dataset he had to just turn off, but actively mucking with it.
Unless they asked the same question multiple times and verified that the AI always gets the right answer, this is a very faulty result.
Even looking at the reasoning, in a majority of the cases you cannot prove that the LLM got it right because it actually found the right pattern instead of on a fluke.
Here's an example reasoning that got the right answer but that is not specific enough and therefore could apply to literally any answer (model is Bing Copilot, picked randomly):
> Option D : A shape resembling a clock. The clock shows the time 9:00.* The pattern involves shifting times across rows and columns in a logical progression. Observing the sequence in the third row, where the first two clocks show times moving forward in increments, the next logical step is a clock displaying 9:00 to fit the established rhythm. This ensures symmetry and continuity within the overall grid.
Here's a comparison to "OpenAI o4 mini high" which is a very specific answer and shows it got the logic of the puzzle correctly:
> D Each row adds +1:30, then +3:00. - Row 1: 12:00 → 1:30 (+1:30), 1:30 → 4:30 (+3:00) - Row 2: 3:00 → 4:30 (+1:30), 4:30 → 7:30 (+3:00) - Row 3: 4:30 → 6:00 (+1:30), so 6:00 → *9:00* (+3:00) (Down each column it’s +3:00 then +1:30, which also fits.)
That applies to humans too. If each question has 6 options, you can assume that everyone will get 16.6% for free and compensate in the grading criteria.
Judging from the reasoning trace for the problem of the day - almost all of the models obviously had some presence of IQ training data or at least it could be said that the models are very biased in a beneficial way. From the beginning of the trace you kinda see that the model had already "figured it out" - the reasoning is done only for applying the basic arithmetics.
None of the models did actually "reason" about what the problem could possibly be - like none of them considered that more intricate patterns are possible in a 3x3 grid (having taken this kinds of test earlier in life, I still had a few seconds of indecision, thinking whether this is the same kind of test that I've seen and not some more elaborate one), and none of them tried solving the problem column-wise (it is still possible by the way) - personally, I think that indicates a strong bias present in the pretraining. For what it's worth, I would consider a model that would come up with at least a few different interpretations of the pattern while "reasoning" to be the most intelligent one - irrespective of the correctness of the answer.
Human IQ is norm-referenced psychometrics under embodied noise. Calling both “IQ” isn’t harmless, it invites bad policy and building decisions on a false equivalence. Don’t promote it.
Human beings' IQ test results can vary significantly based on how much money is in their pockets. For example, if a farmer takes an IQ test before crops are harvested and sold they score lower than after crops are sold, in the same year.
It seems fairly obvious to me that an LLM is the projection intelligence in the language domain. In other words, if you killed Intelligence and gave it a push in direction of language, the chalk outline you could draw around its dead body on the ground would be an LLM.
Full disclosure: I have taken 2 IQ tests, both online and timed. First was in late 90's after graduated electonics eng. was free, scored 149. After 4 years and obtaining theoretical physics degree, I did another scoring 169. The second test was not free, but I did not pay. I got the second test results because the test site owner personally emailed me my results for free with congrads, because they were the highest ever recorded on the site to date. I did both for fun just see the questions, I think both results are meaningless, the same variability occurs on farmers studied as mentioned above.
It's interesting there is a such a large spread for ChatGPT-5 Quite low for "5 Thinking" and high for "5 PRO Vision". They are probably trying to control their compute and energy costs by switching people to the simpler models where they can.
Right now LLMs are like students that study for years, then get their brains frozen into a textbook before they’re released. They can read new stuff during use (context window), but they don’t actually update their core weights on the fly. The “infinite context window” dream would mean every interaction is remembered and folded back into the brain, seamlessly blending inference (using the model) with training (reshaping it).
Within 2–3 years, we’ll see practical “personal LLMs” with effectively infinite memory via retrieval + lightweight updates, feeling continuous but not actually rewriting the core brain.
Within 5–10 years, we’ll likely get true continual-learning systems that can safely update weights live, with mechanisms to prune bad habits and compress knowledge—closer to how a human learns daily.
The rub is less can we and more should we: infinite memory + unfiltered feedback loops risks building a paranoid mirror that learns every user’s quirks, errors, and biases as gospel. In other words, your personal live-updating LLM might become your eccentric twin.
> Note: VERBAL models are asked using the verbalized test prompt. VISION models are asked the test image instead without any text prompts.
Just glancing at the bar graphs, the vision models mostly suck across the board for each question. Whereas verbal ones do OK.
And today's example of clock faces (#17) does a good job of demonstrating why: because when a lot of the diagrams are explained verbally, it makes it significantly easier to solve.
Maybe it's just me, but #17 for example - it's not immediately obvious those are even supposed to represent clocks, and yet the verbal prompt turns each one into clock times for the model (e.g. 1:30) which feels like 50% of the problem being solved before the model does anything at all.
It also would be interesting the other way around: a capable LLM engages in a discussion in order to determine the IQ of the person it's speaking with.
72 comments
[ 4.3 ms ] story [ 100 ms ] threadhttps://news.ycombinator.com/newsguidelines.html
They then hypothesized a general factor, “g,” to explain this pattern. Early tests (e.g., Binet–Simon; later Stanford–Binet and Wechsler) sampled a wide range of tasks, and researchers used correlations and factor analysis to extract the common component, then norm it around 100 with a SD of 15 and call it IQ.
IQ tend to meaningfully predicts performance across some domains especially education and work, and shows high test–retest stability from late adolescence through adulthood. It is also tend to be consistent between high quality tests, despite a wide variety of testing methods.
It looks like this site just uses human rated public IQ tests. But it would have been more interesting if an IQ test was developed specifically for AI. I.e. a test that would aim to Factor out the strength of a model general cognitive ability across a wide variety of tasks. It is probably doable by doing principal component analysis on a large set of benchmarks available today.
The first question to ask is "do LLMs also have a general factor?". How much of an LLMs performance on an IQ test can be explained by a single positive correlation between all questions? I would expect LLMs to perform much better on memory tasks than anything else, and I wouldn't be surprised if that was holding up their scores. Is there a multi factor model that better explains LLM performance on these tests?
My son took an IQ test and it wouldn't score him because he breaks this assumption. He was getting 98% in some tasks and 2% in others. The psychologist giving him the test said it was unlikely enough pattern that they couldn't get an IQ result for him. He's been diagnosed with non-verbal learning disability, and this is apparently common for nvld folks.
There is probably a correlation between how fast a human can do math problems and how intelligent they are in general.
But a very trivial python program running on a normal computer will beat the fastest human at math problems in terms of speed. Even though it does nothing else useful
I'm pretty sure that this is not true, and that the tests were developed to measure children's intellectual development, and whether they were behind or ahead for their age. A bunch of people saw them and decided that it was far better than the primitive tests they had devised in an attempt to limit immigration from southern Europe, or to justify legal discrimination against black people, and wished a universal intelligence scalar into existence.
They justify this by saying that the results on this year's test correlate with the results of last years test. They are not laughed at. The thing it most correlates with is the value of your parent's car or cars.
Isn’t that basically what the ARC tests are?
While a lot of people have used various methods to try to gauge the strength of various AI models, one of my favorites is this time horizon analysis [1] which took coding tasks of various lengths and looked at how long it takes to humans to complete those tasks and compared that to chance that the AI would successfully complete the task. Then they looked at various threshholds to see how long of tasks an AI could generally complete with a certain percent threshold. They found the length of a task that AI is able to complete with a various threshholds is doubling about every 7 months.
The reason I found this to be an interesting approach is both because AI seems to struggling with coding tasks as the problem grows in complexity and also because being able to give it more complex tasks is an important metric both for coding tasks or more generally just asking AIs to act as independent agents. In my experience increasing the complexity of a problem has a much larger performance falloff for AI than in humans where the task would just take longer, so this approach makes a lot of intuitive sense to me.
[1] - https://theaidigest.org/time-horizons
It is interesting to look at the political spectrum as well (https://www.trackingai.org/political-test) - ar are liberals, even Grok 4. The political leaning isn’t surprising either. Mainstream models need to be broadly acceptable, which in practice means being respectful of all groups. An authoritarian right-wing model might work for one country, group, or religion, but would almost certainly be offensive elsewhere.
This website's method doesn't work at all for humans the way it works for LLMs. For humans, there is a strict time limit on these IQ tests (at least in officially recognised settings like Mensa). This kind of sequence completion is mostly a question of how fast your brain can iterate on problems. Being able to solve more questions within the time limit means you get a higher score because your brain essentially switches faster. But for LLMs, they just give them all the time in the world in parallel and see how many questions they can solve at all. If you look at the examples, you'll see some high end models struggling with some the first questions, that most humans would normally get easily. Only the later ones get hard where you really have to think through multiple options. So a 100 IQ LLM in here is not technically more intelligent in IQ test questions than 50% of humans.
If anything, this shows that some LLMs might win against humans because they can spend more time thinking per wall clock time interval thanks to the underlying hardware. Not because they are fundamentally smarter.
As opposed to what you were doing when you wrote that.
There are probably millions of humans that fit this criteria too.
I love when people say that AI can't do things that it can do.
me: give me a synonym for overconfident that is a dactyl
GPT5: A good dactylic synonym for overconfident is arrogant.
It scans as Ár-ro-gant (stressed, unstressed, unstressed), which makes it a perfect dactyl.
Other possible dactyls, though a bit less direct, include:
Ignorant (if you want to imply careless overconfidence)
Insolent (if you want a sharper, defiant edge)
Want me to give you a list of dactylic synonyms with slightly different shades of meaning (e.g., boastful, careless, reckless)?
Even assuming that companies prune out authoritarianism from their models for whatever reason, surely we'd expect at least one of them to drift over into mild economic right-wing territory. It'd be interesting to know what is causing that bias.
I'd imagine it's the same kind of "bias" against hitler that you get by reading any history book. Or that makes the overwhelming majority of scientists liberal or leftist. Just look at what happened when Musk made Grok promote the south african white genocide nonsense. It was evidently not the "make_model_liberal.parquet“ dataset he had to just turn off, but actively mucking with it.
https://www.theatlantic.com/technology/archive/2025/05/elon-...
Even looking at the reasoning, in a majority of the cases you cannot prove that the LLM got it right because it actually found the right pattern instead of on a fluke.
Here's an example reasoning that got the right answer but that is not specific enough and therefore could apply to literally any answer (model is Bing Copilot, picked randomly):
> Option D : A shape resembling a clock. The clock shows the time 9:00.* The pattern involves shifting times across rows and columns in a logical progression. Observing the sequence in the third row, where the first two clocks show times moving forward in increments, the next logical step is a clock displaying 9:00 to fit the established rhythm. This ensures symmetry and continuity within the overall grid.
Here's a comparison to "OpenAI o4 mini high" which is a very specific answer and shows it got the logic of the puzzle correctly:
> D Each row adds +1:30, then +3:00. - Row 1: 12:00 → 1:30 (+1:30), 1:30 → 4:30 (+3:00) - Row 2: 3:00 → 4:30 (+1:30), 4:30 → 7:30 (+3:00) - Row 3: 4:30 → 6:00 (+1:30), so 6:00 → *9:00* (+3:00) (Down each column it’s +3:00 then +1:30, which also fits.)
None of the models did actually "reason" about what the problem could possibly be - like none of them considered that more intricate patterns are possible in a 3x3 grid (having taken this kinds of test earlier in life, I still had a few seconds of indecision, thinking whether this is the same kind of test that I've seen and not some more elaborate one), and none of them tried solving the problem column-wise (it is still possible by the way) - personally, I think that indicates a strong bias present in the pretraining. For what it's worth, I would consider a model that would come up with at least a few different interpretations of the pattern while "reasoning" to be the most intelligent one - irrespective of the correctness of the answer.
Human IQ is norm-referenced psychometrics under embodied noise. Calling both “IQ” isn’t harmless, it invites bad policy and building decisions on a false equivalence. Don’t promote it.
https://emiruz.com/post/2020-12-01-iq-rabbit-hole/
It seems fairly obvious to me that an LLM is the projection intelligence in the language domain. In other words, if you killed Intelligence and gave it a push in direction of language, the chalk outline you could draw around its dead body on the ground would be an LLM.
Full disclosure: I have taken 2 IQ tests, both online and timed. First was in late 90's after graduated electonics eng. was free, scored 149. After 4 years and obtaining theoretical physics degree, I did another scoring 169. The second test was not free, but I did not pay. I got the second test results because the test site owner personally emailed me my results for free with congrads, because they were the highest ever recorded on the site to date. I did both for fun just see the questions, I think both results are meaningless, the same variability occurs on farmers studied as mentioned above.
But that's the first IQ test.
https://www.johnstossel.com/ai-is-coming-for-your-job/
All standard talking points on one page (he forgot lump of labor though). So I'd take this test with a big grain of salt.
Isn't this like saying that a spellchecker is "very smart" because it did well at a spelling bee? It isn't, it just has a list of answers.
Within 2–3 years, we’ll see practical “personal LLMs” with effectively infinite memory via retrieval + lightweight updates, feeling continuous but not actually rewriting the core brain.
Within 5–10 years, we’ll likely get true continual-learning systems that can safely update weights live, with mechanisms to prune bad habits and compress knowledge—closer to how a human learns daily.
The rub is less can we and more should we: infinite memory + unfiltered feedback loops risks building a paranoid mirror that learns every user’s quirks, errors, and biases as gospel. In other words, your personal live-updating LLM might become your eccentric twin.
Just glancing at the bar graphs, the vision models mostly suck across the board for each question. Whereas verbal ones do OK.
And today's example of clock faces (#17) does a good job of demonstrating why: because when a lot of the diagrams are explained verbally, it makes it significantly easier to solve.
Maybe it's just me, but #17 for example - it's not immediately obvious those are even supposed to represent clocks, and yet the verbal prompt turns each one into clock times for the model (e.g. 1:30) which feels like 50% of the problem being solved before the model does anything at all.