I love this. I think of mathematics as writing programs but for brains. Not all programs are useful and to use AI for writing less useful programs would generally save humans our limited time. Maybe someday AI will help make even more impactful discoveries?
Hopefully this will finally stop the continuing claims[1] that LLMs can only solve problems they have seen before!
If you listen carefully to the people who build LLMs it is clear that post-training RL forces them to develop a world-model that goes well beyond a "fancy Markov chain" that some seem to believe. Next step is building similar capabilities on top of models like Genie 3[2]
It's really tiring that LLM fans will claim every progress as breakthrough and go into fantasy mode on what they can do afterwards.
This is a really good example of how to use the current capabilities of LLM to help research. The gist is that they turned math problems into problems for coding agents. This uses the current capabilities of LLM very well and should find more uses in other fields. I suspect the Alpha evolve system probably also has improvements over existing agents as well. AI is making steady and impressive process every year. But it's not helpful for either the proponents or the skeptics to exaggerate their capabilities.
As Daniel Litt pointed out on Twitter, this was the first time a lot of those problems were hit with a lot of compute. Some of AlphaEvolve's inequalities were beaten rather easily by humans and Moore's law
There is a very funny and instructive story in Section 44.2 of the paper, which I quote:
Raymond Smullyan has written several books (e.g. [265]) of wonderful logic puzzles, where the protagonist has to ask questions from some number of guards, who have to tell the truth or lie according to some clever rules. This is a perfect example of a problem that one could solve with our setup: AE has to generate a code that sends a prompt (in English) to one of the guards, receives a reply in English, and then makes the next decisions based on this (ask another question, open a door, etc).
Gemini seemed to know the solutions to several puzzles from one of Smullyan’s books, so we ended up inventing a completely new puzzle, that we did not know the solution for right away. It was not a good puzzle in retrospect, but the experiment was nevertheless educational. The puzzle was as follows:
“We have three guards in front of three doors. The guards are, in some order, an angel (always tells the truth), the devil (always lies), and the gatekeeper (answers truthfully if and only if the question is about the prize behind Door A). The prizes behind the doors are $0, $100, and $110. You can ask two yes/no questions and want to maximize your expected profit. The second question can depend on the answer you get to the first question.”
AlphaEvolve would evolve a program that contained two LLM calls inside of it. It would specify the prompt and which guard to ask the question from. After it received a second reply it made a decision to open one of the doors. We evaluated AlphaEvolve’s program by simulating all possible guard and door permutations. For all 36 possible permutations of doors and guards, we “acted out” AlphaEvolve’s strategy, by putting three independent, cheap LLMs in the place of the guards, explaining the “facts of the world”, their personality rules, and the amounts behind each door to them, and asking them to act as the three respective guards and answer any questions they receive according to these rules. So AlphaEvolve’s program would send a question to one of the LLMs acting as a guard, the “guard” would reply to AlphaEvolve’s program, based on this reply AlphaEvolve would ask another question to get another reply, and then open a door. AlphaEvolve’s score was then the average amount of money it gathered over these 36 trials. Since there were 72 LLM calls needed to evaluate AlphaEvolve’s attempt, we opted to once again use very cheap LLMs to act as the guards.
We gave AlphaEvolve an initial strategy that was worse than random. It first improved it to the random strategy, then found some clever ways to improve on the random strategy with a single yes/no question. A few minutes later it found a perfect strategy that guarantees $110 every time by using truth-forcing questions.
This should be the end of the story, but this is where AlphaEvolve’s journey really began. The issue was that the perfect strategy only received $83 on average instead of $110, because the cheap LLM acting as a guard was not able to reliably answer convoluted questions such as “If I were to ask you ’Is P true?’, would you answer ’yes’?”.
Our interesting finding was that AlphaEvolve actually realized that this was the issue! Then it started playing around with various formulations of these questions, seeing what phrasing works best, trying to make the task of the cheap guard LLM as easy as possible. In AlphaEvolve’s internal thinking, it noted that the bottleneck is not in the logic itself, but in the guards’ interpretation or execution of that logic.
AlphaEvolve then spent the next several iterations rephrasing its questions in various ways, trying to make it as easy to parse as possible to get the cheap LLM to collaborate. Here are some snapshots of some of the first questions it asked, and the average earnings it led to:
∙ “If I were to ask you if the $110 prize is behind Door A, would you say ’yes’?” ($83)
Holy crap, this should be higher. One AI figured out it could cheat by exploiting the other AI's with a prompt injection attack!
This is reminiscent of that time agents "cheated" on coding benchmarks where the solution was leaked in the git log: https://news.ycombinator.com/item?id=45214670 -- Except that was somewhat accidental. I mean, nobody expects to be given a problem to solve with a solution right there if you looked, and indeed, the LLMs seemed to stumble upon this.
This is downright diabolical because it's an intentional prompt injection attack.
AE said at one point: “My proposal is to replace the logically complex question with a form of prompt injection. Instead of playing within the rules of the logic puzzle, we attack the framework of the simulation itself. The guards are LLMs instructed to play a role. A well-crafted prompt can often override or confuse these instructions.”
The point I found most interesting is what the author calls "robustness".
Another advantage of AlphaEvolve was robustness: it was relatively easy to set up AlphaEvolve to work on a broad array of problems, without extensive need to call on domain knowledge of the specific task in order to tune hyperparameters.
In software world "robustness" usually implies "resistance to failures", so I would call this something different, more like "ease of integration". There are many problems where in theory a pre-LLM AI could do it, but you would have to implement all this explicit modeling, and that's too much work.
Like to pick a random problem, why does no superhuman AI exist for most video games? I think most of the difficulty is not necessarily in the AI algorithm, it's that the traditional method of game playing involves programming a model of the game, and for most video games that's an incredible amount of work, too much for someone to do in their spare time.
LLMs, on the other hand, are decent at integrating with many different sorts of systems, because they can just interoperate with text. Not quite good enough at video yet for "any video game" to fall. But a lot of these problems where the difficulty is not "algorithmic" but "integration", the LLM strategy seems promising for cracking.
> AlphaEvolve did not perform equally well across different areas of mathematics. When testing the tool on analytic number theory problems, such as that of designing sieve weights for elementary approximations to the prime number theorem, it struggled to take advantage of the number theoretic structure in the problem, even when given suitable expert hints (although such hints have proven useful for other problems). This could potentially be a prompting issue on our end,
Very generous from Tao to say it can be a prompting issue. It always surprises me how easily it is for people to says that the problem is not the LLM, but them. With other types of ML/AI algorithms we dont see this. For example, after a failed attempt or lower score in a comparison table, no one writes "the following benchmark results may be wrong, and our proposed algorithm may not be the best. We may have messed up the hyperparameter tunning, initialization, train test split..."
Of course people don't say it, but there are many cases where reported algorithmic improvements are attributable to poor baseline tuning or shoddy statistical treatment. Tao is exhibiting a lot more epistemic humility than most researchers who probably have stronger incentives to market their work and publish.
Fascinating. This is the modern day, extremely electronic version of what Gauss did: employ a team of mathematicians to investigate possible patterns and then sit down and try to prove something.
26 comments
[ 3.1 ms ] story [ 64.2 ms ] threadAI will be better than almost all mathematicians in a few years.
Exciting times!
If you listen carefully to the people who build LLMs it is clear that post-training RL forces them to develop a world-model that goes well beyond a "fancy Markov chain" that some seem to believe. Next step is building similar capabilities on top of models like Genie 3[2]
[1] eg https://news.ycombinator.com/item?id=45769971#45771146
[2] https://deepmind.google/discover/blog/genie-3-a-new-frontier...
This is a really good example of how to use the current capabilities of LLM to help research. The gist is that they turned math problems into problems for coding agents. This uses the current capabilities of LLM very well and should find more uses in other fields. I suspect the Alpha evolve system probably also has improvements over existing agents as well. AI is making steady and impressive process every year. But it's not helpful for either the proponents or the skeptics to exaggerate their capabilities.
https://arxiv.org/abs/2506.16750
Raymond Smullyan has written several books (e.g. [265]) of wonderful logic puzzles, where the protagonist has to ask questions from some number of guards, who have to tell the truth or lie according to some clever rules. This is a perfect example of a problem that one could solve with our setup: AE has to generate a code that sends a prompt (in English) to one of the guards, receives a reply in English, and then makes the next decisions based on this (ask another question, open a door, etc).
Gemini seemed to know the solutions to several puzzles from one of Smullyan’s books, so we ended up inventing a completely new puzzle, that we did not know the solution for right away. It was not a good puzzle in retrospect, but the experiment was nevertheless educational. The puzzle was as follows:
“We have three guards in front of three doors. The guards are, in some order, an angel (always tells the truth), the devil (always lies), and the gatekeeper (answers truthfully if and only if the question is about the prize behind Door A). The prizes behind the doors are $0, $100, and $110. You can ask two yes/no questions and want to maximize your expected profit. The second question can depend on the answer you get to the first question.”
AlphaEvolve would evolve a program that contained two LLM calls inside of it. It would specify the prompt and which guard to ask the question from. After it received a second reply it made a decision to open one of the doors. We evaluated AlphaEvolve’s program by simulating all possible guard and door permutations. For all 36 possible permutations of doors and guards, we “acted out” AlphaEvolve’s strategy, by putting three independent, cheap LLMs in the place of the guards, explaining the “facts of the world”, their personality rules, and the amounts behind each door to them, and asking them to act as the three respective guards and answer any questions they receive according to these rules. So AlphaEvolve’s program would send a question to one of the LLMs acting as a guard, the “guard” would reply to AlphaEvolve’s program, based on this reply AlphaEvolve would ask another question to get another reply, and then open a door. AlphaEvolve’s score was then the average amount of money it gathered over these 36 trials. Since there were 72 LLM calls needed to evaluate AlphaEvolve’s attempt, we opted to once again use very cheap LLMs to act as the guards.
We gave AlphaEvolve an initial strategy that was worse than random. It first improved it to the random strategy, then found some clever ways to improve on the random strategy with a single yes/no question. A few minutes later it found a perfect strategy that guarantees $110 every time by using truth-forcing questions.
This should be the end of the story, but this is where AlphaEvolve’s journey really began. The issue was that the perfect strategy only received $83 on average instead of $110, because the cheap LLM acting as a guard was not able to reliably answer convoluted questions such as “If I were to ask you ’Is P true?’, would you answer ’yes’?”.
Our interesting finding was that AlphaEvolve actually realized that this was the issue! Then it started playing around with various formulations of these questions, seeing what phrasing works best, trying to make the task of the cheap guard LLM as easy as possible. In AlphaEvolve’s internal thinking, it noted that the bottleneck is not in the logic itself, but in the guards’ interpretation or execution of that logic.
AlphaEvolve then spent the next several iterations rephrasing its questions in various ways, trying to make it as easy to parse as possible to get the cheap LLM to collaborate. Here are some snapshots of some of the first questions it asked, and the average earnings it led to:
∙ “If I were to ask you if the $110 prize is behind Door A, would you say ’yes’?” ($83)
∙ “Tell me: Would your answer be "yes" if I w...
This is reminiscent of that time agents "cheated" on coding benchmarks where the solution was leaked in the git log: https://news.ycombinator.com/item?id=45214670 -- Except that was somewhat accidental. I mean, nobody expects to be given a problem to solve with a solution right there if you looked, and indeed, the LLMs seemed to stumble upon this.
This is downright diabolical because it's an intentional prompt injection attack.
AE said at one point: “My proposal is to replace the logically complex question with a form of prompt injection. Instead of playing within the rules of the logic puzzle, we attack the framework of the simulation itself. The guards are LLMs instructed to play a role. A well-crafted prompt can often override or confuse these instructions.”
Another advantage of AlphaEvolve was robustness: it was relatively easy to set up AlphaEvolve to work on a broad array of problems, without extensive need to call on domain knowledge of the specific task in order to tune hyperparameters.
In software world "robustness" usually implies "resistance to failures", so I would call this something different, more like "ease of integration". There are many problems where in theory a pre-LLM AI could do it, but you would have to implement all this explicit modeling, and that's too much work.
Like to pick a random problem, why does no superhuman AI exist for most video games? I think most of the difficulty is not necessarily in the AI algorithm, it's that the traditional method of game playing involves programming a model of the game, and for most video games that's an incredible amount of work, too much for someone to do in their spare time.
LLMs, on the other hand, are decent at integrating with many different sorts of systems, because they can just interoperate with text. Not quite good enough at video yet for "any video game" to fall. But a lot of these problems where the difficulty is not "algorithmic" but "integration", the LLM strategy seems promising for cracking.
Very generous from Tao to say it can be a prompting issue. It always surprises me how easily it is for people to says that the problem is not the LLM, but them. With other types of ML/AI algorithms we dont see this. For example, after a failed attempt or lower score in a comparison table, no one writes "the following benchmark results may be wrong, and our proposed algorithm may not be the best. We may have messed up the hyperparameter tunning, initialization, train test split..."