I wonder how we could formalize the notion of „interesting“ problems in a way that would allow us to automatically generate new interesting questions from the existing corpus of mathematics.
This is wildly shortsighted view of mathematics. Historically many of the subfields of math which are presently most valuable were considered useless for decades or centuries. Number theory, non-Euclidean geometry, group theory, and Boolean algebra, to name a few.
This is false dichotomy. Many mathematical achievements were produced while looking at practical problems: Fourier is a good one. For every mathematician that accidentally made a practical contribution, there are many more that produced nothing of value, and would have with a better tuned reward function.
A simple way is to just measure how long it takes to solve. If it takes more than 5 minutes then we don't have a good understanding of that area and its worth potentially investing into.
My understanding is that top tier math programs don't hire based on journal pubs and haven't for a bit. Math journals are pretty slow and hiring is really via reputation building based on one or two big wow ideas / proofs propogated via arxiv and talks.
He is right if model intelligence stalls. If model intelligence continues to improve soon there's no need for the prompter to understand anything or for any workshop as a mathematician will just be able to ask the model to explain how the proof works and models will do a good job at walking them through it step by step.
There will be no gap in understanding. Now there is because the models are discovering things at the edge of what they can do and so suck at explaining it.
That doesn't make sense to me. Terence Tao is great at explaining things, but he could spend months explaining some of his proofs to me without me understanding it. Even if the AI had a superhuman ability to explain things it's no guarantee that it could make a human understand.
It couldn't make every human understand. But it could make Tao and other mathematicians understand. You're not going to become knowledgeable at everything suddenly, even though I doubt what you say. Many months of 1-1 tutoring and hard work from a student with a top mathematician would make you understand a lot. Just not sit down and read it first day.
I think we can say by now that we should not listen to the early nay-sayers and just wait a bit. With every trend, not just "AI". They still have some points (the ethics and environment etc), but the we don't hear from the Stochastic Parrot folks anymore.
Of course it's good to have the discussion... So maybe, we listen to the nay-sayers, but defer judgement on the matter... That's wisdom.
Even when a problem got solved, there has always been value in publishing simpler proofs and corollaries that give better intuition into the broader field.
If I understand Tao correctly, he's saying that's going to have to be the focus going forward. I just default to thinking the models are going to be much better than us at that, too.
> the models are going to be much better than us at that, too.
I wonder if this is true. The code produced by these models are not really getting any more elegant over time. On contrary, the models seem to be getting worse, often proposing really baroque architectures. You can use RL to optimize for correctness, optimizing the vibe seems much more difficult.
For deeper complexity, elegance may turn out to be a hindrance. Better or worse may turn out to be descriptive of taste and nothing else. On the other hand, elegant alternatives might be brought into reach by this first messy contact with new ideas. Who can say at this stage?
This is a distilled version of what people say about the tech industry in the past year or so. Replace math with any field, and the statement is still relevant.
I don't think it's that simple. I divide AI-impacted fields into three buckets:
1. Some present a unified line that the whole point of their craft is the human experience, and that automation is the antithesis of that. Marathon runners don't care that a car can get there faster, poets don't care that Poem Bot 2000 can write poems too. I think this is smart if you can credibly take this position. The difficulty is mostly convincing the buy side, which requires being very outspoken about your views.
2. Some appear to be undecided, with one faction taking the pro-human stance and another rushing to accelerate things with AI. A good example of this is mathematics, and I really wonder where they end up in the long haul. They have a very good claim on #1, because mathematics is pretty close to an art form and is robustly insulated from the pressures of the marketplace. But they can also choose option #3, below.
3. Some crafts prioritize results above all else, practitioners either rushing to extract as much money as possible before it all collapses, or believing that they can out-prompt everyone else forever and that their prompting skills are indispensable to their employers in the long haul. That's software engineering. I think this is going to be interesting to watch.
Seems like neat buckets but you may want to find one example for the first which can actually be done by an llm as running is not its strong suit afaik
I think your second bucket would need to be changed, as it's not really undecided for some and Tao in fact argues (in a way) that there is no need for the field to decide. There is a need for results, but there is also a need for humans to fully understand why the results are the way they are and how they were obtained. This doesn't replace the human in the loop it's more of a cooperative effort.
> This doesn't replace the human in the loop it's more of a cooperative effort.
With respect to Tao, I think this is a very naive / short-sighted perspective. It comes down to aesthetics of your trade. You can argue that there's beauty in humans doing it. Or you can argue that there isn't, in which case, it becomes a race to the bottom as models get better and the value of your contribution becomes more and more dubious.
I don't think there's a functional difference between that and a coder saying "you see, vibecoding is a synergistic relationship, they will always need a human in the loop". Or an artist saying "yeah, I'm using AI to generate artwork, but they will always need my sense of taste". They won't. That's just a signal for labs to fine-tune for the thing you identified as your special insight.
Extrapolating from the list, the "Fully AI" camp is going to push the pedal to the metal, investing time and effort into maximizing their fusion with this technology. Mentally enhanced and amplified 24/7, and likely physically too, fusing their biology with hardware and robotics. Within a few years, not even a generation, they will become unrecognizable to the "Fully Human" category.
I don't think it's smart at all to position yourself in bucket 1.
There's a reason there are maybe a few hundred professional marathon runners in the world vs tens of thousands of professional mathematicians.
Bucket 1 is basically an "amusement for the upper classes" type of deal. Any field that goes in that direction would have to shrink down massively.
It also devalues the field in my opinion from something really profound with actual impact in the world to a somewhat vain leisure activity. (basically going back to gentleman scientists) But I know other people would see it exactly the opposite way.
> There's a reason there are maybe a few hundred professional marathon runners in the world vs tens of thousands of professional mathematicians.
I don't buy this. I think there are other reasons why "professional marathon running" is a niche thing; it's probably just that it's not all that interesting to most people to practice in function of what it demands of your body, and not that interesting to patronize / watch.
Take woodworkers. There's probably more professional woodworkers than professional mathematicians. Everything a woodworker does, a machine can do more cheaply and more quickly, and the craft was definitely impacted industrialization. But the reason it survives is more or less because some craftsmen convinced themselves that there's merit in doing it "the old way", and they managed to convince a decent percentage of customers of the same thing.
And you could argue the same thing you did for marathon runners: custom furniture is just amusement for well-off people. Sure, but there's enough people with money to keep it afloat.
A very balanced perspective, and the concerns he raises are reasonable. He acknowledges that AI is going to transform mathematics, but simply dumping proofs on the math community and expecting others to do the grunt work of verifying, refining, and expanding on them is hardly a productive way to advance the field.
There seems to be more interest in hitting some arbitrary benchmark (we proved X unsolved problems) than in genuinely contributing to mathematics. But what else is to be expected? It's become a maniacal race with too much money. Too much effort is being invested in proving that the exponential curve is still holding.
That seems short sighted though. A few years ago models couldn't do this at all, I'm not sure there's any evidence to suggest exploring and refining results is outside their capabilities or will remain so.
OAI obviously have a fiscal incentive here, but to presume a year from now we won't see improvements and more succinct work on the results coming from models?
The problem is that one a person writes a 60 page proof in theory that person has spent an inordinate amount of time on the proof and can answer questions, describe some insight, etc etc.
If a random person is given a 60 page proof to digest and not the author, those hidden insights that _aren't_ in the paper might be completely inaccessible. Maybe the AI will "just" be able to provide the insights. Maybe. But pedagogy is tricky work, and despite these AIs being able to do all this fancy math we can't get them to write good cover letters yet, so....
Ultimately we might be left with just a bunch of intellectually unsatisfying proofs. This means way less drive to simplify the proofs or rework them.
End result: we generate a layer of "less efficient" mathematics, that won't get built upon. We will not actually have any shoulders upon which to stand.
But why should process of discovering mathematical insights be any less attainable to AI models?
The concern is being raised without evidence, because the evidence points to the gap simply being frontier models have just started to be able to get a raw proof out. Why, given existing progress, should we expect them to be unable to distill insights from those proofs?
Certainly this even more likely doesn't matter at all for applications: if I can send a radio signal further because my AIs design it a certain way, that's an unambiguous result. Which is really the next step here: turn a proof into a "mechanical" application.
According to Scott Aaronsson, OpenAI set their agents on 8000 different problems, and got 372 final proofs. Even spending twice the original effort on simplifying and reworking those proofs so that they do not "feel like something written by someone who’s on psychedelics" would only increase the compute by less than 10% (assuming all the agents had a similar token budget).
The fact that they did not do so can only mean that either (1) their agents currently lack the capability to do it, or (2) OpenAI are completely indifferent and do not care in the slightest if the proofs are understood or not.
> The fact that they did not do so can only mean that either (1) their agents currently lack the capability to do it, or (2) OpenAI are completely indifferent and do not care in the slightest if the proofs are understood or not.
Come now, this is kind of unreasonable. When you're working on a new technology, you first get the ugly, inconvenient-to-use prototypes functioning with the core new thing you need; then you work on packaging it up into a format useable in production. I'm sure the very first digital camera sensors weren't very useful for photographers either; but it isn't really even possible to build the rest of the technology required to turn raw output of a digital sensor into something a professional photographer can use until you have the raw output itself.
The research is still on going on the raw output; getting things to the next stage, where the results are widely useable by professional mathematicians (and then on to engineers and scientists to whom the results would be practically useful), is a whole new research area.
Seems like you are just subscribing to the first option I gave, "their agents currently lack the capability to do it", but saying you think they will be more capable in the future if they can move away from the inconvenient-to-use prototypes after more research. Thinking it might be possible in the future is not in disagreement with anything I said, so I am not sure what you thought was unreasonable about my description.
Imagine someone looking at digital camera researchers showcasing a ground-breaking new sensor, and reacting by saying "Well obviously they're completely indifferent and do not care in the slightest if their work is used by real photographers or not."
Like, "Orr... maybe they care a lot, but haven't gotten to that part yet?"
You are conflating the two options with each other (lack of capability vs indifference). One of them is true, not necessarily both ("or", not "and"). I did not claim a lack of capability was the same as indifference.
They claim each result used three hours of compute on average. Even spending significantly more on simplifying and reworking would delay the release with a single day at most. If avoiding such a minor delay took precedence over (2), I think "indifference" is the correct term. It has also been a while since the release now, so there is ample opportunity to post follow ups if time pressure was the only concern.
Why do that when they can spend more hours extending QRH to a proof of the full Riemann Hypothesis? The opportunity cost of chasing breadcrumbs is the whole enchilada.
Right, easy comparison to make the the open source community for software.
And it's not like this is something where we're loaned some top math genius for a limited amount of time and we have to make the most of it. Rather, this is a new high water mark. The accessibility of the results is no longer scarce. The scarcity has shifted, and that's where the focus of the math ecosystem should shift as well. And it doesn't help for a frontier community to saturate and take over messaging pipelines that were typically managed by the math ecosystem. It's not about "stay in your lane" but rather "we need coherence and be careful not to break the system."
Just two cents from someone who could screw up basic cashier math on any given day.
OpenAI has kicked things over. Dealing with that is the math community's problem. OpenAI had absolutely no obligation to defer to that community's deficiencies. If this has upset their reward system and historical traditions, tough luck.
Those few thousand mathematicians are now getting a taste of their own medicine. After all, it was people with extraordinary mathematical talent who developed machine learning and large language models, leaving hundreds of millions of people who earn their living through speaking, writing, or teaching worried about their future job prospects.
Still, I believe almost everyone will be fine. Perhaps AI will also prove good at coming up with new conjectures, and some mathematicians may shift towards applied mathematics or other sciences.
Proofs are valuable but I believe the mathematical community values understanding more. Proofs were previously a great way to develop understanding. Now, less so.
I recommend RTFA, it explains exactly that: why proofs should not be considered the goalpost, and why dumping all these AI-generated proofs might be an overall negative for mathematics as a whole.
Proofs of open problems are valuable because we are assuming that proving the problem requires some new method or infrastructure in math to prove it. Basically proving open problems isn't actually useful if it doesn't develop new tooling for mathematics, which can help us create new open problems, solve other ones etc.
This argument implicitly makes a few assumptions which will probably not hold in the very near future.
One is that AI will continue hallucinating in a manner that is not easy to verify, second is that AI will not be enhanced to produced more simplified amd robust outputs, and third that a human will be required to do that.
What humans in the loop are doing now is verify the process, propose shortcuts and add legitimacy, through the verification process, if that ends up being succesful its highly likely a lot less mathematicians will be required in the future.
The conclusion that this is not productive focuses on the mathematicians, but it is very productive in terms of hundreds of proofs being produced that had previously consumed uncountable hours of the brightest minds. Unless it ends up being the greatest hallucination ever ofcourse
> One is that AI will continue hallucinating in a manner that is not easy to verify, second is that AI will not be enhanced to produced more simplified amd robust outputs, and third that a human will be required to do that.
There is literally not a single shred of evidence to indicate either of your supposed eventualities. The core technology of an LLM is sampling from a distribution so there is literally no way to make it deterministically robust (only probabilistically).
The direction and pace of capability improvement has already been demonstrated by all models. The latest breakthroughs make that pretty evident, but there have been production systems that are based on probability since the beginning of computing.
What has been demonstrated is a process that outputs lean proofs based on those probabilities. This happened after decades markov chain producing garbled texts and very shortly after gpt2 producing stories about unicorns.
> The core technology of an LLM is sampling from a distribution so there is literally no way to make it deterministically robust (only probabilistically).
An LLM mostly deterministically (except parallel processing nondeterminism that can be mitigated) produces a probability distribution that can be sampled deterministically: just take the highest probability token or use beam search.
Deterministic yes, robust deterministic no. The most likely conjunction is not always the best nor representative of what the model is considering unless it's certainty is high.
I think people on here tend to somewhat fixate on the determinism issue. Even with a deterministic LLM - stabilising the floating point arithmetic, and choosing from the distribution by a fixed method, or just save the random seeds - there is still a kind of a chaotic unpredictability that can exist between its inputs and outputs. However maybe that is a price that needs to be paid to get creativity.
No evidence other than the fact that this has been happening steadily in all areas for many years?
You might have a point if the goal was to have LLMs that spit out a correct proof without chain of thought or tool use. LLMs + agent harnesses are more than capable of self verification and course correction.
> assumptions which will probably not hold in the very near future [...] One is that AI will continue hallucinating in a manner that is not easy to verify
Hold up, that's an even bigger assumption in the opposite direction, and I don't see anything to support it.
At least in terms LLMs getting all the "AI" hype these days, there is no structural/mathematical reason to believe they won't continue to have the same problem they've always had of generating plausible text over rational text, and I don't think anybody even has a clear idea how it could eventually be accomplished.
I've seen "then the magic singularity occurs and somehow it solves the problem for itself", but I would classify that more as mysticism than engineering.
"Plausible" text was preferred over rational text when we trained LLMs using RLHF. It's rapidly shifting the other way now with RLVR, which enforces correctness by default.
Hallucinations are no longer much of a practical problem in software engineering.
Two years ago, hallucinating that the code worked or that a task was accomplished was a common occurrence.
We have seen that now agent swarms across thousands of agents can coordinate to achieve a result.
Clearly hallucinations are no longer the problem they once were, since now we can get working results for long horizon tasks that require massive compute.
Consequently it would seem unwise to assume that current limitations will remain as they are and prevent LLMs from coming up with solutions that they can explain to humans.
The confidence with which you, anonymous user, keep commenting that "hallucination is not much of a practical problem in software engineering anymore" based solely on your own anecdotal evidence is really remarkable, in not a good way.
I find hallucinations in my (mostly perfect) AI output every single day. If you're not finding them, you're just not looking hard enough. It's not surprising when everyone is screaming about how they don't read code these days.
This is just a fact. I'm sorry if it messes with your narrative.
The sum total of all human observations is still not proof of the lack of hallucinations as a problem (even if their observations were perfect, which they aren't considering the volume produced vs reviewed carefully). That's why you can use a counter example only to disprove and not prove anything.
And yeah I get hallucinations all the time still. Maybe it's because I'm working on harder/more niche problems (like a compiler with an unusual type system), but it happens quite a lot. I don't record all of them.
Although the most common one you can find is them misattributing the source of changes from themselves and also other agents (Fable, Opus 5.5, deepseek, whatever). They'll say "your changes" or "you changed" or "your ruling." I didn't decide anything and it's in their own chat log, and yet...
> One is that AI will continue hallucinating in a manner that is not easy to verify
It is an old saw at this point, but what an LLM does still cannot be divided into hallucination and non-hallucination. This is literally an anthropomorphism trap.
Layers and layers of application-specific verification can reduce the risks inherent to LLMs, to a really remarkable degree, but nothing about what these tools are suggests that this problem will go away; it will just bubble up again somewhere else.
For all that I saw over the last few hundred hours with AI on software engineering, hallucinations are no longer a problem at all.
Not once have I seen a task fail due to what would have been a "hallucination". If they still occur, they can apparently be detected and corrected automatically, or are subtle enough to escape notice with presumably no significant impact on the results.
Why would this not also be the case for mathematics?
I think OP is saying that hallucination or not is just semantics. There is nothing qualitatively different about hallucinated vs non-hallucinated output.
That's true in the same sense as "There is nothing qualitatively different about erroneous vs non-erroneous output" for a dog vs. cat image classifier.
If it makes a false claim, then it's an error. If it says the test was passed or a class was implemented but it was not, then it makes a false factual statement.
I'd say a hallucination (very misleading word) or confabulation or "making shit up" happens when an LLM uses factual / evidential language purely based on local statistical expectations of the text, instead of it drawing from actual evidence in its context pointing to it.
This is murkier in the case of general knowledge questions, like when and where was some famous person born. It may then be a spectrum from fully making something up based on how the name sounds, all the way to confidently retrieving it from its weights correctly. In between, we can get hallucinations. But newer models are taught to use Web Search when unsure, and it works pretty well, though not perfectly. I don't see any fundamental limit here. It's just not perfect. Trying to solve "the hallucination problem" is basically like saying "our dog vs. cat classifier is pretty good already with its 99% accuracy, now all we need to do is the tiny little task of eliminating the 1% error, and we will be golden". Like, no shit, there is some error yes. People are working to reduce it. It will never be absolutely 100%. It's not an insight to say we should remove hallucinations.
Something about that analogy bothers me, and I think it's that we've lose track of the fundamental problem with our choice of "hallucination" framing: It falsely assumes "true" thought/sight is occurring the rest of the time.
These models are outputting language that we can decode as factual claims and we can check those, and we can say it made a mistake / error or that it answered correctly. This doesn't require any squishy assertions to how it "feels" while doing it or anything like that. It's an externally observable thing.
I agree that hallucination isn't some kind of "different" operation than "normal". It's not like when a train derails and you can point to it. It just operates as normal and sometimes that yields correct factual outputs, sometimes not. You don't have to metaphysically ascribe any kind of intent to it.
I'm not sure how people conceptualize these things who weren't doing classical machine learning before all this. To me, "hallucination" is shorthand, and we know it's not like humans on drugs or something. It was used in the literature also for any kind of generative imputing of missing information from a learned prior. For example in image inpainting a GAN "hallucinates" the missing part of the image. This terminology was already used in the 2010s and probably earlier. Or in image colorization of grayscale photos, the model "hallucinates" the color information.
Then the word escaped into the mainstream and people have weird connotations about it.
Expression of tech-faith is not intellectually honest argument.
Where does this "will probably not hold in the very near future" come from? People correctly warn about extrapolating current things onto the future, but then just throw some vague "probabilities" without providing any argument why their "probably" is somehow more grounded than others.
Markov chains garbled text to gpt took decades, gpt stories about unicorns to gpt production systems took a few years, gpt production system to gpt astra producing deterministic lean proofs of longstanding mathematical problems happened even faster. Scepticism to the point of requiring proof appears like an academic pursuit while production systems have already been built and are in the process of being enhanced
Putting hallucination aside, LLM "theory of mind" has gotten worse over time. I feel it peaked in Opus 3 and Sonnet 3.5, GPT 4 and then GPT 4.5 for OpenAI. Since then, even with Opus 5.5, phrasing has needed careful crafting, in order that it not be taken too literally. OpenAI models suffer from this much more than Anthropic models but Claudes have backslid over time too.
This means when writing documentation, tutorials or commit messages, their output is often a garbled jumble. Assuming shared context, using invented terminology without explaining, leaking conversational states due to improper epistemic boundaries and failing to model the reader. This all usually leads to their freely generated explanations being terrible. Getting good explanations requires chaining questions that force them to line things up properly, which is not easy the less you know. These failures as something LLMs naturally struggle with make sense, given the nature of attention and RL with weak signals from human data.
Math is not merely a collection of proofs, it's a way of understanding. A proof presented in a manner that cannot be incorporated remains useless. It does not make it's way to physics like Riemannian geometry and matrix math did. This is no less true when done by humans too.
Your hallucination conclusion, checking if a proof is one, is exactly the counterproductive cost.
Most of us cannot verify that the claims in the OpenAI lore dump are in fact all correct. It will take tons of work from experts to do this. It took subject expert mathematicians to identify the discrepancy and disconnect in the Navier Stokes proofs, for example. LLMs will struggle to make use of their own proofs or turn them into knowledge that accumulates over time.
The act of proving is often more valuable than the proof itself. Human constraints and limitations force us to invent tools and abstractions that a 100,000 x 1M context swarm can bypass. The tradeoff from that AI swarm advantage is work that doesn't usually lend itself to being built upon. It's like doing all the side quests and reading all the books of an RPG versus min maxing a straight path with a guide. We might try to identify new abstractions, but the fact that we don't get access to CoT and that much of it will be illegible means mining LLM traces for what human mathematicians produce naturally will be a tedious chore.
> Since then, even with Opus 5.5, phrasing has needed careful crafting, in order that it not be taken too literally.
This is a feature, and a huge step forward.
If you expect AI to do serious work, you can’t have it guessing what you “really meant”. Every sufficiently advanced task depends on very subtle details in the problem statement, and the correct default behavior for advanced AIs is to solve the task exactly as stated, unless a system prompt or other constraint tells it to do otherwise.
Did you even RTFA? His argument absolutely doesn't make any assumptions about hallucinations, implicit or not. It's you who assumes Tao must have surely been complaining about hallucinations or some such. You've not addressed any of his arguments and moreover ask questions his post answers.
> 2. the point of doing mathematics is to be “productive” in the first place. This is silly. Many people get into mathematics because of the beauty of understanding, for example.
Are they independently wealthy? Or do they have a deal with their local supermarket that they can take food for free?
What economic system do you recommend instead? Marxist-Leninist communism? Anarchism? Feudalism? I guess none of these will be fitting for what's coming...
It’s definitely interesting to think about how all of this would play out.
To slightly deflect your question, what do you think will arise naturally?
I’d put my money on feudalism if SOTA model providers can stay far ahead of open weights models. However if that proves infeasible and everyone ends up having access to local AGI does that serve to level the playing field? What resources are still going to be scarce in the coming decades?
Very few people will pay mathematicians to produce as if they were making art. The big problem here is that counting proofs is how mathematicians historically have created legitimacy for the universities they work for. Mathematicians in the past 100 years have been paid solely for the prestige they generate, the work itself is irrelevant to the university. If the prestige signal does universities will spend less on math departments. Every math professor will be expected to teach undergrads as well as a stable of grad students. That’s the value they are actually able to provide to universities.
> The big problem here is that counting proofs is how mathematicians historically have created legitimacy for the universities they work for
This is a really interesting point. This is almost analogous to a metric in software engineering like lines of code which is a proxy for something we actually want to measure.
In a way, the SOTA model providers are cargo culting what appears to be what mathematics is.
> but simply dumping proofs on the math community and expecting others to do the grunt work of verifying, refining, and expanding on them is hardly a productive way to advance the field
This is a transitive period. In a few years, verification and exchange between model instances will happen faster than humans can follow. Human input will be an ethical question, and not a productivity one, because it will be the bottleneck in any science.
Etic is a factor of productivity, and the larger the contextual window is the weightier it becomes. That’s even integrated within the paperclip parabola.
If the focus in placed on maximizing some easily measurable output on a narrow perspective, situation is unlikely going to match a sweet spot of holistic equilibrium which is maximizing harmony and happiness through humanity as a whole.
That's the billion dollar question. Nvidia currently wants to deploy chips with the sole purpose of monitoring agentic workloads and that doesn't seem farfetched but they obviously have a financial motive to sell more products. It's like a cat and mouse game, like cybersecurity in general.
I wonder if top labs will soon abandon math progress like they did go and chess.
In example of go where I'm more familiar Google deep mind poured large resources to get a super human performance first, establish superiority and abandon it. The community then built their own tools starting from reproducing their papers.
I think similar thing might happen to math. Nobody outside of math cares too much about Hamiltonian cycles in some bizarre graphs or proving lower bounds on complexity of some problem.
Once those results stop being worthy of mainstream media attention, they will abandon math and the progress will be done by mathemicians guiding the models and the community will likely establish some new rules about what makes a valuable contribution. Merely solving not yet solved problem might not be it anymore.
This misses the raw advantage of a good proof. It makes conceptualization simpler. In some ways math is like a hash list of of theorems. This list makes it simpler to prove other calculations, and will always be useful, to both humans and AI models. I can see two new directions 1 - the creation of specialist theorem models; that can answer questions efficiently about one topic and 2 - we probably need to incentivize and codify ownership of theorems; charging a proportion of the compute saved by using them. Ultimately enabling mathematicians to be paid our true market value!
Considering how essential math and science is for the prosperity of mankind (not even speaking about the cultural value) the question of how to reward people working and contributing in these fields effectively and appropriately is of extreme importance. (And I think the current decline in our societies is to no small degree caused also by our utter failure to address that issue.)
It is also fascinating, because I don't think there is any solution within our existing system, at least not any I know of. Theorem ownership is not a good solution (and neither are patents in general). Probably the most achievable (or rather the least unachievable) solution is a kind of communist utopia, where people can dedicate their time to a pursuit of any endeavor they see fit, as resources for a decent life are abundant and excessive power capture impossible. (The other option, somewhat dystopian, and which would not require humanity to change too much in its current mode of conduct, would be a totalitarian or caste-like capture of society by the scientific community.)
Incidentally, if AI proves as powerful as some expect it to become, it could bring about another solution of that issue by making all human science and mathematics obsolete, pushing its true market value to zero.
Almost every country in the world has a patent system for a reason...and they are pretty essential for big pharma to function. Granted there should be better calibration of duration - particularly in software, but rewarding first movers is wise in many market conditions. My point regarding "math as a hash list of theorems" is that Theorems will never lose their value, regardless of how powerful AI becomes; they represent compressed knowledge, and as such it will be more efficient for a more advanced AI to query known theorems than reconstruct each one from scratch - in that gap lies a marginal compute saving; an api service or tool call that someone might charge for. Even AI can benefit from specialists...
I wonder if it makes conceptualization simpler for models too, given that they're trained already on human-speak. And I'm also curious as to whether humans currently have an innate advantage into simplifying and contextualizing proofs, or will the machines get good at that as well?
My point is that they will, and that is always going to be useful. Bundles of simplified knowledge (in whatever form) that make research simpler will be useful to share whether humans can understand them or not...
Yeah chess is a good example. DeepMind came for publicity with AlphaZero. Arranged a match with Stockfish with rigged rules to make AlphaZero look better than it really was (it was amazing but the match wasn't fair) and then just published some games and went home.
I was bitter about that back in the day as I hoped for more answers, more matches, more "truth" about chess being shown. Soon after that community project Leela Chess Zero was started and not only surpassed original AlphaZero but added few hundred ELO points over it. Then the combination of NN and classical engines happened with NNUE and current Stockfish is again a few hundred ELO points stronger.
Today we pretty much know the truth in chess for all practical purposes. Human analysts/preparation experts focus on finding interesting path and opponent profiling (what is the most unpleasant for the opponent to face). They don't look for truth anymore. The game is doing great, it's more popular than it ever was.
Stockfish and other classical engines were never intended to be run in matches without opening books (of which there were plenty).
Development assumed the presence of opening book and authors made 0 effort to make engines play well in openings because of it.
This is also the reason classical Stockfish was a very small binary. A little effort to make it even by including even a very small opening book (like 50MB or something that would result in still smaller binary than NN engine with its net) would make it much more interesting.
The result was that Stockfish lost many games by walking into known bad lines and lost way more games than it otherwise would.
> We also played a match that started from the set of opening positions used in the 2016 TCEC world championship, along with a series of additional matches against the most recent development version of Stockfish, and a variant of Stockfish that uses a strong opening book. In all matches, AlphaZero won.
Ok I remember it vaguely but the match that got publicity and the one published results were derived from was 1000 games match from starting position.
Deepmind claimed AlphaZero also won from TCEC positions and vs Stockfish with good opening book but at least back in the day I don't think I could find those games being published or specifics about books/positions they have used. Can you?
I am not claiming AlphaZero wasn't stronger. It wasn't as strong as the PR piece suggested though and we have never seen the games being published. In chess this is extraordinary because basically all games in chess are publicly available - both human and computer games.
Claiming "we have created a strong engine that has beaten Stockfish with opening book" while not showing those games (or details about opening book used) is akin to "we solved this math conjecture" without showing any kind of proof or argument.
Publishing a few 1000 of games costs nothing. Tens/hundreds of thousands of games are published every day.
I disagree. Firstly, people in AI likely care about math on a personal level. Secondly math is useful. Playing go or chess is basically a party trick. Being useful gives it staying power.
But, I do think you are right that there will be some level of moving on. The spotlight is currently on maths and that won't last. It will move to some other area where there is more impact to be had. So while they might shift gears and put less focus on math, it will always be there as part of the portfolio.
I had the same idea recently. You've solved all the famous conjectures (all formulated by humans because humans found them interesting), what next? I doubt "AI formulated a math conjecture that nobody else cares about and immediately solved it" will produce that much hype. The actually interesting thing is indeed how mathematicians themselves will use these AI models going forward and how that will shape mathematics of the future.
> I wonder if top labs will soon abandon math progress like they did go and chess.
I definitely think that this is marketing, just "with good side effects". My doubt is when they will be able to move to "marketing with better side effects", that is, research with more concrete outcomes (health, materials etc.).
While top AI labs no longer focus on chess, the community build way better chess engines.
Stockfish is probably stronger, than everything the top labs build.
Wouldn't we expect the same thing for math? That slowly the broader math community would engineer a harness/program... That will surpass the current labs, and be a community ran project
Yeah its certainly true now that Stockfish is much stronger than alphazero, but it's probably also true that had Deepmind spent another few years working on alphazero it would be enormously stronger than either.
In the case of chess this seems fine, there isn't much value to society in creating an AI capable of beating top humans with a 4 pawn handicap rather than a 2 pawn one, but for maths where there are actual applications it is more complicated.
They aren't trying to 'solve' chess, go, or mathematical proofs as an end in themselves, but mainly in order to learn more about how to build better systems overall. The goal of AlphaZero was ultimately as a stepping stone towards AGI, and it's the same with LLMs.
That’s still an assumption. Since when do you uncritically believe what people are saying without taking into account the different external drivers on those people.
You can believe they believe they are telling the truth or that they have good intentions while still being skeptical of the effective trajectory.
I don't think it will be the case, maths have real utility. I found something interesting at the intersection of combinatorics and information geometry. To be quite frank I don't understand what I'm doing. And yet, when I ask ChatGPT to use the framework we're developing to write an algorithm, it turns out it has quasi-parity with the state of the art. I have to measure absolute perfs to decide which one is better – theirs, not mine. Ok. Time to keep improving on what I have. And this implies dropping the code and going back to the blackboard doing more super abstract math that are way out of my league.
We are at the point where the way in which humans do math and science changes significantly, and I have no good idea at all in what state is it going to settle down. But you are one of (many, I suppose) people exploring the new wilderness, so I wish you best.
If we get AI singularity, then humans will stop doing scientific progress altogether. But if we don't, then it's pretty predictable what's going to happen - things will be much the same as now, except everyone will be using AI for proofs, data analysis, theoretical models and designing experiments, so important discoveries will happen more often. It's also possible that after the AI craze dies down, we'll have enough computational capacity to solve protein folding.
Some math has almost incalculable business value, because math is the biggest driver of game-changer technology.
We'd be nowhere without Laplace and Fourier transforms, Maxwell's equations, elliptic curve cryptography, and many more.
Most math doesn't, but often these techniques are invented first and the applications come later.
And the criticism of the current round of proofs is that while they may be true - likely for some, questionable for others - they're not adding new techniques or insights.
There is a risk of this particularly if it's seen as advertising - at some point "ai model solves hard to explain problem" isn't going to be news and that benefit goes.
However, there's some of this that's a proxy - the compute to solve these problems was very low (they claim a few hours of thinking time on a regular subscription). The large cost would have been the training and if training the models to be better at these things makes them smarter for useful tasks that's beneficial. I believe there was work done earlier on around showing that training the models on code made them better at broader reasoning tasks (not just writing the code itself).
Another side is that if one goal is to improve the models themselves, their ability to work on mathsy problems must be high. That has very direct business value, and ideological value depending on what you think the motivations of the people running the companies are.
Why do you think this has no business value? It would be absolutely wasteful for OpenAI to not be doing this as part of a post-training RL rollout.
There are architectural advancements yes, but lots of progress from LLMs really come from (1) better pre-training [generally through more cleaned data, and ofc more data], and (2) lots and lots of post-training. It's how we get more and more intelligent models for the same param sizes.
The 'marketing' is just a useful side effect they get from their RL rollouts on maths and LEAN.
I think there’s a venue where they start focusing on introducing hypotheses where the model currently can’t solve it, or maybe this is already happening?
Being able to present useful novel ideas would likely generate a lot of press, for a while. I don’t know how this would look since I’m useless at math, but Im sure there are plenty of unknown problems with massive implications, that once formulated can be solved.
AlphaGo and AlphaZero weren't generalized models. Math capability will presumably keep improving along with the other general capabilities, even if there wasn't a special RL focus for math itself.
The amount of money they're lately ploughing into proving math theorems is inconsistent with how societies and markets have priced pure mathematics. The entire US federal budget for math research is something like ~$100M annually; a single college football coach can already earn 10 percent of that.
Pretty much the only enterprise that historically pays some mathematicians handsomely is quant finance, but those people are actually compensated not for proving theorems but rather for statistical modeling and programming skills. And even that industry is so technologically driven these days that pure research mathematicians no longer hold a clear edge over a strong programmer with fluency in undergrad level probability and statistics.
Math is one of the most verifiable domains, esp thanks to LEAN, which also build coding skills.
The $$$ they're pouring isn't just for marketing. Think of these papers/results more as "useful side effects" from large-scale RL rollouts and post-training. Every token being generated contributes to post-training in some way.
There isn't a hard boundary between "training" or "inference", modern post-training is arguably inference-bound :)
It was shown quite some time ago that training LLMs on programming tasks improves their logical reasoning skills also in other natural language domains. So I could see math also being a training gym for AI even if the final use case is not directly math-related. Having to solve math problems efficiently can build in skills that come handy in all kinds of more everyday tasks or science and engineering.
It’s also very useful signal that the reasoning trace is leading to solving open problems - you can be certain that you’re not landing somewhere inside the training data.
I think the entire thing here is that these solutions _are_ inherently interpolations of existing work in the field. That's the "super power" that LLMs have. To interpolate mass amounts of multi-dimensional data.
This depends on some handwavy use of the term "interpolation", not the mathematical definition. Mathematically interpolation usually means that the query point is in the convex hull of the data points, and that almost never happens in high-dimensional spaces.
See:
Learning in High Dimension Always Amounts to Extrapolation
Randall Balestriero, Jerome Pesenti, Yann LeCun
https://arxiv.org/abs/2110.09485
I guess you mean by "interpolation" that it's some kind of nonlinear combination of the training data, but that's an almost vacuous statement. Any input-output relationship has to be so by definition.
No, more like the average can be more important than some outlier's impact, at least for steady progress. So while you do need the Mozart and Einstein, a more productive effort would be to raise the general population's education level.
Humans learn in the side effect of education, nobody is busting out some poor method that schools teach to solve antiquated problems in the real world.
A lot of people have been surprised by the interest that AI companies have in solving maths problems without obvious applications, as well as in the financial precariousness of these companies. I imagine that they thought that the companies were helmed by mere bean-counters, like at Boeing. But no, they're still academics at heart, and so they work on Navier-Stokes and L=BPL, instead of on how to maximise value for their (soon to be) shareholders.
Interesting path forwards, and probably partly true, but there are some important distinctions:
Go was a specialized application. All the math results come as a side effect of reading the whole internet, and it will keep reading the whole internet. It will keep practicing thinking questions. Actually, math might be one of the best ways to keep them contemplating and measure their contemplation abilities, so math will always stay in the loop.
Also, math might not be useful just for humanity, but also for AI, so the system might actively benefit from new math results itself. (Not sure if any of the recent proofs qualify, but future work might.)
I think this is an interesting and good theory. They've probably eked out the large majority of the PR benefit at this point, so whether they continue in this vein will tell us a lot about their motivations for this work.
To take this to the next step, what happened after deep mind pretty much solved Go is that they started looking for the next set of things that hadn't been done yet. It does strike me as very likely that this will follow that same path.
>Merely solving not yet solved problem might not be it anymore.
Never has been.
See: Mochizuki's "proof" of the ABC conjecture [1].
In short, mathematics is an art of storytelling as much as anything else.
Writing a boring story, much less an incomprehensible one is worth exactly nothing, and doesn't matter.
Is it cool that the machine can connect the dots that people put out there in a form that happens to be easily ingestible?
Yeah.
Can the machine answer why give a fuck about a result like that?
Nope. That work has already been put in by people.
The most obvious impact of this BS is that interesting problems with potential to be solved, which were already a scarce resource, will no longer be shared.
It's already become a huge issue without AI being involved, due to the rat race nature of academia.
There's even an industry term for when someone gets a wind of another person's progress on such a problem, and solves it before them - it's called "getting scooped"[2].
It doesn't mean that they got access to the results. It can be as simple as e.g. an experienced mathematician hearing about a problem that a PhD student is working on for their thesis, and scooping it.
Honestly, as a mathematician-turned-software-engineer, I see and hear both sides, and both sides are lying:
* The AI companies are lying that what the LLMs produce is "mathematics". Writing a proof, even a correct proof, is not what mathematics is about. The Fundamental Theorem of Algebra has been "proven" incorrectly four times IIRC, but it was mathematics nonetheless.
Mochizuki's "proof", on the other hand, may be correct, but over a decade later, it's still not really mathematics, because transfer of insight is a fundamental component of it, and insight is a thing that the AI, presently at least, doesn't do, no matter how many proofs it produces.
Proof automation has been a field for a long time, for that matter; hyperscaling it doesn't change the nature of things.
The lie is also that a correct proof is the final point. The Four-Color Theorem[3] was proven in 1976, yet people were still proving it 20 years later, trying to reduce the number of cases to leave to the machine, even as we got it super-duper-verified by theorem-proving software in 2005[4].
The writing was on the wall since computers were invented, so why are mathematicians making Pikachu faces now?
That's where the other lie comes in.
* Mathematicians are lying to everyone else by suggesting that the technology is the problem, or that the issue is with how the technology impacts people, and not with the COMPLETELY, ABSURDLY FUCKED UP ORGANIZATIONAL STRUCTURE OF ACADEMIA, and I can't scream it in bold enough caps lock.
There's a reason websites like "100 reasons to NOT go to grad school"[5] exist, and all who've gone through the masochistic 5+ year journey can attest that actually, the reality is worse.
Closer to the point, there's a reason Grothendieck burned most of his work and fucked off from mathematics in the 90s, and Perelman did the same, in effect, in 2000s[6], while putting a million dollar weight to his statement that MATH SOCIETY IS FUCKED.
To all the mathematicians running in circles screaming AaaaAaaAAAaa, Perelman can now say: told you so.
See, after we discussed what math is about (insight transfer), let's do a peculiar little observation that this is exactly what's rewarded THE LEAST in the field.
"Teaching jobs" are a punishment for those who don't do good enough, and doing good means cranking out proofs in a publish-or-perish race, emitting just enough for others to verify correctness, but not too much lest you don'...
> Too much effort is being invested in proving that the exponential curve is still holding.
Given sustained exponential growth is mathematically impossible to maintain with fininte resources, it's funny to me they're using advanced mathematics to try and achieve this.
This Math 1.0 followed by Math 2.0 framing is wrong. Mathematics did not start a few decades ago when problem solving became the norm. And it will not end now when problem solving turns out to be "easy".
Mathematics will revert to it's main practice, which is to study.
There are lots of weird panic reactions by some prominent problem solvers. See for example the ridiculous cease and desist like statement of AHM shared at Tao blog.
Put this Math 1-2.0 with that AHM statement together and you'll realize that this is a power struggle and that you see only one side of it.
Mathematicians have very diverse opinions about this. I, for example, am for as much as possible automatic harvesting of all these "low hanging" fruits. Should be disclosed as soon as possible, free of any bottleneck, and citable. The mathematics community may do whatever its various members desire to do with these results. Let them decide individually what to do with them. This AI tool is here to stay.
OpenAI does not make money from math papers LOL they just release them for the sake of community. (Because sitting on those results would be considered worse.)
Solving a well-known open problem like N-S "millennium" problem has some marketing value, as this can be referenced in a press release, etc.
OTOH a marginal value of a single paper in a paper dump is basically zero. Nobody gives a flying fuck if they made 100 or 200 papers. It won't give them even a single customer. If they released, say, 5 paper per month it would generate a buzz but not as much backlash from mathematicians.
> This does not affect people building new theories.
Actually, the problem is, somehow the skill of building new theories in math is directly tied to slaving hard over a problem. It's the very experience of slaving away that actually somehow causes ideas to form. Pretty much all mathematicians understand this. Yes, senior mathematicians now can form some new theories, but what about junior ones who will have very little experience in working hard on a problem by hand?
Of course, they could work on the problem by hand anyway, but they won't because no one will pay them when a machine can do it.
Let's start with the fact that mathematicians aren't paid for results. Institution which gives them salary fundamentally doesn't give a fuck about theorems. They might care about having top-grade mathematicians for prestige, or because they believe that countries which are "good at math" also do better in science, engineering, etc.
> somehow the skill of building new theories in math is directly tied to slaving hard over a problem
We don't know if that's the only way. Perhaps collaboration with AI is just as good. Why reject it before it has been tried?
> what about junior ones
Junior guy with brilliant new ideas might benefit the most from AI as it can compensate for lacking technical chops and breadth of knowledge.
> There seems to be more interest in hitting some arbitrary benchmark (we proved X unsolved problems) than in genuinely contributing to mathematics
I feel the same way about academia, the papers, the citations, the ego, the narcissism and the taxpayer codependency that got cut off and turns out wasn’t necessary at all thanks to a private sector entity running laps around them
I don’t feel that academics need to pursue the discipline and distributed brain-wracking that has sometimes resulted in the solved math problems, just because more times they find other nooks and crannies to explore along the way. I think the blueprint is enough. Standing on the shoulders of giants is good enough.
and if the concern is that they can’t figure out what to do with a proof, next year’s AI will
As with any field, convincing people to care about your ideas and your approach is half the battle
Many of the best startup ideas by the best product and engineering minds failed to gain attention and funding. Same with much of the best music - relegated to hard drives with derivative ideas only resurfaced decades later
I would expect much of the recent math dump will be leveraged by other LLM-driven research teams rather than read in depth by a human
Once LLMs pass the threshold to being to invent new general-purpose methods and frameworks, the frontier is irreversibly lost to AI and it becomes simply a hobby that mathematicians pursue. They work through, digest, and maybe write up the proofs for understanding. But the real meat will be growing the LLMs. Who knew software eats the world was so true?
> simply dumping proofs on the math community and expecting others to do the grunt work of verifying, refining, and expanding on them is hardly a productive way to advance the field.
I tend to agree with this, but what is the alternative? Should OpenAI and Anthropic employ hundreds of mathematicians to do this work? Should they just not solve math problems within their reach?
It's unclear what more could be expected than releasing the presumably already verified results and write-ups for each problem. Should they run a mathematics school too?
Then the comment goes on to argue AI labs were not interested in actually advancing mathematics, and that investments into AI were manically excessive.
IMO none of this follows and demand is there to justify the investments.
The comment then goes further to argue that AI labs were putting too much effort into pretending there was exponential progress rather than actually making progress.
The factual basis for this claim seems to be that OpenAI released math results and write-ups, and it's not even clear what more they could do on that topic.
Look at how actual researchers are currently using the tools: they'll generally use the LLMs to create slop papers, sometimes supported by auto-formalizations, just like OpenAI does. Then they will go through the lengthy process of digesting the results, turning the often incomprehensible and poorly organised outputs into something that humans can understand and build upon, they will then give seminars on the results, further helping with dissemination. Doing so still requires expertise, and probably will for a good while.
So yes, that is exactly what they should do. Alternatively, if they are too lazy or incompetent to put in the effort themselves, do what AGMAI proposed and fund a third party to help out.
I think a problem is that math seems like a deeply toxic, ego driven domain.
I think he argued that e.g because the navier stokes millennium problem ist considered solved now, you won't get any recognition for being the first human to solve.(How would you even proof you solved it yourself and not just regurgitated the ai proof?)
And since recognition is the main objective, noone would spend time on dissecting the proof, and perhaps finding some unique approach to solving the problem, that could be transferred to other open issues.
And therefore the problem is now "poisoned". Since it's assumed to be solved noone will research it, and the potential revelations won't be found
The biggest problem is LLM tends to produce over engineered, very complicated proofs that are an eyesore even for relatively simple problems. Give it a beautiful Olympiad geometry problem and LLM will tear it apart into ugly algebraic calculations, turns all lines and circles into equations and calculate their intersection points that spans multiple pages because it is a guaranteed way to solve it. Correct, but hardly any use to the user.
In TFA I read (scroll up from the link anchor) this was explained: OpenAI are not giving talks - because they can't answer any questions about the model's work. There is little follow-up activity - the actual elaboration of human understanding of the new ideas is stifled since the problem is solved.
But you are right, this is not OpenAI's "fault". The problem is - as others have said recently - that many people in mathematics want recognition for solving open questions more than they want the answers to the open questions. Everything about the economics and social environment of Mathematics will have to change.
I think that this is exactly the same split we see in software: there are those who mainly enjoy the craft aspect of building software, and are uninterested in the product or business they are supporting. Others are primarily interested in the production of useful software or building a platform or company.
I've always been in both camps myself. When it became obvious that AI was going to destroy the craft aspect - at least two years before it actually could do so - I became very discouraged, even depressed. But once it was actually good at building software, I became very excited about all the stuff I could now build. Sadly, I think a lot of people in our field have never had something they really wanted to build.
What does it mean to “genuinely” contribute to mathematics? Because the definition you use is load-bearing ;) and it might be different from that of others.
Perhaps mathematics was never about proving things? I know it sounds like moving the goal post, and it certainly was what motivated mathematicians on a day-to-day basis, but bear with me for a moment. I think that beyond being a creative activity that humans enjoy, math was about building new tools and systems of thought. Axiomizing things we take for granted, logic, linear algebra and calculus (on which modern LLMs rely so heavily) are such examples. People chose to participate in this field because they found it enjoyable and satisfying in some way. As a side effect, society reaped the benefits every couple of centuries. Will harvesting open problems with AI ever give us these things, or will we just be left laundry pile of Lean formalizations?
Because they published a ton of slop papers with terrible English, impossible for humans (even experts) to understand, full of non-standard terms, invented jargon etc.
So effectively, stuff got proved, but people don't really understand how, so it's mostly fucking useless and done for OpenAI's marketing team, while also pissing off the maths world at large.
So if god himself comes down from the heavens and hands you the answers to major unsolved problems, but doesn’t walk you through them and you’ll have to still put in effort to understand them, then that’s “not contributing”?
I'm in the same boat, I guess I had this naive idea that unsolved math problems would mean something if they were solved. But it seems like a lot of them at least were more thought experiments than anything else.
Kids on HN have built their whole personalities around math, and now, they are discovering that "problem solving ability" has nothing to do with solving problems.
Mathematics is less about the end product and having a healthy community of people to actually understand the proofs and eventually apply them.
That community won't exist as many people simply won't even enter the field because it's reprehensible and contemptible, not to mention boring, just to read machine-generated proofs and verify them.
Collaborating on, or at least working on unsolved problems is what motivates most people.
AI is like a cheat code in a video game. You get to the end faster but fewer people want to play if the cheat code is always on. You can't turn it off either because the very challenge is to do something unique.
Its pure cope from the math community. One year ago if god came down and asked any of these math guys if they would want to see the proof of the Riemann Hypothesis they would all say yes. Now that the AI math god is here and handing out proofs they are all like 'not like that'!
If math is just about having a community of other mathematicians to hang out with, it still isn't a career. Nobody is paying money you need in order to to eat, just to hang out in a community.
Just like a software engineer, "Well AI can write all my projects now, but I have my local Rust Users Group to hang out with". Nobody is paying me to hang out and hand code Rust.
The other gap in AI is it doesn't explain or lay out how it got to the final proof which is often more fruitful for new techniques etc than the final proof by itself.
In mathematics it's usually a large portion of the paper because they know it's the interesting and useful part of the discovery. Often they come up with new tools and techniques and those often unlock further discoveries.
So you'd prefer if they kept their work secret? Or you don't want them working on these problems at all? Or they should be required to do the work the way you want them to?
I'm not clear what you see as a better option than dumping.
> but simply dumping proofs on the math community and expecting others to do the grunt work of verifying, refining, and expanding on them is hardly a productive way to advance the field.
I assume this will increase the job market for mathematicians though right ? Tens of thousands of dumped proofs will mean math majors will need to be hired for review.
But they are doing the work of verifying, refining and expanding. The humans are irrelevant in this situation. OpenAI is just letting them know that they are now obsolete.
It's been very interesting watching Tao's evolution on his thinking on LLMs. Of course the LLMs have themselves evolved so that shouldn't come as a surprise.
The job of professional mathematician might be the first to be completely eliminated by LLMs, save for those who can make money from a patron. I am hoping they are able to figure something out to save their profession, as other professions could use it as a blueprint as AI comes for them next.
Does this response properly anticipate how math will change further with the next N model generations? Exposition and exploration may fall well within the capabilities of future models.
>the mere knowledge that a solution exists "contaminates" efforts by both humans and AI to find alternate routes to the problem that reveal additional insights
This really expresses the heartburn you see across all fields, not exclusive to careerism. I certainly have friends in decomp and fan translation spaces that have been demotivated by the current rash of efforts happening there.
The rush to be "first" has always been over-celebrated, but it would be nice to believe there's a way to get beyond that thinking.
This has already been the case in AI/ML and computer vision papers via flag planting papers. Have an idea, super quickly publish a hasty work based on it that doest actually work, methodological and eval issues, engineering terrible, slow, bad results etc. But it was the first so now your concurrent work that was much better evaluated, better implemented, etc is suddenly worthless and unpublishable.
Byte-matching has been a side-effect of gaining an understanding of how the software works. With this you are skipping the understanding and you will not get people like Kaze Emanuar on YouTube who have dedicated a lot of time to building upon this understanding to create something better.
I don't think it's motivating to solve a black box by having AI generate another black box if what you want is to understand how the thing worked.
> Byte-matching has been a side-effect of gaining an understanding of how the software works.
No? Decomp sources are often full of comments that claim that there's no clear reason as to why something is done a certain way or straight up full of question marks. Byte matching often is a result of bruteforcing a solution rather than understanding the original idea behind the code.
If your goal was to do something for RHA games you love and understand them on a deeper level, then just handing that off to an LLM doesn't really give you the same sense of accomplishment.
I imagine doctors will also clutch their pearls when Ai starts curing disease. "But curing disease was never the point! These arbitrary dumps of AI cures for cancers is unsustainable! Who will think of the doctors and who will build their communities further? From now on progress in medicine must be redefined as what makes doctors thrive, not what generates cures!"
I think the difference is that with cancer cures we mostly care that it works as proved by trials, and understanding it is a bonus.
Up until now the prize in pure (as opposed to applied) mathematics was the _understanding_ and the machine can't do that for you. What does it mean if we get "super powered alien maths" but humans can't do it? It's like inter univeral teichmuller theory all over again but imagine if Mochizuki was right and it came with a lean proof?
It was the understanding for some, and the result for others. Math is going to bifurcate along those lines. Many are the builder type who use math for a purpose. To accelerate an algorithm, to improve the numerics or convergence of a computation, to verify statements about real things, to use it in angineering applications, etc etc.
In my view, over the last century, math has turned into an intellectual analogue of extreme bodybuilding competitions. A navel gazing runaway optimization in making useless stuff just to demonstrate cleverness. That's fine, why not. But society has no obligation to fund that, just as it doesn't fund other extreme hobbies. Ideally if we ever get something like UBI, math can be still their hobby.
Right. What I not sure about, a lot of times the results will be verifiable but what about when they are not? Like, ai is really good at proving or building a slightly different thing than you thought you asked for, and if you're not at a level where you can understand key details of the formalisation of your ask, you cannot safely use the results.
Maybe AI will take over some roles of doctors, but that's independent of curing diseases. Think about diseases that have a cure - do people with those diseases not go see a doctor?
The same could be said of Software. Instead of giving up on creating novel projects and instead just taking other peoples ideas and porting them to Rust, we could be embracing AI to push software and computers farther.
Im not sure how that will work, but im convinced the current paradigm of just pushing agents into codebases for not much reason other than you can is going to make building software incredibly boring and push creative people away from the field and stagnate progress.
My prediction is software gets boring and building hardware projects will be the new frontier for creative engineers looking to push computing further. Which is probably a good thing.
Sort of. An elegant proof is useful beyond what it shows. It hints at new mathematics, and can prompt discovery in applied fields. I don’t think I’ve heard of elegant code leading to discovery on its own.
> I don’t think I’ve heard of elegant code leading to discovery on its own.
I’ve not heard of it either, but code is an abstraction of math, so I don’t see why this couldn’t theoretically happen.
Anecdotally, I’ve started spending time advancing my math skills beyond the early college level I stopped at and I’ve frequently found I already know concepts of more advanced math - I just didn’t know what they were called or how to apply them to an equation on paper, but I’ve been using them for years and intrinsically grasped the underlying academics.
I think this happens all the time actually, but it's often smaller scale. Like someone writes a neat architecture to do X at their company, and later someone else walks in, looks at this code and sees it's now super easy to do Y, which then turns out to have immense user/business value.
Models have limits too, I don't think software will become boring. I think it'll become more interesting, in the not so distant future one software engineer will be able to do so much more than today.
People wrote many books about software engineering, all from valuable experience from buildng expensive software systems. But in the age of AI, is there still anything learnable from generated code?
Personally I always ask AI to summarize its findings and lessons in a .md file. And I always learn something from it.
But could AI utilize some new patterns and paradigms I wasn't aware of? Very likely. Because we only learn from our personal grave mistakes, a summary from others gets neglected and forgotten
I do agree with you, although I think that there may be still some advancement available in software in terms of programming languages or novel architecture design. But frontier is much more around hardware, just thinking about computing power, electrical transmission, material constraints on power, connectivity, insulation. I would definitely advise kids to study physics and materials engineering than CS.
I mean if we're being honest, most of CS studying (as practiced) was a waste of time anyway.
The S fell short in actual reality for the most part, as it was merely a hiring requirement. A hiring requirement that didn't even make sense, because the skillset of academic CS only marginally overlaps with the skillset one wants to hire for.
Material engineering at least for the most part has actual real-world applications where one can push humanity further. CS (as practiced, not necessarily the idea of real CS but the CS we got due to it being used as a hiring filter) for the most part is just self-referential spinning with mostly unclear results.
There is real impressive work being done in that field, of course, but I'd argue that the majority of it over the last decade or so at least was just performative nonsense.
Maybe by again allocating new resources to other fields, what hides under the label CS can become more pure actual CS again. I think that would also be a much less miserable experience for everyone involved.
What new things are even left to discover in software? You can still use C for all the backend stuff and I don’t think all the endless stream of front end frameworks and libraries are all that innovative. I think the only tangible innovations we’ve made over the last few decades have been in infrastructure management. All heavy lifting is done by mathematics anyways.
Originally the word computer described a person doing the act of computation.
I believe in the future, we're going to see a similar shift in "programmer" - instead of a human programming the computer, you'll give the ai an idea and it will spit out a program.
And just like how automating the act of computation revolutionized what we could compute, automating the act of writing code will change the act of programming - hopefully, as you described, allowing us to do things that simply were not practical in the past.
I suspect the whole field of mathematics will simply disappear as a career path. It seems obvious that the trajectory is for the machines to be able to provide proof on demand for any solvable problem. Whether or not the proof is understandable by humans is perhaps irrelevant in the larger sense. Doing hard math will simply become another black box tool in the larger AI toolkit for goal optimisation. Is this sad and should we try to prevent it? Is it any less sad than the venerable London cabbie who spent a life time memorising every street to gain "the knowledge" and almost overnight supplanted by machine intelligence.
Most of the problems that have been solved are problems on which a great deal of progress had already been made. Those who work on well known problems posed by famous people are those who suffer the most from this. Those who do their own thing and pose new problems, on the contrary, benefit from it. Suddenly raw technical power and great memory are not so valuable as a broad perspective, structural insight, and wild ideas. Who can be successful in this new ecosystem is different. Some of the elites are (correctly) more threatened by it than some "mid tier" mathematicians. I see lots of opportunities to overcome obstacles in my research program some of which had confounded me for years.
On the other hand, it puts a premium on resources. AI is not cheap for mathematicians. Folks are fancy universities in rich countries with forward thinking ministries of science will have an advantage over the rest.
What is clearly in immediate crisis is the traditional model of doctoral education. Most of the problems that were "given" to ordinary doctoral students are solvable (quickly) even by something like Claude pro. Mathematicians need to adopt training models more like what is done in experimental and laboratory sciences - collaborative and structured.
Where Tao is wrong is in regards to exposition. AI already writes better lecture notes, problems, and exercises for mid level undergrad math classes than do most of my colleagues. It's exposition is generally well structured and clear and it can adjust level on request quite well. It writes research better than most professional mathematicians too.
Working heavily with LLMs for the past year has me nodding strongly with Tao's mindset.
AI only take us as far as our imagination thinks to ask it. This can be exhilarating when new models drop every month and we can continually reach a new threshold, basically for free. But it is only a one time gain and ultimately short-sighted. Where I find continuous value is using LLMs to help my understanding, full stop.
I use LLMs all day long as a SWE and I have tried many approaches, but the most satisfying and consistent approach is to lean heavily into understanding a problem space and a solution space. Yes, it whips up architecture and code, but I spend most of my time peppering it with questions about the design and how it handles certain situations, what about this edge case and that security concern and this future product need. I have it write a report breaking down the feature and how it integrates with existing code and if the report is too confusing I have it simplify either the report or the code until it makes sense to me, sometimes scaling back the work to a more manageable state. I do all of this before I look at any of the code it writes.
The difference from this approach is that I am not suffering reading through 3000 lines of AI slop but I am reviewing a PR that I fully understand. I can eyeball it quickly for anything that doesn't fit my mental model and dig deeper or quickly revise it. Only after I am happy with the bones do I consider the meat and skin of the code.
What I find most concerning is how frontier AI companies all seem to have this Math 1.0 perspective that they only want to type "solve Riemann" into the chat box and have the magic to happen. It is the same problem Google ran into, where a simple, no thinking solution serves most of the people best and most profitably, so you fully ignore or remove everything else (boolean operators, exact phrase search, verticals, filters, infinite pages of results, "nothing found" if there isn't, etc.) But that choice leads to the situation Google is in now, scrambling to stay relevant. In a different world, Google would have continuously augmented their search capabilities and eventually built a smooth, guidable AI interface.
But no, we must only have an input box and a Go button.
Everything looks like a nail when you build hammers, sell hammers, have infinite hammers to play with however you like and your company mission is to build a hammer starship to explore the hammerverse, whether or not that is even possible.
Strongly agree. I see the need to understand as back pressure on the system that produces code and features: it limits output to what fits into a working mental model. Agents could turn tickets into code much faster, but I am building a product, and it needs someone who holds that model to steer it long term.
And when the job is done, I run retrospectives on old coding-agent sessions to find areas of friction and confusion. I also journal with pen and paper, as it's supposed to bring cognitive benefits, to help me stay on top of things.
Can we please stop reducing human activity to "taste", conferences, talks, "understanding"? I think this is a very unproductive trap.
There is a world where we get to the edge of AI capabilities, and we build on top of that. As humans have always done with every new technology.
There is another more pessimistic view where LLMs just replace every human capability, and our economic overloads dont need us for anything and we just eat the small pieces of bread that are left.
This comes down to the fact of:
is human existence/intelligence just the simbolic representations we make in our brain? Or are they just a tool?
I tend to think of Godels incompleteness theorem as a proof that on the limit LLMs are useless. The real question for me is at what point approaching this limit becomes an issue, and if it has any practical co sequences.
I know that is the whole point. But i may dream of being as tall as a house, but it does not matter how hard i dream, it wont happen.
In my experience, every new model release allows me to go further, although every time i see every time the limitations, and i identify where i add value. And this value gets bigger every time.
But on the other hand, every model release reduces the amount of people that are able to value this "added value", because it requires more skills.
So we have the paradox that the added value i can bring on top gets bigger and bigger, but the perception of the economic value for the majority of the population gets smaller.
I mean, he still seems to underestimate what future models will be able to do. The exposition, connections, and new research directions he wants to reward are also things future models will do far better than humans. The future of humans in math will be far less important than it is now. Humans are better suited to pursuing math like we do pleasure reading.
The interviewee is worried about the future of math research. He is not strictly worried about being replaced, instead he is worried that he will no longer be able to launder math-as-a-hobby through math-as-something-useful as is the case today. He lays out very clearly that grant proposals claim to have useful outcomes while the proposers know those claims are nonsense.
Business as usual in math, and frankly in all the other sciences, is to do research that furthers the researchers careers or personal interests and pretend that it’s somehow useful. This would be absolutely fine if it were privately funded, but it’s not, this is public money.
In every other endeavour, lying in order to get money is considered fraud.
We have collectively wasted a huge amount of taxpayer money and human time, entire careers, on things not likely to ever matter to anyone.
I look forward to science becoming automated so that we can have real progress instead of the current broken system.
People thought, back in the 17 century, that imaginary were useless (except as a trick for some calculations). Turns out the research into these numbers back then is amazingly useful today, 300 years later, in electronics and such.
Publicly funded maths research should continue, even if some taxpayers feel it's a waste of money.
it's the same - what research getting funded should not be determined by the applied nature of the research.
In fact, the taxpayer should be funding fundamental research because it's so hard to justify profit from it - but that funding would benefit all in the long future.
So leaving the applied research that have commercial value be funded by private, commercial interests, would make more sense.
> usefulness can only be determined after the fact.
Thats great! So in other words it should be perfectly fine for AI to solve all of the supposedly useless math problems so that it might possibly be useful later. No need to worry about the mathematicians hobbies here.
The whole point of public funding for science is that seemingly pointless research yields useful but hard to monetize discoveries. Otherwise VCs would be doing it.
> But at the current time, the opposite is often occurring: problems are being solved autonomously by AI prompters who have no interest in the broader field itself once their initial target is "solved", and do not understand the AI output well enough to answer questions on the result, give talks, or otherwise interact with the rest of the field.
Sounds like Terence Tao would have said the same about Ramanujan who basically just "solved" problems without much explanation / reasoning / communication other than it just arrived from god.
In the case of Ramanujan, others took on the responsibility of socializing and community building knowing that he wouldn't do it himself. Why can't the same approach happen here?
There will be people who want to just "solve" math problems now that they have a new tool that lets them express themselves this way. Maybe the don't want to participate in the broader math community, etc. Why discourage them, or add friction / a barrier to them participating in their own way? Why not take on the burden of socializing, making sense of, and community building yourself?
There may be valid reasons here I'm missing, but to me this seems a bit like wanting others to approach a field in a particular way even though the field can support many ways.
He would never say that because Ramanujan was the one who had the insight that he couldn't explain, not a machine. If a machine solves the problem, then the machine deserves the credit, not the person who prompted it.
This is basically On proof and progress in mathematics by Thurston restated. When Thurston wrote it in 1994, many people didn't understand what he is talking about.
And already on page 2 Thurston cuts the the root of the issue, 30 years ago:
>On a more everyday level, it is common for people first starting to grapple with
computers to make large-scale computations of things they might have done on a
smaller scale by hand. They might print out a table of the first 10,000 primes,
only to find that their printout isn’t something they really wanted after all. They
discover by this kind of experience that what they really want is usually not some
collection of “answers”—what they want is understanding.
The difference in reactions by the maths community should probably be spit up into those who have read and understood On Proof and Progress and actually thought deeply about why they do mathematics, and those who haven't.
Tao is not the one who needs to worry, but it bears pointing out that Tao is probably one of the most famous archetypal "problem solving" mathematicians in history (though at that echelon all mathematicians are both problem solvers and theory builders, in the same way that Chess Super GMs are all universalists).
The people who need to read On Proof and Progress are the undergraduates and PhD students who have ended up down an academic pathway without looking up because they've always been good at proof and understanding mathematics, without ever pausing to ask why.
> The difference in reactions by the maths community should probably be spit up into those who have read and understood On Proof and Progress and actually thought deeply about why they do mathematics, and those who haven't.
I think David Bessis (in an essay from April) has the best thoughts on this since Thurston:
501 comments
[ 2.6 ms ] story [ 144 ms ] threadEverything else is secondary (or the last of our priorities) and would be better automated?
This is a hard pill to swallow
It’s more likely that instead of spending a 100K/year direct grant on two PhD students, PIs will hire 1 and have the student spend 50K on AI.
There will be no gap in understanding. Now there is because the models are discovering things at the edge of what they can do and so suck at explaining it.
AHM Statement on OpenAI's October 6 Release of Mathematical Documents
https://news.ycombinator.com/item?id=50000421 / https://news.ycombinator.com/item?id=49999159
Of course it's good to have the discussion... So maybe, we listen to the nay-sayers, but defer judgement on the matter... That's wisdom.
If I understand Tao correctly, he's saying that's going to have to be the focus going forward. I just default to thinking the models are going to be much better than us at that, too.
I wonder if this is true. The code produced by these models are not really getting any more elegant over time. On contrary, the models seem to be getting worse, often proposing really baroque architectures. You can use RL to optimize for correctness, optimizing the vibe seems much more difficult.
1. Some present a unified line that the whole point of their craft is the human experience, and that automation is the antithesis of that. Marathon runners don't care that a car can get there faster, poets don't care that Poem Bot 2000 can write poems too. I think this is smart if you can credibly take this position. The difficulty is mostly convincing the buy side, which requires being very outspoken about your views.
2. Some appear to be undecided, with one faction taking the pro-human stance and another rushing to accelerate things with AI. A good example of this is mathematics, and I really wonder where they end up in the long haul. They have a very good claim on #1, because mathematics is pretty close to an art form and is robustly insulated from the pressures of the marketplace. But they can also choose option #3, below.
3. Some crafts prioritize results above all else, practitioners either rushing to extract as much money as possible before it all collapses, or believing that they can out-prompt everyone else forever and that their prompting skills are indispensable to their employers in the long haul. That's software engineering. I think this is going to be interesting to watch.
I'd end up with the buckets
a) Fully human
b) Hybrid human-AI
c) Fully AI
With respect to Tao, I think this is a very naive / short-sighted perspective. It comes down to aesthetics of your trade. You can argue that there's beauty in humans doing it. Or you can argue that there isn't, in which case, it becomes a race to the bottom as models get better and the value of your contribution becomes more and more dubious.
I don't think there's a functional difference between that and a coder saying "you see, vibecoding is a synergistic relationship, they will always need a human in the loop". Or an artist saying "yeah, I'm using AI to generate artwork, but they will always need my sense of taste". They won't. That's just a signal for labs to fine-tune for the thing you identified as your special insight.
You say marathons are just for fun, but prior to the wheel it was the only way to get around. (Other than horses in some places)
So it’s not that running is immune to automation, it’s that we’re already post-automation and that only people doing it for fun are left.
There's a reason there are maybe a few hundred professional marathon runners in the world vs tens of thousands of professional mathematicians. Bucket 1 is basically an "amusement for the upper classes" type of deal. Any field that goes in that direction would have to shrink down massively.
It also devalues the field in my opinion from something really profound with actual impact in the world to a somewhat vain leisure activity. (basically going back to gentleman scientists) But I know other people would see it exactly the opposite way.
I don't buy this. I think there are other reasons why "professional marathon running" is a niche thing; it's probably just that it's not all that interesting to most people to practice in function of what it demands of your body, and not that interesting to patronize / watch.
Take woodworkers. There's probably more professional woodworkers than professional mathematicians. Everything a woodworker does, a machine can do more cheaply and more quickly, and the craft was definitely impacted industrialization. But the reason it survives is more or less because some craftsmen convinced themselves that there's merit in doing it "the old way", and they managed to convince a decent percentage of customers of the same thing.
And you could argue the same thing you did for marathon runners: custom furniture is just amusement for well-off people. Sure, but there's enough people with money to keep it afloat.
There seems to be more interest in hitting some arbitrary benchmark (we proved X unsolved problems) than in genuinely contributing to mathematics. But what else is to be expected? It's become a maniacal race with too much money. Too much effort is being invested in proving that the exponential curve is still holding.
OAI obviously have a fiscal incentive here, but to presume a year from now we won't see improvements and more succinct work on the results coming from models?
If a random person is given a 60 page proof to digest and not the author, those hidden insights that _aren't_ in the paper might be completely inaccessible. Maybe the AI will "just" be able to provide the insights. Maybe. But pedagogy is tricky work, and despite these AIs being able to do all this fancy math we can't get them to write good cover letters yet, so....
Ultimately we might be left with just a bunch of intellectually unsatisfying proofs. This means way less drive to simplify the proofs or rework them.
End result: we generate a layer of "less efficient" mathematics, that won't get built upon. We will not actually have any shoulders upon which to stand.
The concern is being raised without evidence, because the evidence points to the gap simply being frontier models have just started to be able to get a raw proof out. Why, given existing progress, should we expect them to be unable to distill insights from those proofs?
Certainly this even more likely doesn't matter at all for applications: if I can send a radio signal further because my AIs design it a certain way, that's an unambiguous result. Which is really the next step here: turn a proof into a "mechanical" application.
The fact that they did not do so can only mean that either (1) their agents currently lack the capability to do it, or (2) OpenAI are completely indifferent and do not care in the slightest if the proofs are understood or not.
Come now, this is kind of unreasonable. When you're working on a new technology, you first get the ugly, inconvenient-to-use prototypes functioning with the core new thing you need; then you work on packaging it up into a format useable in production. I'm sure the very first digital camera sensors weren't very useful for photographers either; but it isn't really even possible to build the rest of the technology required to turn raw output of a digital sensor into something a professional photographer can use until you have the raw output itself.
The research is still on going on the raw output; getting things to the next stage, where the results are widely useable by professional mathematicians (and then on to engineers and scientists to whom the results would be practically useful), is a whole new research area.
Like, "Orr... maybe they care a lot, but haven't gotten to that part yet?"
And it's not like this is something where we're loaned some top math genius for a limited amount of time and we have to make the most of it. Rather, this is a new high water mark. The accessibility of the results is no longer scarce. The scarcity has shifted, and that's where the focus of the math ecosystem should shift as well. And it doesn't help for a frontier community to saturate and take over messaging pipelines that were typically managed by the math ecosystem. It's not about "stay in your lane" but rather "we need coherence and be careful not to break the system."
Just two cents from someone who could screw up basic cashier math on any given day.
Those few thousand mathematicians are now getting a taste of their own medicine. After all, it was people with extraordinary mathematical talent who developed machine learning and large language models, leaving hundreds of millions of people who earn their living through speaking, writing, or teaching worried about their future job prospects.
Still, I believe almost everyone will be fine. Perhaps AI will also prove good at coming up with new conjectures, and some mathematicians may shift towards applied mathematics or other sciences.
> genuinely contributing to mathematics
What's the difference between the two? Proofs are no longer the goalpost?
One is that AI will continue hallucinating in a manner that is not easy to verify, second is that AI will not be enhanced to produced more simplified amd robust outputs, and third that a human will be required to do that. What humans in the loop are doing now is verify the process, propose shortcuts and add legitimacy, through the verification process, if that ends up being succesful its highly likely a lot less mathematicians will be required in the future.
The conclusion that this is not productive focuses on the mathematicians, but it is very productive in terms of hundreds of proofs being produced that had previously consumed uncountable hours of the brightest minds. Unless it ends up being the greatest hallucination ever ofcourse
There is literally not a single shred of evidence to indicate either of your supposed eventualities. The core technology of an LLM is sampling from a distribution so there is literally no way to make it deterministically robust (only probabilistically).
What has been demonstrated is a process that outputs lean proofs based on those probabilities. This happened after decades markov chain producing garbled texts and very shortly after gpt2 producing stories about unicorns.
An LLM mostly deterministically (except parallel processing nondeterminism that can be mitigated) produces a probability distribution that can be sampled deterministically: just take the highest probability token or use beam search.
You might have a point if the goal was to have LLMs that spit out a correct proof without chain of thought or tool use. LLMs + agent harnesses are more than capable of self verification and course correction.
Hold up, that's an even bigger assumption in the opposite direction, and I don't see anything to support it.
At least in terms LLMs getting all the "AI" hype these days, there is no structural/mathematical reason to believe they won't continue to have the same problem they've always had of generating plausible text over rational text, and I don't think anybody even has a clear idea how it could eventually be accomplished.
I've seen "then the magic singularity occurs and somehow it solves the problem for itself", but I would classify that more as mysticism than engineering.
Two years ago, hallucinating that the code worked or that a task was accomplished was a common occurrence.
We have seen that now agent swarms across thousands of agents can coordinate to achieve a result.
Clearly hallucinations are no longer the problem they once were, since now we can get working results for long horizon tasks that require massive compute.
Consequently it would seem unwise to assume that current limitations will remain as they are and prevent LLMs from coming up with solutions that they can explain to humans.
This is just a fact. I'm sorry if it messes with your narrative.
https://arxiv.org/abs/2401.11817
And yeah I get hallucinations all the time still. Maybe it's because I'm working on harder/more niche problems (like a compiler with an unusual type system), but it happens quite a lot. I don't record all of them.
Although the most common one you can find is them misattributing the source of changes from themselves and also other agents (Fable, Opus 5.5, deepseek, whatever). They'll say "your changes" or "you changed" or "your ruling." I didn't decide anything and it's in their own chat log, and yet...
It is an old saw at this point, but what an LLM does still cannot be divided into hallucination and non-hallucination. This is literally an anthropomorphism trap.
Layers and layers of application-specific verification can reduce the risks inherent to LLMs, to a really remarkable degree, but nothing about what these tools are suggests that this problem will go away; it will just bubble up again somewhere else.
An old saw unless something that's widely accepted, but sadly it seems that many people don't recognize this, even many people working in the field.
For all that I saw over the last few hundred hours with AI on software engineering, hallucinations are no longer a problem at all.
Not once have I seen a task fail due to what would have been a "hallucination". If they still occur, they can apparently be detected and corrected automatically, or are subtle enough to escape notice with presumably no significant impact on the results.
Why would this not also be the case for mathematics?
"Test suite passed" when it actually errored? Obvious hallucination, unless it ran a command that returned the wrong error code.
But is running a malformed command that does not achieve the expected effect itself a hallucination?
I'd say a hallucination (very misleading word) or confabulation or "making shit up" happens when an LLM uses factual / evidential language purely based on local statistical expectations of the text, instead of it drawing from actual evidence in its context pointing to it.
This is murkier in the case of general knowledge questions, like when and where was some famous person born. It may then be a spectrum from fully making something up based on how the name sounds, all the way to confidently retrieving it from its weights correctly. In between, we can get hallucinations. But newer models are taught to use Web Search when unsure, and it works pretty well, though not perfectly. I don't see any fundamental limit here. It's just not perfect. Trying to solve "the hallucination problem" is basically like saying "our dog vs. cat classifier is pretty good already with its 99% accuracy, now all we need to do is the tiny little task of eliminating the 1% error, and we will be golden". Like, no shit, there is some error yes. People are working to reduce it. It will never be absolutely 100%. It's not an insight to say we should remove hallucinations.
I agree that hallucination isn't some kind of "different" operation than "normal". It's not like when a train derails and you can point to it. It just operates as normal and sometimes that yields correct factual outputs, sometimes not. You don't have to metaphysically ascribe any kind of intent to it.
I'm not sure how people conceptualize these things who weren't doing classical machine learning before all this. To me, "hallucination" is shorthand, and we know it's not like humans on drugs or something. It was used in the literature also for any kind of generative imputing of missing information from a learned prior. For example in image inpainting a GAN "hallucinates" the missing part of the image. This terminology was already used in the 2010s and probably earlier. Or in image colorization of grayscale photos, the model "hallucinates" the color information.
Then the word escaped into the mainstream and people have weird connotations about it.
To an arbitrary degree.
Just like all of science. Reduce the error to the desired margin.
Where does this "will probably not hold in the very near future" come from? People correctly warn about extrapolating current things onto the future, but then just throw some vague "probabilities" without providing any argument why their "probably" is somehow more grounded than others.
This means when writing documentation, tutorials or commit messages, their output is often a garbled jumble. Assuming shared context, using invented terminology without explaining, leaking conversational states due to improper epistemic boundaries and failing to model the reader. This all usually leads to their freely generated explanations being terrible. Getting good explanations requires chaining questions that force them to line things up properly, which is not easy the less you know. These failures as something LLMs naturally struggle with make sense, given the nature of attention and RL with weak signals from human data.
Math is not merely a collection of proofs, it's a way of understanding. A proof presented in a manner that cannot be incorporated remains useless. It does not make it's way to physics like Riemannian geometry and matrix math did. This is no less true when done by humans too.
Your hallucination conclusion, checking if a proof is one, is exactly the counterproductive cost.
Most of us cannot verify that the claims in the OpenAI lore dump are in fact all correct. It will take tons of work from experts to do this. It took subject expert mathematicians to identify the discrepancy and disconnect in the Navier Stokes proofs, for example. LLMs will struggle to make use of their own proofs or turn them into knowledge that accumulates over time.
The act of proving is often more valuable than the proof itself. Human constraints and limitations force us to invent tools and abstractions that a 100,000 x 1M context swarm can bypass. The tradeoff from that AI swarm advantage is work that doesn't usually lend itself to being built upon. It's like doing all the side quests and reading all the books of an RPG versus min maxing a straight path with a guide. We might try to identify new abstractions, but the fact that we don't get access to CoT and that much of it will be illegible means mining LLM traces for what human mathematicians produce naturally will be a tedious chore.
This is a feature, and a huge step forward.
If you expect AI to do serious work, you can’t have it guessing what you “really meant”. Every sufficiently advanced task depends on very subtle details in the problem statement, and the correct default behavior for advanced AIs is to solve the task exactly as stated, unless a system prompt or other constraint tells it to do otherwise.
Here's a longer article which goes into a bit more of the details: https://terrytao.wordpress.com/2026/10/05/the-future-of-math...
Are they independently wealthy? Or do they have a deal with their local supermarket that they can take food for free?
To slightly deflect your question, what do you think will arise naturally?
I’d put my money on feudalism if SOTA model providers can stay far ahead of open weights models. However if that proves infeasible and everyone ends up having access to local AGI does that serve to level the playing field? What resources are still going to be scarce in the coming decades?
This is a really interesting point. This is almost analogous to a metric in software engineering like lines of code which is a proxy for something we actually want to measure.
In a way, the SOTA model providers are cargo culting what appears to be what mathematics is.
This is a transitive period. In a few years, verification and exchange between model instances will happen faster than humans can follow. Human input will be an ethical question, and not a productivity one, because it will be the bottleneck in any science.
If the focus in placed on maximizing some easily measurable output on a narrow perspective, situation is unlikely going to match a sweet spot of holistic equilibrium which is maximizing harmony and happiness through humanity as a whole.
In example of go where I'm more familiar Google deep mind poured large resources to get a super human performance first, establish superiority and abandon it. The community then built their own tools starting from reproducing their papers.
I think similar thing might happen to math. Nobody outside of math cares too much about Hamiltonian cycles in some bizarre graphs or proving lower bounds on complexity of some problem.
Once those results stop being worthy of mainstream media attention, they will abandon math and the progress will be done by mathemicians guiding the models and the community will likely establish some new rules about what makes a valuable contribution. Merely solving not yet solved problem might not be it anymore.
It is also fascinating, because I don't think there is any solution within our existing system, at least not any I know of. Theorem ownership is not a good solution (and neither are patents in general). Probably the most achievable (or rather the least unachievable) solution is a kind of communist utopia, where people can dedicate their time to a pursuit of any endeavor they see fit, as resources for a decent life are abundant and excessive power capture impossible. (The other option, somewhat dystopian, and which would not require humanity to change too much in its current mode of conduct, would be a totalitarian or caste-like capture of society by the scientific community.)
Incidentally, if AI proves as powerful as some expect it to become, it could bring about another solution of that issue by making all human science and mathematics obsolete, pushing its true market value to zero.
(With apologies for rambling.)
I wonder if it makes conceptualization simpler for models too, given that they're trained already on human-speak. And I'm also curious as to whether humans currently have an innate advantage into simplifying and contextualizing proofs, or will the machines get good at that as well?
I was bitter about that back in the day as I hoped for more answers, more matches, more "truth" about chess being shown. Soon after that community project Leela Chess Zero was started and not only surpassed original AlphaZero but added few hundred ELO points over it. Then the combination of NN and classical engines happened with NNUE and current Stockfish is again a few hundred ELO points stronger.
Today we pretty much know the truth in chess for all practical purposes. Human analysts/preparation experts focus on finding interesting path and opponent profiling (what is the most unpleasant for the opponent to face). They don't look for truth anymore. The game is doing great, it's more popular than it ever was.
https://en.chessbase.com/post/the-full-alphazero-paper-is-pu...
The result was that Stockfish lost many games by walking into known bad lines and lost way more games than it otherwise would.
> We also played a match that started from the set of opening positions used in the 2016 TCEC world championship, along with a series of additional matches against the most recent development version of Stockfish, and a variant of Stockfish that uses a strong opening book. In all matches, AlphaZero won.
https://deepmind.google/blog/alphazero-shedding-new-light-on...
I am not claiming AlphaZero wasn't stronger. It wasn't as strong as the PR piece suggested though and we have never seen the games being published. In chess this is extraordinary because basically all games in chess are publicly available - both human and computer games. Claiming "we have created a strong engine that has beaten Stockfish with opening book" while not showing those games (or details about opening book used) is akin to "we solved this math conjecture" without showing any kind of proof or argument.
Publishing a few 1000 of games costs nothing. Tens/hundreds of thousands of games are published every day.
But, I do think you are right that there will be some level of moving on. The spotlight is currently on maths and that won't last. It will move to some other area where there is more impact to be had. So while they might shift gears and put less focus on math, it will always be there as part of the portfolio.
I definitely think that this is marketing, just "with good side effects". My doubt is when they will be able to move to "marketing with better side effects", that is, research with more concrete outcomes (health, materials etc.).
Problem is, that type of research is much harder. Some doubt that progress in such areas will be quick (https://www.noahpinion.blog/p/wheres-the-intelligence-explos...).
While top AI labs no longer focus on chess, the community build way better chess engines.
Stockfish is probably stronger, than everything the top labs build.
Wouldn't we expect the same thing for math? That slowly the broader math community would engineer a harness/program... That will surpass the current labs, and be a community ran project
In the case of chess this seems fine, there isn't much value to society in creating an AI capable of beating top humans with a 4 pawn handicap rather than a 2 pawn one, but for maths where there are actual applications it is more complicated.
You can believe they believe they are telling the truth or that they have good intentions while still being skeptical of the effective trajectory.
We are at the point where the way in which humans do math and science changes significantly, and I have no good idea at all in what state is it going to settle down. But you are one of (many, I suppose) people exploring the new wilderness, so I wish you best.
Unless they decide that trying p!=np is worth any money.
We'd be nowhere without Laplace and Fourier transforms, Maxwell's equations, elliptic curve cryptography, and many more.
Most math doesn't, but often these techniques are invented first and the applications come later.
And the criticism of the current round of proofs is that while they may be true - likely for some, questionable for others - they're not adding new techniques or insights.
There's a great paper from Abraham Flexner on this topic:
https://worrydream.com/refs/Flexner_1939_-_The_Usefulness_of...
It argues exactly that we should be allowed to pursue the seemingly "useless" knowledge.
Previously discussed on HN:
https://hn.algolia.com/?q=usefulness+of+useless+knowledge
However, there's some of this that's a proxy - the compute to solve these problems was very low (they claim a few hours of thinking time on a regular subscription). The large cost would have been the training and if training the models to be better at these things makes them smarter for useful tasks that's beneficial. I believe there was work done earlier on around showing that training the models on code made them better at broader reasoning tasks (not just writing the code itself).
Another side is that if one goal is to improve the models themselves, their ability to work on mathsy problems must be high. That has very direct business value, and ideological value depending on what you think the motivations of the people running the companies are.
There are architectural advancements yes, but lots of progress from LLMs really come from (1) better pre-training [generally through more cleaned data, and ofc more data], and (2) lots and lots of post-training. It's how we get more and more intelligent models for the same param sizes.
The 'marketing' is just a useful side effect they get from their RL rollouts on maths and LEAN.
Being able to present useful novel ideas would likely generate a lot of press, for a while. I don’t know how this would look since I’m useless at math, but Im sure there are plenty of unknown problems with massive implications, that once formulated can be solved.
Pretty much the only enterprise that historically pays some mathematicians handsomely is quant finance, but those people are actually compensated not for proving theorems but rather for statistical modeling and programming skills. And even that industry is so technologically driven these days that pure research mathematicians no longer hold a clear edge over a strong programmer with fluency in undergrad level probability and statistics.
The $$$ they're pouring isn't just for marketing. Think of these papers/results more as "useful side effects" from large-scale RL rollouts and post-training. Every token being generated contributes to post-training in some way.
There isn't a hard boundary between "training" or "inference", modern post-training is arguably inference-bound :)
See: Learning in High Dimension Always Amounts to Extrapolation Randall Balestriero, Jerome Pesenti, Yann LeCun https://arxiv.org/abs/2110.09485
I guess you mean by "interpolation" that it's some kind of nonlinear combination of the training data, but that's an almost vacuous statement. Any input-output relationship has to be so by definition.
Ah this is an enlightening point. 8 hadn't thought about it this way, but you're right.
-- Me
Go was a specialized application. All the math results come as a side effect of reading the whole internet, and it will keep reading the whole internet. It will keep practicing thinking questions. Actually, math might be one of the best ways to keep them contemplating and measure their contemplation abilities, so math will always stay in the loop.
Also, math might not be useful just for humanity, but also for AI, so the system might actively benefit from new math results itself. (Not sure if any of the recent proofs qualify, but future work might.)
To take this to the next step, what happened after deep mind pretty much solved Go is that they started looking for the next set of things that hadn't been done yet. It does strike me as very likely that this will follow that same path.
Never has been.
See: Mochizuki's "proof" of the ABC conjecture [1].
In short, mathematics is an art of storytelling as much as anything else.
Writing a boring story, much less an incomprehensible one is worth exactly nothing, and doesn't matter.
Is it cool that the machine can connect the dots that people put out there in a form that happens to be easily ingestible?
Yeah.
Can the machine answer why give a fuck about a result like that?
Nope. That work has already been put in by people.
The most obvious impact of this BS is that interesting problems with potential to be solved, which were already a scarce resource, will no longer be shared.
It's already become a huge issue without AI being involved, due to the rat race nature of academia.
There's even an industry term for when someone gets a wind of another person's progress on such a problem, and solves it before them - it's called "getting scooped"[2].
It doesn't mean that they got access to the results. It can be as simple as e.g. an experienced mathematician hearing about a problem that a PhD student is working on for their thesis, and scooping it.
Honestly, as a mathematician-turned-software-engineer, I see and hear both sides, and both sides are lying:
* The AI companies are lying that what the LLMs produce is "mathematics". Writing a proof, even a correct proof, is not what mathematics is about. The Fundamental Theorem of Algebra has been "proven" incorrectly four times IIRC, but it was mathematics nonetheless.
Mochizuki's "proof", on the other hand, may be correct, but over a decade later, it's still not really mathematics, because transfer of insight is a fundamental component of it, and insight is a thing that the AI, presently at least, doesn't do, no matter how many proofs it produces.
Proof automation has been a field for a long time, for that matter; hyperscaling it doesn't change the nature of things.
The lie is also that a correct proof is the final point. The Four-Color Theorem[3] was proven in 1976, yet people were still proving it 20 years later, trying to reduce the number of cases to leave to the machine, even as we got it super-duper-verified by theorem-proving software in 2005[4].
The writing was on the wall since computers were invented, so why are mathematicians making Pikachu faces now?
That's where the other lie comes in.
* Mathematicians are lying to everyone else by suggesting that the technology is the problem, or that the issue is with how the technology impacts people, and not with the COMPLETELY, ABSURDLY FUCKED UP ORGANIZATIONAL STRUCTURE OF ACADEMIA, and I can't scream it in bold enough caps lock.
There's a reason websites like "100 reasons to NOT go to grad school"[5] exist, and all who've gone through the masochistic 5+ year journey can attest that actually, the reality is worse.
Closer to the point, there's a reason Grothendieck burned most of his work and fucked off from mathematics in the 90s, and Perelman did the same, in effect, in 2000s[6], while putting a million dollar weight to his statement that MATH SOCIETY IS FUCKED.
To all the mathematicians running in circles screaming AaaaAaaAAAaa, Perelman can now say: told you so.
See, after we discussed what math is about (insight transfer), let's do a peculiar little observation that this is exactly what's rewarded THE LEAST in the field.
"Teaching jobs" are a punishment for those who don't do good enough, and doing good means cranking out proofs in a publish-or-perish race, emitting just enough for others to verify correctness, but not too much lest you don'...
Given sustained exponential growth is mathematically impossible to maintain with fininte resources, it's funny to me they're using advanced mathematics to try and achieve this.
The amount of Confluence pages of "research" that is just a dump of LLM output someone passed to me to review is staggering
I hate this approach, it's unbelievably selfish
Mathematics will revert to it's main practice, which is to study.
There are lots of weird panic reactions by some prominent problem solvers. See for example the ridiculous cease and desist like statement of AHM shared at Tao blog.
Put this Math 1-2.0 with that AHM statement together and you'll realize that this is a power struggle and that you see only one side of it.
Mathematicians have very diverse opinions about this. I, for example, am for as much as possible automatic harvesting of all these "low hanging" fruits. Should be disclosed as soon as possible, free of any bottleneck, and citable. The mathematics community may do whatever its various members desire to do with these results. Let them decide individually what to do with them. This AI tool is here to stay.
What work do you think mathematicians do normally?
Like they sit whole day and have ideas? And where are the ideas?
The way I see it, _some_ mathematicians enjoy solving puzzles, and now AI is better at solving puzzles.
This does not affect people building new theories.
Also, it's quite prestigious to write a _book_ on some topic. And guess what writing a book entails? Refining and expanding. What you call grunt work.
OTOH a marginal value of a single paper in a paper dump is basically zero. Nobody gives a flying fuck if they made 100 or 200 papers. It won't give them even a single customer. If they released, say, 5 paper per month it would generate a buzz but not as much backlash from mathematicians.
Actually, the problem is, somehow the skill of building new theories in math is directly tied to slaving hard over a problem. It's the very experience of slaving away that actually somehow causes ideas to form. Pretty much all mathematicians understand this. Yes, senior mathematicians now can form some new theories, but what about junior ones who will have very little experience in working hard on a problem by hand?
Of course, they could work on the problem by hand anyway, but they won't because no one will pay them when a machine can do it.
Let's start with the fact that mathematicians aren't paid for results. Institution which gives them salary fundamentally doesn't give a fuck about theorems. They might care about having top-grade mathematicians for prestige, or because they believe that countries which are "good at math" also do better in science, engineering, etc.
> somehow the skill of building new theories in math is directly tied to slaving hard over a problem
We don't know if that's the only way. Perhaps collaboration with AI is just as good. Why reject it before it has been tried?
> what about junior ones
Junior guy with brilliant new ideas might benefit the most from AI as it can compensate for lacking technical chops and breadth of knowledge.
I feel the same way about academia, the papers, the citations, the ego, the narcissism and the taxpayer codependency that got cut off and turns out wasn’t necessary at all thanks to a private sector entity running laps around them
I don’t feel that academics need to pursue the discipline and distributed brain-wracking that has sometimes resulted in the solved math problems, just because more times they find other nooks and crannies to explore along the way. I think the blueprint is enough. Standing on the shoulders of giants is good enough.
and if the concern is that they can’t figure out what to do with a proof, next year’s AI will
Many of the best startup ideas by the best product and engineering minds failed to gain attention and funding. Same with much of the best music - relegated to hard drives with derivative ideas only resurfaced decades later
I would expect much of the recent math dump will be leveraged by other LLM-driven research teams rather than read in depth by a human
I tend to agree with this, but what is the alternative? Should OpenAI and Anthropic employ hundreds of mathematicians to do this work? Should they just not solve math problems within their reach?
It's unclear what more could be expected than releasing the presumably already verified results and write-ups for each problem. Should they run a mathematics school too?
Then the comment goes on to argue AI labs were not interested in actually advancing mathematics, and that investments into AI were manically excessive.
IMO none of this follows and demand is there to justify the investments.
The comment then goes further to argue that AI labs were putting too much effort into pretending there was exponential progress rather than actually making progress.
The factual basis for this claim seems to be that OpenAI released math results and write-ups, and it's not even clear what more they could do on that topic.
That's a very negative opinion.
So yes, that is exactly what they should do. Alternatively, if they are too lazy or incompetent to put in the effort themselves, do what AGMAI proposed and fund a third party to help out.
Hm, kinda reminds me of my college days. "Proof trivial, left as home work." was a sentence my Profs loved to say.
The more information the better.
The entire purpose of published work is to remove noise (and perhaps incentivize work through attributing credit).
This information is now out there. You can choose to ignore it if you wish. You may just find yourself a century behind in research.
And on that point most of this research has been looked at by their mathematics panel and comes with lean certificates, it's not exactly noise.
This to me is more the old guard not willing to let go or change their ways.
I think a problem is that math seems like a deeply toxic, ego driven domain.
I think he argued that e.g because the navier stokes millennium problem ist considered solved now, you won't get any recognition for being the first human to solve.(How would you even proof you solved it yourself and not just regurgitated the ai proof?)
And since recognition is the main objective, noone would spend time on dissecting the proof, and perhaps finding some unique approach to solving the problem, that could be transferred to other open issues.
And therefore the problem is now "poisoned". Since it's assumed to be solved noone will research it, and the potential revelations won't be found
In what world is OpenAI not “genuinely contributing to mathematics”?
I’m getting whiplash from the speed at which people are suddenly accusing them, and AI in general, of not doing enough.
But you are right, this is not OpenAI's "fault". The problem is - as others have said recently - that many people in mathematics want recognition for solving open questions more than they want the answers to the open questions. Everything about the economics and social environment of Mathematics will have to change.
I think that this is exactly the same split we see in software: there are those who mainly enjoy the craft aspect of building software, and are uninterested in the product or business they are supporting. Others are primarily interested in the production of useful software or building a platform or company.
I've always been in both camps myself. When it became obvious that AI was going to destroy the craft aspect - at least two years before it actually could do so - I became very discouraged, even depressed. But once it was actually good at building software, I became very excited about all the stuff I could now build. Sadly, I think a lot of people in our field have never had something they really wanted to build.
So effectively, stuff got proved, but people don't really understand how, so it's mostly fucking useless and done for OpenAI's marketing team, while also pissing off the maths world at large.
That community won't exist as many people simply won't even enter the field because it's reprehensible and contemptible, not to mention boring, just to read machine-generated proofs and verify them.
Collaborating on, or at least working on unsolved problems is what motivates most people.
AI is like a cheat code in a video game. You get to the end faster but fewer people want to play if the cheat code is always on. You can't turn it off either because the very challenge is to do something unique.
This is code for "they keep paying us".
For what?
If math is just about having a community of other mathematicians to hang out with, it still isn't a career. Nobody is paying money you need in order to to eat, just to hang out in a community.
Just like a software engineer, "Well AI can write all my projects now, but I have my local Rust Users Group to hang out with". Nobody is paying me to hang out and hand code Rust.
So you'd prefer if they kept their work secret? Or you don't want them working on these problems at all? Or they should be required to do the work the way you want them to?
I'm not clear what you see as a better option than dumping.
It’s causing unemployment but no real gains unless you have a lot of equity and can benefit from lower costs from fewer people working.
But yes, I agree, there needs to be a solution for the fact that we don't need to work anymore. And it needs to be more like The Jetsons than 1984.
We'll probably have to revolt to get the changes we need. I don't see Elon giving up his power without a fight.
I don't know that the labs expect this at all.
Is the purpose of math to move human knowledge forward or to appease the mathematicians who get their rocks off working on proofs?
I assume this will increase the job market for mathematicians though right ? Tens of thousands of dumped proofs will mean math majors will need to be hired for review.
The job of professional mathematician might be the first to be completely eliminated by LLMs, save for those who can make money from a patron. I am hoping they are able to figure something out to save their profession, as other professions could use it as a blueprint as AI comes for them next.
This really expresses the heartburn you see across all fields, not exclusive to careerism. I certainly have friends in decomp and fan translation spaces that have been demotivated by the current rash of efforts happening there.
The rush to be "first" has always been over-celebrated, but it would be nice to believe there's a way to get beyond that thinking.
This I don't understand, seems like an obvious thing to automate, especially for byte-matching?
I don't think it's motivating to solve a black box by having AI generate another black box if what you want is to understand how the thing worked.
No? Decomp sources are often full of comments that claim that there's no clear reason as to why something is done a certain way or straight up full of question marks. Byte matching often is a result of bruteforcing a solution rather than understanding the original idea behind the code.
Up until now the prize in pure (as opposed to applied) mathematics was the _understanding_ and the machine can't do that for you. What does it mean if we get "super powered alien maths" but humans can't do it? It's like inter univeral teichmuller theory all over again but imagine if Mochizuki was right and it came with a lean proof?
In my view, over the last century, math has turned into an intellectual analogue of extreme bodybuilding competitions. A navel gazing runaway optimization in making useless stuff just to demonstrate cleverness. That's fine, why not. But society has no obligation to fund that, just as it doesn't fund other extreme hobbies. Ideally if we ever get something like UBI, math can be still their hobby.
So make mathematicians proudly wear their flag colors and sing the anthem while lecturing to cheering spectators. Might work.
Maybe AI will take over some roles of doctors, but that's independent of curing diseases. Think about diseases that have a cure - do people with those diseases not go see a doctor?
Im not sure how that will work, but im convinced the current paradigm of just pushing agents into codebases for not much reason other than you can is going to make building software incredibly boring and push creative people away from the field and stagnate progress.
My prediction is software gets boring and building hardware projects will be the new frontier for creative engineers looking to push computing further. Which is probably a good thing.
Sort of. An elegant proof is useful beyond what it shows. It hints at new mathematics, and can prompt discovery in applied fields. I don’t think I’ve heard of elegant code leading to discovery on its own.
I’ve not heard of it either, but code is an abstraction of math, so I don’t see why this couldn’t theoretically happen.
Anecdotally, I’ve started spending time advancing my math skills beyond the early college level I stopped at and I’ve frequently found I already know concepts of more advanced math - I just didn’t know what they were called or how to apply them to an equation on paper, but I’ve been using them for years and intrinsically grasped the underlying academics.
Usually it's the opposite. "That's in prod? And it works? It shouldn't work and I thought it was doing something else. Why does it work?"
Every software design pattern came from elegant code. People wrote code, summarize code, learnt from code, and taught code. That's discovery
People wrote many books about software engineering, all from valuable experience from buildng expensive software systems. But in the age of AI, is there still anything learnable from generated code?
Personally I always ask AI to summarize its findings and lessons in a .md file. And I always learn something from it.
But could AI utilize some new patterns and paradigms I wasn't aware of? Very likely. Because we only learn from our personal grave mistakes, a summary from others gets neglected and forgotten
The S fell short in actual reality for the most part, as it was merely a hiring requirement. A hiring requirement that didn't even make sense, because the skillset of academic CS only marginally overlaps with the skillset one wants to hire for.
Material engineering at least for the most part has actual real-world applications where one can push humanity further. CS (as practiced, not necessarily the idea of real CS but the CS we got due to it being used as a hiring filter) for the most part is just self-referential spinning with mostly unclear results.
There is real impressive work being done in that field, of course, but I'd argue that the majority of it over the last decade or so at least was just performative nonsense.
Maybe by again allocating new resources to other fields, what hides under the label CS can become more pure actual CS again. I think that would also be a much less miserable experience for everyone involved.
The will to implement the stuff we learned, instead of letting dark patterns and churn for the sake of churn get the better of that :P
I believe in the future, we're going to see a similar shift in "programmer" - instead of a human programming the computer, you'll give the ai an idea and it will spit out a program.
And just like how automating the act of computation revolutionized what we could compute, automating the act of writing code will change the act of programming - hopefully, as you described, allowing us to do things that simply were not practical in the past.
Almost certainly not. It's just going to jump to a higher level of abstraction.
On the other hand, it puts a premium on resources. AI is not cheap for mathematicians. Folks are fancy universities in rich countries with forward thinking ministries of science will have an advantage over the rest.
What is clearly in immediate crisis is the traditional model of doctoral education. Most of the problems that were "given" to ordinary doctoral students are solvable (quickly) even by something like Claude pro. Mathematicians need to adopt training models more like what is done in experimental and laboratory sciences - collaborative and structured.
Where Tao is wrong is in regards to exposition. AI already writes better lecture notes, problems, and exercises for mid level undergrad math classes than do most of my colleagues. It's exposition is generally well structured and clear and it can adjust level on request quite well. It writes research better than most professional mathematicians too.
AI only take us as far as our imagination thinks to ask it. This can be exhilarating when new models drop every month and we can continually reach a new threshold, basically for free. But it is only a one time gain and ultimately short-sighted. Where I find continuous value is using LLMs to help my understanding, full stop.
I use LLMs all day long as a SWE and I have tried many approaches, but the most satisfying and consistent approach is to lean heavily into understanding a problem space and a solution space. Yes, it whips up architecture and code, but I spend most of my time peppering it with questions about the design and how it handles certain situations, what about this edge case and that security concern and this future product need. I have it write a report breaking down the feature and how it integrates with existing code and if the report is too confusing I have it simplify either the report or the code until it makes sense to me, sometimes scaling back the work to a more manageable state. I do all of this before I look at any of the code it writes.
The difference from this approach is that I am not suffering reading through 3000 lines of AI slop but I am reviewing a PR that I fully understand. I can eyeball it quickly for anything that doesn't fit my mental model and dig deeper or quickly revise it. Only after I am happy with the bones do I consider the meat and skin of the code.
What I find most concerning is how frontier AI companies all seem to have this Math 1.0 perspective that they only want to type "solve Riemann" into the chat box and have the magic to happen. It is the same problem Google ran into, where a simple, no thinking solution serves most of the people best and most profitably, so you fully ignore or remove everything else (boolean operators, exact phrase search, verticals, filters, infinite pages of results, "nothing found" if there isn't, etc.) But that choice leads to the situation Google is in now, scrambling to stay relevant. In a different world, Google would have continuously augmented their search capabilities and eventually built a smooth, guidable AI interface.
But no, we must only have an input box and a Go button.
Everything looks like a nail when you build hammers, sell hammers, have infinite hammers to play with however you like and your company mission is to build a hammer starship to explore the hammerverse, whether or not that is even possible.
And when the job is done, I run retrospectives on old coding-agent sessions to find areas of friction and confusion. I also journal with pen and paper, as it's supposed to bring cognitive benefits, to help me stay on top of things.
There is a world where we get to the edge of AI capabilities, and we build on top of that. As humans have always done with every new technology.
There is another more pessimistic view where LLMs just replace every human capability, and our economic overloads dont need us for anything and we just eat the small pieces of bread that are left.
This comes down to the fact of:
is human existence/intelligence just the simbolic representations we make in our brain? Or are they just a tool?
I tend to think of Godels incompleteness theorem as a proof that on the limit LLMs are useless. The real question for me is at what point approaching this limit becomes an issue, and if it has any practical co sequences.
We don't build on top of that. No need for us to. AI does. That's sort of the whole point of this endeavor is it not? Humans need not apply.
In my experience, every new model release allows me to go further, although every time i see every time the limitations, and i identify where i add value. And this value gets bigger every time.
But on the other hand, every model release reduces the amount of people that are able to value this "added value", because it requires more skills.
So we have the paradox that the added value i can bring on top gets bigger and bigger, but the perception of the economic value for the majority of the population gets smaller.
Take a look at this interview from two days ago: https://m.youtube.com/watch?v=oQypVVv1u1o
The interviewee is worried about the future of math research. He is not strictly worried about being replaced, instead he is worried that he will no longer be able to launder math-as-a-hobby through math-as-something-useful as is the case today. He lays out very clearly that grant proposals claim to have useful outcomes while the proposers know those claims are nonsense.
Business as usual in math, and frankly in all the other sciences, is to do research that furthers the researchers careers or personal interests and pretend that it’s somehow useful. This would be absolutely fine if it were privately funded, but it’s not, this is public money.
In every other endeavour, lying in order to get money is considered fraud.
We have collectively wasted a huge amount of taxpayer money and human time, entire careers, on things not likely to ever matter to anyone.
I look forward to science becoming automated so that we can have real progress instead of the current broken system.
Do you really think a society with zero human mathematicians or scientists will outperform one with both human and AI ones?
People thought, back in the 17 century, that imaginary were useless (except as a trick for some calculations). Turns out the research into these numbers back then is amazingly useful today, 300 years later, in electronics and such.
Publicly funded maths research should continue, even if some taxpayers feel it's a waste of money.
Note, I think the debate is mainly over what research should be funded, not whether any research should be funded.
In fact, the taxpayer should be funding fundamental research because it's so hard to justify profit from it - but that funding would benefit all in the long future.
So leaving the applied research that have commercial value be funded by private, commercial interests, would make more sense.
Thats great! So in other words it should be perfectly fine for AI to solve all of the supposedly useless math problems so that it might possibly be useful later. No need to worry about the mathematicians hobbies here.
How much..?
Sounds like Terence Tao would have said the same about Ramanujan who basically just "solved" problems without much explanation / reasoning / communication other than it just arrived from god.
In the case of Ramanujan, others took on the responsibility of socializing and community building knowing that he wouldn't do it himself. Why can't the same approach happen here?
There will be people who want to just "solve" math problems now that they have a new tool that lets them express themselves this way. Maybe the don't want to participate in the broader math community, etc. Why discourage them, or add friction / a barrier to them participating in their own way? Why not take on the burden of socializing, making sense of, and community building yourself?
There may be valid reasons here I'm missing, but to me this seems a bit like wanting others to approach a field in a particular way even though the field can support many ways.
Translating LLM proofs to human-ese is instead grunt-work. One that is bound to dissapear in a few years anyways.
https://arxiv.org/abs/math/9404236
>On a more everyday level, it is common for people first starting to grapple with computers to make large-scale computations of things they might have done on a smaller scale by hand. They might print out a table of the first 10,000 primes, only to find that their printout isn’t something they really wanted after all. They discover by this kind of experience that what they really want is usually not some collection of “answers”—what they want is understanding.
The difference in reactions by the maths community should probably be spit up into those who have read and understood On Proof and Progress and actually thought deeply about why they do mathematics, and those who haven't.
The people who need to read On Proof and Progress are the undergraduates and PhD students who have ended up down an academic pathway without looking up because they've always been good at proof and understanding mathematics, without ever pausing to ask why.
I think David Bessis (in an essay from April) has the best thoughts on this since Thurston:
https://davidbessis.substack.com/p/the-fall-of-the-theorem-e...
Also discussed here: https://news.ycombinator.com/item?id=48758048