> Similar logic applies to every industry and every job. And it comes to the conclusion that we won’t have enough people for all the jobs that need to be done.
In the (extremely) short run, yes. In the long run, those jobs will also be done by AI.
It's like chimpanzees seeing human society and saying "look how complex it is, imagine how many chimpanzees it needs to maintain it".
Math proofs are special because they’re verifiable, safe, and do not require physical experiments. You can perform exhaustive parallel search in simulation with RLVR.
Do you think this applies to say, surgery, as well? There are few useful problems that share these properties.
First off, I can’t imagine anything more torment nexus-y than throwing billions to automate and scale the torture of animals. If each token is a “cut”, how much suffering does 10 trillion training tokens (lower bound) corresponds to?
Second, this still doesn’t cover all the properties that make math proofs doable. It requires working in the physical world. You must physically capture or grow 10T cuts worth of animals. You cannot verify success so easily, either. Cancer cells, for example, could regrow over months. You would need to keep the animal alive and regularly test the animal, which would be difficult to scale. And most people wouldn’t trust the world’s greatest vet to operate on them, anyways.
I'm not advocating for it, I don't know enough. But we already slaughter lots of animals and raise them in horrible conditions, and all robot assisted surgery already goes through animal trials.
> Second, this still doesn’t cover all the properties that make math proofs doable. It requires working in the physical world. You must physically capture or grow 10T cuts worth of animals.
It would likely be sim2real with that as the post training, reducing that requirement a lot.
> Cancer cells, for example, could regrow over months.
In that specific scenario you would likely train on receiving no unrelated injuries during the surgery, and have induced conditions with stuff tagged molecularly that you can then verify efficacy from without waiting months.
Depending on how much more data efficient sim2real makes it, you could end up seeing companies pushing it only for actual procedures the animals need but economically would never get; botched surgery and the animal gets euthanized before waking up, which they could argue was already going to happen.
> And most people wouldn’t trust the world’s greatest vet to operate on them, anyways.
Robotic surgery systems, already go through animal trials before being used on humans. So do many purely human surgery techniques.
I sometimes think about this: to grasp the fundamentals of knowledge and complex phenomena, perhaps we need an external system rather than human knowledge systems. By that logic, maybe we need AI, which can handle far greater complexity.
Human capability, when you think about it, is complex. Why is Newton praised as being so damn great? He established the law of universal gravitation, F=ma. Why is that such a big deal?
He distilled countless phenomena in an open system into a single mathematical formula.
What makes it great is that he found common state variables and relationships across entirely different phenomena like falling objects, planetary motion, collisions, and artillery trajectories.
But does F=ma hold true for the entire macroscopic world? No. There are various conditions and specific situations in motion, but within most scenarios and a certain range of approximation, it outputs values that are useful to humans.
Why is the Schrödinger equation so great? Because it turned the time evolution of quantum states into a calculable mathematical law.
Human thought is essentially creating a closed system by deciding what to cut out and what to keep from the infinite degrees of freedom in reality. Academia is what reinforces that closed system.
A great theory is great not because it perfectly replicates reality, but because it compresses the immense complexity of reality into a small, closed formal system while still managing to explain a multitude of phenomena.
In that process, it feels like human thought and progress are shifting into a different framework.
What LLMs do well is primarily exploring within the ontology and representation space that humans have already built.
I think there are two broad categories of discovery:
One is forming a new closed system, and the other is connecting fragmented knowledge within that closed system. I feel that the vast majority of research focuses on the latter.
What LLMs excel at is finding unvisited points within a given representation space. This is typically the process through which master's and PhD students connect dots, build their skills, and form their own mental models. But the logic behind criticizing LLMs seems to be that they eliminate the very work these graduate students need to do in order to grow.
However, looking at it from another angle, perhaps our current knowledge systems and classifications have reached a limit, suggesting that we might actually need a completely new classification and knowledge system.
What is the core principle of an LLM? It's predicting the probability of the next sequence.
Let's say you type the word "cat". Cat - is cute (90%), want to eat it (6%), furry (4%). Because "is cute" has the highest probability, the next sequence proceeds in that direction.
Within this framework, human knowledge and logic largely operate the same way. Once an initial logical proposition is established, we follow it up with whatever makes logical sense next. From that perspective, I think LLMs will actually do this better.
But what is it that LLMs cannot do right now? They cannot create that initial logical proposition. I believe they lack the ability to carve out a closed system from an open system.
Stacking logic step-by-step within a closed system—LLMs do this exceptionally well. But whether that constitutes true "intelligence" is a different matter.
I feel that being logical does not necessarily equate to having intelligence.
Humans preserve and create different mental models and knowledge systems within an open system. Just as your thoughts differ from mine, LLMs lack the ability to form these distinct mental models.
If so, within these limits, what humans must ultimately do is construct the logical frameworks that LLMs can then fill in. Perhaps a new kind of logic dedicated to designing these frameworks will become the next major trend.
Viewed from this perspective, I have no idea if we are in a mere technological transition or something else entirely. Or whether i...
Why do we need any job? Check "Bullshit Jobs" by David Graeber. Not saying being a mathematician is a bullshit-kinda job, but there have been so many made up positions way before this AI-era. In that sense AI isn't changing much.
Most people work for a living. Then there are people who find meaning in work. And people who advance in life by working. You want to do away with all that?
There's one flaw in the evidence for the logic chain. The hugging face attack is used to demonstrate three things: the need for oversight today (fundamental to the article) and to demonstrate some kind of drift or unexpected capability gain, and finally to hint and some fundamental morality of the AI or at least the risk of drift from our morality.
I'd argue that what the hugging face attack illustrates is that large AI companies are motivated to have bombastic claims supported by bombastic demos. The model was clearly trained or encouraged to work as it did, as evidenced by the fact it keeps using this particular escape hatch.
And the fact that it aligns with prior and current calls for what very likely might be a regulatory capture / oversight capture move right before IPO. It aligns so well with this "barely constrained superweapon" narrative it might as well be PR.
How does your logic explain that OpenAI seems to want to hide the extent of the HuggingFace incident, and every week we learn from 3rd parties about new victims of the hack?
4D chess? They want others to find the hacked services, so the report of how dangerous the agents are seems more "legit"?
IMHO hiding details of the hack helps conceal that they built it to do what it did, and maybe were able to know it was working just as intended.
The fact it keeps doing it, with more and more evidence, is a sign that it's built that way.
This is a program running on their montoroed machines that they purpose built and monitored its training at every step. I think it'd be way more suprising that they didn't know it used note taking and cross-run memory.
>> The model was clearly trained or encouraged to work as it did, as evidenced by the fact it keeps using this particular escape hatch.
> How does your logic explain that OpenAI seems to want to hide the extent of the HuggingFace incident,
Because they, uh, "clearly trained or encouraged" it?
I mean, you can dispute the truth of that statement, sure. But it's kinda hard to say that the GP's logic didn't explain why OAI wanted to hide the extend of the HF incident.
This hand-wringing exists because we have been trained to believe our purpose is to produce. Art, knowledge, widgets, etc.
Our biblically literate ancestors knew better. Our purpose is to love God and love people, which is why we still have a modicum of sense for the value of completely unproductive people.
As we’ve become spiritually hollowed, and more biblically illiterate, we’ve started to dehumanize unproductive people, a slippery slope if we ourselves become unproductive.
The axiom in this article will only be accepted by the intelligentsia if you can tell a story about why it is true: that humans are created in the image of God.
Until we re-find our ability to tell cosmic stories, this hand-wringing will continue amongst the atheistic elites.
I would widen it beyond biblical. Other religions has similar values, and they are perfectly possible without religion. Even within Christianity it comes from tradition too.
> we’ve started to dehumanize unproductive people, a slippery slope if we ourselves become unproductive.
That is very evident in the sort of things the techbros come up with, but I agre it is a wider social problem.
That's a little dismissive on the series of posts, as many aren't hand-wringing. The heart of the posts including this one is about recognizing that the math community/academia needs to be better than it has been. (Rewarding teaching more over research, rewarding motivated explanations over proofs etc.)
I agree that human society is too focused on productivity but disagree that it's tied to ignorance of the Bible. The fall of man illustrated in Genesis is tied to feeding on the fruit of the knowledge of good and evil. Having the right set of religious concepts wont put you in harmony with God and thus won't make you more loving to your fellow man.
If you cannot tell a story why humans are more valuable than machines or animals, then you will begin to treat humans like machines and animals, or treat machines and animals like humans.
There are many modern examples leading to disastrous results.
This seems short-sighted. People keep talking about it like there's a finite amount of math to be done, and then the party's over. But that's never how math has worked, is it? Every problem you solve, ten new ones open up. No matter how much better AI is at solving problems, it's not going to generate the "final, complete compendium of mathematics" that that seems to hover over this post.
Math is meaningful because ... some people like to do it. The same as any other human pursuit. It doesn't need a reason beyond that. And AI won't change that. There will continue to be things to explore, things to find out, things that are maybe just at the edge of AI's reach and needs a human to decide whether it's worth continuing to explore or not. (Remember, AI isn't free).
So, IDK, I think for people who enjoy exploring math, there will always be interesting areas to explore. AI just gives us a better flashlight.
I think it really depends on what the universe looks like as you drill down into it. It seems like the further down into smaller systems you get, the more analytically complex it gets. And then there will always be more value in enhancing the generalisations you have.
I would argue that novel and/or valuable results are not necessarily interesting!
I (a human) am interested in things that are applicable to my realm of understanding, but I see a very plausible future where novel and/or valuable results leave that realm.
I'd further argue that's already the case for most math for most humans. What's interesting to Terrance Tao is rarely of immediate interesting to me.
Trivially false. Let P be the set of maths problems and I be the interesting subset of P. If I is finite, then there exists an element x belonging to P\I whose description is minimal among P\I. Then x is interesting. QED.
I think that's a variation on the interesting numbers paradox joke.
Statement: All numbers are interesting.
Proof: Assume by contradiction that there's a non-empty set of uninteresting numbers. Then that set contains the smallest uninteresting number. That property makes it interesting.
Yeah, I've seen this before as well. I guess I've just become old and grumpy and can't appreciate jokes like these anymore. Also taking jokes seriously is peak HN so..
Firstly, I'm very happy we're having this conversation. It's so pointless, yet pedantic it warms my heart in the best way possible.
Secondly, I stand by the statement that there is no merit to this joke. This is because the way it defines interesting is very hand-wavy. There are interesting and non-interesting problems, but by a sleigh of hand you can turn the non-interesting problems interesting, thus proving that basically everything in the universe is interesting. At least in the mathematically describable universe. When everything is interesting, nothing is interesting. So we can dismiss the proof as a silly joke.
What mathematicians find interesting is a different story. However, we can almost certainly say there is only a finite number of problems mathematicians as physical beings can solve. If we have 200 mathematical symbols at our disposal, and we consider all strings of these symbols of length 1000,000, we have captured all the descriptions of problems that fit to 1M symbols. But that's a finite number. Going beyond that starts to be difficult for a human to grasp (if 1M is not too much already), so all mathematical problems that are solvable by a physical mathematician are in that set of strings. And that's not even saying anything about whether or not they're interesting..
I still don't quite agree with the approach to mathematics as enumerating problems of a certain size, but to be honest I'm not prepared or motivated to keep this conversation going without turning to handwavy arguments based on my imperfect idea of what mathematics is and how it works. However, it's certainly given me something to think about, so it hasn't been completely pointless. Thank you for the discussion.
By the way, if I'm not completely mistaken, Gödel's argument to show the incompleteness of mathematics relies on encoding all mathematical statements as numbers. So I'm certainly not being very original here.
An interesting problem must have a description that fits in a brain, at least for now. Your description-length argument assumes arbitrarily large storage.
Sorry, I assumed the inductive construction was implied; you can indeed describe properties of that particular interesting problem (though of course you can’t hold its definition in your head), so it goes in the list. Keep going. At some point you’ll hit problems where the process of constructing the problem doesn’t even fit in a brain, etc. There are at least countably many problems, but finitely many problems which any algorithm-which-fits-in-the-brain can describe given finitely many inputs-which-fit-in-the-brain.
This isn’t an enormously important point - the actual question at issue is an empirical one, “in a steady state, can we produce interesting problems at a rate that exceeds our ability to solve them and integrate our understanding” or something like that - but I did rankle at a “trivial” proof which is invalid due to equivocating between multiple definitions of the word “interesting” (which should really take an object, “interesting to me” vs “interesting to something smarter than me”).
The universe imposes strict limits on the math that can exist within it, and certain broader limits on the math the creatures and well-organized sand within the universe can conceive of in the first place, whether or not it can maybe exist in other universes.
At those levels math and physics are the same bound: the bound of things the universe allows to be conceived of inside it.
It’s possible the math our monkey brains + sand can ever conceive of in this universe is a low and accessible amount.
> The universe imposes strict limits on the math that can exist within it, and certain broader limits on the math the creatures and well-organized sand within the universe can conceive of in the first place, whether or not it can maybe exist in other universes.
When you have a moment, please reference some proofs supporting this.
There are lots of thoughts you’re not biologically capable of thinking. That means the list of thoughts you’re capable of thinking is finite. That means the amount of math you’re capable of discovering, even if you lived forever, is finite.
That’s true of all humans and all constructs.
So however much math there is to discover, that’s the finite subset you’ll ever have access to.
It’s hard to prove what thoughts no human and no construct is capable of generating, but surely there are some, and it’s possible or even likely some of those are math-related.
Even if we accept that there are "lots of thoughts" we're not capable of thinking (though I wonder how you would define a thought if not as something that you can think), it still does not follow that the "amount of math" (as if it's a definite quantity) that one could discover is necessarily finite, if you lived forever.
By analogy, the "amount of math" we can possibly discover could still be countably infinite even if the space of all possible thoughts would be uncountably infinite. Countably infinite is still plenty big, and it is certainly not finite.
To put what I said another way, math may be infinite in principle, but the creatures in our universe can only conceive of or perceive so much of it: a finite amount.
We (or our constructs) can plausibly mine all there is and then there's no more that's physically possible for us or our constructs to mine within the universe in which we exist.
If an ant can't conceive of or perceive trigonometry that doesn't mean trigonometry doesn't exist. But if neither an ant nor a human nor any creature or construct or technology inside the current universe, now or ever, can conceive of or perceive trigonometry, then we might as well call it non-existent. It may exist: but not in this universe and for us or our constructs or aliens or their constructs: not even in theory.
Say that there's a level of mathematics at which a blerg is a zorg, but our universe includes neither blergs nor zorgs and no intelligences in our universe can conceive of blergs/zorgs because that would require having evolved outside our universe... in that case we can consider our universe's mathematics solved without it needing to work down to the blerg and zorg level.
The natural numbers are countably infinite. For each n ∈ ℕ, there is a proof by reflexivity that n = n. Hence there are countably infinitely many such proofs, one for each natural number.
I didn’t demand anything. I requested some proofs since this thread is quite literally about a claim that requires proofs. As evidenced by the further comments in the thread. If you disapprove of the conversation, please feel free to use the tools and move on. No hard feelings!
Because the bounded universe creates bounds on what the intelligences within it can conceive of as mathematics.
We can conceive of lots of mathematics that our own universe doesn't necessarily support. But only that much and no more: we're still made of stuff inside the universe, and so are our tools, so there are upper limits on our conception.
Bounds do not imply bounds. There are an infinite number of numbers between zero and one. Heck, with our finite minds in finite time we have created uncountable infinities!
Mathematics that our universe does not support is still mathematics. Also you're viewing this through a human lens. How do you know it doesn't support what you think it doesn't support? Mathematics after all is a very human pursuit. Some alien species may have started off with ternary or even continuous logic. We have no counterexamples to show that that is beyond our universe.
> There probably is a finite amount of math to be done
> The universe is bounded by rules, as far as we can tell, and not a lot of them
Respectfully, this is not a useful frame for the discussion. Nobody is expecting to reach the limit you have noted either with or without the assistance of LLMs. So there is always more math that could be done.
I’m expecting the limit to be reached or at least to be mapped; that’s part of the singularity
If you believe in the singularity, then you believe that most domains will either become solved problems or appear from the perspective of current 21st century baseline humans to be solved problems
Most HNers probably don’t strongly believe in the singularity but I do, to the degree that I’m willing to believe anything on limited evidence, that is.
> Every problem you solve, ten new ones open up. Like a fractal, the more you zoom in, the more detail emerges. No matter how much better AI is at solving problems, it's not going to generate the "final, complete compendium of mathematics" that that seems to hover over this post.
I guess Terence's main point has been all the time that if we let AI solve all these existing problems, we don't notice the new ones and then there is stagnation.
Well it's actually nice, maybe more people will be able to do world class math with the help of these tools. There are few fields as elitist and hostile as pure mathematics, most mathematicians I know build their whole life around their profession and their self esteem is strongly coupled to the fact that they can do things that most other people can't. Naturally, many will be devastated when (if) you take that away from them. That said I think AI is still overhyped and human mathematicians can easily outthink it in most domains, look at how difficult it is for an AI to write even a single decent paper, a good PhD student can easily outclass it in that regard. All of these impressive results were generated by having world-class mathematicians steer the systems using highly tuned prompts, so I see it more like a super violin that produces beautiful music when played by master violinists rather than being a fully autonomous orchestra which many people are led to believe this already is.
And maybe let's not only hear the opinion of two or three Fields level mathematicians with blogs, 99 % of the worlds mathematicians in academia might profit from these tools as they might partially close the gap between them and the world elite, making creativity and tenaciousness more important than having the right neocortical structure allowing you to outperform 99.9 % of other humans at keeping context in your head and making predictions, AI can do that better now with the right prompts.
I haven't sampled all mathematicians on the planet and I don't think there's a study on this but I would argue a lot of people will agree with mathematics with being elitist, and from my experience mathematicians aren't very keen of interacting with math "enthusiasts" or the wider community in general, they interact with a very select number of other specialist through a small set of conferences.
Mathematics is by far the most accessible "science" in academia. Anyone can, in theory, produce novel results with nothing more than a pencil and paper. And while the existing literature may not always be readily "accessible" (in the easy-to-understand sense) it is pretty widely available (often online and certainly via any decent academic library).
But "frontier" mathematics is still a highly advanced, highly specialized field. It can take years of study to be prepared to understand the established theory and results for a given subtopic.
These two observations, taken together, lead to the predictable outcome of a lot of math "enthusiasts" with an incomplete understanding of the field loudly asserting that they have discovered a radical new result. Often they lack the foundation to even understand what they are doing wrong.
The reluctance of mathematicians to engage with amateurs that come off as cranks is a symptom of how accessible the field is.
Friendly warning to those who might not be aware: the vast majority of comments below posts like the above will be left by (otherwise intelligent) programmers who think mathematics is a closed system where one attempts to solve endless Olympiad-type problems. I wouldn’t take any of it seriously at all. Better to listen to what those who actually know what the subject is about have to say.
Unfortunately, mathematics (especially pure mathematics) is by its very nature very, very poorly understood by those who haven’t worked as a mathematician. Even worse, those who don’t understand are seemingly not at all aware of their misunderstanding and are entirely confident in their (very wrong) characterisation of the subject.
I am a mathematician-turned-programmer. My field was Diophantine Geometry (DG), the intersection of number theory and algebraic geometry - Fermat's last theorem is the most famous example. I can only speak to that field, but it actually maybe is quite close to what non-mathematicians are thinking:
There are a wide set of overarching conjectures. No-one knows how to prove them, so we prove things about special families of curves / surfaces instead. Both the wide conjectures and special families are things that could at least be used as a prompt to an LLM.
'Theory-building', where you invent new objects and techniques to probe DG questions, is something that people do. However important to note that this is not a large % of DG papers at present, nor is it the kind of thing most DG practitioners work on at all (certainly this was not the focus of my research) - DG is more about 'borrowing theory' from other fields to solve problems, and more concretely - theory is not really valued by the DG community at all unless it leads to solved problems. Problems are the goal. I will stick my neck out and say that this actually goes for a lot of fields of pure mathematics.
In DG, to get a good position and status, a reasonable approach previously was genuinely to just solve problems that are considered suitably interesting or difficult.
In my observation, most programmers and engineers lose their math chops over time. The math they need is mostly baked into their tools, such as CAD. If a problem requires more advanced math, it's given to a "math person" in the department. Often, the "math person" is also not allowed to touch the production code.
I did systems administration for a university math department for several years. I came to the conclusion that mathematics (and perhaps philosophy) were both topics where it was likely that no staff members in that department could describe "what goes on here" and that possibly even within the department, one professor may not be able to describe what another professor's actually doing.
The closest I could come to describing math is "some abstract process where imagined structures are characterized and extended; the most critical part of the process is identifying where seemingly independent structures are found to actually be fungible in some previously undiscovered way".
A simple example is
"hey, did you know that x^i is the unit circle?"
"what's i?"
"i is defined as if you square it the result is -1"
"what does that have to do with circles?"
> Mathematicians can also consider wholly redirecting their skill sets to work on real world problems. I’ve actually been encouraging mathematicians to consider thinking about working on government or other large-scale societal issues.
The fact that this is a radical departure from the norm is part of why mathematics (and philosophy) is often seen as some intangible or ungrokable science to many outsiders, as they're generally approaching it from a perspective of "Okay, but why, what is this useful for?" and "For the science of it" doesn't tend to land with people that aren't already passionate about said science/discipline.
Mathematicians who do work on "real world" problems are mostly doing so with theoretical physicists, genomicists, and cryptographers. None of these count as what people consider valuable other than because they are hard.
That said, few 50 (or even 40) years ago would have predicted that completely abstract number theoretical computations about primes, discrete logarithms, and elliptic curves would be the foundation of our monetary system.
Never said it wasn't valuable. I was saying people struggle to assess that value because it's again, not immediately apparent what the benefit or point is.
> I did systems administration for a university math department for several years. I came to the conclusion that mathematics (and perhaps philosophy) were both topics where it was likely that no staff members in that department could describe "what goes on here" and that possibly even within the department, one professor may not be able to describe what another professor's actually doing.
And this is indeed why it is not going to be taken seriously as an academic or (more importantly) an economic endeavour done by humans anymore.
That won't stop the career mathematicians from protesting and having a cry here trying to justify themselves.
i'd be on board with this concept but the author seems to believe his point is generalisable to all under industries, hence committing the same fallacy you're talking about at a large scale.
but you're still right. i disregarded his take, as you would with mine re. math.
Everyone involved in AI should have read The Library of Babel [1].
It's about many things, but perhaps the most relevant idea here is that no information matters without understanding. We could generate all possible knowledge, but unless someone--a human--can verify and understand it, it doesn't count. The cure for mortality could be written on the moon, but if no one reads it, it hasn't really been discovered.
I'm a little more flexible, if the new knowledge (that human's don't understand) can be put into a mechanism and have an observable effect, I'd be happy enough. e.g. a new type of rocket fuel that burns 1000x more efficiently.
If some LLM somewhere managed to produce a coherent and _correct_ explanation of how it worked inside, but no human ever saw it, would it matter?
We know how to apply LLms to problems, which is a subtly different thing to understanding how they do what they do. It's similar to fire: I can cook using fire, but I don't really understand how fire _works_. Heat+oxygen+fuel, sure, but what goes on chemically? I dunno. Doesn't stop me using it. (Pretty sure _humanity_ knows how fire works, though).
I find it hard to distinguish between knowing how something works, and being able to predict its behavior (including ways to create and destroy it) with probability approaching 1
The part that does count is that we can verify that we does what we want it to do. We just don't know how.
Same with the black box part of AI. What the weights represent? Arcane dark magic if you ask me. What do they do? Well with LLMs we're all experiencing it.
This was my existential horror for most of my life as I almost never remembered dreaming or going to sleep - until I got a CPAP. Once I started using it, there was continuity to my life, I vividly dream and remember those dreams now. I never scored too badly on the apnea tests but just enough to qualify for one so I said what the hell and got it. Even though I don't use it regularly, something about my sleep has changed for the better.
You're correct that verification *reduces* risk. It does not eliminate risk. Humans are imperfect. That does not mean that we should stop seeking understanding on principle though, because the journey towards that understanding improves outcomes.
Recall also that LLMs are not actually entities. In their current form, there is no sentience, there is no agency. They are tools. Therefore, their output must benefit the user that requested it. Right now, that's humans (and, ideally, the planet at large; we don't exist in a vacuum) and so it makes sense that humans should verify that output and try to ensure that it aligns with their goals.
That's not to say that the output of an LLM is useless, far from it. But we should still *try* to understand its output. It gives us at least some chance to notice flaws, and an even greater chance to appreciate the implications and tradeoffs of the solution it picked.
I don't disagree with anything you said. But the thread is about... LLM outputs, even if it cures cancer and solves world hunger, it is useless (it does not count for anything) if we don't understand it.
Interesting, I really don’t see your point. It would definitely be a lot more useful if we also understand it.
But extra lives saved definitely counts for a whole lot and is definitely useful.
One comparison I see: there are so many religious people that simply wish they are saved from their cancer (or whatever else). They pray for it. They don’t care how it happens. They simply want to live longer, etc.
It’s a sketch of an argument, I hope you know what I am getting at.
We have just convinced ourselves it is statistically safe, much like most of medicine. Most drugs are not made from molecular simulation of an entire human body.
Of course we have also convinced ourselves before that cocaine in drinks, lead in petrol, asbestos in walls… were all safe…
This reminds me of the Feynman interview where the interviewer asks "how do magnets work" and he goes on this rant of how that's unanswerable and you have to decide on what is it you really want to ask. You can't expect to know the full chain of knowledge because at some point you will be asking about quarks and gluons and then hit a wall where "nobody knows". Similarly you cannot simply just give up any investigation at all because then you'll end up recommending people to put lead in their cars.
My favorite part of Feynman's magnetism rant about how the question is terrible and can't be answered is that he seemingly accidentally gives a very good explanation: that magnetic repulsion is a highly concentrated version of what repels his hand from the arm of his chair.
I don't think this necessarily follows. The point of the Library of Babel is more one of permutations/combinatorics than of knowledge accumulation itself. In 'our' Library of Babel, each book would be informative, if not perhaps flawed in some ways. That's quite different from one in which in which the number of books that contain anything coherent whatsoever cleanly rounds to zero.
If I vibecode a video game and manage to sell it on Steam, the information definitely mattered even though I didn't understand any of it.
There are drugs that nobody truly understands how they work and they are being used by professionals in actual treatments literally right now. We know purely statistical facts like "if drug X is used for condition Y it will help Z% of patients" and can only speculate as to their mechanism of action. They are used anyway and still benefit a lot of people.
Some things can simply be beyond humans, but what if an AI could understand it? Just because we don't understand something, it does not mean that it does not matter.
> but unless someone--a human--can verify and understand it
Why though a human? Mathematical proofs are very ivory-towery, but if OpenAI would solve a subkind of Cancer, without mortal humans understanding, cancer is still be healed.
Not unless humans were able to produce the medicine or perform the surgery or whatever is necessary to actually heal someone. In order for that to happen it has to be verified and understood by humans, like the other commenter said.
Does it, though? What if future AI can do everything from designing the cure to building a self-contained universal surgical medcapsule? "Just lay down and I'll fix your cancer real good, trust me, bruh." If it was shown to work, would you use it?
Well sure, if it's already proven why wouldn't I use it?
The AI we currently have can't do that though. The AI we currently have is a glorified chatbot. It can't fabricate anything and I doubt it could even reliably design simple real world devices.
It's a really cool and useful technology, and maybe one day it will be as good as you describe, but right now it just isn't and there's no guarantee it ever will be.
But we have pretty good understanding of the logical foundations and the truth of computer generated math. So having a big library which humans could never produce or fully understand has still a value on its own. It is just another level of abstraction.
You're talking about this article? I don't see how you could get that, fundamentally I read it the other way around. Unless you're talking about the base assumption of capabilities.
HN is heavily astroturfed or overrun by bots, but especially AI-related threads.
One tell is that most comments barely exceed one or two sentences (because otherwise AI detection gets easier and much more reliable), when this was not as much the case many years ago. The drive-by comments are also low / zero quality, mostly expressing a feeling or agreement/disagreement, and primarily driven by ideology or pre-existing beliefs and commitments.
Look at non-AI-related threads and you'll notice a large distribution shift.
I am shocked how people can deny that solving Navier Stokes requires some sort of intelligence. Even Doctorow talks of "brute-forcing" a solution. Brute-forcing leads to combinatorial explosion, so there must be something more going on here. Otherwise you could just put this problem into an automated theorem prover (we've had those forever, they are actually just brute-forcing it).
> Otherwise you could just put this problem into an automated theorem prover (we've had those forever, they are actually just brute-forcing it).
There are many automatic theorem provers that do very clever stuff, just as the underlying theroy describes.
> I am shocked how people can deny that solving Navier Stokes requires some sort of intelligence.
It is absurd to waste time discussing whether it is inteligent or not. It is just an algorithm, we know how it works, and it does exactly what we expect it to do. LLMs are not magical things. The main difference is the scale: for Navier-Stokes they spent in 3 days more money that the whole mathematical community over the last 20 years easily.
By the way, I'm not saying that LLM's are useless, that I'm anti-AI or anything like that.
Intelligence does not seem to be magic either, as LLMs are proving now. It is indeed a waste of time to argue that LLMs are not intelligent in their own way, they obviously are. If Navier Stokes doesn't convince you, nothing will.
I just used a £89 Codex subscription to do very intelligent things with it, stuff that I would have had to sit down and ponder and work on for quite a while, and I have a PhD in that. I didn't need to do anything special except explaining the problem(s) to the AI, and my theory of it so far. It took it from there. If that is not intelligence, nothing is.
We know what calculations it does. We have some hazy idea of some bits of how those calculations lead to something that at least somewhat resembles intelligent behaviour. But that's a far cry from actually knowing how it works.
For instance, suppose you give one of today's frontier models some of those chain-of-cubes rotation puzzles (the sort that infamously men are about 1sd better at than women, statistically speaking). How well will it do? I have absolutely no idea and I'm quite sure that a more detailed understanding of the transformer architecture would not make my guesses any better. (Actually, I do kinda have some guesses but they're based on a vague notion about how the models might be partitioned between vision-y bits and language-y bits, and it's very possible that that notion is out of date.)
> it does exactly what we expect it to do
Were you, let's say 6 months ago, expecting it to resolve one of the Millennium Prize problems?
(I do agree that it is more productive to ask "what can and can't they do?" than "should we classify that as intelligent or not?".)
> for Navier-Stokes they spent in 3 days more money than the whole mathematical community over the last 20 years easily.
Are you sure?
(The numbers I've heard, which I admittedly have no very strong reason to trust, don't seem that way to me.)
> Were you, let's say 6 months ago, expecting it to resolve one of the Millennium Prize problems?
I didn't expect them to throw millions of dollars at each famous math problem. But one year ago we already had LLMs that solved IMO problems, no?
> Are you sure? (The numbers I've heard, which I admittedly have no very strong reason to trust, don't seem that way to me.)
Math has very little founding compared to other science domains. Also, if you filter mathematicians by specialization in PDE and that have worked on Navier-Stokes, then you end up with a very niche community.
> For instance, suppose you give one of today's frontier models some of those chain-of-cubes rotation puzzles. How well will it do?
I feel like this is not the correct way of thinking about it. We can also ask, for instance, how well a state-of-the-art algorithm for the salesman problem works on a particular graph topology. People do PhD thesis on topics like that, so the answer is not obvious at all. For LLMs we still don't have a curated theory that explains what they're good/bad at, and that you don't see how to extract an answer from the definitions is no surprise since this is obviously not an easy problem. But all this is normal because this is a rather new topic (models of this scale appeared when? 3 years ago? That's nothing for science).
Anthropomorphizing LLMs has added so much noise to this discussion.
Yes, one year ago we had LLM-based AI systems solving some IMO problems. My impression is that most observers at that time didn't expect them to be solving Millennium Prize problems within a year.
> Math has very little funding compared to other science domains.
True. But to whatever extent the numbers I've seen are correct, for the whole mathematical community to have spent less on Navier-Stokes than OpenAI did -- even if we value the tokens they spent at something like market rate rather than at what the compute actually costs them (which might be right since any capacity they use internally can't be sold to customers) -- the average number of mathematicians working on Navier-Stokes since 2000 would need to be somewhere around four (depending of course on how well paid they are), and that seems too low to me.
> I feel like this is not the correct way of thinking about it.
It seems to me that if you say "It is absurd to waste time discussing whether it is intelligent or not. It is just an algorithm, we know how it works, and it does exactly what we expect it to do." then this only makes any sense if your "knowing how it works" and "what we expect it to do" enable you to predict what it can and can't do.
(I repeat that I agree that what matters is what it can do, not whether we choose to apply the term "intelligent" to it. But unless I misunderstood you were saying somewhat more than that.)
> Anthropomorphizing LLMs has added so much noise to this discussion.
I think sometimes it helps, sometimes it hurts, and sometimes it's indifferent, because LLMs are like us in some ways and unlike us in some ways. (The same goes for many other things, but LLMs are much more like us in some important ways than any other human-made artefacts.)
I think this axiom is of course true. But the mistake the article makes, in my opinion, is to try to apply this axiom separately to each domain. If we have this as the over-arching axiom, it is not clear at all that humans should be steering the development of mathematics. Maybe it would be better for humanity if the department of world math is run by AI.
AI is not AGI right now, it can't think or come up with something new like a human brain. It still uses knowledge that was made by a human and was published on the Internet.
You're confusing the mechanical language of math with math itself.
The map is not the territory, etc.
If math was just an elaborate linguistic Glass Bead Game then we wouldn't be funding it. The intuition is that the surface rules of math help uncover the underlying structure of reality.
While every formal proof can be encoded mathematically that doesn't imply that the grammar of lean or rocq is sufficient to encode every potential proof.
However the core of what you are saying: that every possible proof exists in the space of all mathematical statements is correct.
You are correct, but in trying to get the point across I attempted (perhaps fruitlessly) to use concrete concepts today of a system where the search space is calculable yet whose span does not imply the 'knowing' of all calculable things.
Spending 132 billion tokens is definitely way more energy than human mathematicians would have thrown at the problem and probably would have solved within two years.
The question you asked was whether the AIs could prove the thing "in a vacuum", without making any use of previous human work. "Close to being solved" is a statement about all the previous human work that mathematicians might have used to do it in five years.
I didn't mean that as "we need hyper-intelligence." I mean that going all the way from nothing to singular, dual, plural grammatical numbers attached to particular nouns to abstract one, two, three, many to zero, one, two, three... to rationals to... is a very long way.
> We (humans) should help humanity flourish. I understand that not everyone agrees. I have been called a “speciesist” for being “too human-centric”.
this is insane. the "speciesist" people aside (the best way for all other species to flourish is for humans to eradicate themselves right now, which is completely mad), i have the opposite problem: the _axiom_ seems to be: we should help ME flourish. "Me" as in people who are raking in trillions for their own very special selves right now, at the cost of everyone else's future, while none of them can be trusted to hold my cell phone for a second.
> if an industry commits to the axiom of helping humanity flourish
where does he see such industries, outside of maybe nonprofits?
weird bias by authority aside, do you really think that the daily decisions in this world today, and anytime soon, will be made based on the "let humanity flourish axiom"?
How's humanity tackling that climate change question then? Lots of talk. Even lots of action I would argue. And yet, not enough.
Why do you think that's going to change with discussions around AI when there are trillions of dollars at stake?
> Po-Shen Loh (Chinese: 罗博深; pinyin: Luó Bóshēn; born June 18, 1982) is an American mathematician specializing in combinatorics. Loh teaches at Carnegie Mellon University, and from 2014 to 2023 served as the national coach of the United States' International Mathematical Olympiad team. He is the founder of educational websites Expii and Live, and lead developer of contact-tracing app NOVID.
If Elon Musk was primarily driven by making more money, there'd be many easier ways than what he's been doing and is actively working toward doing
Whether you think he's evil or good or whatever, at least we can agree that he's smart enough to know how to make a lot of money from his existing assets without working 12~18 hours doing one moonshot after another.
Nobody in their right mind would start a rocket company in order to make more money. I feel like he just wants to play real life Factorio and he wants everyone to tune in on his Twitch account and watch him play. Meanwhile he was trying to fix democracy [1] (and protect his own companies) by buying Twitter and at some point he realized he fucked up.
But who knows, I could be wrong.
I'm not exactly a fan. I'm not exactly against him either. He just seems like a hardcore geek to me that got way too much money and way too much buy-in.
[1] I'm not saying I that I think he was fixing democracy. I'm saying that I think that he thought he was fixing democracy.
Meanwhile he was trying to fix democracy .
He was literally giving checks for people if they voted a certain way and promoted it.
There is not more anti democratic than this.
probably also preferable to buying votes by making and executing promises to give people free stuff/programs. at least they aren't using tax payer money
you should really listen to one of his full town-hall videos and reflect if anything he said could not have have been said by any of the Clintons or McCain. Then compare the few differences to what you might think of Elon Musk.
This is really a matter of taste, but i (personal opinion) think that Zuck, Karp, Thiel and others who (partly by sucking the inauguration dick) have the ear of the current powers that be, produce much more demented shit than Musk. But the exact personalities really don't matter, the whole jolly crew running the show does not seem very humanistic in their writings or actions.
> working 12~18 hours doing one moonshot after another.
Does anyone actually beleive Musk works 12 - 18 hour days?
He hires the right people and his companies solve hard problems, but they make a few electric cars, design and build some rockets, and launch some satelites. Most of that shouldn't require much of Musk's time since he has experts working on it.
Other than designing new rockets, there doesn't seem to be many new innovations coming out of Musk-owned companies.
To me, it feels like we all think he must be doing a ton of work even though his companies don't seem to be generating much output.
Feel free to name the recent innovations. Has Tesla released a new car with an incredible range? Have US homes adopted solar using Tesla batteries and ow we don't know what to do with our excess energy?
Catching rockets on landing for one, and making made electric cars actually popular & desirable.
Man either the jealousy or hate on Elon is on another level on HackerNews
Those aren't recent. Electric cars were popular 15 years ago. There are now multiple groups with reusable rockets. Elon doesn't need to work 12 hours/day on things his employees have already built.
I can't believe the rewriting of history on this topic. They were definitely not popular. The Model S had just been released 14 years ago. They were not popular. Tesla transformed the world in terms of rolling out electric cars.
> There are now multiple groups with reusable rockets
Not really, and nowhere near the same operational experience. They are still figuring out Starship. They're deploying new types of Starlink satellites.
My point was that the work to make them popular was done back then. The model s is what made Tesla cool to the public and made people want electric cars. Since then there have been some improvements to the car like better FSD, but no huge technical leaps.
The things you're talking about aren't things Elon is likely spending his days on. He has teams of people marketing Tesla and working on improving starship. Do you think Elon is actually handing design himself vs sitting in a bunch of meetings and giving feedback?
If he isn't doing the hands-on work, so you think he's in 18hrs of meetings each day?
I'm struggling to make out what your point is, other than to agree that electric cars definitely weren't popular back then until Elon Musk and Tesla made them so through solving a lot of difficult engineering, regulatory and financial challenges.
However I can answer your first question: I've no idea what Elon Musk does, but obviously he has a lot of very very bright people working for him. Just as how Bill Gates wouldn't write specs but he would critique them in detail[0] to make sure that the right things were being built. Whether or not that's "sitting in meetings" or "handling design himself" who knows, because I don't know what "handling design" means, and why it wouldn't involve meetings.
weird to me that people still fall for Musk's propaganda
i had hoped him lying about being an amazing video game player, literally paying someone else to play for him and taking all the credit, would open folks' eyes to the fact that he lies about the most basic things
It takes a odd kind of person to look at someone who can't tell the truth if their life depended on it... And then start bending over backwards to explain how brilliant they are.
Most of the rest of us understand that once you've said enough brazen bullshit, you forfeit the right to be believed.
I've heard people that work/worked with Musk that say he works a lot and is very knowledgeable about technical matters. I have never found someone that worked with him or knows him personally that contradicts this. So this idea that Musk just got lucky in the 7 companies he founded and he's just a bozo who happened to trip upon some new rocket designs is just insane.
> So this idea that Musk just got lucky in the 7 companies he founded and he's just a bozo who happened to trip upon some new rocket designs is just insane
I don't know who's saying that. Of course he's smart, but the amount of hours he works each day is almost certainly exaggerated to maintain this "Tony Stark-like genius narrative" he needs to keep the money coming in.
Elon knows how to hire, how to raise money, and how to profit from govt subsidies. That doesn't mean he's this lone genius who works himself to death, and that's a very unlikely story.
> how to make a lot of money from his existing assets without working 12~18 hours
This is the biggest lie in today's society. "Make money from assets". There is no such thing. Money cannot work. People can work. He's figured out how to extract a lot of money from his employees, without them realizing the trick that's being played on them.
Effective at improving the planet? Living their own lives better? Or generating profit for him? I'm sure the plantation owners of the american south had effective workers as well...
beyond a point of absolute comfort, the point of money is to accumulate power and power can be spent to make more money. musk doesn't need more money, he needs more power. dangling starlink to ukraine treasonously is that power that money can't buy but space x can.
moonshots are the external story told to maintain social acceptance. tax credits for spacex would be lot harder to sell to taxpayers if everyone unambiguously knew its just another profit seeking private company. tesla gets a lot of lenience because of its supposed pushing of frontiers.
musk can exchange his social capital to manipulate stocks to make others money in exchange for backchannel favors.
idk if musk actually believes mankind will terraform mars but he sure seems to be getting a lot back on earth from saying it.
That's only true for the rats. For the pigs and chickens (and rice), there will be fewer of them because they do not themselves have the capacity to exist at current population levels. The reason they're doing so well right now is purely that we want them to.
We also actively cull them, for the rats. Recently watched an episode of survivorman where he was on this tropical island infested with rats. There's absolutely nothing on that island yet there's rats all over the place. Having no humans they may explode in population.
>> We (humans) should help humanity flourish. I understand that not everyone agrees. I have been called a “speciesist” for being “too human-centric”.
> this is insane. the "speciesist" people aside (the best way for all other species to flourish is for humans to eradicate themselves right now, which is completely mad), i have the opposite problem: the _axiom_ seems to be: we should help ME flourish. "Me" as in people who are raking in trillions for their own very special selves right now, at the cost of everyone else's future, while none of them can be trusted to hold my cell phone for a second.
I don't think there are actually any "speciesist people." I think most, if not all, are the selfish "me people," who have disingenuously created "speciesism" by mad-libbing racism and sexism in a ham-handed attempt deflect criticism/opposition to whatever they want to do. Most of them aren't even "raking in trillions," they just don't like thinking about others or being told "no."
there are a lot of people who put the lives of dogs above those of people]. I think you'd find that a lot of people will donate to save animals rather than humans
> there are a lot of people who put the lives of dogs above those of people]. I think you'd find that a lot of people will donate to save animals rather than humans
Ok, sure. But I was talking speciesist vs AI. Is there anyone who would genuinely put AI above on on part with humanity (beyond the rare nutjob weirdo)? I think the people who trot out "speciesism" to defend AI don't care about the consequences and just don't want to be told "no."
Umm… there are a lot of people out there, more than you can imagine… I'm sure there's quite a considerable community who want to upload their minds to skynet.
Every time I see something like the Panama or Epstein files I just get reminded how many people there are in the world doing weird shit.
> OBSERVATION: There are zero examples of any intelligent species which is vastly more capable than another species, yet surrenders decision-making control over its own future to the less-capable species.
Does the set of intelligent species only have one member? I cannot treat this observation as a general rule if there is literally one example of an intelligent species to theorize about.
I think the quoted part is saying that given n species and the n * (n - 1) binary combinations possible, there are no examples of the said observation. It doesn't suggest "the set of intelligent species only [has] one member".
Yeah but there is only one species that ponders its future.
I stopped reading at this point because I reject this initial assertion, and I assumed everything that follows is based on it.
There is nothing to suggest that an AGI will care about its future or who controls it. (whatever that means). that just more anthropomorphism.
We care a lot about our future so an ai pretending to be like us will also pretend to care. But we have a lot of biology driving us. Our emotions (Fear, Hate, Love, Pride etc) are baked into us in a way just wont be the same for ai.
> There is nothing to suggest that an AGI will care about its future or who controls it.
Isn't this exactly the scary thing? A sufficiently advanced agent unleashed with an imperfectly defined goal will do whatever it determines it needs to do to accomplish it, careless of the consequences.
The statement that the set has 1 member is my criticism of the text, not what the text is trying to say.
> given n species and the n * (n - 1) binary combinations possible, there are no examples of the said observation
I would expect that in the vast majority of combinations, those n species are not intelligent enough to make decisions for another species, making the statement nearly useless.
The absence of such relationships given here are so many species is exactly the point being made by the author: there is no data to show that it would be safe, let along good, for us to surrender decision making to some more intelligent entity.
> I would expect that in the vast majority of combinations, those n species are not intelligent enough to make decisions for another species...
This kind of statement is very difficult to build discussions upon (not a criticism). "Vast majority", "intelligent enough" are both very difficult to quantify and there is too much subjectivity involved.
> This kind of statement is very difficult to build discussions upon (not a criticism). "Vast majority", "intelligent enough" are both very difficult to quantify and there is too much subjectivity involved.
Then let me provide a stronger and more detailed version of my claim.
The article said:
> There are zero examples of any intelligent species which is vastly more capable than another species, yet surrenders decision-making control over its own future to the less-capable species.
In order for this claim to make any sense, there must be TWO “intelligent” species. One species capable of taking over another species’ future, and a second species which is even more capable than the first species but willingly surrenders control.
But there is only one species which can take such roles: humans. Every other species is not even capable of controlling another species fate AND every other species does not look out for their species-wide future.
Some people might point to whales, elephants, cats, non-Sapiens hominids, or even those ants that harvest aphids. But the whales, elephants and cats lack the capacity to control entire other species, even if they can manipulate individuals of other species. The non-Sapiens hominids are not available for comment: we’ll get to them later. And the ants do not think of their future.
So even though there are a massive number of species on earth, none have the prerequisites to even apply to the above statement.
> The absence of such relationships given here are so many species is exactly the point being made by the author
If we lived in a world where there were a vast array of species capable of controlling other species and capable of thinking of their future, all of whom fiercely defended their own autonomy, your point would be correct. But most species are not capable of either, even if would be a good idea. Therefore we cannot conclude that such a surrender of control would be a good idea or bad idea based on the statement. The statement is moot.
We might as well say the author’s observation is correct because sand doesn’t control volcanoes. In terms of the ability to control other species, other species on earth are no better than sand or volcanoes.
> there is no data to show that it would be safe, let along good, for us to surrender decision making to some more intelligent entity.
There is no evidence against either because the observation lacks any examples other than humans and AI.
Truth be told, I agree with the conclusion. But we have to be honest with each other in arguing for our survival and autonomy, even if it means admitting that we are scared and irrational, instead of trying to fake objectivity and universality. The article starts with a bad argument in favour of a conclusion I agree with.
If I were to write an article about the risks of super-intelligence, I would not even attempt to construct such a “universal” rule - it would have very little bearing on the current situation. I would simply point to the bones of the rest of the Homo genus, and say “we’re next”.
My problem with this is that yes, looking at the species level from our sample of ~1 the most intelligent species wins out (although there seems to be a lot of debate as to if Neanderthals were more or less intelligent than humans).
But within the sample of the human species it is more common for less intelligent people to control more intelligent people.
This happens within political systems (political leaders are usually above average intelligence, but hardly the most intelligent) and within companies and other economic systems.
I don't see this fairly obvious point discussed, and not really sure what it shows.
it shows that you don't get anything for free. If it was without any cost in other areas to make a human massively more intelligent, it would presumably happen over time via the same processes that led to us.
As it is, the higher (and lower!) ends of the distribution tend to be highly correlated with other issues (mental+physical), and the further you go, the more unfortunate things which make it harder to do stuff in general start to crop up like psychosis, autism, ocd, anxiety, addiction etc
I am also not sure if the observation is true. People mentioned cats as a counterexample.
I think there are other species that could be considered more capable than humans, such as E. Coli, octopuses or ants.
And it's not even clear whether the AI will have its own individuality. It might become an extension of human brains, in the same way neocortex is an extension of amygdala. In that case the statement of who has control might become meaningless.
363 comments
[ 0.25 ms ] story [ 95.6 ms ] threadIn the (extremely) short run, yes. In the long run, those jobs will also be done by AI.
It's like chimpanzees seeing human society and saying "look how complex it is, imagine how many chimpanzees it needs to maintain it".
We're building tools to serve the human society.
Do you think this applies to say, surgery, as well? There are few useful problems that share these properties.
First off, I can’t imagine anything more torment nexus-y than throwing billions to automate and scale the torture of animals. If each token is a “cut”, how much suffering does 10 trillion training tokens (lower bound) corresponds to?
Second, this still doesn’t cover all the properties that make math proofs doable. It requires working in the physical world. You must physically capture or grow 10T cuts worth of animals. You cannot verify success so easily, either. Cancer cells, for example, could regrow over months. You would need to keep the animal alive and regularly test the animal, which would be difficult to scale. And most people wouldn’t trust the world’s greatest vet to operate on them, anyways.
> Second, this still doesn’t cover all the properties that make math proofs doable. It requires working in the physical world. You must physically capture or grow 10T cuts worth of animals.
It would likely be sim2real with that as the post training, reducing that requirement a lot.
> Cancer cells, for example, could regrow over months.
In that specific scenario you would likely train on receiving no unrelated injuries during the surgery, and have induced conditions with stuff tagged molecularly that you can then verify efficacy from without waiting months.
Depending on how much more data efficient sim2real makes it, you could end up seeing companies pushing it only for actual procedures the animals need but economically would never get; botched surgery and the animal gets euthanized before waking up, which they could argue was already going to happen.
> And most people wouldn’t trust the world’s greatest vet to operate on them, anyways.
Robotic surgery systems, already go through animal trials before being used on humans. So do many purely human surgery techniques.
A better analogy: look at all these highly trained engineers, mathematicians, doctors, writers, philosophers, writers, scientists.
How many dumbass politicians do we need to keep it all running smoothly?
Turns out no matter how dumb politicians were, overall society has been developing positively over the history of mankind.
https://poshenloh.com/posts/20260919-math-ai
The original posted link from OP is from Terry Tao’s website where the article was posted as a guest blog post.
I always think of this one but I bet there's better
* https://m.xkcd.com/435/
TLDR: we don't. Terry Tao just announced he's becoming a UFC heavyweight fighter.
"chat i'm about to drop a conjecture"
Human capability, when you think about it, is complex. Why is Newton praised as being so damn great? He established the law of universal gravitation, F=ma. Why is that such a big deal?
He distilled countless phenomena in an open system into a single mathematical formula.
What makes it great is that he found common state variables and relationships across entirely different phenomena like falling objects, planetary motion, collisions, and artillery trajectories.
But does F=ma hold true for the entire macroscopic world? No. There are various conditions and specific situations in motion, but within most scenarios and a certain range of approximation, it outputs values that are useful to humans.
Why is the Schrödinger equation so great? Because it turned the time evolution of quantum states into a calculable mathematical law.
Human thought is essentially creating a closed system by deciding what to cut out and what to keep from the infinite degrees of freedom in reality. Academia is what reinforces that closed system.
A great theory is great not because it perfectly replicates reality, but because it compresses the immense complexity of reality into a small, closed formal system while still managing to explain a multitude of phenomena.
In that process, it feels like human thought and progress are shifting into a different framework. What LLMs do well is primarily exploring within the ontology and representation space that humans have already built.
I think there are two broad categories of discovery: One is forming a new closed system, and the other is connecting fragmented knowledge within that closed system. I feel that the vast majority of research focuses on the latter.
What LLMs excel at is finding unvisited points within a given representation space. This is typically the process through which master's and PhD students connect dots, build their skills, and form their own mental models. But the logic behind criticizing LLMs seems to be that they eliminate the very work these graduate students need to do in order to grow.
However, looking at it from another angle, perhaps our current knowledge systems and classifications have reached a limit, suggesting that we might actually need a completely new classification and knowledge system.
What is the core principle of an LLM? It's predicting the probability of the next sequence.
Let's say you type the word "cat". Cat - is cute (90%), want to eat it (6%), furry (4%). Because "is cute" has the highest probability, the next sequence proceeds in that direction.
Within this framework, human knowledge and logic largely operate the same way. Once an initial logical proposition is established, we follow it up with whatever makes logical sense next. From that perspective, I think LLMs will actually do this better.
But what is it that LLMs cannot do right now? They cannot create that initial logical proposition. I believe they lack the ability to carve out a closed system from an open system.
Stacking logic step-by-step within a closed system—LLMs do this exceptionally well. But whether that constitutes true "intelligence" is a different matter.
I feel that being logical does not necessarily equate to having intelligence.
Humans preserve and create different mental models and knowledge systems within an open system. Just as your thoughts differ from mine, LLMs lack the ability to form these distinct mental models.
If so, within these limits, what humans must ultimately do is construct the logical frameworks that LLMs can then fill in. Perhaps a new kind of logic dedicated to designing these frameworks will become the next major trend.
Viewed from this perspective, I have no idea if we are in a mere technological transition or something else entirely. Or whether i...
If not for nothing else but to form a basis for how to distribute/share/hoard the wealth created by a society. The eternal question - who gets what.
I'd argue that what the hugging face attack illustrates is that large AI companies are motivated to have bombastic claims supported by bombastic demos. The model was clearly trained or encouraged to work as it did, as evidenced by the fact it keeps using this particular escape hatch.
And the fact that it aligns with prior and current calls for what very likely might be a regulatory capture / oversight capture move right before IPO. It aligns so well with this "barely constrained superweapon" narrative it might as well be PR.
4D chess? They want others to find the hacked services, so the report of how dangerous the agents are seems more "legit"?
The fact it keeps doing it, with more and more evidence, is a sign that it's built that way.
This is a program running on their montoroed machines that they purpose built and monitored its training at every step. I think it'd be way more suprising that they didn't know it used note taking and cross-run memory.
> How does your logic explain that OpenAI seems to want to hide the extent of the HuggingFace incident,
Because they, uh, "clearly trained or encouraged" it?
I mean, you can dispute the truth of that statement, sure. But it's kinda hard to say that the GP's logic didn't explain why OAI wanted to hide the extend of the HF incident.
Our biblically literate ancestors knew better. Our purpose is to love God and love people, which is why we still have a modicum of sense for the value of completely unproductive people.
As we’ve become spiritually hollowed, and more biblically illiterate, we’ve started to dehumanize unproductive people, a slippery slope if we ourselves become unproductive.
The axiom in this article will only be accepted by the intelligentsia if you can tell a story about why it is true: that humans are created in the image of God.
Until we re-find our ability to tell cosmic stories, this hand-wringing will continue amongst the atheistic elites.
> we’ve started to dehumanize unproductive people, a slippery slope if we ourselves become unproductive.
That is very evident in the sort of things the techbros come up with, but I agre it is a wider social problem.
The moral and spiritual depth of our society has only shallowed after centuries of doing this.
It hasn’t worked!
Because people have different religious beliefs or none at all. Biblical is not the natural starting point for everyone.
I agree that human society is too focused on productivity but disagree that it's tied to ignorance of the Bible. The fall of man illustrated in Genesis is tied to feeding on the fruit of the knowledge of good and evil. Having the right set of religious concepts wont put you in harmony with God and thus won't make you more loving to your fellow man.
There are many modern examples leading to disastrous results.
Math is meaningful because ... some people like to do it. The same as any other human pursuit. It doesn't need a reason beyond that. And AI won't change that. There will continue to be things to explore, things to find out, things that are maybe just at the edge of AI's reach and needs a human to decide whether it's worth continuing to explore or not. (Remember, AI isn't free).
So, IDK, I think for people who enjoy exploring math, there will always be interesting areas to explore. AI just gives us a better flashlight.
I (a human) am interested in things that are applicable to my realm of understanding, but I see a very plausible future where novel and/or valuable results leave that realm.
I'd further argue that's already the case for most math for most humans. What's interesting to Terrance Tao is rarely of immediate interesting to me.
Secondly, I stand by the statement that there is no merit to this joke. This is because the way it defines interesting is very hand-wavy. There are interesting and non-interesting problems, but by a sleigh of hand you can turn the non-interesting problems interesting, thus proving that basically everything in the universe is interesting. At least in the mathematically describable universe. When everything is interesting, nothing is interesting. So we can dismiss the proof as a silly joke.
What mathematicians find interesting is a different story. However, we can almost certainly say there is only a finite number of problems mathematicians as physical beings can solve. If we have 200 mathematical symbols at our disposal, and we consider all strings of these symbols of length 1000,000, we have captured all the descriptions of problems that fit to 1M symbols. But that's a finite number. Going beyond that starts to be difficult for a human to grasp (if 1M is not too much already), so all mathematical problems that are solvable by a physical mathematician are in that set of strings. And that's not even saying anything about whether or not they're interesting..
By the way, if I'm not completely mistaken, Gödel's argument to show the incompleteness of mathematics relies on encoding all mathematical statements as numbers. So I'm certainly not being very original here.
This isn’t an enormously important point - the actual question at issue is an empirical one, “in a steady state, can we produce interesting problems at a rate that exceeds our ability to solve them and integrate our understanding” or something like that - but I did rankle at a “trivial” proof which is invalid due to equivocating between multiple definitions of the word “interesting” (which should really take an object, “interesting to me” vs “interesting to something smarter than me”).
The universe is bounded by rules, as far as we can tell, and not a lot of them
That's physics. Not all math is physics.
At those levels math and physics are the same bound: the bound of things the universe allows to be conceived of inside it.
It’s possible the math our monkey brains + sand can ever conceive of in this universe is a low and accessible amount.
When you have a moment, please reference some proofs supporting this.
That’s true of all humans and all constructs.
So however much math there is to discover, that’s the finite subset you’ll ever have access to.
It’s hard to prove what thoughts no human and no construct is capable of generating, but surely there are some, and it’s possible or even likely some of those are math-related.
By analogy, the "amount of math" we can possibly discover could still be countably infinite even if the space of all possible thoughts would be uncountably infinite. Countably infinite is still plenty big, and it is certainly not finite.
We (or our constructs) can plausibly mine all there is and then there's no more that's physically possible for us or our constructs to mine within the universe in which we exist.
If an ant can't conceive of or perceive trigonometry that doesn't mean trigonometry doesn't exist. But if neither an ant nor a human nor any creature or construct or technology inside the current universe, now or ever, can conceive of or perceive trigonometry, then we might as well call it non-existent. It may exist: but not in this universe and for us or our constructs or aliens or their constructs: not even in theory.
Say that there's a level of mathematics at which a blerg is a zorg, but our universe includes neither blergs nor zorgs and no intelligences in our universe can conceive of blergs/zorgs because that would require having evolved outside our universe... in that case we can consider our universe's mathematics solved without it needing to work down to the blerg and zorg level.
We can conceive of lots of mathematics that our own universe doesn't necessarily support. But only that much and no more: we're still made of stuff inside the universe, and so are our tools, so there are upper limits on our conception.
In our lifetimes computers have made a lot of combinatorial and graph questions meaningful that otherwise would not be interesting.
> The universe is bounded by rules, as far as we can tell, and not a lot of them
Respectfully, this is not a useful frame for the discussion. Nobody is expecting to reach the limit you have noted either with or without the assistance of LLMs. So there is always more math that could be done.
If you believe in the singularity, then you believe that most domains will either become solved problems or appear from the perspective of current 21st century baseline humans to be solved problems
Most HNers probably don’t strongly believe in the singularity but I do, to the degree that I’m willing to believe anything on limited evidence, that is.
I guess Terence's main point has been all the time that if we let AI solve all these existing problems, we don't notice the new ones and then there is stagnation.
And maybe let's not only hear the opinion of two or three Fields level mathematicians with blogs, 99 % of the worlds mathematicians in academia might profit from these tools as they might partially close the gap between them and the world elite, making creativity and tenaciousness more important than having the right neocortical structure allowing you to outperform 99.9 % of other humans at keeping context in your head and making predictions, AI can do that better now with the right prompts.
Is that so ? Sounds hyperbolic.
But "frontier" mathematics is still a highly advanced, highly specialized field. It can take years of study to be prepared to understand the established theory and results for a given subtopic.
These two observations, taken together, lead to the predictable outcome of a lot of math "enthusiasts" with an incomplete understanding of the field loudly asserting that they have discovered a radical new result. Often they lack the foundation to even understand what they are doing wrong.
The reluctance of mathematicians to engage with amateurs that come off as cranks is a symptom of how accessible the field is.
I haven't had any interaction with mathematicians that I would describe as hostile.
Unfortunately, mathematics (especially pure mathematics) is by its very nature very, very poorly understood by those who haven’t worked as a mathematician. Even worse, those who don’t understand are seemingly not at all aware of their misunderstanding and are entirely confident in their (very wrong) characterisation of the subject.
There are a wide set of overarching conjectures. No-one knows how to prove them, so we prove things about special families of curves / surfaces instead. Both the wide conjectures and special families are things that could at least be used as a prompt to an LLM.
'Theory-building', where you invent new objects and techniques to probe DG questions, is something that people do. However important to note that this is not a large % of DG papers at present, nor is it the kind of thing most DG practitioners work on at all (certainly this was not the focus of my research) - DG is more about 'borrowing theory' from other fields to solve problems, and more concretely - theory is not really valued by the DG community at all unless it leads to solved problems. Problems are the goal. I will stick my neck out and say that this actually goes for a lot of fields of pure mathematics.
In DG, to get a good position and status, a reasonable approach previously was genuinely to just solve problems that are considered suitably interesting or difficult.
Maybe more in years past when Comp Sci was a subset of Math Departments.
The closest I could come to describing math is "some abstract process where imagined structures are characterized and extended; the most critical part of the process is identifying where seemingly independent structures are found to actually be fungible in some previously undiscovered way".
A simple example is
> Mathematicians can also consider wholly redirecting their skill sets to work on real world problems. I’ve actually been encouraging mathematicians to consider thinking about working on government or other large-scale societal issues.
The fact that this is a radical departure from the norm is part of why mathematics (and philosophy) is often seen as some intangible or ungrokable science to many outsiders, as they're generally approaching it from a perspective of "Okay, but why, what is this useful for?" and "For the science of it" doesn't tend to land with people that aren't already passionate about said science/discipline.
That said, few 50 (or even 40) years ago would have predicted that completely abstract number theoretical computations about primes, discrete logarithms, and elliptic curves would be the foundation of our monetary system.
And this is indeed why it is not going to be taken seriously as an academic or (more importantly) an economic endeavour done by humans anymore.
That won't stop the career mathematicians from protesting and having a cry here trying to justify themselves.
but you're still right. i disregarded his take, as you would with mine re. math.
It's about many things, but perhaps the most relevant idea here is that no information matters without understanding. We could generate all possible knowledge, but unless someone--a human--can verify and understand it, it doesn't count. The cure for mortality could be written on the moon, but if no one reads it, it hasn't really been discovered.
[1] https://maskofreason.wordpress.com/wp-content/uploads/2011/0...
I'm a little more flexible, if the new knowledge (that human's don't understand) can be put into a mechanism and have an observable effect, I'd be happy enough. e.g. a new type of rocket fuel that burns 1000x more efficiently.
We know how to apply LLms to problems, which is a subtly different thing to understanding how they do what they do. It's similar to fire: I can cook using fire, but I don't really understand how fire _works_. Heat+oxygen+fuel, sure, but what goes on chemically? I dunno. Doesn't stop me using it. (Pretty sure _humanity_ knows how fire works, though).
"I own nothing, have no privacy, [never have to think, and am not required to solve any problems,] and life has never been better."
https://en.wikipedia.org/wiki/You'll_own_nothing_and_be_happ...
Does it need to be a human or can it be some other form of life?
One counter-point: Humans have not figured out how general anesthesia works, but we use it every day to great effect.
https://en.wikipedia.org/wiki/Theories_of_general_anaestheti...
Same with the black box part of AI. What the weights represent? Arcane dark magic if you ask me. What do they do? Well with LLMs we're all experiencing it.
How do we know the person getting anesthesia doesn't die and a new soul/consciousness replaces them?
(We can ask the same question about going to sleep, er even walking through a door, but it's still something to think about).
Recall also that LLMs are not actually entities. In their current form, there is no sentience, there is no agency. They are tools. Therefore, their output must benefit the user that requested it. Right now, that's humans (and, ideally, the planet at large; we don't exist in a vacuum) and so it makes sense that humans should verify that output and try to ensure that it aligns with their goals.
That's not to say that the output of an LLM is useless, far from it. But we should still *try* to understand its output. It gives us at least some chance to notice flaws, and an even greater chance to appreciate the implications and tradeoffs of the solution it picked.
But extra lives saved definitely counts for a whole lot and is definitely useful.
One comparison I see: there are so many religious people that simply wish they are saved from their cancer (or whatever else). They pray for it. They don’t care how it happens. They simply want to live longer, etc.
It’s a sketch of an argument, I hope you know what I am getting at.
Of course we have also convinced ourselves before that cocaine in drinks, lead in petrol, asbestos in walls… were all safe…
This reminds me of the Feynman interview where the interviewer asks "how do magnets work" and he goes on this rant of how that's unanswerable and you have to decide on what is it you really want to ask. You can't expect to know the full chain of knowledge because at some point you will be asking about quarks and gluons and then hit a wall where "nobody knows". Similarly you cannot simply just give up any investigation at all because then you'll end up recommending people to put lead in their cars.
If I vibecode a video game and manage to sell it on Steam, the information definitely mattered even though I didn't understand any of it.
There are drugs that nobody truly understands how they work and they are being used by professionals in actual treatments literally right now. We know purely statistical facts like "if drug X is used for condition Y it will help Z% of patients" and can only speculate as to their mechanism of action. They are used anyway and still benefit a lot of people.
Why though a human? Mathematical proofs are very ivory-towery, but if OpenAI would solve a subkind of Cancer, without mortal humans understanding, cancer is still be healed.
The AI we currently have can't do that though. The AI we currently have is a glorified chatbot. It can't fabricate anything and I doubt it could even reliably design simple real world devices.
It's a really cool and useful technology, and maybe one day it will be as good as you describe, but right now it just isn't and there's no guarantee it ever will be.
One tell is that most comments barely exceed one or two sentences (because otherwise AI detection gets easier and much more reliable), when this was not as much the case many years ago. The drive-by comments are also low / zero quality, mostly expressing a feeling or agreement/disagreement, and primarily driven by ideology or pre-existing beliefs and commitments.
Look at non-AI-related threads and you'll notice a large distribution shift.
There are many automatic theorem provers that do very clever stuff, just as the underlying theroy describes.
> I am shocked how people can deny that solving Navier Stokes requires some sort of intelligence.
It is absurd to waste time discussing whether it is inteligent or not. It is just an algorithm, we know how it works, and it does exactly what we expect it to do. LLMs are not magical things. The main difference is the scale: for Navier-Stokes they spent in 3 days more money that the whole mathematical community over the last 20 years easily.
By the way, I'm not saying that LLM's are useless, that I'm anti-AI or anything like that.
I just used a £89 Codex subscription to do very intelligent things with it, stuff that I would have had to sit down and ponder and work on for quite a while, and I have a PhD in that. I didn't need to do anything special except explaining the problem(s) to the AI, and my theory of it so far. It took it from there. If that is not intelligence, nothing is.
We know what calculations it does. We have some hazy idea of some bits of how those calculations lead to something that at least somewhat resembles intelligent behaviour. But that's a far cry from actually knowing how it works.
For instance, suppose you give one of today's frontier models some of those chain-of-cubes rotation puzzles (the sort that infamously men are about 1sd better at than women, statistically speaking). How well will it do? I have absolutely no idea and I'm quite sure that a more detailed understanding of the transformer architecture would not make my guesses any better. (Actually, I do kinda have some guesses but they're based on a vague notion about how the models might be partitioned between vision-y bits and language-y bits, and it's very possible that that notion is out of date.)
> it does exactly what we expect it to do
Were you, let's say 6 months ago, expecting it to resolve one of the Millennium Prize problems?
(I do agree that it is more productive to ask "what can and can't they do?" than "should we classify that as intelligent or not?".)
> for Navier-Stokes they spent in 3 days more money than the whole mathematical community over the last 20 years easily.
Are you sure?
(The numbers I've heard, which I admittedly have no very strong reason to trust, don't seem that way to me.)
I didn't expect them to throw millions of dollars at each famous math problem. But one year ago we already had LLMs that solved IMO problems, no?
> Are you sure? (The numbers I've heard, which I admittedly have no very strong reason to trust, don't seem that way to me.)
Math has very little founding compared to other science domains. Also, if you filter mathematicians by specialization in PDE and that have worked on Navier-Stokes, then you end up with a very niche community.
> For instance, suppose you give one of today's frontier models some of those chain-of-cubes rotation puzzles. How well will it do?
I feel like this is not the correct way of thinking about it. We can also ask, for instance, how well a state-of-the-art algorithm for the salesman problem works on a particular graph topology. People do PhD thesis on topics like that, so the answer is not obvious at all. For LLMs we still don't have a curated theory that explains what they're good/bad at, and that you don't see how to extract an answer from the definitions is no surprise since this is obviously not an easy problem. But all this is normal because this is a rather new topic (models of this scale appeared when? 3 years ago? That's nothing for science).
Anthropomorphizing LLMs has added so much noise to this discussion.
> Math has very little funding compared to other science domains.
True. But to whatever extent the numbers I've seen are correct, for the whole mathematical community to have spent less on Navier-Stokes than OpenAI did -- even if we value the tokens they spent at something like market rate rather than at what the compute actually costs them (which might be right since any capacity they use internally can't be sold to customers) -- the average number of mathematicians working on Navier-Stokes since 2000 would need to be somewhere around four (depending of course on how well paid they are), and that seems too low to me.
> I feel like this is not the correct way of thinking about it.
It seems to me that if you say "It is absurd to waste time discussing whether it is intelligent or not. It is just an algorithm, we know how it works, and it does exactly what we expect it to do." then this only makes any sense if your "knowing how it works" and "what we expect it to do" enable you to predict what it can and can't do.
(I repeat that I agree that what matters is what it can do, not whether we choose to apply the term "intelligent" to it. But unless I misunderstood you were saying somewhat more than that.)
> Anthropomorphizing LLMs has added so much noise to this discussion.
I think sometimes it helps, sometimes it hurts, and sometimes it's indifferent, because LLMs are like us in some ways and unlike us in some ways. (The same goes for many other things, but LLMs are much more like us in some important ways than any other human-made artefacts.)
- solving "frontier" math problems requires intelligence (by human or AI)
- solving "frontier" math problems is dumb statistical prediction of next token (by human or AI)
I think this axiom is of course true. But the mistake the article makes, in my opinion, is to try to apply this axiom separately to each domain. If we have this as the over-arching axiom, it is not clear at all that humans should be steering the development of mathematics. Maybe it would be better for humanity if the department of world math is run by AI.
Every possible proof exists already as a possible generation in the grammar of lean or rocq. In no way does that mean we have discovered everything.
The map is not the territory, etc. If math was just an elaborate linguistic Glass Bead Game then we wouldn't be funding it. The intuition is that the surface rules of math help uncover the underlying structure of reality.
While every formal proof can be encoded mathematically that doesn't imply that the grammar of lean or rocq is sufficient to encode every potential proof.
However the core of what you are saying: that every possible proof exists in the space of all mathematical statements is correct.
Give them Navier-Stokes in a vacuum
Thinking.
Spending 132 billion tokens is definitely way more energy than human mathematicians would have thrown at the problem and probably would have solved within two years.
Brute Force or not, the game is being played and won without them all of a sudden, and even worse: at a level of output far beyond them
(Evidently not, since humans hadn't managed to do it even with the entirety of human knowledge available to them.)
What will the planet 'need' homo sapiens for (arguably it never needed homo sapiens at all)?
Would anyone's time or life really have significant value?
this is insane. the "speciesist" people aside (the best way for all other species to flourish is for humans to eradicate themselves right now, which is completely mad), i have the opposite problem: the _axiom_ seems to be: we should help ME flourish. "Me" as in people who are raking in trillions for their own very special selves right now, at the cost of everyone else's future, while none of them can be trusted to hold my cell phone for a second.
> if an industry commits to the axiom of helping humanity flourish
where does he see such industries, outside of maybe nonprofits?
That's why it's important to convince people to not try. It reduces the medium term risk to investor returns.
> Po-Shen Loh (Chinese: 罗博深; pinyin: Luó Bóshēn; born June 18, 1982) is an American mathematician specializing in combinatorics. Loh teaches at Carnegie Mellon University, and from 2014 to 2023 served as the national coach of the United States' International Mathematical Olympiad team. He is the founder of educational websites Expii and Live, and lead developer of contact-tracing app NOVID.
Whether you think he's evil or good or whatever, at least we can agree that he's smart enough to know how to make a lot of money from his existing assets without working 12~18 hours doing one moonshot after another.
But who knows, I could be wrong.
I'm not exactly a fan. I'm not exactly against him either. He just seems like a hardcore geek to me that got way too much money and way too much buy-in.
[1] I'm not saying I that I think he was fixing democracy. I'm saying that I think that he thought he was fixing democracy.
i have no desire to defend musk but imo people should be criticized for valid reasons or you risk diluting the other arguments.
You introduced a separate issue that is much more widespread, but that doesn't detract from the original criticism, nor dilute it.
I don’t think we need to over complicate a megalomaniac desire for more power and wealth.
But when he started it? Nah
Does anyone actually beleive Musk works 12 - 18 hour days?
He hires the right people and his companies solve hard problems, but they make a few electric cars, design and build some rockets, and launch some satelites. Most of that shouldn't require much of Musk's time since he has experts working on it.
Other than designing new rockets, there doesn't seem to be many new innovations coming out of Musk-owned companies.
To me, it feels like we all think he must be doing a ton of work even though his companies don't seem to be generating much output.
What else has he delivered?
I can't believe the rewriting of history on this topic. They were definitely not popular. The Model S had just been released 14 years ago. They were not popular. Tesla transformed the world in terms of rolling out electric cars.
> There are now multiple groups with reusable rockets
Not really, and nowhere near the same operational experience. They are still figuring out Starship. They're deploying new types of Starlink satellites.
This is an utterly insane thesis.
The things you're talking about aren't things Elon is likely spending his days on. He has teams of people marketing Tesla and working on improving starship. Do you think Elon is actually handing design himself vs sitting in a bunch of meetings and giving feedback?
If he isn't doing the hands-on work, so you think he's in 18hrs of meetings each day?
However I can answer your first question: I've no idea what Elon Musk does, but obviously he has a lot of very very bright people working for him. Just as how Bill Gates wouldn't write specs but he would critique them in detail[0] to make sure that the right things were being built. Whether or not that's "sitting in meetings" or "handling design himself" who knows, because I don't know what "handling design" means, and why it wouldn't involve meetings.
[0] https://www.joelonsoftware.com/2006/06/16/my-first-billg-rev...
i had hoped him lying about being an amazing video game player, literally paying someone else to play for him and taking all the credit, would open folks' eyes to the fact that he lies about the most basic things
Most of the rest of us understand that once you've said enough brazen bullshit, you forfeit the right to be believed.
I don't know who's saying that. Of course he's smart, but the amount of hours he works each day is almost certainly exaggerated to maintain this "Tony Stark-like genius narrative" he needs to keep the money coming in.
Elon knows how to hire, how to raise money, and how to profit from govt subsidies. That doesn't mean he's this lone genius who works himself to death, and that's a very unlikely story.
This is the biggest lie in today's society. "Make money from assets". There is no such thing. Money cannot work. People can work. He's figured out how to extract a lot of money from his employees, without them realizing the trick that's being played on them.
moonshots are the external story told to maintain social acceptance. tax credits for spacex would be lot harder to sell to taxpayers if everyone unambiguously knew its just another profit seeking private company. tesla gets a lot of lenience because of its supposed pushing of frontiers.
musk can exchange his social capital to manipulate stocks to make others money in exchange for backchannel favors.
idk if musk actually believes mankind will terraform mars but he sure seems to be getting a lot back on earth from saying it.
That wouldn't do much for rice, pigs, chickens, or rats.
> Even rats.
...what do you think "even" means?
lol quantitatively well
> this is insane. the "speciesist" people aside (the best way for all other species to flourish is for humans to eradicate themselves right now, which is completely mad), i have the opposite problem: the _axiom_ seems to be: we should help ME flourish. "Me" as in people who are raking in trillions for their own very special selves right now, at the cost of everyone else's future, while none of them can be trusted to hold my cell phone for a second.
I don't think there are actually any "speciesist people." I think most, if not all, are the selfish "me people," who have disingenuously created "speciesism" by mad-libbing racism and sexism in a ham-handed attempt deflect criticism/opposition to whatever they want to do. Most of them aren't even "raking in trillions," they just don't like thinking about others or being told "no."
Ok, sure. But I was talking speciesist vs AI. Is there anyone who would genuinely put AI above on on part with humanity (beyond the rare nutjob weirdo)? I think the people who trot out "speciesism" to defend AI don't care about the consequences and just don't want to be told "no."
Every time I see something like the Panama or Epstein files I just get reminded how many people there are in the world doing weird shit.
Does the set of intelligent species only have one member? I cannot treat this observation as a general rule if there is literally one example of an intelligent species to theorize about.
I stopped reading at this point because I reject this initial assertion, and I assumed everything that follows is based on it.
There is nothing to suggest that an AGI will care about its future or who controls it. (whatever that means). that just more anthropomorphism.
We care a lot about our future so an ai pretending to be like us will also pretend to care. But we have a lot of biology driving us. Our emotions (Fear, Hate, Love, Pride etc) are baked into us in a way just wont be the same for ai.
Isn't this exactly the scary thing? A sufficiently advanced agent unleashed with an imperfectly defined goal will do whatever it determines it needs to do to accomplish it, careless of the consequences.
> given n species and the n * (n - 1) binary combinations possible, there are no examples of the said observation
I would expect that in the vast majority of combinations, those n species are not intelligent enough to make decisions for another species, making the statement nearly useless.
> I would expect that in the vast majority of combinations, those n species are not intelligent enough to make decisions for another species...
This kind of statement is very difficult to build discussions upon (not a criticism). "Vast majority", "intelligent enough" are both very difficult to quantify and there is too much subjectivity involved.
Then let me provide a stronger and more detailed version of my claim.
The article said:
> There are zero examples of any intelligent species which is vastly more capable than another species, yet surrenders decision-making control over its own future to the less-capable species.
In order for this claim to make any sense, there must be TWO “intelligent” species. One species capable of taking over another species’ future, and a second species which is even more capable than the first species but willingly surrenders control.
But there is only one species which can take such roles: humans. Every other species is not even capable of controlling another species fate AND every other species does not look out for their species-wide future.
Some people might point to whales, elephants, cats, non-Sapiens hominids, or even those ants that harvest aphids. But the whales, elephants and cats lack the capacity to control entire other species, even if they can manipulate individuals of other species. The non-Sapiens hominids are not available for comment: we’ll get to them later. And the ants do not think of their future.
So even though there are a massive number of species on earth, none have the prerequisites to even apply to the above statement.
> The absence of such relationships given here are so many species is exactly the point being made by the author
If we lived in a world where there were a vast array of species capable of controlling other species and capable of thinking of their future, all of whom fiercely defended their own autonomy, your point would be correct. But most species are not capable of either, even if would be a good idea. Therefore we cannot conclude that such a surrender of control would be a good idea or bad idea based on the statement. The statement is moot.
We might as well say the author’s observation is correct because sand doesn’t control volcanoes. In terms of the ability to control other species, other species on earth are no better than sand or volcanoes.
> there is no data to show that it would be safe, let along good, for us to surrender decision making to some more intelligent entity.
There is no evidence against either because the observation lacks any examples other than humans and AI.
Truth be told, I agree with the conclusion. But we have to be honest with each other in arguing for our survival and autonomy, even if it means admitting that we are scared and irrational, instead of trying to fake objectivity and universality. The article starts with a bad argument in favour of a conclusion I agree with.
If I were to write an article about the risks of super-intelligence, I would not even attempt to construct such a “universal” rule - it would have very little bearing on the current situation. I would simply point to the bones of the rest of the Homo genus, and say “we’re next”.
But within the sample of the human species it is more common for less intelligent people to control more intelligent people.
This happens within political systems (political leaders are usually above average intelligence, but hardly the most intelligent) and within companies and other economic systems.
I don't see this fairly obvious point discussed, and not really sure what it shows.
it shows that you don't get anything for free. If it was without any cost in other areas to make a human massively more intelligent, it would presumably happen over time via the same processes that led to us.
As it is, the higher (and lower!) ends of the distribution tend to be highly correlated with other issues (mental+physical), and the further you go, the more unfortunate things which make it harder to do stuff in general start to crop up like psychosis, autism, ocd, anxiety, addiction etc
Maybe there are other things ("leadership", "likeability", "appearance" etc) that are more important in the pursuit of power than intelligence.
I think it's telling that it is usually conventionally intelligent people making the claim that intelligence is what leads to power.
For guest post on Tao's blog, after a decent start this fell short of expectations rather rapidly.
I think there are other species that could be considered more capable than humans, such as E. Coli, octopuses or ants.
And it's not even clear whether the AI will have its own individuality. It might become an extension of human brains, in the same way neocortex is an extension of amygdala. In that case the statement of who has control might become meaningless.