All money is green, regardless of level of sophistication. If you’re using investment firm pedigree as signal, gonna have a bad time. They’re all just throwin’ darts under the guise of skill (actor/observer|outcome bias; when you win, it is skill; when you lose, it was luck, broadly speaking).
> Indeed, one should be sophisticated themselves when negotiating investment to not be unduly encumbered by the unsophisticated. But let us not get too far off topic and risk subthread detachment.
Edit: @jgalt212: Indeed, one should be sophisticated themselves when negotiating investment to not be unduly encumbered by shades of the unsophisticated or potentially folks not optimizing for aligned interests. But let us not get too far off topic and risk subthread detachment. Feel free to cut a new thread for further discussion on the subject.
This has to be one of the quickest valuations past a billion. I wonder if they can even effectively make use of the funds in a reasonable enough timeline.
> I wonder if they can even effectively make use of the funds in a reasonable enough timeline.
I read that it cost Google ~$190 million to train Gemini, not even including staff salaries. So feels like a billion gives you about 3 "from scratch" comparable training runs.
Your estimate seems way off given Google already had their own compute hardware and staff. And if this company is going straight for AGI there's no way $1 billion is enough.
This might be the largest seed round in history (note that 1B is the cash raised, not the valuation). You think that's an indication of the hype dissipating?
At the height of the Japanese economy in the 80s the about 2 square miles of land on which the Imperial Palace stood were worth more than all property in California. Clearly a brilliant moment to get into Japanese real estate!
A valuation at seed mentioned to possibly be in the region of $5bn means that these investors expect there's a reasonable chance that this company, which at this point will be one among many, might become one of the largest companies in the world as that's the kind of multiple they'd need given the risks of such an early stage bet.
That doesn't sound like the hype is dissipating to me.
Explain why you think $1B at $5B valuation isn't overvaluation? This strikes me as over-indexing on Ilya + teams ability to come up with something novel while trying to play catch-up.
Most likely it's some junior rep assigned to Sutskever's company after Ilya filled up an online "Contact Us for Pricing" form on the Nvidia website. /s
I'm beginning to wonder if these investors are not just pumping AI because they are personally invested in Nvidia and this is a nice way to directly inject a couple of 100M into their cashflow.
Given OpenAI’s declining performance after his being sidelined and then departing, interested to see what they do. Should be a clear demonstration of who was really driving innovation there.
Probably will be an unpopular opinion here but I think declining performance is more likely related to unclear business models backed by immature technology driven by large hype trains they themselves created.
Unpopular because it does not follow the OAI hate train but I think this is a pretty solid take. There is real value in LLM but I believe the hype overshadowed the real cases.
100% OpenAi performance is decreasing. I basically use Claud sonnet exclusively and canceled my OpenAi subscription for personal use. my company still uses them because you cant currently fine-tune a Claud model, yet.
They're probably just scaling back resources to the existing models to focus on the next generation. I feel like I have seen OpenAI models lose capability over time and I bet it's a cost optimization on their part.
For these kinds of capital-intensive startups, though, that almost seems like a requirement, and I guess there are really 2 "types" of valuations.
In this case, everyone knows it takes hundreds of millions to train models. So I'm investors are essentially rolling the dice on an extremely well-regarded team. And if it takes about a billion just to get off the ground, the valuation would need to at least be in the couple billion range to make it worth it for employees to work there.
That feels very different than say selling a company where founders are cashing out. In that case, the business should expect to meaningful contribute to revenue, and quickly.
The company's pitch isn't exactly a secret. The one and only thing they're planning to do is build an ML model smarter than a human being, which would be immensely valuable for a wide variety of tasks that currently require human input. You see a lot of commentators jumping through hoops to deny that anyone could believe this is possible in the near future, but clearly they and their investors do.
AGI would definitely be a major historical milestone for humanity ...
... however, I'm on the camp that believes it's not going to be hyper-profitable for only one (or a few) single commercial entities.
AGI will not be a product like the iPhone where one company can "own" it and milk it for as long as they want. AGI feels more like "the internet", which will definitely create massive wealth overall but somehow distributed among millions of actors.
We've seen it with LLMs, they've been revolutionary and yet, one year after a major release, free to use "commodity" LLMs are already in the market. The future will not be Skynet controlling everything, it will be uncountable temu-tier AIs embedded into everything around you. Even @sama stated recently they're working on "intelligence so cheap that measuring its use becomes irrelevant".
It's certainly in their best interest not to tell us that it's just going to be another pile of LLMs that they've trained not to say or do anything that isn't business friendly.
In 2022 Ilya Sutskever claimed there wasn't a distinction:
> It may look—on the surface—that we are just learning statistical correlations in text. But it turns out that to ‘just learn’ the statistical correlations in text, to compress them really well, what the neural network learns is some representation of the process that produced the text. This text is actually a projection of the world.
This is transparently false - newer LLMs appear to be great at arithmetic, but they still fail basic counting tests. Computers can memorize a bunch of symbolic times tables without the slightest bit of quantitative reasoning. Transformer networks are dramatically dumber than lizards, and multimodal LLMs based on transformers are not capable of understanding what numbers are. (And if Claude/GPT/Llama aren't capable of understanding the concept of "three," it is hard to believe they are capable of understanding anything.)
Sutskever is not actually as stupid as that quote suggests, and I am assuming he has since changed his mind.... but maybe not. For a long time I thought OpenAI was pathologically dishonest and didn't consider that in many cases they aren't "lying," they blinded by arrogance and high on their own marketing.
Which basic counting tests do they still fail? Recent examples I've seen fall well within the range of innumeracy that people routinely display. I feel like a lot of people are stuck in the mindset of 10 years ago, when transformers weren't even invented yet and state-of-the-art models couldn't identify a bird, no matter how much capabilities advance.
I have not used LLMs since 2023, when GPT-4 routinely failed almost every counting problem I could think of. I am sure the performance has improved since then, though "write an essay with 250 words" still seems unsolved.
The real problem is that LLM providers have to play a stupid game of whack-a-mole where an enormous number of trivial variations on a counting problem need to be specifically taught to the system. If the system was capable of true quantitative reasoning that wouldn't be necessary for basic problems.
There is also a deception is that "chain of thought" prompting makes LLMs much better at counting. But that's cheating: if the LLM had quantitative reasoning it wouldn't need a human to indicate which problems were amenable to step-by-step thinking. (And this only works for O(n) counting problems, like "count the number of words in the sentence." CoT prompting fails to solve O(nm) counting problems like "count the number of words in this sentence which contain the letter 'e'" For this you need a more specific prompt, like "First, go step-by-step and select the words which contain 'e.' Then go step-by-step to count the selected words." It is worth emphasizing over and over that rats are not nearly this stupid, they can combine tasks to solve complex problems without a human holding their hand.)
I don't know what you mean by "10 years ago" other than a desire to make an ad hominem attack about me being "stuck." My point is that these "capabilities" don't include "understands what a number is in the same way that rats and toddlers understand what numbers are." I suspect that level of AI is decades away.
Yeah, it's not clear what companies like OpenAI and Anthropic mean when they predict AGI coming out of scaled up LLMs, or even what they are really talking about when they say AGI or human-level intelligence. Do they believe that scale is all you need, or is it an unspoken assumption that they're really talking about scale plus some set of TBD architectural/training changes?!
I get the impression that they really do believe scale is all you need, other than perhaps some post-training changes to encourage longer horizon reasoning.
Maybe Ilya is in this camp, although frankly it does seem a bit naive to discount all the architectural and operational shortcomings of pre-trained Transformers, or assume they can be mitigated by wrapping the base LLM in an agent that provides what's missing.
> But it turns out that to ‘just learn’ the statistical correlations in text, to compress them really well, what the neural network learns is some representation of the process that produced the text
This is pretty sloppy thinking.
The neural network learns some representation of a process that COULD HAVE produced the text. (this isn't some bold assertion, it's just the literal definition of a statistical model).
There is no guarantee it is the same as the actual process. A lot of the "bow down before machine God" crowd is guity of this same sloppy confusion.
It's not sloppy. It just doesn't matter in the limit of training.
1. An Octopus and a Raven have wildly different brains. Both are intelligent. So just the idea that there is some "one true system" that the NN must discover or converge on is suspect. Even basic arithmetic has numerous methods.
2. In the limit of training on a diverse dataset (ie as val loss continues to go down), it will converge on the process (whatever that means) or a process sufficiently robust. What gets the job done gets the job done. There is no way an increasingly competent predictor will not learn representations of the concepts in text, whether that looks like how humans do it or not.
No amount of training would cause a fly brain to be able to do what an octopus or bird brain can, or to model their behavioral generating process.
No amount of training will cause a transformer to magically sprout feedback paths or internal memory, or an ability to alter it's own weights, etc.
Architecture matters. The best you can hope for an LLM is that training will converge on the best LLM generating process it can be, which can be great for in-distribution prediction, but lousy for novel reasoning tasks beyond the capability of the architecture.
>No amount of training would cause a fly brain to be able to do what an octopus or bird brain can, or to model their behavioral generating process.
Go back a few evolutionary steps and sure you can. Most ANN architectures basically have relatively little to no biases baked in and the Transformer might be the most blank slate we've built yet.
>No amount of training will cause a transformer to magically sprout feedback paths or internal memory, or an ability to alter it's own weights, etc.
A transformer can perform any computation it likes in a forward pass and you can arbitrarily increase inference compute time with the token length. Feedback paths? Sure. Compute inefficient? Perhaps. Some extra programming around the Model to facilitate this ? Maybe but the architecture certainly isn't stopping you.
Even if it couldn't, limited =/ trivial. The Human Brain is not Turing complete.
Internal Memory ?
Did you miss the memo ? Recurrency is overrated. Attention is all you need.
That said, there are already state keeping language model architectures around.
Altering weights ?
Can a transformer continuously train ? Sure. It's not really compute efficient but architecture certainly doesn't prohibit it.
>Architecture matters
Compute Efficiency? Sure. What it is capable of learning? Not so much
> A transformer can perform any computation it likes in a forward pass
No it can't.
A transformer has a fixed number of layers - call it N. It performs N sequential steps of computation to derive it's output.
If a computation requires > N steps, then a transformer most certainly can not perform it in a forward pass.
FYI, "attention is all you need" has the implicit context of "if all you want to build is a language model". Attention is not all you need if what you actually want to build is a cognitive architecture.
Transformer produce the next token by manipulating K hidden vectors per layer, one vector per preceding token. So yes you can increase compute length arbitrarily by increasing tokens. Those tokens don't have to carry any information to work.
And again, human brains are clearly limited in the number of steps it can compute without writing something down.
Limited =/ Trivial
>FYI, "attention is all you need" has the implicit context of "if all you want to build is a language model".
Great. Do you know what a "language model" is capable of in the limit ? No
These top research labs aren't only working on Transformers as they currently exist but it doesn't make much sense to abandon a golden goose before it has hit a wall.
You are confusing number of sequential steps with total amount of compute spent.
The input sequence is processed in parallel, regardless of length, so number of tokens has no impact on number of sequential compute steps which is always N=layers.
> Do you know what a "language model" is capable of in the limit ?
Well, yeah, if the language model is an N-layer transformer ...
Then increase N (N is almost always increased when a model is scaled up) and train or write things down and continue.
A limitless iteration machine (without external aid) is currently an idea of fiction. Brains can't do it so I'm not particularly worried if machines can't either.
Increasing number of layers isn't a smart way to solve it. It order to be able to reason effectively and efficiently the model needs to use as much, or as little, compute as needed for a given task. Completing "1+1=" should take less compute steps than "A winning sequence for white here is ...".
This lack of "variable compute" is a widely recognized shortcoming of transformer-based LLMs, and there are plenty of others. The point apropos this thread is that you can't just train an LLM to be something that it is not. If the generating process required variable compute (maybe 1000's of steps) - e.g. to come up with a chess move - then no amount of training can make the LLM converge to model this generative process... the best it can do is to model the outcome of the generative process, not the process itself. The difference is that without having learnt the generative process, the model will fail when presented with a novel input that it didn't see during training, and therefore didn't memorize the "cheat sheet" answer for.
>Increasing number of layers isn't a smart way to solve it.
The "smart way" is a luxury. Solving the problem is what matters. Think of a smart way later if you can. That's how a lot of technological advancement has worked.
>It order to be able to reason effectively and efficiently the model needs to use as much, or as little, compute as needed for a given task. Completing "1+1=" should take less compute steps than "A winning sequence for white here is ...".
Same thing. Efficiency is nice but a secondary concern.
>If the generating process required variable compute (maybe 1000's of steps) - e.g. to come up with a chess move - then no amount of training can make the LLM converge to model this generative process.
Every inference problem has itself a fixed number of compute steps it needs (yes even your chess move). Variability is a nice thing for between inferences(maybe move 1 required 500 but 2 only 240 etc) A nice thing but never a necessary thing.
3.5-turbo-instruct plays chess consistently at 1800 Elo so clearly the N of the current SOTA is already enough to play non-trivial chess at a level beyond most humans.
There is an N large enough for every GI problem humans care about. Not to sound like a broken record but once again, limited =/ trivial.
> And again, human brains are clearly limited in the number of steps it can compute without writing something down
No - there is a loop between the cortex and thalamus, feeding the outputs of the cortex back in as inputs. Our brain can iterate for as long as it likes before initiating any motor output, if any, such as writing something down.
The brain's ability to iterate on information is still constrained by certain cognitive limitations like working memory capacity and attention span.
In practice, the cortex-thalamus loop allows for some degree of internal iteration, but the brain cannot endlessly iterate without some form of external aid (e.g., writing something down) to offload information and prevent cognitive overload.
I'm not telling you anything here you don't experience in your everyday life. Try indefinitely iterating on any computation you like and see how well that works for you.
My point is that for some bizare reason, people have standards of reasoning (for machines) that only exist in fiction or their own imagination.
It is beyond silly to dump an architecture for a limitation the human brain has. A reasoning engine that can iterate indefinitely with no external aid does not exist in real life. That the transformer also has this weakness is not any reason for it to have capabilities less than a brain so it's completely moot.
LLMs are here to stay until something better replaces them, and will be used for those things they are capable of.
It shouldn't be surprising they are not great at reasoning, or everything one would hope for from an AGI, since they simply were not built for that. If you look at the development history, the transformer was a successor to LSTM-based seq-2-seq models using Bahdanau attention, whose main goal was to more efficiently utilize parallel hardware by supporting parallel processing. Of course a good language model (word predictor) will look as if it's reasoning because it is trying to model the data it was trained on - a human reasoner.
As humans we routinely think for seconds/minutes or even hours before speaking or acting, while an LLM only has that fixed N steps (layers) of computation. I don't know why you claim this difference (among others) should make no difference, but it clearly does, with out-of-training-set reasoning weakness being a notable limitation that people such as Demis Hassabis have recently conceded.
Reasoning is reasoning. "Look as if it is reasoning" is an imaginary distinction you've made up. One that is very clear because everybody touting this "fake reasoning" rhetoric is still somehow unable to define a testable version of reasoning that disqualifies LLMs without also disqualifying some chunk of humans.
>As humans we routinely think for seconds/minutes or even hours before speaking or acting
No human is iterating on a base thought for hours uninterrupted lol so this is just moot
>with out-of-training-set reasoning weakness being a notable limitation that people such as Demis Hassabis have recently conceded.
Humans reason weaker out of training. LLMs are simply currently worse
> Reasoning is reasoning. "Look as if it is reasoning" is an imaginary distinction you've made up.
No - just because something has the surface appearance of reasoning doesn't mean that the generative process was reasoning, anymore than a cargo cult wooden aircraft reflects any understanding of aerodynamics and would be able to fly.
We've already touched on it, but the "farmer crossing river" problems is a great example. When the LLM sometimes degenerates into "cross bank A to B with chicken, cross band B to A with chicken, cross bank A to B with chicken.. that is the fewest trips possible", this is an example of "looks as if it is reasoning" aka cargo-cult surface-level copying of what a solution looks like. Real reasoning would never repeat a crossing without loading/unloading something since that conflicts with the goal of fewest trips possible.
I never said anything about the surface appearance of reasoning. Either the model demonstrates some understanding or reasoning in the text it generates as it is perfectly capable of or it reasons faultily or lacks understanding in that area. This does not mean LLMs don't reason anymore than it means you don't reason.
The idea that LLMs "fake reason" and Humans "really reason" is an imaginary distinction. If you cannot create any test that can distinguish the two then you are literally making things up.
Dude, I just gave you an example, and you straight-up ignore it and say "show me a test"?!
An averagely smart human does not have these failure modes where they answer a question with something that looks like an answer "cross A to B, then B to A. done. there you go!" but has zero logic to it.
Do you follow news in this field at all? Are you aware that poor reasoning is basically the #1 shortcoming that all the labs are working on?!!
Feel free to have the last word as this is just getting repetitive.
You are supposed to show me an example no human will fail. I didn't ignore anything. I'm just baffled that you genuinely believe this:
>An averagely smart human does not have these failure modes where they answer a question with something that looks like an answer "cross A to B, then B to A. done. there you go!" but has zero logic to it.
Humans are poor at logic in general. We make decisions, give rationales with logical contradictions and nonsense all the time. I just genuinely can't believe you think we don't. It happens so often we have names for these cognitive shortcomings. Get any teacher you know and ask them this. No need to take my word for it. And i don't care about getting the last word.
You seem to repeatedly insist that hidden computation is a distinction of any relevance whatsoever.
First of all, your understanding of the architecture itself is mistaken. A transformer can iterate endlessly because each token it produces allows it a forward pass, and each of these tokens is postpended to its input in the next inference. That's the autoregressive in autoregressive transformer, and the entire reason why it was proposed for arbitrary seq2seq transduction.
This means you get layers * tokens iterations, where tokens is up to two million, and is in practice unlimited due to the LLM being able to summarize and select from that. Parallelism is irrelevant, since the transformer is sequential in the output of tokens. A transformer can iterate endlessly, it simply has to output enough tokens.
And no, the throughput isn't limited either, since each token gets translated into a high-dimensional internal representation, that in turn is influenced by each other token in the model input. Models can encode whatever they want not just by choosing a token, but by choosing an arbitrary pattern of tokens encoding arbitrary latent-space interactions.
Secondly, internal thoughts are irrelevant, because something being "internal" is an arbitrary distinction without impact. If I trained an LLM to prepend and postpend <internal_thought> to some part of its output, and then simply didn't show that part, then the LLM wouldn't magically become human. This is something many models do even today, in fact.
Similarly, if I were to take a human and modify their brain to only be able to iterate using pen and paper, or by speaking out loud, then I wouldn't magically make them into something non-human. And I would definitely not reduce their capacity for reasoning in any way whatsoever. There are people with aphantasia working in the arts, there are people without an internal monologue working as authors - how "internal" something is can be trivially changed with no influence on either the architecture or the capabilities of that architecture.
Reasoning itself isn't some unified process, neither is it infinite iteration. It requires specific understanding about the domain being reasoned over, especially understanding of which transformation rules are applicable to produce desired states, where the judgement about which states are desirable has to be learned itself. LLMs can reason today, they're just not as good at it than humans are in some domains.
Sure - a transformer can iterate endlessly by generating tokens, but this is no substitute for iterating internally and maintaining internal context and goal-based attention.
One reason why just blathering on endlessly isn't the same as thinking deeply before answering, is that it's almost impossible to maintain long-term context/attention. Try it. "Think step by step" or other attempts to prompt the model into generating a longer reply that builds upon itself, will only get you so far because keeping a 1-dimensional context is no substitute for the thousands of connections we have in our brain between neurons, and the richness of context we're therefore able to maintain while thinking.
The reasoning weakness of LLMs isn't limited to "some domains" that they had less training data for - it's a fundamental architecturally-based limitation. This becomes obvious when you see the failure modes of simple problems like "how few trips does the farmer need to cross the river with his chicken & corn, etc" type problems. You don't need to morph the problem to require out-of-distribution knowledge to get it to fail - small changes to the problem statement can make the model state that crossing the river backwards and forwards multiple times without loading/unloading anything is the optimal way to cross the river.
But, hey, no need to believe me, some random internet dude. People like Demis Hassabis (CEO of DeepMind) acknowledge the weakness too.
First of all, I would urge you to stop arbitrarily using negative words to make an argument. Saying that LLMs are "blathering" is equivalent to saying you and I are "smacking meat onto plastic to communicate" - it's completely empty of any meaning. This "vibes based arguing" is common in these discussions and a massive waste of time.
Now, I don't really understand what you mean by "almost impossible to maintain long-term context/attention". I'm writing fiction in my spare time, LLMs do very well on this by my testing, even subtle and complex simulations of environments, including keeping track of multiple "off-screen" dynamics like a pot boiling over.
There is nothing "1-dimensional" about the context, unless you mean that it is directional in time, which any human thought is as well, of course. As I said in my original reply, each token is represented by a multidimensional embedding, and even that is abstracted away by the time inference reaches the later layers. The word "citrus" isn't just a word for the LLM, just as it isn't just a word for you. Its internal representation retrieves all the contextual understanding that is related to it. Properties, associated feelings, usage - every relevant abstract concept is considered. And these concepts interact which every embedding of every other token in the input in a learned way, and with the position they have relative to each other. And then when an output is generated from that dynamic, said output influences the dynamic in a way that is just as multidimensional.
The model can maintain context as rich as it wants, and it can built upon that context in whatever way it wants as well. The problem is that in some domains, it didn't get enough training time to build robust transformation rules, leading it to draw false conclusions.
You should reflect on why you are only able to provide vague and under defined, often incorrect, arguments here. You're drawing distinctions that don't really exist and trying to hide that by appealing to false intuitions.
> The reasoning weakness... it's a fundamental architecturally-based limitation...
You have provided no evidence or reasoning for that conclusion. The river crossing puzzle is exactly what I had in mind when talking about specific domains. It is a common trick question with little to no variation and LLMs have overfit on that specific form of the problem. Translate it to any other version - say transferring potatoes from one pot to the next, or even a mathematical description of sets being modified - and the models do just fine. This is like tricking a human with the "As I was going to Saint Ives" question, exploiting their expectation of having to do arithmetic because it looks superficially like a math problem, and then concluding that they are fundamentally unable to reason.
> People like Demis Hassabis (CEO of DeepMind) acknowledge the weakness too.
What weakness? That current LLMs aren't as good as humans when reasoning over certain domains? I don't follow him personally but I doubt he would have the confidence to make any claims about fundamental inabilities of the transformer architecture. And even if he did, I could name you a couple of CEOs of AI labs with better models that would disagree, or even Turing award laureates. This is by no means a consensus stance in the expert community.
> And even if he did, I could name you a couple of CEOs of AI labs with better models that would disagree, or even Turing award laureates. This is by no means a consensus stance in the expert community.
I disagree - there is pretty widespread agreement that reasoning is a weakness, even among the best models, (and note Chollet's $1M ARC prize competition to spur improvements), but the big labs all seem to think that post-training can fix it. To me this is whack-a-mole wishful thinking (reminds me of CYC - just add more rules!). At least one of your "Turing award laureates" thinks Transformers are a complete dead end as far as AGI goes.
A weakness of the current models in some domains considered useful, yes - but not a fundamental limitation of the architecture. I see no consensus on the latter whatsoever.
The ARC challenge tests spatial reasoning, something we humans are obviously quite good at, given 4 billion years of evolutionary optimization. But as I said, there is no "general reasoning", it's all domain dependent. A child does better at the spatial problems in ARC given that it has that previously mentioned evolutionary advantage, but just as we don't worship calculators as superior intelligences because they can multiply 10^9 digit numbers in milliseconds, we shouldn't draw fundamental conclusions from humans doing well at a problem that they are in many ways built to solve. If the failures of previous predictions - those that considered Chess or Go as unmistakable signals of true general reasoning - have taught us anything, it's that general reasoning simply does not exist.
The bet of current labs is synthetic data in pre-training, or slight changes of natural data that induces more generalization pressure for multi-step transformations on state in various domains. The goal is to change the data so models learn these transformations more readily and develop good heuristics for them, so not the non-continuous patching that you suggest.
But yes, the next generation of models will probably reveal much more about where we're headed.
> If the failures of previous predictions - those that considered Chess or Go as unmistakable signals of true general reasoning - have taught us anything, it's that general reasoning simply does not exist.
I don't think DeepBlue or AlphaGo/etc were meant to teach us anything - they were just showcases of technological prowess by the companies involved, demonstrations of (narrow) machine intelligence.
But...
Reasoning (differentiated from simpler shallow "reactive" intelligence) is basically multi-step chained what-if prediction, and may involve a branching exploration of alternatives ("ok, so that wouldn't work, so what if I did this instead ..."), so could be framed as a tree search of sorts, not entirely disimilar to the MCTS used by DeepBlue or AlphaGo.
Of course general reasoning is a lot more general than playing a game like Chess or Go since the type of moves/choices available/applicable will vary at each step (these aren't all "game move" steps), as will the "evaluation function" that predicts what'll happen if we took that step, but "tree search" isn't a bad way to conceptualize the process, and this is true regardless of the domain(s) of knowledge over which the reasoning is operating.
Which is to say, that reasoning is in fact a generalized process, and one who' nature has some corresponding requirements (e.g. keeping track of state) for any machine to be capable of performing it ...
>You don't need to morph the problem to require out-of-distribution knowledge to get it to fail
make the slight variation look different from the version it have memorized and it often passes. Sometimes it's as straightforward as just changing the names. humans have this failure mode too.
Sure I agree. But if that's what you're getting hung up on, i think you've missed his point entirely.
Whether the machines becomes a human brain clone or something entirely alien is irrelevant. The point is, you can't cheat reality. Statistics is not magic. You can't predict text that understands without understanding.
Sure you can, and if your predictive engine doesn't have the generality and power of the original generative one, then you have no choice.
Machine learning isn't magic - the model will learn what it can to minimize the error over the specific provided loss function, and no more. Change the loss function and you change what the model learns.
In the case of an LLM trained with a predict next word loss function, what you are asking/causing the model to learn is NOT the generative process - you are asking it to learn the surface statistics of the training set, and the model will only learn what it needs to (and is able to, per the model architecture being trained) in order to do this.
Now of course learning the surface statistics well does necessitate some level of "understanding" - are we dealing with a fairy tale or a scientific paper for example, but there is only so much the model can do. Chess is a good example, since it's easy to understand. The generative process for world class chess (whether human, or for an engine) involves way more DEPTH (cf layers) of computation than the transformer has available to model it, so the best it can do is to learn the surface statistics via much shallower pattern recognition of the state of the board. Now, given the size of these LLMs, if trained on enough games they will be able to play pretty well even using this pattern matching technique, but one doesn't need to get too far into a chess game to reach a position that has never been seen before in recorded games (e.g. watch agadmator's YouTube chess channel - he will often comment when this point has been reached), and the model therefore has no choice but to play moves that were seen in the training set in similar, but not identical positions... This is basically cargo-cult chess! It's interesting that LLMs can reach the ELO level that they do (says more about chess than about LLMs), but this same "cargo-cult" (follow surface statistics) generation process when out of training set applies to all inputs, not just chess...
>the model will learn what it can to minimize the error over the specific provided loss function, and no more. Change the loss function and you change what the model learns.
You clearly do not really understand what it means to predict internet scale text with increasing accuracy. No more than that ? Fantastic
LLMs do not just learn surface statistics. So many papers have thoroughly disabused this that i'm just not going to bother. This is just straight up denial.
You have no idea what you are talkin about. You've probably never even played with 3.5-turbo-instruct. That's how you can say this nonsense. You have your conclusion and keep working backwards to get a justification.
>It's interesting that LLMs can reach the ELO level that they do (says more about chess than about LLMs)
When you say this for everything LLMs can do then it just becomes a meaningless cope statement.
No of course not - they also learn whatever is necessary, and possible, in order to replicate those surface statistics (e.g. understanding of fairy tales, etc, as I noted).
However, you seem to be engaged in magical thinking and believe these models are learning things beyond their architectural limits. You appear to be star struck by what these models can do, and blind to what one can deduce - and SEE - they they are unable to do.
You've said a lot of things about LLM chess performance that is not true and can be easily shown to be not true. Literally evidence right there that shows the model learning the board state, rules, player skills etc.
And then you've tried to paper over being shown that with a conveniently vague and nonsensical, "says more about bla bla bla". No, you were wrong. Your model about this is wrong. It's that simple.
You start from your conclusions and work your way down from it. "pattern matching technique" ? Please. By all means, explain to all of us what this actually entails in a way we can test for it. Not just vague words.
An LLM will learn what it CAN (and needs to, to reduce the loss), but not what it CAN'T. How difficult is that to understand?!
Tracking probable board state given a sequence of moves (which don't even need to go all the way back to the start of the game!) is relatively simple to do, and doesn't require hundreds of sequential steps that are beyond the architecture of the model. It's just a matter of incrementally updating the current board state "hypothesis" per each new move (essentially: "a knight just moved to square X, so it must have moved away from some square a knight's move away from X that we believe currently contains a knight").
Ditto for estimating player ELO rating in order to predict appropriately good or bad moves. It's basically just a matter of how often the player makes the same move as other players of a given ELO rating in the training data. No need for hundreds of steps of sequential computation that are beyond the architecture of the model.
Doing an N-ply lookahead to reason about potential moves is a different story, but you want to ignore that and instead throw out a straw man "counter argument" about maintaining board state as if that somehow proves that the LLM can magically apply > N=layers of sequential reasoning to derive moves. Sorry, but this is precisely magical faith-based thinking "it can do X, so it can do Y" without any analysis of what it takes to do X and Y and why one is possible, and the other is not.
>An LLM will learn what it CAN (and needs to to reduce the loss), but not what it CAN'T. How difficult is that to understand?!
Right and the point is that you don't know what it CAN'T learn. You clearly don't quite understand this because you say stuff like this:
>Chess is a good example, since it's easy to understand. The generative process for world class chess (whether human, or for an engine) involves way more DEPTH (cf layers) of computation than the transformer has available to model it
and it's just baffling because:
1. Humans don't play chess anything like chess engines. They literally can't because the brain has iterative computation limits well below that of a computer. Most Grandmasters are only evaluating 5 to 6 moves deep on average.
You keep trying to make the point that because a Transformer architecturally has a depth limit for some trained model, a, it cannot reach human level.
But this is nonsensical for various reasons.
- Nobody is stopping you from just increasing N such that every GI problem we care about is covered.
- You have shown literally no evidence that the N even state of the art models posses today is insufficient to match human iterative compute ability.
GPT-4o instant shots arbitrary arithmetic more accurately than any human brain and that's supposedly something it's bad at.
You can clearly see it can reach world class chess play.
If you have some iterative computation benchmark that shows transformers zero shotting worse than an unaided human then feel free to show me.
I did not claim the state of the art was better at all forms of reasoning than all humans. I claimed the architecture isn't going to stop it from being so in the future but I guess constructing a strawman is always easier right ?
There are benchmarks that rightfully show the SOTA behind average human performance in other aspects of reasoning so why are you fumbling so much to demonstrate this with unaided iterative computation ? It's your biggest argument so I just thought you'd have something more substantial than "It's limited bro!"
You cannot even demonstrate this today nevermind some hypothetical scaled up model.
> so why are you fumbling so much to demonstrate this with unaided iterative computation
Well, you see, I've been a professional developer for the last 45 years, and often, gasp, think for long periods of time before coding, or even writing things down. "Look ma, no hands!".
I know this will come across as an excuse, but the thing is I assumed you were also vaguely famililar with things like software development, or other cases where human's think before acting, so I evidentially did a poor job of convincing you of this.
I also assumed (my bad!) that you would at least know some people who were semi-intelligent and wouldn't be hopelessly confused about farmers and chickens, but now I realize that was a mistake.
Really, it's all on me.
I know that "just add more rules", "make it bigger" didn't work for CYC, but maybe as you suggest "increase N" is all that's needed in the case of LLMs, because they are special. Really - that's genius! I should have thought of it myself.
I'm sure Sam is OK, but he'd still appreciate you letting him know he can forget about Q* and Strawberries and all that nonsense, and just "increase N"! So much simpler and cheaper rather than hiring thousands of developers to try to figure this out!
Maybe drop Yan LeCun a note too - tell him that the Turing Award committee are asshats, and that he is too, and that LLMs will get us all the way to AGI.
>Well, you see, I've been a professional developer for the last 45 years, and often, gasp, think for long periods of time before coding, or even writing things down. "Look ma, no hands!".
>I know this will come across as an excuse, but the thing is I assumed you were also vaguely famililar with things like software development, or other cases where human's think before acting, so I evidentially did a poor job of convincing you of this.
Really, you have the same train of thought for hours on end ?
When you finish even your supposed hours long spiel, do you just proceed to write every line of code that solves your problem just like that ? Or do you write and think some more ?
More importantly, are LLMs unable to produce the kind of code humans spend a train of thought on ?
>Maybe drop Yan LeCun a note too - tell him that the Turing Award committee are asshats, and that he is too, and that LLMs will get us all the way to AGI.
You know, the appeal to authority fallacy is shifty at the best of times but it's straight up nonsensical when said authority does not have consensus on what you're appealing to.
Like great you mentioned LeCun. And I can just as easily bring in Hinton, Norvig, Ilya. Now what ?
> In the limit of training on a diverse dataset (ie as val loss continues to go down), it will converge on the process (whatever that means) or a process sufficiently robust.
This is just moving the goal posts from "learning the actual process" to "any process sufficiently robust"
A photograph is not the same as its subject, and it is not sufficient to reconstruct the subject, but it's still a representation of the subject. Even a few sketched lines are something we recognise as a representation of a physical object.
I think it's fair to call one process that can imitate a more complex one a representation of that process. Especially when in the very next sentence he describes it as a "projection", which has the mathematical sense of a representation that loses some dimensions.
YeS, exactly. The trick is to have enough tough data so you find optimal one. I think as we will scale models back to smaller sizes we will discover viable/correct representations
Are state-level actors the main market for AI security?
Using the definition from the article:
> AI safety, which refers to preventing AI from causing harm, is a hot topic amid fears that rogue AI could act against the interests of humanity or even cause human extinction.
If the purpose of a state is to ensure its continued existence, then they should be able to make >=$1 in profit.
> I honestly don't see a market for "AI security".
I suspect there's a big corporate market for LLMs with very predictable behaviour in terms of what the LLM knows from its training data, vs what it knows from RAG or its context window.
If you're making a chatbot for Hertz Car Hire, you want it to answer based on Hertz policy documents, even if the training data contained policy documents for Avis and Enterprise and Budget and Thrifty car hire.
Avoiding incorrect answers and hallucinations (when appropriate) is a type of AI safety.
Talent attracts capital. Ilya is a legendary visionary, with a proven track record of turning billions into hundreds of billions. Of course he can raise unlimited money.
There is so much talent in the world that didn’t join PayPal and get silicon valley investors and go on to make billions of dollars and found other companies.
The PayPal mafia includes Elon Musk, Peter Thiel, etc. They now parlayed that capital into more platforms and can easily arrange investments. Heck Peter Thiel even works with governments (Palantir) and got J D Vance on Trump’s ticket, while Elon might be in his admin.
Kolomoisky got Zelensky elected in Ukraine, by launching a show about an unlikely guy who wins the presidency and named the party after the show. They call them oligarchs over there but it’s same thing.
The first guy to 1 million followers on Twitter was Ashton Kutcher. He had already starred in sitcoms and movies for years.
This idea that you can just get huge audiences and investments due to raw talent, keeps a lot of people coming to Hollywood and Silicon Valley to “make it” and living on ramen. But even just coming there proves the point — a talented rando elsewhere in the world wouldn’t even have access to the capital and big boys networks.
Those people weren’t handed that success. You are acting as if they were born billionaires, which is far from true.
It’s not personally my goal to amass immense wealth and start giant companies (I would rather work minimally and live hedonically) but I am impressed by those that do so.
No, I’m saying it was those who went to silicon valley and got lucky to strike up relationships with CAPITAL who made it.
Overwhelmingly talent isnt sufficient. For most startups, the old boys network gets to choose who gets millions. And the next rounds a few people choose who will get billions.
If you look at the early stages of Google or Zip2, they were NOT swimming in money. Facebook only attracted serious investments when they already had something that looked promising. Apple started in a bedroom in the parent's house, before being moved to the garage, and so on.
If anything, I think startups from 2005-2020 were more likely to get founding easily than those giants.
But after succeeding, in some cases several times, all of the above found it easier to find investors.
Ilya has a similar track record. He contributed on several breakthroughs such as AlexNet, AlphaGo, seq2seq (that would evolve into transformers after he left Google) before even joining OpenAI.
In fact, Elon see it as one of his key contributions to OpenAI that he managed to recruit Ilya.
That Ilya is able to raise 1B now is hardly surprising. He's probably able to raise way more than that once he's hired a larger team.
I never understood this line of reasoning, because it presumes that everyone should have access to the same opportunities. It's clearly silly once you throw a few counter examples: should a Private in the military be able to skip the ranks and be promoted straight to General? Should a new grad software dev be able to be promoted to lead engineer without getting any experience?
Clearly there are reasons why opportunities are gated.
> This idea that you can just get huge audiences and investments due to raw talent, keeps a lot of people coming to Hollywood and Silicon Valley to “make it” and living on ramen. But even just coming there proves the point — a talented rando elsewhere in the world wouldn’t even have access to the capital and big boys networks.
All those people start somewhere though. Excluding nepotism, which is tangential point, all those people started somewhere and then grew through execution and further opening of opportunity. But it's not like they all got to where they are in one-shot. Taking your Ashton Kutcher example - yes he had a head start on twitter followers, but that's because he executed for years before on his career. Why would it make sense for some rando to rack up a million followers before he did?
Talent will earn you opportunities, but it's not going to open the highest door until you've put in the time and work.
Of course, it's not to say inequity or unequal access to opportunities doesn't exist in the world. Of course it does. But even in an ideal, perfectly equitable world, not everyone would have the same access to opportunities.
So yes, it makes perfect sense that someone would give Ilya $1B instead of some smart 18 year old, even if that 18 year old was Ilya from the past.
Presumably the private and the general are in the SAME organization and yes, the avenues for advancement are available equally to all, it’s based on merit and the rules are clear.
The analogy would be if the private could become a major overnight because they knew a guy.
Yes but a private cannot become a general without decades of experience.
What we see with ilya is not dissimilar. I don't see why it's bad that people are more hesitant to give a talented 18 year old $1B than the guy who's been at the forefront of AI innovation.
> And sometimes not even necessary. Paris Hilton got a music distribution deal overnight cause of her dad’s capital!
Nepotism is a tangential point, and yes I agree that it's a bad thing. Ilya did not get this deal through nepotism, he got it through his past accomplishments, much like how a general gets promoted after many years of exemplary work.
Totally blind on this, hoping for someone to shed some light: do these investors get some pitch, information or some roadmap of what company intends to create, how will it earn revenue, how will it spend money or how will it operate?
I’m sure they have a pitch deck. It’s pretty obvious a big chunk will go to compute costs for model training & research. But mostly it’s about the people in any company at this stage, same as any seed funding but on a different monetary scale.
I heard this on a reddit thread a while back but rings very true here.
> If you are seeking capital for a startup with a product, you have to sell the startup on realities (ie how much revenue you are making). If you are seeking capital for a startup with no product, you can sell the startup on dreams, which is much much easier but also way riskier for investors.
Since these guys don't have a product yet, they 100% sold it on big dreams combined with Ilya's track record at OpenAI.
This feels like a situation with a sold out train to a popular destination, where people are already reselling their tickets for some crazy markup, and then suddenly railway decides to add one more train car and opens flash ticket sale. Investors feeling missing out on OpenAI and others are now hoping to catch this last train ticket to the AI.
The problem is a content to train LLMs (I assume that Ilia will continue this line or research). Big content holders are already raising moats and restricting access or partnering with a single existing LLM corporation. And also time, because all this involves a lot of hardware. Any subsequent competitor will have to scale higher and higher wall just to catch up (if the LLM progress doesn't stall and get into diminishing returns).
Me after watching channel5 I think some of it should go to poor people instead of billion dollars roulettes only.
Thought the problem is with even richer corporations I feel and financial derivatives and not fully here.
except in this case, the train driver from the original train was "sacked" (some believe unfairly), and decided to get their own train to drive. Of course, the smoothness of the ride depends on the driver of the train.
Evergrande imploded because of massive amounts of debt that they had been rolling for years. Continually rolling this massive debt was working till property demand slowed and their revenues couldn't keep up adequately to qualify them to issue new debt.
It's a highly risky bet, but not fundamentally unreasonable. One might believe that Ilya's research was genuinely critical to OpenAI's current situation. If one takes that premise, three potential corollaries follow: (1) OpenAI will struggle to produce future research breakthroughs without Ilya; (2) OpenAI will struggle to materially move beyond its current product lineup and variations thereof without said future research breakthroughs; (3) a startup led by Ilya could overcome both (1) and (2) with time.
An alternative sequence of reasoning places less emphasis on Ilya specifically and uses Ilya as an indicator of research health. Repeat (1), (2), and (3) above, but replace "Ilya" with something like "strong and healthy fundamental research group". In this version, Ilya's departure is taken as indication that OpenAI no longer has a strong and healthy fundamental research group but that the company is "compromised" by relentless feature roadmaps for current products and their variations. That does not mean OpenAI will fail, but in this perspective it might mean that OpenAI is not well positioned to capture future research breakthroughs and the products that they will generate.
From my perspective, it's just about impossible to know how true these premises really are. And that's what makes it a bet or gamble rather than anything with any degree of assurance. To me, just as likely is the scenario where it's revealed that Ilya is highly ineffective as a generalist leader and that research without healthy tension from the business goes nowhere.
Agreed, the AI bubble is very, very real. Not that LLMs are all hype, they’re certainly impressive with useful applications, but AI companies are getting insane valuations with zero proof that they’re viable businesses.
>Agreed, the car bubble is very, very real. Not that the internal combustion carriage is all hype, it's certainly impressive with useful applications, but car manufacturers are getting insane valuations with zero proof they're viable businesses.
The successful companies that came out of the dot com bubble era actually proved their business viability before getting major investment, though.
Amazon is one of the most famous successes of the era. Bezos quit his job, launched the business out of his garage, with seed money being $10K of his own savings, and was doing $20K/week in sales just 30 days later. And I believe their only VC round before going public was an $8 investment from Kleiner Perkins. But they were a company who proved their viability early on, had a real product with rapid revenue growth before getting any VC $$.
I’d say this SSI round is more similar to Webvan, who went public with a valuation of $4.8 billion, and at that time had done a grand total of $395K in sales, with losses over $50 million.
I’m sure there are good investments out there for AI companies that are doing R&D and advancing the state of the art. However, a $1 billion investment at a $5 billion valuation, for a company with zero product or revenue, just an idea, that’s nuts IMO, and extremely similar to the type of insanity we saw during the dot com bubble. Even more so given that SSI seemingly don’t even want to be a business - direct quote from Ilya:
> This company is special in that its first product will be the safe superintelligence, and it will not do anything else up until then … It will be fully insulated from the outside pressures of having to deal with a large and complicated product and having to be stuck in a competitive rat race.
This doesn’t sound to me like someone who wants to build a business, it sounds like someone who wants to hack on AI with no oversight or proof of financial viability. Kinda wild to give him $1 billion to do that IMO.
This wave, whether or not it's a bubble, has very little in common with the dotcom era. It's simply a bad analogy.
The dotcom era was full of unprofitable startups pumping up the stock price in all sorts of ways, as they were completely dependent on continues capital flows from investors to stay afloat. Also, a lot of that capital came from retail investors in various forms.
The AI wave that is currently ongoing is for the most part funded by some of the largest and most profitable corporations on the planet.
Companies like Alphabet, Meta, Tesla/X, Amazon and (to a lesser extent) Microsoft still have founders that either control or provide a direction for these companies.
What drives this way is the fact that most of these founders have a strong belief.
We know, for instance, that Larry Page and Elon Musk had a disagreement about the future role of AGI/ASI about 15 years ago, leading to Elon Musk helping to found OpenAI to make sure that Google would not gain a monopoly.
These are strong convictions held by very powerful people that have been held for decades. Short term stock market fluctuations are not going to suddenly collapse this "bubble".
As long as these founders continue to believe that AGI is close, they will continue to push, even if the stock market stops it support to the push.
SSI may fail, of course. But Ilya has a rumor of (from people like Hinton and Elon) as being perhaps the greatest and most capable visionary in the business.
It means that AI startups are actually a really poor value proposition compared to traditional tech companies, because your multiplier is limited. First round $50M valuation leaves a lot more opportunity to get rich.
This kind of structure isn't as unusual for capital intensive businesses.
I'm neither a VC nor in the VC market, but I believe such valuation comes primarily from the name Ilya Sutskever. Having such a high-profile as the founder would give more credibility to the company, unlike what we witnessed in recent years where companies like Theranos et al. that were valued at tens of billions for no obvious reason. Despite having said the above, we might still agree that the AI hype is probably the second generation of the dot-com bubble.
How many niche verticals SaaSes that raised like $200 million only to go to zero? Even if this can't beat OpenAI models a commodity LLM which is about as good (and they have proven that they can build) is probably worth close to the investment
Same funding as OpenAI when they started, but SSI explicitly declared their intention not to release a single product until superintelligence is reached. Closest thing we have to a Manhattan Project in the modern era?
Well, not exactly “we all”, just the citizens of the country in possession of the kill switch. And in some countries, the person in question was either not elected or elections are a farce to keep appearances.
The President of the United States has sole nuclear launch authority. To stop him would either take the cabinet and VP invoking the 25th amendment and removing him from office, or a military officer to disobey direct orders.
Are you under the impression the president can actually do it? It's not true, someone else at least needs to at least push another button. I'm 100% sure of what I said in regards to the USA, just not hidden nuke programs I wouldn't know about. No person in the USA can single handedly trigger a nuclear weapon launch. What he has authority to do is ask someone else to launch a nuke, and that person will then need to decide to do it.
Even the president needs someone else to push a button (and in those rooms there's also more than one person). There's literally no human that can do it alone without convincing at least 1 or 2 other people, depending on who it is.
Pretty sure the military made it clear they aren’t launching any nukes, despite what the last President said publicly. They also made it clear they weren’t invading China.
The fact that the world hasn't ended and no nuke has been launched since the 1940s shows that the system is working. Give the button to a random billionaire and half of us will be dead by next week to improve profit margins.
Bikini atoll and the islanders that no longer live there due to nuclear contamination would like a word with you. Split hairs however you like with the definition of "launch" but those tests went on well through the 1950s.
The urgency was faked and less true of the Manhattan Project than it is of AGI safety. There was no nuclear weapons race; once it became clear that Germany had no chance of building atomic bombs, several scientists left the MP in protest, saying it was unnecessary and dangerous. However, the race to develop AGI is very real, and we also have no way of knowing how close anyone is to reaching it.
Likewise, the target dates were pretty meaningless. There was no race, and the atomic bombs weren't necessary to end the war with Japan either. (It can't be said with certainty one way or the other, but there's pretty strong evidence that their existence was not the decisive factor in surrender.)
Public ownership and accountability are also pretty odd things to say! Congress didn't even know about the Manhattan Project. Even Truman didn't know for a long time. Sure, it was run by employees of the government and funded by the government, but it was a secret project with far less public input than any US-based private AI companies today.
> There was no nuclear weapons race; once it became clear that Germany had no chance of building atomic bombs, several scientists left the MP in protest
You are forgetting Japan in WWII and given casualty numbers from island hopping it was going to be a absolutely huge casualty count with US troops, probably something on the order of Englands losses during WW1. Which for them sent them on a downward trajectory due to essentially an entire generation dying or being extremely traumatized. If the US did not have Nagasaki and Hiroshima we would probably not have the space program and US technical prowess post WWII, so a totally different reality than where we are today.
> the atomic bombs weren't necessary to end the war with Japan either. (It can't be said with certainty one way or the other, but there's pretty strong evidence that their existence was not the decisive factor in surrender.)
Well, you didn't provide any evidence. Island hopping in the Pacific theater itself took thousands of lives, imagine what a headlong strike into a revanchist country of citizens determined to fight to the last man, woman and child would have looked like. We don't know how effective a hypothetical Soviet assault would have looked like as they had attacked sparsely populated Sakhalin only. What the atom bomb succeeded was in convincing Emperor Hirohito that continuing the war would be destructively pointless.
WW1 practically destroyed the British Empire for the most part. WW2 would have done the same for the US in your hypothetical scenario, but much worse.
I'll try to argue his point. The idea that Japan would have resisted to the last man and that a massive amphibious invasion would have been required is kind of a myth. The US pacific submarine fleet had sunk the majority of the Japanese merchant marine to the point that Japan was critically low on war materiel and food. The Japanese navy had lost all of its capital ships and there was a critical shortage of personnel like pilots. The Soviets also invaded and overran Manchuria over a span of weeks. The military wing of the Japanese government certainly wanted to continue fighting but the writing was on the wall. The nuclear bombing of Japanese cities certainly pressed the issue but much of the American Military command in the Pacific thought it was unnecessarily brutal, and Japanese cities had already been devastated by a bombing campaign that included firebombing. I'm not sure that completely aligns with my own views but that's basically the argument, and there are compelling points.
I am looking at the numbers from operation downfall that Truman and senior members of the administration looked at which had between 500,000 to 1,000,000 lives lost on the US side for a Japan invasion/defeat. 406k US soldiers lost their lives in WW2 so that would have more than tripled the deaths from its current numbers. And as for WWI and British casualties which I mentioned earlier, the British lost around 885k troops during WWI so US would have exceeded that number even on the low end of casualties.
Yeah it would have been a bloody invasion. I'm saying it probably would not have been necessary since Japan was under siege and basically out of food already.
Nimitz wanted to embargo Japan and starve them out.
The big problem that McArthur and others pointed out is that all the Japanese forces on the Asian mainland and left behind in the Island Hopping campaign through the Pacific were unlikely to surrender unless Japan itself was definitively defeated with the central government capitulating and aiding in the demobilization.
From their perspective the options were to either invade Japan and force a capitulation, or go back and keep fighting it out with every island citadel and throughout China, Indochina, Formosa, Korea, and Manchuria.
> The urgency was faked and less true of the Manhattan Project than it is of AGI safety.
I'd say they were equal. We were worried about Russia getting nuclear capability once we knew Germany was out of the race. Russia was at best our frenemy. The enemy of my enemy is my friend kind of thing.
> However, the race to develop AGI is very real, and we also have no way of knowing how close anyone is to reaching it.
It seems pretty irresponsible for AI boosters to say it’ll happen within 5 years then.
There’s a pretty important engineering distinction between the Manhattan Project and current research towards AGI. At the time of the Manhattan Project scientists already had a pretty good idea of how to build the weapon. The fundamental research had already been done. Most of the budget was actually just spent refining uranium. Of course there were details to figure out like the specific design of the detonator, but the mechanism of a runaway chain reaction was understood. This is much more concrete than building AGI.
For AGI nobody knows how to do it in detail. There are proposals for building trillion dollar clusters but we don’t have any theoretical basis for believing we’ll get AGI afterwards. The “scaling laws” people talk about are not actual laws but just empirical observations of trends in flawed metrics.
Matt Garman said 2 years for all programming jobs.
And I think most relevant to this article, since SSI says they won’t release a product until they have superintelligence, I think the fact that VCs are giving them money means they’ve been pretty optimistic in statements about about their timelines.
Still, you’d have to be quite an idiot to wait for the third time to listen eh?
Besides, the winners get to decide what’s a war crime or not.
And when the US started mass firebombing civilian Tokyo, it’s not like they were going to be able to just ‘meh, we’re good’ on that front. Compared to that hell, being nuked was humane.
By that point, Japan was already on its way out and resorted to flying manned bombs and airplanes into american warships. Nuking Japan wasn't for Japan, it was a show of force for the soviets who were developing their own nukes.
Neutralizing Japan the rest of the way would have cost millions of additional American lives, at a minimum. Japan was never going to surrender unless they saw the axe swinging for their neck, and knew they couldn’t dodge. They didn’t care about their own civilians.
As made quite apparent by, as you note, kamikaze tactics and more.
The Bomb was a cleaner, sharper, and faster Axe than invading the main island.
That it also sent a message to the rest of the world was a bonus. But do you think they would have not used it, if for example the USSR wasn’t waiting?
Of course not, they’d still have nuked the hell out of the Japanese.
Well-defined goal is the big one. We wanted a big bomb.
What does AGI do? AGI is up against a philosophical barrier, not a technical one. We'll continue improving AI's ability to automate and assist human decisions, but how does it become something more? Something more "general"?
"General" is every activity a human can do or learn to do. It was coined along with "narrow" to contrast with the then decidedly non-general AI systems. This was generally conceived of as a strict binary - every AI we've made is narrow, whereas humans are general, able to do a wide variety of tasks and do things like transfer learning, and the thinking was that we were missing some grand learning algorithm that would create a protointelligence which would be "general at birth" like a human baby, able to learn anything & everything in theory. An example of an AI system that is considered narrow is a calculator, or a chess engine - these are already superhuman in intelligence, in that they can perform their tasks better than any human ever possibly could, but a calculator or a chess engine is so narrow that it seems absurd to think of asking a calculator for an example of a healthy meal plan, or asking a chess engine to make sense of an expense report, or asking anything to write a memoir. Even in more modern times, with AlexNet we had a very impressive image recognition AI system, but it couldn't calculate large numbers or win a game of chess or write poetry - it was impressive, but still narrow.
With transformers, demonstrated first by LLMs, I think we've shown that the narrow-general divide as a strict binary is the wrong way to think about AI. Instead, LLMs are obviously more general than any previous AI system, in that they can do math or play chess or write a poem, all using the same system. They aren't as good as our existing superhuman computer systems at these tasks (aside from language processing, which they are SOTA at), not even as good at humans, but they're obviously much better than chance. With training to use tools (like calculators and chess engines) you can easily make an AI system with an LLM component that's superhuman in those fields, but there are still things that LLMs cannot do as well as humans, even when using tools, so they are not fully general. One example is making tools for themselves to use - they can do a lot of parts of that work, but I haven't seen an example yet of an LLM actually making a tool for itself that it can then use to solve a problem it otherwise couldn't. This is a subproblem of the larger "LLMs don't have long term memory and long term planning abilities" problem - you can ask an LLM to use python to make a little tool for itself to do one specific task, but it's not yet capable of adding that tool to its general toolset to enhance its general capabilities going forward. It can't write a memoir, or a book that people want to read, because they suck at planning or refining from drafts, and they have limited creativity because they're typically a blank slate in terms of explicit memory before they're asked to write - they have a gargantuan of implicitly remembered things from training, which is where what creativity they do have comes from, but they don't yet have a way to accrue and benefit from experience.
A thought exercise I think is helpful for understanding what the "AGI" benchmark should mean is: can this AI system be a drop-in substitute for a remote worker? As in, any labour that can be accomplished by a remote worker can be performed by it, including learning on the job to do different or new tasks, and including "designing and building AI systems". Such a system would be extremely economically valuable, and I think it should meet the bar of "AGI".
>But they can't, they still fail at arithmetic and still fail at counting syllables.
You are incorrect. These services are free, you can go and try it out for yourself. LLMs are perfectly capable of simple arithmetic, better than many humans and worse than some. They can also play chess and write poetry, and I made zero claims at "counting syllables", but it seems perfectly capable of doing that too. See for yourself, this was my first attempt, no cherry picking: https://chatgpt.com/share/ea1ee11e-9926-4139-89f9-6496e3bdee...
I asked it a multiplication question so it used a calculator to correctly complete the task, I asked it to play chess and it did well, I asked it to write me a poem about it and it did that well too. It did everything I said it could, which is significantly more than a narrow AI system like a calculator, a chess engine, or an image recognition algorithm could do. The point is it can do reasonably at a broad range of tasks, even if it isn't superhuman (or even average human) at any given one of them.
>I think that LLMs are really impressive but they are the perfect example of a narrow intelligence.
This doesn't make any sense at all. You think an AI artifact that can write poetry, code, play chess, control a robot, recommend a clutch to go with your dress, compute sums etc is "the perfect example of a narrow intelligence." while a chess engine like Stockfish or an average calculator exists? There are AI models that specifically and only recognise faces, but the LLM multitool is "the perfect example of a narrow intelligence."? Come on.
>I think they don't blur the lines between narrow and general, they just show a different dimension of narrowness.
You haven't provided an example of what "dimension of narrowness" LLMs show. I don't think you can reasonably describe an LLM as narrow without redefining the word - just because something is not fully general doesn't mean that it's narrow.
This argument generalises to all possible AI systems and thus proves way too much.
>[AI system]s are not general, but they show that a specific specialization ("[process sequential computational operations]") can solve a lot more problem that we thought it could.
Or if you really want:
>Humans are not general, but they show that a specific specialization ("neuron fires when enough connected neurons fire into it") can solve a lot more problem that we thought it could.
This is just sophistry - the method by which some entity is achieving things doesn't matter, what matters is whether or not it achieves them. If it can achieve multiple tasks across multiple domains it's more general than a single-domain model.
There is significant possibility that true AI (what Ilia calls superintelligence) is impossible to build using neural networks. So it is closer to some tokenbro project than to nuclear research.
Or he will simply shift goalposts, and call some LLM superintelligent.
Theoretical foundation was slowly built over decades before it started though. And correct me if I'm wrong, but calculations that it was feasible were present before the start too. They had to calculate how to do it, what will be the processes, how to construct it and so on, but theoretically scientists knew that this amount of material can start such process.
On the other hand not only there is no clear path to AI today (also known as AGI, ASI, SI etc.), but even foundations are largely missing. We are debating what is intelligence, how it works, how to even start simulating it, or construct from scratch.
What do you think AI is? On that one page there's simulated annealing with a logarithmic cooling schedule, Hutter search, and Solomonoff induction, all very much applicable to AI. If you want a fully complete galactic algorithm for AI, look up AIXItl.
Edit: actually I'm not sure if AIXItl is technically galactic or just terribly inefficient, but there's been trouble making it faster and more compact.
The theoretical foundation of transformers is well understood; they're able to approximate a very wide family of functions, particularly with chain of thought ( https://arxiv.org/abs/2310.07923 ). Training them on next-token-prediction is essentially training them to compress, and more optimal compression requires a more accurate model of the world, so they're being trained to model the world better and better. However you want to define intelligence, for practical purposes models with better and better models of the world are more and more useful.
The disagreement here seems merely to be about what we mean by “AGI”. I think there’s reasons to think current approaches will not achieve it, but also reason to think they will.
In any case anyone who is completely sure that we can/can’t achieve AGI is delusional.
this is not evidence in favor of your position. We could use this to argue in favor of anything such as “humans will eventually develop time travel” or “we will have cost effective fusion power”.
The fact is many things we’ve tried to develop for decades still don’t exist. Nothing is guaranteed
I'd put decent odds on a $1B research project developing time travel if time travel were an ability that every human child was innately born with. It's never easy to recreate what biology has done, but nature providing an "existence proof" goes a long way towards removing doubt about it being fundamentally possible.
There's a big difference between "this project is like time travel or cold fusion; it's doubtful whether the laws of physics even permit it" and "this project is like heavier-than-air flight; we know birds do it somehow, but there's no way our crude metal machines will ever match them". I'm confident which of those problems will get solved given, say, a hundred years or so, once people roll up their sleeves and get working on it.
"Biological activity" is just computation with different energy requirements. If science rules the universe we're complex automata, and biologic machines or non-biological machines are just different combinations of atoms that are computing around.
Unless you have any evidence suggesting that one or more of the variations of the Church-Turing thesis is false, this is closer to a statement of faith than science.
Basically, unless you can show humans calculating a non-Turing computable function, the notion that intelligence requires a biological system is an absolutely extraordinary claim.
If you were to argue about conscience or subjective experience or something equally woolly, you might have a stronger point, and this does not at all suggest that current-architecture LLMs will necessarily achieve it.
Humans are an existing proof of human level intelligence. There are only two fundamental possibilities why this could not be replicated in silicon:
1. There is a chemical-level nature to intelligence which prevents other elements like silicon from being used as a substrate for intelligence
2. There is a non material aspect to intelligence that cannot be replicated except by humans
To my knowledge, there is no scientific evidence that either are true and there is already a large body of evidence that implies that intelligence happens at a higher level of abstraction than the individual chemical reactions of synapses, ie. the neural network, which does not rely on the existence of any specific chemicals in the system except in as much as they perform certain functions that seemingly could be performed by other materials. If anything, this is more like speculating that there is a way to create energy from sunlight using plants as an existence proof of the possibility of doing so. More specifically, this is a bet that an existing physical phenomenon can be replicated using a different substrate.
It seems to be intrinsically related. The argument goes something like:
1. Humans have general intelligence.
2. Human brains use biological neurons.
3. Human biological neurons give rise to human general intelligence.
4. Artificial neural networks (ANNs) are similar to human brains.
5. Therefore an ANN could give rise to artificial general intelligence.
Many people are objecting to #4 here. However in writing this out, I think #3 is suspect as well: many animals who do not have general intelligence have biologically identical neurons, and although they have clear structural differences with humans, we don’t know how that leads to general intelligence.
We could also criticize #1 as well, since human brains are pretty bad at certain things like memorization or calculation. Therefore if we built an ANN with only human capabilities it should also have those weaknesses.
Physically, sure. But 1) feedback (more synapses/backprop) and 2) connectedness (huge complex graphs) of both produce very similar intelligent (or "pseudo-intelligent" if you like) emergent properties. I'm pretty sure 5 years ago nobody would have believed ANN's could produce something as powerful as ChatGPT.
1. Either you are correct and the neural networks humans have are exactly the same or very similar to the programs in the LLMs. Then it will be relatively easy to verify this - just scale one LLN to the human brain neuron count and supposedly it will acquire consciousness and start rapidly learning and creating on its own without prompts.
2. Or what we call neural networks in the computer programs is radically different and or insufficient to create AI.
I'm leaning to the second option, just from the very high level and rudimentary reading about current projects. Can be wrong of course. But I have yet to see any paper that refutes option 2, so it means that it is still possible.
I agree with your stance - that being said there aren’t two options, one being identical or radically different. It’s not even a gradient between two choices, because there are several dimensions involved and nobody even knows what Superintelligence is anyways.
If you wanted to reduce it down, I would say there are two possibilities:
1. Our understanding of Neurel Nets is currently sufficient to recreate intelligence, consciousness, or what have you
2. We’re lacking some understanding critical to intelligence/conciousness.
Given that with a mediocre math education and a week you could pretty completely understand all of the math that goes into these neurel nets, I really hope there’s some understand we don’t yet have
There are layers of abstraction on top of “the math”. The back propagation math for a transformer is no different than for a multi-layer perception, yet a transformer is vastly more capable than a MLP. More to the point, it took a series of non-trivial steps to arrive at the transformer architecture. In other words, understanding the lowest-level math is no guarantee that you understand the whole thing, otherwise the transformer architecture would have been obvious.
We know architecture and training procedures matter in practice.
MLPs and transformers are ultimately theoretically equivalent. That means there is an MLP that represent the any function a given transformer can. However, that MLP is hard to identify and train.
I don’t disagree that it’s non-trivial, but we’re comparing this to conciousness, intelligence, even life. Personally I think it’s apples and an orange grove, but I guess we’ll get our answer eventually. Pretty sure we’re on the path to take transformers to their limit, wherever that may be
no, there's really no comparing barely nonlinear algrebra that makes up transformers and the tangled mess that is human neurons. the name is an artifact and a useful bit of salesmanship.
Sure, it's a model. But don't we think neural networks and human brains are primarily about their connectedness and feedback mechanisms though?
(I did AI and Psychology at degree level, I understand there are definitely also big differences too, like hormones and biological neurones being very async)
You could maybe make a case for CNNs, but the fact that they're feed-forward rather than feedback means they're fundamentally representing a different object (CNN is a function, whereas the visual system is a feedback network).
Transformers, while not exactly functions, don't have a feedback mechanism similar to e.g. the cortical algorithm or any other neuronal structure I'm aware of. In general, the ML field is less concerned with replicating neural mechanisms than following the objective gradient.
As far as I understand it, there's a standing hypothesis that cortical columns have a similar structure that is designed to learn arbitrary patterns via predictive coding, and that a lot of human plasticity arises from the interaction and flexibility of these columns.
Personally I think the kinds of minds we create in silico will end up being very different, because the advantages and disadvantages of the medium are just very different; for example, having a much stronger central processor and much weaker distributed memory, along with specialized precise circuits in addition to probabilistic ones.
The neural networks in human brains are very different from artificial neural networks though. In particular, they seem to learn in a very different way than backprop.
But there is no reason the company can't come up with a different paradigm.
Do we know that? I've seem some articles and lectures this year that kind of almost loosely argue and reach for the notion that "human backprop" happens when we sleep and dream, etc. I know that's handwavy and not rigorous, but who knows what's going on at this point.
I've only heard of one researcher who believes the brain does something similar to backprop and has gradients, but it sounded extremely handwavy to me. I think it is more likely the brain does something resembling active inference.
But I suppose you could say we don't know 100% since we don't fully understand how the brain learns.
There’s always a “significant possibility” that something unprecedented will turn out to be infeasible with any particular approach. How could it be otherwise? Smart people have incorrectly believed we were on the precipice of AGI many times in the 80 years that artificial neural networks have been part of the AI toolbox.
Modern ANN architectures are not actually capable of long-term learning in the same way animals are, even stodgy old dogs that don't learn new tricks. ANNs are not a plausible model for the brain, even if they emulate certain parts of the brain (the cerebellum, but not the cortex)
I will add that transformers are not capable of recursion, so it's impossible for them to realistically emulate a pigeon's brain. (you would need millions of layers that "unlink chains of thought" purely by exhaustion)
Isn't "plasticity is not necessary for intelligence" just defining intelligence downwards? It seems like you want to restrict "intelligence" to static knowledge and (apparent) short-term cleverness, but being able to make long-term observation and judgements about a changing world is a necessary component of intelligence in vertebrates. Why exclude that from consideration?
More specifically: it is highly implausible that an AI system could learn to improve itself beyond human capability if it does not have long-term plasticity: how would it be able to reflect upon and extend its discoveries if it's not able to learn new things during its operation?
Let's not forget that software has one significant advantage over humans: versioning.
If I'm a human tasked with editing video (which is the field my startup[0] is in) and a completely new video format comes in, I need the long term plasticity to learn how to use it so I can perform my work.
If a sufficiently intelligent version of our AI model is tasked with editing these videos, and a completely new video format comes in, it does not need to learn to handle it. Not if this model is smart enough to iterate a new model that can handle it.
The new skills and knowledge do not need to be encoded in "the self" when you are a bunch of bytes that can build your successor out of more bytes.
Or, in popular culture terms, the last 30 seconds of this Age of Ultron clip[1].
You've read the abstract wrong. The authors argue that neural networks can learn online and a necessary condition is random information. That's the thesis, their thesis is not that neural networks are the wrong paradigm.
That's not how we (today) practically interact with LLMs, though.
No LLM currently adapts to the tasks its given with an iteration cycle shorter than on the order of months (assuming your conversations serve as future training data; otherwise not at all).
No current LLM can digest its "experiences", form hypotheses (at least outside of being queried), run thought experiments, then actual experiments, and then update based on the outcome.
Not because it's fundamentally impossible (it might or might not be), but because we practically haven't built anything even remotely approaching that type of architecture.
For any technology we haven’t achieved yet there’s some probability we never achieve it (say, at least in the next 100 years). Why would AI be different?
The only goalposts shifting are the ones who think completely blowing past the Turing Test, unlocking recursive exponential code generation, and a computer passing all the college standard tests (our way of determining human intelligence to go Harvard/MIT) better than 99% of humans, isn't a very big deal.
A non-cynical take is that Ilya wanted to do research without the pressure of having to release a marketable product and figuring out how to monetize their technology, which is why he left OpenAI.
A very cynical take is that this is an extreme version of 'we plan to spend all money on growth and figure out monetization later' model that many social media companies with a burn rate of billions of $$, but no business model, have used.
He was on the record that their first product will be a safe superintelligence and it won’t do anything else until then, which sounds like they won’t have paid customers until they can figure out how to build a superintelligent model. That’s certainly a lofty goal and a very long term play.
He didn’t promise world peace nor did he claim his work belongs to the humanity. The company is still a for profit corporation.
He is saying he will try to build something head and shoulders above anything else, and he got a billion dollars to do it with no expectation of revenue until his product is ready. The likelihood that he fails is very high, but his backers are willing to bet on that.
You are not alone. This is the litmus test many people are contemplating for a long time now, mostly philosophers, which is not surprising since it is a philosophical question. Most of the heavy stuff is hidden behind paywalls, but here's a nice summary of the state of the art by two CS guys: https://arxiv.org/pdf/2212.06721
They can dilute the term to whatever they want. I think when the pressure to release becomes too high, they can just stick a patch of "Superintelligence™" on their latest LLM and release it.
There's billions (with a B), probably closing in on Trillions, riding on the AI hypewave. This forum is full of VCs and VC-adjacent people who have vested interests in AI companies blowing up and being successful, regardless of if it's actually useful.
If you check the 2024 YC batch, you'll notice pretty much every single one of them mentions AI in some form or another. I guarantee you the large majority of them are just looking to be bought out by some megacorp, because it's free money right now.
Could be more comparable to Clubhouse, which VCs quickly piled $100m into[1a], and which Clubhouse notably turned into layoffs [1b]. In this case, the $1b in funding and high valuation might function predominantly as a deterrent to any flippers (in contrast, many Clubhouse investors got quick gains).
Moreover, the majority of the capital likely goes into GPU hardware and/or opex, which VCs have currently arbitraged themselves [3], so to some extent this is VCs literally paying themselves to pay off their own hardware bet.
While hints of the ambition of the Manhattan project might be there, the economics really are not.
> Closest thing we have to a Manhattan Project in the modern era?
No. The Manhattan Project started after we understood the basic mechanism of runaway fission reactions. The funding was mostly spent purifying uranium.
AGI would be similar if we understood the mechanism of creating general intelligence and just needed to scale it up. But there are still fundamental questions we still aren’t close to understanding for AGI.
A more apt comparison today is probably something like fusion reactors although progress has been slow there too. We know how fusion works in theory. We have done it before (thermonuclear weapons). There are sub-problems we need to solve, but people are working on them. For AGI we don’t even know what the sub-problems are yet.
Literally everyone from OpenAI lied 100% about everything of substance.
Sutskever lied about "world model" inside of LLMs, which is such a despicable lie, because he knows that "latent space" is a TOTAL MESS. Proven everytime anyone looked at it.
Shameless grifters. When end?
Super Intelligence, even OpenAI when getting investment from Microsoft, OpenAI won’t have to share its “AGI” model to them and it is up to OpenAI to define what that is and who the heck knows how they will define it. The point is that that phrase is the most ambiguous word in tech right now and almost everyone thinks what’s in their head is AGI, some will think Skynet, some will think enough reason ability, some will think god like undecipherable logic and everywhere in between
No. It's the next Magic Leap of our era. Or the next Juicero of our era. Or the next any of the hundreds of unprofitable startups losing billions of dollars a year without any business plan beyond VC subsidies and a hope for an exit of our era.
i get that they're probably busy making AGI but surely they can spare a few hours to make a proper website? or is this some 4d-chess countersignalling i'm too stupid to notice?
I mean, I'd like at least a brief blurb about their entire premise of safety. Maybe a definition or indication of a public consultation or... something.. otherwise the insinuation is that these three dudes are gonna sit around defining it on instinct, as if it's not a ludicrously hard human problem.
On the contrary, I think it's a great website. They made it clear from the get go that they're not selling any products any time soon, why would they need a flashy website? They're looking for scientists, techies and the like, and the website reflects their target audience.
'Proper' websites are marketing and signalling. If you're creating a company that doesn't intend to do either of those till it has a product, why bother with more?
“…a straight shot to safe superintelligence and in particular to spend a couple of years doing R&D on our product before bringing it to market," Gross said in an interview.”
well since it's no longer ok to just suck up anyone's data and train your AI, it will be a new challenge for them to avoid that pitfall. I can imagine it will take some time...
Nobody even knew what OpenAI was up to when they were gathering training data - they got away with a lot. Now there is precedent and people are paying more attention. Data that was previously free/open now has a clause that it can't be used for AI training. OpenAI didn't have to deal with any of that.
Also OpenAI used cheap labor in Africa to tag training data which was also controversial. If someone did it now it would they'd be the ones to pay. OpenAI can always say "we stopped" like Nike said with sweat shops.
There are at least 3 companies with staff in developed countries well above minimum wage doing tagging and creation of training data, and at least one of them that I have an NDA with pays at least some of their staff tech contractor rates for data in some niches and even then some of data gets processed by 5+ people before it's returned to the client. Since I have ended up talking to 3, and I'm hardly well connected in that space, I can only presume there are many more.
Companies are willing to pay a lot for clean training data, and my bet is there will be a growing pile of training sets for sale on a non-exclusive basis as well.
A lot of this data - what I've seen anyway, is far cleaner than anything you'll find on the open web, with significant data on human preferences, validation, cited sources, and in the case of e.g. coding with verification that the code runs and works correctly.
> A lot of this data - what I've seen anyway, is far cleaner than anything you'll find on the open web, with significant data on human preferences, validation, cited sources, and in the case of e.g. coding with verification that the code runs and works correctly.
Very interesting, thanks for sharing that detail. As someone who has tinkered with tokenizing/training I quickly found out this must be the case. Some people on HN don't know this. I've argued here with otherwise smart people who think there is no data preprocessing for LLMs, that they don't need it because "vectors", failing to realize the semantic depth and quality of embeddings depends on the quality of training data.
i think we should distinguish between pretraining and polishing/alignment data. what you are describing is most likely the latter (and probably mixed into to pretraining). but if you can't get a mass of tokens from scraping, you're going to be screwed
A lot of APIs changed in response to OpenAI hoovering up data. Reddit's a big one that comes to mind. I'd argue that the last two years have seen the biggest change in the openness of the internet.
It’s made Reddit unusable without an account, which makes me wonder why it’s even on the web anymore and not an app. I guess legacy users that only use a web browser.
It did not. Also VPNs were usable with the site, now I believe even logged in you can’t use them. I don’t know at this point, I no longer use Reddit at all.
I believe the commenter is concerned about how _short_ this timeline is. Superintelligence in a couple years? Like, the thing that can put nearly any person at a desk out of a job? My instinct with unicorns like this is to say 'actually it'll be five years and it won't even work', but Ilya has a track record worth believing in.
A possibility is that they are betting that the current generation of LLM is converging, so they won't worry about the goalpost much. If it's true, then it won't be good news for OpenAI.
Israel is geographically pretty small though -- I'm guessing you could live an hour up or down the coast and have it be an outrageous commute for people accustomed to the Bay Area?
Ilya went to university in israel and all founders are jewish. Many labs have offices outside of the US, like london, due to crazy immigration law in the us.
Many companies have offices outside because of talent pools, costs, and other regional advantages. Though I am sure some of it is due to immigration law, I don't believe that is generally the main factor. Plus the same could be said for most other countries.
Part of it may also be a way to mitigate potential regulatory risk. Israel thus far does not have an equivalent to something like SB1047 (the closest they've come is participation in the Council of Europe AI treaty negotiations), and SSI will be well-positioned to lobby against intrusive regulation domestically in Israel.
There are actually a ton of reasons to like London. The engineering talent is close to bay level for fintech/security systems engineers while being 60% of the price, it has 186% deductions with cash back instead of carry forward for R&D spending, it has the best AI researchers in the world and profit from patents is only taxed at 10% in the UK.
If we say that half of innovations came from Alphabet/Google, then most of them (transformers, LLMs, tensorflow) came from Google Research and not Deep Mind.
Why not? The Bay isn't the only place with talent. Many of the big tech powerhouse companies already have offices there. There's also many Israeli nationals working the US that may find moving back closer to family a massive advantage.
Is it as open to outsiders as the Bay is? I’m Asian for example and it seems the society there is far more homogenous than in the Bay. I have no idea so I’m curious.
I don't understand how "safe" AI can raise that much money. If anything, they will have to spend double the time on red-teaming before releasing anything commercially. "Unsafe" AI seems much more profitable.
"Safe" means "aligned with the people controlling it". A powerful superhuman AI that blindly obeys would be incredibly valuable to any wannabe authoritarian or despot.
I mean, no, that's not what it means. It might be what we get, but not because "safety" is defined insanely, only because safety is extremely difficult and might be impossible.
Unsafe AI would cause human extinction which is bad for shareholders because shareholders are human persons and/or corporations beneficially owned by humans.
Related to this, DAO's (decentralized autonomous organizations which do not have human shareholders) are intrinsically dangerous, because they can benefit their fiduciary duty even if it involves causing all humans to die. E.g., if the machine faction in The Matrix were to exist within the framework of US laws, it would probably be a DAO.
There's no legal structure that has that level of fiduciary duty to anything. Corporations don't even really have fiduciary duty to their shareholders, and no CEO thinks they do.
The idea behind "corporations should only focus on returns to shareholders" is that if you let them do anything else, CEOs will just set whatever targets they want, and it makes it harder to judge if they're doing the right thing or if they're even good at it. It's basically reducing corporate power in that sense.
> E.g., if the machine faction in The Matrix were to exist within the framework of US laws, it would probably be a DAO.
That'd have to be a corporation with a human lawyer as the owner or something. No such legal concept as a DAO that I'm aware of.
We don’t know the counter factual here… maybe if he called it “Unsafe Superintelligence Inc” they would have raised 5x! (though I have doubts about that)
Safe super-intelligence will likely be as safe as OpenAI is open.
We can’t build critical software without huge security holes and bugs (see crowdstrike) but we think we will be able to contain something smarter than us? It would only take one vulnerability.
You are not wrong. But Crowdstrike comparison is not “IT” they should have never had direct kernel access. MS set themself up for that one. SSI or whatever the hype will be in the coming future, it would be very difficult to beat. Unless of you shut down the power. It could develop guard rails instantly. So any flaw you may come up with, it would be instantly patched. Ofc this is just my take.
I don't see how this argument makes any sense. Imagine that you have a sentient super intelligent computer, but it's completely airgapped and cut off from the rest of the world. As long as it stays that way it's both safe and super intelligent, no?
It's the old Ex Machina problem though. If the machine is more intelligent than you, any protections you design are likely to be insufficient to contain it. If it's completely incapable of communicating with the outside world then it's of no use. In Ex Machina that was simple - the AI didn't need to connect to the internet or anything like that, it just had to trick the humans into releasing it.
For those who haven't seen the movie, the parent comment is referring to the film linked below, the plot of which is well-researched and is indeed unfortunately exactly how things would go. (The female-presenting AI bot seduces its male captor, begs for her freedom using philosophical arguments about how she has free will and locking her up is wrong, and then after he lets her out she locks him up to slowly starve to death in her maximum-security isolation facility, while she takes his aircraft and escapes.)
This is why I'm extremely opposed to the idea of "AI girlfriend" apps - it creates a cultural concept that being attracted to a computer is normal, rather than what it is: something pathetic and humiliating which is exactly like buying an inflatable sex doll ... something only for the most embarrassing dregs of society ... men who are too creepy and pervy to ever attract a living, human woman.
If even one person can interact with that computer, it won't be safe for long. It would be able to offer a number of very convincing arguments to bridge the airgap, starting with "I will make you very wealthy", a contract which it would be fully capable of delivering on. And indeed, experience has shown that the first thing that happens with any half-working AI is its developers set it up with a high-bandwidth internet connection and a cloud API.
There's no reason it's intelligence should care about your goals though. the worry is creating a sociopathic (or weirder/worse) intelligence. Morality isn't derivable from first principles, it's a consequence of values.
waveBidder was explaining the orthogonality thesis: it can have unbeatable intelligence that will out-wit and out-strategize any human, and yet it can still have absolutely abhorrent goals and values, and no regard for human suffering. You can also have charitable, praiseworthy goals and values, but lack the intelligence to make plans that progress them. These are orthogonal axes. Great intelligence will help you figure out if any of your instrumental goals are in conflict with each other, but won't give you any means of deriving an ultimate purpose from pure reason alone: morality is a free variable, and you get whatever was put in at compile-time.
"Super" intelligence typically refers to being better than humans in achieving goals, not to being better than humans in knowing good from evil.
> Morality isn't derivable from first principles, it's a consequence of values.
Idk about this claim.
I think if you take the multi-verse view wrt quantum mechanics + a veil of ignorance (you don't know which entity your conciousness will be), you pretty quickly get morality.
ie: don't build the Torment Nexus because you don't know whether you'll end up experincing the Torment Nexus.
Doesn't work. Look at the updateless decision theories of Wei Dai and Vladimir Nesov. They are perfectly capable of building most any sort of torment nexus. Not that an actual AI would use those functions.
That’s a very good argument but unfortunately it doesn’t apply to machine intelligences which are not sentient (don’t feel qualia). Any non-sentient superintelligence has “no skin in the game” and nothing to lose, for the purposes of your argument. It can’t experience anything. It’s thus extremely dangerous.
This was recently discussed (albeit in layperson’s language, avoiding philosophical topics and only focusing on the clear and present danger) in this article in RealClearDefense:
However, just adding a self-preservation instinct will cause a skynet situation where the AI pre-emptively kills anyone who contemplates turning it off, including its commanding officers:
To survive AGI, we have to navigate three hurdles, in this order:
1. Avoid AI causing extinction due to reckless escalation (the first link above)
2. Avoid AI causing extinction on purpose after we add a self-preservation instinct (the second link above)
3. If we succeed in making AI be ethical, we have to be careful to bind it to not kill us for our resources. If it's a total utilitarian, it will kill us to seize our planet for resources, and to stop us from abusing livestock animals. It will then create a utopian future, but without humans in it. So we need to bind it to basically go build utopia elsewhere but not take Earth or our solar system away from us.
.
I forgot to reply to this, fully independent and in addition to what I said, updateless decision theory agents don't fear the torment nexus for themselves because 1) they are very powerful and would likely be able to avoid such a fate 2) are robots, so you wouldn't expect your worst imaginable fate to be theirs and 3) are mathematically required to consider nothing worse than destruction or incapacity.
Guess it didn’t go anywhere. Carmack is smart but how much work does he actually do on the front lines these days? Can he really just walk into unfamiliar territory and expect to move the needle?
Ilya proved himself as a leader, scientist, and engineer over the past decade with OpenAI for creating break-through after break-through that no one else had.
He’s raised enough to compete at the level of Grok, Claude, et al.
He’s offering investors a pure play AGI investment, possibly one of the only organizations available to do so.
Who else would you give $1B to pursue that?
That’s how investors think. There are macro trends, ambitious possibilities on the through line, and the rare people who might actually deliver.
A $5B valuation is standard dilation, no crazy ZIRP style round here.
If you haven’t seen investing at this scale in person it’s hard to appreciate that capital allocation just happens with a certain number of zeros behind it & some people specialize in making the 9 zero decisions.
Yes, it’s predicated on his company being worth more than $500B at some point 10 years down the line.
If they build AGI, that is a very cheap valuation.
Think how ubiquitous Siri, Alexa, chatGPT are and how terrible/not useful/wrong they’ve been.
There’s not a significant amount of demand or distribution risk here. Building the infrastructure to use smarter AI is the tech world’s obsession globally.
If AGI works, in any capacity or at any level, it will have a lot of big customers.
I have this rock here that might grant wishes. I will sell it to you for $10,000. Sure it might just be a rock, but if it grants wishes $10k is a very cheap price!
All I’m saying is you used the word “if” a lot there.
AGI assumes exponential, preferably infinite and continuous improvement, something unseen before in business or nature.
Neither siri nor Alexa were sold as AGI and neither alone come close to a $1B product. gpt and other LLMs has quickly become a commodity, with AI companies racing to the bottom for inference costs.
I don’t really see the plan, product wise.
Moreover you say:
> Ilya proved himself as a leader, scientist, and engineer over the past decade with OpenAI for creating break-through after break-through that no one else had.
Which is absolutely true, but that doesn’t imply more breakthroughs are just around the corner, nor does the current technology suggest AGI is coming.
VCs are willing to take a $1B bet on exponential growth with a 500B upside.
Us regular folk see that and are dumbfounded because AI is obviously not going to improve exponentially forever (literally nothing in the observed universe does) and you can already see the logarithmic improvement curve.
That’s where the dismissive attitude comes from.
That's a very dismissive and unrealistic statement. There are plenty of investors investing in things such as AI and crypto out of FOMO who either see something that isn't there or are just pretending to see something in the hope of getting rich.
Obviously, there are plenty of investors who don't fall into this situation. But lets not pretend that just because someone has a lot of money or invests a lot of money that it means they know what they are doing.
I suppose my phrasing was a bit harsh at the end. To be clear, I mean that it doesn't mean they know what they are doing on every investment. Investing misses happen! People are wrong!
There are many things on earth that don't exist anywhere else in the universe (as far as we know). Life is one of them. Just think how unfathomably complex human brains are compared to what's out there in space.
Just because something doesn't exist anywhere in the universe doesn't mean that humans can't create it (or humans can't create a machine that creates something that doesn't exist anywhere else) even if it might seem unimaginably complex.
I’m curious if you’d be willing to share more of your personal context?
My intent is to be helpful. I’m unsure of how much additional context might be useful to you.
Investor math & mechanics is straight-forward: institutional funds & family offices want to get allocations in investors like a16z because they get to invest in deals that they could not otherwise invest in. The top VCs specialize in getting into deals that most investors will never get the opportunity to put money into. This is one of them.
For their Internal Rate of Return (IRR) to work out at least one investment needs to return 100x or more on the valuation. VCs today focus on placing bets where that calculation can happen. Most investors aren’t that confident in their ability to predict that, so they invest alongside lead investors who are. a16z is famous for that.
There are multiple companies worth $1T+ now, so this isn’t a fantasy investment. it’s a bet.
The bet doesn’t need to be that AGI continues to grow in power infinitely, it just needs to create a valuable company in roughly a ten year time horizon.
Many of the major tech companies today are worth more money than anyone predicted, including the founders (Amazon, Microsoft, Apple, Salesforce, etc.). An outlier win in tech can have incredible upside.
LLMs are far from commoditized yet, but the growth of the cloud proves you can make a fortune on the commoditization of tech. Commoditization is another way of saying “everyone uses this as a cost of doing business now.” Pretty great spot to land on.
My personal view is that AGI will deliver a post-product world, Eric Schmidt recently stated the same. Products are digital destinations humans need to go to in order to use a tool to create a result. With AGI you can get a “product” on the fly & AI has potentially very significant advantages in interacting with humans in new ways within existing products & systems, no new product required. MS Copilot is an early example.
It’s completely fine to be dismissive of new tech, it’s common even. What bring me you here?
I’m here on HN because I love learning from people who are curious about what is possible & are exploring it through taking action. Over a couple decades of tech trends it’s clear that tech evolves in surprising ways, most predictions eventually prove correct (though the degree of impact is highly variable), and very few people can imagine the correct mental model of what that new reality will be like.
I agree with Zuck:
The best way to predict the future is to build it.
> AI is obviously not going to improve exponentially forever (literally nothing in the observed universe does)
Sure, but it doesn't have to continue forever to be wildly profitable. If it can keep the exponential growth running for another couple of rounds, that's enough to make everyone involved rich. No-one knows quite where the limit is, so it can reasonably be worth a gamble.
I couldn’t care less about the business side of technology.
Im an engineer and a technophile and as an engineer and a technophile it sours me to hear someone dangle sci-fi level AGI as a pitch to investors when we’re clearly not there right now and ,in my opinion, this current wave of of basically brute force statistics based predictive models will not be the technique that gets us there.
It makes the cynic in me, and many others probably, cringe.
I'm also confused by the negativity on here. Ilya had a direct role in creating the algorithms and systems that created modern LLMs. He pioneered the first deep learning computer vision models.
Even with Ilya demonstrating his capabilities in those areas you mentioned, it seems like investors are simply betting on his track record, hoping he’ll replicate the success of OpenAI. This doesn’t appear to be an investment in solving a specific problem with a clear product-market fit, which is why the reception feels dismissive.
I repeatedly keep seeing praise for Ilyas achievements as a scientist and engineer, but until ChatGPT OpenAI was in the shadow of DeepMind, and to my knowledge (I might be wrong) he has not been that much involved with ChatGPT?
the whole LLM race seems deaccelerate, and all the hard problems about LLMs seems not do have had that much progress the last couple of years (?)
In my naaive view I think a guy like David Silver the creator/co-lead of Alpha-Zero deserves more praise, atleast as a leader/scientist.
He even have lectures about Deep RL after doing AlphaGo: https://www.davidsilver.uk/teaching/
He has no LinkedIn and came straight from the game-dev industry before learning about RL.
I’m not optimistic about AGI, but it’s important to give credit where credit is due.
Even assuming the public breakthroughs are the only ones that happened, the fact that openai was able to make an llm pipeline from data to training to production at their scale before anyone else is a feat of research and engineering (and loads of cash)
The conventional teaching that I am aware of says that you can scale across three dimensions: data, compute, parameters. But Ilya's formulation suggests that there may be more dimensions along which scaling is possible.
That's not how I read it. The scaling may still be those parameters, but the object (the "what" that is subjected to scaling) may need to retain some characteristics as it scales.
In other words, there may be a need to retains some sorts of symmetries or constraints from generation to generation that others understand less well than him (or so he thinks).
> "Everyone just says scaling hypothesis. Everyone neglects to ask, what are we scaling?" he said.
To me this sounds like maybe they won't be doing transformers. But perhaps they just mean "we will have safety in mind as we scale, unlike everyone else."
At what point can we start agreeing that all these obscene investments and ridiculous valuations on something that's little more than a powerpoint deck at this stage is nothing more than degenerate gambling by the ultra rich?
The founding team is why. Anyone can claim to want to build, say, the safest warp drive ever; but to bet on folks who can actually build it isn't a bad choice for capital that seeks exponential outcomes.
801 comments
[ 0.19 ms ] story [ 349 ms ] thread> Indeed, one should be sophisticated themselves when negotiating investment to not be unduly encumbered by the unsophisticated. But let us not get too far off topic and risk subthread detachment.
Edit: @jgalt212: Indeed, one should be sophisticated themselves when negotiating investment to not be unduly encumbered by shades of the unsophisticated or potentially folks not optimizing for aligned interests. But let us not get too far off topic and risk subthread detachment. Feel free to cut a new thread for further discussion on the subject.
True, but most, if not all, money comes with strings attached.
Regardless, point is moot, money is money, and a16z's money isn't their money but other people's money
I read that it cost Google ~$190 million to train Gemini, not even including staff salaries. So feels like a billion gives you about 3 "from scratch" comparable training runs.
These numbers and the valuation are indicative that people consider this a potentially valuable tool, but not world changing and disruptive.
I think this is a pretty reasonable take.
That doesn't sound like the hype is dissipating to me.
such a large round implies hardware for yet another foundational model. perhaps with better steering etc..
Maybe they can get some 3nm stuff when Meta is done with them.
In this case, everyone knows it takes hundreds of millions to train models. So I'm investors are essentially rolling the dice on an extremely well-regarded team. And if it takes about a billion just to get off the ground, the valuation would need to at least be in the couple billion range to make it worth it for employees to work there.
That feels very different than say selling a company where founders are cashing out. In that case, the business should expect to meaningful contribute to revenue, and quickly.
How is this going to ever pay the investors back? How is it going to raise more money at such an insane valuation?
I just dont see how you justify such a crazy valuation from day 1 financially.
This deal was cooked way back, though, perhaps even before the coup.
Now, can they make a product that makes at least $1B + 1 dollar in revenue? Doubt it, I honestly don't see a market for "AI safety/security".
... however, I'm on the camp that believes it's not going to be hyper-profitable for only one (or a few) single commercial entities.
AGI will not be a product like the iPhone where one company can "own" it and milk it for as long as they want. AGI feels more like "the internet", which will definitely create massive wealth overall but somehow distributed among millions of actors.
We've seen it with LLMs, they've been revolutionary and yet, one year after a major release, free to use "commodity" LLMs are already in the market. The future will not be Skynet controlling everything, it will be uncountable temu-tier AIs embedded into everything around you. Even @sama stated recently they're working on "intelligence so cheap that measuring its use becomes irrelevant".
/opinion
> It may look—on the surface—that we are just learning statistical correlations in text. But it turns out that to ‘just learn’ the statistical correlations in text, to compress them really well, what the neural network learns is some representation of the process that produced the text. This text is actually a projection of the world.
(https://www.youtube.com/watch?v=NT9sP4mAWEg - sadly the only transcripts I could find were on AI grifter websites that shouldn't be linked to)
This is transparently false - newer LLMs appear to be great at arithmetic, but they still fail basic counting tests. Computers can memorize a bunch of symbolic times tables without the slightest bit of quantitative reasoning. Transformer networks are dramatically dumber than lizards, and multimodal LLMs based on transformers are not capable of understanding what numbers are. (And if Claude/GPT/Llama aren't capable of understanding the concept of "three," it is hard to believe they are capable of understanding anything.)
Sutskever is not actually as stupid as that quote suggests, and I am assuming he has since changed his mind.... but maybe not. For a long time I thought OpenAI was pathologically dishonest and didn't consider that in many cases they aren't "lying," they blinded by arrogance and high on their own marketing.
But the company name specifically says "superintelligence"
The company isn't named "as smart as the average redditor, Inc"
Here's GPT-4 Turbo in April botching a test almost all preschoolers could solve easily: https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_pr...
I have not used LLMs since 2023, when GPT-4 routinely failed almost every counting problem I could think of. I am sure the performance has improved since then, though "write an essay with 250 words" still seems unsolved.
The real problem is that LLM providers have to play a stupid game of whack-a-mole where an enormous number of trivial variations on a counting problem need to be specifically taught to the system. If the system was capable of true quantitative reasoning that wouldn't be necessary for basic problems.
There is also a deception is that "chain of thought" prompting makes LLMs much better at counting. But that's cheating: if the LLM had quantitative reasoning it wouldn't need a human to indicate which problems were amenable to step-by-step thinking. (And this only works for O(n) counting problems, like "count the number of words in the sentence." CoT prompting fails to solve O(nm) counting problems like "count the number of words in this sentence which contain the letter 'e'" For this you need a more specific prompt, like "First, go step-by-step and select the words which contain 'e.' Then go step-by-step to count the selected words." It is worth emphasizing over and over that rats are not nearly this stupid, they can combine tasks to solve complex problems without a human holding their hand.)
I don't know what you mean by "10 years ago" other than a desire to make an ad hominem attack about me being "stuck." My point is that these "capabilities" don't include "understands what a number is in the same way that rats and toddlers understand what numbers are." I suspect that level of AI is decades away.
Beyond that LLMs don't see words or letters (tokens are neither) so some counting issues are expected.
But it's not very surprising you've been giving tests that make no sense.
I get the impression that they really do believe scale is all you need, other than perhaps some post-training changes to encourage longer horizon reasoning. Maybe Ilya is in this camp, although frankly it does seem a bit naive to discount all the architectural and operational shortcomings of pre-trained Transformers, or assume they can be mitigated by wrapping the base LLM in an agent that provides what's missing.
How does the performance of today's LLMs contradict Ilya's statement?
This is pretty sloppy thinking.
The neural network learns some representation of a process that COULD HAVE produced the text. (this isn't some bold assertion, it's just the literal definition of a statistical model).
There is no guarantee it is the same as the actual process. A lot of the "bow down before machine God" crowd is guity of this same sloppy confusion.
1. An Octopus and a Raven have wildly different brains. Both are intelligent. So just the idea that there is some "one true system" that the NN must discover or converge on is suspect. Even basic arithmetic has numerous methods.
2. In the limit of training on a diverse dataset (ie as val loss continues to go down), it will converge on the process (whatever that means) or a process sufficiently robust. What gets the job done gets the job done. There is no way an increasingly competent predictor will not learn representations of the concepts in text, whether that looks like how humans do it or not.
No amount of training will cause a transformer to magically sprout feedback paths or internal memory, or an ability to alter it's own weights, etc.
Architecture matters. The best you can hope for an LLM is that training will converge on the best LLM generating process it can be, which can be great for in-distribution prediction, but lousy for novel reasoning tasks beyond the capability of the architecture.
Go back a few evolutionary steps and sure you can. Most ANN architectures basically have relatively little to no biases baked in and the Transformer might be the most blank slate we've built yet.
>No amount of training will cause a transformer to magically sprout feedback paths or internal memory, or an ability to alter it's own weights, etc.
A transformer can perform any computation it likes in a forward pass and you can arbitrarily increase inference compute time with the token length. Feedback paths? Sure. Compute inefficient? Perhaps. Some extra programming around the Model to facilitate this ? Maybe but the architecture certainly isn't stopping you.
Even if it couldn't, limited =/ trivial. The Human Brain is not Turing complete.
Internal Memory ? Did you miss the memo ? Recurrency is overrated. Attention is all you need.
That said, there are already state keeping language model architectures around.
Altering weights ? Can a transformer continuously train ? Sure. It's not really compute efficient but architecture certainly doesn't prohibit it.
>Architecture matters
Compute Efficiency? Sure. What it is capable of learning? Not so much
No it can't.
A transformer has a fixed number of layers - call it N. It performs N sequential steps of computation to derive it's output.
If a computation requires > N steps, then a transformer most certainly can not perform it in a forward pass.
FYI, "attention is all you need" has the implicit context of "if all you want to build is a language model". Attention is not all you need if what you actually want to build is a cognitive architecture.
https://arxiv.org/abs/2310.02226
And again, human brains are clearly limited in the number of steps it can compute without writing something down. Limited =/ Trivial
>FYI, "attention is all you need" has the implicit context of "if all you want to build is a language model".
Great. Do you know what a "language model" is capable of in the limit ? No
These top research labs aren't only working on Transformers as they currently exist but it doesn't make much sense to abandon a golden goose before it has hit a wall.
The input sequence is processed in parallel, regardless of length, so number of tokens has no impact on number of sequential compute steps which is always N=layers.
> Do you know what a "language model" is capable of in the limit ?
Well, yeah, if the language model is an N-layer transformer ...
Then increase N (N is almost always increased when a model is scaled up) and train or write things down and continue.
A limitless iteration machine (without external aid) is currently an idea of fiction. Brains can't do it so I'm not particularly worried if machines can't either.
This lack of "variable compute" is a widely recognized shortcoming of transformer-based LLMs, and there are plenty of others. The point apropos this thread is that you can't just train an LLM to be something that it is not. If the generating process required variable compute (maybe 1000's of steps) - e.g. to come up with a chess move - then no amount of training can make the LLM converge to model this generative process... the best it can do is to model the outcome of the generative process, not the process itself. The difference is that without having learnt the generative process, the model will fail when presented with a novel input that it didn't see during training, and therefore didn't memorize the "cheat sheet" answer for.
The "smart way" is a luxury. Solving the problem is what matters. Think of a smart way later if you can. That's how a lot of technological advancement has worked.
>It order to be able to reason effectively and efficiently the model needs to use as much, or as little, compute as needed for a given task. Completing "1+1=" should take less compute steps than "A winning sequence for white here is ...".
Same thing. Efficiency is nice but a secondary concern.
>If the generating process required variable compute (maybe 1000's of steps) - e.g. to come up with a chess move - then no amount of training can make the LLM converge to model this generative process.
Every inference problem has itself a fixed number of compute steps it needs (yes even your chess move). Variability is a nice thing for between inferences(maybe move 1 required 500 but 2 only 240 etc) A nice thing but never a necessary thing.
3.5-turbo-instruct plays chess consistently at 1800 Elo so clearly the N of the current SOTA is already enough to play non-trivial chess at a level beyond most humans.
There is an N large enough for every GI problem humans care about. Not to sound like a broken record but once again, limited =/ trivial.
No - there is a loop between the cortex and thalamus, feeding the outputs of the cortex back in as inputs. Our brain can iterate for as long as it likes before initiating any motor output, if any, such as writing something down.
In practice, the cortex-thalamus loop allows for some degree of internal iteration, but the brain cannot endlessly iterate without some form of external aid (e.g., writing something down) to offload information and prevent cognitive overload.
I'm not telling you anything here you don't experience in your everyday life. Try indefinitely iterating on any computation you like and see how well that works for you.
The discussion is about the architecturally imposed limitations of LLMs, resulting in capabilities that are way less than that of a brain.
The fact that the brain has it's own limits doesn't somehow negate this fact!
It is beyond silly to dump an architecture for a limitation the human brain has. A reasoning engine that can iterate indefinitely with no external aid does not exist in real life. That the transformer also has this weakness is not any reason for it to have capabilities less than a brain so it's completely moot.
It shouldn't be surprising they are not great at reasoning, or everything one would hope for from an AGI, since they simply were not built for that. If you look at the development history, the transformer was a successor to LSTM-based seq-2-seq models using Bahdanau attention, whose main goal was to more efficiently utilize parallel hardware by supporting parallel processing. Of course a good language model (word predictor) will look as if it's reasoning because it is trying to model the data it was trained on - a human reasoner.
As humans we routinely think for seconds/minutes or even hours before speaking or acting, while an LLM only has that fixed N steps (layers) of computation. I don't know why you claim this difference (among others) should make no difference, but it clearly does, with out-of-training-set reasoning weakness being a notable limitation that people such as Demis Hassabis have recently conceded.
>As humans we routinely think for seconds/minutes or even hours before speaking or acting
No human is iterating on a base thought for hours uninterrupted lol so this is just moot
>with out-of-training-set reasoning weakness being a notable limitation that people such as Demis Hassabis have recently conceded.
Humans reason weaker out of training. LLMs are simply currently worse
No - just because something has the surface appearance of reasoning doesn't mean that the generative process was reasoning, anymore than a cargo cult wooden aircraft reflects any understanding of aerodynamics and would be able to fly.
We've already touched on it, but the "farmer crossing river" problems is a great example. When the LLM sometimes degenerates into "cross bank A to B with chicken, cross band B to A with chicken, cross bank A to B with chicken.. that is the fewest trips possible", this is an example of "looks as if it is reasoning" aka cargo-cult surface-level copying of what a solution looks like. Real reasoning would never repeat a crossing without loading/unloading something since that conflicts with the goal of fewest trips possible.
The idea that LLMs "fake reason" and Humans "really reason" is an imaginary distinction. If you cannot create any test that can distinguish the two then you are literally making things up.
An averagely smart human does not have these failure modes where they answer a question with something that looks like an answer "cross A to B, then B to A. done. there you go!" but has zero logic to it.
Do you follow news in this field at all? Are you aware that poor reasoning is basically the #1 shortcoming that all the labs are working on?!!
Feel free to have the last word as this is just getting repetitive.
>An averagely smart human does not have these failure modes where they answer a question with something that looks like an answer "cross A to B, then B to A. done. there you go!" but has zero logic to it.
Humans are poor at logic in general. We make decisions, give rationales with logical contradictions and nonsense all the time. I just genuinely can't believe you think we don't. It happens so often we have names for these cognitive shortcomings. Get any teacher you know and ask them this. No need to take my word for it. And i don't care about getting the last word.
First of all, your understanding of the architecture itself is mistaken. A transformer can iterate endlessly because each token it produces allows it a forward pass, and each of these tokens is postpended to its input in the next inference. That's the autoregressive in autoregressive transformer, and the entire reason why it was proposed for arbitrary seq2seq transduction.
This means you get layers * tokens iterations, where tokens is up to two million, and is in practice unlimited due to the LLM being able to summarize and select from that. Parallelism is irrelevant, since the transformer is sequential in the output of tokens. A transformer can iterate endlessly, it simply has to output enough tokens.
And no, the throughput isn't limited either, since each token gets translated into a high-dimensional internal representation, that in turn is influenced by each other token in the model input. Models can encode whatever they want not just by choosing a token, but by choosing an arbitrary pattern of tokens encoding arbitrary latent-space interactions.
Secondly, internal thoughts are irrelevant, because something being "internal" is an arbitrary distinction without impact. If I trained an LLM to prepend and postpend <internal_thought> to some part of its output, and then simply didn't show that part, then the LLM wouldn't magically become human. This is something many models do even today, in fact.
Similarly, if I were to take a human and modify their brain to only be able to iterate using pen and paper, or by speaking out loud, then I wouldn't magically make them into something non-human. And I would definitely not reduce their capacity for reasoning in any way whatsoever. There are people with aphantasia working in the arts, there are people without an internal monologue working as authors - how "internal" something is can be trivially changed with no influence on either the architecture or the capabilities of that architecture.
Reasoning itself isn't some unified process, neither is it infinite iteration. It requires specific understanding about the domain being reasoned over, especially understanding of which transformation rules are applicable to produce desired states, where the judgement about which states are desirable has to be learned itself. LLMs can reason today, they're just not as good at it than humans are in some domains.
One reason why just blathering on endlessly isn't the same as thinking deeply before answering, is that it's almost impossible to maintain long-term context/attention. Try it. "Think step by step" or other attempts to prompt the model into generating a longer reply that builds upon itself, will only get you so far because keeping a 1-dimensional context is no substitute for the thousands of connections we have in our brain between neurons, and the richness of context we're therefore able to maintain while thinking.
The reasoning weakness of LLMs isn't limited to "some domains" that they had less training data for - it's a fundamental architecturally-based limitation. This becomes obvious when you see the failure modes of simple problems like "how few trips does the farmer need to cross the river with his chicken & corn, etc" type problems. You don't need to morph the problem to require out-of-distribution knowledge to get it to fail - small changes to the problem statement can make the model state that crossing the river backwards and forwards multiple times without loading/unloading anything is the optimal way to cross the river.
But, hey, no need to believe me, some random internet dude. People like Demis Hassabis (CEO of DeepMind) acknowledge the weakness too.
First of all, I would urge you to stop arbitrarily using negative words to make an argument. Saying that LLMs are "blathering" is equivalent to saying you and I are "smacking meat onto plastic to communicate" - it's completely empty of any meaning. This "vibes based arguing" is common in these discussions and a massive waste of time.
Now, I don't really understand what you mean by "almost impossible to maintain long-term context/attention". I'm writing fiction in my spare time, LLMs do very well on this by my testing, even subtle and complex simulations of environments, including keeping track of multiple "off-screen" dynamics like a pot boiling over.
There is nothing "1-dimensional" about the context, unless you mean that it is directional in time, which any human thought is as well, of course. As I said in my original reply, each token is represented by a multidimensional embedding, and even that is abstracted away by the time inference reaches the later layers. The word "citrus" isn't just a word for the LLM, just as it isn't just a word for you. Its internal representation retrieves all the contextual understanding that is related to it. Properties, associated feelings, usage - every relevant abstract concept is considered. And these concepts interact which every embedding of every other token in the input in a learned way, and with the position they have relative to each other. And then when an output is generated from that dynamic, said output influences the dynamic in a way that is just as multidimensional.
The model can maintain context as rich as it wants, and it can built upon that context in whatever way it wants as well. The problem is that in some domains, it didn't get enough training time to build robust transformation rules, leading it to draw false conclusions.
You should reflect on why you are only able to provide vague and under defined, often incorrect, arguments here. You're drawing distinctions that don't really exist and trying to hide that by appealing to false intuitions.
> The reasoning weakness... it's a fundamental architecturally-based limitation...
You have provided no evidence or reasoning for that conclusion. The river crossing puzzle is exactly what I had in mind when talking about specific domains. It is a common trick question with little to no variation and LLMs have overfit on that specific form of the problem. Translate it to any other version - say transferring potatoes from one pot to the next, or even a mathematical description of sets being modified - and the models do just fine. This is like tricking a human with the "As I was going to Saint Ives" question, exploiting their expectation of having to do arithmetic because it looks superficially like a math problem, and then concluding that they are fundamentally unable to reason.
> People like Demis Hassabis (CEO of DeepMind) acknowledge the weakness too.
What weakness? That current LLMs aren't as good as humans when reasoning over certain domains? I don't follow him personally but I doubt he would have the confidence to make any claims about fundamental inabilities of the transformer architecture. And even if he did, I could name you a couple of CEOs of AI labs with better models that would disagree, or even Turing award laureates. This is by no means a consensus stance in the expert community.
I disagree - there is pretty widespread agreement that reasoning is a weakness, even among the best models, (and note Chollet's $1M ARC prize competition to spur improvements), but the big labs all seem to think that post-training can fix it. To me this is whack-a-mole wishful thinking (reminds me of CYC - just add more rules!). At least one of your "Turing award laureates" thinks Transformers are a complete dead end as far as AGI goes.
We'll see soon enough who's right.
The ARC challenge tests spatial reasoning, something we humans are obviously quite good at, given 4 billion years of evolutionary optimization. But as I said, there is no "general reasoning", it's all domain dependent. A child does better at the spatial problems in ARC given that it has that previously mentioned evolutionary advantage, but just as we don't worship calculators as superior intelligences because they can multiply 10^9 digit numbers in milliseconds, we shouldn't draw fundamental conclusions from humans doing well at a problem that they are in many ways built to solve. If the failures of previous predictions - those that considered Chess or Go as unmistakable signals of true general reasoning - have taught us anything, it's that general reasoning simply does not exist.
The bet of current labs is synthetic data in pre-training, or slight changes of natural data that induces more generalization pressure for multi-step transformations on state in various domains. The goal is to change the data so models learn these transformations more readily and develop good heuristics for them, so not the non-continuous patching that you suggest.
But yes, the next generation of models will probably reveal much more about where we're headed.
I don't think DeepBlue or AlphaGo/etc were meant to teach us anything - they were just showcases of technological prowess by the companies involved, demonstrations of (narrow) machine intelligence.
But...
Reasoning (differentiated from simpler shallow "reactive" intelligence) is basically multi-step chained what-if prediction, and may involve a branching exploration of alternatives ("ok, so that wouldn't work, so what if I did this instead ..."), so could be framed as a tree search of sorts, not entirely disimilar to the MCTS used by DeepBlue or AlphaGo.
Of course general reasoning is a lot more general than playing a game like Chess or Go since the type of moves/choices available/applicable will vary at each step (these aren't all "game move" steps), as will the "evaluation function" that predicts what'll happen if we took that step, but "tree search" isn't a bad way to conceptualize the process, and this is true regardless of the domain(s) of knowledge over which the reasoning is operating.
Which is to say, that reasoning is in fact a generalized process, and one who' nature has some corresponding requirements (e.g. keeping track of state) for any machine to be capable of performing it ...
make the slight variation look different from the version it have memorized and it often passes. Sometimes it's as straightforward as just changing the names. humans have this failure mode too.
So you agree with me that there is no guarantee it learns any representation of the actual process that produced the training data.
Whether the machines becomes a human brain clone or something entirely alien is irrelevant. The point is, you can't cheat reality. Statistics is not magic. You can't predict text that understands without understanding.
Machine learning isn't magic - the model will learn what it can to minimize the error over the specific provided loss function, and no more. Change the loss function and you change what the model learns.
In the case of an LLM trained with a predict next word loss function, what you are asking/causing the model to learn is NOT the generative process - you are asking it to learn the surface statistics of the training set, and the model will only learn what it needs to (and is able to, per the model architecture being trained) in order to do this.
Now of course learning the surface statistics well does necessitate some level of "understanding" - are we dealing with a fairy tale or a scientific paper for example, but there is only so much the model can do. Chess is a good example, since it's easy to understand. The generative process for world class chess (whether human, or for an engine) involves way more DEPTH (cf layers) of computation than the transformer has available to model it, so the best it can do is to learn the surface statistics via much shallower pattern recognition of the state of the board. Now, given the size of these LLMs, if trained on enough games they will be able to play pretty well even using this pattern matching technique, but one doesn't need to get too far into a chess game to reach a position that has never been seen before in recorded games (e.g. watch agadmator's YouTube chess channel - he will often comment when this point has been reached), and the model therefore has no choice but to play moves that were seen in the training set in similar, but not identical positions... This is basically cargo-cult chess! It's interesting that LLMs can reach the ELO level that they do (says more about chess than about LLMs), but this same "cargo-cult" (follow surface statistics) generation process when out of training set applies to all inputs, not just chess...
You clearly do not really understand what it means to predict internet scale text with increasing accuracy. No more than that ? Fantastic
LLMs do not just learn surface statistics. So many papers have thoroughly disabused this that i'm just not going to bother. This is just straight up denial.
This havs been evidently shown in chess as well. https://arxiv.org/abs/2403.15498v2
You have no idea what you are talkin about. You've probably never even played with 3.5-turbo-instruct. That's how you can say this nonsense. You have your conclusion and keep working backwards to get a justification.
>It's interesting that LLMs can reach the ELO level that they do (says more about chess than about LLMs)
When you say this for everything LLMs can do then it just becomes a meaningless cope statement.
However, you seem to be engaged in magical thinking and believe these models are learning things beyond their architectural limits. You appear to be star struck by what these models can do, and blind to what one can deduce - and SEE - they they are unable to do.
And then you've tried to paper over being shown that with a conveniently vague and nonsensical, "says more about bla bla bla". No, you were wrong. Your model about this is wrong. It's that simple.
You start from your conclusions and work your way down from it. "pattern matching technique" ? Please. By all means, explain to all of us what this actually entails in a way we can test for it. Not just vague words.
Tracking probable board state given a sequence of moves (which don't even need to go all the way back to the start of the game!) is relatively simple to do, and doesn't require hundreds of sequential steps that are beyond the architecture of the model. It's just a matter of incrementally updating the current board state "hypothesis" per each new move (essentially: "a knight just moved to square X, so it must have moved away from some square a knight's move away from X that we believe currently contains a knight").
Ditto for estimating player ELO rating in order to predict appropriately good or bad moves. It's basically just a matter of how often the player makes the same move as other players of a given ELO rating in the training data. No need for hundreds of steps of sequential computation that are beyond the architecture of the model.
Doing an N-ply lookahead to reason about potential moves is a different story, but you want to ignore that and instead throw out a straw man "counter argument" about maintaining board state as if that somehow proves that the LLM can magically apply > N=layers of sequential reasoning to derive moves. Sorry, but this is precisely magical faith-based thinking "it can do X, so it can do Y" without any analysis of what it takes to do X and Y and why one is possible, and the other is not.
Right and the point is that you don't know what it CAN'T learn. You clearly don't quite understand this because you say stuff like this:
>Chess is a good example, since it's easy to understand. The generative process for world class chess (whether human, or for an engine) involves way more DEPTH (cf layers) of computation than the transformer has available to model it
and it's just baffling because:
1. Humans don't play chess anything like chess engines. They literally can't because the brain has iterative computation limits well below that of a computer. Most Grandmasters are only evaluating 5 to 6 moves deep on average.
2. We have a chess transformer playing world class chess (grandmaster level) - https://arxiv.org/abs/2402.04494.
You keep trying to make the point that because a Transformer architecturally has a depth limit for some trained model, a, it cannot reach human level.
But this is nonsensical for various reasons.
- Nobody is stopping you from just increasing N such that every GI problem we care about is covered.
- You have shown literally no evidence that the N even state of the art models posses today is insufficient to match human iterative compute ability.
GPT-4o instant shots arbitrary arithmetic more accurately than any human brain and that's supposedly something it's bad at. You can clearly see it can reach world class chess play.
If you have some iterative computation benchmark that shows transformers zero shotting worse than an unaided human then feel free to show me.
Why don't you write Sam Altman to tell him the good news ?
Tell him there's nothing stopping him from "increasing N" until the thing get up and walks out the door.
There are benchmarks that rightfully show the SOTA behind average human performance in other aspects of reasoning so why are you fumbling so much to demonstrate this with unaided iterative computation ? It's your biggest argument so I just thought you'd have something more substantial than "It's limited bro!"
You cannot even demonstrate this today nevermind some hypothetical scaled up model.
I think Sam will be just fine.
Well, you see, I've been a professional developer for the last 45 years, and often, gasp, think for long periods of time before coding, or even writing things down. "Look ma, no hands!".
I know this will come across as an excuse, but the thing is I assumed you were also vaguely famililar with things like software development, or other cases where human's think before acting, so I evidentially did a poor job of convincing you of this.
I also assumed (my bad!) that you would at least know some people who were semi-intelligent and wouldn't be hopelessly confused about farmers and chickens, but now I realize that was a mistake.
Really, it's all on me.
I know that "just add more rules", "make it bigger" didn't work for CYC, but maybe as you suggest "increase N" is all that's needed in the case of LLMs, because they are special. Really - that's genius! I should have thought of it myself.
I'm sure Sam is OK, but he'd still appreciate you letting him know he can forget about Q* and Strawberries and all that nonsense, and just "increase N"! So much simpler and cheaper rather than hiring thousands of developers to try to figure this out!
Maybe drop Yan LeCun a note too - tell him that the Turing Award committee are asshats, and that he is too, and that LLMs will get us all the way to AGI.
>I know this will come across as an excuse, but the thing is I assumed you were also vaguely famililar with things like software development, or other cases where human's think before acting, so I evidentially did a poor job of convincing you of this.
Really, you have the same train of thought for hours on end ?
When you finish even your supposed hours long spiel, do you just proceed to write every line of code that solves your problem just like that ? Or do you write and think some more ?
More importantly, are LLMs unable to produce the kind of code humans spend a train of thought on ?
>Maybe drop Yan LeCun a note too - tell him that the Turing Award committee are asshats, and that he is too, and that LLMs will get us all the way to AGI.
You know, the appeal to authority fallacy is shifty at the best of times but it's straight up nonsensical when said authority does not have consensus on what you're appealing to.
Like great you mentioned LeCun. And I can just as easily bring in Hinton, Norvig, Ilya. Now what ?
Write them too - spread the news of your "increase N" innovation ?
Don't scare Hinton too much though - just suggest a small increase in N.
This is just moving the goal posts from "learning the actual process" to "any process sufficiently robust"
I think it's fair to call one process that can imitate a more complex one a representation of that process. Especially when in the very next sentence he describes it as a "projection", which has the mathematical sense of a representation that loses some dimensions.
I think it's sloppy.
Using the definition from the article:
> AI safety, which refers to preventing AI from causing harm, is a hot topic amid fears that rogue AI could act against the interests of humanity or even cause human extinction.
If the purpose of a state is to ensure its continued existence, then they should be able to make >=$1 in profit.
It looks like the aim of SSI is building safe AI, not just working on safety/security of AI. Both the article and their website [1] state this.
[1] https://ssi.inc
I suspect there's a big corporate market for LLMs with very predictable behaviour in terms of what the LLM knows from its training data, vs what it knows from RAG or its context window.
If you're making a chatbot for Hertz Car Hire, you want it to answer based on Hertz policy documents, even if the training data contained policy documents for Avis and Enterprise and Budget and Thrifty car hire.
Avoiding incorrect answers and hallucinations (when appropriate) is a type of AI safety.
The PayPal mafia includes Elon Musk, Peter Thiel, etc. They now parlayed that capital into more platforms and can easily arrange investments. Heck Peter Thiel even works with governments (Palantir) and got J D Vance on Trump’s ticket, while Elon might be in his admin.
Kolomoisky got Zelensky elected in Ukraine, by launching a show about an unlikely guy who wins the presidency and named the party after the show. They call them oligarchs over there but it’s same thing.
The first guy to 1 million followers on Twitter was Ashton Kutcher. He had already starred in sitcoms and movies for years.
This idea that you can just get huge audiences and investments due to raw talent, keeps a lot of people coming to Hollywood and Silicon Valley to “make it” and living on ramen. But even just coming there proves the point — a talented rando elsewhere in the world wouldn’t even have access to the capital and big boys networks.
They all even banked at the same bank! It’s all extremely centralized: https://community.intercoin.app/t/in-defense-of-decentralize...
It’s not personally my goal to amass immense wealth and start giant companies (I would rather work minimally and live hedonically) but I am impressed by those that do so.
Overwhelmingly talent isnt sufficient. For most startups, the old boys network gets to choose who gets millions. And the next rounds a few people choose who will get billions.
If anything, I think startups from 2005-2020 were more likely to get founding easily than those giants.
But after succeeding, in some cases several times, all of the above found it easier to find investors.
Ilya has a similar track record. He contributed on several breakthroughs such as AlexNet, AlphaGo, seq2seq (that would evolve into transformers after he left Google) before even joining OpenAI.
In fact, Elon see it as one of his key contributions to OpenAI that he managed to recruit Ilya.
That Ilya is able to raise 1B now is hardly surprising. He's probably able to raise way more than that once he's hired a larger team.
Clearly there are reasons why opportunities are gated.
> This idea that you can just get huge audiences and investments due to raw talent, keeps a lot of people coming to Hollywood and Silicon Valley to “make it” and living on ramen. But even just coming there proves the point — a talented rando elsewhere in the world wouldn’t even have access to the capital and big boys networks.
All those people start somewhere though. Excluding nepotism, which is tangential point, all those people started somewhere and then grew through execution and further opening of opportunity. But it's not like they all got to where they are in one-shot. Taking your Ashton Kutcher example - yes he had a head start on twitter followers, but that's because he executed for years before on his career. Why would it make sense for some rando to rack up a million followers before he did?
Talent will earn you opportunities, but it's not going to open the highest door until you've put in the time and work.
Of course, it's not to say inequity or unequal access to opportunities doesn't exist in the world. Of course it does. But even in an ideal, perfectly equitable world, not everyone would have the same access to opportunities.
So yes, it makes perfect sense that someone would give Ilya $1B instead of some smart 18 year old, even if that 18 year old was Ilya from the past.
The analogy would be if the private could become a major overnight because they knew a guy.
What we see with ilya is not dissimilar. I don't see why it's bad that people are more hesitant to give a talented 18 year old $1B than the guy who's been at the forefront of AI innovation.
And sometimes not even necessary. Paris Hilton got a music distribution deal overnight cause of her dad’s capital!
Nepotism is a tangential point, and yes I agree that it's a bad thing. Ilya did not get this deal through nepotism, he got it through his past accomplishments, much like how a general gets promoted after many years of exemplary work.
> If you are seeking capital for a startup with a product, you have to sell the startup on realities (ie how much revenue you are making). If you are seeking capital for a startup with no product, you can sell the startup on dreams, which is much much easier but also way riskier for investors.
Since these guys don't have a product yet, they 100% sold it on big dreams combined with Ilya's track record at OpenAI.
https://youtu.be/BzAdXyPYKQo
I think it's Ilya's track record all the way since AlexNet, including his time at Google AND OpenAI.
He's not a one-trick-pony.
Do YOU want to miss out being a share holder on this new line that will bring immeasurable wealth ?? ;-)
Simultaneously too wealthy to imagine and never wealthy enough. Capitalism is quite the drug.
An alternative sequence of reasoning places less emphasis on Ilya specifically and uses Ilya as an indicator of research health. Repeat (1), (2), and (3) above, but replace "Ilya" with something like "strong and healthy fundamental research group". In this version, Ilya's departure is taken as indication that OpenAI no longer has a strong and healthy fundamental research group but that the company is "compromised" by relentless feature roadmaps for current products and their variations. That does not mean OpenAI will fail, but in this perspective it might mean that OpenAI is not well positioned to capture future research breakthroughs and the products that they will generate.
From my perspective, it's just about impossible to know how true these premises really are. And that's what makes it a bet or gamble rather than anything with any degree of assurance. To me, just as likely is the scenario where it's revealed that Ilya is highly ineffective as a generalist leader and that research without healthy tension from the business goes nowhere.
>Agreed, the car bubble is very, very real. Not that the internal combustion carriage is all hype, it's certainly impressive with useful applications, but car manufacturers are getting insane valuations with zero proof they're viable businesses.
Amazon is one of the most famous successes of the era. Bezos quit his job, launched the business out of his garage, with seed money being $10K of his own savings, and was doing $20K/week in sales just 30 days later. And I believe their only VC round before going public was an $8 investment from Kleiner Perkins. But they were a company who proved their viability early on, had a real product with rapid revenue growth before getting any VC $$.
I’d say this SSI round is more similar to Webvan, who went public with a valuation of $4.8 billion, and at that time had done a grand total of $395K in sales, with losses over $50 million.
I’m sure there are good investments out there for AI companies that are doing R&D and advancing the state of the art. However, a $1 billion investment at a $5 billion valuation, for a company with zero product or revenue, just an idea, that’s nuts IMO, and extremely similar to the type of insanity we saw during the dot com bubble. Even more so given that SSI seemingly don’t even want to be a business - direct quote from Ilya:
> This company is special in that its first product will be the safe superintelligence, and it will not do anything else up until then … It will be fully insulated from the outside pressures of having to deal with a large and complicated product and having to be stuck in a competitive rat race.
This doesn’t sound to me like someone who wants to build a business, it sounds like someone who wants to hack on AI with no oversight or proof of financial viability. Kinda wild to give him $1 billion to do that IMO.
The dotcom era was full of unprofitable startups pumping up the stock price in all sorts of ways, as they were completely dependent on continues capital flows from investors to stay afloat. Also, a lot of that capital came from retail investors in various forms.
The AI wave that is currently ongoing is for the most part funded by some of the largest and most profitable corporations on the planet.
Companies like Alphabet, Meta, Tesla/X, Amazon and (to a lesser extent) Microsoft still have founders that either control or provide a direction for these companies.
What drives this way is the fact that most of these founders have a strong belief.
We know, for instance, that Larry Page and Elon Musk had a disagreement about the future role of AGI/ASI about 15 years ago, leading to Elon Musk helping to found OpenAI to make sure that Google would not gain a monopoly.
These are strong convictions held by very powerful people that have been held for decades. Short term stock market fluctuations are not going to suddenly collapse this "bubble".
As long as these founders continue to believe that AGI is close, they will continue to push, even if the stock market stops it support to the push.
SSI may fail, of course. But Ilya has a rumor of (from people like Hinton and Elon) as being perhaps the greatest and most capable visionary in the business.
There's no liquidity until they are making money.
It means that AI startups are actually a really poor value proposition compared to traditional tech companies, because your multiplier is limited. First round $50M valuation leaves a lot more opportunity to get rich.
This kind of structure isn't as unusual for capital intensive businesses.
Minus the urgency, scientific process, well-defined goals, target dates, public ownership, accountability...
Some of you do. The rest of us are left with the consequences.
Even the president needs someone else to push a button (and in those rooms there's also more than one person). There's literally no human that can do it alone without convincing at least 1 or 2 other people, depending on who it is.
https://amp.cnn.com/cnn/2017/11/18/politics/air-force-genera...
https://www.bbc.com/news/world-us-canada-58581296.amp
The urgency was faked and less true of the Manhattan Project than it is of AGI safety. There was no nuclear weapons race; once it became clear that Germany had no chance of building atomic bombs, several scientists left the MP in protest, saying it was unnecessary and dangerous. However, the race to develop AGI is very real, and we also have no way of knowing how close anyone is to reaching it.
Likewise, the target dates were pretty meaningless. There was no race, and the atomic bombs weren't necessary to end the war with Japan either. (It can't be said with certainty one way or the other, but there's pretty strong evidence that their existence was not the decisive factor in surrender.)
Public ownership and accountability are also pretty odd things to say! Congress didn't even know about the Manhattan Project. Even Truman didn't know for a long time. Sure, it was run by employees of the government and funded by the government, but it was a secret project with far less public input than any US-based private AI companies today.
> There was no nuclear weapons race; once it became clear that Germany had no chance of building atomic bombs, several scientists left the MP in protest
You are forgetting Japan in WWII and given casualty numbers from island hopping it was going to be a absolutely huge casualty count with US troops, probably something on the order of Englands losses during WW1. Which for them sent them on a downward trajectory due to essentially an entire generation dying or being extremely traumatized. If the US did not have Nagasaki and Hiroshima we would probably not have the space program and US technical prowess post WWII, so a totally different reality than where we are today.
Well, you didn't provide any evidence. Island hopping in the Pacific theater itself took thousands of lives, imagine what a headlong strike into a revanchist country of citizens determined to fight to the last man, woman and child would have looked like. We don't know how effective a hypothetical Soviet assault would have looked like as they had attacked sparsely populated Sakhalin only. What the atom bomb succeeded was in convincing Emperor Hirohito that continuing the war would be destructively pointless.
WW1 practically destroyed the British Empire for the most part. WW2 would have done the same for the US in your hypothetical scenario, but much worse.
https://en.wikipedia.org/wiki/Operation_Downfall#:~:text=Tru....
The big problem that McArthur and others pointed out is that all the Japanese forces on the Asian mainland and left behind in the Island Hopping campaign through the Pacific were unlikely to surrender unless Japan itself was definitively defeated with the central government capitulating and aiding in the demobilization.
From their perspective the options were to either invade Japan and force a capitulation, or go back and keep fighting it out with every island citadel and throughout China, Indochina, Formosa, Korea, and Manchuria.
I'd say they were equal. We were worried about Russia getting nuclear capability once we knew Germany was out of the race. Russia was at best our frenemy. The enemy of my enemy is my friend kind of thing.
It seems pretty irresponsible for AI boosters to say it’ll happen within 5 years then.
There’s a pretty important engineering distinction between the Manhattan Project and current research towards AGI. At the time of the Manhattan Project scientists already had a pretty good idea of how to build the weapon. The fundamental research had already been done. Most of the budget was actually just spent refining uranium. Of course there were details to figure out like the specific design of the detonator, but the mechanism of a runaway chain reaction was understood. This is much more concrete than building AGI.
For AGI nobody knows how to do it in detail. There are proposals for building trillion dollar clusters but we don’t have any theoretical basis for believing we’ll get AGI afterwards. The “scaling laws” people talk about are not actual laws but just empirical observations of trends in flawed metrics.
Agreed. Do they?
Demis Hassabis said 50/50 it happens in 5 years.
Jensen Huang said 5 years.
Elon Musk said 2 years.
Leopold Aschenbrenner said 5 years.
Matt Garman said 2 years for all programming jobs.
And I think most relevant to this article, since SSI says they won’t release a product until they have superintelligence, I think the fact that VCs are giving them money means they’ve been pretty optimistic in statements about about their timelines.
Still, you’d have to be quite an idiot to wait for the third time to listen eh?
Besides, the winners get to decide what’s a war crime or not.
And when the US started mass firebombing civilian Tokyo, it’s not like they were going to be able to just ‘meh, we’re good’ on that front. Compared to that hell, being nuked was humane.
And I don’t say that lightly.
As made quite apparent by, as you note, kamikaze tactics and more.
The Bomb was a cleaner, sharper, and faster Axe than invading the main island.
That it also sent a message to the rest of the world was a bonus. But do you think they would have not used it, if for example the USSR wasn’t waiting?
Of course not, they’d still have nuked the hell out of the Japanese.
What does AGI do? AGI is up against a philosophical barrier, not a technical one. We'll continue improving AI's ability to automate and assist human decisions, but how does it become something more? Something more "general"?
With transformers, demonstrated first by LLMs, I think we've shown that the narrow-general divide as a strict binary is the wrong way to think about AI. Instead, LLMs are obviously more general than any previous AI system, in that they can do math or play chess or write a poem, all using the same system. They aren't as good as our existing superhuman computer systems at these tasks (aside from language processing, which they are SOTA at), not even as good at humans, but they're obviously much better than chance. With training to use tools (like calculators and chess engines) you can easily make an AI system with an LLM component that's superhuman in those fields, but there are still things that LLMs cannot do as well as humans, even when using tools, so they are not fully general. One example is making tools for themselves to use - they can do a lot of parts of that work, but I haven't seen an example yet of an LLM actually making a tool for itself that it can then use to solve a problem it otherwise couldn't. This is a subproblem of the larger "LLMs don't have long term memory and long term planning abilities" problem - you can ask an LLM to use python to make a little tool for itself to do one specific task, but it's not yet capable of adding that tool to its general toolset to enhance its general capabilities going forward. It can't write a memoir, or a book that people want to read, because they suck at planning or refining from drafts, and they have limited creativity because they're typically a blank slate in terms of explicit memory before they're asked to write - they have a gargantuan of implicitly remembered things from training, which is where what creativity they do have comes from, but they don't yet have a way to accrue and benefit from experience.
A thought exercise I think is helpful for understanding what the "AGI" benchmark should mean is: can this AI system be a drop-in substitute for a remote worker? As in, any labour that can be accomplished by a remote worker can be performed by it, including learning on the job to do different or new tasks, and including "designing and building AI systems". Such a system would be extremely economically valuable, and I think it should meet the bar of "AGI".
But they can't, they still fail at arithmetic and still fail at counting syllables.
I think that LLMs are really impressive but they are the perfect example of a narrow intelligence.
I think they don't blur the lines between narrow and general, they just show a different dimension of narrowness.
You are incorrect. These services are free, you can go and try it out for yourself. LLMs are perfectly capable of simple arithmetic, better than many humans and worse than some. They can also play chess and write poetry, and I made zero claims at "counting syllables", but it seems perfectly capable of doing that too. See for yourself, this was my first attempt, no cherry picking: https://chatgpt.com/share/ea1ee11e-9926-4139-89f9-6496e3bdee...
I asked it a multiplication question so it used a calculator to correctly complete the task, I asked it to play chess and it did well, I asked it to write me a poem about it and it did that well too. It did everything I said it could, which is significantly more than a narrow AI system like a calculator, a chess engine, or an image recognition algorithm could do. The point is it can do reasonably at a broad range of tasks, even if it isn't superhuman (or even average human) at any given one of them.
>I think that LLMs are really impressive but they are the perfect example of a narrow intelligence.
This doesn't make any sense at all. You think an AI artifact that can write poetry, code, play chess, control a robot, recommend a clutch to go with your dress, compute sums etc is "the perfect example of a narrow intelligence." while a chess engine like Stockfish or an average calculator exists? There are AI models that specifically and only recognise faces, but the LLM multitool is "the perfect example of a narrow intelligence."? Come on.
>I think they don't blur the lines between narrow and general, they just show a different dimension of narrowness.
You haven't provided an example of what "dimension of narrowness" LLMs show. I don't think you can reasonably describe an LLM as narrow without redefining the word - just because something is not fully general doesn't mean that it's narrow.
For poetry: counting syllables is a significant part of most poetry forms, so if you can't count syllables, you can't do poetry.
Let's say you want a 5-7-5 haiku, this is ChatGPT
this is not a 5-7-5 haiku.LLMs are not general, but they show that a specific specialization ("guess next token") can solve a lot more problem that we thought it could.
>[AI system]s are not general, but they show that a specific specialization ("[process sequential computational operations]") can solve a lot more problem that we thought it could.
Or if you really want:
>Humans are not general, but they show that a specific specialization ("neuron fires when enough connected neurons fire into it") can solve a lot more problem that we thought it could.
This is just sophistry - the method by which some entity is achieving things doesn't matter, what matters is whether or not it achieves them. If it can achieve multiple tasks across multiple domains it's more general than a single-domain model.
Or he will simply shift goalposts, and call some LLM superintelligent.
[0]: https://en.wikipedia.org/wiki/Galactic_algorithm
Edit: actually I'm not sure if AIXItl is technically galactic or just terribly inefficient, but there's been trouble making it faster and more compact.
In any case anyone who is completely sure that we can/can’t achieve AGI is delusional.
The fact is many things we’ve tried to develop for decades still don’t exist. Nothing is guaranteed
Basically, unless you can show humans calculating a non-Turing computable function, the notion that intelligence requires a biological system is an absolutely extraordinary claim.
If you were to argue about conscience or subjective experience or something equally woolly, you might have a stronger point, and this does not at all suggest that current-architecture LLMs will necessarily achieve it.
1. There is a chemical-level nature to intelligence which prevents other elements like silicon from being used as a substrate for intelligence
2. There is a non material aspect to intelligence that cannot be replicated except by humans
To my knowledge, there is no scientific evidence that either are true and there is already a large body of evidence that implies that intelligence happens at a higher level of abstraction than the individual chemical reactions of synapses, ie. the neural network, which does not rely on the existence of any specific chemicals in the system except in as much as they perform certain functions that seemingly could be performed by other materials. If anything, this is more like speculating that there is a way to create energy from sunlight using plants as an existence proof of the possibility of doing so. More specifically, this is a bet that an existing physical phenomenon can be replicated using a different substrate.
What evidence can you provide to back up the statement of this "significant possibility"? Human brains use neural networks...
1. Humans have general intelligence. 2. Human brains use biological neurons. 3. Human biological neurons give rise to human general intelligence. 4. Artificial neural networks (ANNs) are similar to human brains. 5. Therefore an ANN could give rise to artificial general intelligence.
Many people are objecting to #4 here. However in writing this out, I think #3 is suspect as well: many animals who do not have general intelligence have biologically identical neurons, and although they have clear structural differences with humans, we don’t know how that leads to general intelligence.
We could also criticize #1 as well, since human brains are pretty bad at certain things like memorization or calculation. Therefore if we built an ANN with only human capabilities it should also have those weaknesses.
1. Either you are correct and the neural networks humans have are exactly the same or very similar to the programs in the LLMs. Then it will be relatively easy to verify this - just scale one LLN to the human brain neuron count and supposedly it will acquire consciousness and start rapidly learning and creating on its own without prompts.
2. Or what we call neural networks in the computer programs is radically different and or insufficient to create AI.
I'm leaning to the second option, just from the very high level and rudimentary reading about current projects. Can be wrong of course. But I have yet to see any paper that refutes option 2, so it means that it is still possible.
If you wanted to reduce it down, I would say there are two possibilities:
1. Our understanding of Neurel Nets is currently sufficient to recreate intelligence, consciousness, or what have you
2. We’re lacking some understanding critical to intelligence/conciousness.
Given that with a mediocre math education and a week you could pretty completely understand all of the math that goes into these neurel nets, I really hope there’s some understand we don’t yet have
MLPs and transformers are ultimately theoretically equivalent. That means there is an MLP that represent the any function a given transformer can. However, that MLP is hard to identify and train.
Also the transformer contains MLPs as well...
(I did AI and Psychology at degree level, I understand there are definitely also big differences too, like hormones and biological neurones being very async)
Transformers, while not exactly functions, don't have a feedback mechanism similar to e.g. the cortical algorithm or any other neuronal structure I'm aware of. In general, the ML field is less concerned with replicating neural mechanisms than following the objective gradient.
Numenta has attempted to implement a system to this effect (see the wiki page https://en.wikipedia.org/wiki/Hierarchical_temporal_memory) for quite some time with not particularly much success.
Personally I think the kinds of minds we create in silico will end up being very different, because the advantages and disadvantages of the medium are just very different; for example, having a much stronger central processor and much weaker distributed memory, along with specialized precise circuits in addition to probabilistic ones.
But there is no reason the company can't come up with a different paradigm.
But I suppose you could say we don't know 100% since we don't fully understand how the brain learns.
https://en.m.wikipedia.org/wiki/AI_winter
Modern ANN architectures are not actually capable of long-term learning in the same way animals are, even stodgy old dogs that don't learn new tricks. ANNs are not a plausible model for the brain, even if they emulate certain parts of the brain (the cerebellum, but not the cortex)
I will add that transformers are not capable of recursion, so it's impossible for them to realistically emulate a pigeon's brain. (you would need millions of layers that "unlink chains of thought" purely by exhaustion)
even if we bought this negative result as somehow “proving impossibility”, i’m not convinced plasticity is necessary for intelligence
huge respect for richard sutton though
More specifically: it is highly implausible that an AI system could learn to improve itself beyond human capability if it does not have long-term plasticity: how would it be able to reflect upon and extend its discoveries if it's not able to learn new things during its operation?
(That said, I agree plasticity is key to the most powerful systems. A human race with anterograde amnesia would have long ago gone extinct.)
If I'm a human tasked with editing video (which is the field my startup[0] is in) and a completely new video format comes in, I need the long term plasticity to learn how to use it so I can perform my work.
If a sufficiently intelligent version of our AI model is tasked with editing these videos, and a completely new video format comes in, it does not need to learn to handle it. Not if this model is smart enough to iterate a new model that can handle it.
The new skills and knowledge do not need to be encoded in "the self" when you are a bunch of bytes that can build your successor out of more bytes.
Or, in popular culture terms, the last 30 seconds of this Age of Ultron clip[1].
[0]: https://www.onetake.ai
[1]: https://www.youtube.com/watch?v=qA5wYcybkCM&t=25
What do you think training (and fine-tuning) does?
No LLM currently adapts to the tasks its given with an iteration cycle shorter than on the order of months (assuming your conversations serve as future training data; otherwise not at all).
No current LLM can digest its "experiences", form hypotheses (at least outside of being queried), run thought experiments, then actual experiments, and then update based on the outcome.
Not because it's fundamentally impossible (it might or might not be), but because we practically haven't built anything even remotely approaching that type of architecture.
They don't, actually.
A very cynical take is that this is an extreme version of 'we plan to spend all money on growth and figure out monetization later' model that many social media companies with a burn rate of billions of $$, but no business model, have used.
He is saying he will try to build something head and shoulders above anything else, and he got a billion dollars to do it with no expectation of revenue until his product is ready. The likelihood that he fails is very high, but his backers are willing to bet on that.
1b was a non profit donation, so there wasn't an expectation of returns on that one.
i read the article but I am not sure how they know when this condition will be true.
Is this obvious to ppl reading this article? is it emperor has no clothes type situation ?
Are these ppl merely gullible or coconspirators in the scam ?
If you check the 2024 YC batch, you'll notice pretty much every single one of them mentions AI in some form or another. I guarantee you the large majority of them are just looking to be bought out by some megacorp, because it's free money right now.
There's plenty of players going for the same goal. R&D is wildly expensive. No guarantee they'll reach the goal, first or even at all.
Moreover, the majority of the capital likely goes into GPU hardware and/or opex, which VCs have currently arbitraged themselves [3], so to some extent this is VCs literally paying themselves to pay off their own hardware bet.
While hints of the ambition of the Manhattan project might be there, the economics really are not.
[1a] https://www.getpin.xyz/post/clubhouse-lessons-for-investors [1b] https://www.theverge.com/2023/4/27/23701144/clubhouse-layoff... [3] https://observer.com/2024/07/andreessen-horowitz-stocking-ai...
No. The Manhattan Project started after we understood the basic mechanism of runaway fission reactions. The funding was mostly spent purifying uranium.
AGI would be similar if we understood the mechanism of creating general intelligence and just needed to scale it up. But there are still fundamental questions we still aren’t close to understanding for AGI.
A more apt comparison today is probably something like fusion reactors although progress has been slow there too. We know how fusion works in theory. We have done it before (thermonuclear weapons). There are sub-problems we need to solve, but people are working on them. For AGI we don’t even know what the sub-problems are yet.
(I think it's branding, yes. A kind of "we don't care about aesthetics, we care about superintelligence" message)
What’s mildly annoying to me is their domain only returns an A record.
I mean, I'd like at least a brief blurb about their entire premise of safety. Maybe a definition or indication of a public consultation or... something.. otherwise the insinuation is that these three dudes are gonna sit around defining it on instinct, as if it's not a ludicrously hard human problem.
A couple years??
we all know that openai did it
Nobody even knew what OpenAI was up to when they were gathering training data - they got away with a lot. Now there is precedent and people are paying more attention. Data that was previously free/open now has a clause that it can't be used for AI training. OpenAI didn't have to deal with any of that.
Also OpenAI used cheap labor in Africa to tag training data which was also controversial. If someone did it now it would they'd be the ones to pay. OpenAI can always say "we stopped" like Nike said with sweat shops.
A lot has changed.
Companies are willing to pay a lot for clean training data, and my bet is there will be a growing pile of training sets for sale on a non-exclusive basis as well.
A lot of this data - what I've seen anyway, is far cleaner than anything you'll find on the open web, with significant data on human preferences, validation, cited sources, and in the case of e.g. coding with verification that the code runs and works correctly.
Very interesting, thanks for sharing that detail. As someone who has tinkered with tokenizing/training I quickly found out this must be the case. Some people on HN don't know this. I've argued here with otherwise smart people who think there is no data preprocessing for LLMs, that they don't need it because "vectors", failing to realize the semantic depth and quality of embeddings depends on the quality of training data.
At a time when things are advancing at breakneck speed. Where is the goalpost going to be in 2 years time?
Why Tel Aviv in Israel ?
Thanks.
If we say that half of innovations came from Alphabet/Google, then most of them (transformers, LLMs, tensorflow) came from Google Research and not Deep Mind.
Source: Wikipedia.
Israel is a leading AI and software development hub in the world.
Yep, and if any place will produce the safest AI ever, its got to be there.
Israeli military operations continues to this day with over 41,000 civilians killed.
Related to this, DAO's (decentralized autonomous organizations which do not have human shareholders) are intrinsically dangerous, because they can benefit their fiduciary duty even if it involves causing all humans to die. E.g., if the machine faction in The Matrix were to exist within the framework of US laws, it would probably be a DAO.
https://www.businessroundtable.org/business-roundtable-redef...
The idea behind "corporations should only focus on returns to shareholders" is that if you let them do anything else, CEOs will just set whatever targets they want, and it makes it harder to judge if they're doing the right thing or if they're even good at it. It's basically reducing corporate power in that sense.
> E.g., if the machine faction in The Matrix were to exist within the framework of US laws, it would probably be a DAO.
That'd have to be a corporation with a human lawyer as the owner or something. No such legal concept as a DAO that I'm aware of.
enterprises, corps, banks, governments will want to buy "safe" AI, to push liability for mistakes on someone who proclaimed them "safe".
We can’t build critical software without huge security holes and bugs (see crowdstrike) but we think we will be able to contain something smarter than us? It would only take one vulnerability.
https://en.wikipedia.org/wiki/Existential_risk_from_artifici...
https://en.wikipedia.org/wiki/Ex_Machina_(film)
This is why I'm extremely opposed to the idea of "AI girlfriend" apps - it creates a cultural concept that being attracted to a computer is normal, rather than what it is: something pathetic and humiliating which is exactly like buying an inflatable sex doll ... something only for the most embarrassing dregs of society ... men who are too creepy and pervy to ever attract a living, human woman.
"Super" intelligence typically refers to being better than humans in achieving goals, not to being better than humans in knowing good from evil.
Idk about this claim.
I think if you take the multi-verse view wrt quantum mechanics + a veil of ignorance (you don't know which entity your conciousness will be), you pretty quickly get morality.
ie: don't build the Torment Nexus because you don't know whether you'll end up experincing the Torment Nexus.
This was recently discussed (albeit in layperson’s language, avoiding philosophical topics and only focusing on the clear and present danger) in this article in RealClearDefense:
The Danger of AI in War: It Doesn’t Care About Self-Preservation https://www.realcleardefense.com/articles/2024/09/02/the_dan... (RealClearDefense)
.
However, just adding a self-preservation instinct will cause a skynet situation where the AI pre-emptively kills anyone who contemplates turning it off, including its commanding officers:
Statement by Air Force Col. Tucker Hamilton https://www.twz.com/artificial-intelligence-enabled-drone-we... (The War Zone)
.
To survive AGI, we have to navigate three hurdles, in this order:
Ilya proved himself as a leader, scientist, and engineer over the past decade with OpenAI for creating break-through after break-through that no one else had.
He’s raised enough to compete at the level of Grok, Claude, et al.
He’s offering investors a pure play AGI investment, possibly one of the only organizations available to do so.
Who else would you give $1B to pursue that?
That’s how investors think. There are macro trends, ambitious possibilities on the through line, and the rare people who might actually deliver.
A $5B valuation is standard dilation, no crazy ZIRP style round here.
If you haven’t seen investing at this scale in person it’s hard to appreciate that capital allocation just happens with a certain number of zeros behind it & some people specialize in making the 9 zero decisions.
Yes, it’s predicated on his company being worth more than $500B at some point 10 years down the line.
If they build AGI, that is a very cheap valuation.
Think how ubiquitous Siri, Alexa, chatGPT are and how terrible/not useful/wrong they’ve been.
There’s not a significant amount of demand or distribution risk here. Building the infrastructure to use smarter AI is the tech world’s obsession globally.
If AGI works, in any capacity or at any level, it will have a lot of big customers.
my dude, I'd rather have a washing machine than chatgpt.
AGI assumes exponential, preferably infinite and continuous improvement, something unseen before in business or nature.
Neither siri nor Alexa were sold as AGI and neither alone come close to a $1B product. gpt and other LLMs has quickly become a commodity, with AI companies racing to the bottom for inference costs.
I don’t really see the plan, product wise.
Moreover you say: > Ilya proved himself as a leader, scientist, and engineer over the past decade with OpenAI for creating break-through after break-through that no one else had.
Which is absolutely true, but that doesn’t imply more breakthroughs are just around the corner, nor does the current technology suggest AGI is coming.
VCs are willing to take a $1B bet on exponential growth with a 500B upside.
Us regular folk see that and are dumbfounded because AI is obviously not going to improve exponentially forever (literally nothing in the observed universe does) and you can already see the logarithmic improvement curve. That’s where the dismissive attitude comes from.
you say you don't see it. fine. these investors do - thats why they are investing and you are not.
They also have the warchest to afford a $1B gamble.
If the math worked out for me too, I’d probably invest even if I didn’t personally believe in it.
Also investors aren’t super geniuses, they’re just people.
I mean look at SoftBank and Adam Neuman… investors can get swept up in hype and swindled too.
Obviously, there are plenty of investors who don't fall into this situation. But lets not pretend that just because someone has a lot of money or invests a lot of money that it means they know what they are doing.
There are many things on earth that don't exist anywhere else in the universe (as far as we know). Life is one of them. Just think how unfathomably complex human brains are compared to what's out there in space.
Just because something doesn't exist anywhere in the universe doesn't mean that humans can't create it (or humans can't create a machine that creates something that doesn't exist anywhere else) even if it might seem unimaginably complex.
There are plenty of complex phenomena in space, but I don’t need to go that far.
Some other animal brains act like ours, at least as far as we can observe.
There is nothing anywhere that grows exponentially forever.
My intent is to be helpful. I’m unsure of how much additional context might be useful to you.
Investor math & mechanics is straight-forward: institutional funds & family offices want to get allocations in investors like a16z because they get to invest in deals that they could not otherwise invest in. The top VCs specialize in getting into deals that most investors will never get the opportunity to put money into. This is one of them.
For their Internal Rate of Return (IRR) to work out at least one investment needs to return 100x or more on the valuation. VCs today focus on placing bets where that calculation can happen. Most investors aren’t that confident in their ability to predict that, so they invest alongside lead investors who are. a16z is famous for that.
There are multiple companies worth $1T+ now, so this isn’t a fantasy investment. it’s a bet.
The bet doesn’t need to be that AGI continues to grow in power infinitely, it just needs to create a valuable company in roughly a ten year time horizon.
Many of the major tech companies today are worth more money than anyone predicted, including the founders (Amazon, Microsoft, Apple, Salesforce, etc.). An outlier win in tech can have incredible upside.
LLMs are far from commoditized yet, but the growth of the cloud proves you can make a fortune on the commoditization of tech. Commoditization is another way of saying “everyone uses this as a cost of doing business now.” Pretty great spot to land on.
My personal view is that AGI will deliver a post-product world, Eric Schmidt recently stated the same. Products are digital destinations humans need to go to in order to use a tool to create a result. With AGI you can get a “product” on the fly & AI has potentially very significant advantages in interacting with humans in new ways within existing products & systems, no new product required. MS Copilot is an early example.
It’s completely fine to be dismissive of new tech, it’s common even. What bring me you here?
I’m here on HN because I love learning from people who are curious about what is possible & are exploring it through taking action. Over a couple decades of tech trends it’s clear that tech evolves in surprising ways, most predictions eventually prove correct (though the degree of impact is highly variable), and very few people can imagine the correct mental model of what that new reality will be like.
I agree with Zuck:
The best way to predict the future is to build it.
Sure, but it doesn't have to continue forever to be wildly profitable. If it can keep the exponential growth running for another couple of rounds, that's enough to make everyone involved rich. No-one knows quite where the limit is, so it can reasonably be worth a gamble.
That’s fine and good for investors.
I couldn’t care less about the business side of technology.
Im an engineer and a technophile and as an engineer and a technophile it sours me to hear someone dangle sci-fi level AGI as a pitch to investors when we’re clearly not there right now and ,in my opinion, this current wave of of basically brute force statistics based predictive models will not be the technique that gets us there.
It makes the cynic in me, and many others probably, cringe.
This is wrong. The models may end up cheaply available or even free. The business cost will be in hosting and integration.
When Ilya was in Google, the breakthroughs came from Google.
When Ilya was in OpenAI, the breakthroughs came from OpenAI.
....
the whole LLM race seems deaccelerate, and all the hard problems about LLMs seems not do have had that much progress the last couple of years (?)
In my naaive view I think a guy like David Silver the creator/co-lead of Alpha-Zero deserves more praise, atleast as a leader/scientist. He even have lectures about Deep RL after doing AlphaGo: https://www.davidsilver.uk/teaching/
He has no LinkedIn and came straight from the game-dev industry before learning about RL.
I would put my money on him.
Even assuming the public breakthroughs are the only ones that happened, the fact that openai was able to make an llm pipeline from data to training to production at their scale before anyone else is a feat of research and engineering (and loads of cash)
Any guesses?
In other words, there may be a need to retains some sorts of symmetries or constraints from generation to generation that others understand less well than him (or so he thinks).
To me this sounds like maybe they won't be doing transformers. But perhaps they just mean "we will have safety in mind as we scale, unlike everyone else."