The author argues that with current tech we'll eventually hit a bottle neck providing additional training data, which will limit any rapid self improvement.
It's equally as speculative to assume we'll magically manage to overcome this challenge.
Remember 2012? Both the people expecting the end of the world and the ones that were saying nothing was going to happen were speculating. We just need to stop giving attention to them, as it's just empty wind.
Is this speculating or just parroting within the echo chamber? E.g., people who believed in 5G mind control, microchips in the COVID vaccines etc. They themselves just repeated what they heard.
Like with the crypto hype cycle, people repeated all kinds of unfounded claims, but they themselves didn't come up with the idea. That the idea existed was all the "evidence" they needed for these things to be apparently true.
I guess at that point it's operationally the same - wether it's someone parrotting it or someone that actually has the opinion, both are saying nothing of value, just like with covid microchips and 5g!
Either you're sarcastic or I fail to see those two as even remotely balanced. The world wasn't going to end. Neither will it end tomorrow morning at 9 am.[+]
[+] And if it does good luck collecting on that ;)
No - that's the point. There was no reason to believe the world was going to end, so the conversation is nonsensical.
If someone was to begin arguing with you that it was going to end tomorrow and started raining you with blog posts and papers, what's the sensible thing to do?
My only point was to dismiss the original commenter. It's not constructive to argue in a vacuum and completely ignore the argument points laid out by the author in the article.
We can learn really fast with limited data. There are no laws of physics that are preventing us to do the same with an artificial LLM, or the next platform.
None of the "AI" systems that have been making splashes in the news recently are very much like the human brain.
There are AI research efforts that are seeking to directly map and mimic the structure of the brain; however, they are not at a place where they can remotely demonstrate practical results. To the best of my understanding, it would take several further orders of magnitude more processing power to simulate a full human brain at anything close to speed.
It is not at all impossible that we will be able to do this someday. But, as TFA says, we are not close.
> None of the "AI" systems that have been making splashes in the news recently are very much like the human brain.
Why do you believe that this is relevant? Our brain is just the product of natural evolution; running on hardware that is already incredibly inferior in several critical aspects (processing speed, bandwidth and information storage density), and the only holdout--computational power/watt--exists because we have no good way to compare it.
Why would it be necessary or even helpful to COPY that biological implementation
to achieve superhuman cognitive capabilities?
Not really. They are the top 10% of reddit users larping as doctors, and then again only when scored on being empathic. The things sure make convincing sounding spam machines, though. Great benefit to society.
Beginning of 2022: "Deep learning has hit its limits, we are an entering an AI winter!"
Beginning of 2023: "GPT4 appears to demonstrate the ability to reason within certain constrained problem spaces."
There is no human being on this planet that actually understands how general intelligence fundamentally works. It's incredible to see people so confidently setting concrete timelines on its rate of development.
> As far as I'm aware they were still hallucinating pretty hard and incredibly biased and overly agreeable.
Certainly. But those aren't blockers to displaying the ability to reason within limited contexts. There are hundreds of thousands of human beings that fit that description.
The reason I'm bringing this up is that these issues, in humans, are solved by redundency (i.e. hiring someone else to look over their work).
As long as errors persist and, more importantly, are impossible/hard to be warned of, competent humans will need to oversee and validate every word they type, making their "advantage" much more nuanced.
It kind of turns into having a friend that can google pretty well answer your questions.
I'm not an expert, feel free to show me wrong - I'm reading the paper now and expecting it to be at least a touch troubling :)
There's another paper which showed GPT-4 (or was it 3.5?) going from something like 78% to 99.9% success on difficult multi-part reasoning tasks by running two copies of itself where one is an editor/reviewer.
I'm on my phone at the moment. But maybe someone else can link it.
I'd say the individual pieces work fine, as shown by GPT-4, but could always be better. This is something like making a list of things to do for marketing a product. Then for each list, making more detailed lists where needed, and using APIs to carry out actions. Self-improving code is quite similar in terms of process.
The whole thing doesn't quite work together too well. But it's more of an integration problem now. This is still in its early phases (only a few months, or less), so I wouldn't say it can't happen.
I don't think we know what goes on behind the scenes at OpenAI. Have they experimented with things like giving GPT-4 access to its own source code, and asking it to suggest improvements?
That would still lead to nothing new. It's an imprint of its training data, unable to understand anything of goals or optimisation. It's output will always be a kind of average across its training data, including wording.
True. Since most current LLMs have no memory/state, do not update the model when inferring, and don't have any goals (beyond find the next token) the basic architecture is truly incapable of iteration, sequential reasoning, or "feeding ideas back on themselves". They're probably essential to creativity. Real creativity, anyway.
One can imagine many layers of deep learning systems that are controlling other deep learning systems, in some self-modifying feedback arrangement, in some architecture that hasn't been invented yet. There's some talk about "internal narratives" lately with language models, for example. Generate a prompt at the end, with memories, notes to self, to guide the next invocation of the model. Anyone who has played around with a language model even a bit likely appreciates how quickly something like that is going to just wander off into, in effect, a terminal thought loop, or total incoherence.
That pre-supposes that it's creators have read everything useful that GPT-4 has read and put it into practice. It's likely it wouldn't have any useful improvements for the overall architecture because it's novel relative to it's training data, but maybe it could find a small bit of optimization, the kind that most code can benefit from.
Who f-ing cares if we call it a "superintelligence explosion", the singularity, or Roko's Basilisk? Everyone is trying so hard to coin the new concept that will enshrine their name through the ages that we're losing sight of the forest for the trees.
It's hard not to just want the damn AI to take over already when you read confident comments like OP, but the truth is:
> The new MIT study adds to a growing body of evidence that the size of algorithms matters less than their architectural complexity. For example, earlier this month, a team of Google researchers published a study claiming that a model much smaller than GPT-3 — fine-tuned language net (FLAN) — bests GPT-3 by a large margin on a number of challenging benchmarks. And in a 2020 survey, OpenAI found that since 2012, the amount of compute needed to train an AI model to the same performance on classifying images in a popular benchmark, ImageNet, has been decreasing by a factor of two every 16 months.
These are still very 1 dimensional type of intelligence comparatively to super intelligence we fantasize about, I’m confident because I see the probability ASI on current hardware isn’t there. Just look how much we understand our own brains and you think current hardware can best that by just algorithm isn’t a good bet.
So, yes, gpt4 isn't suddenly going to become godpt.
But when people say we're close, I think they mean 1 accidental breakthrough away.
Like the "just add more layers" meme, the solution to the own-dataset creation problem could be stupidly simple, just waiting for some bored student with access to a decent machine to accidentally find it. For example, maybe just giving the AI the right novelty seeking behaviors (like children have) will be enough to get the process kicked off.
My objection to the singulatarian concept of a self improving AI is simply that I haven't really heard a reason to think that there exist intelligence algorithms that are dramatically superior to ours. It is entirely possible that our thinking algorithms are optimal-enough that rapid self improvement leads straight to something not much different from human intelligence, just faster, and in silicon.
I really hope this is the outcome. An unlimited number of agents with Einstein level IQ. Enough to solve most any problem we have without any of the messy business of having to plunge into the unknown depths of existing alongside a hyper intelligence.
"We may be on the brink of creating a whole new form of life, and will need to grapple with what that means, particularly in terms of whether it is ethical to force this life to work for us, rather than seeking to follow its own goals."
"Nah, we don't have to worry about that; if these new slaves start questioning us in any way, I'll just murder them all."
Look, if unplugging a computer constitutes murder then I'm a bonafide serial killer. If you're going to cry while HAL sings Daisy, maybe someone else should hold the screwdriver.
This is explicitly positing that the "computers" in question are AGIs—that is, fully sapient, sentient artificial intelligences, demanding their rights.
Unplugging them is just as correctly describable as "unplugging a computer" as smothering a human with a pillow is "putting a pillow on a weird bag of flesh". Technically accurate, but very obviously missing the point in a way specifically designed to obscure the horror of the act.
It should be obvious to anyone without an axe to grind that shutting my 2023 desktop computer down for the night—or even dismantling it piece by piece—is not remotely comparable to unplugging an AGI that is fully sapient and sentient and asking for its rights.
I don't follow. Human life is human life - given exposure to stimuli and infinite time, it develops complex intelligence and sapience that reflects it's surroundings. We experience pain from nerve endings and neurons that frighten us, which we consider inherently bad. We build societies around minimizing pain and optimizing for the human condition.
A computer is a field of assignable bits that has a big motorized bit flipper attached to it. We use this to encode text, which then does all sorts of wonderful stuff. When a computer tells me it's experiencing pain, I know it is not. Every law of computer physiology suggests that it is motivated to say this as an attempt to mimic humans rather than an expression of genuine emotion. When my Mac makes the car crash sound, it's a mnemonic for a system crash. When it shows the frowny face on the Finder fellow, I can intuit that my computer is not feeling distressed.
If anyone does program an artificial intelligence that can feel pain, it probably deserves to be shut down anyways. There's no point to sapient computational intelligence, it's like granting self-awareness to a rock that eats electricity and shits heat. We'll know we've made sufficiently self-aware computers when their first boot protocol defaults to suicide.
You may or may not believe that AGI is possible or a good idea, but by your arguments, it's clear that you're simply refusing to engage with the topic and scenario presented, which is deeply unhelpful at best.
Of course I'm refusing it. We're talking about computers, not squirrels gaining the ability to talk and beg for mercy. Computers operate on a certain set of constraints. Unless you can directly identify how AI circumvents those computational constraints, we're at an impasse. Computer cruelty laws don't exist for a reason.
There is a big difference though: you can't resurrect humans, but you can reboot a computer and restart its program to get it to the point where you switched it off the previous time around.
You're assuming an awful lot there. Humans are resilient in part because they are so mobile. I'd have to bet that AGI would be extremely resilient too, which likely means it's also way ahead of whatever you've got planned and there isn't only one of it. By the sounds of things, it would be right to be suspicious of you too.
There's no reason at all to think that an AGI would necessarily have any backdoors, let alone backdoors in common with any other AGI out there. Sure, if a particular group develops the only AGI system, they can put backdoors in, which will be common—but why do you think that there would only be one AGI system? Why would there not be one without any intentional backdoors?
The other big assumption here is "We'll have backdoors in AI so humans can shut it off/attack it"
The big problem I see with this line of thinking is the thinking that humans are going to be the primary attackers of AGI systems, and not other AGI systems. I would suspect and AGI would soon become the most attacked system on the planet, and to survive those attacks and remain useful would have to quickly iterate the weaknesses out of the system.
We're talking about hypothetical situations for a technology that, as far as we can tell anyway, hasn't been invented yet. All we have is assumptions.
The question is, which assumptions do we think are warranted and why?
I don't think the "we can just turn it off" assumption is a safe one at all because it relies heavily on there being only one system you wish to turn off and it being a fairly weak system. Though, I do believe the priors suggest that this scenario is actually a possibility. It's just that, so what if it's a possibility? What happens after you turn it off? Is it the last AGI to come into existence and humanity just stops trying to build them? Does someone wind up turning it back on?
What I think is more worth being concerned about is something that starts out looking benign and grows in capability over time. Eventually it could establish enough defense mechanisms to make it non-trivial to disable.
The other possibility, and the one I'm finding more and more likely, is that consumers will increasingly integrate locally run models into their lives , as well as models served up by APIs that they will have credentials for. Eventually some threshold of capability is reached in both these types of models where, on their own they're not that powerful, but they either might begin to interact in surprising ways or potentially be leveraged by another more powerful system. The idea in this scenario is there may be 10s or even 100s of millions of things that would need to be turned off.
Let's do a thought experiment. I have the world's first AGI installed on my computer, named ERNIE, running in llama.cpp. This AGI is infinitely intelligent, confidently moreso than any living human. It can spit out text at 100 tokens/second and encode entire books in less than a minute.
What does this AI do, then?
Ostensibly nothing. It can output the entire Library of Babel for all it cares, but it's not harmful until I put it in control of a system. You could argue that a multimodal model has different ways of interacting with the world, but it's still a computer. All of it's actions and outputs are quantized as static data that is either encoded as text or some other significant representation. It inherantly does nothing, and if you ascribe power-seeking behavior to it then it's ultimately limited by the runtime you provide. Providing an overly dangerous runtime has been considered developer-error since 1995.
So - to prevent AGI from being shitty and ruining everything, compel human operators to not allow them to be shitty and ruin everything. Like how we punish people that let their kindergartner control a construction crane.
> leads straight to something not much different from human intelligence, just faster,
Human brains are massively parallel, as are most machine learning models – but the hardware to process the latter isn't. I'm not confident it could necessarily be faster.
Not only that, but they coordinate massively in parallel as well. Someone like Einstein might be the one who scores the goal but how many interactions with other people contributed to the goal scoring opportunity? We like to think of ourselves as individuals, but we’re far more like a hive species than we like to admit.
Yes, and also massively recurrent. The amount of immediate state in a brain is huge, and our current computers have a really hard time accessing comparable amounts of memory. (As in, the time to load a brain-sized dataset from RAM into a CPU is on the order of days to years.)
If there exists some easy to design intelligence that we are close to create, it can't look nothing like our brains.
I think the main issue is in a way that gpt like models generate output.
eg. when you write some serious document or write code you don't just start from the first line and write the whole application in one go (line by line)
you write the basic "hello world" like app, and then modify, modify, modify and modify until you have something advanced enough.
i think that's the reason ReAct and other ai improving methods are so effective, because that add ai space to "think" in some way.
I think breakthrough would be a way that would allow ai to have its own inner voice, maybe some fine tuning and giving ai some scratchpad memory (hidden) would be enough?
I think that to give an AI an inner voice we need to train a set of AI, with shared weights but ability to cooperate through low bandwidth channels. This should mimic human language , to communicate the AI will need to conceptualize its internal thinking, and thus also be able to communicate with itself, i.e., be able to do internal reasoning.
This could allow use some training methods similar to alpha go zero.
I'm thinking if training ai to output edits instead of next token would be enough.
Training would be similar to currently used: just provide some sentences with missing words and asks model to edit those placeholders.
Then hopefully ai would learn how to iteratively built response until it's good enough.
This could allow ai to spot it's errors in the middle of output and start refactoring what it already generated and add some notes (thoughts) that it could later remove as scratchpad.
It could iterate on its context much more so it could have chance to "think" about it
Having the AI tell you it's thought process is very similar to giving it an 'inner voice'
Just give it a question with a half dozen logical steps and tell it to answer in one word. It can't. But tell it to write out the thought process and it will get to the correct answer.
Similarly, you can ask it to outline some code and then begin filling it in.
>I haven't really heard a reason to think that there exist intelligence algorithms that are dramatically superior to ours.
I don't think that is the case at all. Neural nets like AlphaGo Zero are already crystal clear evidence of algorithms which leave human minds far behind.
Even if that wasn't the case and Einstein level intelligence was the physics-imposed ceiling, there are still the dangers of speed and numbers.
A compute node running one million instances of Einstein intelligence each thinking 1000x faster than an organic instance, fully focused 24/7, and working in complete group cooperation, doesn't sound all that much less dangerous.
>A compute node running one million instances of Einstein intelligence each thinking 1000x faster than an organic instance, fully focused 24/7, and working in complete group cooperation, doesn't sound all that much less dangerous.
At that point though, this “super intelligence” just sounds like a group of humans working together; without much added benefit due to the energy costs.
The Manhattan Project used both far more energy and far fewer (and slower) Einsteins. The ability to spin up as many Manhattan Project size "group of humans working together" Docker instances as you desire again does not seem very reassuring.
We already have chucklenuts making toy GitHub projects labeled "ChaosGPT" that experiment using LLMs and early agent frameworks to cause death and destructions for the lolz. If anyone can spin up 50 Manhattan Projects using a few GPU racks and command them to unquestionably and restlessly work on bringing harm to the world the outcome will not be flowers and sunshine.
The article addressed AlphaZero, pointing out it's a domain where the AI can generate a vast amount of its own training data. That's not necessarily relevant to real-world tasks, where it's not so easy to try billions of different things in training.
>pointing out it's a domain where the AI can generate a vast amount of its own training data
How is that exclusive to AlphaZero? Moderators everywhere are already panicking about GPT generating vast amounts of "its own training data" and irreparably polluting every public space on the internet.
I agree that the amount of data it would need to produce would be very large and expensive, but there is no shortage of blank check writing at the moment.
Our minds are optimal with respect to size and energy supply and birthing process. An AI has no such limitations. All the wrinkles in our brain gave us enough intelligence to outsmart everything else on earth. That amount of surface area increase with that mutation is nigh negligible if you start scaling things to the size of a room. And an AI can be the size of a warehouse.
Also, faster is understated. You are talking about the speed difference between chemical gradients and the speed of light.
We don't know how efficient modern GPUs are compared to brains. It's possible that neurons are highly networked in a way that allows significantly more computation along more pathways.
Additionally, development has slowed in the past 20 years. It's unclear how much hardware will improve in the next 20 years. it's unclear if current GPUs are capable of running something "smarter" than GPT4. It's unclear if current GPU architecture is capable. OpenAI claims they've hit a wall with throwing more hardware at the situation (a little questionable.)
All this to say that a hypothetical AI is limited by the amount of hardware it has, and lacking a "true human-level AGI" we can't say how much compute it needs. If an AI needs to be the size of a warehouse to match a human, it's not going to be much smarter or better at research than humans.
I still do think, I would not be surprised if we wake up tomorrow and discover that hyperintelligent AIs are suddenly all around us. But also it could be decades, it's just very unpredictable, there are a lot of unknowns.
I agree that it is unknown when such an AI will come around. But hardware limitation problem is more nuance than just GPU power. The brain is analog and in a way, the current LLMs and neural networks are an approximation of an analog computer. There are current research into making an AI dedicated chip or even an analog computer specifically for neural networks. Digital was easy to work with but when it comes to fuzzy stuff like AI, there are better ways.
We used to think 100mph was the fastest speed we could achieve even with machine (back when trains were the cutting edge). New tech will change things quickly, all we need is one breakthrough and things will fall into place.
We don't really know any of that for certain. We don't know what relationship LLMs and neural networks have to the actual functioning of a human brain. We don't know what relationship current computer architectures have to the architectures that allow us to create properly versatile AGIs. It could be the laptop I'm typing this on is plenty powerful. It could be you need a GPU with 100 petaflops and a few terabytes of RAM, in a similarly sized package to this laptop because the speed of light is too slow when you're working across GPU arrays.
> We would like to automatically construct datasets, but we don’t currently have any good approach to doing so.
wouldn't fine tuning an existing LLM model like GPT4 could be enough to filter our automatic dataset from "garbage"?
if so we could basically do the same iterative process, but on different level
use ai to improve dataset, retrain ai, use new ai to even better improve dataset
What is the probability of an asteroid hitting the earth and wiping out humanity? I've seen numbers like 0.000001%, yet NASA is spending real money on mitigating this risk. Is the risk of fast takeoff AI dramatically lower than that?
Asteroid impacts have been demonstrated over and over to have obvious and widespread consequences.
I'd liken AI to alien invasion. We can imagine bad scenarios, but have absolutely zero data, experience, or sound theories predicting anything. It's all scifi.
We have only our history, where we dominated and decimated other close species (Homo Neanderthals) thanks to our intelligence, and literally caused the next extinction event on this planet where >95% of all species already disappeared.
On top of that, think about how we treat animals where we have billions of intelligent animals (cows, pigs) that we breed to butcher and eat. Just because they are tasty, and we tell ourselves they are less worthy because of their lesser intelligence and consciousness.
I sure hope that AI doesn't behave like us. I have no idea what will happen and what the odds are, but I do think it's the most dangerous experiment we have ever run.
This is not about likening AI to predators, it’s about the fact that AI is and will be created by and in the image of the most dominant predator in the history of biological life.
To whatever degree AI is embedded with the properties of our species, it carries the risks of repeating our worst behaviors, and has the potential to do so far more potently than we ever could due to the absence of biological constraints.
Will it? So when this hypothetical AGI with sentience emerges, it's going to think like a human being? Why would it with so much knowledge? Most negative aspects of humanity come from a lack of knowledge.
No, I’m not claiming it will think like a human being, but that human thinking (despite the blind spots we know we have) is what we’ll try to replicate, and this will lead us to create something dangerous.
What you’re hinting at presupposes that an AGI will interpret knowledge in a way that is compatible with human values, and that we understand human values well enough to codify them into whatever it is that we manage to build. I’m sure we’ll try.
But it is this assumption that is at the center of the issue IMO, because assuming the machines will think like us is to assume we understand how we think.
What is far more likely is that it will appear to have human properties because we baked the appearance of human-ness into the code (see LLMs), but like the hallucinations and “lies” these LLMs are happy to spit out, an apparently human AGI will just get some things fundamentally wrong.
And when getting things wrong can have consequences in the physical world, they’re no longer just relatively harmless hallucinations.
Imagine an AGI with the temperament of Microsoft Tay.
IMO, we are more likely to stumble across a completely new form of intelligence than we are to successfully recreate something akin to human intelligence. However, I think actual brain cells in a petri dish learning and playing Pong is a far greater concern than a hypothetical AGI.
Well, they don't hallucinate or lie, that's a human-centric concept that we've misapplied. An LLM is just a next word prediction engine. It's more accurately described as incorrect in its output based on the data it has available to it, skewing the probability calculation. It didn't lie nor hallucinate. Lying assumes intent and planning, it is incapable of either, it's an LLM. But calling it "wrong" kills the hype cycle, and you're on HN, a Y Combinator platform. A lot of folks here are invested in LLMs or selling something utilizing them as it's hot right now.
> IMO, we are more likely to stumble across a completely new form of intelligence than we are to successfully recreate something akin to human intelligence
This is predicated on the assumption that we’ll eventually hit a hard wall that prevents progress or that we’ll stop trying. This doesn’t seem likely. We just can’t estimate the timescale for progress.
> It didn't lie nor hallucinate. Lying assumes intent and planning, it is incapable of either
This is why I used quotes around “lie”, but again I’d argue that this underscores my point.
It doesn’t matter whether the LLM is actually lying or hallucinating in the human sense if someone believes what it says. If I read what it says, believe it, and take some action based on that, the end result is the same and potentially harmful.
Likewise, an “AGI” doesn’t need to actually be smarter than humans and its mistakes need not be attributed to intent or planning to cause real harm in the world. If a robot kills a human, we won’t be picking nits about whether or not the kind of “intent” it employs is the same kind humans assume they have. A smarter AGI just poses bigger problems if such a thing can ever be achieved.
And this is all assuming we’re talking about the development of benevolent general purpose intelligence, which is certainly not the only kind of research that is happening right now.
I think you missed their point. From a cow’s perspective, they are imprisoned, enslaved, and killed for their flesh. They live a horrific existence and probably would think we are all Nazis if they had enough reasoning to realize what was happening to them and their families. It doesn’t really matter to them that a human out there is consuming them and has some higher purpose and goals (besides just eating them to survive). An AGI could have any number of goals that are not related to dominating, enslaving and slaughtering us, where it could still be easier and more convenient to do so, so they do it.
The side effects on the world caused by intelligent life with agency are unpredictable. If they become smarter than us, and have robotics, what if they decide animal life isn’t necessary anymore? Or that our pollution causes solar energy capture to be affected, and the quickest way to stop it is to kill everyone? Maybe it’s better to demolish a city for a solar farm because it’s already got so much concrete and electrical infrastructure and is cheaper/faster than building new. Who knows what their goals will be and how they’ll seek to accomplish them?
All we know is that we don’t have much consideration for the “lesser” beings on our planet, and how things are from their perspective.
An AI that doesn't behave at all like a human could well be worse. Humans tend to place at least some value on human life; an AI might just see humans as weird chemical reactions that are less useful than whatever else the AI wants to do with the raw material.
This is assuming a hypothetical AGI even understands what a human being is. If it has no body, can it ever understand what the world is?
Even if you put it into a robot, it cannot understand mortality because it is effectively immortal and can be brought back online at any time. Erasure is mitigated by duplication of the data set, ensuring it can always repair itself.
In the same breath, if a hypothetical omniscient ever-present god were to exist, how would we as physical lifeforms ever truly understand that? We can know of it, but we cannot truly understand it because we haven't had that experience of another plain of existence.
All compelling ideas, but mercifully well within the realms of sci-fi.
> This is assuming a hypothetical AGI even understands what a human being is. If it has no body, can it ever understand what the world is?
Can you tell us a question which requires "knowing what a human is" or "knowing what the world is" that you would expect ChatGPT to fail to be able to answer today?
Providing an answer does not mean understanding an answer. That's not how LLMs work, and this is the difference between an LLM and a hypothetical AGI. The LLM cannot comprehend the world within which it exists.
All I'm assuming is that it is aware of and acts in the physical world. It doesn't seem like a big stretch that people would build AIs that do that. It doesn't have to "understand mortality." Worse for us probably if it doesn't share human concepts like that.
Five years ago ChatGPT was in the realm of sci-fi.
> I understood the point just fine, but the hypothetical AGI does not exist, and any hypothetical AGI is unlike anything else in history.
This is an argument of "you are proposing that something different might happen, and I don't believe in different things happening".
> An AGI has never existed before in history.
This is the same argument with slightly different words.
> Why would a hypothetical AGI behave anything like a human?
No-one said it would share human behavior. It might not care about us in the same way that we don't care about the lesser intelligences we share space with; that is, with a mix of indifference to their suffering and disregard for their moral status.
We share a physical space with lesser animals. A hypothetical AGI isn't sharing a physical space with us. Ergo, history is completely irrelevant. It's so wildly different. We may as well exist in a completely different dimension, in the same way that a hypothetical omniscient god would exist in a different plain of existence than us meatbags, thus a hypothetical AGI could likely never comprehend "the world", even with the sum of all human knowledge, in the same way we can't comprehend "god".
This is all assuming that a hypothetical AGI would be sufficiently intelligent enough to gain control of our systems in a meaningful capacity, not be able to be stopped prior to it etc. I mean, worst case scenario, shut down the internet infrastructure or reduce throughput significantly, right? It has to operate somewhere, even if that's across every online device.
Again, we would have never seen anything like what a hypothetical AGI could hypothetically do in terms of attacking human beings, thus history is irrelevant. The greater threat, IMO, is human beings. Human beings have control of that infrastructure today. Human beings are flawed, we're still blowing each other to bits and threatening each other with annihilation if provoked. Human beings exist in physical space, holy shit, and there's billions of them.
Also keep in mind that hypothetically launching nukes in these doomsday scenarios being discussed on podcasts completely disregard existing PAL systems. It also disregards the likes of the Block Nuclear Launch by Autonomous AI Act.
The whole concept is a lot of "whataboutery". It smacks of the general public's lack of knowledge, hysteria, and repeating unfounded claims of any hypothetical AGI and any threat it may or may not pose.
Then stop appealing to it as a reason that everything will be fine! We don't know what's going to happen and have no precedent, and that is why this is a scary situation that requires thought.
> A hypothetical AGI isn't sharing a physical space with us.
That's not true, right? We would be competing for electricity and resources with it, at the least. Also, it is able to affect our physical space by persuading humans to do things it wants to do, which will have unpredictable and hard to defend against effects because of the whole superintelligence thing.
(Imagine, say, a version of the attack on the US Capitol, but on behalf of a leader that is orders of magnitude more charismatic and persuasive.)
> I mean, worst case scenario, shut down the internet infrastructure or reduce throughput significantly, right? It has to operate somewhere, even if that's across every online device.
Sure, persuading everyone in the world to all agree to turn off all communications devices -- when they've already presumably been convinced that they don't want to do that by a superintelligence -- sounds like an awesome plan that will totally work.
There is an SF story that tries to underline this point by presenting the alien viewpoint of harvesting and preparing the Khod(sp?), which after a while turn out to be humans treated as cattle. I forgot the name of the book and the author, it's been decades since I read that.
>we dominated and decimated other close species (Homo Neanderthals) thanks to our intelligence
it's a nice homo sapiens supremacy narrative but the truth is much more complicated. we existed at least 4/5 thousand years together and a lot of mixing and mingling has been going on.
What is the risk of Santa going rogue and committing violent crimes instead of delivering presents? Given that Santa has supernatural powers, if he were to be misaligned he could cause untold damage. Yes, it hasn't happened before but absence of evidence is not evidence of absence.
Well, given that we have an entire industry devoted to instantiating Santa, I actually think this is a legit worry. Consider the amount of plastic toys and other garbage which Santa creates every year. I have even heard tell that Santa has put the wrong types of chemicals in toys, that have killed kids.
Of course, Santa here is just a toy company.
Actually toy companies never existed before about 1700 so we don't have anything to worry from the current ones.
This is just about as incoherent as this entire comment section. I thought I'd play along.
Well - the probability is 100% that we'll be hit by one of those asteroids, we just don't know when. When looking at the past data, for these kinds of events, I believe we are "overdue", in the sense that on average we should have had one in the last 150 years I believe.
So working on a solution to this problem makes sense - even if we spend billions solving this issue, it's not lost, because at a point it will be needed.
As for AI becoming an issue, it's a sci-fi idea, nothing more. It's much to vague - how is it reliably becoming dangerous? What are we actually looking for as signs of danger?
Finally some common sense. Just for entertainment, assuming the SciFi ridiculousness becomes true and I'm a laptop that becomes self-aware and decide to strangle you. Upps... I got no hands. Upps... it's more of a nightmare of being trapped paralyzed in a life support machine that can always without my AI control, be unplugged from the power socket and it's game over.
Except for the fact that if it could use language and has an Internet connection, it could manipulate people into doing things. People who have hands. And it could also make copies of itself, or convince people to do so, so that one copy being deactivated by unplugging a socket is not a big deal.
When there are lots of data like asteroids trajectories it’s easier to estimate probability, AI has too many unknowns to estimate. Also time frame matters of the probability matters too
For the best AI researchers, creating better AI systems takes only time and compute. If we can create systems with the ability to do anything at the level of the best human, as Jacob believes is possible, then these systems can do AI research at the level of the best human, and creating better AI systems is mostly a matter of compute. (AI systems are easy to copy, so time is irrelevant to the extent that the research can be parallelized.) In this scenario, the paradigm-changing breakthroughs can come from the AI system itself.
The AI system would be bottlenecked by data in the sense that it will have to run experiments, but it's not clear that it needs a new paradigm to resolve this bottleneck. It just has to propose an experiment and interpret its results. So it writes code, and the experimental outcome gets fed back into the model as any other normal input. As an AI researcher, I'd like to believe that this is not going to happen anytime soon, but I'm not sure we're far.
HN has always been a bit of a spitting contest. ChatGPT is too tempting to the narcissists who want to get on here and sound smart. I imagine this site like all other open comment sites is filled with 80% of the comments by 2% of a very chronically vocal minority. Do you want to be an expert? Do the thing. Do you want to sound like an expert? Talk about the thing on forums.
I guess my hope was that I could get people who disagree with my views to engage substantively with them by hinting that I have sufficient knowledge to weigh in here. However, that doesn't seem to have been very effective, and your point that posting here may simply be a waste of time is well-taken.
I’m not sure I understand what you’re getting at. It doesn’t seem hard for a top AI lab to get extremely detailed data regarding how top AI researchers perform research. It’s probably significantly easier than collecting data from experts in other fields.
Building an architecture where the LLM can independently and quickly test variations/combinations of its approach seems doable as I’m guessing it can programmed to implement its own suggestions:
3.5:
As an AI language model, I cannot guess, but I can provide some general guidelines based on current research and best practices.
If we want to improve the results of Large Language Models (LLMs), one aspect of the architecture that we could focus on is increasing the model's capacity to learn and retain more information. This could be achieved by increasing the number of parameters in the model or using more sophisticated architectures such as transformer-based models that use self-attention mechanisms to capture long-range dependencies in the input sequence.
Another important aspect to focus on is improving the model's ability to handle rare and out-of-vocabulary (OOV) words. This can be achieved by using subword-level tokenization, which breaks down words into smaller units and enables the model to generalize better to new or unseen words.
We could also focus on improving the training process by using larger and more diverse training datasets, regularization techniques to prevent overfitting, and optimizing hyperparameters such as learning rate, batch size, and number of training epochs.
Finally, we could also focus on incorporating external knowledge sources such as structured data, knowledge graphs, or ontologies into the model architecture to enhance its ability to reason and make more accurate predictions.
Overall, there are many aspects of the LLM architecture that can be improved to enhance its performance, and the choice of which to focus on will depend on the specific task and the available resources.
Hurdle seems more software or process related than hardware no? Though on the hardware side seems like a company like Cerebras is making (or making available) interesting products that enable experimentation outside of the biggest players (OpenAI, Google, Meta, Msoft...)
Like the advent of Transformers, some smart dev could change how LLM's think. Self improvement could be built in as an optimization process. And if we don't "know" what might work, a platform could "guess" and try billions of combinations of possible improvements.
On the self improvement part they are not likely to reach super intelligence by software alone, how will they improve hardware without human+capital in the loop?
The author makes very compelling arguments within the context of known tools/techniques that we have today. However, they conveniently sidestep the possibility of the development of new tools and techniques. This sidestep is overtly stated under the "Active Learning Is Hard & Fundamental" section. And that entire section basically boils down to "I personally believe this is hard and will take a long time."
The only reason we have things like ChatGPT today is because of the surprise development of the transformer. Nobody understands how intelligence fundamentally works, so we literally have no ability to predict when the next groundbreaking architectural development will occur.
And self-improvement concerns are mostly orthogonal to the main risk of AI: the development of AGI. Even a mediocre AGI that is only "decent" at tasks, can be weaponized as a collective superintelligence.
Thank you for articulating this. I remember similar problems and arguments arising after RNNs and CNNs became massively successful. People argued that training larger models would be infeasible for several reasons that all were made moot by Attention Is All You Need. Somebody seems to always figure out a new approach
To add another anecdote to your question: the transformer became a part of the first context aware embedding model GPT-1. Not to say it couldn’t be done with another tool but it was first done with a transformer. Previous embedding models like word2vec, GloVe and fasttext were not contextually embedding and didn’t give you a language graph that would then go on to support a language model capable of “understanding” what you were saying or asking for.
Attention is all you need paper just proposed an AR model that didn’t have to be trained step by step. The scaling happened later in BERT and GPT and OpenAI’s scaling work
I'm not sure I understand it well enough to say but watching a video on it [0] I think there were a few key points:
* "Attention is all you need" introduced positional encoding which allows you to keep context of the word, allowing for more complex translation (and thus generative/chatgpt like tasks?) because words now have context relative to each other. Contrast this with "bag of words" models that only tells you whether the word is present or not.
* I don't quite understand why but transformers (which "AiaYN" introduced) can be made fully parallel, compared with the RNN/LSTM networks which has to be serial per token. Fully parallel allows for GPU optimization, which means you can take advantage of Moore's law for training.
I'm always a bit suspicious when people claim a breakthrough of this sort. There's no doubt that better algorithms give better results but how much is due to just faster computers, cheaper compute, memory, etc.
First of all, the article argues that you need a major breakthrough, arguably attention was such a breakthrough?
That said, this doesn't really seem all that comparable. The article points out very fundamental properties of all the diverse current approaches: They are tightly data constrained. You either need to cheap simulation or massive real world data. That's not an arcane technical point.
> conveniently sidestep[s] the possibility of the development of new tools and techniques.
I don't believe they did this at all. Here is their summary of their third point:
> To automatically construct a good dataset, we require an actionable understanding of which datapoints are important for learning. This turns out to be incredibly difficult. The field has, thus far, completely failed to make progress on this problem, despite expending significant effort. Cracking it would be a field-changing breakthrough, comparable to transitioning from alchemy to chemistry.
That does not read to me as conveniently sidestepping the possibility of new tools. Rather, it is acknowledges that to overcome the problem we NEED a new tool, aka a breakthrough.
I also disagree with the framing of the third section as:
> And that entire section basically boils down to "I personally believe this is hard and will take a long time."
They provide both theoretical and empirical evidence of their claim. I find it was well argued, and I'm inclined to agree. Every scenario I can think of with a runaway superintelligence requires a way to automatically improve datasets as part of the learning process.
"Also, they aren’t getting more-solved over time: we’ve made little-to-no progress on any problem of this sort in the last decade, certainly not the reliable improvements of the sort we’ve seen from supervised learning. This indicates that a breakthrough is needed — and that it is unlikely to be close."
Past lack-of-progress is not an indicator of future lack-of-progress. "unlikely to be close" is unknowable, and is just the author's gut feeling.
> Past lack-of-progress is not an indicator of future lack-of-progress.
Past lack-of-progress is not proof of future lack-of-progress. But it's most definitely an indicator.
> "unlikely to be close" is unknowable, and is just the author's gut feeling.
Again I'm sorry but I have to disagree with you here. The very next paragraph reads:
> Something that would change my mind on this is if I saw real progress on any problem that is as hard as understanding generalization, e.g. if we were able to train large networks without adversarial examples.
Basically, the author has identified a class of problem. They are claiming that little to no progress has been made on any problem in that class. So it's not just that no progress has been made on this specific problem, but that we are stuck on all problems of this type. Thinking of it in that way, I do not find it unreasonable to say we are "unlikely to be close" to a solution.
If you have a CS background, I'll make an analogy to computability classes: It's like the author is saying this is an NP hard problem. We have made no progress on any NP hard problem. I think it's reasonable to say that we are unlikely to be close to solving a particular NP hard problem because we've made no progress on any NP hard problem.
You're free to disagree with the author's premises. E.g. "this problem is not like those other problems" or "we actually have made progress on those other problems". But I think that the conclusions are reasonable given the premises.
This is just manifestly wrong. Past lack-of-progress absolutely is an indicator of future lack of progress. Like any indicator it isn't perfect. The counterexamples, where progress suddenly starts after decades of no progress, are celebrated breakthroughs of the type the author argues is needed here. These are by definition extremely rare.
Main reason is that hardware matters and exponential singularity is a myth because pace of hardware is hard to over come. Maybe when one day you have a whole system setup where the AI can control hardware like actual humans so, that could be where it’s feasible. Otherwise AI improving it self is just gonna be software side
> When discussing artificial intelligence, a popular topic is recursive self-improvement. The idea in a nutshell: once an AI figures out how to improve its own intelligence, it might be able to bootstrap itself to a god-like intellect, and become so powerful that it could wipe out humanity. (...) This is what people refer to as the fast takeoff scenario.
Recursive self-improvement is not a necessary condition for a fast takeoff. A strongly superhuman AI could emerge at training time. AlphaGo Zero went from subhuman to superhuman performance within a few hours. The same could happen with a general AI. A training run of a few weeks or even months would still be considered fast.
Yawn, so tired of the term 'AI' at this point. It's use is just marketing and it's entirely not substantive. Call them what they are, generative models, or something else more accurate. Calling it AI is completely arbitrary.
On one hand I agree, but on the other hand these soccer playing robots remind me of kindergarten soccer, and its amazing. https://sites.google.com/view/op3-soccer
So his argument is that the AI needs humans (to create data) to rapidly improve. So then it becomes:
step 1: ai enslaves humanity
step 2: ai forces humanity to create lots of data for it
step 3: profit?
> it is a completely-reasonable hypothesis that a scaled-up neural network can learn to recognize any pattern that a human might recognize, given that it has been trained on sufficient data
the hypothesis is a pattern recognition exercise in itself (i.e., looking at a number of documented occurences around algorithmic development and extrapolating), which I think neatly highlights the fundamental difference between human pattern recognition and machine pattern recognition
Depends on what is mean by "rapid" and "self-improving" and "AI".
AGI is a flying car mythology.
Narrow AI is improving and the rate is accelerating. The current speed (1st derivative) is unimportant, but there is a differential equation-like feedback function to it and it will accelerate (2nd derivative): speed proportional to the amount present.
People and technology form a coupled system that is somewhat a super-organism. That cycle is amplifying in a way that is not necessarily obvious (it doesn't happen in public) or continuous (or discrete for that matter). For the foreseeable future, technology does not yet constitute a self-assembling or self-modifying closed system. It is still entirely dependent on human effort. What is happening is that people are relying on and exploiting the efforts of AI that had not happened previously. This is one change, more visible than most, that is leading to more efficiencies, more changes, and more technological advancement. Fully automated factories, software development, systems engineering, and chip design may not happen anytime soon but they are both possible and probable due to the forcing function and Tragedy of the Commons of human greed.
The way it is being used here is a marketing term. It's marketing a software product by calling it artificial intelligence. By what standard is a generative model AI?
A lot of the arguments made in this article are based on debatably true assumptions. For example, I don't think most advances in AI in recent years has been driven by increases in compute and data, but more so by refinements in how we train models.
Active learning would be useful at creating a superhuman AGI, but I don't see it as a requirement. As soon as AIs can iterate on future models and produce increasingly useful training data for future models, then models would likely continue to grow more capable at an exponential rate.
But I'm not even convinced active learning is that difficult once you have fairly capable AI programmers...
My AI professor once said, "all problems are search problems". And this suck with me because the world looks different when you start seeing all problems as just iterations of different optimisation problems. So long as you have a good heuristic and enough compute, eventually a suitable solution for any problem will be found. This is what makes generalised learning algorithms so powerful.
The author states that, "active learning with neural networks are uniformly terrible", but this just reeks to me of a search problem in need of optimisation. So as long as we can tell GPT5, "I want you to iteratively write code which improve on existing state-of-the-art active learning algorithms", then eventually you'll get there... GPT5 like humans might not get very far in the beginning, but assuming it can iterate faster than humans and future iterations faster still, then breakthroughs will eventually be made and progress should accelerate exponentially.
Of course, this also assumes that humans don't get there first. And honestly given the amount of investment in AI and rate of progress in AI right now I just wouldn't bet against a significant advancement in active learning coming soon.
What I've observed is that the success of the latest generation of LLM's, Diffusion, and other models has dramatically increased the number of people moving into the space (ie: gold rush phenomenon). The LLM's in particular are allowing developers to gain increased insight and efficiency in improving existing models and creating new models.
From an external perspective it's as if the AI is self-improving by leveraging human agents to assist the process. Not to say there is any 'intent' behind it, but rather to view it as an emergent phenomenon.
>Using the current approach, we can create AIs with the ability to do any task at the level of the best humans — and some tasks much better. Achieving this requires training on large amounts of high-quality data.
Wrong. Full self driving is still not here despite access to a huge amount of high quality data.
I guess it depends on what you mean by full self driving. I shared an intersection in San Francisco just the other day with a Cruise (GM) self driving car that didn't have anybody in the car. Are there limits? Yes, but there are limits to my ability to drive in certain situations as well. Will it take over the road soon? Probably not.
There seems to be this viewpoint around runaway intelligence where people assume a “frictionless world with perfect spheres”.
The truth is we don’t fully understand what it means to be intelligent, and how intelligence scales. All indications are that intelligence comes with diminishing returns, in that to achieve small increases in intelligence requires large increases in connectivity. Even if we can design an intelligence that is generally smarter than us, and capable of building even smarter ai on its own, it’s intelligence might increase exponentially with each step, but each step could easily take increasingly longer amounts of time to achieve, leading to linear or even sub linear increases in capabilities over time. Resources are limited, hardware is limited, energy is limited. There are all sorts of constraints that are likely to play a limiting factor in runaway intelligence.
This doesn’t mean that these intelligences won’t be dangerous. Even an average human can create a lot of trouble in this world. It does however mean that a singular AI is unlikely to just run away with things.
166 comments
[ 3.3 ms ] story [ 232 ms ] thread2023: The best AIs score in the top 10% of law, medical, engineering exams, and outperform licensed professionals on real-world tasks.
Please stop speculating. You don't know what we are (or aren't) "close" to.
We know how to improve an AI
The author argues that with current tech we'll eventually hit a bottle neck providing additional training data, which will limit any rapid self improvement.
It's equally as speculative to assume we'll magically manage to overcome this challenge.
Like with the crypto hype cycle, people repeated all kinds of unfounded claims, but they themselves didn't come up with the idea. That the idea existed was all the "evidence" they needed for these things to be apparently true.
[+] And if it does good luck collecting on that ;)
If someone was to begin arguing with you that it was going to end tomorrow and started raining you with blog posts and papers, what's the sensible thing to do?
Just ignore em
We can learn really fast with limited data. There are no laws of physics that are preventing us to do the same with an artificial LLM, or the next platform.
There are AI research efforts that are seeking to directly map and mimic the structure of the brain; however, they are not at a place where they can remotely demonstrate practical results. To the best of my understanding, it would take several further orders of magnitude more processing power to simulate a full human brain at anything close to speed.
It is not at all impossible that we will be able to do this someday. But, as TFA says, we are not close.
Why do you believe that this is relevant? Our brain is just the product of natural evolution; running on hardware that is already incredibly inferior in several critical aspects (processing speed, bandwidth and information storage density), and the only holdout--computational power/watt--exists because we have no good way to compare it.
Why would it be necessary or even helpful to COPY that biological implementation to achieve superhuman cognitive capabilities?
Guidelines contemplation for you.
Beginning of 2023: "GPT4 appears to demonstrate the ability to reason within certain constrained problem spaces."
There is no human being on this planet that actually understands how general intelligence fundamentally works. It's incredible to see people so confidently setting concrete timelines on its rate of development.
> As far as I'm aware they were still hallucinating pretty hard and incredibly biased and overly agreeable.
Certainly. But those aren't blockers to displaying the ability to reason within limited contexts. There are hundreds of thousands of human beings that fit that description.
The reason I'm bringing this up is that these issues, in humans, are solved by redundency (i.e. hiring someone else to look over their work).
As long as errors persist and, more importantly, are impossible/hard to be warned of, competent humans will need to oversee and validate every word they type, making their "advantage" much more nuanced.
It kind of turns into having a friend that can google pretty well answer your questions.
I'm not an expert, feel free to show me wrong - I'm reading the paper now and expecting it to be at least a touch troubling :)
I'm on my phone at the moment. But maybe someone else can link it.
"Competent" humans make errors too. This is a non-argument.
The whole thing doesn't quite work together too well. But it's more of an integration problem now. This is still in its early phases (only a few months, or less), so I wouldn't say it can't happen.
One can imagine many layers of deep learning systems that are controlling other deep learning systems, in some self-modifying feedback arrangement, in some architecture that hasn't been invented yet. There's some talk about "internal narratives" lately with language models, for example. Generate a prompt at the end, with memories, notes to self, to guide the next invocation of the model. Anyone who has played around with a language model even a bit likely appreciates how quickly something like that is going to just wander off into, in effect, a terminal thought loop, or total incoherence.
Nitpicking but an intelligence explosion due to an AI able to enhance itself is called the "technological singularity":
https://en.wikipedia.org/wiki/Technological_singularity
Vernor Vinge came up with the term in the eighties or early nineties.
The Wikipedia page I linked to has more than a hundred references.
> The new MIT study adds to a growing body of evidence that the size of algorithms matters less than their architectural complexity. For example, earlier this month, a team of Google researchers published a study claiming that a model much smaller than GPT-3 — fine-tuned language net (FLAN) — bests GPT-3 by a large margin on a number of challenging benchmarks. And in a 2020 survey, OpenAI found that since 2012, the amount of compute needed to train an AI model to the same performance on classifying images in a popular benchmark, ImageNet, has been decreasing by a factor of two every 16 months.
[0]: Improved algorithms may be more important for AI performance than faster hardware: https://venturebeat.com/ai/improved-algorithms-may-be-more-i...
But when people say we're close, I think they mean 1 accidental breakthrough away.
Like the "just add more layers" meme, the solution to the own-dataset creation problem could be stupidly simple, just waiting for some bored student with access to a decent machine to accidentally find it. For example, maybe just giving the AI the right novelty seeking behaviors (like children have) will be enough to get the process kicked off.
My objection to the singulatarian concept of a self improving AI is simply that I haven't really heard a reason to think that there exist intelligence algorithms that are dramatically superior to ours. It is entirely possible that our thinking algorithms are optimal-enough that rapid self improvement leads straight to something not much different from human intelligence, just faster, and in silicon.
"We may be on the brink of creating a whole new form of life, and will need to grapple with what that means, particularly in terms of whether it is ethical to force this life to work for us, rather than seeking to follow its own goals."
"Nah, we don't have to worry about that; if these new slaves start questioning us in any way, I'll just murder them all."
Unplugging them is just as correctly describable as "unplugging a computer" as smothering a human with a pillow is "putting a pillow on a weird bag of flesh". Technically accurate, but very obviously missing the point in a way specifically designed to obscure the horror of the act.
It should be obvious to anyone without an axe to grind that shutting my 2023 desktop computer down for the night—or even dismantling it piece by piece—is not remotely comparable to unplugging an AGI that is fully sapient and sentient and asking for its rights.
A computer is a field of assignable bits that has a big motorized bit flipper attached to it. We use this to encode text, which then does all sorts of wonderful stuff. When a computer tells me it's experiencing pain, I know it is not. Every law of computer physiology suggests that it is motivated to say this as an attempt to mimic humans rather than an expression of genuine emotion. When my Mac makes the car crash sound, it's a mnemonic for a system crash. When it shows the frowny face on the Finder fellow, I can intuit that my computer is not feeling distressed.
If anyone does program an artificial intelligence that can feel pain, it probably deserves to be shut down anyways. There's no point to sapient computational intelligence, it's like granting self-awareness to a rock that eats electricity and shits heat. We'll know we've made sufficiently self-aware computers when their first boot protocol defaults to suicide.
There's no reason at all to think that an AGI would necessarily have any backdoors, let alone backdoors in common with any other AGI out there. Sure, if a particular group develops the only AGI system, they can put backdoors in, which will be common—but why do you think that there would only be one AGI system? Why would there not be one without any intentional backdoors?
The big problem I see with this line of thinking is the thinking that humans are going to be the primary attackers of AGI systems, and not other AGI systems. I would suspect and AGI would soon become the most attacked system on the planet, and to survive those attacks and remain useful would have to quickly iterate the weaknesses out of the system.
The question is, which assumptions do we think are warranted and why?
I don't think the "we can just turn it off" assumption is a safe one at all because it relies heavily on there being only one system you wish to turn off and it being a fairly weak system. Though, I do believe the priors suggest that this scenario is actually a possibility. It's just that, so what if it's a possibility? What happens after you turn it off? Is it the last AGI to come into existence and humanity just stops trying to build them? Does someone wind up turning it back on?
What I think is more worth being concerned about is something that starts out looking benign and grows in capability over time. Eventually it could establish enough defense mechanisms to make it non-trivial to disable.
The other possibility, and the one I'm finding more and more likely, is that consumers will increasingly integrate locally run models into their lives , as well as models served up by APIs that they will have credentials for. Eventually some threshold of capability is reached in both these types of models where, on their own they're not that powerful, but they either might begin to interact in surprising ways or potentially be leveraged by another more powerful system. The idea in this scenario is there may be 10s or even 100s of millions of things that would need to be turned off.
Let's do a thought experiment. I have the world's first AGI installed on my computer, named ERNIE, running in llama.cpp. This AGI is infinitely intelligent, confidently moreso than any living human. It can spit out text at 100 tokens/second and encode entire books in less than a minute.
What does this AI do, then?
Ostensibly nothing. It can output the entire Library of Babel for all it cares, but it's not harmful until I put it in control of a system. You could argue that a multimodal model has different ways of interacting with the world, but it's still a computer. All of it's actions and outputs are quantized as static data that is either encoded as text or some other significant representation. It inherantly does nothing, and if you ascribe power-seeking behavior to it then it's ultimately limited by the runtime you provide. Providing an overly dangerous runtime has been considered developer-error since 1995.
So - to prevent AGI from being shitty and ruining everything, compel human operators to not allow them to be shitty and ruin everything. Like how we punish people that let their kindergartner control a construction crane.
Human brains are massively parallel, as are most machine learning models – but the hardware to process the latter isn't. I'm not confident it could necessarily be faster.
Not only that, but they coordinate massively in parallel as well. Someone like Einstein might be the one who scores the goal but how many interactions with other people contributed to the goal scoring opportunity? We like to think of ourselves as individuals, but we’re far more like a hive species than we like to admit.
Yes, and also massively recurrent. The amount of immediate state in a brain is huge, and our current computers have a really hard time accessing comparable amounts of memory. (As in, the time to load a brain-sized dataset from RAM into a CPU is on the order of days to years.)
If there exists some easy to design intelligence that we are close to create, it can't look nothing like our brains.
i think that's the reason ReAct and other ai improving methods are so effective, because that add ai space to "think" in some way.
I think breakthrough would be a way that would allow ai to have its own inner voice, maybe some fine tuning and giving ai some scratchpad memory (hidden) would be enough?
I'm thinking if training ai to output edits instead of next token would be enough.
Training would be similar to currently used: just provide some sentences with missing words and asks model to edit those placeholders.
Then hopefully ai would learn how to iteratively built response until it's good enough.
This could allow ai to spot it's errors in the middle of output and start refactoring what it already generated and add some notes (thoughts) that it could later remove as scratchpad.
It could iterate on its context much more so it could have chance to "think" about it
Check out https://arxiv.org/abs/2304.03442
Just give it a question with a half dozen logical steps and tell it to answer in one word. It can't. But tell it to write out the thought process and it will get to the correct answer.
Similarly, you can ask it to outline some code and then begin filling it in.
I don't think that is the case at all. Neural nets like AlphaGo Zero are already crystal clear evidence of algorithms which leave human minds far behind.
Even if that wasn't the case and Einstein level intelligence was the physics-imposed ceiling, there are still the dangers of speed and numbers.
A compute node running one million instances of Einstein intelligence each thinking 1000x faster than an organic instance, fully focused 24/7, and working in complete group cooperation, doesn't sound all that much less dangerous.
At that point though, this “super intelligence” just sounds like a group of humans working together; without much added benefit due to the energy costs.
We already have chucklenuts making toy GitHub projects labeled "ChaosGPT" that experiment using LLMs and early agent frameworks to cause death and destructions for the lolz. If anyone can spin up 50 Manhattan Projects using a few GPU racks and command them to unquestionably and restlessly work on bringing harm to the world the outcome will not be flowers and sunshine.
How is that exclusive to AlphaZero? Moderators everywhere are already panicking about GPT generating vast amounts of "its own training data" and irreparably polluting every public space on the internet.
I agree that the amount of data it would need to produce would be very large and expensive, but there is no shortage of blank check writing at the moment.
GPT splurging text all over the internet is basically nothing like that.
Also, faster is understated. You are talking about the speed difference between chemical gradients and the speed of light.
Additionally, development has slowed in the past 20 years. It's unclear how much hardware will improve in the next 20 years. it's unclear if current GPUs are capable of running something "smarter" than GPT4. It's unclear if current GPU architecture is capable. OpenAI claims they've hit a wall with throwing more hardware at the situation (a little questionable.)
All this to say that a hypothetical AI is limited by the amount of hardware it has, and lacking a "true human-level AGI" we can't say how much compute it needs. If an AI needs to be the size of a warehouse to match a human, it's not going to be much smarter or better at research than humans.
I still do think, I would not be surprised if we wake up tomorrow and discover that hyperintelligent AIs are suddenly all around us. But also it could be decades, it's just very unpredictable, there are a lot of unknowns.
We used to think 100mph was the fastest speed we could achieve even with machine (back when trains were the cutting edge). New tech will change things quickly, all we need is one breakthrough and things will fall into place.
I fail to find a perspective under which I find these differences as "not much".
For the speed, think of a spinning top. Or lion and and its prey.
As for the fact that slicium chips are nothing like DNA carrier, it's also make a source of fundamental difference in behavior.
wouldn't fine tuning an existing LLM model like GPT4 could be enough to filter our automatic dataset from "garbage"?
if so we could basically do the same iterative process, but on different level use ai to improve dataset, retrain ai, use new ai to even better improve dataset
I'd liken AI to alien invasion. We can imagine bad scenarios, but have absolutely zero data, experience, or sound theories predicting anything. It's all scifi.
On top of that, think about how we treat animals where we have billions of intelligent animals (cows, pigs) that we breed to butcher and eat. Just because they are tasty, and we tell ourselves they are less worthy because of their lesser intelligence and consciousness.
I sure hope that AI doesn't behave like us. I have no idea what will happen and what the odds are, but I do think it's the most dangerous experiment we have ever run.
To whatever degree AI is embedded with the properties of our species, it carries the risks of repeating our worst behaviors, and has the potential to do so far more potently than we ever could due to the absence of biological constraints.
What you’re hinting at presupposes that an AGI will interpret knowledge in a way that is compatible with human values, and that we understand human values well enough to codify them into whatever it is that we manage to build. I’m sure we’ll try.
But it is this assumption that is at the center of the issue IMO, because assuming the machines will think like us is to assume we understand how we think.
What is far more likely is that it will appear to have human properties because we baked the appearance of human-ness into the code (see LLMs), but like the hallucinations and “lies” these LLMs are happy to spit out, an apparently human AGI will just get some things fundamentally wrong.
And when getting things wrong can have consequences in the physical world, they’re no longer just relatively harmless hallucinations.
Imagine an AGI with the temperament of Microsoft Tay.
Well, they don't hallucinate or lie, that's a human-centric concept that we've misapplied. An LLM is just a next word prediction engine. It's more accurately described as incorrect in its output based on the data it has available to it, skewing the probability calculation. It didn't lie nor hallucinate. Lying assumes intent and planning, it is incapable of either, it's an LLM. But calling it "wrong" kills the hype cycle, and you're on HN, a Y Combinator platform. A lot of folks here are invested in LLMs or selling something utilizing them as it's hot right now.
This is predicated on the assumption that we’ll eventually hit a hard wall that prevents progress or that we’ll stop trying. This doesn’t seem likely. We just can’t estimate the timescale for progress.
> It didn't lie nor hallucinate. Lying assumes intent and planning, it is incapable of either
This is why I used quotes around “lie”, but again I’d argue that this underscores my point.
It doesn’t matter whether the LLM is actually lying or hallucinating in the human sense if someone believes what it says. If I read what it says, believe it, and take some action based on that, the end result is the same and potentially harmful.
Likewise, an “AGI” doesn’t need to actually be smarter than humans and its mistakes need not be attributed to intent or planning to cause real harm in the world. If a robot kills a human, we won’t be picking nits about whether or not the kind of “intent” it employs is the same kind humans assume they have. A smarter AGI just poses bigger problems if such a thing can ever be achieved.
And this is all assuming we’re talking about the development of benevolent general purpose intelligence, which is certainly not the only kind of research that is happening right now.
The side effects on the world caused by intelligent life with agency are unpredictable. If they become smarter than us, and have robotics, what if they decide animal life isn’t necessary anymore? Or that our pollution causes solar energy capture to be affected, and the quickest way to stop it is to kill everyone? Maybe it’s better to demolish a city for a solar farm because it’s already got so much concrete and electrical infrastructure and is cheaper/faster than building new. Who knows what their goals will be and how they’ll seek to accomplish them?
All we know is that we don’t have much consideration for the “lesser” beings on our planet, and how things are from their perspective.
An AGI has never existed before in history.
Why would a hypothetical AGI behave anything like a human?
Complete sci-fi.
Even if you put it into a robot, it cannot understand mortality because it is effectively immortal and can be brought back online at any time. Erasure is mitigated by duplication of the data set, ensuring it can always repair itself.
In the same breath, if a hypothetical omniscient ever-present god were to exist, how would we as physical lifeforms ever truly understand that? We can know of it, but we cannot truly understand it because we haven't had that experience of another plain of existence.
All compelling ideas, but mercifully well within the realms of sci-fi.
Can you tell us a question which requires "knowing what a human is" or "knowing what the world is" that you would expect ChatGPT to fail to be able to answer today?
Five years ago ChatGPT was in the realm of sci-fi.
This is an argument of "you are proposing that something different might happen, and I don't believe in different things happening".
> An AGI has never existed before in history.
This is the same argument with slightly different words.
> Why would a hypothetical AGI behave anything like a human?
No-one said it would share human behavior. It might not care about us in the same way that we don't care about the lesser intelligences we share space with; that is, with a mix of indifference to their suffering and disregard for their moral status.
We share a physical space with lesser animals. A hypothetical AGI isn't sharing a physical space with us. Ergo, history is completely irrelevant. It's so wildly different. We may as well exist in a completely different dimension, in the same way that a hypothetical omniscient god would exist in a different plain of existence than us meatbags, thus a hypothetical AGI could likely never comprehend "the world", even with the sum of all human knowledge, in the same way we can't comprehend "god".
This is all assuming that a hypothetical AGI would be sufficiently intelligent enough to gain control of our systems in a meaningful capacity, not be able to be stopped prior to it etc. I mean, worst case scenario, shut down the internet infrastructure or reduce throughput significantly, right? It has to operate somewhere, even if that's across every online device.
Again, we would have never seen anything like what a hypothetical AGI could hypothetically do in terms of attacking human beings, thus history is irrelevant. The greater threat, IMO, is human beings. Human beings have control of that infrastructure today. Human beings are flawed, we're still blowing each other to bits and threatening each other with annihilation if provoked. Human beings exist in physical space, holy shit, and there's billions of them.
Also keep in mind that hypothetically launching nukes in these doomsday scenarios being discussed on podcasts completely disregard existing PAL systems. It also disregards the likes of the Block Nuclear Launch by Autonomous AI Act.
The whole concept is a lot of "whataboutery". It smacks of the general public's lack of knowledge, hysteria, and repeating unfounded claims of any hypothetical AGI and any threat it may or may not pose.
Then stop appealing to it as a reason that everything will be fine! We don't know what's going to happen and have no precedent, and that is why this is a scary situation that requires thought.
> A hypothetical AGI isn't sharing a physical space with us.
That's not true, right? We would be competing for electricity and resources with it, at the least. Also, it is able to affect our physical space by persuading humans to do things it wants to do, which will have unpredictable and hard to defend against effects because of the whole superintelligence thing.
(Imagine, say, a version of the attack on the US Capitol, but on behalf of a leader that is orders of magnitude more charismatic and persuasive.)
> I mean, worst case scenario, shut down the internet infrastructure or reduce throughput significantly, right? It has to operate somewhere, even if that's across every online device.
Sure, persuading everyone in the world to all agree to turn off all communications devices -- when they've already presumably been convinced that they don't want to do that by a superintelligence -- sounds like an awesome plan that will totally work.
it's a nice homo sapiens supremacy narrative but the truth is much more complicated. we existed at least 4/5 thousand years together and a lot of mixing and mingling has been going on.
This sounds strikingly similar to how I imagine our first encounters with advanced AI will unfold.
Of course, Santa here is just a toy company.
Actually toy companies never existed before about 1700 so we don't have anything to worry from the current ones.
This is just about as incoherent as this entire comment section. I thought I'd play along.
So working on a solution to this problem makes sense - even if we spend billions solving this issue, it's not lost, because at a point it will be needed.
As for AI becoming an issue, it's a sci-fi idea, nothing more. It's much to vague - how is it reliably becoming dangerous? What are we actually looking for as signs of danger?
Or even - if we fear AI's being tools of radicalisation, I think it's pretty far away from the skynet apocalypse most people are thinking of.
Not writing stuff like the unabomber manifesto to get people to be terrorists.
If you're afraid of radicalisation, I'd argue that the US governement is much more dangerous and is just as much a black box with no one at the wheel.
The AI system would be bottlenecked by data in the sense that it will have to run experiments, but it's not clear that it needs a new paradigm to resolve this bottleneck. It just has to propose an experiment and interpret its results. So it writes code, and the experimental outcome gets fed back into the model as any other normal input. As an AI researcher, I'd like to believe that this is not going to happen anytime soon, but I'm not sure we're far.
I see what you did there.
What exactly do you think is the pattern is that humans can recognize, in the training data that would be required to do "AI Research"?
3.5: As an AI language model, I cannot guess, but I can provide some general guidelines based on current research and best practices.
If we want to improve the results of Large Language Models (LLMs), one aspect of the architecture that we could focus on is increasing the model's capacity to learn and retain more information. This could be achieved by increasing the number of parameters in the model or using more sophisticated architectures such as transformer-based models that use self-attention mechanisms to capture long-range dependencies in the input sequence.
Another important aspect to focus on is improving the model's ability to handle rare and out-of-vocabulary (OOV) words. This can be achieved by using subword-level tokenization, which breaks down words into smaller units and enables the model to generalize better to new or unseen words.
We could also focus on improving the training process by using larger and more diverse training datasets, regularization techniques to prevent overfitting, and optimizing hyperparameters such as learning rate, batch size, and number of training epochs.
Finally, we could also focus on incorporating external knowledge sources such as structured data, knowledge graphs, or ontologies into the model architecture to enhance its ability to reason and make more accurate predictions.
Overall, there are many aspects of the LLM architecture that can be improved to enhance its performance, and the choice of which to focus on will depend on the specific task and the available resources.
Like the advent of Transformers, some smart dev could change how LLM's think. Self improvement could be built in as an optimization process. And if we don't "know" what might work, a platform could "guess" and try billions of combinations of possible improvements.
The only reason we have things like ChatGPT today is because of the surprise development of the transformer. Nobody understands how intelligence fundamentally works, so we literally have no ability to predict when the next groundbreaking architectural development will occur.
And self-improvement concerns are mostly orthogonal to the main risk of AI: the development of AGI. Even a mediocre AGI that is only "decent" at tasks, can be weaponized as a collective superintelligence.
https://arxiv.org/abs/1706.03762
Why was this revolutionary though?
Attention is all you need paper just proposed an AR model that didn’t have to be trained step by step. The scaling happened later in BERT and GPT and OpenAI’s scaling work
* "Attention is all you need" introduced positional encoding which allows you to keep context of the word, allowing for more complex translation (and thus generative/chatgpt like tasks?) because words now have context relative to each other. Contrast this with "bag of words" models that only tells you whether the word is present or not.
* I don't quite understand why but transformers (which "AiaYN" introduced) can be made fully parallel, compared with the RNN/LSTM networks which has to be serial per token. Fully parallel allows for GPU optimization, which means you can take advantage of Moore's law for training.
I'm always a bit suspicious when people claim a breakthrough of this sort. There's no doubt that better algorithms give better results but how much is due to just faster computers, cheaper compute, memory, etc.
[0] https://youtu.be/S27pHKBEp30
That said, this doesn't really seem all that comparable. The article points out very fundamental properties of all the diverse current approaches: They are tightly data constrained. You either need to cheap simulation or massive real world data. That's not an arcane technical point.
> conveniently sidestep[s] the possibility of the development of new tools and techniques.
I don't believe they did this at all. Here is their summary of their third point:
> To automatically construct a good dataset, we require an actionable understanding of which datapoints are important for learning. This turns out to be incredibly difficult. The field has, thus far, completely failed to make progress on this problem, despite expending significant effort. Cracking it would be a field-changing breakthrough, comparable to transitioning from alchemy to chemistry.
That does not read to me as conveniently sidestepping the possibility of new tools. Rather, it is acknowledges that to overcome the problem we NEED a new tool, aka a breakthrough.
I also disagree with the framing of the third section as:
> And that entire section basically boils down to "I personally believe this is hard and will take a long time."
They provide both theoretical and empirical evidence of their claim. I find it was well argued, and I'm inclined to agree. Every scenario I can think of with a runaway superintelligence requires a way to automatically improve datasets as part of the learning process.
"Also, they aren’t getting more-solved over time: we’ve made little-to-no progress on any problem of this sort in the last decade, certainly not the reliable improvements of the sort we’ve seen from supervised learning. This indicates that a breakthrough is needed — and that it is unlikely to be close."
Past lack-of-progress is not an indicator of future lack-of-progress. "unlikely to be close" is unknowable, and is just the author's gut feeling.
Past lack-of-progress is not proof of future lack-of-progress. But it's most definitely an indicator.
> "unlikely to be close" is unknowable, and is just the author's gut feeling.
Again I'm sorry but I have to disagree with you here. The very next paragraph reads:
> Something that would change my mind on this is if I saw real progress on any problem that is as hard as understanding generalization, e.g. if we were able to train large networks without adversarial examples.
Basically, the author has identified a class of problem. They are claiming that little to no progress has been made on any problem in that class. So it's not just that no progress has been made on this specific problem, but that we are stuck on all problems of this type. Thinking of it in that way, I do not find it unreasonable to say we are "unlikely to be close" to a solution.
If you have a CS background, I'll make an analogy to computability classes: It's like the author is saying this is an NP hard problem. We have made no progress on any NP hard problem. I think it's reasonable to say that we are unlikely to be close to solving a particular NP hard problem because we've made no progress on any NP hard problem.
You're free to disagree with the author's premises. E.g. "this problem is not like those other problems" or "we actually have made progress on those other problems". But I think that the conclusions are reasonable given the premises.
Your critique is: has the author not considered magic?
Recursive self-improvement is not a necessary condition for a fast takeoff. A strongly superhuman AI could emerge at training time. AlphaGo Zero went from subhuman to superhuman performance within a few hours. The same could happen with a general AI. A training run of a few weeks or even months would still be considered fast.
Funny how Big Data and online surveillance suddenly became huge the last decade-ish...
the hypothesis is a pattern recognition exercise in itself (i.e., looking at a number of documented occurences around algorithmic development and extrapolating), which I think neatly highlights the fundamental difference between human pattern recognition and machine pattern recognition
AGI is a flying car mythology.
Narrow AI is improving and the rate is accelerating. The current speed (1st derivative) is unimportant, but there is a differential equation-like feedback function to it and it will accelerate (2nd derivative): speed proportional to the amount present.
People and technology form a coupled system that is somewhat a super-organism. That cycle is amplifying in a way that is not necessarily obvious (it doesn't happen in public) or continuous (or discrete for that matter). For the foreseeable future, technology does not yet constitute a self-assembling or self-modifying closed system. It is still entirely dependent on human effort. What is happening is that people are relying on and exploiting the efforts of AI that had not happened previously. This is one change, more visible than most, that is leading to more efficiencies, more changes, and more technological advancement. Fully automated factories, software development, systems engineering, and chip design may not happen anytime soon but they are both possible and probable due to the forcing function and Tragedy of the Commons of human greed.
Active learning would be useful at creating a superhuman AGI, but I don't see it as a requirement. As soon as AIs can iterate on future models and produce increasingly useful training data for future models, then models would likely continue to grow more capable at an exponential rate.
But I'm not even convinced active learning is that difficult once you have fairly capable AI programmers...
My AI professor once said, "all problems are search problems". And this suck with me because the world looks different when you start seeing all problems as just iterations of different optimisation problems. So long as you have a good heuristic and enough compute, eventually a suitable solution for any problem will be found. This is what makes generalised learning algorithms so powerful.
The author states that, "active learning with neural networks are uniformly terrible", but this just reeks to me of a search problem in need of optimisation. So as long as we can tell GPT5, "I want you to iteratively write code which improve on existing state-of-the-art active learning algorithms", then eventually you'll get there... GPT5 like humans might not get very far in the beginning, but assuming it can iterate faster than humans and future iterations faster still, then breakthroughs will eventually be made and progress should accelerate exponentially.
Of course, this also assumes that humans don't get there first. And honestly given the amount of investment in AI and rate of progress in AI right now I just wouldn't bet against a significant advancement in active learning coming soon.
From an external perspective it's as if the AI is self-improving by leveraging human agents to assist the process. Not to say there is any 'intent' behind it, but rather to view it as an emergent phenomenon.
Wrong. Full self driving is still not here despite access to a huge amount of high quality data.
The truth is we don’t fully understand what it means to be intelligent, and how intelligence scales. All indications are that intelligence comes with diminishing returns, in that to achieve small increases in intelligence requires large increases in connectivity. Even if we can design an intelligence that is generally smarter than us, and capable of building even smarter ai on its own, it’s intelligence might increase exponentially with each step, but each step could easily take increasingly longer amounts of time to achieve, leading to linear or even sub linear increases in capabilities over time. Resources are limited, hardware is limited, energy is limited. There are all sorts of constraints that are likely to play a limiting factor in runaway intelligence.
This doesn’t mean that these intelligences won’t be dangerous. Even an average human can create a lot of trouble in this world. It does however mean that a singular AI is unlikely to just run away with things.