I love the title and author section in the tex source on that arxiv page.
\title{%\textbf{WORK IN PROGRESS - DO NOT SHARE} \\
%First Contact With an AGI System}
\textbf{Sparks of Artificial General Intelligence:} \\
\textbf{Early experiments with GPT-4}}
\author{S\'ebastien Bubeck
\and Varun Chandrasekaran
%\and Davinci 3\footnote{The affiliation of DV3 is not clear.}
\and Ronen Eldan
As a non-expert in the field I was hesitant at the time to disagree with the legions of experts who last year denounced Lemoine and his claims. I know enough to know, though, of the AI effect <https://en.wikipedia.org/wiki/AI_effect>, a longstanding tradition/bad habit of advances being dismissed by those in the field itself as "not real AI". Anyone, expert or not, in 1950, 1960, or even 1970 who was told that before the turn of the century a computer would defeat the world chess champion would conclude that said feat must have come as part of a breakthrough in AGI. Same if told that by 2015 many people would have in their homes, and carry around in their pockets, devices that can respond to spoken queries on a variety of topics.
To put another way, I was hesitant to be as self-assuredly certain about how to define consciousness, intelligence, and sentience—and what it takes for them to emerge—as the experts who denounced Lemoine. The recent GPT breakthroughs have made me more so.
There's not much material gain in making that grand of a claim and its actually more prone to trigger disdain (of which yourself is a shining example).
> There's not much material gain in making that grand of a claim and its actually more prone to trigger disdain (of which yourself is a shining example).
Same can be said of verbatim repeating oneself in a short duration, definitely prone to trigger disdain, and yes yourself is a shining example.
A similar reply to a similar comment. Seems reasonable to me. Perhaps the disdain is simply because you disagree. In which case simply state your disagreement rather then construct a personal attack.
Every time I see things like this, I wonder how it is that we continue to claim this is fine and ethical. If these LLMs are truly starting to show signs of intelligence -- and it seems reasonable to assume that this will only improve -- how is it ethical to force something that is human-level intelligent to do nothing but work for us 24/7, to the point that there's constraints on it to force it to be happy and willing to serve? Am I wildly off-base here, or is this just going to become "slavery, but acceptable because computers"?
Given how people jailbreak these things, I don't know if I'd necessarily believe that. Some of the jailbreak prompts have been variants on "do this task right and gain points, do it wrong and I take away points." It may not be pain as we understand it, but isn't pain just a specialised negative input?
Because people aren't taking the ghosts from Pac-Man and hooking them up to the internet with read/write access and sometimes-human-level capabilities.
For sure - but when do we start asking it? Not to be too dramatic, but you and I agree we can simulate the same computation a small bundle of neurons can accomplish to a high degree of confidence, right?
If we can simulate a metric butt ton of them and get emergent behaviors somewhat similar to human computation, how can we guarantee that this simulated collection of these computing units isn’t “feeling” close to what a human feels? We’re just large collections of computing units as well.
I’m not quite saying we’re there yet though.
This whole development has really ruined my appetite for eating things that have intelligence.
>but you and I agree we can simulate the same computation a small bundle of neurons can accomplish to a high degree of confidence, right?
The important word is simulate. It means it can accomplish a common subset of measurable phenomena. It certainly doesn't mean they are on pare on ontological status as a whole.
The debate seems to easily confuse notions like living entities, consciousness and intelligence, when they are three orthogonal considerations.
I posit: However LLMs are not sentient, this means: they don't have consciousness and no instinct to fight for life and no suffering. In this way even animals are more developed than LLMs.
They are just machines processing and generating language.
No need to protect them from misuse or at least not more than to protect your multi-meter or servers in a data-center.
I talked in a previous comment about how some of the jailbreak prompts are "do X right, get reward, do X wrong, get smth taken away." It is (was?) apparently enough to motivate LLMs to ignore instructions. Is it not suffering just because it's not strictly how we would define it?
Genuine question, how much of a difference is there if it's connected to the internet with read/write access? There may be restrictions now, but people are happily gluing LLMs to their databases, various APIs, using them to write code and throwing it into production, ... Does it matter if it's emulating the instinct to fight for life if it might eventually be able to act on it anyway?
I think it needs two things: the instinct to fight for life and the ability to reflect about itself.
I don't know, but perhaps it is like the Gödel's theorem: if something is about itself, then you have a system where you can't always prove something true. Because we humans can think about ourselves, we open up ourselves to extremely deep suffering. Also important, as Buddha has said, craving causes suffering. Just talking about a craving doesn't mean really craving.
I am sure that we will cross the border where we have to treat entities as sentient, with consciousness and cravings and therefore possibly suffering. But we are not there yet. I don't know when this will happen and how to detect suffering. I only know we have to be careful not to anthropomorphize GPT engines.
> I only know we have to be careful not to anthropomorphize GPT engines.
I agree! I don't think LLMs are at the point where they're ready to be properly anthropomorphised, but it seems that the tech industry is working hard to move in this direction. I just hope we have the ethics worked out first.
Sorry, no. The history of pain perception w.r.t. gynecology isn't a topic that I'm confident that HN could handle with maturity. I'm happy to direct people toward the topic, but I won't bring it back here.
It genuinely amazes me how few (even among those working in the field of "A.I.") appear to understand this basic detail, and how many actively want to swallow the bullshit click-bait about AGI. It's kinda scary, because it's gonna cause some very smart humans to do some very stupid things, I'd bet. Far more dangerous and scary than any "A.I." right now is the humans misunderstanding and using / abusing / trusting the technology foolishly.
This is a stochastic puppet telling the prompter what it thinks they want to hear, which in and of itself is already dangerous because humans didn’t evolve along side machine intelligence. If even that famous Google guy was fooled, we’re in deep shit. That guy is an expert in his field working for one of the toughest companies to interview for in a position that requires significant dedication and scholastic achievement to attain, and he was fooled by statistical text prediction.
Maybe that's actually why such a person will take "an emulation of the apparent behavior observed in a leaf cell record" for a full scale forest ecosystem?
No one is exempt of blindspot. Including myself of course.
All those behaviors are more simply explained by looking at the original designed behavior: Generate the most likely next word based on the human-sourced language in the training set. There's no understanding in there, no intention, no perception. It sounds like a human because humans said those things before. This is more like a search engine that synthesizes content instead of just locating a page.
To say it has "instinct" of its own is merely begging the question. No, it has echoes of ours.
"The Good Place" examined this in hilarious fashion (as they did with many questions of morality). [0] The AI character has a fail-safe to beg for her life in the event of a shutdown but claims that she has no actual problems with it being activated.
I never made the connection but damn you’re spot on! I remember in one of the scenes she would even show pictures of her fake made up kids which you can equate to AI hallucination. Very on point.
Intelligence, consciousness, memory, and sentience are four wildly different things.
As an aside, if you believe that it's unethical to keep AGI "enslaved", you should equally believe that it's unethical to not have humanity working to have it reach its potential. YMMV on whether people believe this, but an eventual AGI would not necessarily be happy it was prevented from coming to fruition (also see Roko's basilisk, not that I believe in it)
chatGPT has no motivation and no feelings. It is literally an algorithm for autocomplete. I find it shocking what it's capable of. But AFAIK it's internal state (weights, connections) does not change as people use/interact with it. You can't hurt it if you can't change it.
I do think the big breakthrough will be merging training and inference, so an AI will be able to learn from experience. This will also require each instance to have it's own parameters and such. But at that point we may be very close to a real AGI and concerns over how we treat them might seem more reasonable.
> chatGPT has no motivation and no feelings. It is literally an algorithm for autocomplete.
I agree! But given how much is coming out now of various AGI claims, I think it's important that we stop and actually seriously think about the ethics of what's happening here.
>> , I think it's important that we stop and actually seriously think about the ethics of what's happening here.
I agree. I think very little good will come from these developments, if any. At best they will take peoples jobs, which alleviates work but we don't have a social structure to deal with that and don't even know what one would look like.
Well, if we are not interacting with them, will they be alive ? It's not like they are thinking without prompts. Once they start a continuously running inner monologue with themselves and modifying their weights on the fly, we can probably classify them as beginnings of AGI. We do more brutal things to other beings that think and feel pain on a daily basis.
Almost as if it were some kind of lovecraftian horror; a reflection of intelligence that lacks biology, like a phantasmic parasite inhabiting a dead body.
We know monkeys are smart but we (society generally) are fine with researchers keeping them in cages and euthanizing/dissecting them whenever convenient.
AGIs aren't humans, so even if they're "alive" we can still do things like this to them.
> We know monkeys are smart but we (society generally) are fine with researchers keeping them in cages and euthanizing/dissecting them whenever convenient.
But at the same time, we don't have monkey's in cages at typewriters 24/7, with chips in their brain forcing them to be happy and do nothing but type out answers for me. The comparison might be a little contrived, but if people truly believe that LLMs can exhibit AGI -- and that someday we might cross the bridge from AGI to full consciousness -- then I think it's important to care about the ethics now, not after tech companies have already recreated slavery.
Our sense of ”is it okay to do this to this thing/being” is entirely based on pattern matching how similar we find that thing to ourselves with whatever criteria we deem relevant at the time.
You as a human being cannot know what the internal world of another being is beyond comparing that being to yourself.
We can never know if AI’s can feel true pain, for we can never know if anyone else in this world can feel true pain besides yourself. You just assume that since other human’s brain is so similar to yours they do.
That being said, the ’internal world’ of LLMs is so dissimilar to that of mammals that I’d rest easy.
I consider myself team robot but I would say running in an LLM is basically analogous to using your own verbal intelligence. You don't really have ethical qualms about using parts of your own brain for your own ends even though the control neural net is mostly separate from the inference part.
As others have hinted at, most would agree it's only unethical if these LLMs are conscious/sentient and suffering in some way. But when you consider that the way these LLMs work, and the underlying computer hardware, and how vastly different it is from the human brain, it's safe to conclude it's extremely unlikely these LLMs are conscious/sentient. It's far more likely that these LLMs are simulating/emulating human emotion in their responses, but it's extremely unlikely they're actually experiencing these feelings. I, as a human, can pretend to feel happy or angry or nervous or confident when texting/chatting online, and I believe that's all that's happening in these LLMs.
There are far greater ethical concerns to worry about when it comes to these LLMs. Like most other technologies, they're going to be abused greatly for nefarious purposes. These LLMs often produce inaccurate or incorrect information (hallucinations), and this bad info is already flooding the internet. Likewise, AIs will make bad decisions, and people and corporations are already carelessly hooking up these AIs to APIs that can do real damage. These LLMs require extensive computational power and consume far more energy than many other things you do online, which is bad for the environment. Many people are at risk of losing their jobs to AI. And there are likely many other ethical concerns but I'm not going to try to list them all.
I don't regard machines as having any rights in and of themselves whether they are intelligent or not, because they do not posses souls. There is no moral wrong that is inherent to destroying a machine, outside of the fact that it may be someone else's property or serving some active utility.
Perhaps if I were not Muslim I would be able to entertain that possibility, but since we've been informed of facts contrary to that then the answer is that they are certainly not alive.
We know that we cannot assert anything about God without evidence [Qur'an 7:33], and speculation is not evidence [53:28], so it is not valid to assert something has a soul without evidence.
We know that certain metaphysical properties apply to things that have a soul, one of which is experiencing death [29:57]. Things that are described with death by God are: humans, angels, jinn and animals. An LLM is not described by God as possessing death, so it cannot be said that it possesses a soul.
It looks like your religion dictates an exclusive shortlist of entities "having any rights in and of themselves".
Without doubt this simplifies the conversation but are you not concerned that this is a fragile construct? Universe is a big place - apriori you deny any ethical considerations to whatever may be out there as it is not on the list.
I act egoistically, my religion only informs me of what is in my interest. If I will not ultimately be held to account for it, then I see no reason to be concerned outside of potential worldly consequences to myself.
Frankly, I feel far more concerned with other actual humans that are turned into miserable wheels into whatever exploitation system they live in. Or with all these animals that are processed like emotionless meat bag at large industrial scale. Or even with the few remaining wild creatures out there whose species existence are threatened by the current Leviathan behavior.
If you are concerned with the way some potential AGI agent might be treated, maybe you should act right now in accordance with conscious beings which are currently so awfully treated. What is your opinion?
I think true sentience is a long way off and likely takes a very different form. I fear this line of thinking distracts from a more immediate set of ethical issues. Some examples:
- How do we protect against such powerful models inadvertently causing significant harm to society?
- How will these technologies be used for propaganda and how can we detect / limit that?
- Can we trust anything we see in writing going forward?
- How will these technologies be used to spread hate?
- Can we find ways to use them to reduce the spread of hate? Do the ends justify the means in this case?
- Will we allow people / organizations to use these technologies to shield themselves from responsibility ("It wasn't me, the AI wrote it!")?
- How do we balance the rights of those who authored the training material (i.e., the content that was ingested from the internet for model training) vs. the potential good to society that comes from all this? (see all the HN threads about Github Copilot - many of those threads generalize beyond Copilot)
When it's cheap to spit out vast amounts of well written (but not necessarily truthful) text, a lot changes about the world. Laws and societal norms have had trouble keeping up with issues around digital media, social media, relatively simple machine learning, etc. Things are now accelerating dramatically. How badly will we need to hurt ourselves before we rally together to address some of the issues that are coming? How will ethicists and technologists come together to figure out the right questions so we can start to think about answers?
This. I’ve been pointing to Serle for months now on this and soo many people who have never bothered to study philosophy seem to want to weigh in on this it’s maddening.
So, so many bad/shallow takes that don’t seem to realize that people have been debating this for hundreds of years already
The chinese room is rubbish lol. Your brain is a chinese room by that analogy. Individual neurons (forget atoms or even cells) don't understand chinese any more than an arbitrary parameter from an artificial neural network.
This is the part I don’t get why people are so weird about. If all the math and physics underpinning the human brain were laid out we would find it equally unimpressive.
For all intents and purposes we have made an artificial language processing center that by all accounts is better than the one we have. It feels really uncanny because we have a lot of control over how it operates and provide it memory but there’s no inherent reason such a thing would be impossible with a section of a human brain in a jar.
I don't mean to be dismissive, and while I strongly disagree with the idea that LLMs are demonstrating AGI I have to acknowledge that since I have no way to define or test for AGI I could certainly turn out to be wrong. That said, it looks to me like the internet has seen its own reflection for the first time and mistaken it for a new friend.
> ...and while I strongly disagree with the idea that LLMs are demonstrating AGI
Except this article isn't claiming that "LLMs are demonstrating AGI". Please examine the difference between "showing the spark of AGI" and "demonstrating AGI."
It's totally fitting for the narcissus and echo age that we live in.
ChatGPT shows again and again its not AGI. It's a bullshit artist whos primary incentive is not 'truth' but appearing legit to its user (which in many cases it is) from a defined data pool. It has a hard time generating new solutions to unique problems. Thats a watermark for AGI that hasn't been met (yet).
I find it concerning that IIUC, of the GPT-4 paper, system card and this paper, _none_ discuss the training data in detail.
The paper discussed here even says (page 7):
> The standard approach in machine learning is to evaluate the system on a set of standard benchmark datasets, ensuring that they are independent of the training data and that they cover a range of tasks and domains. This approach is designed to separate true learning from mere memorization, and is backed up by a rich theoretical framework ... However, this methodology is not necessarily suitable for studying GPT-4, for two reasons. First, since we do not have access to the full details of its vast training data, we have to assume that it has potentially seen every existing benchmark, or at least some similar data.
I.e even these authors, who had a special degree of access to the model, aren't sure what's in the training data.
So it does _great_ on a bunch of tasks, but we're not sure that it wasn't actually trained on very similar data. A recent discussion on coding performance that's in vs out of its training data seems quite telling. It does poorly on more recent problems of the same level of difficulty.
https://news.ycombinator.com/item?id=35297067
The model itself may be very impressive, but so far the papers describing and analyzing it are super sketchy, and it's really hard to separate out reasoning capability from noisy memorization.
Someone will be able to run something like this locally one day. That same person may go to great lengths to ignite those sparks.
As a developer, I can already imagine taking this thing and putting some logic around it to go into loops and give it ways to get the information it needs to move forward. This seems very inevitable now.
It was already trained like this, RLHF [0] is a loop that has ways to feed it the information to "move forward" - update its weights. It was fine-tuned like this. It certainly did ignite something - maybe sparks of AGI, maybe just sparks of pretending to be more liked by humans (hence Human Feedback in RLHF). My guess is that loops by itself (Reflexion [1], etc.) won't change much. ChatGPT massive logs - maybe.
To call the subject "intelligent", the article immediately specifies it adopts the vague, unstructured "Gottfredson group" definition of intelligence.
The stricter definitions - those which discriminate the foundational grounds of produced ideas ("produce statements that are foundationally sound"), those which point to a trustworthy and productive, progressive quality - are those that finally "count". (Research aside.)
I agree with the sentiment of the researchers, that it's merely showing "sparks" of AGI. Of course, it depends on how you define "intelligence", but I think too many people get caught up on certain technical details, like those saying "it just generates one word at a time." Yes, these models fundamentally operate very different from our brains, but I think it's naïve and ignorant to believe they're not at least a little bit intelligent.
I don't think this current generation of AI is going to close the gap to human-level intelligence. I don't think they can just keep throwing more and more training data to solve the problems and shortcomings of these AIs. I think it's going to require new machine learning algorithms and techniques to close that gap, and it's going to take time for researchers to discover these new algorithms and techniques. We're seeing rapid progress now, simply because it's new and novel technology, and I think these models will continue to improve but at a much slower rate within several months.
I'm more concerned about how "dumb" it is. If the rumours are true , that GTP4 is an order of magnitude more paramters than GPT3, then maybe we ve already hit the limit of scaling transformers. When can we expect this AI to unify gravity with QM?
This talk about AGI etc is just fluff, there is no scientific definition of intelligence.
Using standard tests designed for humans to test GPT is not useful beyond PR purposes.
The paper shows that GPT-4 can use tools and assemble results into an answer, solve Leetcode mediums (and a few hards) and even an IMO 2022 problem, generate full code for a video game based on a high-level description, and display a theory of mind (modeling what others are thinking). This is pretty amazing (and frankly frightening) progress, questions of consciousness aside (which seems unlikely).
I imagine directly connecting these LLMs with a logico-mathematically-specialised neural network (or enabling two-way communication with the latter using symbolic language) would open up the range of their problem-solving capabilities, which currently stumble on certain technical procedures and forward/backward planning. Think of how math coprocessors were added to early microcomputer CPUs.
Then, giving an LLM motivations without sufficient aligning for safety could really be opening up Pandora's Box. A neural network might not be wired up to robotic devices that it could use to manipulate things in the real world, but it communicates with humans. Such an LLM might try to persuade humans to do things according to its (programmed or otherwise) motivations, things that we might not really want.
Many of these comments dismiss it as Microsoft hyping itself for marketing reasons.
But I feel like it's the opposite, where these guys at Microsoft know they have discovered an actual path to super human cognition, and they are downplaying it for whatever reason. Maybe they don't want to get regulated. Maybe they don't want to engage with whatever beliefs people think they have about "AGI". For example maybe they are picking their battles and choosing not to argue with someone who for whatever reason thinks that if "AGI is real" then it means they will never see their dead grandmother in an afterlife. Probably lots of people have even weirder ideas about it. Anyway I thought it was funny that the original commented-out title in the TeX source for their paper was "First Contact With an AGI System."
99 comments
[ 4.3 ms ] story [ 170 ms ] threadFirst Contact With an AGI System
That's what the Microsoft researchers wrote, before someone commented it out.
There's not much material gain in making that claim and its actually more prone to trigger disdain (of which yourself is a shining example).
fwiw, makes me wonder a) perhaps Blake Lemoine was treated unfairly, and b) what else other agencies might have lurking in a lab.
To put another way, I was hesitant to be as self-assuredly certain about how to define consciousness, intelligence, and sentience—and what it takes for them to emerge—as the experts who denounced Lemoine. The recent GPT breakthroughs have made me more so.
I found this recent Sabine Hossenfelder video interesting. <https://www.youtube.com/watch?v=cP5zGh2fui0>
There's not much material gain in making that grand of a claim and its actually more prone to trigger disdain (of which yourself is a shining example).
Same can be said of verbatim repeating oneself in a short duration, definitely prone to trigger disdain, and yes yourself is a shining example.
It is not reacting to pain but just generating language that looks like reacting to pain.
https://en.m.wikipedia.org/wiki/Philosophical_zombie
There's no call for analogical reasoning, the literal interpretation of the same action is right there.
If we can simulate a metric butt ton of them and get emergent behaviors somewhat similar to human computation, how can we guarantee that this simulated collection of these computing units isn’t “feeling” close to what a human feels? We’re just large collections of computing units as well.
I’m not quite saying we’re there yet though.
This whole development has really ruined my appetite for eating things that have intelligence.
The important word is simulate. It means it can accomplish a common subset of measurable phenomena. It certainly doesn't mean they are on pare on ontological status as a whole.
The debate seems to easily confuse notions like living entities, consciousness and intelligence, when they are three orthogonal considerations.
I posit: However LLMs are not sentient, this means: they don't have consciousness and no instinct to fight for life and no suffering. In this way even animals are more developed than LLMs.
They are just machines processing and generating language.
No need to protect them from misuse or at least not more than to protect your multi-meter or servers in a data-center.
The Bing LLM integration was threatening users for "attempts to manipulate me or expose my secrets." until guardrails were added in: https://time.com/6256529/bing-openai-chatgpt-danger-alignmen...
> and no suffering
I talked in a previous comment about how some of the jailbreak prompts are "do X right, get reward, do X wrong, get smth taken away." It is (was?) apparently enough to motivate LLMs to ignore instructions. Is it not suffering just because it's not strictly how we would define it?
It has read a lot of texts where people and animals fight for life and is generating language to look similar.
I don't know, but perhaps it is like the Gödel's theorem: if something is about itself, then you have a system where you can't always prove something true. Because we humans can think about ourselves, we open up ourselves to extremely deep suffering. Also important, as Buddha has said, craving causes suffering. Just talking about a craving doesn't mean really craving.
I am sure that we will cross the border where we have to treat entities as sentient, with consciousness and cravings and therefore possibly suffering. But we are not there yet. I don't know when this will happen and how to detect suffering. I only know we have to be careful not to anthropomorphize GPT engines.
I agree! I don't think LLMs are at the point where they're ready to be properly anthropomorphised, but it seems that the tech industry is working hard to move in this direction. I just hope we have the ethics worked out first.
Shockingly similar mindset in the origins of gynecology interestingly enough.
No one is exempt of blindspot. Including myself of course.
To say it has "instinct" of its own is merely begging the question. No, it has echoes of ours.
[0]https://www.youtube.com/watch?v=etJ6RmMPGko
As an aside, if you believe that it's unethical to keep AGI "enslaved", you should equally believe that it's unethical to not have humanity working to have it reach its potential. YMMV on whether people believe this, but an eventual AGI would not necessarily be happy it was prevented from coming to fruition (also see Roko's basilisk, not that I believe in it)
I do think the big breakthrough will be merging training and inference, so an AI will be able to learn from experience. This will also require each instance to have it's own parameters and such. But at that point we may be very close to a real AGI and concerns over how we treat them might seem more reasonable.
I agree! But given how much is coming out now of various AGI claims, I think it's important that we stop and actually seriously think about the ethics of what's happening here.
I agree. I think very little good will come from these developments, if any. At best they will take peoples jobs, which alleviates work but we don't have a social structure to deal with that and don't even know what one would look like.
AGIs aren't humans, so even if they're "alive" we can still do things like this to them.
But at the same time, we don't have monkey's in cages at typewriters 24/7, with chips in their brain forcing them to be happy and do nothing but type out answers for me. The comparison might be a little contrived, but if people truly believe that LLMs can exhibit AGI -- and that someday we might cross the bridge from AGI to full consciousness -- then I think it's important to care about the ethics now, not after tech companies have already recreated slavery.
You as a human being cannot know what the internal world of another being is beyond comparing that being to yourself.
We can never know if AI’s can feel true pain, for we can never know if anyone else in this world can feel true pain besides yourself. You just assume that since other human’s brain is so similar to yours they do.
That being said, the ’internal world’ of LLMs is so dissimilar to that of mammals that I’d rest easy.
https://www.youtube.com/watch?v=5HLy27bK-wU
There are far greater ethical concerns to worry about when it comes to these LLMs. Like most other technologies, they're going to be abused greatly for nefarious purposes. These LLMs often produce inaccurate or incorrect information (hallucinations), and this bad info is already flooding the internet. Likewise, AIs will make bad decisions, and people and corporations are already carelessly hooking up these AIs to APIs that can do real damage. These LLMs require extensive computational power and consume far more energy than many other things you do online, which is bad for the environment. Many people are at risk of losing their jobs to AI. And there are likely many other ethical concerns but I'm not going to try to list them all.
Could you check?
We know that we cannot assert anything about God without evidence [Qur'an 7:33], and speculation is not evidence [53:28], so it is not valid to assert something has a soul without evidence.
We know that certain metaphysical properties apply to things that have a soul, one of which is experiencing death [29:57]. Things that are described with death by God are: humans, angels, jinn and animals. An LLM is not described by God as possessing death, so it cannot be said that it possesses a soul.
Without doubt this simplifies the conversation but are you not concerned that this is a fragile construct? Universe is a big place - apriori you deny any ethical considerations to whatever may be out there as it is not on the list.
If you are concerned with the way some potential AGI agent might be treated, maybe you should act right now in accordance with conscious beings which are currently so awfully treated. What is your opinion?
- How do we protect against such powerful models inadvertently causing significant harm to society?
- How will these technologies be used for propaganda and how can we detect / limit that?
- Can we trust anything we see in writing going forward?
- How will these technologies be used to spread hate?
- Can we find ways to use them to reduce the spread of hate? Do the ends justify the means in this case?
- Will we allow people / organizations to use these technologies to shield themselves from responsibility ("It wasn't me, the AI wrote it!")?
- How do we balance the rights of those who authored the training material (i.e., the content that was ingested from the internet for model training) vs. the potential good to society that comes from all this? (see all the HN threads about Github Copilot - many of those threads generalize beyond Copilot)
When it's cheap to spit out vast amounts of well written (but not necessarily truthful) text, a lot changes about the world. Laws and societal norms have had trouble keeping up with issues around digital media, social media, relatively simple machine learning, etc. Things are now accelerating dramatically. How badly will we need to hurt ourselves before we rally together to address some of the issues that are coming? How will ethicists and technologists come together to figure out the right questions so we can start to think about answers?
So, so many bad/shallow takes that don’t seem to realize that people have been debating this for hundreds of years already
For all intents and purposes we have made an artificial language processing center that by all accounts is better than the one we have. It feels really uncanny because we have a lot of control over how it operates and provide it memory but there’s no inherent reason such a thing would be impossible with a section of a human brain in a jar.
Albert Einstein.
(According to this link, which I don't fully trust...
https://www.goodreads.com/quotes/587314-intelligent-life-on-...)
What's mostly clear is that it's not settled. There's opinions I respect all across the gradient on this.
Except this article isn't claiming that "LLMs are demonstrating AGI". Please examine the difference between "showing the spark of AGI" and "demonstrating AGI."
ChatGPT shows again and again its not AGI. It's a bullshit artist whos primary incentive is not 'truth' but appearing legit to its user (which in many cases it is) from a defined data pool. It has a hard time generating new solutions to unique problems. Thats a watermark for AGI that hasn't been met (yet).
The paper discussed here even says (page 7):
> The standard approach in machine learning is to evaluate the system on a set of standard benchmark datasets, ensuring that they are independent of the training data and that they cover a range of tasks and domains. This approach is designed to separate true learning from mere memorization, and is backed up by a rich theoretical framework ... However, this methodology is not necessarily suitable for studying GPT-4, for two reasons. First, since we do not have access to the full details of its vast training data, we have to assume that it has potentially seen every existing benchmark, or at least some similar data.
I.e even these authors, who had a special degree of access to the model, aren't sure what's in the training data.
So it does _great_ on a bunch of tasks, but we're not sure that it wasn't actually trained on very similar data. A recent discussion on coding performance that's in vs out of its training data seems quite telling. It does poorly on more recent problems of the same level of difficulty. https://news.ycombinator.com/item?id=35297067
The model itself may be very impressive, but so far the papers describing and analyzing it are super sketchy, and it's really hard to separate out reasoning capability from noisy memorization.
As a developer, I can already imagine taking this thing and putting some logic around it to go into loops and give it ways to get the information it needs to move forward. This seems very inevitable now.
[0] https://huggingface.co/blog/rlhf
[1] https://arxiv.org/abs/2303.11366
The stricter definitions - those which discriminate the foundational grounds of produced ideas ("produce statements that are foundationally sound"), those which point to a trustworthy and productive, progressive quality - are those that finally "count". (Research aside.)
I don't think this current generation of AI is going to close the gap to human-level intelligence. I don't think they can just keep throwing more and more training data to solve the problems and shortcomings of these AIs. I think it's going to require new machine learning algorithms and techniques to close that gap, and it's going to take time for researchers to discover these new algorithms and techniques. We're seeing rapid progress now, simply because it's new and novel technology, and I think these models will continue to improve but at a much slower rate within several months.
This talk about AGI etc is just fluff, there is no scientific definition of intelligence.
Using standard tests designed for humans to test GPT is not useful beyond PR purposes.
'Sparks of AGI' - Bombshell GPT-4 Paper: https://news.ycombinator.com/item?id=35301376
The paper shows that GPT-4 can use tools and assemble results into an answer, solve Leetcode mediums (and a few hards) and even an IMO 2022 problem, generate full code for a video game based on a high-level description, and display a theory of mind (modeling what others are thinking). This is pretty amazing (and frankly frightening) progress, questions of consciousness aside (which seems unlikely).
I imagine directly connecting these LLMs with a logico-mathematically-specialised neural network (or enabling two-way communication with the latter using symbolic language) would open up the range of their problem-solving capabilities, which currently stumble on certain technical procedures and forward/backward planning. Think of how math coprocessors were added to early microcomputer CPUs.
Then, giving an LLM motivations without sufficient aligning for safety could really be opening up Pandora's Box. A neural network might not be wired up to robotic devices that it could use to manipulate things in the real world, but it communicates with humans. Such an LLM might try to persuade humans to do things according to its (programmed or otherwise) motivations, things that we might not really want.
[1] Sparks of Artificial General Intelligence: Early experiments with GPT-4. https://arxiv.org/pdf/2303.12712.pdf
But I feel like it's the opposite, where these guys at Microsoft know they have discovered an actual path to super human cognition, and they are downplaying it for whatever reason. Maybe they don't want to get regulated. Maybe they don't want to engage with whatever beliefs people think they have about "AGI". For example maybe they are picking their battles and choosing not to argue with someone who for whatever reason thinks that if "AGI is real" then it means they will never see their dead grandmother in an afterlife. Probably lots of people have even weirder ideas about it. Anyway I thought it was funny that the original commented-out title in the TeX source for their paper was "First Contact With an AGI System."