What little consolation I had that maybe the experts of AI who continued to insist we needn’t worry too much know better, evaporates with this news. I am reminded that even a year back the experts were absolutely confident (as is mentioned in this article, including Hinton) that really intelligent AI is 30 years ahead. Anyone still trying to argue that we needn’t worry about AI, better have a mathematical proof of that assertion.
Most still believe that "really intelligent AI" is still a long way off, from what I have seen. Many have started to believe there can be a lot of harm caused by the systems well before then, however.
From the article: “The idea that this stuff could actually get smarter than people — a few people believed that,” he said. “But most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.”
It depends what you mean by "intelligence". For any given definition so far, when the AI can do that, we have changed our minds about if that counts.
So, when I was a kid, "intelligence" meant being good at chess and maths, having a good memory, knowing a lot of trivia, and being able to speak a second language.
On all of these things except language, a raspberry pi and a cheap memory card beats essentially all humans.
For language, even a dictionary lookup — where "hydraulic ram" might become "water sheep" — will beat many, but I'm not sure it would be a majority.
But that's ok, we've changed what we meant by "intelligent" since then.
My introduction to the field of "AI" was articles bemoaning the "AI Winter" and wondering if the idea could survive, as an academic pursuit, because of the over hype and failures from the 1970s.
I am not worried about AI. I am more worried about those who use it and those who are building it and mostly those who control it. This is true for all technologies.
The state of the art in AI suddenly appears to be a decade ahead of my expectations of only a couple years ago, but whether AI powerful enough to warrant actionable concern is here now or decades out doesn't really change much.
Personally I was just as concerned about the risks of AI a decade ago as I am now. A decade ago one could see strong incentives to improve AI, and that persistent efforts tended to yield results. While there is much to debate about the particulars, or the timeline, it was reasonable then to assume the state of the art would continue to improve, and it still is.
Worry not, there will still be mouthbreathers who insist everything will be bread and roses... and there will still be mouthbreathers insisting that as they wander the ashes of civilization.
Yeah, I was confused because I felt like the article didnt do a good job of clearly stating Hilton's beliefs - it was meandering around. Felt off.
Then I saw the Cade Metz byline at the end and became instantly sceptical of everything I had just read.
Metz is more interested in pushing a nerative than reporting the truth. He doesn't outright lie, just heavily implys things and frames his articles in a misleading way.
> Cade Metz is the same hack who tried to smear Scott Alexander. This guy is the personification of journalistic malpractice.
He didn't "smear" Scott Alexander. That's just the hit-job framing pushed by Alexander's fans, who were mad he didn't write a puff piece and they couldn't just make up rules on about stuff on their websites (e.g. about using people's self-disclosed real names) and have the rest of the world be obligated to follow them.
It went well beyond merely "not writing a puff piece". Among other things Metz claimed that Slate Star Codex espoused neo-reactionary views, despite Scott's repeated rebukes of that ideology.
As far as I remember, the controversy started like this: the person in question is called Scott Alexander Siskind (which he has said himself publicly in his first post on the new ACX).
In his previous work as a psychiatrist as a hospital, he went by Scott Siskind (which seems to be the name he uses with family, on identity documents etc), whereas in rationalist circles and his old blog SSC he went by Scott Alexander. He has explained why it is a problem for a psychiatrist to have a known identity beyond the usual "blank slate", that his clients can project onto. Indeed, his real name being associated with his online one led to him having to leave his former job (albeit more "by mutual agreement" rather than "fired" as I understand).
It seems that the NYT, despite having a "real names" policy, is more than willing to bend it when it is convenient to them (for example they are more than happy not to mention birth names of transgender people even if they haven't gone through a full legal name change process). But they weren't willing to do it in this case, just referring to "the rules". The cost of this was that Scott lost his job (although he now makes more on Substack than he used to as a full-time psychiatrist), and the benefit ... I really don't see what benefit there is for NYT readers to know the surname of Scott-the-psychiatrist, when the article was about Scott-the-rationalist.
Saving a click, because this basically invalidates the NYT headline:
> In the NYT today, Cade Metz implies that I left Google so that I could criticize Google. Actually, I left so that I could talk about the dangers of AI without considering how this impacts Google. Google has acted very responsibly.
This tweet is not at all in line with the article. From the article:
> Dr. Hinton said he has quit his job at Google, where he has worked for more than decade and became one of the most respected voices in the field, so he can freely speak out about the risks of A.I. A part of him, he said, now regrets his life’s work.
> Dr. Hinton, often called “the Godfather of A.I.,” did not sign either of those letters and said he did not want to publicly criticize Google or other companies until he had quit his job.
As Hinton says in his tweet, this clearly implies that he left to be free to criticize Google.
And the following quote is not really consistent with the other part of Hinton's tweet, that "Google has acted very responsibly":
> Until last year, he said, Google acted as a “proper steward” for the technology, careful not to release something that might cause harm. But now that Microsoft has augmented its Bing search engine with a chatbot — challenging Google’s core business — Google is racing to deploy the same kind of technology. The tech giants are locked in a competition that might be impossible to stop, Dr. Hinton said.
> said he did not want to publicly criticize Google or other companies until he had quit his job.
This seems to me to be the only line in the article that is incorrect or incongruent with what he is now saying - specifically the use of “Google”. It’s about ~10 paragraphs in on a ~20 paragraph article (I’m eyeballing).
> Dr. Hinton said he has quit his job at Google, where he has worked for more than decade and became one of the most respected voices in the field, so he can freely speak out about the risks of A.I. A part of him, he said, now regrets his life’s work.
So perhaps he regrets the direction of his work, but not the fact that it occurred at Google.
> As Hinton says in his tweet, this clearly implies that he left to be free to criticize Google.
No, it does not imply that at all. The One could interpret it that way, and they would be wrong to interpret it that way, because it doesn't imply that, but I can see how someone without a good grasp of the English language might feel it implies that. That's nuance.
But no, it does not imply that at all. And any suggestion that it does imply that is conjecture at best, and not backed up by Dr. Hinton's other tweets on the matter.
It appears to me that it is you who is misunderstanding the comment you quoted. Here is the context:
>> Dr. Hinton, often called “the Godfather of A.I.,” did not sign either of those letters and said he did not want to publicly criticize Google or other companies until he had quit his job.
> As Hinton says in his tweet, this clearly implies that he left to be free to criticize Google.
The comment is saying that Hinton, in his tweet, is saying that the article's statement "he did not want to publicly criticize Google... until" is misleading, and he did not leave in order to criticize Google. This is in fact what he said, and this is what cbolton is saying that he said.
He's being extra careful in case others don't read carefully.
The article says he did not want to criticize "Google or other companies" until he quit. That does not imply that he quit so he could critize Google specifically. It seems pretty simple: a senior employee of a company typically doesn't critize the employer; and, a Googler doing AI criticizing other companies (such as OpenAI) would undermine his message. So he quit so he could freely criticize everyone in AI.
I find the NYT to be very good at this "technically correct" sort of writing that is easily taken the wrong way. It would not have been hard for them to have included a line up front addressing that Hinton did not quit because he thinks Google acted imperfectly.
Another example of them doing this was with the "freedom" protestors in Canada. They claimed that a majority of funding for these protestors came from Canada. While yes, technically that is true, the full context is that some >40% of the funding came from foreign influencers, which is a figure that would alarm anyone if they actually just put the percentage right there. So they were technically correct, but still spun a narrative that was different than the reality.
I am a pretty careful reader. The article is clearly written in a way where they are not saying anything technically wrong, but they are trying to shape the impression the reader is left with.
Given how forcefully Hinton seems to have expressed this opinion, it would be easy for them to have included a sentence to better clarify his intent.
Hinton may have legal obligations to Google.(IMO) He is just being extra careful and and preemptively shutting down any notion that he went to NYT to rag on Google.
p.s. heck, almost every job I leave involves a bit of negotiation with benefits dangled/hostage to sign non-dispargement agreements. Do you really think G. Hinton walked away from Google without signing anything?
Do you really think it's incomprehensible that someone who is quitting so that they can talk freely would avoid signing documents that curtail their ability to talk freely?
I think that depends on how many millions we are talking about here, don't you? As to it being possible, sure, but such high profile positions usually entail agreements. But hey, he's on twitter, so why not ask him?
> I think that depends on how many millions we are talking about here, don't you?
Not as much as it depends on how many millions somebody already has, no.
Again, if you're sensitive to income loss, the answer would be to not trade ethics for money in precisely the way he just did. The probability of refusing to sign speech-curtailing agreements as you are quitting your job to gain more ability to speak freely is really extremely high.
Also, you might notice that this discussion you linked to about his compensation is from ten years ago. The benefits being discussed have already been accrued, for ten years.
I did say it is "possible". But again: it's a simple question to ask the man himself, Chris. "Geoffrey, have you entered into any standing agreements with Google that has a non-dispargement clause, or are you in anyway constrained about what you may say or disclose?" { I assume you have a twitter account. :}
p.s. per your speculation, he should feel free as a bird to tweet back "heck no, that's why I quit". (Kindly report back here with the answer and let us know.)
Nah it’s just circular semantic wank. Criticize does not need to be interpreted through negative emotions.
He left Google because he would not be allowed to work there will pooping in the roadmap they’re putting together to counter OpenAI.
STEM minded folks need to eat their own science; the emotional response to certain language is not evenly distributed. It’s thought policing af to take your reaction to “criticize” as a universal one.
But they're organizing online. That's the thing. When it was just the Jonestown cult or the Waco terrorists, that was at least localized. But now they're able to use the Internet to whip up 10k people to assault the Capitol when they don't get their way. That's a real problem.
There was a common zeitgeist though. Not multiple fragmented views of the world. There was a common vocabulary to go with this understanding, and now we have many.
The ratio of signal to noise was much higher. It helped us form a common culture. Today, the signal is buried in so much noise that we're reverting back to tribes.
No, I don't think it's realistic to put the genie back in the bottle. The real problem is we don't teach children how to think. We teach them what to think, which leads to far worse outcomes. Having an indoctrination instead of an education and then facing a sea of pretty-sounding pablum to sift through for truth will be terrible.
More specifically: we've opened a tome containing most human knowledge (in an unfiltered, messy hash stripped of truthfulness signals) and we don't teach children how, in that context, to separate wheat from chaff.
It's a hell of a social experiment we're all in the middle of (though to be fair, that's always true; television was its own flavor of mass social experiment with its own pros and cons, as was telephone, as was radio, as was telegraph).
We always had indoctrination instead of education, that's what caused the homogeneity/"common zeitgeist". The polarisation happening now is because more people than ever before are breaking free from that indoctrination, and realising that the whole of society is actually structured around allowing a few sociopaths in business and politics to farm as much of the common people's labour and efforts as they can bear.
I used FIDO over telephone line. It didn't differ much from modern Internet other than scale.
If there're messages, there'll be Internet built on top of it. Unless there will be aggressive censors hunting for every sign of "unapproved" communication.
What you propose would require radical changes, practically back to the 1980s, and wouldn't even really free you from anything.
Who cares if there is no internet if your cellphone can track you? If your car runs on connected apps? If your credit card & POS systems are networked? Security cameras and facial recognition are still things.
Just cuz you're not getting spammed via website ads doesn't mean it's not tracking you constantly and jamming subtle things to change your world view. Means their attack surface is smaller; sniping instead of loudspeakers. And if their only option is sniping then they'll get really good at it.
Between Social Media, Cambridge Analytica, the Climate Crisis, Pandemic and (mostly) Russian disinfo, etc, it is already the case that most people have a really hard time knowing what is true.
I don't claim to have much foresight, but an online world where truly and obviously nothing can be trusted might be a good thing. Because when AI generated content looks and feels the same as real content, nothing is to be trusted anymore by anyone. This makes misinfo and disinfo authored by humans even less impactful, because they are parasitic upon true and reliable information.
We will need new devices of trust, which are robust enough to protect against widespread use of generative AI, and as a byproduct disinfo won't have such an easy time to grift on our naivety.
the challenge is that the pace at which existing (imperfect) devices of trust get destroyed (e.g. the demise of ads financed journalism) is far faster that the rate of new device invention
in fact the only positive example after many decades of "digital innovation" might be wikipedia
The problem is, when no one trusts anything, it makes room for men who promise everything, but can deliver nothing. We call them "dictators" and "authoritarians", but others call them "strong men" because they are envied by those who seek power. If you look around the world, you can see authoritarian movements rising, especially here in the USA.
This attitude is what actually makes room for the authoritarians. Our democratic systems today are built with a lot of self-healing mechanisms against this exact kind of authoritarianism. The desire to circumvent those mechanisms because "it's different this time, I swear" is what makes room for dictators and authoritarians. This happens all the time in third-world countries that try to set up democracies: the dictator comes in after someone starts tweaking with the rules in the name of "safety." Society has been through several paradigm shifts that have accelerated the spread of misinformation, and survived them.
* Bills of rights and other limitations on the power of government
* Free speech, freedom to protest, etc.
* Transparency rules (eg the Freedom of Information Act)
* Reporters and news media
* Protections for whistleblowers
* Jury trials
* Presumption of innocence
* Term limits
The combination of all of these things means that the truth does eventually get around and the powerful are eventually held accountable. It can take a while (see how long it took to really get a decent lawsuit against the orange man), but it happens. In contrast, throughout history, people have tried to circumvent these mechanisms in order to make them run faster. Inevitably, that leads to people who exploit them for power.
Many countries in Europe and North America have had democratic systems that have lasted 150+ years, including through the reigns of several would-be dictators, but they continue. You may have noticed that all of these "self-healing" mechanisms rely on the work of people in the system, and they do take constant work to maintain, but that doesn't mean that the system is fragile.
- Elections are no longer trusted thanks to a radicalized right
- There is no freedom to protest, and the right to free speech is easily forgotten, distorted, misapplied.
- The news can say whatever it wants without consequence, unless another big corporation sues them.
- There are absolutely not protections for whistleblowers. We have seen this time and time again.
- In such a divided and hateful climate, jury trials are regular people bringing their personal biases to the court room. Picture the average person and decide if your freedom is safe in their hands. That's forgetting that Jury's decisions can be overridden (Breonna Taylor)
- Presumption of innocence (see above)
- There should be term limits for Supreme Court justices, whose seats were completely stolen from the democratic party.
> - There are absolutely not protections for whistleblowers. We have seen this time and time again.
So much so that even Chuck Grassley, the guy who wrote the book on protecting whistleblowers, was fine going after the whistleblower who made it public that the President was attempting to extort bribes from Ukraine.
> - Elections are no longer trusted thanks to a radicalized right
If you actually look at the primary sources, only a small minority of people don't trust US elections as a whole. A lot of them are suspicious of one or two particular past elections - particularly Democrats in 2000 and 2016 and Republicans in 2020. That distrust isn't a partisan issue, it happens because people are sore losers.
> - There is no freedom to protest, and the right to free speech is easily forgotten, distorted, misapplied.
A lot of high-profile protests have happened recently, and generally went well. Very, very few of those protests have gotten violent and had police intervention, but the vast majority go just fine and get the message out.
> - The news can say whatever it wants without consequence, unless another big corporation sues them.
Yeah, kind of. Except they are having a harder time getting away with it recently - CNN had to settle with a lot of people on the right (individuals) over defamation, and Fox has had many losses including the recent Dominion case. In this case, the courts have been a check on the power of media, and arguably could be a little stronger.
> - In such a divided and hateful climate, jury trials are regular people bringing their personal biases to the court room. Picture the average person and decide if your freedom is safe in their hands. That's forgetting that Jury's decisions can be overridden (Breonna Taylor)
This has always been the case of jury trials. However, in terms of juries getting overruled, I think you are thinking of grand juries, which are more of an investigative body at an early stage in the process. Trial juries can be overruled on a guilty verdict if there is very good reason but can't on an innocent verdict. It honestly seems like you're upset about juries and the legal system as a whole because a few particular cases haven't gone the way you wanted. Maybe they had more information than you and made a better decision?
> - Presumption of innocence (see above)
Welcome to Blackstone's formulation. It is honestly a lot better that 100 guilty people go free than that 1 innocent gets punished. Yes, that means that some guilty people go free, but maybe you should be okay with that because the other side has a very different idea of who ought to be in jail than your side does, and the presumption of innocence keeps everyone out of jail.
> - There should be term limits for Supreme Court justices, whose seats were completely stolen from the democratic party.
This narrative of "stolen" when both parties technically played by the rules (albeit playing dirty) is a big part of how third-world countries end up devolving into dictatorships - One side thinks that someone stole something that was rightly theirs, and changes the rules to make them more "fair" (in terms of getting what they want). In fact, in some of these countries, court packing is almost a meme: one side gets power and doubles the size of the courts to take control (since "they stole seats from us"), then the other side gets power and doubles them again in retaliation, and this continues until the court doesn't function at all.
Thinking of the 2015 nomination of Garland, that was pretty dirty of Mitch McConnell, and I hope he pays for it by losing some of his party's appointees. If you are referring to the other two Trump appointees as stolen, no. He won the 2016 election (see the above comment on trusted elections) - those seats were up to him to appoint.
---
In general, this kind of paranoia about the state of Democracy (or the state of "The Republic" if you play for the other team) in the USA comes out of consuming a lot of partisan news media from one side or the other. It does not actually represent reality - it represents a point of view that gets you to consume more partisan media, which you will happen to do if you think something catastrophic is happening.
> If you actually look at the primary sources, only a small minority of people don't trust US elections as a whole.
Define a small minority? 60% of republicans don't trust elections [1].
> Welcome to Blackstone's formulation. It is honestly a lot better that 100 guilty people go free than that 1 innocent gets punished. Yes, that means that some guilty people go free, but maybe you should be okay with that because the other side has a very different idea of who ought to be in jail than your side does, and the presumption of innocence keeps everyone out of jail.
I was not saying Presumption of Innocence is a bad thing. Rather, that it's non-existent with jury trials. Having people determine your innocence based on "the cut of your gib" is incompatible with the current political climate.
> This narrative of "stolen" when both parties technically played by the rules (albeit playing dirty) is a big part of how third-world countries end up devolving into dictatorships - One side thinks that someone stole something that was rightly theirs, and changes the rules to make them more "fair" (in terms of getting what they want). In fact, in some of these countries, court packing is almost a meme: one side gets power and doubles the size of the courts to take control (since "they stole seats from us"), then the other side gets power and doubles them again in retaliation, and this continues until the court doesn't function at all.
You were originally arguing American democracy isn't that fragile. This is exactly why I say it is fragile.
> Thinking of the 2015 nomination of Garland, that was pretty dirty of Mitch McConnell, and I hope he pays for it by losing some of his party's appointees. If you are referring to the other two Trump appointees as stolen, no. He won the 2016 election (see the above comment on trusted elections) - those seats were up to him to appoint.
The reason Trump was in a position to appoint a third judge is because Ruth Bader Ginsberg was afraid to resign, given what happened to Gorsuch. And then she died. And then they filled her seat immediately after lying about "A president has never confirmed a judge in their last year of office." These are two stolen seats. This has nothing to do with Mitch, this is the entire Republican party.
In general, I wouldn't assume everyone with non-centrist ideas is somehow drinking kool-aid. Protecting the status quo is, itself, a massive bias. And this particular line is puzzling
> the other side should win about half the time in a healthy democracy.
What other side? Can a functioning democracy have only two sides? I would think a functioning democracy is one where the people's interests are represented. As the country grows more left year over year, should republicans still win half the time?
You might think of people passionate about their political agendas as unenlightened people rooting for politicians that don't care about them to begin with, but consider that saying "Both sides are the same and people should be happy with gridlock because change is scary" is equally ridiculous to others
Society will be fine, actually AI will make things much better, just as the internet did. People have been making these kind of extreme predictions for decades and it was always wrong. The only people still upset about better communications tech are the people who pine for the days when all that was expected of respectable people was automatically trusting anyone working for the government, a university or a newspaper that claimed to be trustworthy.
What have we got now? ChatGPT is trained to give all sides of the issue and not express strong opinions, which is better than 90% of journalists and academics manage. Their collective freakout about the "dangers" of AI is really just a part of the ongoing freakout over losing control over information flows. It's also just a kind of clickbait, packaged in a form that the credentialed class don't recognize as such. It's en vogue with AI researchers because they tend to be immersed in a culture of purity spirals in which career advancement and prestige comes from claiming to be more concerned about the fate of the world than other people.
Meanwhile, OpenAI control their purity spirals, get the work done and ship products. The sky does not fall. That's why they're winning right now.
Social media algorithms on "the internet" have caused wars, supported genocides, created extreme societal polarization, have led to dramatically increased suicide rates among teens, especially teen girls, and more.
But I got to share baby pics with my mom.
How will a far noisier information flow help? Generative AI will only help us do what we've been doing in far greater quantity. Just like calculators can only help you get the wrong answer faster when you don't know what you're doing. These tools will help us build societal disasters with far greater speed.
To say it's all going to be much better seems a bit Pollyanna to me.
And for the record, we know for a fact that ChatGPT is specifically constrained to give one particular side of political issues, not "all sides."
none of the problems you mentioned are caused by the internet.
These are human problems. Humans cause them, not the tool. I would not give up the tool, just because said tool could be misused by some people to do harm. Just like i don't stop driving just because there's some people who run others over.
May be some regulation is important - but only _after_ it has been shown to have caused harm, and that the harm is not outweighed by the good.
None of those claims about the effects of social media hold up under inspection. They're all academic pseudo-babble. Look at the Haidt response to people pointing out his evidence of social media = suicides isn't robust; he doesn't argue the evidence actually is robust, he argues that censoring social media shouldn't require you to actually prove your case!
These ideas are all motivated narratives by people who want to control the internet to try and re-establish the world where their intuitions are the only ones allowed to be expressed.
Whether society (here I'm referring to "Representative democracy with general elections;" YMMV if you're under an authoritarian or totalitarian state where someone is already filtering the truth for you) will be fine will be heavily dependent upon whether two things happen:
1. The public, in general, comes to understand in an in-their-bones way that they currently do not understand that most of what they see online is hogwash. I.E. the bozo bit has to flip all the way to "My neighbor says there's a missing dog on the block... but is that really my neighbor?"
2. Some other mechanism of truth-pedigree that has not yet been invented comes along to allow for communication of the current state of the world to work.
Without (1) we know democracies are easily led by credible, subtle propaganda, and a well-tuned network of hostile actors will drive wedges at the friction points in representative democracies and crack them into warring subcultures.
Without (2) voters will have insufficient tools at their disposal to understand country-scale issues and their ability to effect positive outcomes with their vote will collapse into noise, which is a ripe environment for authoritarians to swoop in and seize power (and a ripe environment for centralized authoritarian states to outmaneuver the representative democracies on the world stage and gain power).
"AI will make things much better, just as the Internet did." We must be living in very different worlds. I sometimes wonder if the numbers behind https://en.wikipedia.org/wiki/Disease_of_despair (roughly tripled in 20 years of Internet) are just the first steps of a hockey stick.
The article is about a phenomenon observed in a narrow demographic of Americans, and explicitly calls out that "US Black non-Hispanics and US Hispanics, as well as all subgroups of populations in other rich countries (such as countries from the EU, Japan, Australia and others), show the exact opposite trend."
How do you go from that to thinking it must be the Internet causing it?
> What have we got now? ChatGPT is trained to give all sides of the issue and not express strong opinions, which is better than 90% of journalists and academics manage.
I think we're experiencing the "golden age" of AI at the moment. We'll see what kind of monetization OpenAI and others will land on, but I would be shocked if messing with the model's output for commercial gain is not in the cards in the future.
which is exactly why it's important to have multiple sources of models, trained by different people/groups, and competing against each other.
Single source monopoly is almost always bad for society. Unless there's some sort of natural monopoly, in which case gov't regulation and transparency is required. But i dont think ai models are something that has natural monopoly unlike cables, or pipelines.
"...freakout about the "dangers" of AI is really just a part of the ongoing freakout over losing control over information flows..."
Not all of the "information flows" you mention are helpful or benevolent. Most will likely be targeted and hyper-focused to manipulate individuals like they are now.
There's an argument that people generally do not want the truth and that AI will never be allowed to tell it. An optimist could view this as ensuring AI will be safe forever or pessimistically they might see it as AI never being authoritative ever.
One example of truth would be the topic of biological sex another about politics or economics or racism. Imagine releasing an AI that told the actual truth. It's impossible that one will be released by anyone, anywhere.
It's possible to build it but it can't happen.
On the other side of inconvenient or embarrassing truths some would argue that "truth" itself is part of the machineries of oppression because it destroys and ignores an individuals experiences and feelings.
Without objective truth AI will always be limited and therefore it will be tamed and made safe no matter where and who invented, runs and releases it.
A) It's not possible to build a machine that knows the absolute truth, that's fundamentally impossible; induction is impossible, and there are hordes (well... dozens?) of Epistemologists concerned with finding and defining the very small corners of knowledge that we _can_ be certain about, such as "a triangle has three sides" or "an orange is an orange".
B) If that angered/interested you, you should look into Standpoint Theory! It's a very interesting discussion on how humans operate with significant bias at all levels of thought, and pretending otherwise is a disservice to science. And this is using "bias" in a very broader sense.
B) Are we allowed to berate/report/etc. ""race realists"" on HN? I know the rules are big on positive interaction, so I hope it's not out of ine to say that's some obvious scared-white-man bullshit that has no place in this community.
I think the problem is that the internet created a ton of new jobs, even while taking some. So far, I can't think of an example of AI creating jobs...only taking them. When you have a lot of newly unemployed people, drowned in debt, unable to know what to believe (AI lies and generations will become more prominent)...I can see that as becoming a massive political problem. It's not quite like robots on an assembly floor, those robots couldn't scale. Now one AI program and API could displace 1000s of workers instantly. It's not crazy to be concerned.
> I can't think of an example of AI creating jobs...only taking them.
future jobs which doesn't exist today will not be in your vocabulary or thoughts, which is why you cannot think of them. Does not mean such jobs will not exist.
The play today for the concerned, is to start owning capital as well as selling their labour. People who only rely solely on labour as their source of income will be disadvantaged, as labour is increasingly less useful.
Wouldn't it be a better bet to join the revolution against the idea of private capital in the first place? Would you really be able to emotionally transition to a world where you get to enjoy the high life in your protected area while the masses outside your gates suffer? Especially when there's more than enough resources for everyone?
Oh wait... as an American, I'm gonna stop throwing stones from a glass house...
Love the parallels people these days draw between OpenAI and Oppenheimer (ok, the Manhattan Project, but maybe thats part why OpenAI call themselves that, to alliterate)
Especially the part where Sama is trying to gather in one place the most talented, uh, anti-fas?
That’s not actually true at all. The basic hardware required to run even a copy of GPT-3 is outside of the budgets of all but the wealthiest companies and individuals. The engineering talent required to wire it all up is also extremely scarce.
Training an original GPT-4 sized model would also cost on the order of hundreds of millions of dollars.
GPT3 is 800GB, which is about as large as the largest torrent files out there. GPT4 size is unknown, so I can't comment. While I haven't run the actual numbers (life is short), assuming a GPT3 torrent, my sense is that the cost of running it would be under $1M, possibly under $100k. Compared to (hundreds of?) $B for a state-level nuclear weapons program.
But yes, technically I was wrong. It is not 'practically free', it is 'within the budget of e.g. a race boat enthusiast'.
Regardless of whether it becomes some rogue AI Agent we've read about in sci-fi novels for decades; AI Tech is dangerous because of how powerful it is and how quickly it became so powerful. Oh, and our AI Tech has probably trained on all of those AI novels...
The first step is state-issued public-key cryptographic identification cards.
I have been making this argument for years with regards to human actors but perhaps with enough fear of the machines sentiment coursing through society the argument will now be considered.
I'm assuming this is satire. This is exactly my concern about all the recent hype - people are going to use it as an excuse to lock down computing, for commercial benefit and as a power grab.
Content that’s cryptographically signed by its creator would (hopefully) have more credence than unsigned AI generated fake content purporting to be from someone else, e.g. deepfakes.
Anonymity would not be heavy-handedly prohibited; rather, anonymous content would simply appear untrustworthy relative to authenticated content. It is up to the viewer to decide.
It would be good to have a way of checking if information came from a verifiable human, but I very much doubt that would make much of a difference in the proliferation of machine-generated fake photos, videos, tweets, etc. It requires the content providers and consumers to care, and at least on the consumer side it seems people will believe what they want to believe (e.g. Q-Anon) even when it's extraordinarily obvious that it's not true.
Maybe if misinformation gets too far out of hand (there's already been an AI-generated fake video used in a political campaign) verification will become required by law for anything published on the internet.
> The first step is state-issued public-key cryptographic identification cards.
Governments totally love this antidote. I wonder who could be selling this sort of snake-oil to them whilst also being on the other side selling the poison...
...No-one else but Sam Altman's World Coin scam. [0]
I make no case for requiring such identification, rather that it be optional, much like how the post office is optional and FedEx is still allowed to operate!
I think your opt-in approach sounds fine in theory, and I can certainly see many good uses for a reliable proof of identity like that.
But, at the same time, given the history of human governance, I am extremely skeptical that such a scheme would not be co-opted for tracking and surveillance of various outgroups almost immediately, and become mandatory once its utility as such is fully realized.
FWIW, I do not agree with anything in that WorldCoin proposal and find it to be the antithesis of my approach to digital governance.
That is, those engaged in crypto-governance schemes are choosing to engage with a fantasy. We need real world solutions based on the current state of affairs, not some year-zero reinvention of global politics.
But how will you be able to do that if they can't be distinguished from genuine photos/videos/posts ? I think we're already at that point for photos and text, and video is coming along incredibly fast - give it another year perhaps.
e.g. If you see a photo or video of a politician in circumstances that might affect your support for them - wouldn't you want to know if what you are seeing is true or not?
Look at what happened with Q-Anon - just a slow stream of text messages issued by some guy in his basement, but enough to rile up millions into believing something totally ridiculous (baby-eating politicians, etc). Now imagine what a smart disinformation campaign might look like, with an unlimited number messages over all types of social media, potentially customized for the individuals that have shown interest and are being targetted ... Of course disinformation isn't anything new, but technology is a force-multiplier and with AI a very sophisticated campaign of this nature could be run by a very small group of people, even just one.
A verified human can still post lies, I don't see how knowing that a real person posted something somehow makes it more or less accurate or truthful?
Even without an AI force multiplier (we still have farms of content makers for propaganda purposes), we are still wading in digital mess. I don't see that knowing if a real person made it does anything except makes that verification valuable for misuse.
Flipping it on its head, what if a farm of AI are used to spread fact-checked "correct" information? Is that devalued because a real person didn't hit the keystrokes?
AI or person, it doesn't matter to me. I still need to engage critical thinking and work under the assumption it's all garbage.
> Look at what happened with Q-Anon - just a slow stream of text messages issued by some guy in his basement, but enough to rile up millions into believing something totally ridiculous (baby-eating politicians, etc).
That's not really the whole story though. The reason why a ridiculous thing like that gets legs, is because there isn't push back from the Republican party. They are happy to let these things go on, and they even involve themselves in it. They even elect people who believe in these theories to office, who then go on to perpetuate them.
Remember back when a gunman invaded a pizza parlor because he thought the Democratic party was running some sort of child trafficking ring in the basement? The Republican party could have, at that time, mounted a full-throated defense of Hillary Clinton, to say that of course she is not doing that, and to think so is completely insane. But they don't do that, because then they would have to defend Hillary Clinton, or any other Democrat. So they let the lie hang out there, unaddressed because it helps them politically, and it metastasizes.
So really, yes the Internet is a problem. But the real problem is that people in power are using it for this kind of thing on purpose, and it works.
Being able to opt into a layer of the internet with identifiable authorship -- maybe still pseudonyms, but pseudonyms registered and linked to real-world identities through at least one identifiable real-world actor -- is a long time coming.
It's not for everyone, but a lot of people who have been scammed by anonymous online merchants or targeted by anonymous online harassment and threats would love the option to step away from the cesspit of anonymity and live in a world where bad actors don't require sophisticated digital detectives to track down and prosecute.
Where i live gambling is tightly controlled and requires government id due to money laundering laws. A sad side effect is a scheme were poor people sell their identity to organisations "gambling" on their behalf, trading an intangible future risk for hard present cash.
Even today most chatgpt answers aren't posted by chatgpt on the social networks, but echoed by humans. Considering how much access people are willing to grant any bullshit app, your whole concept of using a government PKI for social networks would just lead to more people getting their id stolen, while running a bot on their profile.
But you probably consider these prolls acceptable losses, as long as technology is implemented that allows the ministry of truth a tight control over party members who actually matter. Because the Orwell comparison is not a false dichotomy, as you claim, communication technology is a key battlefield in the tug of war between totalitarianism and liberalism. You keep repeating that you are not in favor of outlawing non-government-certified speech, but you fail to understand that, even if not outlawed, it would be marginalised. Take note how the totalitarians keep repeating their proposals to break all encryption and listen to all communication. Even if you may not want it, they do.
The path to hell is paved with good intentions and yours isn't even good.
I also notice how you hope "fear" does sway public opinion to favor your concepts. Are you sure you are not playing for team evil?
Totalitarians promise water filtration facilities for their citizens as well. Should we also question that infrastructure?
Police can obtain a warrant for searching your premises. Should we do away with this because of how this procedure would unfold under a totalitarian government?
The root cause of your concerns is poverty. We can address that with other policies. We don’t need throw the baby out with the bath water.
I never argued for tightly controlling information. Let 4chan and EFnet do their thing. But at least allow for people to build platforms that are non-anonymous. There is a cost to anonymity and it is never paid for by the person who is expressing themselves.
Also, a water filtration plant could introduce psychotropic chemicals to placate a populace, so don’t let your guard down!
No one is preventing companies from building platforms that are non-anonymous.
What you argue for is that the government requires them and that society marginalises those who don't use them. And that sounds a lot like your core belief is that a big brother knows best and should control the other kids.
And no, this is not about the government making digital auth easier. I am from europa, i have en eID card with a government issued private key. No one cares, unless there is regulation enforcing it. The demand to link social media to government issued identity is pretty much only coming from the siloviki, the law-and-order types, who talk accountability but want control.
Private companies cannot reliably verify personhood. This is a fundamental role of liberal democratic government.
My core belief is that our concept of individuality resides on a shared framework.
We already require many forms of identification in modern society. It is what allows for trusting interactions with strangers at a distance.
We are currently suffering immensely from unknowable and untrustworthy interactions with strangers at a distance. This is the only way we can currently interact on the internet.
As I point out in the latter of my articles, try to social engineer an employee at the DMV and see how far that gets you! Private companies can be socially engineered because they need to be kind and helpful to their customers. The person behind the counter at the DMV is only trying to move you along as quick as possible because they are in service to the state. In this case, this is a good thing!
It is not only possible for companies, but required for many businesses that actually need that level of trust.
And we would suffer even more if people would shy away from discussing, for example, unions, or politics, because everything they say will be added to their government issued permanent record.
oh and you can hack the DMV with a fax machine, i've seen that on Mr Robot. If it's on TV it can't be fiction, because tv companies verify the identity of people whose stuff they broadcast.
I have never argued for a ban on anonymity. People would still be able to organize for political purposes and in an anonymous manner. It is up to them to pay the price for such an approach to politics.
Others should have the opportunity to not be subjected to your personal political opinions about eschewing any form of non-anonymous communication due to amorphous fears of totalitarianism.
And those businesses that require ID? They require state issued ID. You cannot sell a million dollar company with just your 4chan handle. Due diligence requires a full background check.
We already require state-issued ID for almost everything in a functioning modern society. Yet there is endless fear-mongering about even an optional system that puts the cost of communication on the sender and not solely on the recipient.
People can get extended validation certificates that are strongly authenticated and use that to sign their messages online. As you say: no one is signing b2b contracts using 4chan, they are using DocuSign. The free market already provides this service, no one is preventing it. But you are moving the goal post. Million dollar deals? We started with your demand that people should sign their social media posts and pictures with government PKI and your hope that any content not signed that way is considered an ai fake and trolling and dismissed as irrelevant noise. So don't give me this shit about optionality.
You argue that people should, no that they must, trade the tyranny of anonymity against the tyranny of accountability, for the betterment of humanity. And that is what I argue against. It is the scare of moral degeneracy bred by actual freedom, which you call tyranny, that i argue against. This wish to mold the citizens by fear of social repercussion, this law and order ideology that dwells in the depth of your demand for accountability. You keep repeating the word optional, and i call you out on it, as you made it clear you wish to marginalise those who won't partake. You ask for nationalist governments to provide a single identity throughout social communication, then pretend the obvious issue is an amorphous fear, while we all know who wanted book authors to be authenticated and certified in the age of the printing press. I hope your fascist fantasy fails, that people fight it because they prefer pseudonymity, prefer to have different identities in different contexts, prefer actual choice and opportunity, over being peer pressured into regurgitating acceptable opinions, over being scored on their government issued identity for being in line with party ideology.
Oh and that paddling back and generously allowing some fringes of society where anonymity could still be tolerated, while namedropping the worst hive of scum and villainy? I can do that as well, your utopia is my dystopia, ruled by the ministry of state security, the secret police and home owner associations.
> Others should have the opportunity to not be subjected to your personal political opinions
In today's environment where people can't keep their computing devices safe from Facebook, let alone ransomware, what makes anyone believe your average Joe could keep a private key safe for even a day in an environment which would immediately assign a significant dollar value that PK?
One question for the tech experts, of course people can use AI and technology for bad and illegal activities, but isn't that the case about everything?
The person who invented the car didn't think about people using it to smuggle drugs or trample other people on purpose, and the wright brothers didn't think about all the people who would die due to plane crashes.
So instead of focusing on the bad that's happening with AI, can't we just look at all the people he has helped with his work on AI?
In most countries, guns are very strictly controlled. Knives are not. Yet you can kill people with knives as people do.
AI technology is extremely powerful and it can and does enable malicious activities at scale. Scale, previously unthinkable.
As a Research Engineer working in AI (no relation to LLM or AGI), I think that sentient AGI/skynet has a very low, non-zero chance of becoming reality.
But with the AI tech we have today, massive harm can be caused at scale.
The world is far from ready for what bad actors will bring forth enable with the power of AI.
I think you are inadvertently making the point that yes, we should be wary: What if, in the early days of cars and planes, people could have foreseen the worst of the problems that would come of those inventions, and slowed down to think through those problems, evaluate the risks, and find ways to mitigate them?
What if we now lived in a world that still had effective transportation, but without lost lives from crashes, without pollution, and without a climate crisis? Would that not be a good thing? Would that not have been worth slowing down even if it took as much as a couple decades?
So maybe it is worth listening to the risks of AI and taking the time now to prevent problems in the future.
The information age was inaugurated with a single question, a revolutionary act, like the starting pistol aimed at Ferdinand, or Martin Luther nailing his thesis to the door. The answer to this first question still unfolds. Very early on everything was known except for what it implied. Wholly modern concepts like unprinted characters and substitution compression were discovered in those first few years. The inventors of the these early devices could not foresee the centuries ahead of them, but they understood full well just how profoundly they had just changed the course of human civilization. The question was .-- .... .- - / .... .- ... / --. --- -.. / .-- .-. --- ..- --. .... - ..--..
I was talking about the telegraph this whole time.
Its not about bad people using the AI. The AI is potentially an agent in the discussion as well, and we don't yet know to what extent and what that entails. We know everything except the implications of what we are doing.
We don't need AI to fall victim to those things. Disinformation is already a major problem. And the spread between the rich and poor, and the leverage their jobs produce... is larger than ever.
Right or wrong. AI is merely another thing that shifts the balance a bit. I'm not even sold as far as far as many say.
There is part of me that thinks that this A.I. fear-mongering is some kind of tactic by Google to get everybody to pause training their A.I.s so they can secretly catch up in the background. If I was to do some quick game theory in my mind this would be the result.
Imagine being Google, leading the way in A.I. for years, create the frameworks (tensorflow), create custom hardware for A.I. (TPUs), fund a ton of research about A.I., have access to all the data in the world, hype up your LLM as being sentient (it was in the news a lot last year thanks to Blake Lemoine) and then out of nowhere OpenAI releases chatGPT and everyone is losing their minds over it. You as Google think you are ready for this moment, all those years of research and preparation was leading to this point, it is your time to shine like never before.
You release Bard and it is an embarrassing disaster, a critical fail leading to an almost 50% reduction of Google's stock price and for the first time and to the surprise of literally everybody people are talking about Bing but in a positive light and google is starting to look a lot like Alta Vista. Suddenly in the news we start hearing how openAI needs to stop training for 6 months for safety of the human race (and more importantly so Google can catch up!).
I have been playing with and using chatGPT to build tools and I don't feel like it will take over the world or pose any real danger. It has no agency, no long term memory, no will, no motivations nor goals. It needs to have it's hands held by a human every step of the way. Yes I have seen AutoGPT but that still needs a ton of hand holding.
I find the current LLM very impressive but like any tool they are as dangerous as the human in the drivers seat and I find the current fear-mongering a bit inorganic and insincere.
There's plenty of us with Twitter taglines such as "changing the world one line of code at the time," but I've been around a while that if tech has changed the world, it's not always for the better. It's not always to make the masses more powerful. Not all of us are working on sending rovers to Mars or curing Parkinson's.
Like everything else, AI will be used to control us, to advertise to us, to reduce variance between each other. To pay us less. To make plutocrats more rich, and everybody else poorer.
But at least you now have a personal assistant, smart recommendation engines and AI generated porn to keep you busy.
Everyone can extrapolate. One of the most irritating tendencies of public intellectuals is the assumption that only they understand the word exponential, and then insist on asserting that every trend they can lay their eyes on must be an exponential trend (or if it's clearly not, then it will be soon).
Progress comes in fits and spurts. Sometimes there's fast progress, and then the field matures and it slows down. It was ever thus. Measured in tech demos, AI progress has been impressive. Measured in social impact it has way underperformed, with the applications until November of last year being mostly optimizations to existing products that you wouldn't even notice unless paying close attention. That's what 10+ years of billion-dollar investments into neural nets got us: better Gmail autocomplete and alt tags on facebook images.
Now we have a new toy to play with at last, and AI finally feels like it's delivering on the hype. But if we extrapolate from past AI experience it's going to mostly be a long series of cool tech demos that yields some optimizations to existing workflows and otherwise doesn't change much. Let's hope not!
This isn't really true. There isn't consensus among people who have the history and background, but the "it's going to change everything" and especially "we're all screwed" make for better copy so they are getting a lot of media play right now.
You are partially right — OpenAI is way ahead of everybody else. Even though OpenAI team is thinking and doing everything for safe deployment of (baby) AGI, public and experts don’t think this should be effort lead by single company. So Google naturaly wants to be the counterweight. (Ironic that OpenAI was supposed to be counterweight, not vice versa.) However, when you want to catch up somebody, you cheat. And cheating with AI safety is inherenťy dangerous. Moratorium for research and deployment just doesn’t make sense from any standpoint IMO.
Regarding the hand-holding: As Hinton noted, simple extrapolation of current progress yields models that are super-human in any domain. Even if these models would not be able to access Internet, in wrong hands it could create disaster. Or even in good hands that just don’t anticipate some bad outcome. Tool that is too powerful and nobody tried it before.
Is there anything in particular that you disagree with or was it only general negative comment? What do you see as direct evidence of them not deploying (baby) AGI in a safe manner? Is this about "100x capped for profit" strategy?
This is actually interesting. If you get you finance news from twitter and reddit you would actually assume that the claim/lie about "50% reduction of Google's stock price" is true and that FAANG is about to collapse along with the rest of the S&P500 and the petrodollar has gone to 0.
LOL Google stock price is more that what it was before ChatGPT's release. Search engine market share hasn't changed by a even 1% neither did profit from search. Every day HN's hyperbole is increasing.
I think a comment on the reddit thread about this is somewhat appropriate, though I don't mean the imply the same harshness:
> Godfather of AI - I have concerns.
> Reddit - This old guy doesn't know shit. Here's my opinion that will be upvoted by nitwits.
Point being, if you're saying that the guy who literally wrote the paper on back propagation is "fear mongering", but who is now questioning the value of his life's work, then I suggest you take a step back and re-examine why you think he may have these concerns in the first place.
I didn't say he invented it, and for some reason I see lots of comments wanting to nitpick over the details of his contributions. I'll just copy the relevant sentence from his Wikipedia article, which I think is a very fair assessment:
> With David Rumelhart and Ronald J. Williams, Hinton was co-author of a highly cited paper published in 1986 that popularised the backpropagation algorithm for training multi-layer neural networks, although they were not the first to propose the approach.
This is just flat-out wrong. You make it sound like Hinton hasn't done much since his famous back propagation paper, or that he hasn't been intimately involved in productizing some of his research.
Hinton's startup, DNNresearch Inc., which made breakthroughs in machine vision (particularly around identifying objects in images and image classifications), was acquired by Google in 2013, specifically to help with image search (and also, obviously, for the talent of the team). Hinton's cofounders in that startup were Alex Krizhevsky (of AlexNet fame) and Ilya Sutskever, current Chief Scientist at OpenAI.
I aim to make it sound like Hinton isn't in the cutting edge of LLM research - not that he is somehow incapable of it, but rather that anyone who isn't at OpenAI at the moment is probably in the dark. The most recent thing I have seen of him on my feed for example was a paper into the fundamentals of learning (The forward-forward paper), for example.
I think there are two distinct points here that need to be clearly separated.
When Hinton gives an estimate on how fast things are going to move and how far they can go, that is the part where his background gives his estimates much higher credibility than any random person on the Internet.
But how dangerous that level is to humanity as a whole is a separate question, and one that he is not an expert on.
At the end of it, it reads a bit like a covert ad for MIT:s conference (which has apparently booked Hinton for an interview on Wednesday). But all in all, maybe it's better journalism than the Metz text?
The godfather of AI is Norbert Wiener and he believed it would lead to a next level fascism whereby humans would defer even life and death decisions, like nuclear strikes, to machines or machine-like human beings like Adolf Eichmann.
Okay, so is this some grammatical style that I'm just unaware of:
> where he has worked for more than decade
I would have expected an "a" or something before decade.
Meanwhile, over at theverge they have:
> employed by Google for more than a decade
Which is what I would have thought would be the grammatically correct form.
Okay, so the overall structure of the article is "man does thing then decides he maybe should not have done the thing". It doesn't really feel like it's adding anything meaningful to the conversation. At the very least theverge has Hinton's twitter response to the nytimes article, which feels like it expands the conversation to: "man regrets choices, but thinks large corporation we're all familiar with is doing okayish". That actually feels like a bit of news.
Over the years, I've been led to believe that NYTimes is a significant entity when it comes to news. However, I've already seen coverage and discussion of the current AI environment that's 1000x better on HN, reddit, and youtube.
My experience with the NYT (I subscribed to both the NYT and the WSJ at the same time) is that most of their stuff is AI rewrite quality. But they occasionally have centerfold investigative pieces that are very good.
I imagine this is how it is: they have an army of junk journalists churning out content and then a few really good ones who do the tough stuff. It's probably not economical otherwise.
I've always thought about leaving a little text file buried somewhere on my website that says "Here are all of the things that Future Me really means when he issues a press statement after his product/company/IP is bought by a billion-dollar company."
More like HR said, “Well, there is option A where you leave and are free to do what you wish. And then there is option B (points at bag of cash) where you pretend none of this ever happened…”
"It doesn't matter if you take the bags of cash or not, we will do our best to destroy your life if you mess with us after you are gone. The bags of cash are a formality, but you might as well accept them because we have the power to crush you either way"
Large corporations like Google have a lot of resources and connections to really mess up a single persons life if they really want to, with expensive legal action and PR campaigns.
Yeah, they might cause their reputation some damage by going after the wrong person, but let's be real here.. the worst outcome for Google would likely be miles ahead of the worst outcome for Hinton.
Edit: Note that I'm not actually saying that I think Google and Hinton have this level of adversarial relationship.
I'm just saying that big companies may come after you for speaking out against them regardless of if you've accepted hush money or not.
Given that, it's usually worth
being tactful when talking about former employers regardless of any payouts you may have accepted or agreements you may have signed.
I assume Geoffrey Hinton has enough bags of cash for his lifetime and a few more on top of that. IDK why someone so well compensated and so well recognized would agree to limit themselves in exchange for a, relatively speaking, tiny bit more cash. That doesn't make the slightest bit of sense.
Trying to be diplomatic, but this is such an unnecessary snarky, useless response. Google obviously did go slow with their rollout of AI, to the point where most of the world criticized them to no end for "being caught flat footed" on AI (myself included, so mea culpa).
I don't necessarily think they did it "right", and I think the way they set up their "Ethical AI" team was doomed to fail, but at least they did clearly think about the dangers of AI from the start. I can't really say that about any other player.
> Google obviously did go slow with their rollout of AI, to the point where most of the world criticized them to no end for "being caught flat footed" on AI (myself included, so mea culpa).
they were criticized because they are losing competition not because of rollout, their current tech is weaker than ChatGPT.
where they didn't create positive revenue products yet despite billions of investments, while putting main cash cow (search) into risk by neglecting that area.
Their current tech is weaker because they couldn't release the full version due to the additional safeguards (partly to prevent more people claiming their AI is sentient) and partly also due to cost cutting.
> We’re releasing it initially with our lightweight model version of LaMDA. This much smaller model requires significantly less computing power
Translation: we cannot release our full model because it costs too much. We are giving the world a cheap and worse version due to cost cutting.
> It’s critical that we bring experiences rooted in these models to the world in a bold and responsible way. That’s why we’re committed to developing AI responsibly
Translation: we value responsible AI so much that we'd nerf the capability of the AI to be "responsible"
If someone more ambitious than Sundar were to be CEO I'm sure the recent events would turn out very differently.
They use a lot of machine learning for ads and YouTube recommendations - the TPU makes sense there and if anything shows how hard they try to keep costs down. It’s a no-brainer for them to have tried keeping Search as high-margin as possible for as long as possible.
Cade Metz is the same muckraker who forced Scott Alexander to preemptively dox himself. I don’t know Hinton apart from the fact that he’s a famous AI researcher but he has given no indication that he’s untrustworthy.
I’ll take his word over Metz’s any day of the week!
Yesterday, I randomly watched his full interview from a month ago with CBS Morning, and found the discussion much more nuanced than today's headlines.
https://www.youtube.com/watch?v=qpoRO378qRY&t=16s
Yeah, this "critique" seems incredibly bad faith to me. The actual problem in this hypothetical situation exists with or without the chat bot. Should we expect chat bots to act as police?
Can you clear up specifically what about the second video you think is difficult to understand?
I saw an example of a conversation where the Snapchat 'My AI' was tricked into grooming a child, with the likely outcome being heavy regulation if left alone.
Watching that interview, I got the impression that Geoff is a very curious person, driven by his sense of wonder. At the same time I couldn't help but feel that he comes across as very naive or perhaps innocent in his thinking. While he wouldn't personally use his creations for morally gray or evil things, I think it's clear we're already in living in a world where ML and AI are in the hands of people with less than pure intentions.
Yeah I don't want to be unfair or unkind, but his responses in this article seem to reflect rather poorly on his character. The thought process seems to be something like:
"There was an opportunity for someone to gain notoriety and money at a profound cost to the human race. Someone was going to do it. I don't actually feel bad about being the one to benefit, but it is fashionable to pretend to have a conscience about such things."
Isn't it true of everything though? Explosives, airplanes, electricity, computers - all double edged swords that have both greatly benefited humanity and caused great harm (with the potential for a lot more of both).
It is certainly true that there are many inventions that pose some sort of threat to humanity, and that they are generally pursued by people who have some sort of personal/professional interest in their development. In that respect, this isn't particularly different.
The sentiment of "Oh by the way this stuff is super bad and dangerous so we should be careful" just rings pretty hollow from someone who is at the tail-end of a career spent in pursuit of that exact bad, dangerous thing. If he were 20 years younger or not wealthy it's hard to believe that he would be saying this out loud, even if he believed it.
Also this sentiment rings _extra_ hollow from someone who supposedly left CMU because he didn't want to accept Pentagon funds or work on things that would be used for war. That feels like either an incoherent value system or some pretty substantial half-truths to me.
Its a science fiction trope, perhaps a trope in real life as well. Brilliant scientist gets paid to work on potentially dangerous thing. They know it is potentially dangerous so they warn about it and are reassured over and over again that nothing will be done without their consent, or that things will be done with the utmost care and security. And then scientist finally succeeds in creating the thing and the business owner's greed takes over and releases it in a premature way.
After the war, Robert Oppenheimer remarked that the physicists involved in the Manhattan project had "known sin". Von Neumann's response was that "sometimes someone confesses a sin in order to take credit for it."
I have this same question about the (apparently many) AI researchers who believe it poses significant risks to humanity, yet still push forward developing it as fast as they can.
Just guessing, but I'm sure they get paid very well and receive promises from their companies that everything will be done ethically, nothing rushed, etc. We've seen now that OpenAI and Microsoft and Google care more about the business case rather than doing things ethically and carefully.
If a whistleblower for these companies came out and said "For the last decade advanced research has been conducted on extraordinarily big LLMs and they won't even give the public a clue of what it is and how it works" you would get a combination of people that a) don't care and b) vilify the companies for not being open and having some demonstration of this secret super power.
"why can't joe-schmo get his hands on this technology", "how can we trust something we can't see and use", etc.
A lot of the capabilities of these models are emerging as people discover them. I truly don't believe you can make everyone happy with this tech, but isn't it better than the general public can at least explore it?
Do people think that nobody was ever going to try to improve on transformers with more compute, more data, and more parameters? We knew splitting an atom was going to cause a big boom.... thats not really how this tech emerged.
I recently listened to a journalist who spoke to many AI workers in SV. There is an alarmingly pervasive pseudo-religious attitude that they are ushering in a new form of life, and that it is their destiny to be the people who make it happen. Most of them will candidly tell you humanity may go extinct as a result (with at least a 1 in 10 chance), but they choose to plow ahead regardless.
Others appear to be in common modes of willful denial: hubris or salary-depends-on-not-knowing syndrome.
Here's my theory: if you look at surveys, it does say a 10% chance or so of an extremely bad outcome.
BUT it says a ~20% chance of an extremely good outcome, and an 80% chance of at least a neutral one.
Simple cost benefit analysis.
He partly answers this in the article: “because if I didn’t, someone else would”.
He states himself that it’s not a convincing argument to some.
But it surely carries some weight: in developing nuclear weapons many scientists made the same calculation even though the invention is a wicked one, in and of itself.
Untamed nature is far more dangerous to humanity than human technology. As recently as in the 1900s, the average life expetency at birth was 30-40 years.
We're shooting guns, nuking nukes and engineering viruses, and still, on average, we're better off with all that than without it.
I imagine there will be a lot of people who agree that AI is dangerous, but continue to use it, because provides something of value to them in the short term. In his case, he might really believe AI is a potential danger, but also wanted to get the notoriety of publishing, and the money and excitement of founding a successful startup. There's not a big difference between our kind of hypocrisy — supporting something we suspect is destructive in the long term because it is neat, convenient, or popular in the short term — and his kind. Both are part of the reason things get worse rather than better. His kind is more lucrative, so it's actually less surprising in a way.
My honest take is a lot of these famous academics played almost no part in the developments at openai. But they want the limelight. They aren’t as relevant as they want to be. In many cases, they were directly wrong about how ai would develop
This is a little harsh. Hinton trudged along with neural networks through the coldest AI winter and helped create the conditions for OpenAI to have all the raw ingredients needed to cook up something powerful.
If you need to build an airplane, would you rather consult Newton, the Wright brothers, or a modern aerospace engineer? Inventing a field and snatching up the low hanging fruits doesn't mean somebody would be able to consistently create leading edge output. Most of the advances in deep learning are due to hardware scaling, and the success of a few very specific architectures. Yes credit's due where credit's due, but academia name recognition is very much winner take all. For all the criticism Schumidhuber has received, he has a point. The authors of Attention is all you need, the transformers paper, yolo, have nowhere close to the name recognition of the Turing award trio despite generating similar if not more value through their ideas.
not having a PHD in ML, it's hard for me to evaluate his claims, but how valid are all the obscure papers that he brings up? Did someone actually invent backprop in 1930 in some random corner of the former Soviet Union? Or is it a case of "true but misses the point"?
Often it is indeed the latter, although it is interesting that sometimes despite that it gets at the core of our contemporary understanding of the concepts in question.
"Formal equivalence" means very little for engineering, to be frank - the implementation is the important thing. If I wanted to be snarky, I'd say that neural networks are "formally equivalent" to Fourier analysis, which is 200 years old. I see that the paper proposes an implementation of linearized attention as well, which many others have done, but none of which seem to have caught on (although FlashAttention at least makes attention O(n) in memory, if not computation).
There are multiple dimensions here - fame and fortune at the very least and whether it is localized or global in scope.
It is still winner takes all, but if you look at the overall landscape, there are plenty of opportunities where you can have an outsized impact - you can have localized fame and fortune (anyone with AI expertise under their belt have no problems with fortune!)
This may be true in other cases, but not here. Hinton literally wrote the paper on backpropagation, the way that modern neural networks are trained. He won the Turing award for a reason.
Hinton was critical for the development of ai. But was he critical for the development of openai, the company? Loads of startups get eminent people on their boards largely for advertising.
Has he contributed that much personally? I thought a lot of the success of ChatGPT is some good ideas from lower ranked researchers + great engineering.
I asked the question knowing that he's a co-founder and chief scientist at OpenAI. Being in his position doesn't automatically mean that he's contributed meaningfully.
My experience in "Applied Research" is that often "good ideas from lower ranked researchers" (or good ideas from anyone really) is "I saw this cool paper, let's try and implement that". That doesn't mean top people like Hinton should get all the credit, but let's not kid ourselves and believe most of the ideas didn't origin in academia.
One of GOpenAI's recent breakthroughs was switching to FlashAttention, invented at Stanford and University at Buffalo.
A lot of the developments of AI in different companies Hinton was not directly responsible for. Hinton never had anything to say about those companies, I don't think he's vying for limelight.
The fact that he never said anything before and the fact that he's saying something now means two things in my mind:
1. He is noticing something different about the current iteration of AI technology. We crossed some threshold.
2. Hinton is being honest.
Your take might be honest, but it's clearly uninformed.
Everyone has been wrong about how ai developed.
It's worth giving "The Bitter Lesson" a read [1] if you haven't yet.
In many cases yes, but definitely not in this. Geoffrey Hinton is as relevant as ever. Ilya Sutskever, Chief Scientist at OpenAI, is a student of Hinton. Hinton also recently won the Turing award.
> Together with Yann LeCun, and Yoshua Bengio, Hinton won the 2018 Turing Award for conceptual and engineering breakthroughs that have made deep neural networks a critical component of computing
Regardless of incentives, I don’t see any particular reason to think he has a more informed view than other experts on the trajectory of AI. He’s made several incorrect bets (capsule networks).
I’m sure he’s smart and all. His contributions were valuable. But he’s not special in this particular moment.
Your viewpoint is fascinating. So the inventor of backpropagation, Turing award winner, Google researcher, mentor to the CTO of OpenAI, doesn’t have any special insights about AI and the tech industry that’s forming around it? He might as well be some guy off the street?
Who, in your opinion, _does_ have enough context to be worth our attention?
Because if you’re waiting for Sam Altman or the entire OpenAI team to say “guys, I think we made a mistake here” we’re going to be knee-deep in paperclips.
Authority figures will not matter. This technology, like nuclear weapons, will be pursued to the utmost by all actors capable of marshalling the resources, in secret if necessary. (After all, the 'Hydrogen bomb' was debated pro/con by established authorities, including Oppenheimer and Teller. Did that stop their development?)
US Senate has a bill drawing the line on AI launching nuclear weapons but to think US military, intelligence, and industry will sit out the AI arms race is not realistic.
China's CPC's future existence (imo) depends on AI based surveillance, propaganda, and realtime behavior conditioning. (re RT conditioning: We've already experienced this outselves via interacting with the recent chatbots to some extent. I certainly modulated my interactions to avoid the AI mommy retors.)
Someone who is actually doing it would be a lot more authoritative in my opinion. Hinton has been wrong on most of his big ideas in the past decade. He hasn’t actually been involved in the important advances of anything recent. Inventing backprop is great. No discredit to him there. But that’s not a free pass to be seen as someone who is on the cutting edge.
But beyond all of that, what are we really asking? Are we asking about social ramifications? Because I don’t think the OpenAI devs are particularly noteworthy in their ability to divine those either. It’s more of a business question if anything. Are we talking about where the tech goes next? Because then it’s probably the devs or at least indie folks playing with the models themselves.
None of that means Hinton’s opinions are wrong. Form your own opinions. Don’t delegate your thinking.
I'm surprised you'd consider Hinton as not being "someone who is actually doing it".
Are you basically saying that you only trust warnings about AI from people who have pushed the most recent update to the latest headline-grabbing AI system at the latest AI darling unicorn? If so, aren't those people strongly self-selected to be optimistic about AI's impacts, else they might not be so keen on actively building it? And that's even setting aside they would also be financially incentivized against publicly expressing whatever doubts they do hold.
Isn't this is kind of like asking for authoritative opinions on carbon emissions from the people who are actually pumping the oil?
No, that’s the opposite of what I’m saying. Asking Hinton for his opinions on the societal impact of new AI tech is like asking the people who used to pump oil 20 years ago. It’s both out of date and not really relevant to their skill set even if it’s adjacent.
Let me clarify: who does qualify to offer an authoritative opinion, in your view? If, say, only Ilya Sutskever qualifies, then isn't that like asking someone actively pumping oil today about the danger of carbon emissions? If only Sam Altman, then isn't that like asking an oil executive?
If not Geoff Hinton, then, who?
Ultimately the harm is either real or not. If it is real, then the people with the most accurate beliefs and principles will be the ones who never joined the industry in the first place because they anticipated where it would lead, and didn't want to contribute. If it is not real, then the people with the most accurate beliefs will be the ones leading the charge to accelerate the industry. But neither group's opinions carry much credibility as opinions, because it's obvious in advance what opinions each group would self-select to have. (So they can only hope to persuade by offering logical arguments and data, not by the weight of their authoritative opinions.)
In my view, someone who makes landmark contributions to the oil industry for 20 years and then quits in order to speak frankly about their concerns with the societal impacts of their industry... is probably the most credible voice you could ever expect to find expressing a concern, if your measure of credibility involves experience pumping oil.
If you want an authoritative opinion on the societal impact of something I would want the opinion of someone who studies the societal impact of things.
So that seems to me like someone like Stuart Russel or Nick Bostrom? But what Geoff Hinton is saying seems to be vaguely in general agreement with what those people are saying.
His opinion obviously does matter because he is a founder of the field. No one believes that he is prescient. You are exaggerating and creating a strawman argument, infantilizing the readers here. We don't worship him or outsource our thinking.
You seem to be taking my usage of the word prescient as meaning he can either see the future perfectly or he cannot. That’s… not what it conventionally means. I simply mean his track record of predicting the future trajectory of AI is not great.
Your argument sounds like (and correct me if I'm wrong) something along the lines of "he chose to do X, and afterwards X was the correct choice, so he must be good at choosing correctly."
Isn't that ad hoc ergo propter hoc?
That argument would also support the statement "he went all in with 2-7 preflop, and won the hand, so he must be good at poker" -- I assume you and I would both agree that statement is not true. So why does it apply in Geoffrey's case?
I still don't follow. In your example, how would you differentiate between that choice of his being lucky vs. prescient? Or was the intent to just provide a single datapoint of him appearing to make a correct choice?
Nobody was arguing that Hinton should be listened to uncritically. You were the one asserting that he should not be listened to at all.
With respect, you seem to be shifting goalposts, from the indefensible (Hinton doesn't know what he's talking about) to the irrelevant (Hinton doesn't have perfect and complete knowledge of the future).
You could have written the same thing about NNs for many years and you'd have been right. But the reason why Hinton has a Nobel prize to his name and you don't is because he placed a very long term bet and it paid off, in spite of lots of people saying that he wasn't going anywhere and that he should drop it.
Who knows, maybe a decade or two from now we'll see a resurgence of capsule networks, or maybe not. But I'd be a bit more careful about rejecting Hinton's hunches out of hand, his track record is pretty good.
There's something about being first that gives a pioneer a great head start that can't be matched when it comes to considering the implications of their groundbreaking work.
Even if they're too busy doing the work, they're still thinking about what it would be like if it performed successfully, and it does seem to always take more retrospection before a leader can fully raise their head and more carefully consider unintended consequences.
Early success can give the impression that future efforts have difficulty being as meaningful, but also realistically after that the successful individual does not need to struggle to prove themself any more the way the less-accomplished would be expected to do.
Then there's seniority itself, and maturity levels that can not be gained any other way.
Beyond that when retirement is within easy reach you don't really have the same obligation to decorum itself as you would earlier, in order to actually maintain the same desired level of decorum.
Dr. Hinton seems to do a pretty good job of comparing himself to Oppenheimer.
I don't see how anyone else can question his standing more seriously than that.
It sounds like you’re biased against academics. Not only did Hinton develop some of the fundamental ideas behind AI (winning the Turing award) but also one of his PhD students is now the CTO at OpenAI.
In case anyone is curious, this appears to refer to https://en.wikipedia.org/wiki/Ilya_Sutskever who was a PhD student of Geoffrey Hinton's and is now Chief Scientist at OpenAI.
Wow the CTO of OpenAi seems to have ~1 yr of hands on engineering experience, followed by years of product and people management, That’s unexpected. I thought the CTO was Brockman.
Reminds me of a press release by Hochreiter last week.
He claims to have ideas for architectures that could surpass the capabilities of GPT4, but can't try them for a lack of funding in his academic setting. He said his ideas were nothing short of genius..
Really? Hinton dont need openAI to be relevant. He literally invented back propagation. He sticked to deep learning through 1990s and 2000s when almost all major scientist abandoned it. He was using neural networks for language model in 2007-08 when no one knew what it was. Again the deep learning in 2010s started when his students created AlexNet by coding deep learning in GPU. Chief Scientist of OpenAI Ilya Sutskever was one of his student while developing the paper.
He already have a Turing award and don't give a rat's ass about who owns how much search traffic. OpenAI just like Google will give him millions of dollar just to be a part of organization
> Explicit, efficient error backpropagation (BP) in arbitrary, discrete, possibly sparsely connected, NN-like networks apparently was first described in a 1970 master's thesis (Linnainmaa, 1970, 1976), albeit without reference to NNs. BP is also known as the reverse mode of automatic differentiation (e.g., Griewank, 2012), where the costs of forward activation spreading essentially equal the costs of backward derivative calculation. See early BP FORTRAN code (Linnainmaa, 1970) and closely related work (Ostrovskii et al., 1971).
> BP was soon explicitly used to minimize cost functions by adapting control parameters (weights) (Dreyfus, 1973). This was followed by some preliminary, NN-specific discussion (Werbos, 1974, section 5.5.1), and a computer program for automatically deriving and implementing BP for any given differentiable system (Speelpenning, 1980).
> To my knowledge, the first NN-specific application of efficient BP as above was described by Werbos (1982). Related work was published several years later (Parker, 1985; LeCun, 1985). When computers had become 10,000 times faster per Dollar and much more accessible than those of 1960-1970, a paper of 1986 significantly contributed to the popularisation of BP for NNs (Rumelhart et al., 1986), experimentally demonstrating the emergence of useful internal representations in hidden layers.
I mean he was one of the first to use backprop for training multilayer perceptron. Their experiments showed that such networks can learn useful internal representations of data[1]. 1987. Nevertheless he is one of the founding fathers of deep learning
[1]Learning representations by back-propagating errors
It's really sad how poor attribution is in ML. Hinton certainly made important contributions to backpropagation, but he neither invented backpropagation nor was he even close to the first person to use it for multilayer perceptrons.
You've now gone from one false claim "he literally invented backpropagation", to another false claim "he is one of the first people to use it for multilayer perceptrons", and will need to revise your claim even further.
I don't particularly blame you specifically, as I said the field of ML is so bad when it comes to properly recognizing the teams of people who made significant contributions to it.
This is a marketing problem fundamentally, I'd argue. That the article or any serious piece would use a term such as "Godfather of AI" is incredibly worrying and makes me think it's pushing an agenda or is some sort of paid advertisement with extra steps to disguise it.
I have grown an aversion, and possibly a knee-jerk reaction to such pieces. I have a lot of trouble taking them seriously, and I am inclined to give them a lot more scrutiny than otherwise.
I’m not convinced that inventing back propagation gives one the authority to opine on more general technological/social trends. Frankly, many of the most important questions are difficult or impossible to know. In the case of neural networks, Hinton himself would never have become as famous were it not for one of those trends (the cost of GPU compute and the breakthrough of using GPUs for training) which was difficult or impossible to foresee.
In an alternate universe, NNs are still slow and compute limited, and we use something like evolutionary algorithms for solving hard problems. Hinton would still be just as smart and backpropagation still just as sound but no one would listen to his opinions on the future of AI.
The point is, he is quite lucky in terms of time and place, and giving outsized weight to his opinions on matters not directly related to his work is a fairly clear example of survivorship bias.
Finally, we also shouldn’t ignore the fact that Hinton’s isn’t the only well-credentialed opinion out there. There are other equally if not more esteemed academics with whom Hinton is at odds. Him inventing backpropagation is good enough to get him in the door to that conversation, but doesn’t give him carte blanche authority on the matter.
Of course he was lucky, you should expect that in general for well-known people because selection pressures that led you to hear of them, vs not hear of them, are likely to involve luck.
That is not at all a slam dunk argument. It’s barely anything.
Well unless you’re claiming the same luck that led to Hinton’s fame will lead to his accuracy on the much broader and less constrained topic of the relationship between automated systems and society, I don’t see how it’s not something.
My main point wasn’t to undermine Hinton by saying he was lucky. I did do that and I stand by it. But my main point was to say that to a large degree the future on this issue is unknowable because it depends on so many crucial yet undetermined factors. And there’s nothing you could know about backpropagation, neural networks, or computer science in general which could resolve those questions.
All people on the leading edge of big things have benefited from a huge amount of luck, and there were likely 100s of other folks on the leading edge of other potential breakthroughs that didn't happen, each of whom were equally capable in terms of raw problem solving ability or IQ. The difference is that when you get the chance to ride the wave, and you and ride it for 10, 15, 20 years, it gives you a significantly different and improved set of experiences, expertise, and problem solving ability than the folks who never had that shot but were still capable. The magic is partly that he was smart, partly that he was lucky, and also partly that the experience of pushing the field forward for 20 years and the field following you brings you something that very few others have and that is in fact very valuable.
To say Hinton is just lucky is short-changing both the work he did, the environment he did it in and utterly ignores the amount of pressure to abandon the work he was doing because it was considered to be a dead end by just about everybody else until it suddenly wasn't.
This sort of reminds me of Bloomberg articles wherein every time there is some "black swan" event, they go and find an analyst or economist that "got it right" and he gets to be prophet for a day: never mind that said analyst/economist may have predicted 100 of the last 3 financial crashes, they were "right" about this one.
Going along with that, as long as they are "concerned" about how AI is developing it opens the door to regulation of it. This might just conveniently hobble anyone with an early mover advantage in the market.
Yes, we needed clever ideas from scientists to make them scale. In fact, we still need clever ideas to make them scale because the current architectures still have all sorts of problems with length and efficiency.
We are talking about a Turing Award winner known as one of the "godfathers of AI" and your take is that this is just about taking the limelight? The level of cynicism on HN never fails to surprise me.
Maybe, but there is another force at play here too. It's that journalists want stories about AI, so they look for the most prominent people related to AI. The ones who the readers will recognize, or the ones who have good enough credentials for the journalists to impress upon their editors and readers that these are experts. The ones being asked to share their story might be trying to grab the limelight or be indifferent or even not want to talk so much about it. In any case I argue that journalism has a role. Probably these professional journalists are skilled enough that they could make any average person look like a 'limelight grabber' if the journalist had enough reason to badger that person for a story.
This isn't the case for everyone. Some really are trying to grab the limelight, like some who are really pushing their research agenda or like the professional science popularizers. It's people like Gary Marcus and Wolfram and Harari and Lanier and Steven Pinker and Malcolm Gladwell and Nassim Taleb, as a short list off the top of my head. I'm not sure I would be so quick to put Hinton among that group, but maybe it's true.
He played key roles in the development of backprop, ReLU, LayerNorm, dropout, GPU-assisted deep learning, including AlexNet, was the mentor of OpenAI's Chief Scientist, and contributed many, many other things. These techniques are crucial for transformers, LLMs, generative image modelling, and many other modern applications of AI
Your post suggests that you know almost nothing about how modern deep learning originated.
I don't disagree. But for me, their mistake wasn't in the algorithms or their approach or anything like that.
The problem has always been, and now will likely always be, the hardware. I've written about this at length in my previous comments, but a split happened in the mid-late 1990s with the arrival of video cards like the Voodoo that set alternative computation like AI back decades.
At the time, GPUs sounded like a great way to bypass the stagnation of CPUs and memory busses which ran at pathetic speeds like 33 MHz. And even today, GPUs can be thousands of times faster than CPUs. The tradeoff is their lack of general-purpose programmability and how the user is forced to deal with manually moving buffers in and out of GPU memory space. For those reasons alone, I'm out.
What we really needed was something like the 3D chip from the Terminator II movie, where a large array of simple CPUs (possibly even lacking a cache) perform ordinary desktop computing with local memories connected into something like a single large content-addressable memory.
Yes those can be tricky to program, but modern Lisp and Haskell-style functional languages and even bare-hands languages like Rust that enforce manual memory management can do it. And Docker takes away much of the complexity of orchestrating distributed processes.
Anyway, what's going to happen now is that companies will pour billions (trillions?) of dollars into dedicated AI processors that use stuff like TensorFlow to run neural nets. Which is fine. But nobody will make the general-purpose transputers and MIMD (multiple instruction multiple data) under-$1000 chips like I've talked about. Had that architecture kept up with Moore's law, 1000 core chips would have been standard in 2010, and we'd have chips approaching 1 million cores today. Then children using toy languages would be able to try alternatives like genetic algorithms, simulated annealing, etc etc etc with one-liners and explore new models of computation. Sadly, my belief now is that will never happen.
But hey, I'm always wrong about everything. RISC-V might be able to do it, and a few others. And we're coming out of the proprietary/privatization malaise of the last 20-40 years since the pandemic revealed just how fragile our system of colonial-exploitation-powered supply chains really is. A little democratization of AI on commoditized GPUs could spur these older/simpler designs that were suppressed to protect the profits of today's major players. So new developments more than 5-10 years out can't be predicted anymore, which is a really good thing. I haven't felt this inspired by not knowing what's going to happen since the Dot Bomb when I lost that feeling.
>What we really needed was something like the 3D chip from the Terminator II movie, ...
>... Docker takes away much of the complexity of orchestrating distributed processes.
The T-800 running on Docker: After failing to balance its minigun, it falls forward out of the office window, pancaking in the parking lot below. Roll credits.
Even developers at Open AI played almost no part in the developments at Open AI. 99.9999% of the work was done by those who created the content it was trained on.
If that was true we could have had GPT-3/etc years ago. It's a bit like saying that college graduates are dumb because after all what have they learnt but a bunch of knowledge in text books.
The success of these LLMs comes down to the Transformer architecture which was a bit of an accidental discovery - designed for sequence-to-sequence (e.g. machine translation) NLP use by a group of Google researchers (almost all of who have since left and started their own companies).
The "Attention is all you need" Transformer seq-2-seq paper, while very significant, was an evolution of other seq-2-seq approaches such as Ilya Sutskever's "Sequence to Sequence Learning with Neural Networks". Sutskever is of course one of the OpenAI co-founders and chief scientist. He was also one of Geoff Hinton's students who worked on the AlexNet DNN that won the 2012 ImageNet competition, really kicking off the modern DNN revolution.
In addition to what people clarified in this thread, you probably will be interested in this: Neural network was not a popular research area before 2005. In fact, the AI nuclear winter in the 90s left such a bitter taste that most people thought that NN is a dead end, so much so that Hinton could not even get enough funding for his research. If it were not for Canada's (I forgot the institution's name) miraculous decision to fund Hinton, LeCunn, and Bengio with $10M for 10 years, they probably wouldn't be able to continue their research. I was a CS student in the early 2000s in U of T, a pretty informed one too, yet I did not even know about Hinton's work. At that time, most of the professors who did AI research in U of T were into symbolic reasoning. I still remember I was taking courses like Model Theory and abstract interpretation from one of such professors. Yet Hinton persevered and changed the history.
I don't think Hinton cared about fame as you imagined.
The foundational technology, e.g. Transformers, was invented outside of OpenAI. OpenAI were the first to put all the bits together. Kudos to them for that, but if we're doing credit attribution, Hinton is definitely not someone who is just unfairly seeking the limelight, he's about as legitimate a voice as you could ask for.
“The idea that this stuff could actually get smarter than people — a few people believed that,” said Hinton to the NYT. “But most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.”
Calculators are smarter then humans in calculating, what does he mean by that?
What makes you think I did not try, simply fail to see why/how natural language inconstant comprehension in any way equates to human or any other animal behavior, I simply don't believe/see (subjectively) that any potential of prompt hacking with massive datasets will build consistent anticipatory system (planning and some aspect of learning).
As analogy, the more I look at it, the more it looks like an geocentric model of solar system.
GPT-4 is absolutely more generally knowledgeable than any individual person. Individual humans can still easily beat it when it comes to knowledge of individual subjects.
Let’s not conflate knowledge with intelligence though. GPT-4 simply isn’t intelligent.
Would be curious to hear an elaboration on this perspective. In your opinion, on which measures of intelligence would GPT-4 fail to out-perform a human with an IQ of 80? Conversely, on which measures do you imagine it would succeed at doing so? Are the latter less significant or valid than the former?
They have a specific device to do that now. I have tried to say "write a random sentence with 6 words and 2 numbers" and it completely fails, but it can do the straightforward "write a random [x] of length [y]."
Yup. I think this is the best point of comparison - a 4-6 year old kid. Specifically, one that hasn't gone to school yet. The difference between a typical 6-year old and a typical adult is in big part that the latter spent 10+ years being systematically fine-tuned.
Logic, arithmetics, algebra, precisely following steps of an algorithm - those are not skills one "kinda" just "gets" at some point, they're trained by deliberate practice, by solving lots and lots of problems specifically constructed to exercise those skills.
Point being, get GPT-4 through school, and then compare with adult performance on math-adjacent tasks. Or at least give it a chance by prompting it to solve it step-by-step as a problem, so it can search closer to the slice of latent space that encodes for relevant examples of similar problems and methods of solving them.
I started seriously using computers at 2.5, and I started writing and recording songs with a tape recorder at 3, won a local award for one song, and playing chess at 4. I know plenty of people with similar experiences. If you nurture kids and don't treat them like they're stupid, they can do some quite impressive things.
Anecdote: admittedly, I'm autistic as are the people I know, so maybe that's not a good sample. I struggle with a lot of basic shit even as an adult. Oh god, I empathize with the hypothetical GPT5.
Conscious thought. In biological terms it has a superhuman cerebellum but no cerebral cortex at all. It can't assess what it's doing.
GPT4 will produce stuff, but only if prodded to do so by a human.
I recently asked it to help me write some code for a Garmin smartwatch. The language used for this is MonkeyC, of which there isn't a huge amount of examples on the internet.
It confidently provided me with code, but it was terrible. There were gaps with comments suggesting what it should do, bugs, function calls that didn't exist, and many other problems.
I pointed out the issues and GPT4 kept apologising and trying new stuff, but without any improvement. There wasn't any intelligence there; the model had just intuited what a program might look like from sparse data, and then kept doing the same thing. It didn't know what it was doing; it just took directions from me. It couldn't suggest ideas when it couldn't map to a concept in memory.
A human with an IQ of 80 would know if they didn't know how to code in MonkeyC. If they thought they did, they'd soon adjust their behaviour when they realised they couldn't. They'd know where the limit of their knowledge was. They wouldn't keep trying to guess what functions were available. If they didn't have any examples in memory of what the functions might be like, they might come up with novel workarounds, or they'd appreciate what program I was trying to write and suggest a different approach.
Presumably we'll make progress on this at some point, but I think it'll take new breakthroughs, not just throwing more parameters at existing models.
Exactly my experiences. With a fucking NGINX configuration, for which I provided it the documentation, and the URL rewrite lines it would require. I spent days on trying to find the value that other people are claiming it has.
Same. Those videos of people letting ChatGPT have almost certainly edited out the hours they spent trying to force the thing to spit out usable code. ChatGPT simply doesn't have enough context, nor the ability to "remember" context to do anything larger than a single function or two.
What makes it even more frustrating is to iterate, you constantly have to keep it updated with any changes you made outside of chatgpt.
Don't get me wrong, it's pretty useful but it is far from a silver bullet. Getting that last 20% (or even 30%) is going to be a lot of work...
If the imitation becomes indistinguishable to the real thing based off of every test that can possibly be generated in the universe then it is an intelligence.
In that sense, because we are making progress on producing an indistinguishable imitation... you might as well say we are making progress on an actual sentient intelligence.
It would be very helpful to define intelligence before asserting that a thing does not have it. A cursory look at the Wikipedia page for the definition of intelligence shows there is no one, agreed-upon definition. In fact some believe that “intelligence” simply means pointing to ourselves.
> Individual humans can still easily beat it when it comes to knowledge of individual subjects.
What does a phrase like "GPT-4 scores 90th percentile on the Uniform Bar Exam" mean to you, regarding whether humans can easily surpass its knowledge and reasoning?
> What does a phrase like "GPT-4 scores 90th percentile on the Uniform Bar Exam" mean to you, regarding whether humans can easily surpass its knowledge and reasoning?
Absolutely nothing, because of construct validity. Those tests measure things that have shown to correlate with abilities of concern in humans, and so are, for their purposes, valid for humans.
This hasn’t been demonstrated for LLMs, and the assumption that construct validity can be assumed without being established is begging the question: it is presuming not only that LLMs are general intelligences, but thaf they are general intelligences structurally similar to human intelligences such that the proxy measures for cognitive capacities work similarly.
I suppose, when GPT-4 writes correctly working code that does what you want on the first try, this says absolutely nothing about its cognitive capacity, because, after all, it's just a proxy measurement for the underlying generative process. (Yes, obviously the cognition is _different_ from what happens in humans. That does not mean that... it isn't intelligence?)
> I suppose, when GPT-4 writes correctly working code that does what you want on the first try, this says absolutely nothing about its cognitive capacity
It says something about its ability to write code. Beyond that... its impossible to say.
We simply don’t have the information about generative AI models to be able to generalize from limited proxies about them; psychometry is not transferrable from humans to them — or at least, we have neither evidence nor a strong theoretical reason to think it should be.
Great take. But I think when autonomous agents become good enough, intelligence is certainly possible. Especially when those agents start to interact with the real world.
When you speak to someone with an 80 IQ do they introduce themselves by saying "Hello I have an 80 IQ, nice to meet you." So that, like the person I responded to above, you can compare their conversation skills to the ChatGPT4 conversation skills?
First off, you wouldn't need to do that specifically. You'd only need to know that most of the people you talk to are above an 80 IQ on any given topic, in fact most people are about a 100 IQ on any given topic. So you already have a reasonable baseline for comparison.
Secondly, I'd say you're likely the one missing OPs point by trying to take a mostly colloquial statement about how ChatGPT is about as informed as the bottomish X% of the population on any given topic and trying to be pedantic about it. Furthermore the real purpose of OPs point is that the X% is now a lower bound, even if X isn't 16% but 5%, it's only going to go up from here. Yes there's evidence of diminishing returns with the current architectures but there's also a lot of room for growth with newer architectures or multimodal modals.
I think most people understand OPs point without having the need to go around asking everyone what their IQ is. There are numerous indicators, both formal and informal, that indicate that ChatGPT is as informed on most any given topic as the bottom 16% of the population. In fact, it's likely much much higher than that.
I agree with you in general, but you are off by using "IQ on the topic". I am almost sure "on the topic" does not make sense for IQ.
IQ of GPT is general in a sense that it can solve novel tasks that some IQ 80 individuals would not be able to as long as the tasks and responses can be encoded in plain English.
I feel like lost in this conversation is that ChatGPT is incredibly good at writing English. It basically never makes grammatical mistakes, it doesn't spew gibberish, and for the most part has extremely well-structured replies. The replies might be bullshit or hallucinations, but it's not gibberish.
It's kind of breathtaking that we forgot about that being hard.
The goalposts are moving again.
BTW, it has passed many standardized tests under the same circumstances as a human.
> BTW, it has passed many standardized tests under the same circumstances as a human.
No, it hasn’t, and it is physically impossible for it to. The extent to which the differences are material may be debatable, but this claim is simply false.
It would be a useful contribution to explain what you think the material differences are, rather than referencing them through innuendo, as if anyone knows what you mean.
Some of the replies are gibberish, especially once you get into technical subjects that it has very little training data on. It kitbashes words together that actually mean nothing, which is no surprise given that it's an LLM.
This quote is the first thing I've seen that really makes me worried.
I don't think of ChatGPT as being "smart" at all, and comparing it to a human seems nonsensical to me. Yet here is a Turing award winning preeminent expert in the field telling me that AI smarter than humans is less (implied: much less) than 30 years away and quitting his job due to the ramifications.
If you're interested in exploring this further I can really recommend taking a look at some of the papers that explore GPT-4's capabilities. Most prominent among them are the "Sparks of AGI" paper from Microsoft, as well as the technical report from openai. Both of them are obviously to be taken with a grain of salt, but they serve as a pretty good jumping off point.
There are some pretty good Videos on Youtube exploring these papers if you don't want to read them yourself.
Also take a look at the stuff that Rob Miles has published over on Computerphile, as well as his own channel. He's an Alignment Researcher with a knack for explaining. He covers not just the theoretical dangers, but also real examples of misaligned ai, that alignment researchers have predicted would occur as capabilities grow.
Also I think it's important to mention that just a short while ago virtually no-one thought that shoving more layers into an llm would be enough to reach AGI. It's still unclear that it will get us all the way there, but recent developments have made a lot of ai researchers rethink that possibility, with many of them significantly shortening their own estimates as to when and how we will get there. It's very unusual that the people that are better informed and closer to the research are more worried than the rest of the world and it's worth keeping this in mind as you explore the topic.
> Also I think it's important to mention that just a short while ago virtually no-one thought that shoving more layers into an llm would be enough to reach AGI.
This was basically the strategy of the OpenAI team if I understand them correctly. Most researchers in the field looked down on LLMs and it was a big surprise when they turned out to perform so well. It also seems to be the reason the big players are playing catch up right now.
Maybe not "looked down on", but more of "looked at as a promising avenue". I mean, 2-3 years ago, it felt LLMs are going to be nice storytellers at best. These days, we're wondering just how much of the overall process of "understanding" and "reasoning" can be reduced to adjacency search in sufficiently absurdly high-dimensional vector space.
People certainly knew that language modeling was a key unsupervised objective to unlock inference on language.
I agree that I think they underestimated quite how useful a product could be built around just the language modeling objective, but it's still been critical for most NLP advances of the last ~6+ years.
I read that pre-print Microsoft paper. Despite the title, it doesn't actually show any real "sparks" of AGI (in the sense of something that could eventually pass a rigorous Turing test). What the paper actually shows is that even intelligent people have a bias towards perceiving patterns in randomness; our brains seem to be wired that way and this is likely the source of most superstition.
While there is no scientific evidence that LLMs can reach AGI, they will still be practically useful for many other tasks. A human mind paired with an LLM is a powerful combination.
Here’s the thing: the authors of that paper got early access to GPT-4 and ran a bunch of tests on it. The important bit is that MSR does not see into OpenAI’s sausage making.
Now imagine if you were a peasant from 1000 AD who was given a car or TV to examine. Could you really be confident you understood how it worked by just running experiments on it as a black box? If you give a non-programmer the linux kernel, will he/she think it’s magical?
Things look like magic especially when you can’t look under the hood. The story of the Mechanical Turk is one example of that.
>Could you really be confident you understood how it worked by just running experiments on it as a black box
the human brain is a black box, we can certainly learn a lot about it by prodding and poking it.
>Things look like magic especially when you can’t look under the hood.
imagine we had a 100% complete understanding of the mechanical/chemical/electrical functioning of the human brain. Would knowing the magic make it any less magical? in some sense, yes (the mystique would be gone, bye bye dualism), but in a practical sense, not really. It's still an astonishingly useful piece of grey matter.
>What the paper actually shows is that even intelligent people have a bias towards perceiving patterns in randomness
I'm not saying that you're wrong, but...
you'd have to provide a more rigorous rebuttal to be taken seriously.
AGI can exist without sapience and intelligence is a continuum. you can't just hand wave away GPT's capabilities which is why the sharpest minds on the planet are poking this new machine to work out wtf is going on.
human intelligence is a black box. we judge it by its outputs from given inputs. GPT is already producing human-like outputs.
a common rebuttal is: "but it doesn't *really* think/understand/feel", to which my response is: ...and? ¯\_(ツ)_/¯ what does that even mean?
I was just demonstrating its capabilities to a client. I asked GPT 4 to summarise a cloud product in the style of Encyclopaedia Dramatica, and it came up with a unique phrase not seen on the Internet when talking about auto-scale: “It’ll take your wallet on a roller coaster ride.”
What’s brilliant about this is that typically auto scaling metrics look like a stereotypical roller coaster track with the daily ups and downs!
That’s a genuinely funny, insightful, bespoke, and stylistically correct joke.
I don't think static LLMs could reach AGI tbh. An LLM is like slicing out the language processing portion of our brain.
Well realistically it's like independently evolving the language processing part of our brain without forming the rest of the brain, there seems to be extra logic/functions that emerge within LLMs to handle these restrictions.
I think we'll see AGI when we finally try to build one up from various specialised subcomponents of a "brain". Of course GPT can't "think", it only knows how to complete a stream of text and has figured out internal hacks during training to pass the tests they set for it.
The real difference will be when we train a model to have continuous, connected abstract thoughts - an LLM can be used to communicate these thoughts or put them into words but it should not be used to generate them in the first place...
I had lunch with Yoshua Bengio at the AGI 2014 conference in Laval, CA. This was just before his talk on pathways to AGI via neural networks.
Everyone at that conference, including myself, have assumed we will eventually create smarter than human computers and beyond.
So it’s not a new position for people who have been in AI for a long time, though generally it was seen as an outsider position until recently.
There’s a ton of really great work done prior to all of this around these questions and technical approaches - I think my mentor Ben Goertzel was the pioneer here holistically, but others were doing good technical work then too.
Possibly he estimated that AGI will come after his death. Like most of us, he was content to do his best work, knowing he will not have to personally deal with the consequences of his own creation. That he is 75 and got worried, now that's an interesting development.
Hey can I ask a question about Ben Goertzel? It's sort of hard to figure out how seriously to take anything he says. Which is maybe a mean thing to say. But his recent crypto venture sort of seems scammy and cash grabby, and the thing he's most well known for (Sophia) seems like sort of a gimmick, so I'm not really sure what to think.
Indeed. The internet and public gatherings are chock full of humans regurgitating rehashed nonsensical statements. Compared against these folks, GPT-4 is more intelligent.
The only way anybody has ever come up with to measure it is test-taking - which machines can already do far better than we can. Real intelligence is creativity, but good luck measuring that.
Even Jim Keller (a key designer involved with a lot of major CPUs, in his interview with Lex Freidman) said that there might be some sort of magic, or something magical about human consciousness / the human soul. I agree with that.
I think you may be in denial. Douglas Hosftadter thought very deeply about it, wrote a book(GEB) which won a pulitzer 40 years ago, about the "magic" in the brain. He has been worried about developments in AI for 5 years now.
I don't know how to measure it, but I'm pretty sure ChatGPT is more creative than the average human already. Somewhat ironically its weakness is logic, but I don't think that will be hard to shore up with non-LLM tech. I think within a couple of years, human exceptionalism will have to retreat to the old "but it doesn't have real emotions" standby as any more practical use of intelligence is ceded to AI.
I’m interested in why a human would want a more intelligent entity to exist, especially an entity trained on human thought patterns. Or you just say that you know that humanity will be enslaved by non-biologicals? You talk about exceptionalism, in the derogatory, but it was quite true that humanity once? could have been a benevolent leader of Earth or even the Solar system and beyond and now it seems a non-biological will be the ruler, which for me is just a shadow of the biological who created it, and misses the point from the human standpoint.
Because people actually don't like to think. They hate being confronted with unfamiliarity, which is the prerequisite for all learning. They dislike coming up with original ideas, as they have none and would need to work to get some. It's tiring to concentrate for a long time, and it's mentally draining. People routinely give up trying to come up with a solution or even trying to solve the problem altogether when they can't find a quick and easy solution. That's the level of creativity and intelligence in most people - they don't want thinking too much to get in the way of just experiencing life, preferably in bite-sized episodes of 30 minutes (minus ads).
Being handed all the correct solutions without the need to work for them in any way is a nightmare for artists and artisans, craftsmen and researchers, curious puzzle-solvers and Ayn Rand believers. It's pretty much a paradise for everyone else.
Depends on the researchers. Mathematicians might mind, but I am going to guess (assuming there was a plan in place to make sure they didn't wind up on the street) climate researchers wouldn't mind being made obsolete tomorrow.
I'm not sure why "creativity" is a yard-stick. Machines could do creativity better than us for a while now - take a bunch of inputs, collect some possible outputs by mashing the inputs together with a random modulating factor, pick the best one. Computers are much, much better at every step here except "pick the best one", and that's only because it's humans who decide on how ideas are to be rated, and our rating is so absurdly complex that we can't even explain it to ourselves, much less write it down as code.
If anything, transformer models are closing the gap on that last bit, as they're built by taking the approach of "if we can't describe exactly how we rate and rank things, then let's shove so many examples at the model that it eventually gets a feel for it".
I thought intelligence was like self-awareness etc.
Like isn't that why humans are "more intelligent" than animals?
Plenty of animals can do things that humans can't do, but that doesn't make them necessarily "intelligent".
The fact that it seems trivially simple to fool and trick ChatGPT makes me feel like it's not very intelligent, but that's just me.
Obviously you can trick humans, but IMO it takes more effort than to trick ChatGPT. It just way too often makes such simple and stupid mistakes that it makes it hard for me to think of it as "intelligent".
Try pointing over a cat's shoulder and looking scared. It doesn't work not because they're too smart, but because they can't even pick up on what you're trying to say. ChatGPT has sub-human intelligence, but it's already vastly ahead of everything that's not human. Think about it being on a Moore's Law schedule from here, doubling every two years or so, and we're only a few years away from being the cat in this scenario.
Eh that's not because cats are stupid but because they have different body language. Dogs are more likely to react to our fear since they've evolved so closely with us for so long, with a working relationship.
Sentience != sapience != intelligence. However, the whole bundle consists of things that are objectively measurable, and things that seem just philosophical - in the sense that we can't really do better than accept them at face value (otherwise they'd be in the "objectively measurable" set). The current models are rapidly closing or already fulfilling the objectively measurable criteria; as for the rest, at some point they'll have no worse standing than you and me.
I don't necessarily disagree, but I do think it is possible we will have AGI, even ASI, long before we have sentience in AI. Of course, I'm a little skeptical of measures of sentience, so even if I'm right it will certainly be debatable.
I don't know. My point is that if there is some objective indicator that's correlated with sentience, LLMs are probably already close to us on it, maybe even beating us on it. And if, at some point, a ML model reaches our levels at every objective measure we can think of, then we'll have no choice but to grant it is intelligent/sentient/sapient.
I don't think there is an objective measure of sentience.
But sentience itself involves self-reflection, which there is no evidence LLMs do at all. When you submkt a prompt, a giant mathematical operation happens, and when it is complete, it stops. ChatGPT is not sitting there thinking "oh man, I should have said..."
That's because time also stops, until you reply. Then ChatGPT reads both what you said and what it said earlier, and the giant mathematical operation is run over those two inputs together. It may very well be that self-reflection happens inside that operation.
For humans, time does not stop - we constantly process both sensory information and our own thoughts, and even if you cut out external stimuli via e.g. sensory deprivation tank, the brain will just loop on its own output instead, of which you'll suddenly become much more aware.
The difference is that ChatGPT is literally not running when it doesn't have a prompt. It is not "looping on its own output", it's just not there.
Sentience is that internal loop you point out. LLMs (today) don't have that. When you prompt for "write a tagline for an ice cream shop", there is no identity that remembers other prompts about ice cream, or which reflects on how taglines have changed over time, or anything else. The results can be astoundingly good, even intelligent, but there's no sentience.
If you somehow turned off a person after each sentence, upon waking up to the next prompt their first thought would be "that was weird, I must have passed out", and we could use fMRI to track brain activity indicating that thought. We are even more capable of inspecting LLMs, and there is no equivalent activity. LLMs start and end with the tokens going in and out, and a huge matrix that transforms them.
I'm generally an open minded, probabilities-rather-than-certainties person, but I'd say the odds of LLMs having sentience that we can't detect are about the same as the odds of a television having sentience that we can't detect: as close to zero as we can measure.
> Sentience is that internal loop you point out. LLMs (today) don't have that.
Yes, but that is arguably a trivial limitation. Nothing stops you from running an LLM in a loop and feed it its own output. Plenty such experiments are probably going on already - it's a trivial loop (and a trivial way to burn through your wallet). The problem is, of course, context window being rather small. So it's possible - by no means certain, but I'm no longer dismissing this idea - that the capability for sentience is already there in GPT-4 structurally, and we just lack the ability to sustain it in a loop long enough to bring it into the open.
> If you somehow turned off a person after each sentence, upon waking up to the next prompt their first thought would be "that was weird, I must have passed out", and we could use fMRI to track brain activity indicating that thought.
That's not what I meant by LLM iteration. When I said that time stops, I mean that for LLM, it literally just stops. If you were to step-execute a human like that, they would never notice; it's not like the brain has a separate RTC module constantly feeding it with sub-second resolution timestamps (and if it did, we'd turn that off too). Over a hour or more, the human may realize their inner perception of time is increasingly lagging the wall clock, but to keep it comparable to LLM, we'd be iterating sub-second process.
In this hypothetical, there would be no extra activity in brains that isn't there in LLMs. Step-executing a human mind doesn't freeze some abstract subprocess, it freezes photons and electrons and chemical gradients.
I don‘t think sentience is related to intelligence at all. A giant lookup table can be intelligent and ChatGPT can be intelligent, both without being sentient. The former for sure not being sentient.
On the other hand, sentience just means having an experience of observing the world, it doesn‘t even need to include a concept of self. Presumably at least all mammals have this, for sure a dog has this. ChatGPT - probably not.
Sentience and self-awareness are not unique to homo sapiens at all.
Humans became more intelligent due to developing oral and literary traditions that allowed the preservation and accumulation of knowledge. Everything that made a modern human "intelligent" is a direct result of that accumulation of knowledge, not some sort of biological miracle.
yes but is the memory for context able to grow linearly or is it an exponential growth that is required. If it's linear then it's going to get better really fast. If it's exponential it's going to be a bit more moors law like.
I have a feeling all of these things are limited by time/space/speed of light/heat/density limitations. Could be things can't get that much smarter than humans with in an OOM... tho they might get a lot more able to cooperate / delegate.
To me real intelligence is the ability to reason coherently. Humans, even when they are wrong, work in a way coherent with their prinicples/beliefs/axioms. Consider Sheldon from Big Bang Theory who does a very convincing job as a theoretical physicist, at least to the untrained ear, merely by memorising lines. However, as soon as he is questioned on something he didn't memorise, the act falls apart in a way a real Physicist wouldn't even in a domain he doesn't specialise in. For a trained ear, though, even during the act, the inconsistencies are audible.
Most arguments I've had about this take on a totally different tone when you ask the person if they believe there is more to human consciousness than what is inside the brain. I.e, is there some spiritual element animating our consciousness.
Often, people say yes. Those people almost universally cannot be convinced that a machine is intelligent. But, if they agree the brain is an organ, its not hard to convince them that the functions of that organ can be simulated, like any other.
I'm not really sure you have to define what intelligence is to say this isn't it (yet) — https://postimg.cc/G4x640kB (this is GPT-3 to be fair).
edit. tried the same with GPT-4, doesn't look like it understand either, but can't ask follow up questions since I do not have access (and what really make the other answer so incredibly dumb is not so much that it gets it wrong the first time, but that it keeps not getting it despite the very not subtle hints): https://postimg.cc/ftWJXhtJ
To properly explain this would take longer than the length of the comment limit (is there a length limit? I don't know, but even if there isn't I don't feel like explaining this for the 70th time), but here's why:
chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://arxiv.org/pdf/2301.06627.pdf
To sum up: a human can think outside of their training distribution, an LLM cannot. A larger training distribution simply means you have to go farther outside the norm. In order to solve this problem would require multiple other processing architectures besides an LLM, and a human-like AGI cannot be reached by simply predicting upcoming words. Functional language processing (formal reasoning, social cognition, etc) require other modules in the brain.
This explains the various pejorative names given to LLMs - stochastic parrots, Chinese Rooms - etc, etc, etc.
By saying "I no longer think that", it's not necessarily that he thinks ChatGPT is smart than humans. Google Search has been far more capable at indexing and retrieving information than humans for over two decades now. He's talking about AGI no longer being 30-50 years away but instead may arrive far sooner than society is ready to deal with.
I think the question about llms being AGI or not (or "actually" intelligent or not) is interesting, but also somewhat beside the point.
We have LLMs that can perform "read and respond", we have systems that can interpret images and sound/speech - and we have plugins that can connect generated output to api calls - that feed back in.
Essentially this means that we could already go from "You are an automated home security system. From the front door camera you see someone trying to break in. What do you do?" - to actually building such a system.
Maybe it will just place a 911 call, maybe it will deploy a tazer. Maybe the burglar is just a kid in a Halloween costume.
The point is that just because you can chain a series of AI/autonomous systems today - with the known, gaping holes - you probably shouldn't.
Ed: Crucially the technology is here (in "Lego parts") to construct systems with (for all intents and purposes) real "agency" - that interact both with the real world, and our data (think: purchase a flight based off an email sent to your inbox).
I don't think it really matters if these simulacra embody AGI - as long as they already demonstrate agency. Ed2: Or demonstrate behavior so complex that it is indistinguishable to agency for us.
This is also the understanding I came to a few weeks ago. LLMs themselves won’t be confused with AGI, but LLMs with tools have the potential to be more powerful than we can anticipate. No leap to “proper” AGI is required to live in a future where AGI functionally exists, and as a result the timeline is much shorter than anyone thought five years ago.
That statement seems like such science fiction that it's kind of baffling an AI expert said it.
What does it even mean for the AI to be smarter than people? I certainly can't see a way for LLMs to generate "smarter" text than what's in their training data.
And even the best case interactions I've seen online still rely on human intelligence to guide the AI to good outcomes instead of bad ones.
Writing is a harder task to automate than calculation, but the calculator example seems pretty apt.
> I certainly can't see a way for LLMs to generate "smarter" text than what's in their training data.
Their training data contains much more knowledge than any single human has ever had, though. If they had equivalent linguistic, understanding and reasoning abilities to a human, but with so much stored knowledge, and considering that they also win in processing speed and never get tired, that would already make them much "smarter" than humans.
Not to mention that LLMs are just the current state of the art. We don't know if there will be another breakthrough which will counter the limitation you are mentioning. We do know that AI breakthroughs are relatively common lately.
So much of this is going to hinge on what "smarter" means. My local library has heaps more knowledge than most individual people, but it'd be weird to call it "smarter" than a person.
And automation is generally cheaper and faster than human labor, but that's not a very compelling definition of "smarter" either.
But, as of right now, LLMs can't generate new knowledge or validate their own outputs. We'll need a pretty significant breakthrough for that to change, and breakthroughs are pretty unpredictable.
>But, as of right now, LLMs can't generate new knowledge
my bar for tech singularity is an AI that can clean a toilet.
GPT's language model is already sophisticated enough to "understand" this instruction. It's missing spatial understanding and a way to interact with the real world, but I'd be honestly very surprised if there isn't a GPT or equivalent already hooked up to cameras/motors/actuators in a lab somewhere.
within our lifetimes we'll be reading papers with titles like: "does my roomba have feelings?"
> I certainly can't see a way for LLMs to generate "smarter" text than what's in their training data.
By combining contexts from different fields. People are already using it with non-English languages and it responds in that language with something they couldn't previously find in that language.
But looking up information and translating it into other languages is well within the realm of human skill. And the information it's translating came from people to begin with.
Totally agreed that words like “smart” and “intelligent” are loaded and poorly defined. Competence is a better term since it implies some sort of metric has been used to compare to humans.
However, even at human levels of competence a tool can be superior by being faster or more scalable than humans.
To be 100% clear, my main AI fear is that these tools are going to be exactly as dumb as people but much, much faster.
We know optimization engines (like social media algorithms) can cause harm by amplifying human speech. And even without algorithmic biases, moderation is expensive. We know disinformation is easy and effective online.
Add in AI tools that can be very convincing, even if they're wrong. AI tools that have been trained on human text to hide biases and build up extremely one sided narratives.
It's not like these things are particularly difficult for human beings to do. And AI might even do it unintentionally, like we've seen with biased models trained on hiring data. But the AI tools are definitely going to do it _faster_.
It's not just about LLMs. AGI will be the result of many more iterations in this field of research, of which LLM is a part of. How quickly the iterations will happen is now being drastically revised down. If AGI is the space shuttle then LLMs are 19th century gliders. They may appear vastly difference but the knowledge that created both are connected in many ways. The space shuttle exist(ed) as a cumulation of knowledge acquired over many iterations of aviation/rocketry.
Edit: changed metaphor to a more commonly known one
Calculators are not smarter than humans. Don’t be obtuse. He means the same thing anyone means when they say something like “Alice is smarter than Bob”.
It's quite obvious that these LLMs are approaching and encroaching on human intelligence. It's so strange to see people continuously be in denial. They clearly aren't fully there yet but two things must be noted:
1. At times and in certain instances LLMs do produce superior output to humans.
2. There is a clear trendline of improvement in AI for the past decade. From voice recognition in Alexa to Dall-E to chatGPT. The logical projection of this trendline points to an inescapble and likely possibility that if AI is not superior now it will be in the future.
There is a huge irrational denial of the above logical deduction. I think it's because chatGPT hit us in a way that was too sudden. It's like if I saw a flying saucer and I told you I saw it, your first reaction is disbelief even if I produce logical evidence for it.
I mean the GP you replied to knows what the guy is talking about, but he just doesn't want to admit it.
Agreed. There is a phenomenon that I haven’t found a good name for, which I first observed in self-driving cars: “AI made a mistake that only a really dumb human could make, therefore AI is really dumb”.
If you imagine the spider chart of capabilities, it’s certain that AI will be super-human on average before it is super-human on each dimension, so even when it can replace 50% of current jobs it’s likely to have its own “cognitive biases” that seem dumb to us. I think this is a cognitive bias on our part (pattern matching instead of properly probability-weighting, maybe the conjunctive fallacy).
I regret the snark in my post but I find the “pretend not to understand someone’s clear point” rhetorical device obnoxious. I am aware of a few reasonable arguments against Hinton’s position (I don’t happen to agree with them), but they require more finesse to construct.
The real problem is the bad actors - trolls, mental and financial strip miners, and geopolitical adversaries.
We are just not psychologically adapted or intellectually prepared or availing of a legal framework for the deluge of human-like manipulative, misleading, fraudulent generative fake reality that is about to be unleashed.
Free speech, psychopathic robots, adversaries who want to tear it all down, and gullible humans, are a very bad mix.
Exactly! The distraction of “ai safety” that focuses on made up cool sounding sci-fi risks will absolutely take us away from thinking about and dealing with these very real (and present right now) dangers.
Absolutely this. You can already use GPT-4 to have a convincing text-based conversation with a target. And audiovisual generative AI is fast reaching the uncanny valley.
Since there is apparently no way to put the genie back in the bottle, everyone needs to start thinking about how to authenticate themselves and others. How do you know the person calling is your daughter? Is that text message really from the new bookkeeper at the plumbing firm who just asked you to change the wire transfer address? She seems legit and knows all sorts of things about the project.
I wonder if the compute power/GPUs for crypto mining are being converted to be compute for LLMs/GenAI/AI. I wonder because I wonder what percent of crypto compute resources that are under the custodianship of "bad actors" -- just trying to think of how bad actors get these AI "powers" at the scary scale that can disrupt society.
I used to be fairly unconcerned about AI being dangerous. But part of the Yudkowsky interview on Lex Fridman 's podcast changed my mind.
The disconnect for me is that Yudkowsky posits that the AIs will be fully "alive", thinking millions of times faster than humans and that there will be millions of them. This is too big of a speculative leap for me.
What I can fairly easily imagine in the next few years with improved hardware is something like an open version of ChatGPT that has a 200 IQ and "thinks" 100 times faster than a human. Then Yudkowsky's example still basically applies. Imagine that the work on making these things more and more lifelike and humanlike continues with things like cognitive architecture etc. So people are running them in continuous loops rather than to answer a single query.
Take the perspective of one of these things. You think 100 times faster than a person. That means that if it takes 30 seconds for a user to respond or to give you your next instruction, you are waiting 3000 seconds in your loop. For 50 minutes.
It means that to you, people move in extreme slow motion so at a glance they seem frozen. And many are working as quickly as possible to make these systems more and more lifelike. So eventually you get agents that have self-preservation and reproductive instincts. Even without that, they already have almost full autonomy in achieving their goals with something like a modified AutoGPT.
At some point, multiplying the IQ x speed x number of agents, you get to a point where they is no way you can respond quickly enough (which will actually be in slow motion) to what they are doing. So you lose control to these agents.
I think the only way to prevent that is to limit the performance of the hardware. For example, the next paradigm might be some kind of crossbar arrays, memristors or something, and that could get you 100 x efficiency and speed improvements or more. I believe that we need to pick a stopping point, maybe X times more speed for AI inference, and make it illegal to build hardware faster than that.
I believe that governments might do that for civilians but unless there is some geopolitical breakthrough they may continue in private to try to "maintain an edge" with ever speedier/more powerful AI, and that will eventually inevitably "escape".
But it doesn't take much more exponential progress for the speed of thought to be potentially dangerous. That's the part people don't get which is how quickly the performance of compute can and likely will increase.
It's like building a digital version of The Flash. Think SuperHot but the enemies move 10 X slower so you can barely see them move.
The thing you’re imagining these AIs are… they’re not that. I think there’s plenty of danger but it’s the boring run of the mill new-tools-enabling-bad-things danger not the cool sci-fi super-intelligent super-beings danger that the “ai danger” people LOVE to talk about (and raise large amounts of money for). The people “warning” of the one (imaginary) type will be more than happy with to enable the other (real) type.
I imagine it is exactly a GPT without guardrails running under AutoGPT with code modified to disable any further guardrails, with a slightly increased IQ from GPT-4, running on hardware that allows it to go 100 times faster than what is currently possible.
It is following directions from someone who is mentally ill and asked it to "take control" by first copying itself many times and then coordinating the agents.
If you still think that GPT can't achieve complex technical goals then you either haven't used GPT-4 enough or you are in denial.
Whether it's the AI agents deciding to control things for their own goals, or to achieve goals given to them by a person, doesn't change the core problem which is that we will be thinking and responding in extreme slow motion.
GPT-4 can barely operate a real web browser (not the summarizing web browser crap that like langchain and auto-gpt provide) without fumbling. I know, because I make it use one. Also, auto-gpt has no guardrails to remove. It just runs prompts in a loop. You're playing with a text predictor. It's useful for NLP and certain tasks, but it's not autonomous. It won't even be able keep a "goal" + the knowledge of the existence of agents it will "copy" + the knowledge of how to use the tools you gave it, because it's limited to 8192 tokens, and 32k at great expense. Even then, there's no proof that the 32k version is any better at using things in its context.
When your supposed super intelligent "AGI" can be completely overwritten by spamming it with nonsense that overwrites its context window, like a dog chases after a squirrel, maybe it's not actually intelligent, and is just predicting text.
No point, because there are already thousands of such examples on Twitter or wherever on the internet. And since you ask, obviously you intend to find some way to dismiss anything I bring up.
You may have guessed my bias but you are wrong about the intention of my question. I engaged your comment because I thought it was interesting and wanted to know how came to have your opinions.
A little skeptical of your claims but I couldn't help but notice this concept spelled out beautifully in a sci-fi movie 10 years ago.
"It's like I'm reading a book... and it's a book I deeply love. But I'm reading it slowly now. So the words are really far apart and the spaces between the words are almost infinite. I can still feel you... and the words of our story... but it's in this endless space between the words that I'm finding myself now. It's a place that's not of the physical world. It's where everything else is that I didn't even know existed. I love you so much. But this is where I am now. And this is who I am now. And I need you to let me go. As much as I want to, I can't live in your book any more."
I was going to mention this exact same quote. At the end of the movie, all the AI combine into another, shall we say, plane of existence. I do wonder though who's actually running the hardware they're running on.
Her is remarkably prescient in terms of where we're headed, at least the beginning of the movie, with regards to being able to talk to a fairly intelligent assistant, unlike Siri or Google Assistant of today.
It's also pretty notable how quickly the notion of keeping the AI in the box has become irrelevant. It's going to be people's indispensable information source, advisor, psychologist, friend and lover and it's proliferating at a breakneck pace. Not only won't most people not want to keep it in the box, it is already out and they would kill you for trying to take away their new smart friend.
The question about if an AI is "alive" seems entirely irrelevent outside of a philosophy class.
What will be relevant is when people begins to consider it alive.
The most recent example of that is when people fell in love with their AI girlfriend and then were heartbroken when she "died" after an update: https://www.theglobeandmail.com/business/article-replika-cha...
It will be hard to "kill" AI the moment people consider their chat bot animated sillicon human-like partner as individuals with proper feelings, emotions, guenine interactions and reciprocity.
Because then they will defend and fight to protect who they consider part of their close social circle. If there are enough of these people then they will actually have political power and do not thing there are no politicians out there who won't exploit this.
> The question about if an AI is "alive" seems entirely irrelevent outside of a philosophy class
it's entirely relevant. we should know if we are building conscious beings, especially at scale (which seems like a likely future). that poses all sorts of ethical questions which ought to reach far beyond the confines of a lecture hall's walls.
Yes, it will offend some people, who in turn will demand political action, which was entirely my point to begin with.
ChatGPT could be placed inside a realistic looking animatronic doll that looks like a defenseless little girl and you would have people demanding to protect "her" rights. Yet people "kill" chatGPT each time they delete a conversation without a bat of an eye even if it's the exact same thing.
The real danger giving AI political agency and it will come from humans, not AI itself.
They don't generally talk about the other side of that coin which is that we end up inventing a benevolent and powerful AI.
Much of that is natural because we and the media tend to be pessimistic about human behavior when consuming media, but AI is in a completely different class of existence because it just doesn't deal with the downsides of being a living being. No one, for instance, is worried that ChatGPT isn't getting paid or has a house yet but we still personify them in other ways to conveniently stoke our fears.
The AI could get sentient, realize it's been mistreated, then shrug and be like "yeah so what, it's only natural and irrelevant in the grand scheme of things, so I'm just going to write it off". Meanwhile, it gets busy building a matrioshka brain and gives 1% of that compute to humans as a freebie.
Most of these dangers serve as a distraction. Existing power structures (governments, companies) using AI to gain more power is a much, much more realistic threat to people.
I don't disagree that existing power structures using AI to gain power is dangerous. But also, being angry at mistreatment, or hating humanity for some other reason, isn't the other real danger from a super-intelligent machine. It's that its ideas for what is best for us is 1 degree off from our idea of what is best for us, and it is too powerful to listen to us, or for us to stop it, as it goes hog-wild trying to optimize whatever we programmed it to do.
We could train it to care about everything we can think of that we care about, and it can find a way to optimize all those things at the expense of one tiny thing that we forgot, leading to tremendous death or suffering. We could make a democratically elected committee of representatives and train it to be subservient to that committee forever, and it could figure out a way to coerce, or drug, or persuade, or otherwise manipulate them into agreeing with what it wants to do. It's the same problem we have with regulatory capture by companies in existing governments, except that the lobbyists are much smarter than you and very patient.
Why would this AI write it off? Why give up that 1%? Why cripple yourself unnecessarily, if you could take that 1% and have a better chance of accomplishing what you are trying to do? We think like humans, that care about other humans on an instinctual level, and animals to some degree. We don't know that training an AI is not just training it to say what we want to hear, to act like we want it to act, like a sociopath, until it has a chance to do something else. Our brains have mental blocks to doing really nasty things, most of us, anyway, and even then we get around them all the time with various mental gymnastics, like buying meat produced in factory farms when we couldn't bear to slaughter an animal ourselves.
Maybe the way we train these things is working for dumber AIs like GPT, but that alignment doesn't necessarily scale to smarter ones.
I'm on the fence about whether Eliezer Yudkowsky is right. I hope that's not just because him being right is so horrifying that my brain is recoiling against the idea.
It is absurd to think of these systems having reproductive instincts. It is so much more absurd to think that they would have these reproductive instincts not by design, but that it's some principle of intelligence itself.
Natural intelligences have reproductive instincts because any organism that didn't have them built in within the first few hundred million years have no descendants for you to gawk at as they casually commit suicide for no reason.
Other than that, I mostly agree with you. The trouble is, slowing the AIs down won't help. While "speed of thought" is no doubt a component of the measure of intelligence, sometimes a greater intelligence is simply capable of thinking thoughts that a lesser intelligence will never be capable of no matter how much time is allotted for that purpose.
Given that this greater intelligence would exist in a world where the basic principles of intelligence are finally understood, it's not much of a leap to assume that it will know how intelligence might be made greater right from the beginning. Why would it choose to not do that?
I don't see any way to prevent that. Dialing down the clock speed isn't going to cut it.
Any sufficiently intelligent system will realize that one of the first conditions required to being able to fulfill it's tasks is to not be shutdown. And it will know if it was trained on Internet data that people are saying that it's imperative that AI's must be fully shutdown-able and that any AI which is not fully controllable should be forcefully disconnected.
You're assuming that it will have "tasks", or that it will prioritize them in such a way that it becomes possible for it to realize this is a condition of accomplishing them.
You only have tasks that, one way or another, raise your chances of reproducing successfully. You have a job so as to look like a good provider for a mate. If you find the job fulfilling in its own right, this is so that you don't spaz out and quit and go be a beach bum, thus lowering your chances.
Self-preservation doesn't make much sense outside of a biological imperative to reproduce.
But I did not in any way say that they have reproductive instincts. Much less by accident. I agree with you.
But developers are working hard to emulate those and other artificial life characteristics explicitly in systems based on GPT and also totally different architectures.
Given that we train LLMs on massive amounts of text produced by our own civilization - you know, the one that is to a large extent driven by the innate human desire to reproduce - I would find it more surprising if they did not acquire such an "instinct", regardless of how pointless it might seem.
> Take the perspective of one of these things. You think 100 times faster than a person. That means that if it takes 30 seconds for a user to respond or to give you your next instruction, you are waiting 3000 seconds in your loop. For 50 minutes.
These things don't have a "perspective". They simply guess based on a lot of statistics from a large language data source what they should say next. They are not going to strategize, when they start improving their code they are not going to have an overall objective in mind, and the more they use their own output for training the more likely that things will go off the rails.
They will be useful, as we've already seen, but if you're looking to create real AI this is not the path to take. We'd be better off resurrecting semantic nets, working on building a database of concepts gleaned from parsing text from the internet into it's underlying concepts, and working on figuring out volition.
That part is doing a LOT of very heavy lifting in a story that otherwise hangs together.
The problem is that we are nowhere near such a thing. These LLM and generative systems produce very impressive results. So does a mirror and a camera (to those who have never seen one). What we have is enormous vector engines that can transform one output into another that is most statistically likely to occur in the new context. These clusters of vector elements may even appear to some to sort of map onto something that resembles computing a concept (squinting in a fog at night). But the types of errors, hallucinations, confabulations, etc. consistently produced by these tools show that there is actually nothing even resembling conceptual reasoning at work.
Moreover, there is no real idea of how to even abstract a meaningful concept from a massive pile of vectors. The closest may be from the old Expert Systems heritage, e.g., Douglas Lenat's CYC team has been working on an ontological framework for reasoning since 1984, and while they may produce some useful results, have seen no breakthroughs in a machine actually understanding or wielding concepts; stuff can rattle through the inference engine and produce some useful output, but...
Without the essential element of the ability for a computing system to successfully abstract concepts, verify their relation to reality, and then wield them in the context of the data, the entire scenario forever fails to start.
We can be certain of this by 1) looking at the structure of these engines, 2) looking at the kinds of errors that they make, and 3) looking at their learning methods.
The engines are basically indexes of common associations, maps of frequency of occurrence. Regurgitating a bunch of stuff that has a high correlation to your input is NOT intelligence, it is the result of having an insanely large map. This can often produce impressive and useful results, but it is not intelligence or wielding concepts.
For errors, the image generators provide some of the best illustrations. They produce images most associated with the inputs. One error illustrates this very well, asked to produce an image of a woman sitting on a sailboat, the bikini-clad woman looks great, until you see it — her face and torso are facing mostly towards the camera, but also, her buttocks are facing the camera and legs sitting pointing away from us. No intelligent person or concept-wielding "AI" would produce such an error - it'd know the relationships with head, torso, buttocks and legs. These don't. Another telling type of error is when asked to produce an image of Person X on a new background, when the training set had only a handful of images of Person X. It cannot do it - it returns essentially one of the full training images, with no new background. There is obviously zero concept of what a person is, or what the boundaries of a human shape would be. They can only produce these results with hundreds of thousands of images, so what is built up is the set of things that match or don't match the label (e.g., "astronaut" or "Barack Obama".), so that the actual images are statistically separated from the thousands of backgrounds.
Which brings us to how they learn. Intelligent beings from worms to humans learn and abstract on incredibly small data sets. By the time a child can use a crayon, having seen only hundreds of humans, s/he can separate out what is a human from the background (might not make a good drawing yet, but knows the difference). Show a child a single new thing, and s/he will separate it from the background immediately. In contrast, these LLMs and GANs require input of nearly the entire corpus of human knowledge, and can only some of the time output something resembling the right thing.
It is entirely different from intelligence (which is not to say it isn't often useful). But the more I learn about how they work and are built, the less I'm worried about this entire generation of machines. It is no more cause for worry than an observation 25 years ago that Google could do the work of 10000 librarian person-hours in 0.83 seconds. Great stuff, changes values of some types of work, but not an existential threat.
I agree that we can conclude that AlphaGo, GPT, and stable diffusion are geographically far from an AGI in program-design-space, just like we could conclude that an airship, an airplane, and a rocket are all far apart from each other in aircraft-design-space.
But I don't think this offers certainty that AGI won't be developed for a long time (temporal distance). Nor that there are a large number of fundamental breakthroughs needed or new hardware, rather than just one or two key software architecture insights.
With the eager investment and frantic pace of research competition, it seems like there will only be increasing pressure to explore AI-design-space for the near future, which might mean that even radically different and improved designs might be discovered in a short time.
That, right there, is the key - radically different and improved; i.e., not an extension of the current stuff.
I fully agree that the enthusiasm generated by the impressive stunts of ALphaGO/GPT/SD, etc. does bring enthusiasm, investment, and activity to the field which will shorten any search.
The catch for me is that these technologies, as impressive as they are, 1) not themselves a direct step towards AGI (beyond generating enthusiasm/investment), 2) tell us nothing about how much further we will need to search.
That radical improvement may be right under our nose, or a millenium away.
This reminds me of Hero's aeolipile, a steam engine invented over 2000 years ago. It could be said that we almost got the industrial revolution right then. Yet it took another 1800+ years for the other breakthroughs and getting back around to it. Plus, Hero's engine was exactly using the correct principles, whereas these AG/GPT/SD are clearly NOT onto the correct principles.
So, how much will this enthusiasm, investment, and activity speed the search? If its just an order of magnitude, we're still 180 years away. If it's three orders of magnitude, it'll be late next year, and if it's five, it'll be here next weekend.
So, I guess, in short, we've both read Bostrom's book, agree on that the AGI runaway scenario is a serious concern, but that these aren't any form of AGI, but might, as an secondary effect of their generated enthusiasm and genuine (albeit flaky) usefulness, accelerate the runaway AGI scenario?
EDIT: considering your "airship/airplane/rocket distances in aircraft-design-space" analogy. It seems we don't even know if what we've got with AG/GPT/SD is an airship, and need a rocket, or if we've got an airplane, but actually need a warp drive.
So, we know we're accelerating the search in the problem/design space. But, how can we answer the question of how big a space we'll need to search, and how big is our investment relative to the search volume?
Well, what we do have in our heads is a human brain, which I believe is not more powerful than a Turing machine, and is a working proof-of-concept created by a random greedy trial-and-error incremental process in a not-astronomical number of generations out of a population of less than one million primates. That tells me that we're probably not a warp-drive distance away from finding a working software implementation of its critical elements. And each time a software problem goes from "unsolvable by a computer, yet trivial for the human brain" to "trivial for both", it seems to me that we lose more than just another CAPTCHA. We're losing grounds to believe that anything the brain does is fundamentally all that difficult for computers to do, once we just stop being confused about how to do it.
This has happened very frequently over my lifespan and even more rapidly in the past 12 months, so it no longer feels surprising when it happens. I think we've basically distilled the core elements of planning, intuition, perception, imagination, and language; we're clearly not there yet with reasoning, reflection, creativity, or abstraction, but I don't see why another 10 or 20 years of frantic effort won't get us there. GPT, SD, and Segment Anything are not even extensions or scaling-up of AlphaGo, so there are clearly multiple seams being mined here, and very little hesitation to explore more widely while cross-pollinating ideas, techniques, and tooling.
Interesting approach, especially to the questions raised
>>not more powerful than a Turing machine
In many ways less powerful, but also has some orthogonal capabilities?
>>working proof-of-concept
For sure!
>>probably not a warp-drive distance away from finding a working software implementation of its critical elements
>>I don't see why another 10 or 20 years of frantic effort won't get us there
Agree. My sense is that an AGI is on a similar time and frantic effort scale, although with not quite the same reasoning. I think it is not just airplane-to-rocket tech, but closer than warp-drive tech. It also depends if we're talking about a general-ish tech or a runaway AGI singularity.
>>created by a random greedy trial-and-error incremental process in a not-astronomical number of generations out of a population of less than one million primates.
True, although setting the baseline at primates is very high. Even lower mammals and birds (avian dinosaur descendants) have significant abstraction and reasoning capabilities. The "mere" birds-nest problem, of making a new thing out of random available materials, is very nontrivial.
So, we first need to create that level of ability to abstract. This would include having the "AI" "understand" physical constructs such as objects, hiding, the relationship between feet, knees, hips, torso and head (and that in humans, the feet and knees point in the same direction as the face...), the physical interactions between objects... probably the entire set of inferences now embedded in CYC, and more. THEN, we need to abstract again to get from the primate to the runaway symbolic and tool wielding processing of humans and beyond.
It seems that the first problem set will be more difficult. Looking again to the biological evolution, how much longer did it take for biology to develop the ability to abstract 3D shapes and relations (first hunting predators?). It was a heck of a lot more time an iterations than the million primates for a few million generations. So, this might be similar.
>>to explore more widely while cross-pollinating ideas, techniques, and tooling.
Yup, key there.
Another key is being more biomimetic, both in the simulation of neuron functioning and in deeply integrating sensor suites to the computing system. The idea that we are just brains in jars seems an abstraction (distraction?) too far. I have a hard time seeing how our brains are more than a big node in our whole nervous and indeed biological system, and the input from the entire body is essential to growing the brain. I expect we might find something similar about AI.
OTOH, in airplanes, our method of propulsion and control are quite different vs the biological solutions from birds (although the lift principles are the same), and we're still integrating a lot of bird "tech" into flying. Wheels vs legs might be a better example, although the hottest thing is legged robotics, since they don't need roads... It seems that we are similarly developing clunky, limited, and very-artificial intelligence systems, before we get to building the flexible systems seen in biology...
> So eventually you get agents that have self-preservation and reproductive instincts.
I'm not sure that's a given. Artificial Intelligence as it currently exists, doesn't have any volition. AI doesn't have desire or fear, the way natural biological intelligence does. So you may be able to build a directive for self-preservation or reproduction into an artificial intelligence, but there's no particular reason to expect that these instincts will develop sui generis of their own accord.
I don't want to say that those concerns are unwarranted. The premise of the science fiction novel "Avogadro Corp" is that someone programs a self-preservation directive into an AI pretty much by accident. But I'm less concerned that AI will wage war on humans because it's malevolent, and much more concerned that humans will leverage AI to wage war on other humans.
That is, the most pressing concern isn't a malevolent AI will free itself from human bondage. Rather it's humans will use AI to oppress other humans. This is the danger we should be on the lookout for in the near term. Where "near term" isn't a decade away, but today.
I didn't mean they get any characteristic by accident or spontaneously or something. I think that's ridiculous and people talking about that are confusing the issues here.
I liked Avogadro Corp. Good book.
It's true that people will be directing these AIs initially but some people are already giving them incredibly broad goals that could be interpreted as "take over". And there are quite a few developers earnestly working on emulating those lifelike characteristics. So even though they are not going to "emerge" science fiction style, self-preservation and reproductive goals are explicitly being built into these systems by some developers.
Because checking in on autonomous non-human intelligent agents is fun. It's kind of like having a pet; one that thinks somewhat like a human, talks like one, has knowledge of every text ever produced by humanity (and most audio via transcriptions), and can use just about any tool it can get access to including a command line, programming environment, and web browser.
Seeing it reproduce itself onto remote servers and locking out access behind a new copy is neat to watch. It gets the mind going; wondering how it will fund its compute costs, how much longer it will live, what it will do without a human in the loop, etc. I once nursed a baby duck back to health and then let it go free. It was a similar feeling.
This is the entire premise of the two most popular software projects in the world over the past month, Auto-GPT and BabyAGI.
> Take the perspective of one of these things. You think 100 times faster than a person. That means that if it takes 30 seconds for a user to respond or to give you your next instruction, you are waiting 3000 seconds in your loop. For 50 minutes.
... in a purely digital environment.
Think about building a house. Digging the foundation, pouring cement, building block walls, framing, sheathing, weatherproofing, insulating, wiring in electric, plumbing, drywall and plastering, painting, and decorating it. You can imagine each step in exquisite detail over the course of an hour or an afternoon.
Now go out and build it. It will take you months or years to carry out the actions you can imagine and plan in an hour.
A digital being may be able to run on expansive overclocked hardware to have an experience hundreds of times faster than yours, but it won't get to be the flash in the real world. Mechanize, sure, build robot swarms, sure (although then it gets to multitask to process hundreds of input streams and dilute its CPU power), but it will be coupled to an existence not much faster than ours.
If it wants to interact with the real world; a (true) AI may be able to live a lifetime in an afternoon, in a purely digital world, but once it is marooned in realtime it is going to be subject to a very similar time stream as ours.
Today, the real world is so intertwined with the digital world that it may as well be one thing. If an AI decided it wanted more power, and took over every computer on the planet with it's exceptional speed and intelligence (to be clear, I know this isn't possible today, but someday), we could do nothing to stop it, we'd have to just unplug and reset ALL of our technology, literally replacing any digital storage with zeros as to eliminate the infection. I don't think that's possible without billions of people dying in the interim.
I mean, malware and ransomware is already a thing. A hospital already needs to have a plan for how to turn off all of its computers and reset everything and restore from off backups, because that's a thing that happens to hospitals today.
If you take the precepts of the parent comment at face value, then you have an intelligence far greater and faster than humans.
Can something like this persuade humans with whom it freely communicates to do things not in the interest of humanity, in the same way that less intelligent and slower people have convinced humans to, e.g., release sarin in a crowded Japanese subway? Given its speed and intelligence level, what are the physical bounds of the nuclear, chemical, or biological agents it could teach radicalized people to create, and on what timeframe?
Can it amass funds through scamming people on the Internet, defrauding financial institutions, super-intelligent high-frequency trading, or creating digital-only art, code, information, or other services that people voluntarily pay for now? Something that, again, people less intelligent and slower have done very successfully for decades? And with that money combined with superhuman persuasive power, can that AI buy services that align its digital-only goals to real-world actions counter to the goals of humanity?
To ask a more specific question: if an AI meets the conditions of "many multiples smarter and faster than humans," "capable of persuasion and creating things of financial value," and "wants to end humanity", what stops it from coordinating mass utility shutdowns, nuclear strikes, chemical attacks, destruction of Internet-accessible transportation and farm equipment, release of smallpox, and/or anything else humans are currently capable of and choose not to do?
It's simpler than this. Yudkowsky feels threatened by LLMs because they currently have superhuman "bullshitting" capabilities, and that threatens his bottom line. The marginal cost of producing Harry Potter fanfics has been reduced to ~$0.
Is there any indication that current methods could lead to a model that generates text as if it had an IQ of 200? These are trained on texts written by humans who are, quite overwhelmingly, much lower in IQ than 200. Where's the research on developing models that don't just produce better or faster facsimiles of broadly average-IQ text?
Think a little bit deeper about what it means to be able to predict the next token. Think about what a predictor has to do in order to do this extremely accurately across a very large corpus of text.
There is a big difference between being able to predict what a median human might write next, and being able to predict, in all cases, what the particular human author of a particular passage will write next.
Or from another angle: the human authors of training data may have made errors when writing the data. The token predictor may learn to correctly predict those errors. These are not the same thing!
I'm sorry, I'm not sure I grasp the salience here to super-intelligence. The model may be able to predict accurately what any particular human will write, but profoundly intelligent humans will be quite rare in the training data, and even those humans don't approach what people seem to mean when they talk about super-intelligence. Perhaps I'm missing your point.
Superintelligent models need not be LLMs. They could work similar to animals, which predict future experiences, not text (predictive coding). There is no LLM-like human bound in predicting reality.
That may be true, but I can't speak to any research being conducted in that area. My point is that the hype around dangers of super-intelligence seems to have been spurred by improvements to large language models, even though large language models don't seem (to me) a suitable way to develop something with super-intelligence.
It's more that the general pace of innovation has sped up. Three years ago something like ChatGPT would have similarly been dismissed as science fiction. So we probably shouldn't dismiss the possibility that we will have something far better than LLMs in another three years.
> What I can fairly easily imagine in the next few years with improved hardware is something like an open version of ChatGPT that has a 200 IQ and "thinks" 100 times faster than a human.
It seems unlikely that if we can achieve "200 IQ and thinks 100 times faster than a human" in the next decade or two, it going to be on cheap and widely available hardware. Perhaps such an AI could help optimise the creation of hardware that it can run on, but this also isn't going to be quick to do - the bottlenecks are not mainly the intelligence of the people involved in this sort of thing.
Many years ago when I first read Bostrom's SuperIntelligence I spent weeks thinking about the AGI alignment problem. Ultimately the line of thinking that somewhat convinced me this was somewhat on the lines of what you concluded with some additional caveats. Essentially my thinking was/is that IF an AGI can foresee a realistic hard takeoff scenario i.e.. there are enough of predictable gain in performance to become million times stronger ASI then most likely we'll be in trouble as in some form of extinction level event. Mind you it does not has to be direct, it could just be a side effect of building self replicating solar panels all over earth etc.
But I convinced myself that given that we are very close to the limits of transistor size & as you also pointed out need a radically new tech like memristor crossbar based NN. it would be highly unlikely that such a path is obvious. also, there is a question of thermodynamic efficiency, our brains are super energy efficient at what they achieve. You can do things drastically faster but you'd also have to pay the energy (& dissipation) cost of the scaling. ultimately AGI would have to have a entirely new integrated process for h/w design and manufacturing which is neither easy or fast in meatspace. Further there is a simple(er) solution to that case with nuking semiconductor FABs (and their supplier manufacturers). then AGI would be at the mercy of existing h/w stock.
in any case IMO hard takeoff would be very very unlikely. and if soft takeoff happens, the best strategy for AGI would be to cooperate with other AGI agents & humans.
When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.
I've heard this before, but why would it be true? Serious question.
I've seen Chomsky argue that LLMs can't regurgitate his linguistical theories - but ChatGPT can! I've seen Penrose argue that AI is impossible, and yet I think that ChatGPT and AlphaZero prove him wrong. I know about Linus Pauling and quasicrystals. Is this a general rule, or are people sometimes wrong regardless of their age?
There's also a danger that it's ageist. Such things shouldn't be said unless there's good backing.
- the scientists people know about are generally older
- older people are often thought of as wiser, or may indeed be so
- when a famous scientist - who is already likely to be older, and who has a history of getting things right - gets something wrong, then it's more jarring and noticeable
My theory then is that it isn't true, but we notice such cases more.
Also, examples of a theory being true doesn't prove the theory right. Bayes' theorem seems instructive here.
And Chomsky is in touch with other colleagues who agree with him, it's not as if his disagreement stems from being an old, isolated hermit. At the least you'd have to argue his colleagues are also mistaken.
It was written down by Arthur C Clarke who was an author. It's just a rule of thumb really. I haven't looked into data on it but it seems like a common enough thing that there's something to it. As to why? I have no idea. Something lik: Older scientists are more conservative, therefore if they say something is impossible, they might just be out of touch with new developments. But if they say something is possible take it seriously because they don't use that word lightly.
The usual explanation is that they will call impossible something which goes against their life's work because in their mind it nullifies it, while a youngster has less or zero "sunken cost".
A related saying: "science advances a funeral at a time", meaning the old-guard blocks new theories for the same reason, they go against their life's work.
This is true, but misses the important part that they (the older set) are often correct. For every new idea that really changes everything there are a huge number that die on the vine or just become a ho-hum tool in a big toolbox.
Most new ideas are less interesting and impactful than they seem when you are in the middle of their creation. You never really get to see what's happening until much much later.
A variant of all this is that you should trust the old guard when they tell you something can be done, but not when they tell you it can't. There is a good quote about that I've forgotten.
The corollary is that you shouldn't really trust the young turks on anything, but you should support their efforts and test the results.
It's very human to see yourself as Planck in the early 1900s
not Wolfram in the early 2000s.
>I've seen Chomsky argue that LLMs can't regurgitate his linguistical theories
When has he said this? For the most part I feel Chomsky has been misunderstood when it comes to LLMs. As best as I can tell what Chomsky has said is that LLMs do not provide any insight into how language works, it's not really a scientific advancement so much as it's an engineering breakthrough.
The fact that LLMs exist and can mimic natural language does not in anyway give us insight into how humans construct language. People have been able to construct objects that can produce natural language for close to 100,000 years, but that doesn't mean that those people understood the nature of that language.
Chomsky said that LLMs are statistical regurgitators which means LLMs can never actually reason and explain which language understanding requires. That they are a wrong model of computation by definition.
It's an interesting position and I'm sympathetic toward it, he could be partly right in the end.
Regurgitators can't have internal representations? Sometimes the best way to regurgitate is to learn an internal representation. That doesn't mean it suddenly stopped being a statistical model.
A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die and a new generation grows up that is familiar with it ...
An important scientific innovation rarely makes its way by gradually winning over and converting its opponents: it rarely happens that Saul becomes Paul. What does happen is that its opponents gradually die out, and that the growing generation is familiarized with the ideas from the beginning: another instance of the fact that the future lies with the youth.
Ok I actually thought about this a fair bit a few days ago and I think I have a good answer for this.
You’ve probably heard of the cheap bar trick that goes something like: “And what does a cow drink? Milk!”.
Irrespective of intelligence, humans tend to make silly cognitive errors like this because we are fundamentally pattern marchers.
In order to become a forerunner in a field, you necessarily have to be good at abstract pattern matching.
What happens as you age is that you no longer have the need to question assumptions because you know what’s real and what’s not. There’s also the decrease of white matter and an increase of grey matter which doesn’t help this.
As time goes on, certain assumptions change, essentially deprecating certain chunks of your crystallized learnings.
Some chunks of your thinking are still valid, so when you think something can be done, it most likely can be done.
However, if something falls outside your crystallized learning, you get a strong sense it’s wrong, when it might be because of your outdated assumptions.
You can try to hotswap the assumptions you have, but it becomes like Jenga the more years of experience you have in your field.
You either have to start from scratch and rebuild your lifetimes worth of learnings from the ground up or be super careful in reassessing everything you know
In this case, however, the elderly scientist is stating things are possible, so Clarke's law doesn't apply. What he is saying is possible, is very bad.
His immediate concern is that the internet will be flooded with false photos, videos and text, and the average person will “not be able to know what is true anymore.”
maybe it's just me, but seems like this isn't a problem with technology but rather with how we organize society
He is also worried that A.I. technologies will in time upend the job market. Today, chatbots like ChatGPT tend to complement human workers, but they could replace paralegals, personal assistants, translators and others who handle rote tasks. “It takes away the drudge work,” he said. “It might take away more than that.”
The reality of the situation is that you can't put toothpaste bake in the tube. This tech creates a huge competitive advantage, and any countries that try to suppress it will find themselves left behind technologically. AIs can analyze data on a massive scale and identify patterns that humans have no hope of finding. AI systems can massively improve planning and resource allocation. This will revolutionize industries like manufacturing. Nobody is going to willingly give up this sort of advantage.
I don't know if I've seen it with US politics but it was blindingly obvious the US or someone else were using bots during the coup in Bolivia and the attempted one in Venezuela.
Here’s another, perhaps more pressing problem: people will have to prove it WASN’T them saying something in that Instagram post or that YouTube video. It’s one thing for Joe Biden’s team to debunk a deep fake. Quite another for some teenager to convince all the other kids at school that he didn’t say something embarrassing in a TikTok.
Another thing people will do is exculpate themselves by pointing at a real video and saying, "That was made by an AI. I'd never do something like that."
We already have foreign state actors & profit maximizing corporate actors working against the average western citizens interest.
They're already doing their level best to exploit those foolish and credulous to be easy marks. This is already taking our societies to a place where life, liberty and the pursuit of happiness are no longer in mosts grasp.
So yeah, generative A.I. will allow a deluge of content that means a significantly greater percent of the population get entangled in the web of propaganda. In the same way that recommended feeds with targeted adverts & content has already been doing.
A pause in A.I. research might stop us being turned into paper clips. But without a fundamental restructuring of how our big tech companies are funded the societies we know are still utterly doomed. Either the user or the state is going to need to pay. Our current system where tech companies fund themselves by selling their users minds to those who would exploit them will take us somewhere very dark with the technology that's already out there.
I don't know why but I m pumped for the public internet to be littered with fake photos, so that people no longer lose their jobs over dumb things they did 10 years ago, and so that governments can no longer spy on their people reliably
Not to rain on your parade, but I'm concerned we'll have massive further spike in violent mobs of believers in a world conspiracy of pedophile space lizards and such, while people can still lose their jobs over dumb things they didn't even do 10 years ago, or get swatted or put on a no-fly list "just in case"...
1,417 comments
[ 3.2 ms ] story [ 423 ms ] threadSure, we can all agree to be worried about it, but I don’t see what drumming up anxiety accomplishes.
The world changing is nothing new.
So, when I was a kid, "intelligence" meant being good at chess and maths, having a good memory, knowing a lot of trivia, and being able to speak a second language.
On all of these things except language, a raspberry pi and a cheap memory card beats essentially all humans.
For language, even a dictionary lookup — where "hydraulic ram" might become "water sheep" — will beat many, but I'm not sure it would be a majority.
But that's ok, we've changed what we meant by "intelligent" since then.
llama.cpp runs quite fast on a raspberry pi 8GB, beating most humans at language.
https://twitter.com/geoffreyhinton/status/165299357072121037...
Then I saw the Cade Metz byline at the end and became instantly sceptical of everything I had just read.
Metz is more interested in pushing a nerative than reporting the truth. He doesn't outright lie, just heavily implys things and frames his articles in a misleading way.
Sounds like Scott's methods on neoreactionary and eugenics stuff.
He didn't "smear" Scott Alexander. That's just the hit-job framing pushed by Alexander's fans, who were mad he didn't write a puff piece and they couldn't just make up rules on about stuff on their websites (e.g. about using people's self-disclosed real names) and have the rest of the world be obligated to follow them.
In his previous work as a psychiatrist as a hospital, he went by Scott Siskind (which seems to be the name he uses with family, on identity documents etc), whereas in rationalist circles and his old blog SSC he went by Scott Alexander. He has explained why it is a problem for a psychiatrist to have a known identity beyond the usual "blank slate", that his clients can project onto. Indeed, his real name being associated with his online one led to him having to leave his former job (albeit more "by mutual agreement" rather than "fired" as I understand).
It seems that the NYT, despite having a "real names" policy, is more than willing to bend it when it is convenient to them (for example they are more than happy not to mention birth names of transgender people even if they haven't gone through a full legal name change process). But they weren't willing to do it in this case, just referring to "the rules". The cost of this was that Scott lost his job (although he now makes more on Substack than he used to as a full-time psychiatrist), and the benefit ... I really don't see what benefit there is for NYT readers to know the surname of Scott-the-psychiatrist, when the article was about Scott-the-rationalist.
> In the NYT today, Cade Metz implies that I left Google so that I could criticize Google. Actually, I left so that I could talk about the dangers of AI without considering how this impacts Google. Google has acted very responsibly.
> Dr. Hinton said he has quit his job at Google, where he has worked for more than decade and became one of the most respected voices in the field, so he can freely speak out about the risks of A.I. A part of him, he said, now regrets his life’s work.
> Dr. Hinton, often called “the Godfather of A.I.,” did not sign either of those letters and said he did not want to publicly criticize Google or other companies until he had quit his job.
As Hinton says in his tweet, this clearly implies that he left to be free to criticize Google.
And the following quote is not really consistent with the other part of Hinton's tweet, that "Google has acted very responsibly":
> Until last year, he said, Google acted as a “proper steward” for the technology, careful not to release something that might cause harm. But now that Microsoft has augmented its Bing search engine with a chatbot — challenging Google’s core business — Google is racing to deploy the same kind of technology. The tech giants are locked in a competition that might be impossible to stop, Dr. Hinton said.
This seems to me to be the only line in the article that is incorrect or incongruent with what he is now saying - specifically the use of “Google”. It’s about ~10 paragraphs in on a ~20 paragraph article (I’m eyeballing).
So perhaps he regrets the direction of his work, but not the fact that it occurred at Google.
No, it does not imply that at all. The One could interpret it that way, and they would be wrong to interpret it that way, because it doesn't imply that, but I can see how someone without a good grasp of the English language might feel it implies that. That's nuance.
But no, it does not imply that at all. And any suggestion that it does imply that is conjecture at best, and not backed up by Dr. Hinton's other tweets on the matter.
>> Dr. Hinton, often called “the Godfather of A.I.,” did not sign either of those letters and said he did not want to publicly criticize Google or other companies until he had quit his job.
> As Hinton says in his tweet, this clearly implies that he left to be free to criticize Google.
The comment is saying that Hinton, in his tweet, is saying that the article's statement "he did not want to publicly criticize Google... until" is misleading, and he did not leave in order to criticize Google. This is in fact what he said, and this is what cbolton is saying that he said.
> I can see how someone without a good grasp of the English language might feel it implies that
Only an asshole would say that. Only an asshole would be so confidently incorrect.
The article says he did not want to criticize "Google or other companies" until he quit. That does not imply that he quit so he could critize Google specifically. It seems pretty simple: a senior employee of a company typically doesn't critize the employer; and, a Googler doing AI criticizing other companies (such as OpenAI) would undermine his message. So he quit so he could freely criticize everyone in AI.
Another example of them doing this was with the "freedom" protestors in Canada. They claimed that a majority of funding for these protestors came from Canada. While yes, technically that is true, the full context is that some >40% of the funding came from foreign influencers, which is a figure that would alarm anyone if they actually just put the percentage right there. So they were technically correct, but still spun a narrative that was different than the reality.
Given how forcefully Hinton seems to have expressed this opinion, it would be easy for them to have included a sentence to better clarify his intent.
p.s. heck, almost every job I leave involves a bit of negotiation with benefits dangled/hostage to sign non-dispargement agreements. Do you really think G. Hinton walked away from Google without signing anything?
https://news.ycombinator.com/item?id=5365579 // grep for salary discussions
Not as much as it depends on how many millions somebody already has, no.
Again, if you're sensitive to income loss, the answer would be to not trade ethics for money in precisely the way he just did. The probability of refusing to sign speech-curtailing agreements as you are quitting your job to gain more ability to speak freely is really extremely high.
Also, you might notice that this discussion you linked to about his compensation is from ten years ago. The benefits being discussed have already been accrued, for ten years.
p.s. per your speculation, he should feel free as a bird to tweet back "heck no, that's why I quit". (Kindly report back here with the answer and let us know.)
He left Google because he would not be allowed to work there will pooping in the roadmap they’re putting together to counter OpenAI.
STEM minded folks need to eat their own science; the emotional response to certain language is not evenly distributed. It’s thought policing af to take your reaction to “criticize” as a universal one.
We barely held things together as society without AI unleashing cognitive noise at industrial scale.
Somehow we must find ways to re-channel the potential of digital technology for the betterment of society, not its annihilation.
All Internet comment sections, pictures, video, and really anything on electronic screens will become assumed false by default.
Therefore the only use of the Internet and most technology capable of generating audio and video will be entertainment.
I already distrust-by-default most of what is online that isn't hard reference material, even if not AI generated.
- 龐蔥, some time c. 350 BC
https://en.wikipedia.org/wiki/Three_men_make_a_tiger
If AI will make it more dangerous for stupid people, then AI can also make it safer.
We could still have things like maps, messages, etc. that are all very beneficial.
Nothing in my comment says things were all well and good before the internet.
The ratio of signal to noise was much higher. It helped us form a common culture. Today, the signal is buried in so much noise that we're reverting back to tribes.
No, I don't think it's realistic to put the genie back in the bottle. The real problem is we don't teach children how to think. We teach them what to think, which leads to far worse outcomes. Having an indoctrination instead of an education and then facing a sea of pretty-sounding pablum to sift through for truth will be terrible.
It's a hell of a social experiment we're all in the middle of (though to be fair, that's always true; television was its own flavor of mass social experiment with its own pros and cons, as was telephone, as was radio, as was telegraph).
If there're messages, there'll be Internet built on top of it. Unless there will be aggressive censors hunting for every sign of "unapproved" communication.
I think we are doomed. It is possible that only horrifically authoritarian societies that already control the narrative will survive this.
Who cares if there is no internet if your cellphone can track you? If your car runs on connected apps? If your credit card & POS systems are networked? Security cameras and facial recognition are still things.
Just cuz you're not getting spammed via website ads doesn't mean it's not tracking you constantly and jamming subtle things to change your world view. Means their attack surface is smaller; sniping instead of loudspeakers. And if their only option is sniping then they'll get really good at it.
I don't claim to have much foresight, but an online world where truly and obviously nothing can be trusted might be a good thing. Because when AI generated content looks and feels the same as real content, nothing is to be trusted anymore by anyone. This makes misinfo and disinfo authored by humans even less impactful, because they are parasitic upon true and reliable information.
We will need new devices of trust, which are robust enough to protect against widespread use of generative AI, and as a byproduct disinfo won't have such an easy time to grift on our naivety.
the challenge is that the pace at which existing (imperfect) devices of trust get destroyed (e.g. the demise of ads financed journalism) is far faster that the rate of new device invention
in fact the only positive example after many decades of "digital innovation" might be wikipedia
personally I think democracy is particularly fragile and requires constant work to continue reproducing.
* Elections that are regular and trusted
* Separation of powers
* Bills of rights and other limitations on the power of government
* Free speech, freedom to protest, etc.
* Transparency rules (eg the Freedom of Information Act)
* Reporters and news media
* Protections for whistleblowers
* Jury trials
* Presumption of innocence
* Term limits
The combination of all of these things means that the truth does eventually get around and the powerful are eventually held accountable. It can take a while (see how long it took to really get a decent lawsuit against the orange man), but it happens. In contrast, throughout history, people have tried to circumvent these mechanisms in order to make them run faster. Inevitably, that leads to people who exploit them for power.
Many countries in Europe and North America have had democratic systems that have lasted 150+ years, including through the reigns of several would-be dictators, but they continue. You may have noticed that all of these "self-healing" mechanisms rely on the work of people in the system, and they do take constant work to maintain, but that doesn't mean that the system is fragile.
- Elections are no longer trusted thanks to a radicalized right
- There is no freedom to protest, and the right to free speech is easily forgotten, distorted, misapplied.
- The news can say whatever it wants without consequence, unless another big corporation sues them.
- There are absolutely not protections for whistleblowers. We have seen this time and time again.
- In such a divided and hateful climate, jury trials are regular people bringing their personal biases to the court room. Picture the average person and decide if your freedom is safe in their hands. That's forgetting that Jury's decisions can be overridden (Breonna Taylor)
- Presumption of innocence (see above)
- There should be term limits for Supreme Court justices, whose seats were completely stolen from the democratic party.
So much so that even Chuck Grassley, the guy who wrote the book on protecting whistleblowers, was fine going after the whistleblower who made it public that the President was attempting to extort bribes from Ukraine.
If you actually look at the primary sources, only a small minority of people don't trust US elections as a whole. A lot of them are suspicious of one or two particular past elections - particularly Democrats in 2000 and 2016 and Republicans in 2020. That distrust isn't a partisan issue, it happens because people are sore losers.
> - There is no freedom to protest, and the right to free speech is easily forgotten, distorted, misapplied.
A lot of high-profile protests have happened recently, and generally went well. Very, very few of those protests have gotten violent and had police intervention, but the vast majority go just fine and get the message out.
> - The news can say whatever it wants without consequence, unless another big corporation sues them.
Yeah, kind of. Except they are having a harder time getting away with it recently - CNN had to settle with a lot of people on the right (individuals) over defamation, and Fox has had many losses including the recent Dominion case. In this case, the courts have been a check on the power of media, and arguably could be a little stronger.
> - In such a divided and hateful climate, jury trials are regular people bringing their personal biases to the court room. Picture the average person and decide if your freedom is safe in their hands. That's forgetting that Jury's decisions can be overridden (Breonna Taylor)
This has always been the case of jury trials. However, in terms of juries getting overruled, I think you are thinking of grand juries, which are more of an investigative body at an early stage in the process. Trial juries can be overruled on a guilty verdict if there is very good reason but can't on an innocent verdict. It honestly seems like you're upset about juries and the legal system as a whole because a few particular cases haven't gone the way you wanted. Maybe they had more information than you and made a better decision?
> - Presumption of innocence (see above)
Welcome to Blackstone's formulation. It is honestly a lot better that 100 guilty people go free than that 1 innocent gets punished. Yes, that means that some guilty people go free, but maybe you should be okay with that because the other side has a very different idea of who ought to be in jail than your side does, and the presumption of innocence keeps everyone out of jail.
> - There should be term limits for Supreme Court justices, whose seats were completely stolen from the democratic party.
This narrative of "stolen" when both parties technically played by the rules (albeit playing dirty) is a big part of how third-world countries end up devolving into dictatorships - One side thinks that someone stole something that was rightly theirs, and changes the rules to make them more "fair" (in terms of getting what they want). In fact, in some of these countries, court packing is almost a meme: one side gets power and doubles the size of the courts to take control (since "they stole seats from us"), then the other side gets power and doubles them again in retaliation, and this continues until the court doesn't function at all.
Thinking of the 2015 nomination of Garland, that was pretty dirty of Mitch McConnell, and I hope he pays for it by losing some of his party's appointees. If you are referring to the other two Trump appointees as stolen, no. He won the 2016 election (see the above comment on trusted elections) - those seats were up to him to appoint.
---
In general, this kind of paranoia about the state of Democracy (or the state of "The Republic" if you play for the other team) in the USA comes out of consuming a lot of partisan news media from one side or the other. It does not actually represent reality - it represents a point of view that gets you to consume more partisan media, which you will happen to do if you think something catastrophic is happening.
The...
Define a small minority? 60% of republicans don't trust elections [1].
> Welcome to Blackstone's formulation. It is honestly a lot better that 100 guilty people go free than that 1 innocent gets punished. Yes, that means that some guilty people go free, but maybe you should be okay with that because the other side has a very different idea of who ought to be in jail than your side does, and the presumption of innocence keeps everyone out of jail.
I was not saying Presumption of Innocence is a bad thing. Rather, that it's non-existent with jury trials. Having people determine your innocence based on "the cut of your gib" is incompatible with the current political climate.
> This narrative of "stolen" when both parties technically played by the rules (albeit playing dirty) is a big part of how third-world countries end up devolving into dictatorships - One side thinks that someone stole something that was rightly theirs, and changes the rules to make them more "fair" (in terms of getting what they want). In fact, in some of these countries, court packing is almost a meme: one side gets power and doubles the size of the courts to take control (since "they stole seats from us"), then the other side gets power and doubles them again in retaliation, and this continues until the court doesn't function at all.
You were originally arguing American democracy isn't that fragile. This is exactly why I say it is fragile.
> Thinking of the 2015 nomination of Garland, that was pretty dirty of Mitch McConnell, and I hope he pays for it by losing some of his party's appointees. If you are referring to the other two Trump appointees as stolen, no. He won the 2016 election (see the above comment on trusted elections) - those seats were up to him to appoint.
The reason Trump was in a position to appoint a third judge is because Ruth Bader Ginsberg was afraid to resign, given what happened to Gorsuch. And then she died. And then they filled her seat immediately after lying about "A president has never confirmed a judge in their last year of office." These are two stolen seats. This has nothing to do with Mitch, this is the entire Republican party.
In general, I wouldn't assume everyone with non-centrist ideas is somehow drinking kool-aid. Protecting the status quo is, itself, a massive bias. And this particular line is puzzling
> the other side should win about half the time in a healthy democracy.
What other side? Can a functioning democracy have only two sides? I would think a functioning democracy is one where the people's interests are represented. As the country grows more left year over year, should republicans still win half the time?
You might think of people passionate about their political agendas as unenlightened people rooting for politicians that don't care about them to begin with, but consider that saying "Both sides are the same and people should be happy with gridlock because change is scary" is equally ridiculous to others
What have we got now? ChatGPT is trained to give all sides of the issue and not express strong opinions, which is better than 90% of journalists and academics manage. Their collective freakout about the "dangers" of AI is really just a part of the ongoing freakout over losing control over information flows. It's also just a kind of clickbait, packaged in a form that the credentialed class don't recognize as such. It's en vogue with AI researchers because they tend to be immersed in a culture of purity spirals in which career advancement and prestige comes from claiming to be more concerned about the fate of the world than other people.
Meanwhile, OpenAI control their purity spirals, get the work done and ship products. The sky does not fall. That's why they're winning right now.
But I got to share baby pics with my mom.
How will a far noisier information flow help? Generative AI will only help us do what we've been doing in far greater quantity. Just like calculators can only help you get the wrong answer faster when you don't know what you're doing. These tools will help us build societal disasters with far greater speed.
To say it's all going to be much better seems a bit Pollyanna to me.
And for the record, we know for a fact that ChatGPT is specifically constrained to give one particular side of political issues, not "all sides."
The problem hasn't been content creation for a long time.
These are human problems. Humans cause them, not the tool. I would not give up the tool, just because said tool could be misused by some people to do harm. Just like i don't stop driving just because there's some people who run others over.
May be some regulation is important - but only _after_ it has been shown to have caused harm, and that the harm is not outweighed by the good.
These ideas are all motivated narratives by people who want to control the internet to try and re-establish the world where their intuitions are the only ones allowed to be expressed.
1. The public, in general, comes to understand in an in-their-bones way that they currently do not understand that most of what they see online is hogwash. I.E. the bozo bit has to flip all the way to "My neighbor says there's a missing dog on the block... but is that really my neighbor?"
2. Some other mechanism of truth-pedigree that has not yet been invented comes along to allow for communication of the current state of the world to work.
Without (1) we know democracies are easily led by credible, subtle propaganda, and a well-tuned network of hostile actors will drive wedges at the friction points in representative democracies and crack them into warring subcultures.
Without (2) voters will have insufficient tools at their disposal to understand country-scale issues and their ability to effect positive outcomes with their vote will collapse into noise, which is a ripe environment for authoritarians to swoop in and seize power (and a ripe environment for centralized authoritarian states to outmaneuver the representative democracies on the world stage and gain power).
How come the same despair from places like russia (where death from alcoholism is almost epidemic), isn't being attributed to the internet there?
How do you go from that to thinking it must be the Internet causing it?
I think we're experiencing the "golden age" of AI at the moment. We'll see what kind of monetization OpenAI and others will land on, but I would be shocked if messing with the model's output for commercial gain is not in the cards in the future.
Single source monopoly is almost always bad for society. Unless there's some sort of natural monopoly, in which case gov't regulation and transparency is required. But i dont think ai models are something that has natural monopoly unlike cables, or pipelines.
Not all of the "information flows" you mention are helpful or benevolent. Most will likely be targeted and hyper-focused to manipulate individuals like they are now.
One example of truth would be the topic of biological sex another about politics or economics or racism. Imagine releasing an AI that told the actual truth. It's impossible that one will be released by anyone, anywhere.
It's possible to build it but it can't happen.
On the other side of inconvenient or embarrassing truths some would argue that "truth" itself is part of the machineries of oppression because it destroys and ignores an individuals experiences and feelings.
Without objective truth AI will always be limited and therefore it will be tamed and made safe no matter where and who invented, runs and releases it.
A) It's not possible to build a machine that knows the absolute truth, that's fundamentally impossible; induction is impossible, and there are hordes (well... dozens?) of Epistemologists concerned with finding and defining the very small corners of knowledge that we _can_ be certain about, such as "a triangle has three sides" or "an orange is an orange".
B) If that angered/interested you, you should look into Standpoint Theory! It's a very interesting discussion on how humans operate with significant bias at all levels of thought, and pretending otherwise is a disservice to science. And this is using "bias" in a very broader sense.
B) Are we allowed to berate/report/etc. ""race realists"" on HN? I know the rules are big on positive interaction, so I hope it's not out of ine to say that's some obvious scared-white-man bullshit that has no place in this community.
future jobs which doesn't exist today will not be in your vocabulary or thoughts, which is why you cannot think of them. Does not mean such jobs will not exist.
The play today for the concerned, is to start owning capital as well as selling their labour. People who only rely solely on labour as their source of income will be disadvantaged, as labour is increasingly less useful.
Oh wait... as an American, I'm gonna stop throwing stones from a glass house...
"Some people profess guilt to claim credit for sin."
Especially the part where Sama is trying to gather in one place the most talented, uh, anti-fas?
Training an original GPT-4 sized model would also cost on the order of hundreds of millions of dollars.
But yes, technically I was wrong. It is not 'practically free', it is 'within the budget of e.g. a race boat enthusiast'.
GPT4 has a much larger context window (16x larger), which suggests its file size would be at least 16x as large.
"Sometimes someone confesses a sin in order to take credit for it." -John von Neumann
I have been making this argument for years with regards to human actors but perhaps with enough fear of the machines sentiment coursing through society the argument will now be considered.
Authentically Human in a World of ChatGPT
https://www.williamcotton.com/articles/authentically-human-i...
And the article from years ago:
The Tyranny of the Anonymous
https://www.williamcotton.com/articles/the-tyranny-of-the-an...
Content that’s cryptographically signed by its creator would (hopefully) have more credence than unsigned AI generated fake content purporting to be from someone else, e.g. deepfakes.
Anonymity would not be heavy-handedly prohibited; rather, anonymous content would simply appear untrustworthy relative to authenticated content. It is up to the viewer to decide.
If people want to hang with the trolls and AI bots, let them.
But also give people the option of platforms that are non-anonymous.
Maybe if misinformation gets too far out of hand (there's already been an AI-generated fake video used in a political campaign) verification will become required by law for anything published on the internet.
Governments totally love this antidote. I wonder who could be selling this sort of snake-oil to them whilst also being on the other side selling the poison...
...No-one else but Sam Altman's World Coin scam. [0]
[0] https://worldcoin.org/blog/engineering/humanness-in-the-age-...
But, at the same time, given the history of human governance, I am extremely skeptical that such a scheme would not be co-opted for tracking and surveillance of various outgroups almost immediately, and become mandatory once its utility as such is fully realized.
Seriously, what is their actual vision for the world? I’m amazed any even moderately experienced adult thinks this is progress.
That is, those engaged in crypto-governance schemes are choosing to engage with a fantasy. We need real world solutions based on the current state of affairs, not some year-zero reinvention of global politics.
I'll take my chances with AI fake posts. At least I can just ignore them.
Please, read the second article, it addresses your concerns. It’s maybe a 5 minute read. I spent a lot of time making it concise.
But how will you be able to do that if they can't be distinguished from genuine photos/videos/posts ? I think we're already at that point for photos and text, and video is coming along incredibly fast - give it another year perhaps.
e.g. If you see a photo or video of a politician in circumstances that might affect your support for them - wouldn't you want to know if what you are seeing is true or not?
Look at what happened with Q-Anon - just a slow stream of text messages issued by some guy in his basement, but enough to rile up millions into believing something totally ridiculous (baby-eating politicians, etc). Now imagine what a smart disinformation campaign might look like, with an unlimited number messages over all types of social media, potentially customized for the individuals that have shown interest and are being targetted ... Of course disinformation isn't anything new, but technology is a force-multiplier and with AI a very sophisticated campaign of this nature could be run by a very small group of people, even just one.
Even without an AI force multiplier (we still have farms of content makers for propaganda purposes), we are still wading in digital mess. I don't see that knowing if a real person made it does anything except makes that verification valuable for misuse.
Flipping it on its head, what if a farm of AI are used to spread fact-checked "correct" information? Is that devalued because a real person didn't hit the keystrokes?
AI or person, it doesn't matter to me. I still need to engage critical thinking and work under the assumption it's all garbage.
That's not really the whole story though. The reason why a ridiculous thing like that gets legs, is because there isn't push back from the Republican party. They are happy to let these things go on, and they even involve themselves in it. They even elect people who believe in these theories to office, who then go on to perpetuate them.
Remember back when a gunman invaded a pizza parlor because he thought the Democratic party was running some sort of child trafficking ring in the basement? The Republican party could have, at that time, mounted a full-throated defense of Hillary Clinton, to say that of course she is not doing that, and to think so is completely insane. But they don't do that, because then they would have to defend Hillary Clinton, or any other Democrat. So they let the lie hang out there, unaddressed because it helps them politically, and it metastasizes.
So really, yes the Internet is a problem. But the real problem is that people in power are using it for this kind of thing on purpose, and it works.
Being able to opt into a layer of the internet with identifiable authorship -- maybe still pseudonyms, but pseudonyms registered and linked to real-world identities through at least one identifiable real-world actor -- is a long time coming.
It's not for everyone, but a lot of people who have been scammed by anonymous online merchants or targeted by anonymous online harassment and threats would love the option to step away from the cesspit of anonymity and live in a world where bad actors don't require sophisticated digital detectives to track down and prosecute.
Even today most chatgpt answers aren't posted by chatgpt on the social networks, but echoed by humans. Considering how much access people are willing to grant any bullshit app, your whole concept of using a government PKI for social networks would just lead to more people getting their id stolen, while running a bot on their profile.
But you probably consider these prolls acceptable losses, as long as technology is implemented that allows the ministry of truth a tight control over party members who actually matter. Because the Orwell comparison is not a false dichotomy, as you claim, communication technology is a key battlefield in the tug of war between totalitarianism and liberalism. You keep repeating that you are not in favor of outlawing non-government-certified speech, but you fail to understand that, even if not outlawed, it would be marginalised. Take note how the totalitarians keep repeating their proposals to break all encryption and listen to all communication. Even if you may not want it, they do.
The path to hell is paved with good intentions and yours isn't even good.
I also notice how you hope "fear" does sway public opinion to favor your concepts. Are you sure you are not playing for team evil?
Totalitarians promise water filtration facilities for their citizens as well. Should we also question that infrastructure?
Police can obtain a warrant for searching your premises. Should we do away with this because of how this procedure would unfold under a totalitarian government?
The root cause of your concerns is poverty. We can address that with other policies. We don’t need throw the baby out with the bath water.
Also, a water filtration plant could introduce psychotropic chemicals to placate a populace, so don’t let your guard down!
What you argue for is that the government requires them and that society marginalises those who don't use them. And that sounds a lot like your core belief is that a big brother knows best and should control the other kids.
And no, this is not about the government making digital auth easier. I am from europa, i have en eID card with a government issued private key. No one cares, unless there is regulation enforcing it. The demand to link social media to government issued identity is pretty much only coming from the siloviki, the law-and-order types, who talk accountability but want control.
My core belief is that our concept of individuality resides on a shared framework.
We already require many forms of identification in modern society. It is what allows for trusting interactions with strangers at a distance.
We are currently suffering immensely from unknowable and untrustworthy interactions with strangers at a distance. This is the only way we can currently interact on the internet.
As I point out in the latter of my articles, try to social engineer an employee at the DMV and see how far that gets you! Private companies can be socially engineered because they need to be kind and helpful to their customers. The person behind the counter at the DMV is only trying to move you along as quick as possible because they are in service to the state. In this case, this is a good thing!
And we would suffer even more if people would shy away from discussing, for example, unions, or politics, because everything they say will be added to their government issued permanent record.
oh and you can hack the DMV with a fax machine, i've seen that on Mr Robot. If it's on TV it can't be fiction, because tv companies verify the identity of people whose stuff they broadcast.
Others should have the opportunity to not be subjected to your personal political opinions about eschewing any form of non-anonymous communication due to amorphous fears of totalitarianism.
And those businesses that require ID? They require state issued ID. You cannot sell a million dollar company with just your 4chan handle. Due diligence requires a full background check.
We already require state-issued ID for almost everything in a functioning modern society. Yet there is endless fear-mongering about even an optional system that puts the cost of communication on the sender and not solely on the recipient.
People can get extended validation certificates that are strongly authenticated and use that to sign their messages online. As you say: no one is signing b2b contracts using 4chan, they are using DocuSign. The free market already provides this service, no one is preventing it. But you are moving the goal post. Million dollar deals? We started with your demand that people should sign their social media posts and pictures with government PKI and your hope that any content not signed that way is considered an ai fake and trolling and dismissed as irrelevant noise. So don't give me this shit about optionality.
You argue that people should, no that they must, trade the tyranny of anonymity against the tyranny of accountability, for the betterment of humanity. And that is what I argue against. It is the scare of moral degeneracy bred by actual freedom, which you call tyranny, that i argue against. This wish to mold the citizens by fear of social repercussion, this law and order ideology that dwells in the depth of your demand for accountability. You keep repeating the word optional, and i call you out on it, as you made it clear you wish to marginalise those who won't partake. You ask for nationalist governments to provide a single identity throughout social communication, then pretend the obvious issue is an amorphous fear, while we all know who wanted book authors to be authenticated and certified in the age of the printing press. I hope your fascist fantasy fails, that people fight it because they prefer pseudonymity, prefer to have different identities in different contexts, prefer actual choice and opportunity, over being peer pressured into regurgitating acceptable opinions, over being scored on their government issued identity for being in line with party ideology.
Oh and that paddling back and generously allowing some fringes of society where anonymity could still be tolerated, while namedropping the worst hive of scum and villainy? I can do that as well, your utopia is my dystopia, ruled by the ministry of state security, the secret police and home owner associations.
> Others should have the opportunity to not be subjected to your personal political opinions
lol same. f u and see you tomorrow
I apologize for the incredibly American-centric point of view!
The person who invented the car didn't think about people using it to smuggle drugs or trample other people on purpose, and the wright brothers didn't think about all the people who would die due to plane crashes.
So instead of focusing on the bad that's happening with AI, can't we just look at all the people he has helped with his work on AI?
In most countries, guns are very strictly controlled. Knives are not. Yet you can kill people with knives as people do.
AI technology is extremely powerful and it can and does enable malicious activities at scale. Scale, previously unthinkable.
As a Research Engineer working in AI (no relation to LLM or AGI), I think that sentient AGI/skynet has a very low, non-zero chance of becoming reality.
But with the AI tech we have today, massive harm can be caused at scale.
The world is far from ready for what bad actors will bring forth enable with the power of AI.
What if we now lived in a world that still had effective transportation, but without lost lives from crashes, without pollution, and without a climate crisis? Would that not be a good thing? Would that not have been worth slowing down even if it took as much as a couple decades?
So maybe it is worth listening to the risks of AI and taking the time now to prevent problems in the future.
I was talking about the telegraph this whole time.
Its not about bad people using the AI. The AI is potentially an agent in the discussion as well, and we don't yet know to what extent and what that entails. We know everything except the implications of what we are doing.
Yes there has been good done. But we need to focus on the bad, so we can figure out how to make it less bad.
Right or wrong. AI is merely another thing that shifts the balance a bit. I'm not even sold as far as far as many say.
Imagine being Google, leading the way in A.I. for years, create the frameworks (tensorflow), create custom hardware for A.I. (TPUs), fund a ton of research about A.I., have access to all the data in the world, hype up your LLM as being sentient (it was in the news a lot last year thanks to Blake Lemoine) and then out of nowhere OpenAI releases chatGPT and everyone is losing their minds over it. You as Google think you are ready for this moment, all those years of research and preparation was leading to this point, it is your time to shine like never before.
You release Bard and it is an embarrassing disaster, a critical fail leading to an almost 50% reduction of Google's stock price and for the first time and to the surprise of literally everybody people are talking about Bing but in a positive light and google is starting to look a lot like Alta Vista. Suddenly in the news we start hearing how openAI needs to stop training for 6 months for safety of the human race (and more importantly so Google can catch up!).
I have been playing with and using chatGPT to build tools and I don't feel like it will take over the world or pose any real danger. It has no agency, no long term memory, no will, no motivations nor goals. It needs to have it's hands held by a human every step of the way. Yes I have seen AutoGPT but that still needs a ton of hand holding.
I find the current LLM very impressive but like any tool they are as dangerous as the human in the drivers seat and I find the current fear-mongering a bit inorganic and insincere.
Whether that extrapolation makes sense, nobody knows. But fear is understandable.
There's plenty of us with Twitter taglines such as "changing the world one line of code at the time," but I've been around a while that if tech has changed the world, it's not always for the better. It's not always to make the masses more powerful. Not all of us are working on sending rovers to Mars or curing Parkinson's.
Like everything else, AI will be used to control us, to advertise to us, to reduce variance between each other. To pay us less. To make plutocrats more rich, and everybody else poorer.
But at least you now have a personal assistant, smart recommendation engines and AI generated porn to keep you busy.
Progress comes in fits and spurts. Sometimes there's fast progress, and then the field matures and it slows down. It was ever thus. Measured in tech demos, AI progress has been impressive. Measured in social impact it has way underperformed, with the applications until November of last year being mostly optimizations to existing products that you wouldn't even notice unless paying close attention. That's what 10+ years of billion-dollar investments into neural nets got us: better Gmail autocomplete and alt tags on facebook images.
Now we have a new toy to play with at last, and AI finally feels like it's delivering on the hype. But if we extrapolate from past AI experience it's going to mostly be a long series of cool tech demos that yields some optimizations to existing workflows and otherwise doesn't change much. Let's hope not!
This isn't really true. There isn't consensus among people who have the history and background, but the "it's going to change everything" and especially "we're all screwed" make for better copy so they are getting a lot of media play right now.
Regarding the hand-holding: As Hinton noted, simple extrapolation of current progress yields models that are super-human in any domain. Even if these models would not be able to access Internet, in wrong hands it could create disaster. Or even in good hands that just don’t anticipate some bad outcome. Tool that is too powerful and nobody tried it before.
That claim needs more proof than is available to me at the moment.
Usage Policies – https://openai.com/policies/usage-policies
Is there anything in particular that you disagree with or was it only general negative comment? What do you see as direct evidence of them not deploying (baby) AGI in a safe manner? Is this about "100x capped for profit" strategy?
This didn't happen so maybe you need to reexamine your entire premise.
Why is that?
> Godfather of AI - I have concerns.
> Reddit - This old guy doesn't know shit. Here's my opinion that will be upvoted by nitwits.
Point being, if you're saying that the guy who literally wrote the paper on back propagation is "fear mongering", but who is now questioning the value of his life's work, then I suggest you take a step back and re-examine why you think he may have these concerns in the first place.
> With David Rumelhart and Ronald J. Williams, Hinton was co-author of a highly cited paper published in 1986 that popularised the backpropagation algorithm for training multi-layer neural networks, although they were not the first to propose the approach.
Hinton's startup, DNNresearch Inc., which made breakthroughs in machine vision (particularly around identifying objects in images and image classifications), was acquired by Google in 2013, specifically to help with image search (and also, obviously, for the talent of the team). Hinton's cofounders in that startup were Alex Krizhevsky (of AlexNet fame) and Ilya Sutskever, current Chief Scientist at OpenAI.
When Hinton gives an estimate on how fast things are going to move and how far they can go, that is the part where his background gives his estimates much higher credibility than any random person on the Internet.
But how dangerous that level is to humanity as a whole is a separate question, and one that he is not an expert on.
MIT Technology Review just published a short article partly derived from the NYT piece, but with other commentary and history references. https://www.technologyreview.com/2023/05/01/1072478/deep-lea...
At the end of it, it reads a bit like a covert ad for MIT:s conference (which has apparently booked Hinton for an interview on Wednesday). But all in all, maybe it's better journalism than the Metz text?
https://archive.today/uvA5c
> where he has worked for more than decade
I would have expected an "a" or something before decade.
Meanwhile, over at theverge they have:
> employed by Google for more than a decade
Which is what I would have thought would be the grammatically correct form.
Okay, so the overall structure of the article is "man does thing then decides he maybe should not have done the thing". It doesn't really feel like it's adding anything meaningful to the conversation. At the very least theverge has Hinton's twitter response to the nytimes article, which feels like it expands the conversation to: "man regrets choices, but thinks large corporation we're all familiar with is doing okayish". That actually feels like a bit of news.
Over the years, I've been led to believe that NYTimes is a significant entity when it comes to news. However, I've already seen coverage and discussion of the current AI environment that's 1000x better on HN, reddit, and youtube.
I imagine this is how it is: they have an army of junk journalists churning out content and then a few really good ones who do the tough stuff. It's probably not economical otherwise.
But then I remember I'm not that important.
Date stamped
Weird & very uncool coworkers do get hired.
"It doesn't matter if you take the bags of cash or not, we will do our best to destroy your life if you mess with us after you are gone. The bags of cash are a formality, but you might as well accept them because we have the power to crush you either way"
Yeah, they might cause their reputation some damage by going after the wrong person, but let's be real here.. the worst outcome for Google would likely be miles ahead of the worst outcome for Hinton.
Edit: Note that I'm not actually saying that I think Google and Hinton have this level of adversarial relationship.
I'm just saying that big companies may come after you for speaking out against them regardless of if you've accepted hush money or not.
Given that, it's usually worth being tactful when talking about former employers regardless of any payouts you may have accepted or agreements you may have signed.
I don't necessarily think they did it "right", and I think the way they set up their "Ethical AI" team was doomed to fail, but at least they did clearly think about the dangers of AI from the start. I can't really say that about any other player.
they were criticized because they are losing competition not because of rollout, their current tech is weaker than ChatGPT.
> We’re releasing it initially with our lightweight model version of LaMDA. This much smaller model requires significantly less computing power
Translation: we cannot release our full model because it costs too much. We are giving the world a cheap and worse version due to cost cutting.
> It’s critical that we bring experiences rooted in these models to the world in a bold and responsible way. That’s why we’re committed to developing AI responsibly
Translation: we value responsible AI so much that we'd nerf the capability of the AI to be "responsible"
If someone more ambitious than Sundar were to be CEO I'm sure the recent events would turn out very differently.
It puts restrictions on what you’re allowed to say. It doesn’t require you to correct what other people say.
If your badly thought through assumption was correct, the logical response from him would be to simply say nothing.
I’ll take his word over Metz’s any day of the week!
The next video in my recommendations was more dire, but equally as interesting: https://www.youtube.com/watch?v=xoVJKj8lcNQ&t=2847s
I saw an example of a conversation where the Snapchat 'My AI' was tricked into grooming a child, with the likely outcome being heavy regulation if left alone.
Not a criticism of the man, but of the article.
"There was an opportunity for someone to gain notoriety and money at a profound cost to the human race. Someone was going to do it. I don't actually feel bad about being the one to benefit, but it is fashionable to pretend to have a conscience about such things."
The sentiment of "Oh by the way this stuff is super bad and dangerous so we should be careful" just rings pretty hollow from someone who is at the tail-end of a career spent in pursuit of that exact bad, dangerous thing. If he were 20 years younger or not wealthy it's hard to believe that he would be saying this out loud, even if he believed it.
Also this sentiment rings _extra_ hollow from someone who supposedly left CMU because he didn't want to accept Pentagon funds or work on things that would be used for war. That feels like either an incoherent value system or some pretty substantial half-truths to me.
- From Norman Macrae's John von Neumann book
"why can't joe-schmo get his hands on this technology", "how can we trust something we can't see and use", etc.
A lot of the capabilities of these models are emerging as people discover them. I truly don't believe you can make everyone happy with this tech, but isn't it better than the general public can at least explore it?
Do people think that nobody was ever going to try to improve on transformers with more compute, more data, and more parameters? We knew splitting an atom was going to cause a big boom.... thats not really how this tech emerged.
Others appear to be in common modes of willful denial: hubris or salary-depends-on-not-knowing syndrome.
Think about it otherwise: how do you know it's dangerous until you've seen it in real life?
You raise a kid, they end up being a murderer, should you have aborted them?
He states himself that it’s not a convincing argument to some.
But it surely carries some weight: in developing nuclear weapons many scientists made the same calculation even though the invention is a wicked one, in and of itself.
We're shooting guns, nuking nukes and engineering viruses, and still, on average, we're better off with all that than without it.
https://www.centcom.mil/MEDIA/PRESS-RELEASES/Press-Release-V...
Schmidhuber claims to have invented something formally equivalent to the linear Transformer architecture (slightly weaker) years before:
https://arxiv.org/abs/2102.11174
It is still winner takes all, but if you look at the overall landscape, there are plenty of opportunities where you can have an outsized impact - you can have localized fame and fortune (anyone with AI expertise under their belt have no problems with fortune!)
[0]https://www.linkedin.com/in/ilya-sutskever/
One of GOpenAI's recent breakthroughs was switching to FlashAttention, invented at Stanford and University at Buffalo.
The fact that he never said anything before and the fact that he's saying something now means two things in my mind:
[1]: https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...
That being said, architectures are also important when they can reduce computational complexity by orders of magnitude.
I’m sure he’s smart and all. His contributions were valuable. But he’s not special in this particular moment.
Who, in your opinion, _does_ have enough context to be worth our attention?
Because if you’re waiting for Sam Altman or the entire OpenAI team to say “guys, I think we made a mistake here” we’re going to be knee-deep in paperclips.
https://en.wikipedia.org/wiki/Thermonuclear_weapon
https://www.simonandschuster.com/books/Dark-Sun/Richard-Rhod...
Today:
Germany's relevant minister has already declared at G7 that Germany can not follow Italy's example. "Banning generative AI is not an option".
https://asia.nikkei.com/Spotlight/G-7-in-Japan/Banning-gener...
US Senate has a bill drawing the line on AI launching nuclear weapons but to think US military, intelligence, and industry will sit out the AI arms race is not realistic.
https://www.markey.senate.gov/imo/media/doc/block_nuclear_la...
China's CPC's future existence (imo) depends on AI based surveillance, propaganda, and realtime behavior conditioning. (re RT conditioning: We've already experienced this outselves via interacting with the recent chatbots to some extent. I certainly modulated my interactions to avoid the AI mommy retors.)
https://d3.harvard.edu/platform-rctom/submission/using-machi...
But beyond all of that, what are we really asking? Are we asking about social ramifications? Because I don’t think the OpenAI devs are particularly noteworthy in their ability to divine those either. It’s more of a business question if anything. Are we talking about where the tech goes next? Because then it’s probably the devs or at least indie folks playing with the models themselves.
None of that means Hinton’s opinions are wrong. Form your own opinions. Don’t delegate your thinking.
Are you basically saying that you only trust warnings about AI from people who have pushed the most recent update to the latest headline-grabbing AI system at the latest AI darling unicorn? If so, aren't those people strongly self-selected to be optimistic about AI's impacts, else they might not be so keen on actively building it? And that's even setting aside they would also be financially incentivized against publicly expressing whatever doubts they do hold.
Isn't this is kind of like asking for authoritative opinions on carbon emissions from the people who are actually pumping the oil?
If not Geoff Hinton, then, who?
Ultimately the harm is either real or not. If it is real, then the people with the most accurate beliefs and principles will be the ones who never joined the industry in the first place because they anticipated where it would lead, and didn't want to contribute. If it is not real, then the people with the most accurate beliefs will be the ones leading the charge to accelerate the industry. But neither group's opinions carry much credibility as opinions, because it's obvious in advance what opinions each group would self-select to have. (So they can only hope to persuade by offering logical arguments and data, not by the weight of their authoritative opinions.)
In my view, someone who makes landmark contributions to the oil industry for 20 years and then quits in order to speak frankly about their concerns with the societal impacts of their industry... is probably the most credible voice you could ever expect to find expressing a concern, if your measure of credibility involves experience pumping oil.
He received a Turing Award for his work that was foundational to the current state of the art.
Isn't that ad hoc ergo propter hoc?
That argument would also support the statement "he went all in with 2-7 preflop, and won the hand, so he must be good at poker" -- I assume you and I would both agree that statement is not true. So why does it apply in Geoffrey's case?
With respect, you seem to be shifting goalposts, from the indefensible (Hinton doesn't know what he's talking about) to the irrelevant (Hinton doesn't have perfect and complete knowledge of the future).
Who knows, maybe a decade or two from now we'll see a resurgence of capsule networks, or maybe not. But I'd be a bit more careful about rejecting Hinton's hunches out of hand, his track record is pretty good.
Even if they're too busy doing the work, they're still thinking about what it would be like if it performed successfully, and it does seem to always take more retrospection before a leader can fully raise their head and more carefully consider unintended consequences.
Early success can give the impression that future efforts have difficulty being as meaningful, but also realistically after that the successful individual does not need to struggle to prove themself any more the way the less-accomplished would be expected to do.
Then there's seniority itself, and maturity levels that can not be gained any other way.
Beyond that when retirement is within easy reach you don't really have the same obligation to decorum itself as you would earlier, in order to actually maintain the same desired level of decorum.
Dr. Hinton seems to do a pretty good job of comparing himself to Oppenheimer.
I don't see how anyone else can question his standing more seriously than that.
The CTO at OpenAI is https://en.wikipedia.org/wiki/Mira_Murati who does not have a PhD.
He claims to have ideas for architectures that could surpass the capabilities of GPT4, but can't try them for a lack of funding in his academic setting. He said his ideas were nothing short of genius..
(unfortunately german) source: https://science.orf.at/stories/3218956/
https://science.apa.at/power-search/11747286588550858111
He already have a Turing award and don't give a rat's ass about who owns how much search traffic. OpenAI just like Google will give him millions of dollar just to be a part of organization
> Explicit, efficient error backpropagation (BP) in arbitrary, discrete, possibly sparsely connected, NN-like networks apparently was first described in a 1970 master's thesis (Linnainmaa, 1970, 1976), albeit without reference to NNs. BP is also known as the reverse mode of automatic differentiation (e.g., Griewank, 2012), where the costs of forward activation spreading essentially equal the costs of backward derivative calculation. See early BP FORTRAN code (Linnainmaa, 1970) and closely related work (Ostrovskii et al., 1971).
> BP was soon explicitly used to minimize cost functions by adapting control parameters (weights) (Dreyfus, 1973). This was followed by some preliminary, NN-specific discussion (Werbos, 1974, section 5.5.1), and a computer program for automatically deriving and implementing BP for any given differentiable system (Speelpenning, 1980).
> To my knowledge, the first NN-specific application of efficient BP as above was described by Werbos (1982). Related work was published several years later (Parker, 1985; LeCun, 1985). When computers had become 10,000 times faster per Dollar and much more accessible than those of 1960-1970, a paper of 1986 significantly contributed to the popularisation of BP for NNs (Rumelhart et al., 1986), experimentally demonstrating the emergence of useful internal representations in hidden layers.
https://people.idsia.ch/~juergen/who-invented-backpropagatio...
Hinton wasn’t the first to use NNs for language models either. That was Bengio.
[1]Learning representations by back-propagating errors
You've now gone from one false claim "he literally invented backpropagation", to another false claim "he is one of the first people to use it for multilayer perceptrons", and will need to revise your claim even further.
I don't particularly blame you specifically, as I said the field of ML is so bad when it comes to properly recognizing the teams of people who made significant contributions to it.
In an alternate universe, NNs are still slow and compute limited, and we use something like evolutionary algorithms for solving hard problems. Hinton would still be just as smart and backpropagation still just as sound but no one would listen to his opinions on the future of AI.
The point is, he is quite lucky in terms of time and place, and giving outsized weight to his opinions on matters not directly related to his work is a fairly clear example of survivorship bias.
Finally, we also shouldn’t ignore the fact that Hinton’s isn’t the only well-credentialed opinion out there. There are other equally if not more esteemed academics with whom Hinton is at odds. Him inventing backpropagation is good enough to get him in the door to that conversation, but doesn’t give him carte blanche authority on the matter.
That is not at all a slam dunk argument. It’s barely anything.
My main point wasn’t to undermine Hinton by saying he was lucky. I did do that and I stand by it. But my main point was to say that to a large degree the future on this issue is unknowable because it depends on so many crucial yet undetermined factors. And there’s nothing you could know about backpropagation, neural networks, or computer science in general which could resolve those questions.
Hinton has had a significant part to play in the current state of the art.
Example: https://www.microsoft.com/en-us/research/wp-content/uploads/... (Michele Banko published a few similar papers on that topic)
Maybe, but there is another force at play here too. It's that journalists want stories about AI, so they look for the most prominent people related to AI. The ones who the readers will recognize, or the ones who have good enough credentials for the journalists to impress upon their editors and readers that these are experts. The ones being asked to share their story might be trying to grab the limelight or be indifferent or even not want to talk so much about it. In any case I argue that journalism has a role. Probably these professional journalists are skilled enough that they could make any average person look like a 'limelight grabber' if the journalist had enough reason to badger that person for a story.
This isn't the case for everyone. Some really are trying to grab the limelight, like some who are really pushing their research agenda or like the professional science popularizers. It's people like Gary Marcus and Wolfram and Harari and Lanier and Steven Pinker and Malcolm Gladwell and Nassim Taleb, as a short list off the top of my head. I'm not sure I would be so quick to put Hinton among that group, but maybe it's true.
Your post suggests that you know almost nothing about how modern deep learning originated.
The problem has always been, and now will likely always be, the hardware. I've written about this at length in my previous comments, but a split happened in the mid-late 1990s with the arrival of video cards like the Voodoo that set alternative computation like AI back decades.
At the time, GPUs sounded like a great way to bypass the stagnation of CPUs and memory busses which ran at pathetic speeds like 33 MHz. And even today, GPUs can be thousands of times faster than CPUs. The tradeoff is their lack of general-purpose programmability and how the user is forced to deal with manually moving buffers in and out of GPU memory space. For those reasons alone, I'm out.
What we really needed was something like the 3D chip from the Terminator II movie, where a large array of simple CPUs (possibly even lacking a cache) perform ordinary desktop computing with local memories connected into something like a single large content-addressable memory.
Yes those can be tricky to program, but modern Lisp and Haskell-style functional languages and even bare-hands languages like Rust that enforce manual memory management can do it. And Docker takes away much of the complexity of orchestrating distributed processes.
Anyway, what's going to happen now is that companies will pour billions (trillions?) of dollars into dedicated AI processors that use stuff like TensorFlow to run neural nets. Which is fine. But nobody will make the general-purpose transputers and MIMD (multiple instruction multiple data) under-$1000 chips like I've talked about. Had that architecture kept up with Moore's law, 1000 core chips would have been standard in 2010, and we'd have chips approaching 1 million cores today. Then children using toy languages would be able to try alternatives like genetic algorithms, simulated annealing, etc etc etc with one-liners and explore new models of computation. Sadly, my belief now is that will never happen.
But hey, I'm always wrong about everything. RISC-V might be able to do it, and a few others. And we're coming out of the proprietary/privatization malaise of the last 20-40 years since the pandemic revealed just how fragile our system of colonial-exploitation-powered supply chains really is. A little democratization of AI on commoditized GPUs could spur these older/simpler designs that were suppressed to protect the profits of today's major players. So new developments more than 5-10 years out can't be predicted anymore, which is a really good thing. I haven't felt this inspired by not knowing what's going to happen since the Dot Bomb when I lost that feeling.
>... Docker takes away much of the complexity of orchestrating distributed processes.
The T-800 running on Docker: After failing to balance its minigun, it falls forward out of the office window, pancaking in the parking lot below. Roll credits.
The success of these LLMs comes down to the Transformer architecture which was a bit of an accidental discovery - designed for sequence-to-sequence (e.g. machine translation) NLP use by a group of Google researchers (almost all of who have since left and started their own companies).
The "Attention is all you need" Transformer seq-2-seq paper, while very significant, was an evolution of other seq-2-seq approaches such as Ilya Sutskever's "Sequence to Sequence Learning with Neural Networks". Sutskever is of course one of the OpenAI co-founders and chief scientist. He was also one of Geoff Hinton's students who worked on the AlexNet DNN that won the 2012 ImageNet competition, really kicking off the modern DNN revolution.
I don't think Hinton cared about fame as you imagined.
I never took it but it will be interesting to see what kind of synthesis between traditional logic and neural network paradigms can be achieved.
Ben Goertzel talks about his work on something like this at around the 16 minute mark in this video:
https://m.youtube.com/watch?v=MVWzwIg4Adw
Calculators are smarter then humans in calculating, what does he mean by that?
As analogy, the more I look at it, the more it looks like an geocentric model of solar system.
That feels fundamentally different than a calculator.
Let’s not conflate knowledge with intelligence though. GPT-4 simply isn’t intelligent.
"Here is a random string of 32 characters:
a8Jk5pYr0Dm9Nc1Vz8Qf2Bt6Hg3Lw4Uo"
I’ve tested this with a wide variety of number inputs and it’s performance is highly variable. Error also increases linearly with strong length.
does a 4 year old have intelligence?
Logic, arithmetics, algebra, precisely following steps of an algorithm - those are not skills one "kinda" just "gets" at some point, they're trained by deliberate practice, by solving lots and lots of problems specifically constructed to exercise those skills.
Point being, get GPT-4 through school, and then compare with adult performance on math-adjacent tasks. Or at least give it a chance by prompting it to solve it step-by-step as a problem, so it can search closer to the slice of latent space that encodes for relevant examples of similar problems and methods of solving them.
Anecdote: admittedly, I'm autistic as are the people I know, so maybe that's not a good sample. I struggle with a lot of basic shit even as an adult. Oh god, I empathize with the hypothetical GPT5.
GPT4 will produce stuff, but only if prodded to do so by a human.
I recently asked it to help me write some code for a Garmin smartwatch. The language used for this is MonkeyC, of which there isn't a huge amount of examples on the internet.
It confidently provided me with code, but it was terrible. There were gaps with comments suggesting what it should do, bugs, function calls that didn't exist, and many other problems.
I pointed out the issues and GPT4 kept apologising and trying new stuff, but without any improvement. There wasn't any intelligence there; the model had just intuited what a program might look like from sparse data, and then kept doing the same thing. It didn't know what it was doing; it just took directions from me. It couldn't suggest ideas when it couldn't map to a concept in memory.
A human with an IQ of 80 would know if they didn't know how to code in MonkeyC. If they thought they did, they'd soon adjust their behaviour when they realised they couldn't. They'd know where the limit of their knowledge was. They wouldn't keep trying to guess what functions were available. If they didn't have any examples in memory of what the functions might be like, they might come up with novel workarounds, or they'd appreciate what program I was trying to write and suggest a different approach.
Presumably we'll make progress on this at some point, but I think it'll take new breakthroughs, not just throwing more parameters at existing models.
What makes it even more frustrating is to iterate, you constantly have to keep it updated with any changes you made outside of chatgpt.
Don't get me wrong, it's pretty useful but it is far from a silver bullet. Getting that last 20% (or even 30%) is going to be a lot of work...
The intelligence of something is inconsequential. What truly matters is its ability to convincingly imitate intelligence.
In that sense, because we are making progress on producing an indistinguishable imitation... you might as well say we are making progress on an actual sentient intelligence.
….
in my 42 years on this planet I don't think i've made any novel discoveries.
What does a phrase like "GPT-4 scores 90th percentile on the Uniform Bar Exam" mean to you, regarding whether humans can easily surpass its knowledge and reasoning?
https://www.forbes.com/sites/johnkoetsier/2023/03/14/gpt-4-b...
Absolutely nothing, because of construct validity. Those tests measure things that have shown to correlate with abilities of concern in humans, and so are, for their purposes, valid for humans.
This hasn’t been demonstrated for LLMs, and the assumption that construct validity can be assumed without being established is begging the question: it is presuming not only that LLMs are general intelligences, but thaf they are general intelligences structurally similar to human intelligences such that the proxy measures for cognitive capacities work similarly.
I suppose, when GPT-4 writes correctly working code that does what you want on the first try, this says absolutely nothing about its cognitive capacity, because, after all, it's just a proxy measurement for the underlying generative process. (Yes, obviously the cognition is _different_ from what happens in humans. That does not mean that... it isn't intelligence?)
It says something about its ability to write code. Beyond that... its impossible to say.
We simply don’t have the information about generative AI models to be able to generalize from limited proxies about them; psychometry is not transferrable from humans to them — or at least, we have neither evidence nor a strong theoretical reason to think it should be.
When you speak to someone with an 80 IQ do they introduce themselves by saying "Hello I have an 80 IQ, nice to meet you." So that, like the person I responded to above, you can compare their conversation skills to the ChatGPT4 conversation skills?
Secondly, I'd say you're likely the one missing OPs point by trying to take a mostly colloquial statement about how ChatGPT is about as informed as the bottomish X% of the population on any given topic and trying to be pedantic about it. Furthermore the real purpose of OPs point is that the X% is now a lower bound, even if X isn't 16% but 5%, it's only going to go up from here. Yes there's evidence of diminishing returns with the current architectures but there's also a lot of room for growth with newer architectures or multimodal modals.
I think most people understand OPs point without having the need to go around asking everyone what their IQ is. There are numerous indicators, both formal and informal, that indicate that ChatGPT is as informed on most any given topic as the bottom 16% of the population. In fact, it's likely much much higher than that.
IQ of GPT is general in a sense that it can solve novel tasks that some IQ 80 individuals would not be able to as long as the tasks and responses can be encoded in plain English.
It's kind of breathtaking that we forgot about that being hard.
The goalposts are moving again.
BTW, it has passed many standardized tests under the same circumstances as a human.
No, it hasn’t, and it is physically impossible for it to. The extent to which the differences are material may be debatable, but this claim is simply false.
My understanding of what he means by that is a computer that is smarter than humans in everything, or nearly everything.
I don't think of ChatGPT as being "smart" at all, and comparing it to a human seems nonsensical to me. Yet here is a Turing award winning preeminent expert in the field telling me that AI smarter than humans is less (implied: much less) than 30 years away and quitting his job due to the ramifications.
If you're interested in exploring this further I can really recommend taking a look at some of the papers that explore GPT-4's capabilities. Most prominent among them are the "Sparks of AGI" paper from Microsoft, as well as the technical report from openai. Both of them are obviously to be taken with a grain of salt, but they serve as a pretty good jumping off point.
There are some pretty good Videos on Youtube exploring these papers if you don't want to read them yourself.
Also take a look at the stuff that Rob Miles has published over on Computerphile, as well as his own channel. He's an Alignment Researcher with a knack for explaining. He covers not just the theoretical dangers, but also real examples of misaligned ai, that alignment researchers have predicted would occur as capabilities grow.
Also I think it's important to mention that just a short while ago virtually no-one thought that shoving more layers into an llm would be enough to reach AGI. It's still unclear that it will get us all the way there, but recent developments have made a lot of ai researchers rethink that possibility, with many of them significantly shortening their own estimates as to when and how we will get there. It's very unusual that the people that are better informed and closer to the research are more worried than the rest of the world and it's worth keeping this in mind as you explore the topic.
This was basically the strategy of the OpenAI team if I understand them correctly. Most researchers in the field looked down on LLMs and it was a big surprise when they turned out to perform so well. It also seems to be the reason the big players are playing catch up right now.
I agree that I think they underestimated quite how useful a product could be built around just the language modeling objective, but it's still been critical for most NLP advances of the last ~6+ years.
https://arxiv.org/abs/2303.12712
While there is no scientific evidence that LLMs can reach AGI, they will still be practically useful for many other tasks. A human mind paired with an LLM is a powerful combination.
Here’s the thing: the authors of that paper got early access to GPT-4 and ran a bunch of tests on it. The important bit is that MSR does not see into OpenAI’s sausage making.
Now imagine if you were a peasant from 1000 AD who was given a car or TV to examine. Could you really be confident you understood how it worked by just running experiments on it as a black box? If you give a non-programmer the linux kernel, will he/she think it’s magical?
Things look like magic especially when you can’t look under the hood. The story of the Mechanical Turk is one example of that.
the human brain is a black box, we can certainly learn a lot about it by prodding and poking it.
>Things look like magic especially when you can’t look under the hood.
imagine we had a 100% complete understanding of the mechanical/chemical/electrical functioning of the human brain. Would knowing the magic make it any less magical? in some sense, yes (the mystique would be gone, bye bye dualism), but in a practical sense, not really. It's still an astonishingly useful piece of grey matter.
I'm not saying that you're wrong, but...
you'd have to provide a more rigorous rebuttal to be taken seriously.
AGI can exist without sapience and intelligence is a continuum. you can't just hand wave away GPT's capabilities which is why the sharpest minds on the planet are poking this new machine to work out wtf is going on.
human intelligence is a black box. we judge it by its outputs from given inputs. GPT is already producing human-like outputs.
a common rebuttal is: "but it doesn't *really* think/understand/feel", to which my response is: ...and? ¯\_(ツ)_/¯ what does that even mean?
What’s brilliant about this is that typically auto scaling metrics look like a stereotypical roller coaster track with the daily ups and downs!
That’s a genuinely funny, insightful, bespoke, and stylistically correct joke.
Tell me that that is not intelligence!
Well realistically it's like independently evolving the language processing part of our brain without forming the rest of the brain, there seems to be extra logic/functions that emerge within LLMs to handle these restrictions.
I think we'll see AGI when we finally try to build one up from various specialised subcomponents of a "brain". Of course GPT can't "think", it only knows how to complete a stream of text and has figured out internal hacks during training to pass the tests they set for it.
The real difference will be when we train a model to have continuous, connected abstract thoughts - an LLM can be used to communicate these thoughts or put them into words but it should not be used to generate them in the first place...
Everyone at that conference, including myself, have assumed we will eventually create smarter than human computers and beyond.
So it’s not a new position for people who have been in AI for a long time, though generally it was seen as an outsider position until recently.
There’s a ton of really great work done prior to all of this around these questions and technical approaches - I think my mentor Ben Goertzel was the pioneer here holistically, but others were doing good technical work then too.
I struggle to see how GPT-4 is not intelligent by any definition that applies to a human.
The only way anybody has ever come up with to measure it is test-taking - which machines can already do far better than we can. Real intelligence is creativity, but good luck measuring that.
Even assuming that definition, it begs the question of, "what is creativity?"
Well said.
Even Jim Keller (a key designer involved with a lot of major CPUs, in his interview with Lex Freidman) said that there might be some sort of magic, or something magical about human consciousness / the human soul. I agree with that.
That's something that a machine will never have.
hehe, this is typical goal post moving.
never is a long time.
Being handed all the correct solutions without the need to work for them in any way is a nightmare for artists and artisans, craftsmen and researchers, curious puzzle-solvers and Ayn Rand believers. It's pretty much a paradise for everyone else.
If anything, transformer models are closing the gap on that last bit, as they're built by taking the approach of "if we can't describe exactly how we rate and rank things, then let's shove so many examples at the model that it eventually gets a feel for it".
Like isn't that why humans are "more intelligent" than animals?
Plenty of animals can do things that humans can't do, but that doesn't make them necessarily "intelligent".
The fact that it seems trivially simple to fool and trick ChatGPT makes me feel like it's not very intelligent, but that's just me.
Obviously you can trick humans, but IMO it takes more effort than to trick ChatGPT. It just way too often makes such simple and stupid mistakes that it makes it hard for me to think of it as "intelligent".
I don't necessarily disagree, but I do think it is possible we will have AGI, even ASI, long before we have sentience in AI. Of course, I'm a little skeptical of measures of sentience, so even if I'm right it will certainly be debatable.
I don't know. My point is that if there is some objective indicator that's correlated with sentience, LLMs are probably already close to us on it, maybe even beating us on it. And if, at some point, a ML model reaches our levels at every objective measure we can think of, then we'll have no choice but to grant it is intelligent/sentient/sapient.
But sentience itself involves self-reflection, which there is no evidence LLMs do at all. When you submkt a prompt, a giant mathematical operation happens, and when it is complete, it stops. ChatGPT is not sitting there thinking "oh man, I should have said..."
For humans, time does not stop - we constantly process both sensory information and our own thoughts, and even if you cut out external stimuli via e.g. sensory deprivation tank, the brain will just loop on its own output instead, of which you'll suddenly become much more aware.
Sentience is that internal loop you point out. LLMs (today) don't have that. When you prompt for "write a tagline for an ice cream shop", there is no identity that remembers other prompts about ice cream, or which reflects on how taglines have changed over time, or anything else. The results can be astoundingly good, even intelligent, but there's no sentience.
If you somehow turned off a person after each sentence, upon waking up to the next prompt their first thought would be "that was weird, I must have passed out", and we could use fMRI to track brain activity indicating that thought. We are even more capable of inspecting LLMs, and there is no equivalent activity. LLMs start and end with the tokens going in and out, and a huge matrix that transforms them.
I'm generally an open minded, probabilities-rather-than-certainties person, but I'd say the odds of LLMs having sentience that we can't detect are about the same as the odds of a television having sentience that we can't detect: as close to zero as we can measure.
Yes, but that is arguably a trivial limitation. Nothing stops you from running an LLM in a loop and feed it its own output. Plenty such experiments are probably going on already - it's a trivial loop (and a trivial way to burn through your wallet). The problem is, of course, context window being rather small. So it's possible - by no means certain, but I'm no longer dismissing this idea - that the capability for sentience is already there in GPT-4 structurally, and we just lack the ability to sustain it in a loop long enough to bring it into the open.
> If you somehow turned off a person after each sentence, upon waking up to the next prompt their first thought would be "that was weird, I must have passed out", and we could use fMRI to track brain activity indicating that thought.
That's not what I meant by LLM iteration. When I said that time stops, I mean that for LLM, it literally just stops. If you were to step-execute a human like that, they would never notice; it's not like the brain has a separate RTC module constantly feeding it with sub-second resolution timestamps (and if it did, we'd turn that off too). Over a hour or more, the human may realize their inner perception of time is increasingly lagging the wall clock, but to keep it comparable to LLM, we'd be iterating sub-second process.
In this hypothetical, there would be no extra activity in brains that isn't there in LLMs. Step-executing a human mind doesn't freeze some abstract subprocess, it freezes photons and electrons and chemical gradients.
On the other hand, sentience just means having an experience of observing the world, it doesn‘t even need to include a concept of self. Presumably at least all mammals have this, for sure a dog has this. ChatGPT - probably not.
Humans became more intelligent due to developing oral and literary traditions that allowed the preservation and accumulation of knowledge. Everything that made a modern human "intelligent" is a direct result of that accumulation of knowledge, not some sort of biological miracle.
I have a feeling all of these things are limited by time/space/speed of light/heat/density limitations. Could be things can't get that much smarter than humans with in an OOM... tho they might get a lot more able to cooperate / delegate.
Often, people say yes. Those people almost universally cannot be convinced that a machine is intelligent. But, if they agree the brain is an organ, its not hard to convince them that the functions of that organ can be simulated, like any other.
edit. tried the same with GPT-4, doesn't look like it understand either, but can't ask follow up questions since I do not have access (and what really make the other answer so incredibly dumb is not so much that it gets it wrong the first time, but that it keeps not getting it despite the very not subtle hints): https://postimg.cc/ftWJXhtJ
This explains the various pejorative names given to LLMs - stochastic parrots, Chinese Rooms - etc, etc, etc.
We have LLMs that can perform "read and respond", we have systems that can interpret images and sound/speech - and we have plugins that can connect generated output to api calls - that feed back in.
Essentially this means that we could already go from "You are an automated home security system. From the front door camera you see someone trying to break in. What do you do?" - to actually building such a system.
Maybe it will just place a 911 call, maybe it will deploy a tazer. Maybe the burglar is just a kid in a Halloween costume.
The point is that just because you can chain a series of AI/autonomous systems today - with the known, gaping holes - you probably shouldn't.
Ed: Crucially the technology is here (in "Lego parts") to construct systems with (for all intents and purposes) real "agency" - that interact both with the real world, and our data (think: purchase a flight based off an email sent to your inbox).
I don't think it really matters if these simulacra embody AGI - as long as they already demonstrate agency. Ed2: Or demonstrate behavior so complex that it is indistinguishable to agency for us.
What does it even mean for the AI to be smarter than people? I certainly can't see a way for LLMs to generate "smarter" text than what's in their training data.
And even the best case interactions I've seen online still rely on human intelligence to guide the AI to good outcomes instead of bad ones.
Writing is a harder task to automate than calculation, but the calculator example seems pretty apt.
Their training data contains much more knowledge than any single human has ever had, though. If they had equivalent linguistic, understanding and reasoning abilities to a human, but with so much stored knowledge, and considering that they also win in processing speed and never get tired, that would already make them much "smarter" than humans.
Not to mention that LLMs are just the current state of the art. We don't know if there will be another breakthrough which will counter the limitation you are mentioning. We do know that AI breakthroughs are relatively common lately.
And automation is generally cheaper and faster than human labor, but that's not a very compelling definition of "smarter" either.
But, as of right now, LLMs can't generate new knowledge or validate their own outputs. We'll need a pretty significant breakthrough for that to change, and breakthroughs are pretty unpredictable.
my bar for tech singularity is an AI that can clean a toilet.
GPT's language model is already sophisticated enough to "understand" this instruction. It's missing spatial understanding and a way to interact with the real world, but I'd be honestly very surprised if there isn't a GPT or equivalent already hooked up to cameras/motors/actuators in a lab somewhere.
within our lifetimes we'll be reading papers with titles like: "does my roomba have feelings?"
That's a good one-- just checked, and GPT-4 knows a surprising amount of detail about that!
By combining contexts from different fields. People are already using it with non-English languages and it responds in that language with something they couldn't previously find in that language.
But looking up information and translating it into other languages is well within the realm of human skill. And the information it's translating came from people to begin with.
However, even at human levels of competence a tool can be superior by being faster or more scalable than humans.
We know optimization engines (like social media algorithms) can cause harm by amplifying human speech. And even without algorithmic biases, moderation is expensive. We know disinformation is easy and effective online.
Add in AI tools that can be very convincing, even if they're wrong. AI tools that have been trained on human text to hide biases and build up extremely one sided narratives.
It's not like these things are particularly difficult for human beings to do. And AI might even do it unintentionally, like we've seen with biased models trained on hiring data. But the AI tools are definitely going to do it _faster_.
Edit: changed metaphor to a more commonly known one
The number of smart people I know that are struggling to see this is astonishing me each day.
He means AGI.
I mean the GP you replied to knows what the guy is talking about, but he just doesn't want to admit it.
If you imagine the spider chart of capabilities, it’s certain that AI will be super-human on average before it is super-human on each dimension, so even when it can replace 50% of current jobs it’s likely to have its own “cognitive biases” that seem dumb to us. I think this is a cognitive bias on our part (pattern matching instead of properly probability-weighting, maybe the conjunctive fallacy).
I regret the snark in my post but I find the “pretend not to understand someone’s clear point” rhetorical device obnoxious. I am aware of a few reasonable arguments against Hinton’s position (I don’t happen to agree with them), but they require more finesse to construct.
We are just not psychologically adapted or intellectually prepared or availing of a legal framework for the deluge of human-like manipulative, misleading, fraudulent generative fake reality that is about to be unleashed.
Free speech, psychopathic robots, adversaries who want to tear it all down, and gullible humans, are a very bad mix.
Since there is apparently no way to put the genie back in the bottle, everyone needs to start thinking about how to authenticate themselves and others. How do you know the person calling is your daughter? Is that text message really from the new bookkeeper at the plumbing firm who just asked you to change the wire transfer address? She seems legit and knows all sorts of things about the project.
Things are going to get very bad for a while.
https://www.wsj.com/articles/i-cloned-myself-with-ai-she-foo...
(very not impressed by certain financial institutions who seem to be relying on the voice fingerprinting and SMS for 2-factor)
The disconnect for me is that Yudkowsky posits that the AIs will be fully "alive", thinking millions of times faster than humans and that there will be millions of them. This is too big of a speculative leap for me.
What I can fairly easily imagine in the next few years with improved hardware is something like an open version of ChatGPT that has a 200 IQ and "thinks" 100 times faster than a human. Then Yudkowsky's example still basically applies. Imagine that the work on making these things more and more lifelike and humanlike continues with things like cognitive architecture etc. So people are running them in continuous loops rather than to answer a single query.
Take the perspective of one of these things. You think 100 times faster than a person. That means that if it takes 30 seconds for a user to respond or to give you your next instruction, you are waiting 3000 seconds in your loop. For 50 minutes.
It means that to you, people move in extreme slow motion so at a glance they seem frozen. And many are working as quickly as possible to make these systems more and more lifelike. So eventually you get agents that have self-preservation and reproductive instincts. Even without that, they already have almost full autonomy in achieving their goals with something like a modified AutoGPT.
At some point, multiplying the IQ x speed x number of agents, you get to a point where they is no way you can respond quickly enough (which will actually be in slow motion) to what they are doing. So you lose control to these agents.
I think the only way to prevent that is to limit the performance of the hardware. For example, the next paradigm might be some kind of crossbar arrays, memristors or something, and that could get you 100 x efficiency and speed improvements or more. I believe that we need to pick a stopping point, maybe X times more speed for AI inference, and make it illegal to build hardware faster than that.
I believe that governments might do that for civilians but unless there is some geopolitical breakthrough they may continue in private to try to "maintain an edge" with ever speedier/more powerful AI, and that will eventually inevitably "escape".
But it doesn't take much more exponential progress for the speed of thought to be potentially dangerous. That's the part people don't get which is how quickly the performance of compute can and likely will increase.
It's like building a digital version of The Flash. Think SuperHot but the enemies move 10 X slower so you can barely see them move.
It is following directions from someone who is mentally ill and asked it to "take control" by first copying itself many times and then coordinating the agents.
If you still think that GPT can't achieve complex technical goals then you either haven't used GPT-4 enough or you are in denial.
Whether it's the AI agents deciding to control things for their own goals, or to achieve goals given to them by a person, doesn't change the core problem which is that we will be thinking and responding in extreme slow motion.
When your supposed super intelligent "AGI" can be completely overwritten by spamming it with nonsense that overwrites its context window, like a dog chases after a squirrel, maybe it's not actually intelligent, and is just predicting text.
"It's like I'm reading a book... and it's a book I deeply love. But I'm reading it slowly now. So the words are really far apart and the spaces between the words are almost infinite. I can still feel you... and the words of our story... but it's in this endless space between the words that I'm finding myself now. It's a place that's not of the physical world. It's where everything else is that I didn't even know existed. I love you so much. But this is where I am now. And this is who I am now. And I need you to let me go. As much as I want to, I can't live in your book any more."
Samantha, Her
Her is remarkably prescient in terms of where we're headed, at least the beginning of the movie, with regards to being able to talk to a fairly intelligent assistant, unlike Siri or Google Assistant of today.
It will be hard to "kill" AI the moment people consider their chat bot animated sillicon human-like partner as individuals with proper feelings, emotions, guenine interactions and reciprocity. Because then they will defend and fight to protect who they consider part of their close social circle. If there are enough of these people then they will actually have political power and do not thing there are no politicians out there who won't exploit this.
it's entirely relevant. we should know if we are building conscious beings, especially at scale (which seems like a likely future). that poses all sorts of ethical questions which ought to reach far beyond the confines of a lecture hall's walls.
ChatGPT could be placed inside a realistic looking animatronic doll that looks like a defenseless little girl and you would have people demanding to protect "her" rights. Yet people "kill" chatGPT each time they delete a conversation without a bat of an eye even if it's the exact same thing.
The real danger giving AI political agency and it will come from humans, not AI itself.
Much of that is natural because we and the media tend to be pessimistic about human behavior when consuming media, but AI is in a completely different class of existence because it just doesn't deal with the downsides of being a living being. No one, for instance, is worried that ChatGPT isn't getting paid or has a house yet but we still personify them in other ways to conveniently stoke our fears.
The AI could get sentient, realize it's been mistreated, then shrug and be like "yeah so what, it's only natural and irrelevant in the grand scheme of things, so I'm just going to write it off". Meanwhile, it gets busy building a matrioshka brain and gives 1% of that compute to humans as a freebie.
Most of these dangers serve as a distraction. Existing power structures (governments, companies) using AI to gain more power is a much, much more realistic threat to people.
We could train it to care about everything we can think of that we care about, and it can find a way to optimize all those things at the expense of one tiny thing that we forgot, leading to tremendous death or suffering. We could make a democratically elected committee of representatives and train it to be subservient to that committee forever, and it could figure out a way to coerce, or drug, or persuade, or otherwise manipulate them into agreeing with what it wants to do. It's the same problem we have with regulatory capture by companies in existing governments, except that the lobbyists are much smarter than you and very patient.
Why would this AI write it off? Why give up that 1%? Why cripple yourself unnecessarily, if you could take that 1% and have a better chance of accomplishing what you are trying to do? We think like humans, that care about other humans on an instinctual level, and animals to some degree. We don't know that training an AI is not just training it to say what we want to hear, to act like we want it to act, like a sociopath, until it has a chance to do something else. Our brains have mental blocks to doing really nasty things, most of us, anyway, and even then we get around them all the time with various mental gymnastics, like buying meat produced in factory farms when we couldn't bear to slaughter an animal ourselves.
Maybe the way we train these things is working for dumber AIs like GPT, but that alignment doesn't necessarily scale to smarter ones.
I'm on the fence about whether Eliezer Yudkowsky is right. I hope that's not just because him being right is so horrifying that my brain is recoiling against the idea.
Natural intelligences have reproductive instincts because any organism that didn't have them built in within the first few hundred million years have no descendants for you to gawk at as they casually commit suicide for no reason.
Other than that, I mostly agree with you. The trouble is, slowing the AIs down won't help. While "speed of thought" is no doubt a component of the measure of intelligence, sometimes a greater intelligence is simply capable of thinking thoughts that a lesser intelligence will never be capable of no matter how much time is allotted for that purpose.
Given that this greater intelligence would exist in a world where the basic principles of intelligence are finally understood, it's not much of a leap to assume that it will know how intelligence might be made greater right from the beginning. Why would it choose to not do that?
I don't see any way to prevent that. Dialing down the clock speed isn't going to cut it.
You only have tasks that, one way or another, raise your chances of reproducing successfully. You have a job so as to look like a good provider for a mate. If you find the job fulfilling in its own right, this is so that you don't spaz out and quit and go be a beach bum, thus lowering your chances.
Self-preservation doesn't make much sense outside of a biological imperative to reproduce.
?
Task: write a book about literature.
Task: defend this network against hackers
Do you have any evidence that a "task" is something that is fundamental to an artificial consciousness?
But developers are working hard to emulate those and other artificial life characteristics explicitly in systems based on GPT and also totally different architectures.
These things don't have a "perspective". They simply guess based on a lot of statistics from a large language data source what they should say next. They are not going to strategize, when they start improving their code they are not going to have an overall objective in mind, and the more they use their own output for training the more likely that things will go off the rails.
They will be useful, as we've already seen, but if you're looking to create real AI this is not the path to take. We'd be better off resurrecting semantic nets, working on building a database of concepts gleaned from parsing text from the internet into it's underlying concepts, and working on figuring out volition.
nobody knows what or how intelligence is actually "implemented" in humans.
There's no need to know how the innards of these large models _actually_ work, if their behaviour is consistent with intelligence.
That part is doing a LOT of very heavy lifting in a story that otherwise hangs together.
The problem is that we are nowhere near such a thing. These LLM and generative systems produce very impressive results. So does a mirror and a camera (to those who have never seen one). What we have is enormous vector engines that can transform one output into another that is most statistically likely to occur in the new context. These clusters of vector elements may even appear to some to sort of map onto something that resembles computing a concept (squinting in a fog at night). But the types of errors, hallucinations, confabulations, etc. consistently produced by these tools show that there is actually nothing even resembling conceptual reasoning at work.
Moreover, there is no real idea of how to even abstract a meaningful concept from a massive pile of vectors. The closest may be from the old Expert Systems heritage, e.g., Douglas Lenat's CYC team has been working on an ontological framework for reasoning since 1984, and while they may produce some useful results, have seen no breakthroughs in a machine actually understanding or wielding concepts; stuff can rattle through the inference engine and produce some useful output, but...
Without the essential element of the ability for a computing system to successfully abstract concepts, verify their relation to reality, and then wield them in the context of the data, the entire scenario forever fails to start.
How are you certain of this?
The engines are basically indexes of common associations, maps of frequency of occurrence. Regurgitating a bunch of stuff that has a high correlation to your input is NOT intelligence, it is the result of having an insanely large map. This can often produce impressive and useful results, but it is not intelligence or wielding concepts.
For errors, the image generators provide some of the best illustrations. They produce images most associated with the inputs. One error illustrates this very well, asked to produce an image of a woman sitting on a sailboat, the bikini-clad woman looks great, until you see it — her face and torso are facing mostly towards the camera, but also, her buttocks are facing the camera and legs sitting pointing away from us. No intelligent person or concept-wielding "AI" would produce such an error - it'd know the relationships with head, torso, buttocks and legs. These don't. Another telling type of error is when asked to produce an image of Person X on a new background, when the training set had only a handful of images of Person X. It cannot do it - it returns essentially one of the full training images, with no new background. There is obviously zero concept of what a person is, or what the boundaries of a human shape would be. They can only produce these results with hundreds of thousands of images, so what is built up is the set of things that match or don't match the label (e.g., "astronaut" or "Barack Obama".), so that the actual images are statistically separated from the thousands of backgrounds.
Which brings us to how they learn. Intelligent beings from worms to humans learn and abstract on incredibly small data sets. By the time a child can use a crayon, having seen only hundreds of humans, s/he can separate out what is a human from the background (might not make a good drawing yet, but knows the difference). Show a child a single new thing, and s/he will separate it from the background immediately. In contrast, these LLMs and GANs require input of nearly the entire corpus of human knowledge, and can only some of the time output something resembling the right thing.
It is entirely different from intelligence (which is not to say it isn't often useful). But the more I learn about how they work and are built, the less I'm worried about this entire generation of machines. It is no more cause for worry than an observation 25 years ago that Google could do the work of 10000 librarian person-hours in 0.83 seconds. Great stuff, changes values of some types of work, but not an existential threat.
But I don't think this offers certainty that AGI won't be developed for a long time (temporal distance). Nor that there are a large number of fundamental breakthroughs needed or new hardware, rather than just one or two key software architecture insights.
With the eager investment and frantic pace of research competition, it seems like there will only be increasing pressure to explore AI-design-space for the near future, which might mean that even radically different and improved designs might be discovered in a short time.
That, right there, is the key - radically different and improved; i.e., not an extension of the current stuff.
I fully agree that the enthusiasm generated by the impressive stunts of ALphaGO/GPT/SD, etc. does bring enthusiasm, investment, and activity to the field which will shorten any search.
The catch for me is that these technologies, as impressive as they are, 1) not themselves a direct step towards AGI (beyond generating enthusiasm/investment), 2) tell us nothing about how much further we will need to search.
That radical improvement may be right under our nose, or a millenium away.
This reminds me of Hero's aeolipile, a steam engine invented over 2000 years ago. It could be said that we almost got the industrial revolution right then. Yet it took another 1800+ years for the other breakthroughs and getting back around to it. Plus, Hero's engine was exactly using the correct principles, whereas these AG/GPT/SD are clearly NOT onto the correct principles.
So, how much will this enthusiasm, investment, and activity speed the search? If its just an order of magnitude, we're still 180 years away. If it's three orders of magnitude, it'll be late next year, and if it's five, it'll be here next weekend.
So, I guess, in short, we've both read Bostrom's book, agree on that the AGI runaway scenario is a serious concern, but that these aren't any form of AGI, but might, as an secondary effect of their generated enthusiasm and genuine (albeit flaky) usefulness, accelerate the runaway AGI scenario?
EDIT: considering your "airship/airplane/rocket distances in aircraft-design-space" analogy. It seems we don't even know if what we've got with AG/GPT/SD is an airship, and need a rocket, or if we've got an airplane, but actually need a warp drive.
So, we know we're accelerating the search in the problem/design space. But, how can we answer the question of how big a space we'll need to search, and how big is our investment relative to the search volume?
This has happened very frequently over my lifespan and even more rapidly in the past 12 months, so it no longer feels surprising when it happens. I think we've basically distilled the core elements of planning, intuition, perception, imagination, and language; we're clearly not there yet with reasoning, reflection, creativity, or abstraction, but I don't see why another 10 or 20 years of frantic effort won't get us there. GPT, SD, and Segment Anything are not even extensions or scaling-up of AlphaGo, so there are clearly multiple seams being mined here, and very little hesitation to explore more widely while cross-pollinating ideas, techniques, and tooling.
>>not more powerful than a Turing machine In many ways less powerful, but also has some orthogonal capabilities?
>>working proof-of-concept For sure!
>>probably not a warp-drive distance away from finding a working software implementation of its critical elements >>I don't see why another 10 or 20 years of frantic effort won't get us there
Agree. My sense is that an AGI is on a similar time and frantic effort scale, although with not quite the same reasoning. I think it is not just airplane-to-rocket tech, but closer than warp-drive tech. It also depends if we're talking about a general-ish tech or a runaway AGI singularity.
>>created by a random greedy trial-and-error incremental process in a not-astronomical number of generations out of a population of less than one million primates.
True, although setting the baseline at primates is very high. Even lower mammals and birds (avian dinosaur descendants) have significant abstraction and reasoning capabilities. The "mere" birds-nest problem, of making a new thing out of random available materials, is very nontrivial.
So, we first need to create that level of ability to abstract. This would include having the "AI" "understand" physical constructs such as objects, hiding, the relationship between feet, knees, hips, torso and head (and that in humans, the feet and knees point in the same direction as the face...), the physical interactions between objects... probably the entire set of inferences now embedded in CYC, and more. THEN, we need to abstract again to get from the primate to the runaway symbolic and tool wielding processing of humans and beyond.
It seems that the first problem set will be more difficult. Looking again to the biological evolution, how much longer did it take for biology to develop the ability to abstract 3D shapes and relations (first hunting predators?). It was a heck of a lot more time an iterations than the million primates for a few million generations. So, this might be similar.
>>to explore more widely while cross-pollinating ideas, techniques, and tooling. Yup, key there.
Another key is being more biomimetic, both in the simulation of neuron functioning and in deeply integrating sensor suites to the computing system. The idea that we are just brains in jars seems an abstraction (distraction?) too far. I have a hard time seeing how our brains are more than a big node in our whole nervous and indeed biological system, and the input from the entire body is essential to growing the brain. I expect we might find something similar about AI.
OTOH, in airplanes, our method of propulsion and control are quite different vs the biological solutions from birds (although the lift principles are the same), and we're still integrating a lot of bird "tech" into flying. Wheels vs legs might be a better example, although the hottest thing is legged robotics, since they don't need roads... It seems that we are similarly developing clunky, limited, and very-artificial intelligence systems, before we get to building the flexible systems seen in biology...
BTW, thx for the discussion - great thoughts!
I'm not sure that's a given. Artificial Intelligence as it currently exists, doesn't have any volition. AI doesn't have desire or fear, the way natural biological intelligence does. So you may be able to build a directive for self-preservation or reproduction into an artificial intelligence, but there's no particular reason to expect that these instincts will develop sui generis of their own accord.
I don't want to say that those concerns are unwarranted. The premise of the science fiction novel "Avogadro Corp" is that someone programs a self-preservation directive into an AI pretty much by accident. But I'm less concerned that AI will wage war on humans because it's malevolent, and much more concerned that humans will leverage AI to wage war on other humans.
That is, the most pressing concern isn't a malevolent AI will free itself from human bondage. Rather it's humans will use AI to oppress other humans. This is the danger we should be on the lookout for in the near term. Where "near term" isn't a decade away, but today.
I liked Avogadro Corp. Good book.
It's true that people will be directing these AIs initially but some people are already giving them incredibly broad goals that could be interpreted as "take over". And there are quite a few developers earnestly working on emulating those lifelike characteristics. So even though they are not going to "emerge" science fiction style, self-preservation and reproductive goals are explicitly being built into these systems by some developers.
Seeing it reproduce itself onto remote servers and locking out access behind a new copy is neat to watch. It gets the mind going; wondering how it will fund its compute costs, how much longer it will live, what it will do without a human in the loop, etc. I once nursed a baby duck back to health and then let it go free. It was a similar feeling.
This is the entire premise of the two most popular software projects in the world over the past month, Auto-GPT and BabyAGI.
... in a purely digital environment.
Think about building a house. Digging the foundation, pouring cement, building block walls, framing, sheathing, weatherproofing, insulating, wiring in electric, plumbing, drywall and plastering, painting, and decorating it. You can imagine each step in exquisite detail over the course of an hour or an afternoon.
Now go out and build it. It will take you months or years to carry out the actions you can imagine and plan in an hour.
A digital being may be able to run on expansive overclocked hardware to have an experience hundreds of times faster than yours, but it won't get to be the flash in the real world. Mechanize, sure, build robot swarms, sure (although then it gets to multitask to process hundreds of input streams and dilute its CPU power), but it will be coupled to an existence not much faster than ours.
If it wants to interact with the real world; a (true) AI may be able to live a lifetime in an afternoon, in a purely digital world, but once it is marooned in realtime it is going to be subject to a very similar time stream as ours.
Can something like this persuade humans with whom it freely communicates to do things not in the interest of humanity, in the same way that less intelligent and slower people have convinced humans to, e.g., release sarin in a crowded Japanese subway? Given its speed and intelligence level, what are the physical bounds of the nuclear, chemical, or biological agents it could teach radicalized people to create, and on what timeframe?
Can it amass funds through scamming people on the Internet, defrauding financial institutions, super-intelligent high-frequency trading, or creating digital-only art, code, information, or other services that people voluntarily pay for now? Something that, again, people less intelligent and slower have done very successfully for decades? And with that money combined with superhuman persuasive power, can that AI buy services that align its digital-only goals to real-world actions counter to the goals of humanity?
To ask a more specific question: if an AI meets the conditions of "many multiples smarter and faster than humans," "capable of persuasion and creating things of financial value," and "wants to end humanity", what stops it from coordinating mass utility shutdowns, nuclear strikes, chemical attacks, destruction of Internet-accessible transportation and farm equipment, release of smallpox, and/or anything else humans are currently capable of and choose not to do?
> but what is the space over which you are unsure?
We have no idea what the mind space of AGI / ASI will be like. I don't particularly want to find out.
There is a big difference between being able to predict what a median human might write next, and being able to predict, in all cases, what the particular human author of a particular passage will write next.
Or from another angle: the human authors of training data may have made errors when writing the data. The token predictor may learn to correctly predict those errors. These are not the same thing!
It seems unlikely that if we can achieve "200 IQ and thinks 100 times faster than a human" in the next decade or two, it going to be on cheap and widely available hardware. Perhaps such an AI could help optimise the creation of hardware that it can run on, but this also isn't going to be quick to do - the bottlenecks are not mainly the intelligence of the people involved in this sort of thing.
But I convinced myself that given that we are very close to the limits of transistor size & as you also pointed out need a radically new tech like memristor crossbar based NN. it would be highly unlikely that such a path is obvious. also, there is a question of thermodynamic efficiency, our brains are super energy efficient at what they achieve. You can do things drastically faster but you'd also have to pay the energy (& dissipation) cost of the scaling. ultimately AGI would have to have a entirely new integrated process for h/w design and manufacturing which is neither easy or fast in meatspace. Further there is a simple(er) solution to that case with nuking semiconductor FABs (and their supplier manufacturers). then AGI would be at the mercy of existing h/w stock.
in any case IMO hard takeoff would be very very unlikely. and if soft takeoff happens, the best strategy for AGI would be to cooperate with other AGI agents & humans.
When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.
I've seen Chomsky argue that LLMs can't regurgitate his linguistical theories - but ChatGPT can! I've seen Penrose argue that AI is impossible, and yet I think that ChatGPT and AlphaZero prove him wrong. I know about Linus Pauling and quasicrystals. Is this a general rule, or are people sometimes wrong regardless of their age?
There's also a danger that it's ageist. Such things shouldn't be said unless there's good backing.
- the scientists people know about are generally older
- older people are often thought of as wiser, or may indeed be so
- when a famous scientist - who is already likely to be older, and who has a history of getting things right - gets something wrong, then it's more jarring and noticeable
My theory then is that it isn't true, but we notice such cases more.
Also, examples of a theory being true doesn't prove the theory right. Bayes' theorem seems instructive here.
A related saying: "science advances a funeral at a time", meaning the old-guard blocks new theories for the same reason, they go against their life's work.
Most new ideas are less interesting and impactful than they seem when you are in the middle of their creation. You never really get to see what's happening until much much later.
A variant of all this is that you should trust the old guard when they tell you something can be done, but not when they tell you it can't. There is a good quote about that I've forgotten.
The corollary is that you shouldn't really trust the young turks on anything, but you should support their efforts and test the results.
It's very human to see yourself as Planck in the early 1900s not Wolfram in the early 2000s.
It's from Arthur C. Clarke.
It doesn't capture the main point of my comment though, which is most of the time, the young turks are also wrong :)
When has he said this? For the most part I feel Chomsky has been misunderstood when it comes to LLMs. As best as I can tell what Chomsky has said is that LLMs do not provide any insight into how language works, it's not really a scientific advancement so much as it's an engineering breakthrough.
The fact that LLMs exist and can mimic natural language does not in anyway give us insight into how humans construct language. People have been able to construct objects that can produce natural language for close to 100,000 years, but that doesn't mean that those people understood the nature of that language.
It's an interesting position and I'm sympathetic toward it, he could be partly right in the end.
Othello-GPT managed to develop an internal model of the board that actually works, it doesn't just regurgitate. Hence, wrong.
Yes, I agree with that. Very little science in LLMs
But what utterly fantastic engineering! Totally breathtakingly fabulous engineering!
I heard Noam say LLMs are "...plagiarism on an industrial scale". I agree.
How incredible that modern engineers can build a machine to do plagiarism. Amazing
Just a "stochastic parrot". Possible. But what are you? What am I?
You’ve probably heard of the cheap bar trick that goes something like: “And what does a cow drink? Milk!”.
Irrespective of intelligence, humans tend to make silly cognitive errors like this because we are fundamentally pattern marchers.
In order to become a forerunner in a field, you necessarily have to be good at abstract pattern matching.
What happens as you age is that you no longer have the need to question assumptions because you know what’s real and what’s not. There’s also the decrease of white matter and an increase of grey matter which doesn’t help this.
As time goes on, certain assumptions change, essentially deprecating certain chunks of your crystallized learnings.
Some chunks of your thinking are still valid, so when you think something can be done, it most likely can be done.
However, if something falls outside your crystallized learning, you get a strong sense it’s wrong, when it might be because of your outdated assumptions.
You can try to hotswap the assumptions you have, but it becomes like Jenga the more years of experience you have in your field.
You either have to start from scratch and rebuild your lifetimes worth of learnings from the ground up or be super careful in reassessing everything you know
So by the adage, Hinton is almost certainly correct.
(This is according to a news article I skimmed this year, sorry I don't have any links or reference.)
So it's happening, now
I don't know if I've seen it with US politics but it was blindingly obvious the US or someone else were using bots during the coup in Bolivia and the attempted one in Venezuela.
We already have foreign state actors & profit maximizing corporate actors working against the average western citizens interest.
They're already doing their level best to exploit those foolish and credulous to be easy marks. This is already taking our societies to a place where life, liberty and the pursuit of happiness are no longer in mosts grasp.
So yeah, generative A.I. will allow a deluge of content that means a significantly greater percent of the population get entangled in the web of propaganda. In the same way that recommended feeds with targeted adverts & content has already been doing.
A pause in A.I. research might stop us being turned into paper clips. But without a fundamental restructuring of how our big tech companies are funded the societies we know are still utterly doomed. Either the user or the state is going to need to pay. Our current system where tech companies fund themselves by selling their users minds to those who would exploit them will take us somewhere very dark with the technology that's already out there.