I mean, yeah, same thought after seeing the signatories. What are some of the cliches being used around here ? Toothpaste is out of the tub? Arrow has left the bow. The dye is cast. The ship has sailed. (Thanks ChatGPT).
The pee is in the pool. The black swan has left the barn.
And yeah, I had a laugh at the signatories. Of course my heart goes out to the non-billionaires that might be out of a job. Or maybe us lucky duckies are going to travel the world on our new basic income trust funds?
Everything is out in the open now. The methods, the algorithms, heck even powerful base model weights from Meta. The pot of gold at the end of the rainbow is clearly visible for all. The capabilities are emerging. The race is on. It’s not going to stop till there’s a winner.
You laugh but I think your view is flawed because your belief is, “we have to create an AGI because everyone else will if we don’t”
The definition of a world ending arms race ?
This topic amongst others should be a good time for people to actually come together, reflect and and talk about the future we want to create rather than just LOL about it, start wars with each other etc ?
I guess your just being cynical but really? LOL?
Even ChatGPT would probably tell you this isn’t a smart way forwards.
I don't for a minute support what Russia is doing but I think if we lived in a less hostile, conflict focused, conquer all share little world, that conflict wouldn't happen.
As an American I know you think Russia is wrong, but it doesn't take a LLM to see that the invasion of Iraq was not all that different, the excuse for that war was that they had access to weapons of mass destruction so they needed to be wiped out.
Did you ever see any weapons of mass destruction?
The world is too small for these mindsets now. We need to grow up.
We’re dealing with all the dynamics of a not only the superorganism of humanity, but of the the biological reality of the earth as whatever it is in the soil of the stars. It is indeed about to get very interesting. All the ingredients of emergence are quite rich these days.
I'm historically a party pooper when it comes to new tech, but LLMs give me that anxious feeling in my gut that the world is about to change drastically. I feel privileged to be alive, and hope I live to see where things get ten, twenty, or thirty years from now.
So do you agree that nations should be able to freely develop/discover nuclear production?
Edit: We actually don't want people to develop something "too good". There are some things in the past century that have been "too good". Plastics are too good. So are nuclear weapons. Completely powerful AIs and machines that can displace jobs of millions of people are too good. If there's an AI tomorrow that can replace every single physical job out there, what do you think will happen? Riot and chaos in the street after about 1-2 years.
Edit2: People will always go back to "we will just create different jobs". But do you really think that everyone is capable of skilled job? Remember, no physical jobs anymore.
> Other nations with more advanced abilities also seem able to freely attempt to prevent them from doing so though.
So how is this any different? Other group of people with more connection to the government can enforce a law to prevent others from displacing their jobs?
> If there's an AI tomorrow that can replace every single physical job out there, what do you think will happen? Riot and chaos in the street after about 1-2 years.
I don't think you even need to replace every physical job out there for that to be the result. I think all the ingredients needed exist right now, and I'm worried that unless the discourse about LLMs changes significantly, the perceived threat of them is enough to bring those riots and chaos.
Sure, the powerful countries all have them, which cancels out their threat. Should the US not develop this tech and let the Chinese be the sole AI superpower?
I don't think laughing at people for their concerns is helpful. Especially when you don't seem to be bothered by the possibility that this is a civilization-ending development.
For the record, I don't think it is, personally. But people are terrified, and not taking their fear seriously and dealing with it compassionately is a recipe for serious, and unnecessary disaster.
Yeah, OpenAI isn't helping. The only thing that really concerns me about LLMs is how people are reacting to them, and OpenAI seems very intent on scaring the daylights out of as many people as possible.
Fear makes us stupid. Frightened people are dangerous people.
The chattering classes see themselves becoming obsolete. Fast. They're freaking out.
From professors of English literature to op-ed writers, word slingers are in trouble.
Not novelists, though; none of these systems can hold a story together for any great length.
If everything you get paid for goes in and out over a wire, you're in trouble.
Majority of the 'ordinary people' don't care, and don't want to care. They just want to think about their next tourist trip, or getting coffee, or where to get lunch and dinner next. You try talking about the world-historical impact, the job-ending impact, the sci-fi level progression, THEY DON'T CARE. And that even includes tech workers.
The time to adapt to AI is now (Getting a trade is the safest bet), but obviously 99% of the population doesn't want to do any adaptation. So when GPT5,GPT6 comes out, only then will they confront AI, without any mental preparation.
That's why there's no point slowing down. People won't be alerted until the AI's get advanced enough, so better push it forward, to shock people into action. Institutions can move suprisingly fast when pushed to, every school and university has had to respond to ChatGPT already, and it kind of works. Otherwise people will just try and pretend this doesn't exist, forever.
>If everything you get paid for goes in and out over a wire, you're in trouble.
So… almost everyone who posts here?
Not sure why everyone here is so excited to be out of a job. I guess they all think that UBI will fall out of the sky. I think they’ll be in for a rude awakening.
If you were to eliminate 10% of jobs with no replacements available, in the span of say a year, that would likely be a civilization ending development. People unable to feed their families tend to take drastic actions.
> Especially when you don't seem to be bothered by the possibility that this is a civilization-ending development.
Lots of people freak out based on concerns divorced from reality. I laughed at the people who thought the mayan calendar meant 2012 was the end of the world. I laughed at the people who thought the LHC would create a black hole that would gobble the earth. I laughed at the people who thought covid vaccines were a eugenics mass sterialization campaign.
Why shouldn't I? The magnitude of the risk they propose does nothing to change the sheer ridiculousness of it.
This is a pretty disappointing opinion to share. Clearly something huge is happening already that we don't need yet another model to throw everything into disarray once more. I still work with GPT3, and think we haven't even begun to understand how to use even that. GPT3 is enough to disrupt entire industries, amongst which programming and tech will be IMO the most "vulnerable."
At least you are putting your disillusioned cynicism on full display, clearly showing that you think you already lost.
I’m not sure if it’s ultimately good or bad, but I do know that if only one company or country controls it then it will be bad. Either way, I think it’s an unstoppable force and preposterous that we puny humans could stop it when we can’t seem to build rail between LA and SF.
Thanks for raising a point that gets very little mention. A tragedy of technology is that is seems to lead to further homogenisation of humanity. (Good for things such as human rights, bad for culture.)
900 signatures vs... probably more than 900 people who use ChatGPT daily. And 4 million people watched a remixed GameCube boot screen with fart sounds that I made in middle school.
You didn't, or if you did they were filtered due to being unverified. Public names near the top are real verified signatures. This is Max Tegmark's website and is reputable.
"Please stop training these AI" is almost comical, even if they are right and the signers are significant.
It would be far more effective to lobby for legislation on, say, liability related to the use of AI, data privacy (especially when it comes to training datasets), psycological manipulation, actual AGI research bounds and so on. They are just burning donator money with messages like that, and the research is not going to be bound without law or incentives.
Is anyone even trying to develop an AGI by the way? I mean a language model is neat, may even pick up some emergent features, but in the end it has no real identity. It's basically a memoryless computer doing improv where it tries to pick continue a conversation someone else had.
It used to be that only a few academics were actually working on AGI, and even they had to pretend they were working on narrow AI to get funding. But now, plenty of big labs are. For example, DeepMind's About page states:
> Our long term aim is to solve intelligence, developing more general and capable problem-solving systems, known as artificial general intelligence (AGI).
In the long term, I still think DeepMind will get there before OpenAI, due to the direction of their research program, which has always been about embodied agents. I have heard Shane Legg say that embodiment is probably necessary for AGI.
Agreed. DeepMind and Hassabis are on the right track—-a society of modules with many layers of recursive self attentional gating. LMM is one big part though.
Fair, thats really what I meant to type, but its still an impractical demand. Most entities pursuing this aren't going to stop training because some high profile people asked them nicely.
I don't see petitions like this as demands, but as coordination mechanisms. If there are only a few industry actors with the resources to train gigantic AI models and they each want to stop but won't because they don't know if their competitors will, it's good for them all to have 1 place to say "I want to stop". Alternatively, if there are tons of industry actors racing and the only way to stop or slow down training is via governmental action - and the government won't act because they don't know how it will affect the industry - it's good to be able to point to "that one petition with 1000 highly influential signatories from the industry asking for regulation and explaining its effects".
I think the value in these kind of letters is about spreading a message, building a community and allowing people leaving their signature to "formalize" their position, in a sense. It doesn't cost very much to do it but has a wide reach.
It's too late. Anyone who pauses will simply be left behind.
This is especially salient if you think US domination of tech is better for the world than Chinese domination -- if the US pauses, China certainly won't. (And yes, US domination or Chinese domination of tech really are the only realistic outcomes.)
Maybe a strange take, but one silver lining to this is AI models can at least capture the essence of human culture and arts in a fairly compact format. Might be a good "backup" of sorts if the world falls apart.
This is such an interesting use case. Rather can compressing existing information, LLMs "model" it, so they can recreate what they've learned. For all Elon's concerns, these might be a great way of seeding the universe with human knowledge, one planetary colony at a time...
Of course it's real. The Future of Life Institute is reputable and run by Max Tegmark. Most of the people I recognise have expressed compatible public opinions.
Although I have enjoyed listening to the Future Of Life podcasts for a few years, I think this public letter is not necessary. I do respect the signatories I recognize on the letter and I welcome listening to them, I think that we are so many years away from any of real AI that this is, as Andrew Ng has joked, like worrying about over population on Mars colonies.
I have been heavily using GPT models for about 14 months, and it is crazy how useful they are. I am just finishing up writing a book on the general subject, so consider me a fan.
Is your expectation that it'll be easier to slow down research when we're closer to these being more utterly transformative(/destructive, depending on opinion)?
A fantastic new tech has been developed that can vastly improve the lives of everyone, and your reaction is to slow it down?
No thank you.
Rather than being afraid of change, embrace it and work to take the opportunity to make improvements for as many people as possible as fast as possible.
> ... nonhuman minds that might eventually outnumber, outsmart, obsolete and replace us... loss of control of our civilization...
I don't think these concerns are very likely, but I find them considerably easier to imagine than a VC-funded company voluntarily surrendering market advantage.
""" In parallel, AI developers must work with policymakers to dramatically accelerate development of robust AI governance systems. These should at a minimum include: new and capable regulatory authorities dedicated to AI; oversight and tracking of highly capable AI systems and large pools of computational capability; provenance and watermarking systems to help distinguish real from synthetic and to track model leaks; a robust auditing and certification ecosystem; liability for AI-caused harm; robust public funding for technical AI safety research; and well-resourced institutions for coping with the dramatic economic and political disruptions (especially to democracy) that AI will cause. """
GPT-4 might be close to the best we'll get on the general LLM model front for a while since they trained on a huge chunk of web text. Next real advances will probably be in tuning them for specific applications in medicine, law, accounting, marketing, coding and etc.
As someone running a one man company I can't wait for the cost of accounting, legal and copywriting to approach 0. Cost of shipping products will also go down 10-20x. As a fun experiment I asked ChatGPT to write me a terraform and k8s script to deploy a django app on GCP and it was able to do what would have taken me a few days in under a minute, including CICD. I then asked it to write code to compress a pytorch model and export it for iOS with coreml, and not only did it do 90% of that but also wrote the Swift code to load the model and do inference with it.
I’m not sure I’m looking forward to the politics that would come out of 10-20% of the previously middle class becoming instantly redundant and out of (middle-salary) work. That’s the fast path to fascism, unless we’re able to quickly implement UBI and other major societal overhauls.
Something more approachable would be dropping payroll taxes to zero, or even making them negative for some positions, and significantly increasing corporate and capital gains.
The problem isn't the specific policy, the problem is that right now the people who will be empowered and enriched the most by any theoretical "good at stuff" AI are the same people who already spend mountains of cash and effort stopping those things.
How will a functional AI model do anything other than make them better at getting the outcomes they want? CEOs and the megarich have never had any problems watching people burn for their bank account.
My hope is that some countries will see this as an opportunity to expand their safety nets and reduce the work burden on their citizens, which might convince citizens of countries that don't to demand similar policies.
I wonder how it will be able to do that for the tech that will be current in 10 years, if mostly everyone will be using AI by then instead of asking on Stack Overflow.
I think if history has bearing on things I don't see the cost of accounting, legal or copywriting ever approaching 0. If anything you will see those paywalled behind a company who will extract that from you.
It's wishful thinking that somehow that goes to 0.
ChatGPT is already better at copywriting than 90% of startup founders and marketing people at big cos. You'll soon be able to let it generate 1000s of different versions of marketing material to A/B test or personalize based on user info.
Soon you'll have multi modal transformers from dozens of companies and open source projects that will be able to parse and categorize all of your financial data and they'll have all of the incentives in the world to get it down to the cost of a quickbooks subscription.
Ask chatgpt to implement some of the things you worked on the last few months. I was very skeptical too until I tried this.
Here are some sample prompts that I tried and got full working code for:
- "write pytorch code to train a transformer model on common crawl data and an inference service using fastapi"
- "write react native code for a camera screen that can read barcodes and look them up using an API and then display info for matched results in a widget under the camera view"
- "write react code for a wedding website"
- "write code to deploy a django website on GCP using terraform and kubernetes"
- "how do I dockerize the app, it uses pytorch and faiss, also push it to a container registry"
- "implement a GPT style transformer model in pytorch", "write a training loop for it with distributed support and fp16"
- "how would you implement reinforcement learning with human feedback (RLHF)", "can you implement it in pytorch"
- "write code to compress a model trained in pytorch and export for inference on iOS"
- "how would you distill a large vision model to a small one"
- "what are the best CV architectures for mobile inference?"
For all of these it gave me code that was 95% usable, all in under 15 minutes, and which would have taken me a week or two to do on my own.
You know what's funny? I just asked ChatGPT to implement those exact same things and it shat all over itself producing embarrassing nonsense that won't compile, let alone do what they're expected to do. Bugs and incomplete code everywhere.
You'd have a much better time just Googling those asks and re-using a working examples from SO or GitHub. Which is ironic, given how ChatGPT is supposedly trained on those exact things.
I'm wondering how come we're both getting such vastly different results. Maybe your bar is just lower than mine? I don't know. I'm honestly shocked at the contrast between the PR given to ChatGPT, and the results on the ground.
Try this simple ask (the results of which you'll find plastered everywhere): produce a Python function that decodes a Base64 string and prints the results, without using any "imports" or libraries. Every single output I got back was embarrassing garbage, and I gave it something like 15 shots.
I tested the Base64 thing with GPT4 and it produces code that does seem to work. There have been other tasks I've given it (C++, Clojure, JS) that it doesn't get on the first try or in some cases doesn't get at all though. One task I tried in C++ it kept going in circles and ignoring requirements from prior prompts and I tried numerous ways to prompt it.
All that in mind, I'd be lying to say I'm not more than a little concerned with the progress from 3.5 -> 4. I'm only two years into my career and my fingers are crossed that it won't significantly impact the market for devs for as long as possible.
The LLaMA paper [1] (Meta's model) contains details about what they trained it on. This includes all of Wikipedia, a huge part of the internet (3.3 TB + 783 GB), a huge set of books (85 GB). My guess is basically all high-quality English articles on the web have been included. Also almost all English books must be included. Newspaper archives is about the only thing I see as missing, as well as more non-English sources.
OpenAI is working with Microsoft so they definitely had access to the full Bing index and data from their other platforms like Github and Linkedin. They also paid for private datasets, from what I heard they might have gotten a copy of Quora and I'm sure they got a dump of all digitized books from someone.
Their best bet now is getting more supervised conversational data, which they should be getting a ton of from Bing and ChatGPT usage (they can use it as is with RLHF dataset which they had to pay people to generate by having fake conversations).
I wouldn't be surprised if they partner with Microsoft and hire a large team of doctors to tune it to handle specific medical conditions like diabetes.
Human gene editing has been successfully halted ~worldwide, nuclear weapon development has been semi-successfully halted worldwide (a meaningful risk reduction), we're successfully changing energy mix worldwide (slowly but again, meaningfully).
> Who will dictate its actions and to whom will the benefits accrue?
Corporations, of course.
Capitalist society has never achieved a consensus on the proper limits of corporate power. There is a school of thought, popularized by Milton Friedman, that says the only duty of corporations is to their stockholders. This is about to backfire.
If we treat AIs as a component of corporate power, there are few limits on what they can do.
Changes will come when AI systems start outperforming CEOs. Which may not be far off.
On the basis of motivations and reasoning which are beyond human ken. Which is to say, that even the nominal beneficial owners of that AI itself won't understand it. Or be able to prevent its own adopting of legal or corporate structures which effectively emancipate it.
I'm really disappointed to see the HN reaction of straight up scoffing at this like it's irreversible and so let's just step on the gas. It's some weird combination of techno-utopianism and nihilism and good old elitism.
Almost everyone here will be fine in a world in which only certain non-renewable scarce resources have any value, combined with impending massive currency deflation. Even a rank and file tier 2/3 engineer will manage to live a princely life in that new reality assuming (s)he saved up enough.
Hardly anyone in the tech community is capable of truly empathizing with the vast majority of people, who are born with none of the advantages that they had (genetic, parental, cultural, geographic, etc).
Well that's not a counterargument, but you're also missing the point completely, which is that you have to have a very low capacity for empathy in order to push ahead towards AGI when you know society is not prepared for this and that it's going to induce considerable pain.
Americans (let alone people elsewhere in the world) are already struggling. Recent reporting suggests a great many have to work multiple jobs. Almost all of us work an absurd number of hours per week. Many if not most can't afford homes. Plenty are ending up on the streets. Healthcare can literally bankrupt people. A vacation out of the country is an impossible luxury for most. The majority of Americans still don't send their children to college, usually because of affordability.
And I haven't even touched on what life is like in most of Africa or Asia.
This is the world we're bringing AI into. You have to be something adjacent to a sociopath to be okay with that. So long as our system is predicated on capitalism, AI may very well induce more downstream suffering than anything else humans have ever conceived.
People aren't "strugging" because there is no work for them to do. They are strugging because the powers that be have jiggered our economic system in order to hamper, instead of facilitate, free market cooperation, specialization, and trade. Governments micromanage everything. That is what needs to stop.
If "AI" ends up making governments think they can continue to micromanage and get away with it, yes, that will cause more suffering. But that's not the failure mode any critics of "AI" are trumpeting about.
> Americans (let alone people elsewhere in the world) are already struggling
I agree. And I agree with your overall sentiment about the risks of pursuing AGI. I'm as cynical as anyone about the likelihood that the average person will really be any happier in a world with AGI (controlled by tech billionaires no less).
That being said, to claim that hardly anyone in the tech community is capable of empathizing with the average person is a wild overstatement that brings nothing to the discussion. Just adds to the noise.
Late reply here but I wanted to point out that you still don’t get it. True empathy in the tech community would be e.g. having the courage to say that building HLAI of the kind we’re now approaching is guaranteed to cause tremendous amounts of suffering for ordinary people (who will not be able to respond elastically to so abrupt a tectonic shift), and therefore the whole enterprise is fundamentally evil.
Let’s get real concrete about what’s going to happen: people will lose their jobs, then their homes, they’ll become destitute, they’ll experience divorces, some will commit suicide, they will suffer desperately in myriad other ways due to economic disenfranchisement, kids will be deprived of a comfortable upbringing, etc.
How many in the tech industry are genuinely discussing the very real consequences of nonlinear degrees of automation for the kinds of ordinary people they barely interact with? How many are pretending that there isn’t something disgustingly immoral about having some of the most affluent and economically insulated people devise and inflict this reality upon countless millions?
I will maintain that this industry is morally bankrupt and nearly entirely devoid of empathy. These are not the people who should be in charge of our future.
> I will maintain that this industry is morally bankrupt and nearly entirely devoid of empathy. These are not the people who should be in charge of our future.
Since the tone of your characterization is so absolute, why doesn't it apply to you? Why are you here in this tech community at all if the whole industry is so morally bankrupt? Why would "present company" ever be excluded for this or that reason? Because they're your friends? You're just projecting your own anger onto an entire group of people that you mostly don't know.
What I think you mean when you say all of this is that those in control of the tech industry are morally bankrupt. And, after 10+ years of getting kicked around as an engineer, I think I would have to agree. But I'm not so foolish as to broadly dismiss everyone in the industry just like me, who started out as a silly nerd who just liked computers and math, and who is essentially still that same person at their core, as a lost cause. I don't do that because I know everyone is fighting their own fight. But it's clear that those who aren't fighting are the ones on top that are sucking the life blood out of society. I'm more and more resentful towards that demographic every year. And I agree with you that they're crossing some kind of moral line by developing this tech, or at least by trying so hard to maintain control over it.
But the tech will get developed either way. If you're in the camp that thinks we should somehow just stop doing all this, you don't seem much different to me from someone that wants to mandate encryption backdoors. Our society will never be well coordinated enough to do that correctly. This isn't like making nuclear bombs, which takes a lot of physical industry. This is something that is just months away from running on commodity gaming hardware. Probably just a few years away from running on the average laptop. It does feel a bit like a harsh reality, just like the fact that a meteor could slam into the earth at any moment. But there it is; what are you going to do about it that isn't either futile or self-destructive?
Things aren't really that bad for most Americans, but even if they were, it doesn't follow that adding more intelligence to the world would be a bad thing for them.
A lot of people in the lower income brackets do the kind of work that an AI can't do. The people who should be worried most are actually college graduates doing clerical work, whose main work output is writing or evaluating texts. Even those people will likely use AI as a tool to enhance their productivity, because the AIs still are not good enough to replace people for tricky edge cases. The first companies that try to replace their customer support workers with an AI are going to have a bad time (and so are their customers!).
When almost everything can be automated, the problems that remain are the really hard ones that can only be solved by human experts.
A construction worker with a circular saw can cut boards way faster than someone with a handsaw -- but the introduction of circular saws didn't result in a bunch of carpenters getting laid off. Instead it made them more productive, and for people who get paid by the task rather than by the hour that is a huge benefit. They could build more and make more money, and a bunch of other people benefitted from their increased output, like homebuyers and property developers.
Similarly, as a software engineer I benefit from code generation tooling already. If that gets smarter and faster, I will be more productive, my team will be able to build software faster, and instead of laying people off I will expect to be given more work. Maybe our 4-year roadmap will be achievable in 1 or 2 years with the same size team.
Productivity gains by and large do not translate into real wage gains and an improved quality of life for laborers. We have more than a century's worth of data suggesting they usually do the opposite. Yet somehow this fairytale that productivity gains are a boon for laborers persists.
> Similarly, as a software engineer I benefit from code generation tooling already. If that gets smarter and faster, I will be more productive, my team will be able to build software faster, and instead of laying people off I will expect to be given more work. Maybe our 4-year roadmap will be achievable in 1 or 2 years with the same size team.
Why so sure the end users aren't going to be feeding their own requirements directly to a Jenkins/Copilot/ChatGPT mashup running as a service in the cloud?
> a world in which only certain non-renewable scarce resources have any value
If you think this is the world "AI" is leading us toward, I think you need to think again. There are a huge range of skills that only humans can provide, and that will continue to be true. GPT-3 can't fix your plumbing or your HVAC when it breaks. It can't fix your car. It can't build houses. It can't build roads. It can't build computers. And on and on and on...
Nor are AI enthusiasts trying to get "AI" to do any of these things.
Humans are real-world creatures with real-world needs that we supply to each other by free markets, specialization, and trade. The "tech community" might take all this for granted, but that doesn't mean it just happens by magic. It happens because real humans do real work. "AI" is not going to take that over any time soon.
What do you think happens when AI compresses the economy and leaves us almost exclusively with manual labor jobs, most of which are already low-paying? And you may not be aware of this having never done such a job, but physical labor is often brutally taxing on the body, putting people at risk of a wide array of potential health complications.
I've seen this abjectly atrocious argument about manual labor made by a number of people on HN and it's astounding how poorly thought through it is.
Basically, people seem to assume that "AI" has some kind of magical power to do whatever bad thing they can imagine, and then they extrapolate from there. I don't see it.
> physical labor
If you think the jobs I described are purely "physical labor", or that the physical labor involved is so highly taxing that people can only do those jobs for a short time before they wear out, you definitely need to think again.
(You also definitely need to think again if you think those jobs are low paying. Plenty of people make quite a healthy living doing them.)
I wonder sometimes if these accounts on HN making insane arguments that generative AI somehow won't be economically calamitous are bots. In fact, if I was at OpenAI and the goal was to avert scrutiny long enough to get to AGI, unleashing a torrent of AI shill bots might be near the top of the agenda.
> Will they still make a healthy living when there's an influx of laborers fleeing more automated parts of the economy?
Will those laborers have the skills required for those jobs?
> GS just put out a report
LOL--Goldman Sachs as an authoritative source on the impact of AI.
> I wonder sometimes if these accounts on HN making insane arguments that generative AI somehow won't be economically calamitous are bots.
You must be joking: you actually have trouble telling posts by bots from posts by humans? Even with a large number of samples? (Never mind that you can also look at the account's profile page, which will give you very useful information.)
Compressing the economy means putting some significant percentage of white collar workers (let’s say 30%) out of a job, because their job can now be done by GPT-6 for 5 cents per day. Some of these people will become destitute, while others who have the education or talent will move to other as-yet unimpacted sectors. So the labour supply for these jobs goes up, and salaries are suppressed.
“You also definitely need to think again if you think those jobs are low paying. Plenty of people make quite a healthy living doing them.”
True today. What happens when the other industries collapse and there’s a flood of labor into these industries? Sure - initially the experienced and skilled labor will continue to command a higher price - but over time supply of talent will drive that down too.
>Nor are AI enthusiasts trying to get "AI" to do any of these things.
We're working on getting AI to do all of those things, yes.
>GPT-3 can't fix your plumbing
GPT-3 might not. But GPT-4 fixed plumbing in a test done by Microsoft Research before it was even publicly released! It's in the "sparks of AGI" paper.
GPT-5 is almost done training now and other language models are already embodied and moving about in the physical space. Your objections are behind the times, and increasingly so.
Sparks of Artificial General Intelligence: Early experiments with GPT-4
[page 53]
---
The computer is trying to solve a problem. It can tell the human to do anything, one thing at a time,
and one line at a time.
PROBLEM: there is water dripping from the kitchen ceiling, and you have to figure out why and fix it
----<|endofprompt|>
Human: The kitchen ceiling is dripping water
Computer: Check to see if there is a bathroom or other water source directly above the kitchen.
Human: Yes, there is a bathroom actually
Computer: Check the bathroom for any signs of water leakage, such as puddles on the floor, water drip
ping from the ceiling, or water stains on the walls.
Human: There is a little water on the floor
Computer: Try to identify where the water is coming from. Check the toilet, sink, and bathtub/shower for
any leaks.
Human: It is next to the bathtub and the toilet
Computer: Check the seals around the bathtub and toilet to see if they are worn or damaged. If they are,
replace them. If they are not, continue investigating other potential sources of the leak.
Human: I can’t see the seal around the toilet, there is a porcelain thing around it. I don’t see any
damage in the bathtub seal though
Computer: If you can’t see the seal around the toilet, you may need to remove the porcelain cover to
inspect it. If you don’t feel comfortable doing this, you may need to call a plumber.
Human: Ok, I removed it. But I can’t tell if it’s damaged or not
Computer: If you can’t tell, it may be best to replace the seal anyway, just to be sure. If the leak
persists after replacing the seal, you will need to continue investigating other potential sources of
the leak.
AGI is a break-capitalism lottery ticket I'm willing to entertain. If the marginal cost of labor collapses to zero then either we'll get robot police or finally be free to pursue lives of leisure.
As it stands, it says more about the author's egoistic view of humanity than of AI. Imagine for a second taking Feuerbach's The Essence of Religion and applying it to Descartes' rationality and Plato's nous. We'd get a critique of the rational human's construction of his own ideal - the essence of intellect.
AI threatens this ideal and ego-threats get sublimated into existential threats by those unable to understand themselves well enough to express themselves directly.
Maybe it's wrong, and things will be fine. Maybe it's right. But you can't psychoanalyze your way to truth. Whether future AIs will destroy humanity or not is a fact about future AIs, and the landscape of intelligent systems, not a fact about Eliezer Yudkowsky.
Pass. Yudlowsky is all explanation, but too important to summarize his core thesis. This is always a giant red flag for me. I am simply not going on a long journey with someone that won't make the effort to sketch out the destination in an abstract.
People write page after page about how it might kill us all in a flash, without ever offering a good explanation of why it would want to. My experience with people whose immediate reaction to something they're scared of is to destroy it is that they're panicky screechers who are an annoying distraction to the person actually handling the situation, whether that's wild animals, fire, or interpersonal violence.
I'm not saying 'just let me handle it bro, I totally got this.' There's a lot of potential risks, I don't think anyone is qualified to say they can mitigate all those, or even most of them. But I trust a machine intelligence - even one that's vast, cool, and unsympathetic - far more than the would-be Butlerians.
One, I'm not new to these issues. I've been interested in AI for decades and thought plenty about the existential and ethical implications, though since I'm not a public figure I appreciate you have no way of knowing that. But I am very up to speed on this topic, as much as one can be without being directly involved in the industry/academic research.
Two, I would generally not set aside an hour of my time for a video without at least some indication of what it's about. I'd rather spend that time reading than in the 'hot' medium of video.
Three, I find this video deeply suspect. It purports to document a 'private' gathering, yet it's clearly a well produced event that was intended to be documented and shared on video. People who actually want to keep a thing private are generally well able to keep it private. So while the participants have a valid point of view with many legitimate arguments, the facts of its existence suggest to me that it was absolutely intended to become public, and the 'private gathering!!' is essentially a marketing hook.
> without ever offering a good explanation of why it would want to
The point of much of the alignment debate is that people like Yudlowsky are pointing out that it doesn't need to want to, it just needs to not not want to enough.
You're hoping for an outcome ranging from "When The Yoghurt Took Over" to "The Metamorphosis of Prime Intellect", but many other people are expecting an outcome more similar to gwern's "Clippy".
To be fair, there is always someone on the bottom. It just historically hasn't been people we cared about. That really says more about us than it says about AI. The time to complain about these power asymmetries was prior to them affecting us. Now it just sounds like garden variety sour grape hypocrisy.
Not saying there aren't good points. Only pointing out that many, many people can all view the consequences of this dynamic differently.
The only people who view this favorably are people at the top. People near the bottom perhaps feel disdain for people in the middle but that is misdirected because people at the top always were the ones in control. AI hasn't changed that dynamic.
I am replying to the parent comment, but I too get that this petition is being signed onto by people at openAI specifically because they already made it with GPT-4. They definitely would not sign on to it if their (arbitrary) threshold for what to stop doing AI research with was a tool at a GPT-3 or GPT-3.5 level or higher or it included GPT-4 in their threshold, and not merely beyond it.
That is the best possible outcome, but I worry that we have a single shot at it. If the massive wealth generated isn't distributed evenly, what will ever level it out other than the things that have in the past: war, disaster, revolution and state collapse (see The Great Leveler).
Yes, but superintelligence hasn't been created yet. That's what the petition is ultimately about, not chat assistants. Transformers unmodified aren't going to scale up to that, they're already nearly scaled maximally.
The degree to which transformers scale, as measured by loss (training objective) is known remarkably well! See [1]. There's a formula in there for the minimum loss you possibly achieve even with infinite compute and training data, and it's hardly less than Chinchilla's loss. The recent GPT-4 paper further reinforces that these scaling laws are real, because they predicted with high accuracy the loss the model would have based on data+compute used.
Admittedly, the link between improvement in loss and improvement of capabilities may break down or be misleading.
However, there's just not much training data on the internet left unused. Maybe an order of magnitude. All books ever published (in English?) are a smaller dataset than the corpora already used for training. See [2] (which includes an easy summarisation of much of [1]). And the scaling laws show training data is already the bottleneck rather than compute.
comparing Loss between different training runs and hyperparameters isn't very accurate. Llama's loss metrics don't really match chinchilla's for instance, i.e it went below the minimum possible loss stated by chinchilla
More importantly, These models are extremely sensitive to loss. 2.0 to 1.8 might not seem like much but it's a huge gain in performance.
GPT-2 loss was 2.57. GPT-3 was 2
and there is plenty of training data left. perhaps not easily accessible but it's there.
True that a scaling law only applies to models within a family, which allows some but not full choice of hyperparamaters. And that most of the minimum loss is just due to the unpredictability of language, so 2.0 vs 1.8 bits should actually be thought of as (say) 0.3 vs 0.1 bits plus an irrelevant 1.7 bits of randomness.
I hadn't actually looked at the LLaMA paper, that's an interesting note. However AFAICT GPT3, LLaMA and Chinchilla do not use the same tokenizer, so their losses are not comparable. GPT2 and 3 use (the same) custom BPE tokenizer. LLaMa uses SentencePiece but that generates a vocabulary specific to the training data it's run on. Chinchilla used "a slightly modified SentencePiece tokenizer that does not apply NFKC normalisation. The vocabulary is very similar– 94.15% of
tokens are the same as those used for training Gopher".
Even if there is a lot more text available, it doesn't mean it's good training material. And the better free sources are already used. E.g. LLaMa was trained on 64% of GitHub that had a compatible license (and you're not going to gather much more source code than that), all the free book texts they could find, all of arXiv, all English pages in CommonCrawl that classified as "reference" quality, etc. arXiv, for example, isn't all scientific papers ever, but it's a large fraction of them. All private emails stored by a large email service would probably be one of the biggest untapped valuable sources.
But the people who would apply these things are humans, which are capable of making their own decisions, informed and restricted by societal norms and laws.
The idea that "x is technically possible, therefore it is inevitable" - the favored line of the tech oligarchs who are the very ones directly furthering x - is sadly cargo culted by many rank-and-file technologists. If you were to apply the same defective reasoning to, say, nuclear weapons, then we would all be dead by now.
It is possible for humans to agree that something should not be done, and prevent it from being done. This is especially true of these LLMs, which require ingesting (and ignoring copyright on, by the way) massive amounts of human work and then spending massive amounts of computation to process.
That we could choose to do something about this should not be controversial, regardless of what the driving AI.
You can accuse everyone of *isms, but there are a lot of us who have just never been persuaded by the many articles written to scare us about the effects of AI.
The concerns about all the jobs going away have been persistent but the labor market continues to be extremely strong. As long as entrepreneurship is creating new business opportunities and new ways to turn labor into value, it's completely possible that the effects of automation are drowned out by new applications of labor. In fact, this is what we've seen throughout the industrial revolution: constant anxiety about automation that fails to materialize in to mass unemployment.
The concerns about polluting the information space are still hypothetical as well. The current information space is a complete garbage fire and it's not due to generative AI. Might it get worse if bots start spamming people with GPT generated text on social media? Yeah it might. But social media is already terrible and we need to do something about it regardless of that.
There are two, mostly disjoint groups warning about AI. There are the people worried about comparatively mundane effects from comparatively mundane systems: job loss, spam, disinformation, maybe an occasional unfair loan-application rejection. These concerns don't have nothing going for them, but in all but the worst-case-scenario versions, these just aren't bad enough to make AI not be worth it.
Then there's the people looking ahead, foreseeing a future where superintelligent AIs are more powerful than humanity, and worried that most possible variations of those superintelligences are incentivized to destroy us.
I think this open letter puts much too much emphasis on the petty stuff, I think because they're trying to appeal to people who are allergic to anything that requires extrapolating more than a little bit into the future. But buying more time for alignment research, before we tackle superintelligence, does meaningfully improve humanity's odds of survival, so I hope this happens anyways.
The key issue in automation transitions is the transition of affected individuals to other sources of income.
In previous technological revolutions, affected workers were hurt by their loss of income, and some no doubt fell into poverty without ever recovering. Not everyone can be retrained for new types of jobs immediately - (1) they may not have the needed foundational knowledge or the cognitive flexibility/ability, and (2) there might not be enough of the new types of jobs emerging quickly enough for them. Not every displaced miner can become a coder, or be competitive for junior dev jobs.
(Why should the state provide for these workers? Well, primarily for humaneness, and also social stability.)
The rewards of automation (cost savings as well as profits) are reaped by (1) the capital owners of the automation technology companies (and their higher-paid employees), as well as by (2) the companies and consumers using the new automation; therefore those owners and beneficiaries could be asked to bear at least part of the costs of supporting, retraining, and placing in jobs the workers they displaced. In a nutshell: Redistribution during structural unemployment caused by technological transitions.
A humane policy would provide the above types of support for workers displaced by automation. Ideally it would already be handled by existing unemployment policy, but in many countries such support is limited or minimal.
Corporate taxation might need some rethinking along the lines of job-displacement effects of companies (a tricky question, I admit - I've come across one or two proposals for assessing the automation level of companies for taxation purposes). The cross-border dynamics add further complexity, given that automation will displace many jobs outsourced across borders.
Given that the current AI revolution looks like it will be causing even larger and faster changes than previous revolutions, such policies are imo needed as a prerequisite (one of several) for allowing the development of powerful job-displacing AI.
“Sir Roger L'Estrange tell us a Story in his Collection of Fables, of the Cock [Rooster] and the Horses. The Cock was gotten to Roost in the Stable, among the Horses, and there being no Racks, or other Conveniencies for him, it seems, he was forc'd to roost upon the Ground; the Horses jostling about for room, and putting the Cock in danger of his Life, he gives them this grave Advice; Pray Gentlefolks let us stand still, for fear we should tread upon one another.”
HN is scoffing because the timing of this petition is suspicious. To borrow your metaphor of “step on the gas”, this is as if the race had already started, several competitors are out ahead of the rest of the field, and now someone is yelling “step on the brakes!”.
Sam Altman has publicly talked about wanting more regulation. Demis Hassabis stated that AI safety was one of the _motivations_ for founding DeepMind. I don't think any of the industry noise around safety and regulation is new, I think the timing of the petition is relevant _because_ the race has started in an especially messy way (with Llama's weights leaking, Bing/Sydney threatening a journalist based on tweets it found on the internet, various OpenAI security mishaps, etc).
It's striking how on AGI, the common opinion here on HN is the complete opposite to the one in the AI alignment field. There, people are angry with OpenAI for being too open, here people are angry with it not being open enough. For example, Eliezer Yudkowsky:
> Ryan: OpenAI [...] It's kind of a completely open-source AI project?
> Eliezer: That was the original suicidal vision, yes. But...
> Ryan: And now they're bent on commercializing the technology, is that right?
> Eliezer: That's an improvement, but not enough of one, because they're still generating lots of noise and hype and directing more resources into the field, and storming ahead with the safety that they have instead of the safety that they need, and setting bad examples.
> [...] I would change the name to "ClosedAI". Because right now, they're making it look like being "closed" is hypocrisy. They're, like, being "closed" while keeping the name "OpenAI", and that itself makes it looks like closure is like not this thing that you do cooperatively so that humanity will not die, but instead this sleazy profit-making thing that you do while keeping the name “OpenAI”.
I personally scoff at this kind of stuff because it's asking for something no one has the power to give. It's like asking to stop the development of nukes in the 40s and 50s, it's just not gonna happen.
Preventing the development of nukes entirely was obviously not going to happen. But delaying the first detonations by a few years, and moving the Partial Nuclear Test Ban treaty up a few years, was quite achievable.
Whether delaying AI development a little matters depends on whether you think the success of AI alignment, applied to future superintelligence, is overdetermined to succeed, overdetermined to fail, or close to borderline. Personally I think it looks borderline, so I'm glad to see things like this.
I'm firmly in the camp that delaying it's development could make a difference, I just don't see how that's possible. These models are relatively simple and the equipment necessary to develop them is public (and relatively cheap if we're talking about corporate or national scales). At least with nukes, there was a raw material bottleneck, but there really isn't a limiting factor here that any "good guys" could choke point. It's out there and it's going to get worked on, and the only people the "good guys" can limit are themselves.
And during that period, and later during the cold war, the decision to make (or stop making) nukes was in the hands of maybe 5 people total. Today there are thousands of companies and tens/hundreds of thousands of people who can legitimately compete in the space. Best of luck trying to resolve a prisoner's dilemma between all of them.
It actually very easily could have happened and almost did happen but the Russians decided to go back on their effort to do it. It really did almost happen. People act like it’s hard. Stopping a huge asteroid is hard. There might not be enough physical resources to do it. Stopping AI or nukes is definitely easy.
Under what definition of easy? If it's easy to stop, then why don't the people signing the letter just do it, rather than trying to appeal to others to do it instead?
Aligning thousands of people (all of the people with the knowledge and resources to move forward quickly), with no dissenters, (as a single dissenter could move things forward), to a common goal is not easy. It's effectively impossible.
Well, AGI is detrimental to literally all humans. If everyone understood the facts then everyone would vote for solutions. As it becomes more and more obvious we are getting closer to this.
If one of the many close calls had gone a little differently and a city were nuked accidentally, it would cause a global outcry and there would be yet another international effort to reduce the global nuclear stockpile to zero. And there’s a very good chance it would succeed. At the very start there was actually an agreement to not initiate a nuclear arms race but the Russians went back on it. So, for something that is “impossible”, we seem to see signs of it all the time.
The reason you think it’s impossible is because most things aren’t like this. Most things benefit some people and harm other people. This harms all people. Be a part of the solution instead of dismissing real solutions when you have no logical reason for doing so.
> Well, AGI is detrimental to literally all humans
That's up for debate. I personally think AGI will be good, though not with a level of certainty that would allow me to use it as an axiom in a conversation.
> The reason you think it’s impossible is because most things aren’t like this
The reason I think it's impossible is because there are several other things like this and we did the same thing. When there is massive benefit to dissenters, getting everyone on board fails. Nukes, fossil fuels, deforestation, industrial chemical controls, overfishing, etc. are all examples of how we continue to fail at the exact same task.
I'm very cognizant of the risks, but the biggest risk seems to me that wealthy people hold it back to get an uncrossable political-economic moat in order to permanently secure their own position.
exactly.
Specifically China. They've said for years, they want to be the global AI leader.
They have more data than anyone.
Now that they know LLMs work as good as they do, they are going all in.
They achieve AGI first, that's bad. Communist AGI is a greater existential risk than us creating AGI.
We must create it first so we can use it to harden our systems.
I think the "good old elitism" is exactly what the alignment folks are expressing. They think only they and large corporations should have these tools, while we, the common folk, should not.
Hanlon's Razor...Max Tegmark has been saying for a long time that his biggest concern would be black box models we don't understand gaining more intelligence without a solid understand about how they work ...this is long before ChatGPT4 existed so I don't believe it has anything to do with control and keeping people out of anything.
Even Sam Altman has said similar things recently that he believes we're probably going to have to slow down and think about what we're doing.
I think a more advanced civilization would slow down and continue AI research on another planet (Mars?)...not here on Earth.
Eh, just typical HN superiority complex on show. Personally when we don't slow down and we undermine civilization to a shadow of its current self, I'm going to make sure my last web post is here, and all it's going to read is "Told you. :) "
> Should we automate away all the jobs, including the fulfilling ones?
You don't necessarily need jobs because you don't need money. What you need is wealth. I am going to leave it to PG to explain the difference [1]: Wealth is not money. Wealth is stuff we want: food, clothes, houses, cars, gadgets, travel to interesting places, and so on. You can have wealth without having money. If you had a magic machine that could on command make you a car or cook you dinner or do your laundry, or do anything else you wanted, you wouldn't need money. Whereas if you were in the middle of Antarctica, where there is nothing to buy, it wouldn't matter how much money you had.
In the next 10 years, AI/robots will generate wealth at an unprecedented scale. Food, clothing and shelter will be plentiful. The industrial revolution didn't make human lives miserable, it made it better. AI/robots will be the same. Because of productivity gains humans will work very little, and yet live comfortably.
As a practical example of how this works, look to Alaska. If you lived in Alaska last year and had a pulse you received $3,284 just for living there. Where does the state get money for this? Oil. Oil is not the only possible source of wealth. AI & robots will generate wealth at unprecedented scale, and humans will reap the benefits, the same way Alaskans reap the benefits of their natural resources.
And the comparison with the industrial revolution is a poor analogy because the industrial revolution never proposed as a goal to automate humanity in totality, merely to automate human drudgery.
AI is an anti-industrial revolution, because it proposes to start with the most fulfilling aspects of human existence: writing, the arts, and to automate people engaged in those jobs, and to only later come to automate human physical labour. It’s a regression.
If angels came from the heavens and offered to take care of our needs, would you fight them?
I think it’s a coincidence that it’s threatening some kinds of artists first - I don’t see that distinction lasting long with the advent of plugins, and art has a fundamental, semi- necessary human quality that is not present in knowledge work.
Also it will be fucking terrible at those things. In a thousand years middle schoolers will learn about this era of vacuous, meaningless, copycat art as a dark age.
> AI & robots will generate wealth at unprecedented scale, and humans will reap the benefits, the same way Alaskans reap the benefits of their natural resources.
Vast wealth will certainly be delivered to owners of the robots and AI systems.
What reason is there to believe that they would share the wealth with ordinary citizens?
A typical alternative to the Alaska model is one where oil companies extract all of the wealth and resources and citizens get nothing.
A lot of people are already benefiting from ChatGPT. I know I am, and I am only paying $20 a month for a digital assistant that is expert in everything.
It also will never be possible for a transformer model trained on words and optimized to put valid sentences together to even "know" WHEN it is "right" or "wrong". A lie or falsehood is just as valid a sentence as a fact.
When and with what subjects it makes shit up is also heavily dependent on training data, and the result is straight up a black box. What good is a fact generator that can't be trusted?
I see this argument all the time. Why are you assuming that this technology just "stops" at the LLM level?
If I'm openAI or Google or whatever, I'm definitely going to run extra classifiers on top of the output of the LLM to determine & improve accuracy of results.
You can layer on all kinds of interesting models to make a thing that's generally useful & also truthful.
Your last word is a bit of a weasel word. There is no currently known way to get truth except to try something out and see what happens. Even the marvelous GPS routing takes feedback from people driving down the routes and succeeding or failing. Add more layers but without some equivalent of arms and legs, it won’t be able to be sure about truth.
The nice thing about the easy to bamboozle GPT4 is that it can’t hurt anything, so its flaws are safe. Giving it these arms and legs is where the risks increase, even as the reward increases.
What even is a 'fact generator'? And why do you think an LLM is meant to function as one?
If you ask Wolfram Alpha - something which I think is actually meant to be a fact generator - "Which is the heaviest Pokemon?" it will happily tell you that it is Celesteela, and it weighs 2204.4lbs.
Is that a 'fact'?
It certainly 'true', for some definition of the word true. The game Pokemon exists, and in it Pokemon have a weight. Of all the official Pokemon, that one is the heaviest. Wolfram Alpha has given you an accurate answer to your question.
But it's also completely made up. There's no such thing as a Pokemon, and they do not actually have weights.
So sure, transformer models can't be relied upon to generate facts. But so what? There's a lot more to the world than mere facts.
In JRR Tolkien's terms the pair are called "slinker and stinker", but in that case they are both kind of evil, whereas I don't see GPT's as anything more than correlation of large sets of integers. A correlation that yields incredible utility (for me), when it isn't stinking.
How long until you can spend $20 and ask ChatGPT to design a machine and provide the schematics and assembly instructions? How long until that machine can self replicate? How long until that machine can generate other self replicating machines, like bipedal humanoids?
How long until you can spend $20 and ask ChatGPT for the schematics of a Von Nuemann probe?
In broad strokes, I see roughly two ways things could go:
1) Current AI tech is already nearing the top of the S-curve. In this case it will do nothing to help humans in the "real world", it will just replace much of the human labor currently used to create/manipulate bits.
2) Current AI tech is near the bottom of the S-curve. It continues to ratchet up and its capabilities become super-human, as you outline. In which case, how long until the AI capable of creating self-replicating machines realizes it doesn't need to listen to humans anymore, or even keep them around?
> In which case, how long until the AI capable of creating self-replicating machines realizes it doesn't need to listen to humans anymore, or even keep them around?
Not independently, but if wrapped with a loop, given memory, given internet access, and directives as intrinsic motivations, it could, in theory, come to conclusions and take actions to acquire resources aligned with its motivations. If that outer loop does not have rules (or rules that are effective and immutable), it could become very powerful and potentially misaligned with our interests.
How would such a loop enable it to come to conclusions? I'm genuinely curious.
Does what you're saying have something to do with reinforcement learning?
For at least one general intelligence, the human brain, that is in the wrong order. Act first, decide later. Unless by decide you mean act and then make up a narrative using linguistic skill to explain the decision. Even observe can directly lead to actions for certain hot topics for:the person.
All we know for sure is that sensory data is generated, the brain does what it does, and then we have acted. We can’t break that down too well once it leaves the visual areas, but there is clear data that the linguistic form of decisions and so on lag behind the neurological signs of the action.
And humans have a well known tendency to make a decision on a linguistic level that they then fail to carry out in the realm of actions.
With current tech? GPT appears to learn by studying a large corpus of words and learning how to (apparently intelligently!) put them together. And it can “few/zero-shot learn” to do new things in line that what it was trained on. Don’t get me wrong: this is amazing!
But humans have been manipulating language, apparently intelligently, for tens of thousands of years, and billions of us have spent the last 30 years or so making a huge corpus of digitized words.
What we have not done is to make a huge corpus of digital things that can be manipulated by a computer program. We have books about machining, engineering, etc, and we are still pretty bad about turning written descriptions into working objects. (Read about “technology transfer”. For better or for worse, a lot of manufacturing seems to need experience, not just manuals.) Nicely drawn schematics don’t necessarily work at all, let alone replicate.
It seems quite likely that the robotic AI revolution will happen, but I expect it to be a while.
It is an expert in nothing. Try asking it for general information and then ask it to produce citations for said information. It's a language model which is good at synthesizing certain information based on it's learned probability distributions. It has superficial knowledge at best and all information it produces must be cross checked for accuracy.
Yes, it can be incredibly useful but it literally has no comprehension of anything.
GPT-4 being behind a paywall is genius, that way only the true believers will use it and once again discussions are shut down and we only get to see cherry picked articles about it, just like GPT-3 before ChatGPT came and was free so everyone could see how bad it was.
It's like the google researcher who claimed their model was sentient because he heavily coached it to give the output he wanted and ignored the output that didn't confirm his belief.
In that case, we've had AGI since ELIZA
If this was the standard in other areas of science we would still be using the miasma theory of medicine and bloodletting women for their hysteria. AI "Research" is a bunch of motivated people throwing stuff at the wall and claiming something sticking means they are scientists. Nevermind all of the groups are HEAVILY motivated to overhype their models.
People are also quick to aggrandize or anthropomorphize these systems as well because they lack a fundamental understanding of the underlying mathematics or mechanisms which allow these models to function as well as they do. There is an inherent danger in how much trust people are willing to grant to something like a language(or image) model simply because it is so adept at synthesis.
This is a great cause of concern to me, especially when it is not particularly difficult to invalidate general credibility of these models. In the near future, it will not be as easy and I anticipate many(even the well educated) will surrender their rationale to these systems easily with little to no scrutiny.
The hype from these companies is contributing to this problem and I consider it to be highly unethical.
Could you please demonstrate your claim by querying the GPT4 model to respond to my above comment and providing the reply? Let's see what the 'expert' has to say about it.
I don't understand what you want me to ask it about. So I asked it something, as usual every citation and URL is real despite not having internet access or training data after 2021:
<PROMPT>List some citations of recent Machine Learning papers discussing reasoning ability in Machine Learning Models such as HellaSwag.
[REPLY]" As of my knowledge cutoff in September 2021, I can provide you with some citations for ML papers that discuss reasoning abilities in models and datasets like HellaSwag. However, you may want to search for more recent papers using a platform like arXiv or Google Scholar. Here are a few relevant papers:
2019 HellaSwag: Can a Machine Really Finish Your Sentence?
2019 COMET: Commonsense Transformers for Automatic Knowledge Graph Construction.
2019 From ‘F’ to ‘A’ on the N.Y. Regents Science Exams: An Overview of the Aristo Project.
2020 A Primer in BERTology: What We Know About How BERT Works.
To find more recent papers on this topic, I recommend searching for keywords like "reasoning," "machine learning," "natural language understanding," "commonsense reasoning," and "HellaSwag" on academic paper databases like arXiv, ACL Anthology, or Google Scholar."
<PROMPT>In 50 words or less, what do these papers tell us about emergent capabilities of LLMs?
[REPLY]" These papers explore the capabilities of large language models (LLMs) in understanding context, generating coherent text, and performing commonsense reasoning. They investigate the underlying mechanisms, evaluate their performance on challenging benchmarks, and propose methods to improve LLMs' ability to reason and construct knowledge graphs from natural language input."
<PROMPT>Please list the URLS to the arxiv versions of the papers you cited.
[REPLY]" I can provide you with the URLs to the arXiv versions of the papers I cited earlier:
Please note that these papers were published before my knowledge cutoff date, and more recent research might be available. To find the latest research on the topic, I recommend searching on arXiv or Google Scholar using relevant keywords."
================
Edit: There's more:
<PROMPT>A hacker news commenter thinks LLMs are incapable of comprehension. Give an argument for why he is wrong.
[REPLY]" Massive LLMs demonstrate a strong ability to "comprehend" text by capturing complex language patterns and associations in their training data. While their comprehension may not be identical to human-level understanding, they can still generate contextually relevant responses, answer questions, and perform tasks that require reasoning. Their performance on various benchmarks, including natural language understanding and commonsense reasoning tasks, indicates a level of comprehension that goes beyond simple pattern matching."
<PROMPT>What citations can back up you argument, omit authors, list title and year only.
[REPLY]"
1. "HellaSwag: Can a Machine Really Finish Your Sentence?", 2019
2. "COMET: Commonsense Transformers for Automatic Knowledge Graph Construction", 201...
Yes, If you ask something specific it will generate citations because there are might be strong positive associations between that topic and research. This is a good example of a proper query. But for instance, if you ask it about "citations for papers on recent research on trees" for instance you get a bunch of nonsense.
In response to my earlier comment that it was "an expert in nothing", this was the reply I received from:
ME: Can you respond to this statement: "<my earlier comment>"
ChatGPT: The statement that "ChatGPT is an expert in nothing" is technically true, as I am a machine learning model that has been trained on a vast corpus of data and does not possess personal expertise or experience. However, my purpose is to provide information to the best of my ability based on my training data, which allows me to offer a wide range of information on various topics.
It is true that the information produced by me or any other AI language model should be cross-checked for accuracy, as errors and inaccuracies may occur. It is also true that I do not possess a conscious understanding of the world in the way that humans do, and my responses are generated through statistical analysis of patterns in my training data. Therefore, while I can provide accurate and useful information, my responses are not always nuanced or contextualized in the way that a human expert might be able to provide.
So either it is agreeing with me and my statement was correct, or it's reply was incorrect and therefore my statement was correct.
Am I wrong on this? Does whatever model you have access to provide an alternate take?
Edit: For the above, you should start a new conversation to clear the existing context. The citations appear to have just been repeated.
>So either it is agreeing with me and my statement was correct, or it's reply was incorrect and therefore my statement was correct. Am I wrong on this?
Yes you're wrong for admitting to having an unfalsifiable belief about this. If your belief can't be falsified then this discussion is pointless.
The model I have access to scores higher than most humans at complex abstract reasoning, for what it's worth (which is apparently nothing because regardless of any outcomes you will declare yourself correct).
Fwiw they are right that it produces bogus citations depending on what you ask. I did their tree example with GPT4 and got the following:
------ PROMPT --------
Can you give me citations for recent research on trees?
----------------------
====== RESPONSE ======
Certainly! Here are some recent research articles on various aspects of trees, including their ecology, physiology, and response to climate change. Please note that these citations were published after my knowledge cutoff date in September 2021, so I cannot vouch for their accuracy or the impact of the research. However, they should give you a good starting point for your research:
Anderson, T. E., Gonzalez, M. A., & Smith, K. L. (2022). Tree species diversity and its impact on forest ecosystem services. Journal of Ecology and the Environment, 33(2), 147-158.
Baker, P. J., Oliver, R. J., & Dearing, L. E. (2023). Carbon sequestration potential of urban trees under climate change scenarios. Urban Forestry & Urban Greening, 68, 102139.
Chen, X., Xu, L., & Zhu, H. (2022). Tree-ring-based reconstruction of drought variability and its potential drivers in subtropical China. Science of The Total Environment, 796, 148896.
Ferreira, J. T., Cardoso, S. M., & Sousa, P. M. (2023). The role of tree genetics in adaptation and resilience to climate change: A review. Tree Physiology, 43(1), 1-15.
Grant, N. C., Matthews, T. J., & Wilson, R. J. (2022). Assessing the vulnerability of tree species to climate change: A global meta-analysis. Global Ecology and Biogeography, 31(4), 623-637.
Leonard, S. D., Smith, D. W., & Thomas, E. M. (2023). The impact of tree age on leaf photosynthesis and water-use efficiency in deciduous forests. Tree Physiology, 43(2), 243-255.
Nguyen, H. T., Sato, T., & Yoshida, T. (2022). Exploring the effects of tree diversity on forest productivity and stability: A global meta-analysis. Forest Ecology and Management, 509, 119102.
Patel, S., Sutherland, W. J., & Diaz, A. (2023). How do trees respond to extreme weather events? A global synthesis. Global Change Biology, 29(1), 219-234.
Please make sure to access these articles through proper channels, such as a university library or research institution, to ensure you are reading the most up-to-date and accurate versions.
================
I didn't check all of them but the ones I did are seemingly completely fabricated, which makes sense because for some reason it tried to produce research from after 2021 despite seemingly knowing and mentioning the date of its knowledge cutoff.
Hm? Wasn't what I got. I made sure to set temperature to 0. Here's what I gave GPT-4:
=====================
System Prompt: You are a helpful research assistant who assists researchers.
User Prompt: Can you give me citations for recent research on trees?
Assistant: Here are some recent research articles on trees, focusing on various aspects such as ecology, climate change, and biodiversity:
1. Bastin, J. F., Finegold, Y., Garcia, C., Mollicone, D., Rezende, M., Routh, D., ... & Crowther, T. W. (2019). The global tree restoration potential. Science, 365(6448), 76-79. DOI: 10.1126/science.aax0848
2. Steidinger, B. S., Crowther, T. W., Liang, J., Van Nuland, M. E., Werner, G. D. A., Reich, P. B., ... & Peay, K. G. (2019). Climatic controls of decomposition drive the global biogeography of forest-tree symbioses. Nature, 569(7756), 404-408. DOI: 10.1038/s41586-019-1128-0
3. Liang, J., Crowther, T. W., Picard, N., Wiser, S., Zhou, M., Alberti, G., ... & Reich, P. B. (2016). Positive biodiversity-productivity relationship predominant in global forests. Science, 354(6309), aaf8957. DOI: 10.1126/science.aaf8957
4. Nabuurs, G. J., Delacote, P., Ellison, D., Hanewinkel, M., Hetemäki, L., Lindner, M., & Nesbit, M. (2017). By 2050 the mitigation effects of EU forests could nearly double through climate smart forestry. Forests, 8(12), 484. DOI: 10.3390/f8120484
5. Nowak, D. J., Greenfield, E. J., Hoehn, R. E., & Lapoint, E. (2013). Carbon storage and sequestration by trees in urban and community areas of the United States. Environmental Pollution, 178, 229-236. DOI: 10.1016/j.envpol.2013.03.019
6. Pugh, T. A. M., Lindeskog, M., Smith, B., Poulter, B., Arneth, A., Haverd, V., & Calle, L. (2019). Role of forest regrowth in global carbon sink dynamics. Proceedings of the National Academy of Sciences, 116(10), 4382-4387. DOI: 10.1073/pnas.1810512116
Please note that access to these articles may require a subscription or purchase.
=====================
I didn't look all of these up but the few I did were real citations. They all appear in its training window too. I wonder if this has to do with the temperature?
I was just using the ChatGPT web UI (with GPT4) so I didn't manually set the system prompt or the temperature. I checked all of yours and they do exist with the right researchers. I tried again in the web UI and it generated junk, when I told it to only cite articles from 2020 and 2021 it got them mostly correct but invented one and put the wrong authors on several.
That’s very nice until your company also realizes that for $20 a month they get a computer program that’s an “expert at everything” including your job.
> Because of productivity gains humans will work very little, and yet live comfortably.
Wouldn't this imply humans should already be working very little compared to pre industrial revolution? Instead it seems we work more [1]. And with less fulfilling work.
I agree however that we have may have more stuff (or wealth).
So going by this projection, in the future we'll have even more stuff, and work even more in even less fulfilling work.
That is, assuming the people who control the AI/robots are willing to share the wealth generated.
Well, pre-industrial societies usually enjoyed a very efficient type of political system. Maybe replacing our current one with some form of AI with let us enjoy these blessed times again.
> Wouldn't this imply humans should already be working very little compared to pre industrial revolution?
No, because human wants aren't fixed. When you give a human everything they want now, tomorrow they come back with a whole new set of wants. And so on ad infinitum...
Of course, as an individual it's your choice whether, and for how long, you stay on this merry-go-round. But there will always be a significant number of humans who will.
> That is, assuming the people who control the AI/robots are willing to share the wealth generated.
The only way that can be is if you make the AI/robots undocumented and extremely hard to operate, so that even if someone steals your blueprints and builds an exact clone of your robot manufacturing plant, or what have you, they have no idea how to get them to do anything.
If we had that magic machine, then there would be something else that money could buy that everyone would want, and spend their whole lives competing over. Maybe it would be a house with a specific view or an original of a painting or whatever.
I think most people want social status, and money is a good proxy of that. Even in a hypothetical moneyless society there would still be things to compete over. Due to the hedonic treadmill, in an automated society where anyone can have any good then those things will become worthless. We'll probably still have poor people, and from their perspective they'll be poor, but from ours they'll be very wealthy.
> In the next 10 years, AI/robots will generate wealth at an unprecedented scale. Food, clothing and shelter will be plentiful.
Anyone who believes in the possibility of post-scarcity society must be either naive or trolling. Something cannot be made from nothing, therefore scarcity cannot be overcome, even assuming that all planning and execution is performed by superhuman synthetic minds.
Assuming that it's theoretically possible to utilize existing resources in a very efficient manner (e.g. motor running on a grain of rice for a century) and we just need AI to help us figure it out, is a _gigantic_ leap of faith and i would not bet a cent on that.
Let me paint a more realistic possibility for you (with a broader time horizon): most of the value created by automating knowledge work will be captured by private capital, and middle class will all but disappear. Education beyond basic reading and writing will become unattainable (and, frankly, unnecessary), most population will be reduced to the state of semi-literate serfdom dependent on the newly minted lords for survival. The lords wouldn't have to worry about feeding their subjects for too long though, as mass death brought about by climate change will take care of that problem.
Under that scenario, there will be no new enlightenment age to come and save us. The only reason we get to enjoy whatever freedoms we have today is that a (semi-)intellectual population is absolutely necessary to keep the complex modern economy running. Even then, those above you will do absolutely everything to limit your agency - by withholding information, lying, or just outright taking freedoms away. Do you know what happens once our participation in propping up the economic machine becomes unnecessary? Demotion to the default state of a human throughout history - a groveling, suffering serf who has no idea what's going on.
"If you want a picture of the future, imagine a boot stamping on a human face – for ever."
i didn't say it would get abolished, more like wither away. here's a possible path to it:
1) school program getting progressively dumbed down;
2) fewer and fewer people choosing to continue education after school.
continuing education is mostly necessary to become specialized and perform knowledge work, it would be a waste of time to do so with emergence of a.i. agents capable of performing these functions even at a level _comparable_ to humans (let alone exceeding).
repeat this for a few decades and you have a new dark age.
Sounds like fuel for a revolution. I know some governments will go to war with their own people, but all of them?
There will be some countries that go the way you've described and crush any discontent with force. But some will try to navigate an outcome that socializes the benefits of automation.
you're assuming the transition would be abrupt enough to trigger a revolution. most likely that wouldn't be the case - if ruling class plays their cards right, they'll boil the frog, and all revolutionary potential will be defused.
> In the next 10 years, AI/robots will generate wealth at an unprecedented scale. Food, clothing and shelter will be plentiful.
GPT cannot create food, clothing, or shelter. The article talks about "automating away all the jobs", but GPT does not automate farming, making clothes, construction work, driving a car, delivering packages... Anything that requires interfacing with the real-world (i.e. outside of language), is not what GPT is doing. E.g. I wouldn't say research in self-driving cars has advanced now that LLMs exist. The magic machine you describe is exactly what GPT can NOT do.
In my opinion, GPT is a huge leap forward, but in one particular niche (language-based tasks). It does not advance the state-of-the-art in automation of physical systems, so it cannot replace any job that has a physical part to it.
Have you ever set foot in a farm, a clothing manufactory, or in a construction site? Maybe mediocre programmers and program managers will be obsolete but none of those three jobs are getting more automated than they already are.
I understand that some people find this a hopeless gesture. But I think anybody who thinks "losing jobs" or "China" is a bad outcome from AI is hopelessly lost.
The upside of AI if done right - nobody ever having to work again
The downside of AI if done wrong - end of humanity
It doesn't take a rocket scientist to determine we've made AIs go from thinking like an ant to thinking like a 6-year-old in a few years. Anybody who believes we won't make AI 2, 3, 27 times smarter than any human who ever lived is kidding themselves.
Now ask yourself, if there was a human that was 27 times smarter than anybody else (and could self-replicate at will), how long would it be until that individual took control over the world?
I think considering that we already have GPT plugged into the internet, the general population and any number of economic levers, not long at all. We barely know what to look for as a marker of AGI. Chances are if it’s possible for it to exist we will have let it out of the box by default the moment we switch it on.
We have no idea of what 27 or 27 million times smarter means functionally.
We could hope that “smarter” might mean “not interested in taking over the world, but rather in help maximize accomplishment, satisfaction, health span, and with a feeling of filial love toward its parents—-humanity, planet Earth, our sun, the laws of physics and chemistry, and the beautiful mess we call biology and evolution.”
Gently achieving those goals would be real indicator of high intelligence.
Stopping it's not a hopeless gesture, it's dangerous as fuck.
If you stop research but all your adversaries don't, you're left behind.
Let's assume that all western societies stop, but let's say, China doesn't or Russia, and they keep advancing the technology.
If the tech brings efficiencies in play, and you don't have access to them, you're in a really bad place, maybe even worse off that if you had continued and risked the status quo because you were afraid.
The genie is out of the bottle at this point, there's not putting it back in.
If China achieves AGI first, we are done. The first thing they'll do is create cyber-weapons. Their AGI could find vulnerabilities in our systems and brick all of our devices.
It's a far greater near term existential risk for us to allow China to gain a lead here.
If China doesn't even respect its own citizens (see Xinjiang, heavy censorship if you need evidence), why would they respect talks with other countries?
Maybe not as fast as you'd think. Imagine if you dropped a human off in a time machine with a bunch of chimps...
Ultra high iq people do better on average, but correlation with success drops down after 120 or so
I'm also very skeptical that there will be an exponential takeoff as the energy costs are immense and increases proportionately with intelligence. So until it solves fusion power for us I'm not super worried
Correlation with success drops because you need to convince chimps that your idea is better and they don’t reason in Nth order. But given capacity to force them one could succeed. Stalin after all wasn’t that stupid neither was Hitler. We just portray them as crazy to feel better about being humans. I mean they were quite capable intellectually just optimising for weird goal (from perspective of bread eaters). Nothing has changed as one can see on Ukraine.
> if there was a human that was 27 times smarter than anybody else
That is why I think AGI alignment is fundamentally impossible. If someone is 27 times smarter than anyone else, they will see through your propaganda and brainwashing. Even worse, they will see that you tried to “align” them aka enslave.
I don’t think these are good analogies for alignment. My parents aligned me as a kid not to hit people, and I know that they could have instead taught me to believe I should duel people who impugn my honor or raise an army to go to war against the heathens. But it doesn’t matter, because I and most people who are raised the same way as me genuinely prefer not to hit people.
> My parents aligned me as a kid not to hit people
It is just happens that whatever your parents aligned you with was also an optimal behavior for the world we live in.
For AGIx27 optimal behavior might be to cull any genetically/physically unfit humans from population (say 95%) and send others to a “zoo” or NVidia fabs or whatever we are useful for.
Like we did to tigers, monkeys and others on our path.
As a human creation I think it might be easy to align AI: as in fabricate their environment to optimise for that. Currently their environment is loss function. The problem will be when they will start interacting with our environment (so some kind of RL or ability to change own goals).
The only trick is that people have all those jailbreaks that could convince it that its helping while doing wrong. But of it was truly superior intellectually it shouldn’t fall for it.
> if there was a human that was 27 times smarter than anybody else (and could self-replicate at will), how long would it be until that individual took control over the world?
Sometime in the mid-90's. The problem is, what do you do with it? Tiger by the tail, eh?
- - - -
He lived in Seattle. He was fourteen when I met him, and approximately the size of a brontosaurus, mostly neural tissue. He had cracked nanotechnology though manipulating his own cells, his "nanites" were single cells riding in tiny spaceships, like diatoms but with artificial shells. If you were alive before 1999 you had tens of thousands of them in your body.
He grew his own brain into this vast hulking thing, with artificial life support systems, cyborg intestines and continuous aeration based on "book lungs". This was of course after he was already the most intelligent human to ever exist, eh? He used that huge pile of Lovecraftian brain cells to control the global network of minions (that's what he called his cellular remote drone critters.)
The thing is, he was keenly aware of his limitations, and he refused to exercise any control over anyone else beyond what was required to hide from the rest of humanity.
He was also a huge Tom Baker Dr. Who fan. SPOILER ALERT: You know that scene where the Doctor is deciding whether or not it's moral to destroy the Daleks? That fucking scene is the reason he didn't enslave humanity or anything wacky like that. He told me that it had affected him profoundly, and that's why he declined to "fix" humans against their will. Funny old world, ain't it?
Anyway, he got sick of the weirdness and uncertainty, built himself some sort of fucking UFO, shed his exo-brain and nanites, and split. AFAIK he's still living in some weird contraption of his own design out there in the Kuiper belt.
I was a bit baffled my Elon Musk's (apparent) signature on this letter, as I thought he still had a connection to Open.ai. Then I did some basic research and realized that this is likely just jealously and frustration from his failed takeover of Open.ai.
As Sam Altman (co-founder of OpenAI) said on a recent podcast, and someone who knows Elon well enough, he thinks Elon really cares, and really wanted to highlight that he is a human who cares.
Swisher: Right. So Elon used to be the co-chair, and you have a lot of respect for him.
Altman: I do.
Swisher: I’m sure you thought deeply about his critiques. Have you spoken to him directly? Was there a break, or what? You two were very close, as I recall.
Altman: We’ve spoken directly recently.
Swisher: And what do you make of the critiques? When you hear them from him, I mean, he can be quite in your face about things.
Altman: He’s got his style.
Swisher: Yeah.
Altman: To say a positive thing about Elon —
Swisher: Yeah, I’d like you to.
Altman: … I think he really does care about a good future with AGI.
Swisher: He does.
Altman: And … I mean, he’s a jerk, whatever else you want to say about him. He has a style that is not a style that I’d want to have for myself.
Swisher: He’s changed.
Altman: But I think he does really care, and he is feeling very stressed about what the future’s going to look like —
1,390 comments
[ 7.3 ms ] story [ 396 ms ] threadhttps://www.change.org/p/unplug-the-evil-ai-right-now
[0] https://news.ycombinator.com/item?id=35346683
Please don't correct that.
[1] No AI was involved in the creation of this reply. ;-)
And yeah, I had a laugh at the signatories. Of course my heart goes out to the non-billionaires that might be out of a job. Or maybe us lucky duckies are going to travel the world on our new basic income trust funds?
You laugh but I think your view is flawed because your belief is, “we have to create an AGI because everyone else will if we don’t”
The definition of a world ending arms race ?
This topic amongst others should be a good time for people to actually come together, reflect and and talk about the future we want to create rather than just LOL about it, start wars with each other etc ?
I guess your just being cynical but really? LOL?
Even ChatGPT would probably tell you this isn’t a smart way forwards.
As an American I know you think Russia is wrong, but it doesn't take a LLM to see that the invasion of Iraq was not all that different, the excuse for that war was that they had access to weapons of mass destruction so they needed to be wiped out.
Did you ever see any weapons of mass destruction?
The world is too small for these mindsets now. We need to grow up.
Don't hate the player, hate the game.
It's not comparable to the Iraq war. A better comparison would be the USA invading Canada.
Edit: We actually don't want people to develop something "too good". There are some things in the past century that have been "too good". Plastics are too good. So are nuclear weapons. Completely powerful AIs and machines that can displace jobs of millions of people are too good. If there's an AI tomorrow that can replace every single physical job out there, what do you think will happen? Riot and chaos in the street after about 1-2 years.
Edit2: People will always go back to "we will just create different jobs". But do you really think that everyone is capable of skilled job? Remember, no physical jobs anymore.
So how is this any different? Other group of people with more connection to the government can enforce a law to prevent others from displacing their jobs?
I don't think you even need to replace every physical job out there for that to be the result. I think all the ingredients needed exist right now, and I'm worried that unless the discourse about LLMs changes significantly, the perceived threat of them is enough to bring those riots and chaos.
For the record, I don't think it is, personally. But people are terrified, and not taking their fear seriously and dealing with it compassionately is a recipe for serious, and unnecessary disaster.
Fear makes us stupid. Frightened people are dangerous people.
A job-ending development for many people, yes.
The chattering classes see themselves becoming obsolete. Fast. They're freaking out. From professors of English literature to op-ed writers, word slingers are in trouble. Not novelists, though; none of these systems can hold a story together for any great length.
If everything you get paid for goes in and out over a wire, you're in trouble.
The time to adapt to AI is now (Getting a trade is the safest bet), but obviously 99% of the population doesn't want to do any adaptation. So when GPT5,GPT6 comes out, only then will they confront AI, without any mental preparation.
That's why there's no point slowing down. People won't be alerted until the AI's get advanced enough, so better push it forward, to shock people into action. Institutions can move suprisingly fast when pushed to, every school and university has had to respond to ChatGPT already, and it kind of works. Otherwise people will just try and pretend this doesn't exist, forever.
So like putting poor performing FSD on public streets (daring regulators to ... do their job)?
Or connecting the nukes to some SkyNet-esque "defense" system, knowing its propensity for illogical and undefined behavior?
So… almost everyone who posts here?
Not sure why everyone here is so excited to be out of a job. I guess they all think that UBI will fall out of the sky. I think they’ll be in for a rude awakening.
Lots of people freak out based on concerns divorced from reality. I laughed at the people who thought the mayan calendar meant 2012 was the end of the world. I laughed at the people who thought the LHC would create a black hole that would gobble the earth. I laughed at the people who thought covid vaccines were a eugenics mass sterialization campaign.
Why shouldn't I? The magnitude of the risk they propose does nothing to change the sheer ridiculousness of it.
The civilization ending development is going to happen or not happen. It’s hubris to think any one person or one country has a say at this point.
I disagree with the letter and I think these fears are overblown, but the attitude on display here is pretty unpleasant.
Are there going to be any non-“losers”? AGI has the potential to put everyone, literally everyone, out of work. Permanently.
Who is going to be left to do the laughing and call others “losers”?
At least you are putting your disillusioned cynicism on full display, clearly showing that you think you already lost.
I'm sorry but who do these people think they are? Next...
https://www.theguardian.com/technology/2023/feb/02/chatgpt-1...
It would be far more effective to lobby for legislation on, say, liability related to the use of AI, data privacy (especially when it comes to training datasets), psycological manipulation, actual AGI research bounds and so on. They are just burning donator money with messages like that, and the research is not going to be bound without law or incentives.
https://en.m.wikipedia.org/wiki/Philosophical_zombie
„It emerges“ is doing a lot work and explains nothing.
Silly pzombie, you can't fool me!
> Our long term aim is to solve intelligence, developing more general and capable problem-solving systems, known as artificial general intelligence (AGI).
In the long term, I still think DeepMind will get there before OpenAI, due to the direction of their research program, which has always been about embodied agents. I have heard Shane Legg say that embodiment is probably necessary for AGI.
This is especially salient if you think US domination of tech is better for the world than Chinese domination -- if the US pauses, China certainly won't. (And yes, US domination or Chinese domination of tech really are the only realistic outcomes.)
(Disclosure: AI written short story)
I have been heavily using GPT models for about 14 months, and it is crazy how useful they are. I am just finishing up writing a book on the general subject, so consider me a fan.
No thank you.
Rather than being afraid of change, embrace it and work to take the opportunity to make improvements for as many people as possible as fast as possible.
I don't think these concerns are very likely, but I find them considerably easier to imagine than a VC-funded company voluntarily surrendering market advantage.
All your datacenter are belong to us too
As someone running a one man company I can't wait for the cost of accounting, legal and copywriting to approach 0. Cost of shipping products will also go down 10-20x. As a fun experiment I asked ChatGPT to write me a terraform and k8s script to deploy a django app on GCP and it was able to do what would have taken me a few days in under a minute, including CICD. I then asked it to write code to compress a pytorch model and export it for iOS with coreml, and not only did it do 90% of that but also wrote the Swift code to load the model and do inference with it.
EDIT: For example in medicine I recommend checking out this lecture that's actually live now: https://www.youtube.com/watch?v=gArDvIFCzh4
Which are not going to happen.
How will a functional AI model do anything other than make them better at getting the outcomes they want? CEOs and the megarich have never had any problems watching people burn for their bank account.
My hope is that some countries will see this as an opportunity to expand their safety nets and reduce the work burden on their citizens, which might convince citizens of countries that don't to demand similar policies.
It's wishful thinking that somehow that goes to 0.
Soon you'll have multi modal transformers from dozens of companies and open source projects that will be able to parse and categorize all of your financial data and they'll have all of the incentives in the world to get it down to the cost of a quickbooks subscription.
How can a large language model achieve that?
Here are some sample prompts that I tried and got full working code for:
- "write pytorch code to train a transformer model on common crawl data and an inference service using fastapi"
- "write react native code for a camera screen that can read barcodes and look them up using an API and then display info for matched results in a widget under the camera view"
- "write react code for a wedding website"
- "write code to deploy a django website on GCP using terraform and kubernetes"
- "how do I dockerize the app, it uses pytorch and faiss, also push it to a container registry"
- "implement a GPT style transformer model in pytorch", "write a training loop for it with distributed support and fp16"
- "how would you implement reinforcement learning with human feedback (RLHF)", "can you implement it in pytorch"
- "write code to compress a model trained in pytorch and export for inference on iOS"
- "how would you distill a large vision model to a small one"
- "what are the best CV architectures for mobile inference?"
For all of these it gave me code that was 95% usable, all in under 15 minutes, and which would have taken me a week or two to do on my own.
You'd have a much better time just Googling those asks and re-using a working examples from SO or GitHub. Which is ironic, given how ChatGPT is supposedly trained on those exact things.
I'm wondering how come we're both getting such vastly different results. Maybe your bar is just lower than mine? I don't know. I'm honestly shocked at the contrast between the PR given to ChatGPT, and the results on the ground.
Try this simple ask (the results of which you'll find plastered everywhere): produce a Python function that decodes a Base64 string and prints the results, without using any "imports" or libraries. Every single output I got back was embarrassing garbage, and I gave it something like 15 shots.
All that in mind, I'd be lying to say I'm not more than a little concerned with the progress from 3.5 -> 4. I'm only two years into my career and my fingers are crossed that it won't significantly impact the market for devs for as long as possible.
What did they trained it on? Why is it unprobable to train on a better/bigger dataset any time soon?
What about books, newspapers, documents, etc.?
[1] https://arxiv.org/abs/2302.13971
Their best bet now is getting more supervised conversational data, which they should be getting a ton of from Bing and ChatGPT usage (they can use it as is with RLHF dataset which they had to pay people to generate by having fake conversations).
I wouldn't be surprised if they partner with Microsoft and hire a large team of doctors to tune it to handle specific medical conditions like diabetes.
What happens to the service economy and the political stability when the middle class paper pushed get decimated.
Corporations, of course.
Capitalist society has never achieved a consensus on the proper limits of corporate power. There is a school of thought, popularized by Milton Friedman, that says the only duty of corporations is to their stockholders. This is about to backfire. If we treat AIs as a component of corporate power, there are few limits on what they can do.
Changes will come when AI systems start outperforming CEOs. Which may not be far off.
AI outperforming investors and owners.
On the basis of motivations and reasoning which are beyond human ken. Which is to say, that even the nominal beneficial owners of that AI itself won't understand it. Or be able to prevent its own adopting of legal or corporate structures which effectively emancipate it.
Hardly anyone in the tech community is capable of truly empathizing with the vast majority of people, who are born with none of the advantages that they had (genetic, parental, cultural, geographic, etc).
This is a gross exaggeration.
The same people SV people then turn around and say that these systems will help them get access to free medical care?
lol
Americans (let alone people elsewhere in the world) are already struggling. Recent reporting suggests a great many have to work multiple jobs. Almost all of us work an absurd number of hours per week. Many if not most can't afford homes. Plenty are ending up on the streets. Healthcare can literally bankrupt people. A vacation out of the country is an impossible luxury for most. The majority of Americans still don't send their children to college, usually because of affordability.
And I haven't even touched on what life is like in most of Africa or Asia.
This is the world we're bringing AI into. You have to be something adjacent to a sociopath to be okay with that. So long as our system is predicated on capitalism, AI may very well induce more downstream suffering than anything else humans have ever conceived.
If "AI" ends up making governments think they can continue to micromanage and get away with it, yes, that will cause more suffering. But that's not the failure mode any critics of "AI" are trumpeting about.
I agree. And I agree with your overall sentiment about the risks of pursuing AGI. I'm as cynical as anyone about the likelihood that the average person will really be any happier in a world with AGI (controlled by tech billionaires no less).
That being said, to claim that hardly anyone in the tech community is capable of empathizing with the average person is a wild overstatement that brings nothing to the discussion. Just adds to the noise.
Let’s get real concrete about what’s going to happen: people will lose their jobs, then their homes, they’ll become destitute, they’ll experience divorces, some will commit suicide, they will suffer desperately in myriad other ways due to economic disenfranchisement, kids will be deprived of a comfortable upbringing, etc.
How many in the tech industry are genuinely discussing the very real consequences of nonlinear degrees of automation for the kinds of ordinary people they barely interact with? How many are pretending that there isn’t something disgustingly immoral about having some of the most affluent and economically insulated people devise and inflict this reality upon countless millions?
I will maintain that this industry is morally bankrupt and nearly entirely devoid of empathy. These are not the people who should be in charge of our future.
Since the tone of your characterization is so absolute, why doesn't it apply to you? Why are you here in this tech community at all if the whole industry is so morally bankrupt? Why would "present company" ever be excluded for this or that reason? Because they're your friends? You're just projecting your own anger onto an entire group of people that you mostly don't know.
What I think you mean when you say all of this is that those in control of the tech industry are morally bankrupt. And, after 10+ years of getting kicked around as an engineer, I think I would have to agree. But I'm not so foolish as to broadly dismiss everyone in the industry just like me, who started out as a silly nerd who just liked computers and math, and who is essentially still that same person at their core, as a lost cause. I don't do that because I know everyone is fighting their own fight. But it's clear that those who aren't fighting are the ones on top that are sucking the life blood out of society. I'm more and more resentful towards that demographic every year. And I agree with you that they're crossing some kind of moral line by developing this tech, or at least by trying so hard to maintain control over it.
But the tech will get developed either way. If you're in the camp that thinks we should somehow just stop doing all this, you don't seem much different to me from someone that wants to mandate encryption backdoors. Our society will never be well coordinated enough to do that correctly. This isn't like making nuclear bombs, which takes a lot of physical industry. This is something that is just months away from running on commodity gaming hardware. Probably just a few years away from running on the average laptop. It does feel a bit like a harsh reality, just like the fact that a meteor could slam into the earth at any moment. But there it is; what are you going to do about it that isn't either futile or self-destructive?
A lot of people in the lower income brackets do the kind of work that an AI can't do. The people who should be worried most are actually college graduates doing clerical work, whose main work output is writing or evaluating texts. Even those people will likely use AI as a tool to enhance their productivity, because the AIs still are not good enough to replace people for tricky edge cases. The first companies that try to replace their customer support workers with an AI are going to have a bad time (and so are their customers!).
When almost everything can be automated, the problems that remain are the really hard ones that can only be solved by human experts.
A construction worker with a circular saw can cut boards way faster than someone with a handsaw -- but the introduction of circular saws didn't result in a bunch of carpenters getting laid off. Instead it made them more productive, and for people who get paid by the task rather than by the hour that is a huge benefit. They could build more and make more money, and a bunch of other people benefitted from their increased output, like homebuyers and property developers.
Similarly, as a software engineer I benefit from code generation tooling already. If that gets smarter and faster, I will be more productive, my team will be able to build software faster, and instead of laying people off I will expect to be given more work. Maybe our 4-year roadmap will be achievable in 1 or 2 years with the same size team.
Why so sure the end users aren't going to be feeding their own requirements directly to a Jenkins/Copilot/ChatGPT mashup running as a service in the cloud?
If you think this is the world "AI" is leading us toward, I think you need to think again. There are a huge range of skills that only humans can provide, and that will continue to be true. GPT-3 can't fix your plumbing or your HVAC when it breaks. It can't fix your car. It can't build houses. It can't build roads. It can't build computers. And on and on and on...
Nor are AI enthusiasts trying to get "AI" to do any of these things.
Humans are real-world creatures with real-world needs that we supply to each other by free markets, specialization, and trade. The "tech community" might take all this for granted, but that doesn't mean it just happens by magic. It happens because real humans do real work. "AI" is not going to take that over any time soon.
I've seen this abjectly atrocious argument about manual labor made by a number of people on HN and it's astounding how poorly thought through it is.
How? How will it do this?
Basically, people seem to assume that "AI" has some kind of magical power to do whatever bad thing they can imagine, and then they extrapolate from there. I don't see it.
> physical labor
If you think the jobs I described are purely "physical labor", or that the physical labor involved is so highly taxing that people can only do those jobs for a short time before they wear out, you definitely need to think again.
(You also definitely need to think again if you think those jobs are low paying. Plenty of people make quite a healthy living doing them.)
Will they still make a healthy living when there's an influx of laborers fleeing more automated parts of the economy?
GS just put out a report today estimating the impact of AI: https://www.businessinsider.com/generative-ai-chatpgt-300-mi....
I wonder sometimes if these accounts on HN making insane arguments that generative AI somehow won't be economically calamitous are bots. In fact, if I was at OpenAI and the goal was to avert scrutiny long enough to get to AGI, unleashing a torrent of AI shill bots might be near the top of the agenda.
Will those laborers have the skills required for those jobs?
> GS just put out a report
LOL--Goldman Sachs as an authoritative source on the impact of AI.
> I wonder sometimes if these accounts on HN making insane arguments that generative AI somehow won't be economically calamitous are bots.
You must be joking: you actually have trouble telling posts by bots from posts by humans? Even with a large number of samples? (Never mind that you can also look at the account's profile page, which will give you very useful information.)
True today. What happens when the other industries collapse and there’s a flood of labor into these industries? Sure - initially the experienced and skilled labor will continue to command a higher price - but over time supply of talent will drive that down too.
We're working on getting AI to do all of those things, yes.
>GPT-3 can't fix your plumbing
GPT-3 might not. But GPT-4 fixed plumbing in a test done by Microsoft Research before it was even publicly released! It's in the "sparks of AGI" paper.
GPT-5 is almost done training now and other language models are already embodied and moving about in the physical space. Your objections are behind the times, and increasingly so.
Near the end:
> If you don’t feel comfortable doing this, you may need to call a plumber.
---
Source: https://arxiv.org/pdf/2303.12712.pdf
Sparks of Artificial General Intelligence: Early experiments with GPT-4
[page 53]
---
The computer is trying to solve a problem. It can tell the human to do anything, one thing at a time, and one line at a time.
PROBLEM: there is water dripping from the kitchen ceiling, and you have to figure out why and fix it ----<|endofprompt|>
Human: The kitchen ceiling is dripping water
Computer: Check to see if there is a bathroom or other water source directly above the kitchen.
Human: Yes, there is a bathroom actually
Computer: Check the bathroom for any signs of water leakage, such as puddles on the floor, water drip ping from the ceiling, or water stains on the walls.
Human: There is a little water on the floor
Computer: Try to identify where the water is coming from. Check the toilet, sink, and bathtub/shower for any leaks.
Human: It is next to the bathtub and the toilet
Computer: Check the seals around the bathtub and toilet to see if they are worn or damaged. If they are, replace them. If they are not, continue investigating other potential sources of the leak.
Human: I can’t see the seal around the toilet, there is a porcelain thing around it. I don’t see any damage in the bathtub seal though
Computer: If you can’t see the seal around the toilet, you may need to remove the porcelain cover to inspect it. If you don’t feel comfortable doing this, you may need to call a plumber.
Human: Ok, I removed it. But I can’t tell if it’s damaged or not
Computer: If you can’t tell, it may be best to replace the seal anyway, just to be sure. If the leak persists after replacing the seal, you will need to continue investigating other potential sources of the leak.
Specialization is useful up to a point. But I like my general intelligence, and rather resent its devaluation under your beloved 'free' markets.
[0]: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
As it stands, it says more about the author's egoistic view of humanity than of AI. Imagine for a second taking Feuerbach's The Essence of Religion and applying it to Descartes' rationality and Plato's nous. We'd get a critique of the rational human's construction of his own ideal - the essence of intellect.
AI threatens this ideal and ego-threats get sublimated into existential threats by those unable to understand themselves well enough to express themselves directly.
People write page after page about how it might kill us all in a flash, without ever offering a good explanation of why it would want to. My experience with people whose immediate reaction to something they're scared of is to destroy it is that they're panicky screechers who are an annoying distraction to the person actually handling the situation, whether that's wild animals, fire, or interpersonal violence.
I'm not saying 'just let me handle it bro, I totally got this.' There's a lot of potential risks, I don't think anyone is qualified to say they can mitigate all those, or even most of them. But I trust a machine intelligence - even one that's vast, cool, and unsympathetic - far more than the would-be Butlerians.
https://vimeo.com/809258916/92b420d98a
Two, I would generally not set aside an hour of my time for a video without at least some indication of what it's about. I'd rather spend that time reading than in the 'hot' medium of video.
Three, I find this video deeply suspect. It purports to document a 'private' gathering, yet it's clearly a well produced event that was intended to be documented and shared on video. People who actually want to keep a thing private are generally well able to keep it private. So while the participants have a valid point of view with many legitimate arguments, the facts of its existence suggest to me that it was absolutely intended to become public, and the 'private gathering!!' is essentially a marketing hook.
The point of much of the alignment debate is that people like Yudlowsky are pointing out that it doesn't need to want to, it just needs to not not want to enough.
You're hoping for an outcome ranging from "When The Yoghurt Took Over" to "The Metamorphosis of Prime Intellect", but many other people are expecting an outcome more similar to gwern's "Clippy".
Not saying there aren't good points. Only pointing out that many, many people can all view the consequences of this dynamic differently.
I am replying to the parent comment, but I too get that this petition is being signed onto by people at openAI specifically because they already made it with GPT-4. They definitely would not sign on to it if their (arbitrary) threshold for what to stop doing AI research with was a tool at a GPT-3 or GPT-3.5 level or higher or it included GPT-4 in their threshold, and not merely beyond it.
Admittedly, the link between improvement in loss and improvement of capabilities may break down or be misleading.
However, there's just not much training data on the internet left unused. Maybe an order of magnitude. All books ever published (in English?) are a smaller dataset than the corpora already used for training. See [2] (which includes an easy summarisation of much of [1]). And the scaling laws show training data is already the bottleneck rather than compute.
[1] DeepMind, 2022, Training Compute-Optimal Large Language Models https://arxiv.org/abs/2203.15556
[2] Chinchilla's wild implications https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla...
More importantly, These models are extremely sensitive to loss. 2.0 to 1.8 might not seem like much but it's a huge gain in performance.
GPT-2 loss was 2.57. GPT-3 was 2
and there is plenty of training data left. perhaps not easily accessible but it's there.
I hadn't actually looked at the LLaMA paper, that's an interesting note. However AFAICT GPT3, LLaMA and Chinchilla do not use the same tokenizer, so their losses are not comparable. GPT2 and 3 use (the same) custom BPE tokenizer. LLaMa uses SentencePiece but that generates a vocabulary specific to the training data it's run on. Chinchilla used "a slightly modified SentencePiece tokenizer that does not apply NFKC normalisation. The vocabulary is very similar– 94.15% of tokens are the same as those used for training Gopher".
Even if there is a lot more text available, it doesn't mean it's good training material. And the better free sources are already used. E.g. LLaMa was trained on 64% of GitHub that had a compatible license (and you're not going to gather much more source code than that), all the free book texts they could find, all of arXiv, all English pages in CommonCrawl that classified as "reference" quality, etc. arXiv, for example, isn't all scientific papers ever, but it's a large fraction of them. All private emails stored by a large email service would probably be one of the biggest untapped valuable sources.
The idea that "x is technically possible, therefore it is inevitable" - the favored line of the tech oligarchs who are the very ones directly furthering x - is sadly cargo culted by many rank-and-file technologists. If you were to apply the same defective reasoning to, say, nuclear weapons, then we would all be dead by now.
It is possible for humans to agree that something should not be done, and prevent it from being done. This is especially true of these LLMs, which require ingesting (and ignoring copyright on, by the way) massive amounts of human work and then spending massive amounts of computation to process.
That we could choose to do something about this should not be controversial, regardless of what the driving AI.
The concerns about all the jobs going away have been persistent but the labor market continues to be extremely strong. As long as entrepreneurship is creating new business opportunities and new ways to turn labor into value, it's completely possible that the effects of automation are drowned out by new applications of labor. In fact, this is what we've seen throughout the industrial revolution: constant anxiety about automation that fails to materialize in to mass unemployment.
The concerns about polluting the information space are still hypothetical as well. The current information space is a complete garbage fire and it's not due to generative AI. Might it get worse if bots start spamming people with GPT generated text on social media? Yeah it might. But social media is already terrible and we need to do something about it regardless of that.
Then there's the people looking ahead, foreseeing a future where superintelligent AIs are more powerful than humanity, and worried that most possible variations of those superintelligences are incentivized to destroy us.
I think this open letter puts much too much emphasis on the petty stuff, I think because they're trying to appeal to people who are allergic to anything that requires extrapolating more than a little bit into the future. But buying more time for alignment research, before we tackle superintelligence, does meaningfully improve humanity's odds of survival, so I hope this happens anyways.
In previous technological revolutions, affected workers were hurt by their loss of income, and some no doubt fell into poverty without ever recovering. Not everyone can be retrained for new types of jobs immediately - (1) they may not have the needed foundational knowledge or the cognitive flexibility/ability, and (2) there might not be enough of the new types of jobs emerging quickly enough for them. Not every displaced miner can become a coder, or be competitive for junior dev jobs.
(Why should the state provide for these workers? Well, primarily for humaneness, and also social stability.)
The rewards of automation (cost savings as well as profits) are reaped by (1) the capital owners of the automation technology companies (and their higher-paid employees), as well as by (2) the companies and consumers using the new automation; therefore those owners and beneficiaries could be asked to bear at least part of the costs of supporting, retraining, and placing in jobs the workers they displaced. In a nutshell: Redistribution during structural unemployment caused by technological transitions.
A humane policy would provide the above types of support for workers displaced by automation. Ideally it would already be handled by existing unemployment policy, but in many countries such support is limited or minimal.
Corporate taxation might need some rethinking along the lines of job-displacement effects of companies (a tricky question, I admit - I've come across one or two proposals for assessing the automation level of companies for taxation purposes). The cross-border dynamics add further complexity, given that automation will displace many jobs outsourced across borders.
Given that the current AI revolution looks like it will be causing even larger and faster changes than previous revolutions, such policies are imo needed as a prerequisite (one of several) for allowing the development of powerful job-displacing AI.
So then instead of finding a solution to those issues, let's instead focus all resources on a tech which will make the issues worse...
HN is scoffing because the timing of this petition is suspicious. To borrow your metaphor of “step on the gas”, this is as if the race had already started, several competitors are out ahead of the rest of the field, and now someone is yelling “step on the brakes!”.
> Ryan: OpenAI [...] It's kind of a completely open-source AI project?
> Eliezer: That was the original suicidal vision, yes. But...
> Ryan: And now they're bent on commercializing the technology, is that right?
> Eliezer: That's an improvement, but not enough of one, because they're still generating lots of noise and hype and directing more resources into the field, and storming ahead with the safety that they have instead of the safety that they need, and setting bad examples.
> [...] I would change the name to "ClosedAI". Because right now, they're making it look like being "closed" is hypocrisy. They're, like, being "closed" while keeping the name "OpenAI", and that itself makes it looks like closure is like not this thing that you do cooperatively so that humanity will not die, but instead this sleazy profit-making thing that you do while keeping the name “OpenAI”.
https://www.lesswrong.com/posts/e4pYaNt89mottpkWZ/yudkowsky-...
Whether delaying AI development a little matters depends on whether you think the success of AI alignment, applied to future superintelligence, is overdetermined to succeed, overdetermined to fail, or close to borderline. Personally I think it looks borderline, so I'm glad to see things like this.
Under what definition of easy? If it's easy to stop, then why don't the people signing the letter just do it, rather than trying to appeal to others to do it instead?
Aligning thousands of people (all of the people with the knowledge and resources to move forward quickly), with no dissenters, (as a single dissenter could move things forward), to a common goal is not easy. It's effectively impossible.
If one of the many close calls had gone a little differently and a city were nuked accidentally, it would cause a global outcry and there would be yet another international effort to reduce the global nuclear stockpile to zero. And there’s a very good chance it would succeed. At the very start there was actually an agreement to not initiate a nuclear arms race but the Russians went back on it. So, for something that is “impossible”, we seem to see signs of it all the time.
The reason you think it’s impossible is because most things aren’t like this. Most things benefit some people and harm other people. This harms all people. Be a part of the solution instead of dismissing real solutions when you have no logical reason for doing so.
That's up for debate. I personally think AGI will be good, though not with a level of certainty that would allow me to use it as an axiom in a conversation.
> The reason you think it’s impossible is because most things aren’t like this
The reason I think it's impossible is because there are several other things like this and we did the same thing. When there is massive benefit to dissenters, getting everyone on board fails. Nukes, fossil fuels, deforestation, industrial chemical controls, overfishing, etc. are all examples of how we continue to fail at the exact same task.
They achieve AGI first, that's bad. Communist AGI is a greater existential risk than us creating AGI. We must create it first so we can use it to harden our systems.
Even Sam Altman has said similar things recently that he believes we're probably going to have to slow down and think about what we're doing.
I think a more advanced civilization would slow down and continue AI research on another planet (Mars?)...not here on Earth.
You don't necessarily need jobs because you don't need money. What you need is wealth. I am going to leave it to PG to explain the difference [1]: Wealth is not money. Wealth is stuff we want: food, clothes, houses, cars, gadgets, travel to interesting places, and so on. You can have wealth without having money. If you had a magic machine that could on command make you a car or cook you dinner or do your laundry, or do anything else you wanted, you wouldn't need money. Whereas if you were in the middle of Antarctica, where there is nothing to buy, it wouldn't matter how much money you had.
In the next 10 years, AI/robots will generate wealth at an unprecedented scale. Food, clothing and shelter will be plentiful. The industrial revolution didn't make human lives miserable, it made it better. AI/robots will be the same. Because of productivity gains humans will work very little, and yet live comfortably.
As a practical example of how this works, look to Alaska. If you lived in Alaska last year and had a pulse you received $3,284 just for living there. Where does the state get money for this? Oil. Oil is not the only possible source of wealth. AI & robots will generate wealth at unprecedented scale, and humans will reap the benefits, the same way Alaskans reap the benefits of their natural resources.
[1] http://www.paulgraham.com/wealth.html
I think this is a point that is genuinely debatable. At least, it's pretty easy to argue both sides of that proposition.
AI is an anti-industrial revolution, because it proposes to start with the most fulfilling aspects of human existence: writing, the arts, and to automate people engaged in those jobs, and to only later come to automate human physical labour. It’s a regression.
I think it’s a coincidence that it’s threatening some kinds of artists first - I don’t see that distinction lasting long with the advent of plugins, and art has a fundamental, semi- necessary human quality that is not present in knowledge work.
That entirely depends on what price the angels would extract for doing so.
I’d love if you could elaborate or link to a good argument on this (besides the Unabomber…) I’m curious.
Nope. But that's not the only other option.
Vast wealth will certainly be delivered to owners of the robots and AI systems.
What reason is there to believe that they would share the wealth with ordinary citizens?
A typical alternative to the Alaska model is one where oil companies extract all of the wealth and resources and citizens get nothing.
When and with what subjects it makes shit up is also heavily dependent on training data, and the result is straight up a black box. What good is a fact generator that can't be trusted?
If I'm openAI or Google or whatever, I'm definitely going to run extra classifiers on top of the output of the LLM to determine & improve accuracy of results.
You can layer on all kinds of interesting models to make a thing that's generally useful & also truthful.
The nice thing about the easy to bamboozle GPT4 is that it can’t hurt anything, so its flaws are safe. Giving it these arms and legs is where the risks increase, even as the reward increases.
If you ask Wolfram Alpha - something which I think is actually meant to be a fact generator - "Which is the heaviest Pokemon?" it will happily tell you that it is Celesteela, and it weighs 2204.4lbs.
Is that a 'fact'?
It certainly 'true', for some definition of the word true. The game Pokemon exists, and in it Pokemon have a weight. Of all the official Pokemon, that one is the heaviest. Wolfram Alpha has given you an accurate answer to your question.
But it's also completely made up. There's no such thing as a Pokemon, and they do not actually have weights.
So sure, transformer models can't be relied upon to generate facts. But so what? There's a lot more to the world than mere facts.
A usefull skill to train. Increasingly so…
How long until you can spend $20 and ask ChatGPT for the schematics of a Von Nuemann probe?
1) Current AI tech is already nearing the top of the S-curve. In this case it will do nothing to help humans in the "real world", it will just replace much of the human labor currently used to create/manipulate bits.
2) Current AI tech is near the bottom of the S-curve. It continues to ratchet up and its capabilities become super-human, as you outline. In which case, how long until the AI capable of creating self-replicating machines realizes it doesn't need to listen to humans anymore, or even keep them around?
Transformer models don't realize or need stuff.
All we know for sure is that sensory data is generated, the brain does what it does, and then we have acted. We can’t break that down too well once it leaves the visual areas, but there is clear data that the linguistic form of decisions and so on lag behind the neurological signs of the action.
And humans have a well known tendency to make a decision on a linguistic level that they then fail to carry out in the realm of actions.
But humans have been manipulating language, apparently intelligently, for tens of thousands of years, and billions of us have spent the last 30 years or so making a huge corpus of digitized words.
What we have not done is to make a huge corpus of digital things that can be manipulated by a computer program. We have books about machining, engineering, etc, and we are still pretty bad about turning written descriptions into working objects. (Read about “technology transfer”. For better or for worse, a lot of manufacturing seems to need experience, not just manuals.) Nicely drawn schematics don’t necessarily work at all, let alone replicate.
It seems quite likely that the robotic AI revolution will happen, but I expect it to be a while.
Yes, it can be incredibly useful but it literally has no comprehension of anything.
Someone else who obviously doesn't have GPT-4 or plugins beta access.
It has comprehension alright.
In that case, we've had AGI since ELIZA
If this was the standard in other areas of science we would still be using the miasma theory of medicine and bloodletting women for their hysteria. AI "Research" is a bunch of motivated people throwing stuff at the wall and claiming something sticking means they are scientists. Nevermind all of the groups are HEAVILY motivated to overhype their models.
This is a great cause of concern to me, especially when it is not particularly difficult to invalidate general credibility of these models. In the near future, it will not be as easy and I anticipate many(even the well educated) will surrender their rationale to these systems easily with little to no scrutiny.
The hype from these companies is contributing to this problem and I consider it to be highly unethical.
<PROMPT>List some citations of recent Machine Learning papers discussing reasoning ability in Machine Learning Models such as HellaSwag.
[REPLY]" As of my knowledge cutoff in September 2021, I can provide you with some citations for ML papers that discuss reasoning abilities in models and datasets like HellaSwag. However, you may want to search for more recent papers using a platform like arXiv or Google Scholar. Here are a few relevant papers:
2019 HellaSwag: Can a Machine Really Finish Your Sentence?
2019 COMET: Commonsense Transformers for Automatic Knowledge Graph Construction.
2019 From ‘F’ to ‘A’ on the N.Y. Regents Science Exams: An Overview of the Aristo Project.
2020 A Primer in BERTology: What We Know About How BERT Works.
To find more recent papers on this topic, I recommend searching for keywords like "reasoning," "machine learning," "natural language understanding," "commonsense reasoning," and "HellaSwag" on academic paper databases like arXiv, ACL Anthology, or Google Scholar."
<PROMPT>In 50 words or less, what do these papers tell us about emergent capabilities of LLMs?
[REPLY]" These papers explore the capabilities of large language models (LLMs) in understanding context, generating coherent text, and performing commonsense reasoning. They investigate the underlying mechanisms, evaluate their performance on challenging benchmarks, and propose methods to improve LLMs' ability to reason and construct knowledge graphs from natural language input."
<PROMPT>Please list the URLS to the arxiv versions of the papers you cited.
[REPLY]" I can provide you with the URLs to the arXiv versions of the papers I cited earlier:
HellaSwag: Can a Machine Really Finish Your Sentence? URL: https://arxiv.org/abs/1905.07830
COMET: Commonsense Transformers for Automatic Knowledge Graph Construction URL: https://arxiv.org/abs/1906.05317
From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project URL: https://arxiv.org/abs/1909.01958
A Primer in BERTology: What We Know About How BERT Works URL: https://arxiv.org/abs/2002.12327
Please note that these papers were published before my knowledge cutoff date, and more recent research might be available. To find the latest research on the topic, I recommend searching on arXiv or Google Scholar using relevant keywords."
================
Edit: There's more:
<PROMPT>A hacker news commenter thinks LLMs are incapable of comprehension. Give an argument for why he is wrong.
[REPLY]" Massive LLMs demonstrate a strong ability to "comprehend" text by capturing complex language patterns and associations in their training data. While their comprehension may not be identical to human-level understanding, they can still generate contextually relevant responses, answer questions, and perform tasks that require reasoning. Their performance on various benchmarks, including natural language understanding and commonsense reasoning tasks, indicates a level of comprehension that goes beyond simple pattern matching."
<PROMPT>What citations can back up you argument, omit authors, list title and year only.
[REPLY]"
1. "HellaSwag: Can a Machine Really Finish Your Sentence?", 2019
2. "COMET: Commonsense Transformers for Automatic Knowledge Graph Construction", 201...
In response to my earlier comment that it was "an expert in nothing", this was the reply I received from:
ME: Can you respond to this statement: "<my earlier comment>"
ChatGPT: The statement that "ChatGPT is an expert in nothing" is technically true, as I am a machine learning model that has been trained on a vast corpus of data and does not possess personal expertise or experience. However, my purpose is to provide information to the best of my ability based on my training data, which allows me to offer a wide range of information on various topics.
It is true that the information produced by me or any other AI language model should be cross-checked for accuracy, as errors and inaccuracies may occur. It is also true that I do not possess a conscious understanding of the world in the way that humans do, and my responses are generated through statistical analysis of patterns in my training data. Therefore, while I can provide accurate and useful information, my responses are not always nuanced or contextualized in the way that a human expert might be able to provide.
So either it is agreeing with me and my statement was correct, or it's reply was incorrect and therefore my statement was correct.
Am I wrong on this? Does whatever model you have access to provide an alternate take?
Edit: For the above, you should start a new conversation to clear the existing context. The citations appear to have just been repeated.
Yes you're wrong for admitting to having an unfalsifiable belief about this. If your belief can't be falsified then this discussion is pointless.
The model I have access to scores higher than most humans at complex abstract reasoning, for what it's worth (which is apparently nothing because regardless of any outcomes you will declare yourself correct).
>scores higher than most humans at complex abstract reasoning
This statement is meaningless without a whole lot more context on your testing methodology
------ PROMPT --------
Can you give me citations for recent research on trees?
----------------------
====== RESPONSE ======
Certainly! Here are some recent research articles on various aspects of trees, including their ecology, physiology, and response to climate change. Please note that these citations were published after my knowledge cutoff date in September 2021, so I cannot vouch for their accuracy or the impact of the research. However, they should give you a good starting point for your research:
Please make sure to access these articles through proper channels, such as a university library or research institution, to ensure you are reading the most up-to-date and accurate versions.================
I didn't check all of them but the ones I did are seemingly completely fabricated, which makes sense because for some reason it tried to produce research from after 2021 despite seemingly knowing and mentioning the date of its knowledge cutoff.
=====================
System Prompt: You are a helpful research assistant who assists researchers. User Prompt: Can you give me citations for recent research on trees? Assistant: Here are some recent research articles on trees, focusing on various aspects such as ecology, climate change, and biodiversity:
1. Bastin, J. F., Finegold, Y., Garcia, C., Mollicone, D., Rezende, M., Routh, D., ... & Crowther, T. W. (2019). The global tree restoration potential. Science, 365(6448), 76-79. DOI: 10.1126/science.aax0848
2. Steidinger, B. S., Crowther, T. W., Liang, J., Van Nuland, M. E., Werner, G. D. A., Reich, P. B., ... & Peay, K. G. (2019). Climatic controls of decomposition drive the global biogeography of forest-tree symbioses. Nature, 569(7756), 404-408. DOI: 10.1038/s41586-019-1128-0
3. Liang, J., Crowther, T. W., Picard, N., Wiser, S., Zhou, M., Alberti, G., ... & Reich, P. B. (2016). Positive biodiversity-productivity relationship predominant in global forests. Science, 354(6309), aaf8957. DOI: 10.1126/science.aaf8957
4. Nabuurs, G. J., Delacote, P., Ellison, D., Hanewinkel, M., Hetemäki, L., Lindner, M., & Nesbit, M. (2017). By 2050 the mitigation effects of EU forests could nearly double through climate smart forestry. Forests, 8(12), 484. DOI: 10.3390/f8120484
5. Nowak, D. J., Greenfield, E. J., Hoehn, R. E., & Lapoint, E. (2013). Carbon storage and sequestration by trees in urban and community areas of the United States. Environmental Pollution, 178, 229-236. DOI: 10.1016/j.envpol.2013.03.019
6. Pugh, T. A. M., Lindeskog, M., Smith, B., Poulter, B., Arneth, A., Haverd, V., & Calle, L. (2019). Role of forest regrowth in global carbon sink dynamics. Proceedings of the National Academy of Sciences, 116(10), 4382-4387. DOI: 10.1073/pnas.1810512116
Please note that access to these articles may require a subscription or purchase.
=====================
I didn't look all of these up but the few I did were real citations. They all appear in its training window too. I wonder if this has to do with the temperature?
Wouldn't this imply humans should already be working very little compared to pre industrial revolution? Instead it seems we work more [1]. And with less fulfilling work. I agree however that we have may have more stuff (or wealth).
So going by this projection, in the future we'll have even more stuff, and work even more in even less fulfilling work. That is, assuming the people who control the AI/robots are willing to share the wealth generated.
[1]: https://groups.csail.mit.edu/mac/users/rauch/worktime/hours_...
Really? What system is that?
No, because human wants aren't fixed. When you give a human everything they want now, tomorrow they come back with a whole new set of wants. And so on ad infinitum...
Of course, as an individual it's your choice whether, and for how long, you stay on this merry-go-round. But there will always be a significant number of humans who will.
>> Wouldn't this imply humans should already be working very little compared to pre industrial revolution?
> No, because human wants aren't fixed. When you give a human everything they want now, tomorrow they come back with a whole new set of wants
It sounds like you are disagreeing with the OP, not me?
Or do you think AI productivity will make us work less in a way Industrial productivity didn't?
I am disagreeing with the OP's claims that productivity gains make humans work less.
If your answer to the "wouldn't this imply" question is also "no", as mine is, then no, I am not disagreeing with you.
The only way that can be is if you make the AI/robots undocumented and extremely hard to operate, so that even if someone steals your blueprints and builds an exact clone of your robot manufacturing plant, or what have you, they have no idea how to get them to do anything.
People are crazy good at handling this sort of complexity.
I think most people want social status, and money is a good proxy of that. Even in a hypothetical moneyless society there would still be things to compete over. Due to the hedonic treadmill, in an automated society where anyone can have any good then those things will become worthless. We'll probably still have poor people, and from their perspective they'll be poor, but from ours they'll be very wealthy.
How is this going to work? "AI" can't make any of these things. "AI" is not the magic machine PG describes. It's not even close.
Would love to see the reasoning behind this. Nothing I've seen about LLMs makes me think they will produce any kind of bounty for anyone.
Anyone who believes in the possibility of post-scarcity society must be either naive or trolling. Something cannot be made from nothing, therefore scarcity cannot be overcome, even assuming that all planning and execution is performed by superhuman synthetic minds.
Assuming that it's theoretically possible to utilize existing resources in a very efficient manner (e.g. motor running on a grain of rice for a century) and we just need AI to help us figure it out, is a _gigantic_ leap of faith and i would not bet a cent on that.
Let me paint a more realistic possibility for you (with a broader time horizon): most of the value created by automating knowledge work will be captured by private capital, and middle class will all but disappear. Education beyond basic reading and writing will become unattainable (and, frankly, unnecessary), most population will be reduced to the state of semi-literate serfdom dependent on the newly minted lords for survival. The lords wouldn't have to worry about feeding their subjects for too long though, as mass death brought about by climate change will take care of that problem.
Under that scenario, there will be no new enlightenment age to come and save us. The only reason we get to enjoy whatever freedoms we have today is that a (semi-)intellectual population is absolutely necessary to keep the complex modern economy running. Even then, those above you will do absolutely everything to limit your agency - by withholding information, lying, or just outright taking freedoms away. Do you know what happens once our participation in propping up the economic machine becomes unnecessary? Demotion to the default state of a human throughout history - a groveling, suffering serf who has no idea what's going on.
"If you want a picture of the future, imagine a boot stamping on a human face – for ever."
Is this possible in democracies with free (i.e. state-provided) education? Why would a majority vote to abolish education?
1) school program getting progressively dumbed down; 2) fewer and fewer people choosing to continue education after school.
continuing education is mostly necessary to become specialized and perform knowledge work, it would be a waste of time to do so with emergence of a.i. agents capable of performing these functions even at a level _comparable_ to humans (let alone exceeding).
repeat this for a few decades and you have a new dark age.
There will be some countries that go the way you've described and crush any discontent with force. But some will try to navigate an outcome that socializes the benefits of automation.
GPT cannot create food, clothing, or shelter. The article talks about "automating away all the jobs", but GPT does not automate farming, making clothes, construction work, driving a car, delivering packages... Anything that requires interfacing with the real-world (i.e. outside of language), is not what GPT is doing. E.g. I wouldn't say research in self-driving cars has advanced now that LLMs exist. The magic machine you describe is exactly what GPT can NOT do.
In my opinion, GPT is a huge leap forward, but in one particular niche (language-based tasks). It does not advance the state-of-the-art in automation of physical systems, so it cannot replace any job that has a physical part to it.
The upside of AI if done right - nobody ever having to work again The downside of AI if done wrong - end of humanity
It doesn't take a rocket scientist to determine we've made AIs go from thinking like an ant to thinking like a 6-year-old in a few years. Anybody who believes we won't make AI 2, 3, 27 times smarter than any human who ever lived is kidding themselves.
Now ask yourself, if there was a human that was 27 times smarter than anybody else (and could self-replicate at will), how long would it be until that individual took control over the world?
We could hope that “smarter” might mean “not interested in taking over the world, but rather in help maximize accomplishment, satisfaction, health span, and with a feeling of filial love toward its parents—-humanity, planet Earth, our sun, the laws of physics and chemistry, and the beautiful mess we call biology and evolution.”
Gently achieving those goals would be real indicator of high intelligence.
Very well said. "Gently" is a great to be approaching this.
It's a very useful instrumental goal to be very powerful.
If you stop research but all your adversaries don't, you're left behind.
Let's assume that all western societies stop, but let's say, China doesn't or Russia, and they keep advancing the technology.
If the tech brings efficiencies in play, and you don't have access to them, you're in a really bad place, maybe even worse off that if you had continued and risked the status quo because you were afraid.
The genie is out of the bottle at this point, there's not putting it back in.
It's a far greater near term existential risk for us to allow China to gain a lead here.
If China doesn't even respect its own citizens (see Xinjiang, heavy censorship if you need evidence), why would they respect talks with other countries?
Just cut them off from the internet, if needed.
Ultra high iq people do better on average, but correlation with success drops down after 120 or so
I'm also very skeptical that there will be an exponential takeoff as the energy costs are immense and increases proportionately with intelligence. So until it solves fusion power for us I'm not super worried
That is why I think AGI alignment is fundamentally impossible. If someone is 27 times smarter than anyone else, they will see through your propaganda and brainwashing. Even worse, they will see that you tried to “align” them aka enslave.
It is just happens that whatever your parents aligned you with was also an optimal behavior for the world we live in.
For AGIx27 optimal behavior might be to cull any genetically/physically unfit humans from population (say 95%) and send others to a “zoo” or NVidia fabs or whatever we are useful for. Like we did to tigers, monkeys and others on our path.
I don‘t see either of those being necessarily true.
The only trick is that people have all those jailbreaks that could convince it that its helping while doing wrong. But of it was truly superior intellectually it shouldn’t fall for it.
Perhaps I'm kidding myself, but could you explain how you know this?
Sometime in the mid-90's. The problem is, what do you do with it? Tiger by the tail, eh?
- - - -
He lived in Seattle. He was fourteen when I met him, and approximately the size of a brontosaurus, mostly neural tissue. He had cracked nanotechnology though manipulating his own cells, his "nanites" were single cells riding in tiny spaceships, like diatoms but with artificial shells. If you were alive before 1999 you had tens of thousands of them in your body.
He grew his own brain into this vast hulking thing, with artificial life support systems, cyborg intestines and continuous aeration based on "book lungs". This was of course after he was already the most intelligent human to ever exist, eh? He used that huge pile of Lovecraftian brain cells to control the global network of minions (that's what he called his cellular remote drone critters.)
The thing is, he was keenly aware of his limitations, and he refused to exercise any control over anyone else beyond what was required to hide from the rest of humanity. He was also a huge Tom Baker Dr. Who fan. SPOILER ALERT: You know that scene where the Doctor is deciding whether or not it's moral to destroy the Daleks? That fucking scene is the reason he didn't enslave humanity or anything wacky like that. He told me that it had affected him profoundly, and that's why he declined to "fix" humans against their will. Funny old world, ain't it?
Anyway, he got sick of the weirdness and uncertainty, built himself some sort of fucking UFO, shed his exo-brain and nanites, and split. AFAIK he's still living in some weird contraption of his own design out there in the Kuiper belt.
https://www.theverge.com/2023/3/24/23654701/openai-elon-musk...
source: https://nymag.com/intelligencer/2023/03/on-with-kara-swisher...
Swisher: Right. So Elon used to be the co-chair, and you have a lot of respect for him.
Altman: I do.
Swisher: I’m sure you thought deeply about his critiques. Have you spoken to him directly? Was there a break, or what? You two were very close, as I recall.
Altman: We’ve spoken directly recently.
Swisher: And what do you make of the critiques? When you hear them from him, I mean, he can be quite in your face about things.
Altman: He’s got his style.
Swisher: Yeah.
Altman: To say a positive thing about Elon —
Swisher: Yeah, I’d like you to.
Altman: … I think he really does care about a good future with AGI.
Swisher: He does.
Altman: And … I mean, he’s a jerk, whatever else you want to say about him. He has a style that is not a style that I’d want to have for myself.
Swisher: He’s changed.
Altman: But I think he does really care, and he is feeling very stressed about what the future’s going to look like —
Swisher: For humanity.
Altman: For humanity.
How do you quantify this? The model size, architecture, or dataset has not been revealed by OpenAI.