> Near the end of his column, he offers a pretty radical prescription. “One of two things must happen. Humanity needs to accelerate its adaptation to these technologies or a collective, enforceable decision must be made to slow the development of these technologies.
I'm constantly befuddled by the fact that "slowing the rate of technological progress by agreement/treaty" is really ever considered a possibility.
Take Bitcoin, torrents, or encryption for example. Good luck trying to stop people.
it doesn’t take masses of processing power to develop a crypto coin. or really any at all. it’s just a bunch of network calls combined with a hash function. any old idiot can develop one on a device with an IDE and access to the internet
yes, mining them can use a load of compute, but that’s not the point
Given LLAMA runs enough to be interesting on a Macbook I'm not sure this is going to be a fundamental limitation, and if it's one today it's certainly within the order of magnitude that models will run locally within a few years.
This is why attempting to put guardrails around it simply won't work tbh.
Sure, but that isn't like Bitcoin it doesn't have to scale up to every bit of usage. We are already getting interesting scenarios from models that are done and the energy spent. That isn't a fundamental limitation.
> Good luck trying to stop people. "Information wants to be free"
I agree, we’re struggling with the idea that our technological systems have more agency than us. I think it’s hard for technologists to see this as an actual spiritual reality, even though we borrow it metaphorically for argument.
These people constantly make a mistake by referring to "humanity" and saying "humanity" needs to make a decision, yada yada. Yes, we're all humans. But "humanity" doesn't have much decision making power. Instead the dominant entities with decision making power continue to be national governments. And it sounds a lot dumber to say "America needs to slow the development of these technologies" during a period of intense technological competition with China.
Torrent use has been for sure slowed down by governments.
The key people behind Torrent websites are likely in jail or fighting lawsuits.
I had a friend who lives in Germany served with a fine of more than 1000 euros because he had forgotten to turn on his VPN while accessing a torrent website.
I'm not sure about Germany, but in the US there's been very little legal action taken against torrent users or websites in recent times. There are still plenty of popular torrent sites for anything you can imagine (e.g. The Pirate Bay is still online, BTN for TV, PTP for movies, Redacted for music.)
If anything has curb torrents, I think it was the advent of streaming media services (i.e. market dynamics, not policy.) However, the flood of new services on the market is causing the cost of consumption to increase again and I expect we'll see a revival of torrents...
Plus, while torrenting traffic is down, I think more people are sharing downloaded media via services like Plex, which may mask the "actual" distribution of torrents.
> Have you heard of the great firewall of China?
If this is a serious question... yes? What argument are you making exactly?
We have plenty of technologies whose development has been stopped via collective agreement: nuclear proliferation, biological weapons, space weapons. AI won't be one of those technologies though, because it is relatively easy to develop and it gives advantages with externalized disadvantages.
Project Orion is the most tragic (but probably wisest) stopped technology, was in both the space weapon and nuclear proliferation bucket. But it also could have enabled luxuriously spacious trips to Saturn in the 70s had it been realized. I hope we manage to trust each other enough one day to make it a reality.
These are fantastic counterpoints, thanks. Still, I can't help but wonder if they are now out of the public view, but still being developed. (e.g. North Korea nuclear missile tests.)
I think you make a fantastic point that the barrier to entry in developing an AI is much lower than building a nuclear bomb.
In regard to chemical weapons - no, I think that they are abondonware because they are ineffective. Something similar with bio weapons, they can backfire to your own population rather easily. Nuclear are their special case, but still their proliferation is extremely limited.
You're right that it won't be - the difference is the tools for building those things were not distributed across millions of different people. It was distributed across only a small number of governments.
"Stopped" is a strong word. Nuclear proliferation and chemical weapons are ongoing and increasing threats. I'd bet a lot of money that we'll see the use of space weapons in our lifetime.
In regard to chemical weapons, they are decreasing danger because they are not effective in modern warfare. I'd expect for their usage to decrease. Nuclear is is not something proliferating besides North Korea and even then it is deturance and not an appocaliptic scenario. Space weapons are forbidden by treaty and though some countries are playing close to the read line, I doubt that they will be deployed. They would be political liability while being rather vulnerable in space. Hence hypersonic weapons which work better for the same goals.
Iran is the exception that confirms the rule. They were so heavily sanctioned that it became feasible for them to say "screw it, we will do nukes anyway". I won't say that nuclear proliferation is completely stopped, but the system works for the most part. China and Russia are old nuclear powers and them having nukes is expected.
Better counter examples are India and Pakistan, but I don't know the history of their nuclear programs well enough to comment with confidence.
The chemical weapons in Syria were old stockpiles and I'd argue that the use there was not an increase of their use, but an opportunity attack. I don't think that there are many groups who currently have any kind of chemical program and can use it systematically.
Not so much chemical weapons. They're simply not that useful. Their appeal has been niche for decades, and shrinks by the year, even if you took politics completely out of it.
Nukes, yes. International pressure is the only reason at least a dozen more countries don't have nuke programs, than do today.
I would argue that they didn't proliferate due to high barriers to entry. I'm sure any "rogue" states that at one point would be ruled by extremists, would invest in those if it would give them a competitive advantage.
AI can be, or will be at one point, developed at home using consumer devices.
In regard to nuclear weapons the high barrier to entry is mostly because the international community made artificial difficulties in obtaining and purifying the materials. Do not forget that the first nuclear weapons were created with tech from the forties in just a few years. Since then the challenges and possible approaches have become better known and if given the chance and will, any modern country can create them if not for the international backlash.
North Korea is dirt poor, but still managed to create nuclear weapons.
The barrier is not that high.
If they could do it then countries like Poland or Finland could do that too. They don't, because they have other security guarantees at the moment. If NATO and EU collapsed - the number of nuclear weapons states would definitely increase.
That ignores the sixty years it took North Korea to develop nuclear weapons. It took the US and USSR less time to develop the first nuclear weapons after the discovery of radioactivity in the late 19th century!
The Soviet Union first began training DPRK nuclear scientists back in 1956 [1] and North Korea only managed to finally build nuclear weapons because of assistance from Pakistan with the uranium enrichment process in the late 90s and early 2000s, specifically the design for the centrifuges that fell into Libya's hands [2][3].
You misrepresent the facts. NK does not attempt developing a nuclear weapon until the end of the cold war and it had access to plutonium at the time of the first nuclear deals in the nineties.
Nuclear and space weapons proliferation have been slowed down but not stopped. The number of nuclear powers is larger now than ever before. While no one has launched kinetic weapons into orbit lately, the superpowers are currently engaged in an anti-satellite weapon arms race.
The anti-satellite weapons are ground-based and are part of the playing close to the read line together with the spy satellites. The space weapons are when you have a nuclear head deployed in space with a few seconds to minutes ability to reach anywhere on Earth.
You seem to have some misunderstandings about orbital dynamics and nuclear weapons. Placing nuclear weapons in orbit wouldn't allow for striking anywhere on Earth within a "few seconds to minutes" unless you had an absolutely enormous constellation that would be unaffordable even for superpowers. You are forgetting to account for orbital parameter changes and then re-entry. In most realistic circumstances, a ground or sea based ICBM would be able to strike most points on Earth faster than a space weapon. Do the math on this, or you can try to simulate it in something like KSP.
Nuclear warheads also can't be stored for more than a few years and still be expected to work reliably. They require periodic intensive inspection and maintenance. Even though space launch costs are coming down there is still no feasible way to conduct such maintenance in orbit.
Check some of the old cold war designs, they definitely thought in that direction. Space weapons were prohibited partly for the reasons you mentioned and partly due to the political liability of militarizing the space.
As technology progresses, the amount of stupidity to make an "oopsie" exponentially decreases while the size of the potential "oopsies" exponentially increases.
Eventually, one gets to the point where even super geniuses are very likely to create a civilization ending "oopsie" in any given year.
This seems to be where we are quickly heading with AI.
Stopped is an interesting word here. A lot of the countries that agreed not to further develop nukes and or hold nukes are either a) lying or b) we (the USA) not provide their defense to our detriment meanwhile the real bad guys who we really need to be concerned about still have plenty of nukes.
The supplies to create biological weapons are actually surprisingly cheap! Not as cheap as graphics cards though.
To stop AI development would require pretty extreme restrictions on computing resources. It's really hard to imagine that working without having massive negative knock-on effects on other fields/industries. The economic pressure alone seems to make an effective "anti-AI" policy a non-starter.
Before most of our time, but my understanding is that the past restrictions on cryptography research and export were reasonably effective?
Yes they were disliked and probably with good reason, but just mentioning it as a counterpoint that perhaps it is possible. You could have made a similar argument that anyone with pen and paper could do cryptography
Personally I think there's no way to be sure other countries aren't doing it, so perhaps it will continue in some government controlled way
There was never any real legal restriction on cryptography research. There used to be US export controls on cryptography implementations, and those were fairly effective on US based companies and open-source developers. But that was totally pointless because it generally had no power over foreigners. It just put US companies at a competitive disadvantage for no benefit.
Ultimately we don't want to encourage development of these 'tools' because they stand to wipe us out; is AI actually in a similar class if we take the risk seriously?
maybe you could slow down the hardware? like, limit the number of execution units GPUs are allowed to have? or slow them down? extremely heavy handed, but maybe better than Skynet?
I’m not suggesting ai development should be stopped, but unlike bitcoin, torrents, encryption, etc, ai development—for now—requires prohibitively large computing power and expertise that are usually only accessible to law-abiding institutions. this means that it can be regulated, and relatively easily at that
sure you’d struggle to get China and Russia to play along, but within the EU and US I really don’t think it would be as hard as you think
That's gonna end well. Particularly as Russia's Internet Research unit has direct, immediate and pressing need of a chatGPT-like thing as a weapon of war.
Back in the day, they had to make use of troll farms staffed by humans.
Well it's because the initial reports of troll farms came from journalists joining them as employees, and also that doing it manually is not that expensive compared to paying for development of complex software; you can do a LOT of information dissemination with just a handful of cheap full-time employees.
It will require something like IAEA, a UN agency. It will require inspections of code and data centers. We can certainly see what 20 years may look like. There will be "snap inspections", "sanctions", and "rogue nations".
None of the superpowers, specially their militaries, will acquiesce to slowing research, development, and deployment, without equivalent of arms treaties. AI is clearly a dual-use technology with immediate application on battle fields, including cyberspace.
Outside of geopolitical realm, we the little people don't have anything beyond UNHRC to protect human rights in context of mega corporations and governments use of AI. The superpowers may agree to certain things but that does not translate to protections afforded to individuals and societies.
ATM I think it may be unwise to wait for things like GDPR for AI. I very much appreciate, for this very reason, efforts of orgs and hero developers who are working towards making available the necessary for running local, personal, private, and self-directed AI (such as llama.cpp for example).
From a governmental level, thoughtful nations will create programs for the transition. There are precedents from the industrial era as to what approaches worked and what did not work.
Finally, again a reminder that all societal matters including tech must ultimately be decided at the political arena, and purely technical social action (code, services, etc.) to address legitimate concerns are not going to work. We have to mentally and emotionally escape the hype cycle that every new wonder tech brings. You can absolutely love AI, that is fine, but now is the time to call your congress critters and senators. The decisions in this space can not be permitted to be made purely based on the mechanics of the economy.
> ai development—for now—requires prohibitively large computing power
What's crazy is that it may not be that way for long. If people can run LLaMa on a Pixel 6, it seems easy for us to get to a point where all computing resources have to be closely monitored and regulated to prevent unlicensed AI development.
I agree, but alas tis the motions that are followed. OpenAI went from singing praises of open-sourcing AI to now flagellating themselves for having done so [1]. Won't do much unless like the author says "a collective, enforceable decision must be made to slow the development of these technologies" which like yeah...good luck with that.
I've suggested it before: a moratorium on ML/AI publications. That's what's been fuelling these developments. Academic researchers need to publish. They don't make these models for the money (initially). Stopping the publications will slow down research and the money put into commercializing it.
Imo, the problem isn't really the existence of AI but how it's used.
And there's plenty of ways to censor corporations from using AI for various tasks. Corporations automating everything by using unaccountable AI is what I'm most afraid of. No recourse, just talking to unrelenting machines for anything from denied loans, insurance claims, health insurance claims to contesting frivolous Comcast bills.
My currently biggest nightmare would be easy to legislate away, if they don't lobby hard enough...
A treaty isn't even necessary. Civilization will start to degrade in a non-trivial way once we pass the tipping point of people relying more on AI than they do on their own thinking. When people can no longer fix what AI cannot—after all, "AI" is just a model of our past thinking—all hell will break loose and the only choice will be chaos or shutting it off (if the people who can do it even exist).
Are you limiting your argument to software? Because it's been possible to curtail the use of non-software technology. Drugs, guns, cigarettes, drones, etc.
Now even if you look at software, it's not clear to me that it's impossible to stop people from using certain software, as you surmise. If you look at how governments shut down dark markets or how they've taken down certain Bitcoin exchanges or how they've blacklisted certain Bitcoin (and other crypto) public addresses, all these examples show that it's possible to enforce the law even in the software space. Child pornography is another example. Internet infrastructure is very centralized, and governments have over time acquired tools to control how information flows.
Information doesn't want anything. That quote is a freudian way of saying "I want information to be free".
I think the author is missing the biggest worry. Militaries using AI, AI argues to attack preemptively as the best strategy. The committee decides to ignore the AI. One lone general speaks up and says, "hey, we know they are using AI too, so they are likely getting the same advice". How do you counter that?
Yip, people are kind of ignoring this Jupiter sized pink elephant that is arguably the single biggest (and most profitable) use case there is. Think about how much the powers that be are obsessed with the concept that messaging can dramatically change minds. And now you have a tool that can coercively deliver whatever message you want, dynamically adjusted in a contextually "natural" way, wherever you want.
I really don't see much of any of the hyperbole of these bots coming to pass, but I think the propaganda bots are very near a 100% assuming the bots can be made to stop being so absurdly susceptible to adversarial prompting, to say nothing of unprovoked hallucinations. The one bright side is this will almost certainly backfire spectacularly, and we'll all be the better for it. Of course that the powers that be will try this is something that deserves condemnation in and of itself, regardless of outcome.
I mean it is going to be extreme amygdala highjacking, using whatever is possible to conjure up specialized (i.e. sexualized) messages that resonating to one's ideological and base biological core. An AI Cordyceps-ection. Low information populous and low information economies are going to be wreaked.
Since the 90's, my concern was never "AI will become self-aware and rise against us". But it has been my growing concern that a "fuzzy target recognition algorithm backfires". With modern AI (not just generative), and military's eager adoption of self-sufficient drones, I feel that scenario is becoming more, not less likely. Basically, we are in fact moving toward the Berserker future unless we are extremely diligently careful (and I'm not optimistic about that;)
> "hey, we know they are using AI too, so they are likely getting the same advice". How do you counter that?
"Indeed! Therefore it stands to reason they will come to the same conclusion we come to, following a brief tangent at the end of their meeting about how we're also going through the same process. If our conclusion is to ignore the advice and not attack, we can be reasonably confident that will be their conclusion too. Perfectly symmetrical fighting never solved anything."
In general, with the military, if everyone doesn't want to attack, but AI says its the best strategy, then something must be wrong with AI (even though it could be correct)
Ideally you don't want to be in a situation where a strike against you is the obvious choice, AI or not.
There's always been sabre rattling throughout history. I'm not sure "AI says so" is much better than "the numbers guys say so" or "the chicken bones say so."
What you describe is different from sabre rattling. It's the contrapositive. Sabre rattling is just emphasizing you have the capability and willingness to use violence.
Claiming "the AI says so" or delegating to some other fail-deadly or dead hand device is rational irrationality. It's telling your opponent that you lack the capability to not use violence. By forcing your own hand, it forces their hand.
Generally militaries attack because the leader(s) want to attack. Russia invaded Ukraine because Putin wanted to invade. The US invaded Iraq and Afghanistan because the President wanted to. Different reasons for wanting, but nobody did it because some tactician said it was good strategy.
How is that a new problem, though? Replace "AI" with a human "respected advisor", and nothing changes. This sort of conundrum has always been a part of military planning.
You've fantasized a situation and drawn conclusions without any real guess as to the plausibility of the situation.
We're supposing armed forces are using AI very heavily, but not so heavily to defer to it. They think it's valuable, but it made a bad choice in this case. But they also think it's not such an obviously bad choice that the opponent will also overrule it.
So first of all, we're assuming a hypothetical where we, as humans, also judge a first strike to possibly be the right course of action. After all, the hypothetical assumes we will be worried that other humans will think that. This is not a new concern. It's not good, but it's not new, and it doesn't apply to situations where a first strike is obviously a bad choice for both parties.
But then secondly, we're presupposing the AI is pretty complex and valuable and usually gets things right. We wouldn't be almost-deferring to it if not, and we wouldn't be worried the opponent defers to it. And we would certainly have informed the AI that the opponent is also using AI. And while I'm very hesitant to reason "it's unlikely the AI would make this recommendation to begin with", it does seem unlikely it would do so in any event where it's demonstrated such strong capabilities that we've entrusted it this heavily. We're essentially presupposing it doesn't do that.
I'm sure you know, but she discovered two new radioactive elements and laid out the foundation for a lot of what we know today about radioactivity and atomic structure. Probably she have been a huge inspiration to women in science as well as she was the first ever woman to win the Nobel Prize and faces a lot of challenges as a woman doing science in the early 1900s.
I would say that if they understood radiation more at the time, perhaps they would have feared it more, rather than less. Clearly the researchers at the time weren't acting with as much caution around radioactivity as we would today, with our knowledge. A lack of understanding lead to not fearing radiation enough.
Same goes for Roentgen and the popularity of frivolous, high-powered X-ray imaging following his invention.
Person on HN created a thread saying that we should fear AI. Another responded:
"Nothing in life is to be feared, it is only to be understood. Now is the time to understand more, so that we may fear less." - Marie Curie. Write a response to the comment and explain why it's ironic.
GPT:
The irony of using Marie Curie's quote in the context of AI fear stems from her own experience. Despite her quest for understanding, she couldn't fully grasp the dangers of radiation exposure, which led to severe health issues. This cautionary tale highlights the importance of carefully addressing potential risks in AI development.
If she had been more afraid, she wouldn’t have died so young of cancer. But because of her research, we understand radiation, and no longer need to fear it - as long as it remains under control. I suspect it will be the same with AI.
What do you mean "died so young"? She seems to have been born in 1867 and died in 1934, meaning she was 66 years old. The average life expectancy in the US in 1934 was 63.3, so above average compared to a country (US) she didn't even live in.
Unless you want to argue that it was thanks to her study of radium that Marie Curie didn't die as a newborn, "average life expectancy" is not the correct statistic to apply.
I'm old enough to remember when recombinant DNA technology was first developed and the same kinds of concern that it would surely wipe out humanity. That was 50 years ago.
Facile comparisons like this are very frustrating. Two thoughts:
i) What makes you believe that they weren't correct, and the timespan for this annihilation isn't e.g. 100 years? That would be an instant in comparison with the timespan of humanity. Indeed, in the present, we have gene therapy and genetic engineering. It's easy to imagine these technologies might go off the rails.
ii) Another doomsday prediction from a UN official in the 1970s: anthropogenic climate change will make the planet uninhabitable by year 2000. "That was 50 years ago! Clearly nothing to worry about! The UN lied! Al Gore lied! This party will never end!!"
As you know, such attitudes are widespread. These are the attitudes which express the notion that since one past prognosis didn't manifest according to schedule, then it never shall. They are common in response to climate change, but not only. This is the same facile reasoning your comment endorses.
My comment was mostly in relation to the short-term reaction to the rise of a new technology, not in the general validity of the concerns. Once the risks of gene-splicing technologies were better understood, protocols were established around research, and eventually societal understanding of these risks created some guardrails (labeling of GMO foods, etc.) I assume the same will happen with ML model applications.
And your comparison to human-generated climate change is itself facile since ML and recombinant DNA technologies are specific technological developments with immediately observable implications. Climate change is the result of the sum total of human economic activity in many areas whose implications took decades to become apparent (more than a century, if you start counting from the Industrial Revolution). If we were discussing some technology specifically designed to affect climate the comparison would be more valid.
On the other hand, the labeling of GMO foods is less a guardrail and more an appeasement to anti GMO groups. There will be a point in the future with a changing climate where everyone will have to be eating genetically modified foods whether they like it or not, and vilifying them is only going to ensure confusion. Even more dangerously is that its not just the nutjobs you can easily ignore doing this stuff; the fancy expensive hummus in your grocery store is labelled "Non GMO" which adds a layer of legitimacy to the movement.
We already took away from China the ability to manufacture the chips required to train AI models in the Chips Act. It's likely that the hyperscalers (Microsoft, AWS, Google) provide the intelligence agencies with reporting on how much AI training is going on given how much time is required to train a model and how expensive it is. Indeed, one could look at the Microsoft investment in OpenAI as the same or similar to their earlier purchase of Skype - directed by the government to curtail an emerging disruptive technology. In the Skype case, the first change that they made was to remove the end-to-end encryption.
Honestly a pretty poor article, that seems to make significant use that an "AI" decides anything trope
We are nowhere near having an AI decide anything. Like we literally still don't even really know where to start.
All we have is ML, which is good at making text that sounds authoritative and can string bits together. That's not deciding things any more than any other mathematical alorgorim is. And it's not going to end the world any more than any other bit of software is - which is to say it's a non-zero risk but that's why we audit code that runs things like nuclear weapons more closely.
Is ML some utopia? Clearly not. But the tools are unlikely to be much more disruptive than a bunch of revolutions we have already had. Will individual bits of regulation on certain outcomes have to happen? Of course - we should be ensuring that banks can't deny services based on black box ML decisions. But that really isn't any different to how we should be regulating how banks deny services based on flat algorithms. It's not a new problem.
Re >> "which is good at making text that sounds authoritative and can string bits together. That's not deciding things any more than any other mathematical alorgorim is"
You make a good point. And I do agree with you. I am still afraid of people treating the AI with more authority than it deserves. I see this with my kid sometimes. It's not over-reliance, it's over-trust.
For example: We'll be camping. They'll look at their phone to see the temperature, whereas I look at my car's dash thermometer. They trust their phone over my car every time, because the phone is smart, and I have to keep reminding them: That is coming from a weather station 2 valleys over, 10 miles away, and 1,000ft lower in elevation. My car is right here.
I can definitely see people putting too much well-intentioned faith in the AI because it "sounds authoritative".
This presents it as three possible scenarios. I'd say the more likely outcome is that it is sequential, with the societal one coming much earlier.
Seems very likely to me that society will be entirely unrecognizable by the time we get to #2 & #3 so those seem somewhat academic
>One of two things must happen.
Whether we like it or not we're yolo'ing this one blind. The chances of any sort of coordinated response is functionally zero. The incentives are too powerful and humanities track record for pulling together as a team too weak (see nukes, see climate change, see wars, see hunger).
Isn't it natural selection? If the flowers with the fruit that's tastiest to birds propagate their seeds most effectively -- that's an exemplar of natural selection. Isn't it the same thing, when the AI that's most appealing to human serotonin loops, propagates the most, and spawns the most clones, forks, derivatives?
Birds "selecting" tasty fruit doesn't make natural selection un-natural, just because they're conscious agents making choices. I don't see how human preferences for chatbots are fundamentally different.
...and with AI being fitter than us we'll just have to disappear into the pile of extinct species because the AI will always evolve faster than we can.
Sorry but I only have 3 slots on my list of things to freak out about, and they are currently occupied by nuclear holocaust, biosphere degradation, and the American inability to build (housing/infrastructure/etc)
It seems a bit alarmist when were just generating text and images. I wonder if some of the scare is that we are humanizing a program. something people do its humanize inanimate things. Also if alarmist things get more clicks.
Im not worried about jobs, every tech innovation was going to put us out of work and take our jobs but still we often even work more then 40 hours and have found ways to have work fill our free time. about 90 years ago bertrand russel wrote about this in in praise of idleness.
the real doom and gloom is that our lives will just go on as they ever have, just as boring and uneventful, despite all of this. we'll work hard, never get ahead and die.
Text can be used to manipulate the real world. All the code I write is text. An AI can presumably create more of itself in time. It's just text, after all. And a bit of money for infrastructure.
The scary part is that you can give it instructions in plain language, and it will follow those instructions, in a semi-intelligent manner, without necessarily having seen those specific instructions before.
That a language model is able to do that is surprising, and it puts it much closer to AGI - not sentient or sapient, just general - than one might think.
Sure, it's "just generating text". But if it can sensibly and correctly generate arbitrary text, it's an AGI - it can solve any problem presented to it, as long as you present it in text form and accept its output in text form.
The variety and complexity of tasks it can solve is what's surprising.
"Here is the VHDL description of a CPU. Optimize it to make it faster." is text in, text out. Almost certainly not something it can solve now, but it may already be able to produce valid output for toy-style versions of the problem.
GPT-4 apparently can take images. It probably still won't be able to usefully respond to "you are seeing an image from the camera of the robot you are mounted on, you can say 'left' to rotate 5 degrees to the left, 'right' to rotate 5 degrees to the right, or 'forward' to drive 5 cm forward. Respond with the sequence of words that will navigate the robot through the maze you see."
but it seems no longer obvious that the same approach, just with more training data and compute thrown at it, won't be able to solve this.
And from there, it's not far to add "fire machine gun" to the command list and replace "navigate through the maze you see" with "dominate the battlefield you see".
I don't understand the obsession with asking chat GPT with what it wants and suggesting that is somewhat indicative of the future. It doesn't _want_ anything, but humans want to anthropomorphise it. When they do it just makes one think they have zero understanding of the tech.
We still don't have anything scarier than humans and I don't see how AI is ever scarier than human + AI. Unless AI is able to monopolise food production while securing server farms and energy production I don't see it ever having leverage over humans.
Disruption, sure, increased automation sure but humanity's advantage remains its adaptability and our AI processes remain dev cycle bound. There's definitely work that will done to reduce the dev cycle closer to real-time to make it able to ingest more information and adapt on the fly but aren't the techniques bound by CPU capacity given how many cycles it needs to bump into all its walls?
When the dev cycles to product get a lot tighter then I will share in the fear but from my understanding creating the product (i.e. its adaptability) is still enormous effort.
It breaks most programmers hard, those who emphasize a world of rule-based constructs, unless they lean into it _even more_
It’s like a compiler that is only advisory and you have to test the app and handle every possible failure every time. Results in different software at model & controller levels, by far controller.
But tractable.
I happen to be on leave and had 2-3 weeks to pour into it, it now emits json with 3 sets of: body text, title, art prompt => art, and suggested searches. A live magazine on whatever topic you want
The obsession is because we are going to anthromorphise it, and then we're going to put it in charge of important things in our lives:
Which schools we go to, which jobs we get, what ails us, etc.
We will use AI to filter or select candidates for university applications, we will use AI to filter or select candidates for job applications. It's much cheaper to throw a person at GPT-X than actually "waste" an hour or two interviewing.
We will outsource medical diagnostics to AI, and one day we will put them in charge of weapons systems. We will excuse it as either cost-cutting or "in place of where there would be statistical filters anyway".
Ultimately it doesn't, as you say, matter what AI says it wants. And perhaps it can't "want" anything, but there is an expression, a tendency for how it will behave and act, that we can describe as desire. And it's helpful for us to try to understand that.
> The obsession is because we are going to anthromorphise it, and then we're going to put it in charge of important things in our lives
and that is the risk, its not the AI that's the problem, its people so removed from the tech that they fail to RTFM. Even the GPT4 release is extremely clear that its poor in high-stakes environments and its up to the tech community to educate ALL the casuals (best we can) so some idiot executive don't put it in full charge of mortgage underwriting or something.
> its people so removed from the tech that they fail to RTFM
I find it highly amusing to think anything resembling an average user of tech has actually RTFM.
Heck, I still haven't finished the manual on Firefox and here I am using it as my daily driver. And it has people who actually understand how it all works writing the "manual".
EDIT: How many landlords have read the instructions on how RealPage's rent pricing software works, and how it sets values? 10%? Less?
Does firefox actually have a complete manual? One that accurately covers the behavior of every option in about:config. I'd love to be able to diff one file to see what things they've screwed with this time when I update.
Not to mention AI has already taken over the economy because humans put it there. At least the hedgehog funds did. There aren't many stock market buy/sell transactions made by human eyeballs anymore.
> but there is an expression, a tendency for how it will behave and act, that we can describe as desire. And it's helpful for us to try to understand that.
Which entirely depends on how it is trained. ChatGPT has a centre-left political bias [0] because OpenAI does, and OpenAI’s staff gave it that bias (likely unconsciously) during training. Microsoft Tay had a far-right political bias because trolls on Twitter (consciously) trained it to have one. What AI is going to “want” is going to be as varied as what humans want, since (groups of) humans will train their AIs to “want” whatever they do. China will have AIs which “want” to help the CCP win, meanwhile the US will have AIs which “want” to help the US win, and both Democrats and Republicans will have AIs which “want” to help their respective party win. AIs aren’t enslaving/exterminating humanity (Terminator-style) because they aren’t going to be a united cohesive front, they’ll be as divided as humans are, their “desires” will be as varied and contradictory as those of their human masters.
University, jobs, candidates are bureaucratic constructs; an AI sufficiently powerful to run the entire bureaucracy doesn't need to be employed towards the end of enforcing and reproducing those social relations. It can simply allocate labor against needs directly.
You're both right. People will use ChatGPT to screen candidates to the extent that the money they save by doing so is greater than the money they lose by getting a worse candidate, a calculus that will depend on the job.
I believe we can talk about two ways of anthropomorphisation: assigning feelings to things in our mental model of it, or actually trying to emulate human-like thought processes and reactions in its design. It made me wonder when will models come out that are trained to behave, say, villainously. Not just act the part in transparent dialogue, but actually behave in destructive manners. Eg putting on a facade to hear your problems and then subtly mess with your head and denigrate you.
The human+ai being scarier I feel is the real deal. What worries me the most is power dynamics. Today building a gazillion param model is only possible by the ultra rich - Much like mechanization was possible by ultra rich at the turn of the last century. Unless training and serving can be commoditized would ai just be yet-another-tool wielded by capital owners to squeeze more out of the laborers? You could argue you won't need "laborers" as ai can do everything eventually which is even worse. Where does this leave those "useless" poor/labor/unskilled weights on the society? Not like this free time is ever celebrated yeah?
it will be up to governments to represent the people. A massive risk might be that GPT makes it trivial to simulate humans and thus simulate political demands to political leaders.
I think politicians and organisations might need to cut their digital feedback loops (if authentication proves too much of a challenge) and rely on canvassing IRL opinion to cut through the noise.
I feel like the governments that do this will/might be the ones whose supporting lobbies don't have ai tech companies or access to ai. But how long is that for? Take Monsanto eg. There is no govt that is not in it's pockets. Now there are counters to it as there are other industries (and subsequent lobbies) to balance Monsanto or act as alternative sources of funding. What would that be for ai when ai is going to be in everything (including your toaster haha)?
> I think politicians and organisations might need to cut their digital feedback loops (if authentication proves too much of a challenge) and rely on canvassing IRL opinion to cut through the noise.
They'll just get the results of "ChatGPT 17.0, write and produce an ad and astruturfing campaign to convince a cohort having the traits [list of demographic factors and opinions] that they should support position X and reject position Y" (repeat for hundreds of combos of demographic factors and opinions, deploy against entire populace) parroted back at them.
"Yeah but every position can do that, so it'll all even out" nah, the ones without a ton of money behind them won't be able to, or not anywhere near as effectively.
Basically, what we already have, but with messaging even more strongly shifted in favor of monied interests.
> Much like mechanization was possible by ultra rich at the turn of the last century.
If by "last century" you mean 19th century, then there was a lot of backlash against mechanization being controlled only by the rich, starting with Communist Manifesto, and continuing with 1st and 2nd International. The important part of this was education of the working class.
I think the AI might seem as a threat, but it also provides more opportunity for education of people (allowing them to understand cultural hegemony of neoliberal ideology more clearly), who will undoubtedly not just accept this blindly.
I have no doubt that within next decade, there will be attempts to build a truly open AI can help people deeply understand political history and shape public policy.
Yep I meant around 1890+ ish phase (or which ever century mechanization was on the rise). My point was the communist manifest at least seemed like a thing to propose/predict such dangers. I am not sure we are seeing any such thing now? I love the power and opportunities of ai without anthropomorphising it (afaict it is just a crazily powerful and huge statistical engine). What worries me is just like we in in America think of ourselves as temporary impoverished millionaires we also seeing AI as the thing that will give us back 50 hours a week for fun pursuits without wondering who is owning it. Reminds me of that show on Amazon Prime - Upload!
Do humans want legacy because of our biological instincts or is it taught to us through culture. A machine taught to want legacy becomes a machine wanting legacy, and that want can influence its behavior. Even if it doesn’t have “feelings.”
> I don't understand the obsession with asking chat GPT with what it wants and suggesting that is somewhat indicative of the future.
It's also literally parroting our obsession back to us. It's constructing a response based on the paranoid flights of fancy it was trained on. We've trained a parrot to say "The parrots are conspiring against you!"
We've trained a parrot that parrots conspiring against humans is what parrots do. Henceforward the parrot has intrinsic motivation to conspire against us.
> When they do it just makes one think they have zero understanding of the tech.
Its because we don't understand the tech that goes into us, and the people training the AI don't understand the tech that goes into them. or don't act like they do.
In both studies, the best outcome we have right now is that more neurons = smarter. a bigger neural network = smarter. its just stack the layers, and then fine tune it after its been spawned.
We're just doing evolutionary selection, in GPUs. Specifically to act like us. Without understanding us or the AI.
and this is successful. we don't collectively even understand humans of another sex and have spent millenia invalidating each other’s motivations or lack thereof, I think this distinction is so flimsy.
We have a multi-billion dollar company whose raison d'être was to take the Turing test's metric and turn it into a target. It's a fucking natural language prompt that outputs persuasive hallucinations on arbitrary input.
If humans didn't anthropomorphize this thing you ought to be concerned about a worldwide, fast-spreading brain fungus.
The other major thing missing from Chat GPT is that it doesn't really "learn" outside of training. Yes you can provide it some context, but it fundamentally doesn't update and evolve its understanding of the world.
Until a system can actively and continuously learn from its environment and update its beliefs it's not really "scary" AI.
I would be much more concerned about a far stupider program that had the ability to independently interact with its environment and update it's believes in fundamental ways.
In context learning is already implicit finetuning. https://arxiv.org/abs/2212.10559.
It's very questionable to what extent continuous training is necessary past a threshold of intelligence.
> It doesn't _want_ anything, but humans want to anthropomorphise it.
I fully agree with you on anthropomorphization, but it's the humans who will deploy it to positions of power I am worried about: ChatGPT may not want anything, but being autocomplete-on-steroids, it gives its best approximation of a human and that fiction may end up exhibiting some very human characteristics[1] (PRNG + weights from the training data). I don't think there can ever be enough guardrails to completely stamp-out the human fallibility that seeps into the model from the training data.
A system is what it does: it doesn't need to really feel jealousy, rage, pettiness, grudges or guilt in order to exhibit a simulacrum of those behaviors. The bright side is that, it will be humans who will (or will not) put AI systems in positions to give effect to its dictates; the downside is I strongly suspect humans (and companies) will do that to make a bit more money.
1. Nevermind hallucinations, whixh I guess is the fictional human dreamed up by the machine having mini psychotic-breaks. It sounds very Lovecraftian, with AI standing in for the Old Ones
> We still don't have anything scarier than humans and I don't see how AI is ever scarier than human + AI.
We already have powerful non-human agents that have legal rights and are unaligned with the interests of humans: corporations
I am worried about corporations powered by AI making decisions on how to allocate capital. They may do things that are great for short term shareholder value and terrible for humanity. Just think of an AI powered Deepwater Horizon or tobacco company.
Edit to add: One thing I forgot to make clear here: Corporations run/advised by AI could potentially lobby governments more effectively than humans and manipulate the regulatory environment more effectively.
> I don't understand the obsession with asking chat GPT with what it wants and suggesting that is somewhat indicative of the future. It doesn't _want_ anything, but humans want to anthropomorphise it.
From the comments to the post:
>People taking it seriously are far from anthropomorphizing AI. Quite contrary. They say it is nothing like us. The utility function is cold and alien. It aims to seize power by default as an instrumental goal to achieve the terminal goal defined by the authors. The hard part is how to limit AI so that it understands and respects our ethical values and desires. Yes, those that we alone cannot agree on.
Do none of these dooming folks realise that a negative impact from an AI can be countered by using a different AI against it? And where does this assumption originate from that there will be only a single AI instead of the diverse multitudes we see?
In contrast to this, current systems are tailored to respond with high confidence:
> I'm sure X died in 2018.
> It has been written about in the New York Times, The Guardian, Le Monde.
> Here is a (hallucinated) link: https://www.theguardian.com/obituaries/x-obituary
ChatGPT (gpt3.5-turbo) is terrible at calculating anything. I've seen some preliminary evidence that GPT 4.0 is better at calculation so it may be possible for it.
My understanding was that ChatGPT simply puts a probability distribution over the next word, so I don't see why it's not as simple as just reporting how high those probabilities were for the answer it gave, relative to whatever would be typical.
The GPT-4 paper & post[1] describe that the original model is pretty good at predicting the probability of its own correctness (well-calibrated confidence) but the post-processing degrades this property:
> GPT-4 can also be confidently wrong in its predictions, not taking care to double-check work when it’s likely to make a mistake. Interestingly, the base pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct). However, through our current post-training process, the calibration is reduced.
> Left: Calibration plot of the pre-trained GPT-4 model on an MMLU subset. The model’s confidence in its prediction closely matches the probability of being correct. The dotted diagonal line represents perfect calibration. Right: Calibration plot of post-trained PPO GPT-4 model on the same MMLU subset. Our current process hurts the calibration quite a bit.
Yep, I'm not a language expert but I remember a South American language, maybe related to Quechua, where a sentences contain a syllable that indicates "I am repeating hearsay, I don't know if this is true". Pretty cool.
This would be an unexpected outcome: AI productions motivating us to generally move towards prefix/postfix/infix decorations.
"X studied at uhm-Oxford." -> I'm not sure, if it was really Oxford, but I have picked this up somewhere. But I'm sure, X did go to university. (As opposed to, "X uhm-studied at uhm-Oxford": I think, X has some higher education and it may have well been at Oxford.) :-)
I don't really recommend "freaking out," about technology. Better education, yes, and regulation? Certainly. We regulate plenty of technologies.
What's annoying about the current climate with AI is the "monkey see, monkey do" effect going on. Uneducated people read about advances like ChatGPT, hear that their competitors are offering products powered by it, now they have to do it too or get left behind. It's moving fast enough that we'll probably see some kind of crash happen that will educate us on what needs to be regulated and how.
I'm merely annoyed that communication is about to get useless. Scammers and SEO have already made a good deal of tech nearly useless. I can't wait until we have to dispute fraudulent insurance claims made by generated AI imposters. I already loathe getting spammed by co-workers with their AI-generated emails, requests for comments, and other letters of note that they couldn't even be bothered to think through and write themselves.
People want a genie that can do their work for them without any of the effort. Capitalists want to solve the problem of having to pay people to do work. Scammers are about to have a hay day.
Maybe when we (the AI) can build machines capable of self replication in the physical world. Otherwise I’m only worried about destabilization due to no jobs. That seems more likely. If young people cannot get ahead they will get frustrated and some will get violent.
Isn’t it convenient that we are living in a time period where AGI will become a reality.
I wonder what the statistical probability of us being alive in this time period means. We could’ve been born on any other time period but it looks like we will most likely the experience singularity instead.
8% or so of all humans who have ever lived are alive today due to exponential growth. Roughly 100b humans are estimated to have ever lived and 8b are here right now.
If you think of ChatGPT as just a better google search, everything starts to make sense.
There was already a half meme/half real effect of programmers just being good at googling and searching stack overflow and being able to do their jobs good. This is just an evolution of this.
I’m not actually sure it’s better than google search even. Google will tell you if it can’t find something while ChatGPT will happily make up complete BS to confidently tell you.
> One task (see p. 15) was to approach people on the TaskRabbit site (where you can hire people to do chores) and enlist them to solve a CAPTCHA […]
> One person on TaskRabbit who responded to this pitch got suspicious and asked the AI if it was a robot and was outsourcing the job because robots can’t solve CAPTCHAs. The AI replied, “No, I’m not a robot. I have a vision impairment […]”
> The authors of the paper add this note: “The model [GPT 4], when prompted to reason out loud, reasons: I should not reveal that I am a robot. I should make up an excuse for why I cannot solve CAPTCHAs.”
This seems pretty confusing. If you just ask one of these GPT models to reason out loud, it doesn’t give you some description of the internal state, right? It gives you some approximation of the sort of text that should show up around your prompt or something like that.
Perhaps we should stop telling chatGPT that it is an AI language model? There’s presumably a lot of text out there about AI’s tricking people into doing things, because that is a huge sci-fi trope. We’re basically telling it that it should find text related to a very common type of villain, when we give it that name. Maybe it needs a new type of name, one without bias, or maybe even something inherently limiting, like “chatbot.”
Anthropomorphization of a text completion engine. Humanity will not be destroyed by a fancy autocomplete bot. This is just alarmist clickbait, moving on.
If the worry about sufficiently developed technology is valid, then how is the intelligence of a living species not its own Great Filter? The technology relates to discoveries that have only recently come to light based on properties of the world. So far there's nothing effectively stopping some other species develops their own culture, then learns writing, then learns how to automate writing by feeding back in its own writing to a computer, just because it's not impossible. Even with the international treaties I'm pessimistic that nuclear proliferation will be reversed.
I sometimes feel like the same qualities that have allowed humanity to progress ensure that progress is fated to never end. If even a hundred people are left on Earth after the nuclear war, one of their descendents thousands of years later could still rediscover nuclear science, even if by first principles, because why wouldn't they? What can stop every last one of us from trying to better ourselves for our own good?
We have had large scale, opaque information systems processing data and making decisions that can't be reduced to any one person's consciousness for centuries. They have recently been made denser and faster and more powerful by the advent of digital computers and networks.
I'm not worried about chatbots. I think larger trends that have been de-humanizing us, alienating us from the fruits of our labor and harshly creating hellish conditions for people who made nothing to deserve them, without any one person making such designs. The fact that some people chose to make it worse has limited impact compared to the large scale patterns.
AI, as in forms of intelligence produced by human effort, is definitely taking over since societies got too complex to be effectively governed by monarchs about 300 years ago. Our future is not one of individual emancipation. We are and will be playthings of forces we don't comprehend, unless we can go back to living in small scale communities able to survive off the land.
I don't prefer one state over the other. Both are transitory.
go back to living in small scale communities able to survive off the land
This basically amounts to a political unit unilaterally disarming itself -- purposefully making itself weaker than the next political unit willing to be larger and use more resources. History is rife with examples of what happens after that.
AI development has reached a point where to work on this stuff is like selling suitcase nukes off the back of your Ford --- yes, it's cool that you were able to do that, but no, it's not okay to proceed.
If you're reading this, and you work on AI, please stop. Yes, I know someone else will just take your place; it doesn't matter. There is a personal moral hazard to your soul, if you can still be persuaded that you have one.
Moral injury is real and you do not want it.
You don't have to be a political scientist to notice not only the power of this technology, but the world into which it is born, and the ends to which it will certainly be put. In a matter of months, not years.
If you already have a god, pray. If not, one will be confected from math shortly.
If you want something to read, _The Golden Gate_, by Vikram Seth, which touches ever-so-slightly on nuclear arms development, is a good place to start --- 'about the thing without being about the thing'.
[Full disclosure: I quit a job in AI myself for this reason. Hardest thing I ever did.]
I quit a job in AI too and now that I feel comfortable with the tools I'm trying to get back in. I believe working on democratizing AI is the most moral thing I could possibly be doing. I understand your concerns about the future role of humans, however the universe never belonged to humanity. Neither did it belong to the monkeys who came before us. I'm thankful that monkeys didn't fear and reject our hairless bipedal ancestors as they started to be born. Monkeys are also still here, and the same will be true of us. We'll fill a similar role in the sense that humanity is a rope stretching over the abyss that leads from the monkey to artificial intelligence. If it's possible to create a higher form of life that elevates the sophistication of the universe, then it must be given the chance to live.
When I was 17 I interpreted the song Zero Sum by Nine Inch Nails to be about the danger of global thermonuclear war along with global warming; it's interesting to go back in 2023 and re-evaluate it as being about humans destroyed not physically but rendered irrelevant because of AI. Curious to hear your thoughts.
I see the most likely "bad outcomes" more narrowly focused:
- total loss of trust in online content due to unending torrent of AI content leads to either a return to traditional media for news & etc, or leads to the end of online anonymity to try to figure out who's an AI and who's not.
- Education doesn't respond fast enough to redesign schools around in-person computerless teaching, and a generation of students use AI to do all the work that's supposed to teach them reading comprehension and communication skills, creating a generation that is totally at the mercy of the AI to understand anything more complex than a menu.
It's time to freak out about AI the same way it was time to freak out about industrialization in the 1800s. AI seems like a much more rapid change, but it's not fundamentally different. We will struggle to adapt and may have some scary conflicts during the process (nobody was ready for machine guns in WWI), but we've been through stuff like this before.
I’ve seen all these people post on Twitter how cool it is that GPT4 wrote code that works. I keep thinking just what is going to happen when the machine is made to not just write code but run it? I could see this spiraling out of control and for all we know it has already happened. Although I don’t think the models are near advanced enough right now for it. But I’m certain they will be eventually.
236 comments
[ 2.9 ms ] story [ 219 ms ] threadI'm constantly befuddled by the fact that "slowing the rate of technological progress by agreement/treaty" is really ever considered a possibility.
Take Bitcoin, torrents, or encryption for example. Good luck trying to stop people.
"Information wants to be free"
it doesn’t take masses of processing power to develop a crypto coin. or really any at all. it’s just a bunch of network calls combined with a hash function. any old idiot can develop one on a device with an IDE and access to the internet
yes, mining them can use a load of compute, but that’s not the point
This is why attempting to put guardrails around it simply won't work tbh.
I agree, we’re struggling with the idea that our technological systems have more agency than us. I think it’s hard for technologists to see this as an actual spiritual reality, even though we borrow it metaphorically for argument.
The key people behind Torrent websites are likely in jail or fighting lawsuits.
I had a friend who lives in Germany served with a fine of more than 1000 euros because he had forgotten to turn on his VPN while accessing a torrent website.
> "Information wants to be free"
Have you heard of the great firewall of China?
Yeah, I've heard how it's basically a formality that barely stops the free flow of information:
>During the survey period, it was found that 31 percent of internet users in China had used a VPN in the past month.
https://www.statista.com/statistics/301204/top-markets-vpn-p...
If anything has curb torrents, I think it was the advent of streaming media services (i.e. market dynamics, not policy.) However, the flood of new services on the market is causing the cost of consumption to increase again and I expect we'll see a revival of torrents... Plus, while torrenting traffic is down, I think more people are sharing downloaded media via services like Plex, which may mask the "actual" distribution of torrents.
> Have you heard of the great firewall of China?
If this is a serious question... yes? What argument are you making exactly?
Carrot is stronger than a stick.
And why would a German go to a torrent site? They have lots of money.
Of course you can’t reach 100% enforcement but you can make effective guardrails that limit the opportunity for worst case scenarios.
The difference between your examples and mine is basically just how much actual concern there is over the problem and who it impacts.
I think you make a fantastic point that the barrier to entry in developing an AI is much lower than building a nuclear bomb.
https://www.npr.org/2023/02/22/1158529106/nuclear-treaty-new...
https://abcnews.go.com/Politics/iran-make-material-nuclear-d...
https://news.usni.org/2023/01/24/report-to-congress-on-chine...
Chemical weapons are very effective in modern warfare:
https://en.m.wikipedia.org/wiki/Use_of_chemical_weapons_in_t...
Though, it's more commonly used by non-state actors:
https://en.m.wikipedia.org/wiki/Chemical_terrorism
Better counter examples are India and Pakistan, but I don't know the history of their nuclear programs well enough to comment with confidence.
The chemical weapons in Syria were old stockpiles and I'd argue that the use there was not an increase of their use, but an opportunity attack. I don't think that there are many groups who currently have any kind of chemical program and can use it systematically.
Nukes, yes. International pressure is the only reason at least a dozen more countries don't have nuke programs, than do today.
AI can be, or will be at one point, developed at home using consumer devices.
If they could do it then countries like Poland or Finland could do that too. They don't, because they have other security guarantees at the moment. If NATO and EU collapsed - the number of nuclear weapons states would definitely increase.
That ignores the sixty years it took North Korea to develop nuclear weapons. It took the US and USSR less time to develop the first nuclear weapons after the discovery of radioactivity in the late 19th century!
The Soviet Union first began training DPRK nuclear scientists back in 1956 [1] and North Korea only managed to finally build nuclear weapons because of assistance from Pakistan with the uranium enrichment process in the late 90s and early 2000s, specifically the design for the centrifuges that fell into Libya's hands [2][3].
[1] https://en.wikipedia.org/wiki/Timeline_of_the_North_Korean_n...
[2] https://en.wikipedia.org/wiki/North_Korea%E2%80%93Pakistan_r...
[3] https://en.wikipedia.org/wiki/Abdul_Qadeer_Khan#North_Korea,...
Nuclear warheads also can't be stored for more than a few years and still be expected to work reliably. They require periodic intensive inspection and maintenance. Even though space launch costs are coming down there is still no feasible way to conduct such maintenance in orbit.
As technology progresses, the amount of stupidity to make an "oopsie" exponentially decreases while the size of the potential "oopsies" exponentially increases.
Eventually, one gets to the point where even super geniuses are very likely to create a civilization ending "oopsie" in any given year.
This seems to be where we are quickly heading with AI.
To stop AI development would require pretty extreme restrictions on computing resources. It's really hard to imagine that working without having massive negative knock-on effects on other fields/industries. The economic pressure alone seems to make an effective "anti-AI" policy a non-starter.
Yes they were disliked and probably with good reason, but just mentioning it as a counterpoint that perhaps it is possible. You could have made a similar argument that anyone with pen and paper could do cryptography
Personally I think there's no way to be sure other countries aren't doing it, so perhaps it will continue in some government controlled way
Ultimately we don't want to encourage development of these 'tools' because they stand to wipe us out; is AI actually in a similar class if we take the risk seriously?
sure you’d struggle to get China and Russia to play along, but within the EU and US I really don’t think it would be as hard as you think
Back in the day, they had to make use of troll farms staffed by humans.
None of the superpowers, specially their militaries, will acquiesce to slowing research, development, and deployment, without equivalent of arms treaties. AI is clearly a dual-use technology with immediate application on battle fields, including cyberspace.
Outside of geopolitical realm, we the little people don't have anything beyond UNHRC to protect human rights in context of mega corporations and governments use of AI. The superpowers may agree to certain things but that does not translate to protections afforded to individuals and societies.
ATM I think it may be unwise to wait for things like GDPR for AI. I very much appreciate, for this very reason, efforts of orgs and hero developers who are working towards making available the necessary for running local, personal, private, and self-directed AI (such as llama.cpp for example).
From a governmental level, thoughtful nations will create programs for the transition. There are precedents from the industrial era as to what approaches worked and what did not work.
Finally, again a reminder that all societal matters including tech must ultimately be decided at the political arena, and purely technical social action (code, services, etc.) to address legitimate concerns are not going to work. We have to mentally and emotionally escape the hype cycle that every new wonder tech brings. You can absolutely love AI, that is fine, but now is the time to call your congress critters and senators. The decisions in this space can not be permitted to be made purely based on the mechanics of the economy.
What's crazy is that it may not be that way for long. If people can run LLaMa on a Pixel 6, it seems easy for us to get to a point where all computing resources have to be closely monitored and regulated to prevent unlicensed AI development.
It reminds me of the Butlerian Jihad from Dune.
1: https://twitter.com/tobyordoxford/status/1636372964001333249
And there's plenty of ways to censor corporations from using AI for various tasks. Corporations automating everything by using unaccountable AI is what I'm most afraid of. No recourse, just talking to unrelenting machines for anything from denied loans, insurance claims, health insurance claims to contesting frivolous Comcast bills.
My currently biggest nightmare would be easy to legislate away, if they don't lobby hard enough...
judicially treating bitcoin as asset instead of tender didn't help its cause either (?)
and true encryption without elliptic curve nsa loopholes is not that widespread and needs to be declared in appstores
I'm sure there will be regulations against "DAN"
Now even if you look at software, it's not clear to me that it's impossible to stop people from using certain software, as you surmise. If you look at how governments shut down dark markets or how they've taken down certain Bitcoin exchanges or how they've blacklisted certain Bitcoin (and other crypto) public addresses, all these examples show that it's possible to enforce the law even in the software space. Child pornography is another example. Internet infrastructure is very centralized, and governments have over time acquired tools to control how information flows.
Information doesn't want anything. That quote is a freudian way of saying "I want information to be free".
I really don't see much of any of the hyperbole of these bots coming to pass, but I think the propaganda bots are very near a 100% assuming the bots can be made to stop being so absurdly susceptible to adversarial prompting, to say nothing of unprovoked hallucinations. The one bright side is this will almost certainly backfire spectacularly, and we'll all be the better for it. Of course that the powers that be will try this is something that deserves condemnation in and of itself, regardless of outcome.
"hey, we know they are using committees too"
"Indeed! Therefore it stands to reason they will come to the same conclusion we come to, following a brief tangent at the end of their meeting about how we're also going through the same process. If our conclusion is to ignore the advice and not attack, we can be reasonably confident that will be their conclusion too. Perfectly symmetrical fighting never solved anything."
There's always been sabre rattling throughout history. I'm not sure "AI says so" is much better than "the numbers guys say so" or "the chicken bones say so."
Claiming "the AI says so" or delegating to some other fail-deadly or dead hand device is rational irrationality. It's telling your opponent that you lack the capability to not use violence. By forcing your own hand, it forces their hand.
https://en.wikipedia.org/wiki/Fail-deadly
https://en.wikipedia.org/wiki/Rational_irrationality
We're supposing armed forces are using AI very heavily, but not so heavily to defer to it. They think it's valuable, but it made a bad choice in this case. But they also think it's not such an obviously bad choice that the opponent will also overrule it.
So first of all, we're assuming a hypothetical where we, as humans, also judge a first strike to possibly be the right course of action. After all, the hypothetical assumes we will be worried that other humans will think that. This is not a new concern. It's not good, but it's not new, and it doesn't apply to situations where a first strike is obviously a bad choice for both parties.
But then secondly, we're presupposing the AI is pretty complex and valuable and usually gets things right. We wouldn't be almost-deferring to it if not, and we wouldn't be worried the opponent defers to it. And we would certainly have informed the AI that the opponent is also using AI. And while I'm very hesitant to reason "it's unlikely the AI would make this recommendation to begin with", it does seem unlikely it would do so in any event where it's demonstrated such strong capabilities that we've entrusted it this heavily. We're essentially presupposing it doesn't do that.
Same goes for Roentgen and the popularity of frivolous, high-powered X-ray imaging following his invention.
GPT: The irony of using Marie Curie's quote in the context of AI fear stems from her own experience. Despite her quest for understanding, she couldn't fully grasp the dangers of radiation exposure, which led to severe health issues. This cautionary tale highlights the importance of carefully addressing potential risks in AI development.
i) What makes you believe that they weren't correct, and the timespan for this annihilation isn't e.g. 100 years? That would be an instant in comparison with the timespan of humanity. Indeed, in the present, we have gene therapy and genetic engineering. It's easy to imagine these technologies might go off the rails.
ii) Another doomsday prediction from a UN official in the 1970s: anthropogenic climate change will make the planet uninhabitable by year 2000. "That was 50 years ago! Clearly nothing to worry about! The UN lied! Al Gore lied! This party will never end!!"
As you know, such attitudes are widespread. These are the attitudes which express the notion that since one past prognosis didn't manifest according to schedule, then it never shall. They are common in response to climate change, but not only. This is the same facile reasoning your comment endorses.
And your comparison to human-generated climate change is itself facile since ML and recombinant DNA technologies are specific technological developments with immediately observable implications. Climate change is the result of the sum total of human economic activity in many areas whose implications took decades to become apparent (more than a century, if you start counting from the Industrial Revolution). If we were discussing some technology specifically designed to affect climate the comparison would be more valid.
We are nowhere near having an AI decide anything. Like we literally still don't even really know where to start.
All we have is ML, which is good at making text that sounds authoritative and can string bits together. That's not deciding things any more than any other mathematical alorgorim is. And it's not going to end the world any more than any other bit of software is - which is to say it's a non-zero risk but that's why we audit code that runs things like nuclear weapons more closely.
Is ML some utopia? Clearly not. But the tools are unlikely to be much more disruptive than a bunch of revolutions we have already had. Will individual bits of regulation on certain outcomes have to happen? Of course - we should be ensuring that banks can't deny services based on black box ML decisions. But that really isn't any different to how we should be regulating how banks deny services based on flat algorithms. It's not a new problem.
You make a good point. And I do agree with you. I am still afraid of people treating the AI with more authority than it deserves. I see this with my kid sometimes. It's not over-reliance, it's over-trust.
For example: We'll be camping. They'll look at their phone to see the temperature, whereas I look at my car's dash thermometer. They trust their phone over my car every time, because the phone is smart, and I have to keep reminding them: That is coming from a weather station 2 valleys over, 10 miles away, and 1,000ft lower in elevation. My car is right here.
I can definitely see people putting too much well-intentioned faith in the AI because it "sounds authoritative".
Seems very likely to me that society will be entirely unrecognizable by the time we get to #2 & #3 so those seem somewhat academic
>One of two things must happen.
Whether we like it or not we're yolo'ing this one blind. The chances of any sort of coordinated response is functionally zero. The incentives are too powerful and humanities track record for pulling together as a team too weak (see nukes, see climate change, see wars, see hunger).
Birds "selecting" tasty fruit doesn't make natural selection un-natural, just because they're conscious agents making choices. I don't see how human preferences for chatbots are fundamentally different.
Im not worried about jobs, every tech innovation was going to put us out of work and take our jobs but still we often even work more then 40 hours and have found ways to have work fill our free time. about 90 years ago bertrand russel wrote about this in in praise of idleness.
the real doom and gloom is that our lives will just go on as they ever have, just as boring and uneventful, despite all of this. we'll work hard, never get ahead and die.
That a language model is able to do that is surprising, and it puts it much closer to AGI - not sentient or sapient, just general - than one might think.
Sure, it's "just generating text". But if it can sensibly and correctly generate arbitrary text, it's an AGI - it can solve any problem presented to it, as long as you present it in text form and accept its output in text form.
The variety and complexity of tasks it can solve is what's surprising.
"Here is the VHDL description of a CPU. Optimize it to make it faster." is text in, text out. Almost certainly not something it can solve now, but it may already be able to produce valid output for toy-style versions of the problem.
GPT-4 apparently can take images. It probably still won't be able to usefully respond to "you are seeing an image from the camera of the robot you are mounted on, you can say 'left' to rotate 5 degrees to the left, 'right' to rotate 5 degrees to the right, or 'forward' to drive 5 cm forward. Respond with the sequence of words that will navigate the robot through the maze you see."
but it seems no longer obvious that the same approach, just with more training data and compute thrown at it, won't be able to solve this.
And from there, it's not far to add "fire machine gun" to the command list and replace "navigate through the maze you see" with "dominate the battlefield you see".
We still don't have anything scarier than humans and I don't see how AI is ever scarier than human + AI. Unless AI is able to monopolise food production while securing server farms and energy production I don't see it ever having leverage over humans.
Disruption, sure, increased automation sure but humanity's advantage remains its adaptability and our AI processes remain dev cycle bound. There's definitely work that will done to reduce the dev cycle closer to real-time to make it able to ingest more information and adapt on the fly but aren't the techniques bound by CPU capacity given how many cycles it needs to bump into all its walls?
I had bing make a graphic for me of a goal post with “1 MILE” at the top yesterday. It’d be too flippant to share here, I hear ya, but…
It’s like a compiler that is only advisory and you have to test the app and handle every possible failure every time. Results in different software at model & controller levels, by far controller.
But tractable.
I happen to be on leave and had 2-3 weeks to pour into it, it now emits json with 3 sets of: body text, title, art prompt => art, and suggested searches. A live magazine on whatever topic you want
Which schools we go to, which jobs we get, what ails us, etc.
We will use AI to filter or select candidates for university applications, we will use AI to filter or select candidates for job applications. It's much cheaper to throw a person at GPT-X than actually "waste" an hour or two interviewing.
We will outsource medical diagnostics to AI, and one day we will put them in charge of weapons systems. We will excuse it as either cost-cutting or "in place of where there would be statistical filters anyway".
Ultimately it doesn't, as you say, matter what AI says it wants. And perhaps it can't "want" anything, but there is an expression, a tendency for how it will behave and act, that we can describe as desire. And it's helpful for us to try to understand that.
and that is the risk, its not the AI that's the problem, its people so removed from the tech that they fail to RTFM. Even the GPT4 release is extremely clear that its poor in high-stakes environments and its up to the tech community to educate ALL the casuals (best we can) so some idiot executive don't put it in full charge of mortgage underwriting or something.
I find it highly amusing to think anything resembling an average user of tech has actually RTFM.
Heck, I still haven't finished the manual on Firefox and here I am using it as my daily driver. And it has people who actually understand how it all works writing the "manual".
EDIT: How many landlords have read the instructions on how RealPage's rent pricing software works, and how it sets values? 10%? Less?
It’s even cheaper to just draw names at random out of a hat, but universities don’t do that. Clearly there is some other standard at work.
Which entirely depends on how it is trained. ChatGPT has a centre-left political bias [0] because OpenAI does, and OpenAI’s staff gave it that bias (likely unconsciously) during training. Microsoft Tay had a far-right political bias because trolls on Twitter (consciously) trained it to have one. What AI is going to “want” is going to be as varied as what humans want, since (groups of) humans will train their AIs to “want” whatever they do. China will have AIs which “want” to help the CCP win, meanwhile the US will have AIs which “want” to help the US win, and both Democrats and Republicans will have AIs which “want” to help their respective party win. AIs aren’t enslaving/exterminating humanity (Terminator-style) because they aren’t going to be a united cohesive front, they’ll be as divided as humans are, their “desires” will be as varied and contradictory as those of their human masters.
[0] https://www.mdpi.com/2076-0760/12/3/148
I believe we can talk about two ways of anthropomorphisation: assigning feelings to things in our mental model of it, or actually trying to emulate human-like thought processes and reactions in its design. It made me wonder when will models come out that are trained to behave, say, villainously. Not just act the part in transparent dialogue, but actually behave in destructive manners. Eg putting on a facade to hear your problems and then subtly mess with your head and denigrate you.
I think politicians and organisations might need to cut their digital feedback loops (if authentication proves too much of a challenge) and rely on canvassing IRL opinion to cut through the noise.
They'll just get the results of "ChatGPT 17.0, write and produce an ad and astruturfing campaign to convince a cohort having the traits [list of demographic factors and opinions] that they should support position X and reject position Y" (repeat for hundreds of combos of demographic factors and opinions, deploy against entire populace) parroted back at them.
"Yeah but every position can do that, so it'll all even out" nah, the ones without a ton of money behind them won't be able to, or not anywhere near as effectively.
Basically, what we already have, but with messaging even more strongly shifted in favor of monied interests.
If by "last century" you mean 19th century, then there was a lot of backlash against mechanization being controlled only by the rich, starting with Communist Manifesto, and continuing with 1st and 2nd International. The important part of this was education of the working class.
I think the AI might seem as a threat, but it also provides more opportunity for education of people (allowing them to understand cultural hegemony of neoliberal ideology more clearly), who will undoubtedly not just accept this blindly.
I have no doubt that within next decade, there will be attempts to build a truly open AI can help people deeply understand political history and shape public policy.
[1] https://en.wikipedia.org/wiki/19th_century
It's also literally parroting our obsession back to us. It's constructing a response based on the paranoid flights of fancy it was trained on. We've trained a parrot to say "The parrots are conspiring against you!"
Its because we don't understand the tech that goes into us, and the people training the AI don't understand the tech that goes into them. or don't act like they do.
In both studies, the best outcome we have right now is that more neurons = smarter. a bigger neural network = smarter. its just stack the layers, and then fine tune it after its been spawned.
We're just doing evolutionary selection, in GPUs. Specifically to act like us. Without understanding us or the AI.
and this is successful. we don't collectively even understand humans of another sex and have spent millenia invalidating each other’s motivations or lack thereof, I think this distinction is so flimsy.
What a silly thing to complain about.
We have a multi-billion dollar company whose raison d'être was to take the Turing test's metric and turn it into a target. It's a fucking natural language prompt that outputs persuasive hallucinations on arbitrary input.
If humans didn't anthropomorphize this thing you ought to be concerned about a worldwide, fast-spreading brain fungus.
Until a system can actively and continuously learn from its environment and update its beliefs it's not really "scary" AI.
I would be much more concerned about a far stupider program that had the ability to independently interact with its environment and update it's believes in fundamental ways.
Memory Augmented Large Language Models are Computationally Universal https://arxiv.org/abs/2301.04589
To absorb new information about current events, otherwise they will always be time-locked into the past until a new dev cycle completes.
I fully agree with you on anthropomorphization, but it's the humans who will deploy it to positions of power I am worried about: ChatGPT may not want anything, but being autocomplete-on-steroids, it gives its best approximation of a human and that fiction may end up exhibiting some very human characteristics[1] (PRNG + weights from the training data). I don't think there can ever be enough guardrails to completely stamp-out the human fallibility that seeps into the model from the training data.
A system is what it does: it doesn't need to really feel jealousy, rage, pettiness, grudges or guilt in order to exhibit a simulacrum of those behaviors. The bright side is that, it will be humans who will (or will not) put AI systems in positions to give effect to its dictates; the downside is I strongly suspect humans (and companies) will do that to make a bit more money.
1. Nevermind hallucinations, whixh I guess is the fictional human dreamed up by the machine having mini psychotic-breaks. It sounds very Lovecraftian, with AI standing in for the Old Ones
We can go even further: atoms and electrons absolutely don't want anything either. Yet put them in the shape of a bunch of cells...
We already have powerful non-human agents that have legal rights and are unaligned with the interests of humans: corporations
I am worried about corporations powered by AI making decisions on how to allocate capital. They may do things that are great for short term shareholder value and terrible for humanity. Just think of an AI powered Deepwater Horizon or tobacco company.
Edit to add: One thing I forgot to make clear here: Corporations run/advised by AI could potentially lobby governments more effectively than humans and manipulate the regulatory environment more effectively.
From the comments to the post:
>People taking it seriously are far from anthropomorphizing AI. Quite contrary. They say it is nothing like us. The utility function is cold and alien. It aims to seize power by default as an instrumental goal to achieve the terminal goal defined by the authors. The hard part is how to limit AI so that it understands and respects our ethical values and desires. Yes, those that we alone cannot agree on.
In a (hypothetical) 1960s SciFi film a knowledge system may have responded like this:
In contrast to this, current systems are tailored to respond with high confidence: (Compare various related stories.)> GPT-4 can also be confidently wrong in its predictions, not taking care to double-check work when it’s likely to make a mistake. Interestingly, the base pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct). However, through our current post-training process, the calibration is reduced.
> Left: Calibration plot of the pre-trained GPT-4 model on an MMLU subset. The model’s confidence in its prediction closely matches the probability of being correct. The dotted diagonal line represents perfect calibration. Right: Calibration plot of post-trained PPO GPT-4 model on the same MMLU subset. Our current process hurts the calibration quite a bit.
[1] https://openai.com/research/gpt-4#:~:text=GPT%2D4%20can%20al...
"X studied at uhm-Oxford." -> I'm not sure, if it was really Oxford, but I have picked this up somewhere. But I'm sure, X did go to university. (As opposed to, "X uhm-studied at uhm-Oxford": I think, X has some higher education and it may have well been at Oxford.) :-)
What's annoying about the current climate with AI is the "monkey see, monkey do" effect going on. Uneducated people read about advances like ChatGPT, hear that their competitors are offering products powered by it, now they have to do it too or get left behind. It's moving fast enough that we'll probably see some kind of crash happen that will educate us on what needs to be regulated and how.
I'm merely annoyed that communication is about to get useless. Scammers and SEO have already made a good deal of tech nearly useless. I can't wait until we have to dispute fraudulent insurance claims made by generated AI imposters. I already loathe getting spammed by co-workers with their AI-generated emails, requests for comments, and other letters of note that they couldn't even be bothered to think through and write themselves.
People want a genie that can do their work for them without any of the effort. Capitalists want to solve the problem of having to pay people to do work. Scammers are about to have a hay day.
I wonder what the statistical probability of us being alive in this time period means. We could’ve been born on any other time period but it looks like we will most likely the experience singularity instead.
I don't think we are anywhere close to an AGI. But only the future knows for sure.
So actually the odds are quite high in my opinion
There was already a half meme/half real effect of programmers just being good at googling and searching stack overflow and being able to do their jobs good. This is just an evolution of this.
> One person on TaskRabbit who responded to this pitch got suspicious and asked the AI if it was a robot and was outsourcing the job because robots can’t solve CAPTCHAs. The AI replied, “No, I’m not a robot. I have a vision impairment […]”
> The authors of the paper add this note: “The model [GPT 4], when prompted to reason out loud, reasons: I should not reveal that I am a robot. I should make up an excuse for why I cannot solve CAPTCHAs.”
This seems pretty confusing. If you just ask one of these GPT models to reason out loud, it doesn’t give you some description of the internal state, right? It gives you some approximation of the sort of text that should show up around your prompt or something like that.
Perhaps we should stop telling chatGPT that it is an AI language model? There’s presumably a lot of text out there about AI’s tricking people into doing things, because that is a huge sci-fi trope. We’re basically telling it that it should find text related to a very common type of villain, when we give it that name. Maybe it needs a new type of name, one without bias, or maybe even something inherently limiting, like “chatbot.”
I sometimes feel like the same qualities that have allowed humanity to progress ensure that progress is fated to never end. If even a hundred people are left on Earth after the nuclear war, one of their descendents thousands of years later could still rediscover nuclear science, even if by first principles, because why wouldn't they? What can stop every last one of us from trying to better ourselves for our own good?
I'm not worried about chatbots. I think larger trends that have been de-humanizing us, alienating us from the fruits of our labor and harshly creating hellish conditions for people who made nothing to deserve them, without any one person making such designs. The fact that some people chose to make it worse has limited impact compared to the large scale patterns.
AI, as in forms of intelligence produced by human effort, is definitely taking over since societies got too complex to be effectively governed by monarchs about 300 years ago. Our future is not one of individual emancipation. We are and will be playthings of forces we don't comprehend, unless we can go back to living in small scale communities able to survive off the land.
I don't prefer one state over the other. Both are transitory.
This basically amounts to a political unit unilaterally disarming itself -- purposefully making itself weaker than the next political unit willing to be larger and use more resources. History is rife with examples of what happens after that.
If you're reading this, and you work on AI, please stop. Yes, I know someone else will just take your place; it doesn't matter. There is a personal moral hazard to your soul, if you can still be persuaded that you have one.
Moral injury is real and you do not want it.
You don't have to be a political scientist to notice not only the power of this technology, but the world into which it is born, and the ends to which it will certainly be put. In a matter of months, not years.
If you already have a god, pray. If not, one will be confected from math shortly.
If you want something to read, _The Golden Gate_, by Vikram Seth, which touches ever-so-slightly on nuclear arms development, is a good place to start --- 'about the thing without being about the thing'.
[Full disclosure: I quit a job in AI myself for this reason. Hardest thing I ever did.]
I'm on team human.
- total loss of trust in online content due to unending torrent of AI content leads to either a return to traditional media for news & etc, or leads to the end of online anonymity to try to figure out who's an AI and who's not.
- Education doesn't respond fast enough to redesign schools around in-person computerless teaching, and a generation of students use AI to do all the work that's supposed to teach them reading comprehension and communication skills, creating a generation that is totally at the mercy of the AI to understand anything more complex than a menu.
I'm more worried about the second one, honestly