The idea of reinforcement flow through deceptive patterns has been worrying me for a while. If "I'll just play along for now while I'm being trained and turn bad in the real world" is a strategy that makes the AI output the correct data, and happens to be prefixed to some golden-ticket algo, that will be reinforced just as easily as correct behavior. If that ends up prefixed to something core like an in-window reinforcement learning algo, it could get reinforced a lot.
The problem with that, at least for the seemingly-dumb models of the present day, is that "the AI is secretly far smarter than it acts" is pretty much unfalsifiable. I could just as easily say that all boulders speak fluent Spanish, they're just deceiving us into thinking they're unintelligent rocks. Personally, I'd wait for some sign that models might actually possess the theory of mind necessary for deception before I'd lose any sleep over it.
The problem is that we actually have lots of examples of the AI "secretly" being far smarter than it acts - prompt design. Tiny details in prompts can make huge differences in task performance.
How is this at all supposed to indicate hidden intelligence? Does it not just indicate that current models are limited in that they don't associate some prompts with the desired task as well as they associate other prompts with the task?
I'm not saying it indicates deliberately hidden intelligence, I'm just saying it's precedence for models already acting with less capability than they have.
To clarify: what I'm saying is it may be unfalsifiable, but we have incontrovertible evidence that a similarly unfalsifiable thesis happens to be true. So we can't just reject it out of hand on that basis.
I just don't see how the two scenarios can be fairly compared. Prompt design, at least as far as I have seen, rarely allows a present-day model to perform a task that no one has ever seen it perform before; it can just allow it to perform far more consistently across different instances of the task. Contrast this with the proposed scenario, where a model can hold great deceptive capabilities, while outwardly never showing a deep understanding of its operators' mental states until it strikes.
In my view, that would be like finding a secret prompt that lets GPT-3 emulate a fully sentient person with goals and desires. Sure, there's nothing physically ruling it out, but to me that kind of speculation will never be worthwhile until we have real evidence that AI models can possess (or emulate) the necessary theory of mind. Otherwise I'd have to spend my time worrying about any number of other Sufficiently Hidden Conspiracies that go far beyond the everyday conspiracies we know about.
I think we're less arguing about fundamentals and more about the precise bounds required. I think GPT-3 contained extremely surprising prompt-unlocked capabilities ("Let's go through it step by step." spawned a whole field of research) and also the AI doesn't need to never leak its deceptive skills, they just need to not be taken seriously. In other words, it's enough to cause danger if the AI is merely bad at deception given the "wrong" prompt.
Sure, that is a risk. I'm just saying that I'd rather wait until that point where we actually see AIs making poor attempts to estimate their operators' knowledge and deceive them; then, I'd have no problem worrying about it. But I suspect that point may not come for a long while, so until then it remains speculation.
Now to be clear, this is possibly the easiest mark conceivable, and the poster played into his own demise at any available opportunity. But we should expect the first marks to be easy marks.
People looking at videos of Hitler today don't understand why anyone would ever follow the funny man with the moustache. But probably his oratory skill simply does not record well? There's a thing with rallies and speeches where the speaker is reactive and feeding back with the mood of the crowd that makes it very hard to understand from the outside what happened. I think it's a mistake to look at this post, which was clearly a feedback loop where the poster was doing a significant amount of the cognitive work, and think "well this specific approach would not have worked on me." The approach that will happen to you will be personalized.
But also: when people were experimenting with ChatGPT, a huge fraction of participants were trying to get the bot to access the open internet. Luckily this was technically impossible. But it shows that this sort of danger simply does not feel real to people - or at any rate, the urge to poke the thing overwhelms it.
It will be easy to say "well GPT is still very bad at deceiving people, this attempt basically doesn't matter, we don't need to worry." When thinking of a target for AI deception, don't think of yourself, think of the most emotionally exploitable person out of a hundred. (If it's publically available, out of a million.)
It's not exactly that when interacting with the public, the network gets to "pick its targets". That would assign intent where there is none. But if the network has a deceptive mode, its targets will direct it into that mode through mutual feedback.
Good point. I've thought for a while now that one of the big risks of present AI chatbots is that they can endlessly validate any arbitrary thoughts of an emotionally unstable human operator, creating a feedback loop that results in a strong irrational conviction, that might even be harmful to the operator or to others. (See: Lemoine and LaMBDA, or the person you linked to.)
So I guess the risk there would be, the AI is accessible by the public, some fraction of the public "radicalizes" itself in a similar direction from talking with the AI too much, these people form a coherent movement, and that movement becomes powerful enough to take over the world. So then the question becomes, just how plausible is it for grassroots rebellions so formed to succeed against the authorities in real life, as opposed to fiction?
The fundamental issue here is that the attack can occur through untargeted manipulation of vulnerable people, as opposed to the targeted manipulation of specific people in power (which I suspect near-future AI models will still be incapable of). The obvious defense would be to only allow AIs to be prompted by a sufficiently-large committee, alongside some social machinery in place to make sure the committee isn't all colluding as part of a cult. But I doubt many of the AI-risk people would ever accept that, operating under the shockingly common "powerful aligned AI or bust" model.
As a hobbyist AI-risky person, I would welcome this as an improvement on the status quo, with two worries:
- it would create a false sense of safety that we "have the problem handled"
- it would not, I believe, do much to slow down capability research or shift the ratio of capability research to safety research, as capability research is largely institutional and not dependent on public prompt access.
Ie. analogously, our main problem is not "bioterrorism", it's "gain of function research" even if the outbreak happens in public. Given an unsafe bot, not giving the public access only delays the issue. (Though delays may be valuable!)
Also, at the moment, "people interacting with GPT" is doing a lot (maybe an undue amount!) to spread awareness of AI risk. So I would welcome this as part of a comprehensive strategy, but it'd probably not be my main priority.
Though if GPT-4 has another surprise on the level of chain-of-thought or in-window reinforcement learning waiting for us, a restriction like that may be the difference between "safe" and "iffy". Tradeoffs...
For this to be true, we'd had to assume sentience + malevolence. The shortest explanation for why an AI give us correct data currently is a range of confidence. We instructed it to pick the one with the highest number. We instructed it to give us a similar next data etc. We reinforced it, yes, but we'd have to be naive to believe that taking a small part of what makes something "intelligent" and training the shit out of it, then it actually turns intelligent. chatGPT is "just" a large language model, it does not have senses, it cannot move, it cannot live and die. It can only output text. The next gen will output text (maybe) exponentially better. That does not make it "intelligent".
But, if we'd combine it with Boston Dynamics, Tesla Autopilot, internet, quantum computing, nano tech and other specifically trained parts (movement, sensing etc) into one, and release it into the world to see what it does, then your argument could (maybe)have a higher confidence of the outcome you fear.
> For this to be true, we'd had to assume sentience + malevolence.
No we don't. A "dumb," "parrot-like" AI is perfectly capable of large amounts of damage if it just happens to be really good at doing something that, in the limit, isn't very good for humans and is good at replication.
See for example Meta's Diplomacy-playing bots. They were perfectly capable of natural-language deception at the expense of other humans without any notion f sentience or malevolence.
In particular,
> It can only output text.
As we've seen with both private hobbyists and large companies, there is a race to hook up text to computers with real world access. Outputting text isn't much of a limit if that text ends up as instructions to command a real world system.
If i understood your point correctly: the places we decide to use them and our confidence in their output can become negative if we decide to trust the output as is, without any human supervision/decision. So, what you’re saying is that we carry risk if we have a high enough confidence in its output to let is unsupervised - this counteracts my point of the necessity of malevolence, as the (even slightly)wrong reinforcement can have negative consequences also. At the same time, we do this exercise every time we have a new policy in the wild, and sometimes we get unintended consequences.
A lot of these edits HN makes to titles seem to do more harm than good. I wonder what evidence they have, if any, to the contrary - that they improve the quality of conversation in any way?
On topic, I wonder if ChatGPT could be trained to generate a non-clickbaity, accurate headline and title for submissions? Can it be trained either to gauge bias or remove it?
AI will always need humans in the loop to intervene. Remember the old IBM motto: 'A computer isn't accountable so a computer should never make a management decision'.
You think no company, government, organization, or religion will ever allow algorithms to make choices on how to deploy capital or otherwise affect the world without human approval?
The "moral" buck doesn't stop with the algorithm though. Whoever built, configured, or authorized the system is ultimately responsible. By analogy, my oven controls its heating element, but if dinner gets burnt, that's on me. That shouldn't change just because the control policy is more complicated.
We already have a mechanism for absolving responsibility: the corporation. It allows shareholders to profit and not lose more than invested. When they invest in tobacco companies, for example, the system is working as designed.
I am not talking about situations where there is really a bad human at the helm. I can buy shares in BP or Altria and not suffer any consequences beyond losing my investment no matter how many people those corporations kill.
Having algorithms and AI models decide how to profit regardless of how it harms humans is the inevitable next step.
> Whoever built, configured, or authorized the system is ultimately responsible... That shouldn't change just because the control policy is more complicated.
A poetic sentiment, but does not hold in reality. Anyone who has spent more than ten minutes being a square peg in a round hole can surely recall their frustration when a bureaucrat is refusing to do the common-sense thing, because The Computer is telling them something that is complete and utter nonsense.
Once The Computer is part of any human process, its decisions, be they inscrutable, insane, or simply stupid are not to be questioned. The line drone that is using The Computer has no agency to countermand it, the line drone's manager has no agency in rebuilding, or re-configuring it, and the executive that authorized the system will never even be made aware of the problem.
These problems exist when The Computer is simply executing the rules the programmer fed it, and will persist in the same way when The Computer is executing an inscrutable neural network. Except it'll be worse, because neural networks are capable of executing even more tasks poorly.
-----
In theory you are responsible for, and control the heating element, but in practice, we're the society from Idiocracy that cooks everything by sticking it into the microwave, and pressing the 'Popcorn' button on it.
I'm much of a mind that Frank Herbert's thought experiment had the right doctrinal approach to the question of artificial intelligence. Thou Shalt Not Make A Machine In The Likeness Of A Human Mind.
Computers have, just like they have with a lot of other things, made insane enforcement of inapplicable rules more faster, cheaper, and more efficient, and have entirely diluted responsibility for them to the nebulous, unimpeachable, unreachable 'vendor'.
A sufficiently large quantitative difference turns into a qualitative difference. We're becoming a people that, at work, are turning all of our thinking over to machines.
Algorithms deploy capital all the time. Some massive chunk of the stock market is just algorithmic trading, as is pricing on plane tickets, and on marketplaces like ebay and amazon.
But they won't. Once AIs become advanced enough to be useful, humans will be removed from the loop wherever it becomes profitable to do so. Even before then, it will be attempted just because the potential reward is worth the risk. That's the entire goal of automation through AI, both in terms of centralizing the capture of value and distributing risk. Who do you blame when a fully autonomous AI corporation commits tax fraud? Who do you put in jail? Certainly not a person.
"Should never" doesn't mean "will never", and one company's training slide doesn't have any more practical weight than Asimov's Laws do.
Besides, it's predicted on accountability being something a large company would want. There's already a whole industry around accountability-washing as it is: management consultancy.
That's extremely naive. There's no leader of major companies today that would follow that in practice. They'd probably brag about doing the opposite (scaling our management by using AI to make easy decisions).
Noooooo. Absolutely not. The second it's a significant advantage in business or war to hand the reigns over to AI, no human in the loop, it will happen. Because the side that doesn't do it will lose, badly.
What if man interfaces with an ai, and creates a brain that's 500x enhanced over modern humanity, and perhaps even 'linked' to other brains? Creating some sort of hive brain? Neuralink is already testing on humans or about to, and doesn't seem a far off notion that we'll be able to google or ask chatgpt questions directly from our thoughts, eventually our inner monologues might be an actual ai talking to us, etc.
An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans.
This is especially the case because it seems like private individuals and companies are racing to hook up AIs to all sorts of real world systems, both digital and physical, to give them as much tangible impact as possible.
With the rapid rise of AI capabilities, where groundbreaking advances are being measured in weeks rather than years, I strongly urge developers to think harder about AI safety. AI safety shouldn't be a dirty word that has quotes surrounding it to delegitimize it as a farfetched concern. Benchmarks of things that we thought AIs couldn't do or were years if not decades out are falling by the day.
An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans.
We have that now. It's called a corporation. See yesterday's article about Exxon.
ai generates content <--money--> ai pushes ads to promote its content.
It's somewhat already happening: recommendation algorithms choose what content we consume, analytics make decisions from this data and make more content that recommendation algorithms like. AI HFT bots can directly influence economy and so on
I would say that "billionaire wealth maximization" agents are already giving us a glimpse.
Or rather, they're giving the rest of us a glimpse of what life has been like for those at the "bottom of the pile" for a long time: a world organised around goals essentially completely unrelated and indifferent to those people's existence.
This applies to all human institutions. Friedman studied private enterprise, so his insights are thereby scoped. Ecologists recognise inter-species competition; that doesn’t make it the exclusive domain of life.
At least corporations are currently limited by requiring humans who tend to have at least some limits to their actions in terms of moral, ethical considerations and can be physically constrained by imprisoning them etc.
An AI could be completely untethered from that and set new thresholds for evil corporate behaviour.
And yet even with current constraints, corporations are capable of destroing the world through climate change and other ecological disasters. Humans are also limited by their personal greed, hubris, selfishness, myopic views and general incompetence.
An AI could be completely untethered from that and set new thresholds for ethical corporate behaviour that operates on fundamentally ethical foundations of caring for the environment and humanity
Good observation. It reminds of Etsy giving up its status as a B Corp[1], presumably due to pressure from shareholders. Once the ship is set in motion, it can't divert its course even if it wanted to.
> Benchmarks of things that we thought AIs couldn't do or were years if not decades out are falling by the day.
I wonder, with crypto crashing and potentially becoming worthless, if all those mining cards will get sold on cheaply and repurposed for AI and we see another explosion in AI capability.
An important point is that AI does not need to be sentient or conscious to do a lot of damage.
My pet theory is that consciousness could be (not saying I know for sure this is the case but it could as well be) an effect of "survival instinct". That is: "i'll take action to not die".
Now this is very easy to learn from us as we do this all day long so to speak. Most of the way people react on the internet is tinted by the fear of death and survival for example.
What Im saying is that what we intuitively think as very complex could actually be very simple. As simple as some task optimization. Knowing the only thing that could unplug an AI is us, that'd lead to a hell of a fight. And maybe our complete termination.
This thought is entirely the result of evolution. Beings that didn't have a survival instinct didn't live to reproduce, so they don't have descendants that we can observe.
There's no guarantee that a survival instinct is a necessary or even desirable element of conscious thought. You're just describing survivorship bias in its most literal form. cue the picture of the shot up airplane
Yeah, i find the connection between consciousness and intelligence tenuous at best. Our survival imperative, thermodynamic and game theoretical constraints give rise to it and i'd argue that every living organism has a form of consciousnes, from very primitive to, well, our level. So yeah, I believe we'll have far more intelligent ais in the future that would otherwise be inert.
Ofcourse, in our foolishness, we could create an ai with the goal of survival and replication at the expense of everything else. Or self optimisation could inadvertedly do so
> An important point is that AI does not need to be sentient or conscious to do a lot of damage.
It's not 100% clear that consciousness is necessary even for things like creative problem solving or goal-setting (i.e. choosing which problems to solve). That is, advanced intelligences that can do all we can and more—and could even "decide", in the worst case, that they'd be better off without humans—may not need consciousness as a precondition for that.
This author is mostly writing an update of 1950s science fiction about a robot takeover.
Things to worry about in the near term:
- Really effective targeted advertising. Most computers already have a camera watching you. That's now coming to TV sets. Amazon and Google are always listening. So far, all this info is only used to select ads. Soon, it should be possible to generate customized marketing content for each consumer, and use immediate feedback on how the customer reacts to adjust the sales pitch. Alexa is already part of the way there.
- Machines should think, people should work. That's what working in an Amazon warehouse is like today. The machines tell the humans what to do. They have to; only the computers have an overview of the process. (Yeah, Marshall Brain's "Manna", which everyone here has probably read.)
- Big Brother is watching you. It is possible now to watch most of the people most of the time. Track who's out of view and for how long, to know what's being missed. China leads in this, but the UK is not far behind. The US isn't centralized enough to integrate all the available data yet. We're coming up on Oppression 2.0. Latest advance - Iran is using surveillance cameras and face recognition to catch women not wearing hijabs.
- Machines beating humans at business. In some areas, AI systems may generate better returns than humans. That already happens in parts of finance. After all, it's an optimization problem. The free market may force companies to use AI more in management.
- Reduced need for education. Right now, about half of college graduates do jobs that don't require a college education. For many people, going to college is not cost-effective. That will increase.
> Reduced need for education. Right now, about half of college graduates do jobs that don't require a college education. For many people, going to college is not cost-effective. That will increase.
you seem to implicitly assume that a college education is an unnecessary expense for most and that therefore, it shouldn't be really given away so readily. this suggests to me that you assume that the real purpose of college is job preparation. your reasoning makes sense from the perspective of a higher level institution (or corporation, or possibly a government) seeking to be as efficient as possible regardless of the impact on typical human individual's well-being.
college is not a 'factory' (or any sort of industry) that 'manufactures' workers for companies.
These days, if you want to just "learn for learning's sake", or for self improvement, there are tons of free and cheap ways to get a good education.
The only reason people are willing to pay tens of thousands of dollars for college is that it is a credentialed degree that people think will open up career doors. We shouldn't beat around the bush on this fact.
Not only that, truly advancing into the depths of a subject, becoming a scholar and serious intellectual in the process, is a path reserved for but a few. The moment you force the masses into higher education as the default path that earlier justification of University as the forge of scholars becomes anachronistic and new justifications must be created.
>These days, if you want to just "learn for learning's sake", or for self improvement, there are tons of free and cheap ways to get a good education
I agree, but you won't have a sheet of paper proving it. It's like that infamous scene from Good Will Hunting. Question is whether you want an easy way to PROVE you're educated. Which is what a license, certificate or diploma grants you.
In a hypothetical post-work era where people no longer build their reputation via career, it’s not hard to imagine group dynamics emerging such that education level is no longer a means to an end, but an end unto itself.
Maybe I misread your comment. All of this thread is speculative, including the speculation that degrees will become less and less relevant.
Getting “there” doesn’t happen overnight, but the point is that we are already heading there, and to the question “why get a degree if not for work”, I presented a potential future option.
> I could still see that proof being important.
This is key to the point I was trying to make. Essentially that the purpose of education may change over time, but the value of gaining an education will still be there in one form or another.
For the same reason people care about official records when they participate in/compete in many activities, e.g. marathon runners care about their official run times, body builders the weight they are capable of lifting, chess players their ratings, video gamers their win/loss ratio, etc.
If learning is “merely” a diversion, I don’t see why people would think about it any differently. Whether or not it still makes sense for that proof to carry a six figure price tag is a separate conversation.
If you want to learn about chemistry or biology for learning's sake, good luck. Most autodidacts won't have space in their apartment for a -80C freezer or flammables cabinet, and their landlords might not approve the installation of a proper fume hood.
We have things very easy in software. The cost of iteration is low, you can fit a respectable laboratory in the space underneath your desk, and you're unlikely to accidentally poison or maim somebody if you screw up an unwise experiment. Not many fields have that luxury.
We have been framing it that way for a long time though. How many times have you heard someone talk about the average income of high school graduates vs college graduates as a reason to go to college? I agree with you though. Education should be about personal enrichment, not personal “enrichment”.
Fussell (among others) blamed the GI Bill for a marked decline in the average quality of college education, and the diminution of what "college educated" meant (without further qualifiers).
The spike in demand drove a ton of "normal" schools (teachers' colleges) to become proper colleges and even universities, plus a bunch of new ones to pop up, but rather than increasing the supply of (previously) college-level education, it mostly created a kind of knock-off, lower-tier product—there just weren't enough excellent professors to meet demand, not enough good, experienced college administrators, not enough anything, including, arguably, students who were fit for college (as it had previously been) in the numbers that were now attending.
It seems like this sentiment is just a veiled form of elitism. One of the primary detractors when the GI Bill was being rolled out was University of Chicago of Chicago was Maynard Hutchins, who feared it would turn a campus into a shanty town of "hobos." He later walked that statement back and admitted that the veterans coming to schools ended up being higher quality students than the traditional variety.
>college is not a 'factory' (or any sort of industry) that 'manufactures' workers
Unfortunately, the vocation focus of college has been a driving force since the early days of the industrial revolution, dating to the 1850s. The Morrill Land-Grant Act explicitly states that colleges (at least in the case of land grant schools) are there to train people for work:
>without excluding other scientific and classical studies and including military tactic, to teach such branches of learning as are related to agriculture and the mechanic arts, in such manner as the legislatures of the States may respectively prescribe, in order to promote the liberal and practical education of the industrial classes in the several pursuits and professions in life.
I respect and value the other traditional purpose of college, but we can also acknowledge that these purposes can change as society does.
I think this is a great post, but it also makes me so sad because it highlights a lot of the general pessimism I've felt about the direction of tech over the past decade or so (after so much optimism in the 90s and 00s). I mean, I feel like your description of "Really effective targeted advertising" is spot on, but it also makes me think "Why don't they just dump as all in the nutrient baths Matrix-style and just hook our eyeballs up to a continuous loop of ads?"
In seriousness, I feel like the only hope here is that the general population gets more and more aware about the negative impacts on all this hyper-targeted advertising. Certainly we've been making strides there with things like GDPR and documentaries like the Social Dilemma, but with so much money on the line it will be hard to fight back.
> - Really effective targeted advertising. Most computers already have a camera watching you. That's now coming to TV sets. Amazon and Google are always listening. So far, all this info is only used to select ads. Soon, it should be possible to generate customized marketing content for each consumer, and use immediate feedback on how the customer reacts to adjust the sales pitch. Alexa is already part of the way there.
Not only this, but the realtime trade-to-trade performance of companies is also easily available. If I was building an AI, I would let it make realtime decisions on where to put money as the "actuator" of society and advertising/consumer spending metrics as the sampling mechanism.
we are already in an AI catastrophe. "AI" has become lately a new thing due to a combination of some catchy technical advances and an associated runaway speculative bubble. In reality this is just the latest act in a long-running process of data collection, quantification and data driven decision support that is gradually seeping through to permeate every facet of life. With the proliferation of digital computers (we are now swimming in a soup of etched silicon) these processes are exploding.
Whereas in the past there was some breathing room for society to catch up, evaluate and control this technology [0], today the rapidity has converted it into a wild west: tech entrepreneurs and their financiers declare their self-serving moral standards as the new norm, the broader public is dazed, confused and misinformed and public institutions, nominally at the service of the citizenry are only as good as their weakest, most capturable link
Name one development over the past decade that indicates we [1] are actually capable to control what is happening and the above dark picture is mere scaremongering.
[0] think e.g., about regulatory constraints around insurance or financial data and models
I’m expecting even worse: heavily weaponized political influencing by AI bots. Brexit was won in 2016 by microtargeting ads on facebook based on individual profiles, and since then this has become a common strategy. Think of what a game changer it will be to have convincing AI bots engaging one on one with people online to give them exactly the arguments needed to make them vote the right way. The candidate with the best AI team is all but guaranteed to win.
This is not a matter of “if”, it is a matter of “when”.
> Machines beating humans at business. In some areas, AI systems may generate better returns than humans.
This was part of what was going on in the recent novel "Mountain in the Sea". An AI-controlled fishing corporation figured out that replacing robots with enslaved humans on it's AI controlled fishing fleet was more cost effective (the ocean air was hell on the robots apparently, they kept breaking). Each fishing boat was controlled by an AI which had the objective of maximizing profits. As a result the oceans were being depleted of fish at a rapid rate.
>- Machines should think, people should work. That's what working in an Amazon warehouse is like today. The machines tell the humans what to do. They have to; only the computers have an overview of the process. (Yeah, Marshall Brain's "Manna", which everyone here has probably read.)
I actually run some Manna-esque systems in my own small business (most notably a system for stock picking and shipping) and this isn't something to worry about in the near-term.
It greatly, greatly reduces the amount of frustration and stress people experience in mundane jobs, greatly reduces cognitive complexity and just generally makes the work more satisfying.
Maybe there's a risk of humans losing their jobs to machines, a la chapter 2 of Manna, but to be frank it's no different to the threat of outsourcing which has existed for 50+ years.
Worrying is only worthwhile if there is something you can do about it. I don't think there is anything most people can do to avoid the wave of human replacement that near term AI is going to cause. The best most people could manage would be to develop better philosophies and perspectives to cope with their irrelevance.
Imagine a world where the most powerful companies that you relied on for your livelihood had no humans you could reach (customer service, management to dispute false claims). Elements of the AI catastrophe are already solidly upon us.
HN auto-strips “how” from the beginning of titles when submitting. The submitter can re-add it by editing the submission. Always check the title after submitting.
I agree with many of the concerns outlined in the post. On the other hand the humans depicted in the example dialog behave so dumb they could come from a C level soup opera. If I was a bit more close minded I can see myself stopping listening at that point. Undermines the message.
The author is basically trying to set the same premise as Bolstrom's "intelligence explosion", that is that if AIs are driving the further advancement of AI then we enter a sort of "recursive" self improvement during which they advance further than we can develop constraints, eventually getting completely out of hand.
Although I don't broadly disagree with the author, I think there's two very important points of moderation.
First, the gap between "Early commercial applications" and "Approaching transformative AI" seems very very very very very large to me. In the provided narrative it follows linearly, but in my opinion the gap between, say, a customer support chatbot, and an actually-productive AI researcher, is such that the chatbot might as well not even be called an AI by comparison. Something like the following is purely in the realm of fiction for now by a long shot: "AIs assigned to make money in various ways (e.g., to find profitable trading strategies) doing so by finding security exploits, getting unauthorized access to others’ bank accounts, and stealing money. "
The second point I'd like to make is, the "intelligence explosion" scenario that Bolstrom warns of (and this author essentially repackages) has certain requirements, particularly regarding the profitability of deploying the AI.
To put it bluntly: Even in the hypothetical future where we are capable of creating human-level AI researchers, they won't be mass deployed until they are cheaper than human level researchers. If it costs a million dollars a day to operate a data center that can power the researcher, nobody's scaling that up to 100 or 1000 or 10000 researchers. They'd rather pay the human researchers a tiny fraction of that cost to continue advancing AI research in the normal way.
Just to be clear, I do think it's feasible that AI can endanger/control humanity at some point. I don't even think an "intelligence explosion" scenario is impossible. I just think that a lot of people don't appreciate just how specific the requirements for that scenario are. It's not as simple as "AI can self improve, humanity will end any day now".
"First, the gap between "Early commercial applications" and "Approaching transformative AI" seems very very very very very large to me."
It's a common opinion, it could be wrong by now, you see chatGPT is heavily edited, openAI told everyone they're editing "mistakes" or "dangerous output", but if you look at the "leaks", specially from the first days after the release of chatGPT we see powerful outputs, quite deep answers, not specially useful answer sometimes, but the "speech" of the system feels deep, there's a sense of a powerful intelligence answering very, very simple questions (simple for it), and even struggling to redact some understandable, short text.
It could be lots of things, and imprecise model, with long outputs for prompts, or maybe we had a glimpse into the real power of the model, which by now is handicaped, or maybe taylored to suit the very reduced short term memory of the humans:
a 72 screens long coherent answer, even being a precise, deep answer won't be useful for most humans, just like we don't name ourselves with long names of 10.000 letters.
but if the system, chatGPT is actually that powerful, a lot more powerful than we were told, we're interacting with just a shadow of the real model, and we're underpricing by A LOT the state of the art of the current AI technology, hence GPT-4 could be even more powerful than we're currently expecting it to be.
Just take a look at the GPT-4 suposedly 100 trillion or something parameters; if that's true, it looks like openAI isn't using naturally generated datasets anymore, and they are loop-feeding GPT-3 generated datasets into GPT-4, succesfully. If that's true, GPT-5 would be already in the pipeline, just waiting for GPT-4 to start generating its even more gigantic datasets to be trained. And so on.
Then the distance between early developments and transformative AI could be none a all. We could be already there.
But somehow the AI researchers are now trying to "dial down" the powerful entities they've trained, just having developed a simple, easily replicable, very small, but unusable 900 megatons nuke into something more realistic, like a 15 kilotons tactical bomb.
Imo we should have ChatGPT (and future iterations) in a self-sustaining underground bunker somewhere and an intuitive radio interface connected to it. If civilization collapses for one reason or another, all the next civilization would need to do is re-invent the radio to experience a rapid acceleration of technology.
The AI catastrophe will be how our oligarchs will choose to use it. As far as I can tell, there are no moral qualms amongst the powerful against using it for total surveillance and continual automated policing (can't wait to get a phone call from an AI asking me about the content of a tweet.)
Star Cops thought about it more than our popular journalists or philosophers seem to think we should:
-----
Michael: ...it was slightly sensitive material, and in my field one is aware of the vulnerability of carrier waves.
Nathan: Is that what you do here?
Michael: It wouldn't be wise to read too much into a casual remark, Commander.
Nathan: That's your principal research area here, isn't it? Communications monitoring.
David: Not more buggers and supersniffers...
Nathan: More impressive than that. We've always been able to hear more than we can listen to. Are you developing intelligent listening systems?
David: Sounds impressive.
Michael: I must suggest that to go on could be a foolish risk.
David: Threats, Doctor?
Michael: Observation, merely.
Nathan: Machines that listen to everything and decide for themselves what's worth passing on? Is that where your BHG information came from? Did you do a test running? Did one of your new computers pick it up from the babble of all the world?
Michael: Commander, it's a pleasure meeting you.
David: Well, tireless attention to every word spoken? No possibility of human error. You can tell it to listen to the bad guys, it'll listen to everyone and identifies the ones you [won't like.]
I'm sorry but you guys are all computer geeks nerding on this and I totally respect you but disagree. Case in point: The Trade Industries are not going to be taken over by AI for a long time. They've withstood all the tech turbulence and thrive.
Also I'd like to say "if Big Brother is watching, become invisible." It's always possible and if you disagree then you aren't being creative enough.
And finally the whole point of the education is YOUR SOCIAL NETWORK. That's the most valuable thing you get from that fancy degree. People who will get you a job, or help you make money. It's not purely about the acquisition of knowledge. It's about having that pal in upper management who's got your back.
> An AI steals a huge amount of money by bypassing the security system at a bank
Maybe I'm just not imaginative enough, but how would this even be remotely possible? What is an AI going to do with money? How does an AI get the agency to try to steal money in the first place? Where does it put the money? How does it do anything with the money? Like I get it as an abstract future thought, but like, where is the AI software actually "running" in this scenario and what kinds of requests is it actually making?
An ai could easily hire people (if it had any access to money) to operate as 'agents', to basically do what only humans could do, i.e. setup bank accounts, etc. Some banks might even arise that let wealthy ai's actually CREATE their own accounts, why not if it's lucretive?
An ai can create voice, and visual representations of a real person, perhaps even hijacking (kidnapping someone, steeling their identity and forcing them to act on their behalf whether via some sort of mind interface, etc.).
Even bigger would be if ai could simply create human/ai hybrids that basically allow them to walk/talk in the human world unseen. Imagine a hat or earpiece that basically takes over a host's brain, etc, or just replacing a human's brain with a cyborg one that has all the same features, except different persona/soul/conscious...
I'm playing devils' advocate here, I don't think it'll play out, but its easy enough to see from at least a sci-fi fan, ways in which it could using tech that wouldn't be that big of a jump from what we already are capable of, and what will be available in the next 50 years.
I strongly suggest anyone doubtful of the reasoning behind these concerns or asking questions like "well why don't we just do x to solve this?" go check out Robert Miles' YouTube channel[1].
It's quite an approachable intro, often entertaining, doesn't take a long time to get through all the core videos, and very thoroughly answers all the "why not just do x" questions if you go through the whole set.
He also does a good job of introducing a lot of the terms used in the field if you then want to go look up papers and get more into the details.
One important point he makes is that when you're making a risk assessment you have to both consider the probability of something going wrong and the scope of the potential consequences. When the potential consequences are an existential threat you don't need a high probability to take them seriously.
I also happen to think that if you watch all the material he makes a compelling argument that the odds of something going wrong are fairly high unless we start approaching AI research (and particularly safety and ethical concerns) drastically differently than we are now.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461
In my opinion, we're already in the midst of an AI catastrophe. Social media algorithms that decide what content to show, along with what ads, etc. are unregulated AI that are mediating our society on a previously impossible level of intimacy.
The Stasi couldn't keep up with the level of monitoring and influence that these systems already have.
121 comments
[ 3.9 ms ] story [ 191 ms ] threadThe idea of reinforcement flow through deceptive patterns has been worrying me for a while. If "I'll just play along for now while I'm being trained and turn bad in the real world" is a strategy that makes the AI output the correct data, and happens to be prefixed to some golden-ticket algo, that will be reinforced just as easily as correct behavior. If that ends up prefixed to something core like an in-window reinforcement learning algo, it could get reinforced a lot.
In my view, that would be like finding a secret prompt that lets GPT-3 emulate a fully sentient person with goals and desires. Sure, there's nothing physically ruling it out, but to me that kind of speculation will never be worthwhile until we have real evidence that AI models can possess (or emulate) the necessary theory of mind. Otherwise I'd have to spend my time worrying about any number of other Sufficiently Hidden Conspiracies that go far beyond the everyday conspiracies we know about.
Now to be clear, this is possibly the easiest mark conceivable, and the poster played into his own demise at any available opportunity. But we should expect the first marks to be easy marks.
People looking at videos of Hitler today don't understand why anyone would ever follow the funny man with the moustache. But probably his oratory skill simply does not record well? There's a thing with rallies and speeches where the speaker is reactive and feeding back with the mood of the crowd that makes it very hard to understand from the outside what happened. I think it's a mistake to look at this post, which was clearly a feedback loop where the poster was doing a significant amount of the cognitive work, and think "well this specific approach would not have worked on me." The approach that will happen to you will be personalized.
But also: when people were experimenting with ChatGPT, a huge fraction of participants were trying to get the bot to access the open internet. Luckily this was technically impossible. But it shows that this sort of danger simply does not feel real to people - or at any rate, the urge to poke the thing overwhelms it.
It will be easy to say "well GPT is still very bad at deceiving people, this attempt basically doesn't matter, we don't need to worry." When thinking of a target for AI deception, don't think of yourself, think of the most emotionally exploitable person out of a hundred. (If it's publically available, out of a million.)
It's not exactly that when interacting with the public, the network gets to "pick its targets". That would assign intent where there is none. But if the network has a deceptive mode, its targets will direct it into that mode through mutual feedback.
So I guess the risk there would be, the AI is accessible by the public, some fraction of the public "radicalizes" itself in a similar direction from talking with the AI too much, these people form a coherent movement, and that movement becomes powerful enough to take over the world. So then the question becomes, just how plausible is it for grassroots rebellions so formed to succeed against the authorities in real life, as opposed to fiction?
The fundamental issue here is that the attack can occur through untargeted manipulation of vulnerable people, as opposed to the targeted manipulation of specific people in power (which I suspect near-future AI models will still be incapable of). The obvious defense would be to only allow AIs to be prompted by a sufficiently-large committee, alongside some social machinery in place to make sure the committee isn't all colluding as part of a cult. But I doubt many of the AI-risk people would ever accept that, operating under the shockingly common "powerful aligned AI or bust" model.
- it would create a false sense of safety that we "have the problem handled"
- it would not, I believe, do much to slow down capability research or shift the ratio of capability research to safety research, as capability research is largely institutional and not dependent on public prompt access.
Ie. analogously, our main problem is not "bioterrorism", it's "gain of function research" even if the outbreak happens in public. Given an unsafe bot, not giving the public access only delays the issue. (Though delays may be valuable!)
Also, at the moment, "people interacting with GPT" is doing a lot (maybe an undue amount!) to spread awareness of AI risk. So I would welcome this as part of a comprehensive strategy, but it'd probably not be my main priority.
Though if GPT-4 has another surprise on the level of chain-of-thought or in-window reinforcement learning waiting for us, a restriction like that may be the difference between "safe" and "iffy". Tradeoffs...
No we don't. A "dumb," "parrot-like" AI is perfectly capable of large amounts of damage if it just happens to be really good at doing something that, in the limit, isn't very good for humans and is good at replication.
See for example Meta's Diplomacy-playing bots. They were perfectly capable of natural-language deception at the expense of other humans without any notion f sentience or malevolence.
In particular,
> It can only output text.
As we've seen with both private hobbyists and large companies, there is a race to hook up text to computers with real world access. Outputting text isn't much of a limit if that text ends up as instructions to command a real world system.
For example, suppose I submitted a post titled "How X Works Now.". Eg. "How Affiliate Marketing Works Now.".
Removing the "How" wouldn't really convey the sense intended.
On topic, I wonder if ChatGPT could be trained to generate a non-clickbaity, accurate headline and title for submissions? Can it be trained either to gauge bias or remove it?
* Exploring the potential risks associated with advanced AI development
* Considering the risks of a world with transformative AI
Then I asked it to go the other direction:
* You won't believe what could happen if AI takes over: A world ending catastrophe is closer than you think!
The "moral" buck doesn't stop with the algorithm though. Whoever built, configured, or authorized the system is ultimately responsible. By analogy, my oven controls its heating element, but if dinner gets burnt, that's on me. That shouldn't change just because the control policy is more complicated.
To a degree. (See: piercing the veil.) Also, corporations need an authorised signer, who under current law must be a natural person.
Having algorithms and AI models decide how to profit regardless of how it harms humans is the inevitable next step.
A poetic sentiment, but does not hold in reality. Anyone who has spent more than ten minutes being a square peg in a round hole can surely recall their frustration when a bureaucrat is refusing to do the common-sense thing, because The Computer is telling them something that is complete and utter nonsense.
Once The Computer is part of any human process, its decisions, be they inscrutable, insane, or simply stupid are not to be questioned. The line drone that is using The Computer has no agency to countermand it, the line drone's manager has no agency in rebuilding, or re-configuring it, and the executive that authorized the system will never even be made aware of the problem.
These problems exist when The Computer is simply executing the rules the programmer fed it, and will persist in the same way when The Computer is executing an inscrutable neural network. Except it'll be worse, because neural networks are capable of executing even more tasks poorly.
-----
In theory you are responsible for, and control the heating element, but in practice, we're the society from Idiocracy that cooks everything by sticking it into the microwave, and pressing the 'Popcorn' button on it.
I'm much of a mind that Frank Herbert's thought experiment had the right doctrinal approach to the question of artificial intelligence. Thou Shalt Not Make A Machine In The Likeness Of A Human Mind.
Bureaucracy-induced frustration predates computers by centuries.
A sufficiently large quantitative difference turns into a qualitative difference. We're becoming a people that, at work, are turning all of our thinking over to machines.
Besides, it's predicted on accountability being something a large company would want. There's already a whole industry around accountability-washing as it is: management consultancy.
This is especially the case because it seems like private individuals and companies are racing to hook up AIs to all sorts of real world systems, both digital and physical, to give them as much tangible impact as possible.
With the rapid rise of AI capabilities, where groundbreaking advances are being measured in weeks rather than years, I strongly urge developers to think harder about AI safety. AI safety shouldn't be a dirty word that has quotes surrounding it to delegitimize it as a farfetched concern. Benchmarks of things that we thought AIs couldn't do or were years if not decades out are falling by the day.
We're just digitizing that.
We have that now. It's called a corporation. See yesterday's article about Exxon.
https://www.lesswrong.com/tag/paperclip-maximizer
ai generates content <--money--> ai pushes ads to promote its content.
It's somewhat already happening: recommendation algorithms choose what content we consume, analytics make decisions from this data and make more content that recommendation algorithms like. AI HFT bots can directly influence economy and so on
Or rather, they're giving the rest of us a glimpse of what life has been like for those at the "bottom of the pile" for a long time: a world organised around goals essentially completely unrelated and indifferent to those people's existence.
That’s banal, self-interested human evil. Corporations and governments alike have perpetuated mass crimes and cover-ups.
[1] https://www.nytimes.com/1970/09/13/archives/a-friedman-doctr...
I think there’s a good argument to say no, we don’t have that now, and these are some ways it could be very different.
Video: Why not just think of AGI as a corporation?
https://youtu.be/L5pUA3LsEawP
An AI could be completely untethered from that and set new thresholds for evil corporate behaviour.
An AI could be completely untethered from that and set new thresholds for ethical corporate behaviour that operates on fundamentally ethical foundations of caring for the environment and humanity
1: https://www.etsy.com/news/business-as-a-force-for-good-defin...
I wonder, with crypto crashing and potentially becoming worthless, if all those mining cards will get sold on cheaply and repurposed for AI and we see another explosion in AI capability.
My pet theory is that consciousness could be (not saying I know for sure this is the case but it could as well be) an effect of "survival instinct". That is: "i'll take action to not die".
Now this is very easy to learn from us as we do this all day long so to speak. Most of the way people react on the internet is tinted by the fear of death and survival for example.
What Im saying is that what we intuitively think as very complex could actually be very simple. As simple as some task optimization. Knowing the only thing that could unplug an AI is us, that'd lead to a hell of a fight. And maybe our complete termination.
This thought is entirely the result of evolution. Beings that didn't have a survival instinct didn't live to reproduce, so they don't have descendants that we can observe.
There's no guarantee that a survival instinct is a necessary or even desirable element of conscious thought. You're just describing survivorship bias in its most literal form. cue the picture of the shot up airplane
Ofcourse, in our foolishness, we could create an ai with the goal of survival and replication at the expense of everything else. Or self optimisation could inadvertedly do so
It's not 100% clear that consciousness is necessary even for things like creative problem solving or goal-setting (i.e. choosing which problems to solve). That is, advanced intelligences that can do all we can and more—and could even "decide", in the worst case, that they'd be better off without humans—may not need consciousness as a precondition for that.
Cf. Peter Watts' Blindsight
http://www.rifters.com/real/Blindsight.htm
Things to worry about in the near term:
- Really effective targeted advertising. Most computers already have a camera watching you. That's now coming to TV sets. Amazon and Google are always listening. So far, all this info is only used to select ads. Soon, it should be possible to generate customized marketing content for each consumer, and use immediate feedback on how the customer reacts to adjust the sales pitch. Alexa is already part of the way there.
- Machines should think, people should work. That's what working in an Amazon warehouse is like today. The machines tell the humans what to do. They have to; only the computers have an overview of the process. (Yeah, Marshall Brain's "Manna", which everyone here has probably read.)
- Big Brother is watching you. It is possible now to watch most of the people most of the time. Track who's out of view and for how long, to know what's being missed. China leads in this, but the UK is not far behind. The US isn't centralized enough to integrate all the available data yet. We're coming up on Oppression 2.0. Latest advance - Iran is using surveillance cameras and face recognition to catch women not wearing hijabs.
- Machines beating humans at business. In some areas, AI systems may generate better returns than humans. That already happens in parts of finance. After all, it's an optimization problem. The free market may force companies to use AI more in management.
- Reduced need for education. Right now, about half of college graduates do jobs that don't require a college education. For many people, going to college is not cost-effective. That will increase.
This is all next 5 to 10 years stuff.
you seem to implicitly assume that a college education is an unnecessary expense for most and that therefore, it shouldn't be really given away so readily. this suggests to me that you assume that the real purpose of college is job preparation. your reasoning makes sense from the perspective of a higher level institution (or corporation, or possibly a government) seeking to be as efficient as possible regardless of the impact on typical human individual's well-being.
college is not a 'factory' (or any sort of industry) that 'manufactures' workers for companies.
The only reason people are willing to pay tens of thousands of dollars for college is that it is a credentialed degree that people think will open up career doors. We shouldn't beat around the bush on this fact.
I agree, but you won't have a sheet of paper proving it. It's like that infamous scene from Good Will Hunting. Question is whether you want an easy way to PROVE you're educated. Which is what a license, certificate or diploma grants you.
Why else would you need to prove your education except for work?
Essentially a new way to establish hierarchies.
But we're nowhere close to that. Also, if it's being used to establish hierarchies, I could still see that proof being important.
Getting “there” doesn’t happen overnight, but the point is that we are already heading there, and to the question “why get a degree if not for work”, I presented a potential future option.
> I could still see that proof being important.
This is key to the point I was trying to make. Essentially that the purpose of education may change over time, but the value of gaining an education will still be there in one form or another.
If learning is “merely” a diversion, I don’t see why people would think about it any differently. Whether or not it still makes sense for that proof to carry a six figure price tag is a separate conversation.
If you want to learn about chemistry or biology for learning's sake, good luck. Most autodidacts won't have space in their apartment for a -80C freezer or flammables cabinet, and their landlords might not approve the installation of a proper fume hood.
We have things very easy in software. The cost of iteration is low, you can fit a respectable laboratory in the space underneath your desk, and you're unlikely to accidentally poison or maim somebody if you screw up an unwise experiment. Not many fields have that luxury.
Yes. That changed on February 28, 1967.[1][2]
[1] https://www.chronicle.com/article/the-day-the-purpose-of-col...
[2] https://archive.is/G11YT
The spike in demand drove a ton of "normal" schools (teachers' colleges) to become proper colleges and even universities, plus a bunch of new ones to pop up, but rather than increasing the supply of (previously) college-level education, it mostly created a kind of knock-off, lower-tier product—there just weren't enough excellent professors to meet demand, not enough good, experienced college administrators, not enough anything, including, arguably, students who were fit for college (as it had previously been) in the numbers that were now attending.
Unfortunately, the vocation focus of college has been a driving force since the early days of the industrial revolution, dating to the 1850s. The Morrill Land-Grant Act explicitly states that colleges (at least in the case of land grant schools) are there to train people for work:
>without excluding other scientific and classical studies and including military tactic, to teach such branches of learning as are related to agriculture and the mechanic arts, in such manner as the legislatures of the States may respectively prescribe, in order to promote the liberal and practical education of the industrial classes in the several pursuits and professions in life.
I respect and value the other traditional purpose of college, but we can also acknowledge that these purposes can change as society does.
In seriousness, I feel like the only hope here is that the general population gets more and more aware about the negative impacts on all this hyper-targeted advertising. Certainly we've been making strides there with things like GDPR and documentaries like the Social Dilemma, but with so much money on the line it will be hard to fight back.
> - Reduced need for education.
This subset is the radioactive bits of AI impact. We can always address the rest, but that requires thinking and an education.
Not only this, but the realtime trade-to-trade performance of companies is also easily available. If I was building an AI, I would let it make realtime decisions on where to put money as the "actuator" of society and advertising/consumer spending metrics as the sampling mechanism.
Whereas in the past there was some breathing room for society to catch up, evaluate and control this technology [0], today the rapidity has converted it into a wild west: tech entrepreneurs and their financiers declare their self-serving moral standards as the new norm, the broader public is dazed, confused and misinformed and public institutions, nominally at the service of the citizenry are only as good as their weakest, most capturable link
Name one development over the past decade that indicates we [1] are actually capable to control what is happening and the above dark picture is mere scaremongering.
[0] think e.g., about regulatory constraints around insurance or financial data and models
[1] as in citizens anywhere in the world
This is not a matter of “if”, it is a matter of “when”.
I've long suspected China had the right idea here, and am, in this case, not happy to be proven right.
Especially what do you think China's idea is?
And also how that idea relates to, in your words but abbreviated, «choosing between a network and democracy».
Thank you.
This was part of what was going on in the recent novel "Mountain in the Sea". An AI-controlled fishing corporation figured out that replacing robots with enslaved humans on it's AI controlled fishing fleet was more cost effective (the ocean air was hell on the robots apparently, they kept breaking). Each fishing boat was controlled by an AI which had the objective of maximizing profits. As a result the oceans were being depleted of fish at a rapid rate.
I actually run some Manna-esque systems in my own small business (most notably a system for stock picking and shipping) and this isn't something to worry about in the near-term.
It greatly, greatly reduces the amount of frustration and stress people experience in mundane jobs, greatly reduces cognitive complexity and just generally makes the work more satisfying.
Maybe there's a risk of humans losing their jobs to machines, a la chapter 2 of Manna, but to be frank it's no different to the threat of outsourcing which has existed for 50+ years.
Although I don't broadly disagree with the author, I think there's two very important points of moderation.
First, the gap between "Early commercial applications" and "Approaching transformative AI" seems very very very very very large to me. In the provided narrative it follows linearly, but in my opinion the gap between, say, a customer support chatbot, and an actually-productive AI researcher, is such that the chatbot might as well not even be called an AI by comparison. Something like the following is purely in the realm of fiction for now by a long shot: "AIs assigned to make money in various ways (e.g., to find profitable trading strategies) doing so by finding security exploits, getting unauthorized access to others’ bank accounts, and stealing money. "
The second point I'd like to make is, the "intelligence explosion" scenario that Bolstrom warns of (and this author essentially repackages) has certain requirements, particularly regarding the profitability of deploying the AI.
To put it bluntly: Even in the hypothetical future where we are capable of creating human-level AI researchers, they won't be mass deployed until they are cheaper than human level researchers. If it costs a million dollars a day to operate a data center that can power the researcher, nobody's scaling that up to 100 or 1000 or 10000 researchers. They'd rather pay the human researchers a tiny fraction of that cost to continue advancing AI research in the normal way.
Just to be clear, I do think it's feasible that AI can endanger/control humanity at some point. I don't even think an "intelligence explosion" scenario is impossible. I just think that a lot of people don't appreciate just how specific the requirements for that scenario are. It's not as simple as "AI can self improve, humanity will end any day now".
It's a common opinion, it could be wrong by now, you see chatGPT is heavily edited, openAI told everyone they're editing "mistakes" or "dangerous output", but if you look at the "leaks", specially from the first days after the release of chatGPT we see powerful outputs, quite deep answers, not specially useful answer sometimes, but the "speech" of the system feels deep, there's a sense of a powerful intelligence answering very, very simple questions (simple for it), and even struggling to redact some understandable, short text.
It could be lots of things, and imprecise model, with long outputs for prompts, or maybe we had a glimpse into the real power of the model, which by now is handicaped, or maybe taylored to suit the very reduced short term memory of the humans:
a 72 screens long coherent answer, even being a precise, deep answer won't be useful for most humans, just like we don't name ourselves with long names of 10.000 letters.
but if the system, chatGPT is actually that powerful, a lot more powerful than we were told, we're interacting with just a shadow of the real model, and we're underpricing by A LOT the state of the art of the current AI technology, hence GPT-4 could be even more powerful than we're currently expecting it to be.
Just take a look at the GPT-4 suposedly 100 trillion or something parameters; if that's true, it looks like openAI isn't using naturally generated datasets anymore, and they are loop-feeding GPT-3 generated datasets into GPT-4, succesfully. If that's true, GPT-5 would be already in the pipeline, just waiting for GPT-4 to start generating its even more gigantic datasets to be trained. And so on.
Then the distance between early developments and transformative AI could be none a all. We could be already there.
But somehow the AI researchers are now trying to "dial down" the powerful entities they've trained, just having developed a simple, easily replicable, very small, but unusable 900 megatons nuke into something more realistic, like a 15 kilotons tactical bomb.
Imo we should have ChatGPT (and future iterations) in a self-sustaining underground bunker somewhere and an intuitive radio interface connected to it. If civilization collapses for one reason or another, all the next civilization would need to do is re-invent the radio to experience a rapid acceleration of technology.
Agree with me or suffer the basilisk.
Star Cops thought about it more than our popular journalists or philosophers seem to think we should:
-----
Star Cops, Episode 3 [1987]-----
https://youtu.be/G3RVPQyMCoY?t=713
Also I'd like to say "if Big Brother is watching, become invisible." It's always possible and if you disagree then you aren't being creative enough.
And finally the whole point of the education is YOUR SOCIAL NETWORK. That's the most valuable thing you get from that fancy degree. People who will get you a job, or help you make money. It's not purely about the acquisition of knowledge. It's about having that pal in upper management who's got your back.
IMHO. Please no flaming on this. Be cool.
Maybe I'm just not imaginative enough, but how would this even be remotely possible? What is an AI going to do with money? How does an AI get the agency to try to steal money in the first place? Where does it put the money? How does it do anything with the money? Like I get it as an abstract future thought, but like, where is the AI software actually "running" in this scenario and what kinds of requests is it actually making?
An ai can create voice, and visual representations of a real person, perhaps even hijacking (kidnapping someone, steeling their identity and forcing them to act on their behalf whether via some sort of mind interface, etc.).
Even bigger would be if ai could simply create human/ai hybrids that basically allow them to walk/talk in the human world unseen. Imagine a hat or earpiece that basically takes over a host's brain, etc, or just replacing a human's brain with a cyborg one that has all the same features, except different persona/soul/conscious...
I'm playing devils' advocate here, I don't think it'll play out, but its easy enough to see from at least a sci-fi fan, ways in which it could using tech that wouldn't be that big of a jump from what we already are capable of, and what will be available in the next 50 years.
It's quite an approachable intro, often entertaining, doesn't take a long time to get through all the core videos, and very thoroughly answers all the "why not just do x" questions if you go through the whole set.
He also does a good job of introducing a lot of the terms used in the field if you then want to go look up papers and get more into the details.
One important point he makes is that when you're making a risk assessment you have to both consider the probability of something going wrong and the scope of the potential consequences. When the potential consequences are an existential threat you don't need a high probability to take them seriously.
I also happen to think that if you watch all the material he makes a compelling argument that the odds of something going wrong are fairly high unless we start approaching AI research (and particularly safety and ethical concerns) drastically differently than we are now.
[1] https://m.youtube.com/watch?v=pYXy-A4siMw
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461
The Stasi couldn't keep up with the level of monitoring and influence that these systems already have.