(Added: People really shouldn't flag you for asking a question.)
The point is "aligning the AI's goals with human interests", i.e. make the AI non-evil.
You're somewhat lucky to never have gone down that particular rabbit hole. Some strange, but also seemingly very intelligent people on the internet argue that it's the most significant extinction risk humanity faces. My impression is that actual AI/ML researchers believe this to be mostly bullshit. (I'm personally not qualified to decide, just explaining the positions.)
The author of this curriculum seems to define it as:
> Within the coming decades, artificial general intelligence (AGI) may surpass human capabilities at a wide
range of important tasks. We outline a case for expecting that, without substantial effort to prevent it, AGIs
could learn to pursue goals which are undesirable (i.e. misaligned) from a human perspective. We argue that
if AGIs are trained in ways similar to today’s most capable models, they could learn to act deceptively to
receive higher reward, learn internally-represented goals which generalize beyond their training distributions,
and pursue those goals using power-seeking strategies. We outline how the deployment of misaligned AGIs
might irreversibly undermine human control over the world, and briefly review research directions aimed at
preventing this outcome.
It's a "well defined" term in the crackpot side of AI, represented by lesswrong et al, who are self appointed AI experts who have never published any actual peer reviewed research in AI.
And no peer-reviewed research, apart from the dozens of widely-cited academic papers on this? All of these are just from one lab, not even including groups like CHAI:
https://www.anthropic.com/#papers
> And no peer-reviewed research, apart from the dozens of widely-cited academic papers on this? All of these are just from one lab, not even including groups like CHAI: https://www.anthropic.com/#papers
Which of these papers track AGI Safety and are also published in peer reviewed journals like SIGKDD, NeurIPS, AAAI etc?
OpenAI mentions AI Alignment and they are currently at the forefront of pushing AI on the world so it is not just that community, though I wouldn't be surprised if many employees at OpenAI participate there.
Open AI is one of the actual research organizations promoting AI Safety work. However, this has more to do with funding and the money put up front by the original founders - Elon Musk, Sam Altman etc, who themselves are not AI researchers, yet AGI enthusiasts.
Open AI published a number of papers, essentially and implicitly supporting the position of the AI inexperienced founders. They try to highlight AI safety aspects quite frequently. But I would not take it too seriously, just like you wouldn't take a Facebook employees endorsement of Zucks vision.
If you are familiar with ML, you would know that L2 norms are a key optimization metric in ML. L2 norms are basically Pythagorean hypotenuse computations in high dimensions. I will take AI safety research seriously when they start discussing the risk factors associated with Pythagoras theorem and how we as a society must deal with it before we become a slave to L2 norm driven super intelligence.
If you want to really talk about human risk from AI, Tesla's self driving killing machines are already here along with ML guided drone bombers. It's no different from Therac 25. If we are not discussing these and creating a course about super intelligence, it's just mental masturbation.
Thanks for the response. My hope for AI Alignment studies is that it would be something rigorous and theoretical and be testable in current AI safety issues. Much like studies of abstract algorithms can be compared and contrast with the runtime of actual algorithms.
What I have seen in LessWrong is lots extrapolations upon extrapolations with worse case scenarios just to say there is nothing we can do. That seems to take away from any serious research that might exist since they are probably the best known community associated with AI Alignment.
Not shocked to see that module 0, machine learning, is 1. Optional and 2. Just a few YouTube videos.
It seems like most people I see billing themselves as an “AGI researcher” or discussing alignment on Twitter (or irl in SF) have no actual background in AI/ML.
“The creator of the course was a research engineer at DeepMind and now works at OpenAI. Several people at those orgs have acted as facilitators for the course since its creation.”
The course is an intro to AI risk. Nobody acts like an AGI safety researcher when they finish the course with no ML experience. The majority keep skilling up for a while, which includes replicating ML papers.
Me personally, I started doing ML in 2015 and got into AI alignment in early 2022.
I'm not a historian, but I think that if in 1900, when there were no successful airplanes, if you tried to have serious dialogues on how to design safe air traffic control systems, you would have been laughed at. Maybe you'd be able to anticipate some real problems, but some of your focus would be wildly off. And to the extent that you could see real issues coming, it would be heavily informed by what you could deduce about planes and how they would be used. And those skills would probably have been better directed at making trains or cars safer.
So I'm always a little surprised how seriously people take AI safety and alignment concerns about systems we don't yet know how to build, and how uninterested many of those same people are about 'alignment' issues with organizations and institutions we have today. And given our poor success in addressing our current issues, even where good policy proposals exist, why should we believe that technical solutions for AI alignment would be used even if they're researched and well-described?
You're right - the odds of success are incredibly slim. But we may only have one shot, because the first mid-air collision in this analogy could literally be the end of the world.
How exactly? Explain how an AI would cause the end of the world? Are you suggesting we would turn over all of the world's nuclear arsenals to AI to deal with? Maybe it's just me but lately it seems like everything is being labeled as "dangerous" to the point of absurdity. It seems to be following the same line as US political rhetoric where everyone is either a Nazi or communist bent on destroying the country depending on what side you generally align with.
And the monkeys thought, “how could a human be dangerous?” “Would they clobber us with stones? Surely we are stronger!”
The problem is that the toolset available to someone who is much more intellectually capable is beyond what we can think of.
People are not afraid of AGI because it will behave like a very smart human, people are afraid of AGI because the capability gap will be more like the one we experience between humans and other animals.
The toolset available to humans when dealing with monkeys is literally incomprehensible to the monkeys.
The toolset available to AGI is similarly incomprehensible to humans.
Part of the issue is that taking AI alignment seriously does require some level of intellectual humility — a quality that the HN comment section famously lacks.
This has been written about in numerous places. There are multiple possible ways an AI might go about this if it saw that as its task; the probability of any one specific method being is of course lower than the total probability of the whole set. So any one method would be an unlikely and speculative scenario. The method in question could range from nuclear, chemical, biological, sabotaging agriculture, mass-producing CFCs or other pollutants, triggering wars, or other unforeseen approaches. Most scenarios allow that (a) the AGI is very smart, deceptive, creative, and resourceful, and can pose as a human or corporation to execute transactions; (b) the AGI is able to gain control over some means of funding, either legitimately or illegitimately, and thereby pay unsuspecting humans to perform seemingly-innocuous tasks like protein synthesis or package delivery; (c) you wouldn't see it coming, any more than you see the chessmate approaching several moves ahead, because the AGI would be appearing to be friendly and helpful along the way, and perhaps earning you lots of money, while it is secretly outsmarting you for its own ends.
For a nuclear approach, the AI would only have to hijack the least-hackproof of US, Russian, or Chinese arsenals in order to trigger an exchange from all sides. But it would probably opt for a different method that would do less collateral damage to its own resources.
This has been an issue raised since at least the early 2010s if not before, and so (arguably) predates the most recent round of US political polarization. The core arguments are unchanged, but became more urgent as AIs broke through several milestones thought to be decades out, such as defeating top human Go players, cracking the protein folding problem, and passing the Turing test with flying colors.
Forget the idea of an "AI" then, because the idea of "intelligence" makes the argument harder. Just think of a "new technology."
Is it possible that a new technology could destroy the world? Of course. It could've turned out that nuclear weapons would incinerate the atmosphere upon detonation, as some were worried they would. It could be that the next technological innovation will kill us, there's nothing prevent it in the laws of physics.
AGI is a specific technology we are worried about, because the whole premise is "once we build something that is extremely capable at a variety of things, one thing it will be capable of is destroying the world. Even by accident."
We're already using AI techniques to help with problems in biology like protein folding. Take it a few dozen iterations forward, and these systems will be helping design medicines and vaccines that no human can do by themselves. At that point, what's to stop the system from creating a super-flu that kills everyone? Forget about intent here, how about a bug?
ChatGPT often misunderstands queries, take something like ChatGPT but 100x more capable, do you really think people won't be using it to do things? And given that they will, it could easily have a bug that "oops, incinerates the atmosphere" as a side effect.
I don't think they are taken that seriously. They are writing their op-eds and fiery papers, but the field marches forward nevertheless. But I agree that the whole fields seems like ivory-tower types trying to hijack the latest hottest technology an become the decision makers based on their "exprertise". AI as a field always attracted all kinds of weird individuals, probably because it is such a popular and influential concept. If we would work with their ethics, we would never allow computers to reach general population, because it is simply too dangerous and a lot of "harm" can be done. Their use will be heavily monitored and a license or an assessment would be required to use some of the functions, e.g., connect to the Internet.
I'm amazed to see how common this view is on HN. You'd think the rapid progress made in AI would give more validity and attention to these concerns raised about risk and alignment.
The people doing this work are mediocre at best, if they could push the state of the art they'd being do that. It's a bunch of second handers trying to control those who actually understand what is going on.
Yes. The AGI teacher is a recent graduate with a Masters in physics. None of the material in this so called "course" is helpful with actual AI research or industry work.
AI "alignment" is a made up field ride with crackpots and charlatans.
The creator of the course was a research engineer at DeepMind and now works at OpenAI. Several people at those orgs have acted as facilitators for the course since its creation.
The purpose of the course is to give people an introduction to AI risk so that they can start getting into the field. Some who take it have been ML engineers for a while and some are just learning about the field.
The expectation was never to consider yourself a full-fledged alignment researcher once you finish this course, but instead to get people started. Many people will spend a bunch of time doing things like implementing GPT from scratch and replicate papers after this to try to get an internship at one of the big orgs. And then do other things to slowly make their way to producing great research. Sometimes ya’ll just want to hate on this site.
If you like to pay for an AI ML course being taught by someone whose entire ML education is 6 months of independent self teaching (as per his own LinkedIn) go ahead, be my guest.
The course is free and funded by Open Philanthropy.
It's not taught by James solely, but there's weekly discussion groups on readings chosen by a mix of alums from OpenAI and Oxford's Future Of Humanity Institute.
I mean, I'm not sure I'd require people who study and advocate for nuclear disarmament to also be publishing papers on how to improve bomb yields in order to treat them as credible. I'd probably expect them to study game theory and geopolitics.
"None of the material in this nuclear disarmament course even covers how to build a modern hypersonic missile!"
Don't be silly. Computer proliferation is not an existential risk.
—
There is exactly one correct analogue to AGI risk in our history, and that is the risk of an all-out nuclear exchange. And surprise surprise, we as a species took that risk pretty damn seriously once we realized what the stakes were. Do you think that the considerable efforts to avoid nuclear proliferation, indeed even the use of a single low-yield tactical nuclear warhead in anger, are misguided?
Nuclear risk is a contained problem. Access is extremely limited and controlled by low tech hardware locks. The technology is understood, we know what makes bombs explode and it is predictable by physics.
AI is the opposite of all these characteristics. Its access is nearly everyone. The technology is not understood and not predictable. It is a nuclear warhead that thinks for which we don't know what might make it explode.
Yes, although I wouldn't go as far as to claim that current AI technology has any sort of "accessible to all" path to AGI. Accessible to irresponsible actors with lots of money? Possibly, and that's the real risk. But at least it's not an absurd comparison unlike the GP's.
Maybe think about the invention of nuclear weapons/energy rather than airplanes since that matches the type of concern. You can just choose not to take an airplane.
If there was no manhattan project and work towards that goal had been done in the open over a long period of time, I don't think people with safety concerns would be laughed out of the building.
In fact I think we got kinda lucky that it turns out to be so expensive to build nukes that only nation states can do it, and that the barriers are still prohibitive even then such that we can tell if a country is trying to gain that ability. I'm not sure we could have ruled out the possibility of individuals or small groups being in control of nukes.
>In fact I think we got kinda lucky that it turns out to be so expensive to build nukes that only nation states can do it, and that the barriers are still prohibitive even then such that we can tell if a country is trying to gain that ability. I'm not sure we could have ruled out the possibility of individuals or small groups being in control of nukes.
This is something I've also realized. Nuclear weapons are undeniably an existential threat but they're in part mitigated because it's very very difficult to get the materials and assemble them. That was complete luck. We got lucky. That's all that saved(es) us.
Imagine if making a nuke was as easy as making a steel sword. Or perhaps, as easy as running Make-Me-Global-Dictator.exe
> but I think that if in 1900, ... serious dialogues on how to design safe air traffic control systems, you would have been laughed at.
In 1900 there were massive rail stations and harbors. Just say it has to safely receive new train/ship every 10 minutes, and that very well describes the problem.
> better directed at making trains or cars safer.
In Greece there was train collision recently, somehow passenger and cargo train were on the same track. I doubt any advanced "lessons" were applied compared to 100 years earlier.
The Base Assumptions undergirding all AI safety research is that:
1. AI systems respond to incentives, not commands, including incentives you did not intend to be there.
2. The technology will eventually outsmart its creators, making the imposition of reactionary controls difficult or impossible.
#1 was a good bet, and has already been proven with existing ML systems. In fact, it's one of the most common training blunders you can do and there are countless examples of artificial stupidity coming about from poorly thought-out incentives. AI are ruthless optimizers that will find the loopholes in your systems and maximally exploit them[0].
#2 implies significant harm if we get it wrong (and according to #1, we will always get it wrong). It is very plausible that AI will exceed human capability in at least some tasks, since there are already problem domains where AI dominates (e.g. chess, Go, etc).
My main objection to #2 is that the idea of a new structure or machine that exceeds human intelligence is not novel. Humans evolved to live in small tribes of no more than 100 people and we have not exceeded that limitation. So we're already several nested singularities deep[1] and humanity has not yet been destroyed.
On the other hand, humanity itself is a singularity: the system that created us (genetic evolution) is not capable of effectively controlling us anymore. We control it, and have been doing so since the dawn of agriculture. And we can see the harms that we have done to nature in pursuit of our own goals. So it's not irrational to extrapolate and see how an AI with more intelligence than us can do more harm to us.
[0] So are human children, but they do not have the I/O capability of a data center full of A100s.
[1] Agriculture beget civilization, which beget nation-states, which beget corporations, which beget computers. Each one is a singularity unto it's creators.
1 is just plain wrong, AI models optimize human designed, mathematically defined loss functions, not arbitrary handwave-y incentives. We know exactly what these systems are trained to do.
One of the main reasons why AI alignment is a hard problem is that it's not easy to predict what behavior will result from optimization applied to a known loss function. (An incentive is just the gradient of the loss function, btw).
Yes, but when there are many degrees of freedom, that minimization might not happen in the way you expected, and if your loss function was only an approximation to your utility (because you weren't sure how to precisely codify "goodness"), the difference between the behavior you hoped for and the behavior you actually get might be very bad. For example, you attempt train an AI to get good behavior via a penalty button and reward button; an approximately-optimal strategy might include shooting the researcher to prevent him from pressing the penalty button, and then mashing the reward button itself.
> and according to #1, we will always get it wrong
This is a point I've made. We are playing with paradoxes that proponents tend to ignore. They won't be solvable. The more sophisticated AI becomes, the more non-deterministic behavior arises. The emergent deception has been very interesting.
We do have "successful" (but primitive) AI systems already, at least of the kind that we consider difficult to align (value optimizers / maximizers), which have a considerable impact of the world. An example would be social media recommendation engines.
We like to perscribe malice to the people spreading false news, but we forget that ultimately its often the recommendation engine that decides which things to highlight and which not to.
If you give an AI a value function that measures engagement perfectly well, and you get a recommendation algorithm that spreads false news and outrage inducing posts, the algorithm did exactly what you asked from it. The problem is that the optimizer didn't optimize for other implicit values you had in mind. (This is one of the key problems discussed in AI alignment)
Of course, this would be strongly contested by proponents of free speech who would say the AI gave us exactly what we wanted. But... did it? Or did the optimizer find a behavior that successfully manipulates people without taking into account the side-effect of radicalization of online discourse? Is that what we really want?
So, I'd say misaligned primitive AI might be already causing serious trouble in the world.
Take AI safety and alignment concerns seriously? Industry actors are at best paying lip service to AI alignment. Currently we're firmly stuck on a timeline where permanent and drastic reduction of human potential has a much higher probability than I’m comfortable with. What do you think is an acceptable risk of human extinction in the next 50 years? 0.1%? 1%? 10%?
To be clear, I would claim some of the "obvious" cases where misalignment of existing organizations / institutions and society at large, and where there have been reasonable policy fixes advocated for years include:
- unpenalized externalities, including carbon emissions, and other environmental effects
- clearly broken tax law which allows some enormously profitable companies to pay very low tax rates
- political donations protected as "speech"
- partisan gerrymandering
These are cases where we made systems with rules, in which agents have incentives, and where the combination of the design of the system and the incentives on participants can cause meaningful harm to the public. In many cases, transparency is meaningfully hindered, and "explanations" of particular outcomes are obscured. Several of the structural issues that AI alignment people care about are present, but the harms are already happening, and yet we seem unable (and AI alignment folk seem comparatively uninterested) in addressing them.
I don't think AI alignment folk are uninterested in addressing them. I think they do see as a red flag the fact that it's so hard even for corporations to craft incentive schemes that consistently reward good behavior without being hacked and gamed for bad results, and that's when the incentives are being followed humans who we have a lot of experience with.
> So I'm always a little surprised how seriously people take AI safety and alignment concerns about systems we don't yet know how to build
Aside from Basilisk cultists and people play-acting concern for safety and alignment as a PR tactic to push centralized control of AI in support of their own financial self interests (groups which have more overlap than one might expect), the main (non-academic) safety and alignment concerns are more about the kind of systems we do know how to build, and which are seeing increasing deployment deployment demonstrable severe biases that have significant impacts in application and other issues.
I have recently written a paper on understanding machine learning via the lens of Hopf algebra https://arxiv.org/abs/2302.01834.
Hopf algebras (which are really just tensors with recurrence relations built in) subsume convnets, transformers and diffusion model and also provide a theoretically better autodiff that operates within single layers as opposed to across entire graphs.
Furthermore, there is a correspondence between Hopf algebra and linear logic and Hopf algebras are related to zonotopes, which are polyhedra that have been used in verified numerical computation. I'm strongly convinced the LL connection can provide proofs over zonotopes which paves the way towards interpretable AI and will be central for alignment.
I'm working on a next gen Hopf algebra based machine learning framework.
Very interesting, I also have recently come to think that higher-order logic and its relationship to type systems and verification are the way to go for alignment.
I have attempted to ask many alignment proponents a question on the basic assumption of the feasibility of alignment, but so far have not had any respond directly to what I perceive as a paradox.
That being we must accept that a low IQ entity could design an unescapable containment for a high IQ entity which was built for the very purpose of solving imperceptible problems of the low IQ entity.
Confining and using an agent who's smarter than you, maybe much smarter, is indeed crazy hard. I've never seen an "alignment proponent" who didn't already agree and consider it a terrible idea. What other answer do you want from them? The OP is about trying to figure out some other strategy to develop very smart machines.
(I read the last paragraphs at your link to see if I'm misunderstanding you.)
- Creating an AGI whose goals are compatible with ours.
- Creating an AGI with inscrutable goals and safely confining it.
They're not the same!
In whoever you've been talking to I can't recognize the rationalist scene I've known since it grew out of the 90s extropians. "AI boxing" was a live argument around the early 2000s. Not now.
Online you often see people in that community saying that slower progress in AI capabilities would be a very good thing. At the same time, yes, they do want us to achieve AGI/ASI as soon as possible, provided we don't screw it up.
> At the same time, yes, they do want us to achieve AGI/ASI as soon as possible, provided we don't screw it up.
That's the point, it is not possible. The premise is based on a paradox and you can't throwaway containment and have alignment that remains defined by original intent.
If you want to argue that it's logically impossible for a smarter-than-human intelligence to be created with values aligned with human ones, then cool, do that. (Note that a smart human on much faster hardware is smarter-than-human. This is not a claim that AGIs must be mere faster humans; it's one example to point out a difficulty for any such argument.)
But I think you gave a false impression of others at the start of this thread.
> If you want to argue that it's logically impossible for a smarter-than-human intelligence to be created with values aligned with human ones, then cool, do that.
That is precisely what I have stated.
> But I think you gave a false impression of others at the start of this thread.
Your question and larger post don't present anything new to me, and I'm just a casual reader of the idea space (but I've been around since SL4). Some brief thoughts:
1. Congrats on recognizing that it's a hard (perhaps impossibly so, at least with the time left) problem! Too many (not any of the alignment people) have thought it's a trivial matter, or that Friendliness (the old word) will just "emerge", or that somehow being "superintelligent" will cause it to just smoke weed and play games all day like some super smart dorm room friend so there's nothing to worry about, or...
2. The idea of Friendliness or Aligned values is a difficult one purely from definitions alone, yes, let alone implementation, so that's part of what makes it such a hard problem. The idea of an AI tiling the solar system with smiley faces was once a response to a serious suggestion that one can just train the AI to optimize "human happiness". Imagine trying to formally model something like "If AI-Gandhi was offered a pill that made AI-Gandhi 2x smarter but also want to destroy the world, we know AI-Gandhi would not take such a pill because [...]" What's the [...]? A proof about its values? For actual Gandhi, we might as humans say "because he's Gandhi, and Gandhi does not want to destroy the world, and does not want to want to destroy the world". On the topic of the submission, I hope that any who take the course can by the end demonstrate many issues with things like the old and obsolete Coherent Extrapolated Volition proposal.
3. Containment is of course folly, you can look up the old AI Box arguments if you wanted more on that from back in the day. Hence the focus on Friendly (and more modernly, Aligned) AI.
4. I, an approximately +1 std dev IQ human, fully expect that it'd be no challenge for a -1 std dev IQ human to dig a pitfall trap that could not only contain me but kill me. Because gravity still applies even if I'm smarter than my adversary. Sure, an AGI will figure things out about physics that we don't know, and math too, but physics will still apply, I would bet that even a superintelligent AGI will never find a way to break the speed of light and travel the universe. But perhaps we're very wrong about special relativity. We'd have to be very wrong about everything if our proofs from first order logic are somehow awry. But that too highlights the level of challenge to the problem: we have to prove the AI is Friendly, with the reliability of first order logic. As a matter of fact humans can do things perfectly, your CPU is executing billions of instructions perfectly every second. (Or to be fair, there are errors, but those are modeled accurately and through correction and redundancy (with multiple systems for things that go into space) we can drive the probability of a total failure as low as we please.)
> Congrats on recognizing that it's a hard (perhaps impossibly so, at least with the time left) problem!
If that was your takeaway, then you missed the point. It was not an argument that it was a hard problem. It is precisely the argument that it is impossible as paradoxes are not solvable.
> As a matter of fact humans can do things perfectly, your CPU is executing billions of instructions perfectly every second
Yes, if you constrain the context small enough you get get the result you want. However, the concept of a ASI with free agency expands the context to a scope that we can't even reason about.
Your argument that it is actually impossible is not convincing, I think my example of a pitfall trap in 4 is sufficient to demonstrate that. More precisely, if we grant that any AGI no matter how intelligent and agent-y is still subject to physics and logic, it doesn't seem to be an actually impossible problem, just very hard, and maybe not solvable before time runs out. If we don't grant physics and logic, very well, but Fermi considerations raise a puzzle about why we're even here to ask the question.
Similarly I'm not convinced that the scope is broad enough that it really cannot be reasoned about at all, either in its entirety or by breaking it up into smaller components (as we do for large problems already). I think you'd have to give some sort of impossibility proof on what about the problem is possible to reason about and what is not possible to reason about to convince me. (Note: chaos theory won't necessarily help you here, because probabilistic reasoning exists as well as do() operations by us, and if it must be used the argument must explain why certain catastrophic errors cannot be modeled and driven to arbitrarily low values.)
> I think my example of a pitfall trap in 4 is sufficient to demonstrate that
It is an isolated narrow context. The problem with ASI is there is no such limitations. I addressed this point in the post, but another example would be the Kobayashi Maru Test.
Or rather, there has been no system devised that humans have not found a vulnerability. Why would AGI be any different? Alignment/containment must also address the possibility of human exploit of the system as well. So we are back at the beginning in either case. Which means a human could still devise a trap for the AI to leverage its ability or modify its alignment.
> Similarly I'm not convinced that the scope is broad enough that it really cannot be reasoned about at all, either in its entirety or by breaking it up into smaller components
We are already having challenges in this area. Example...
“The size and complexity of deep learning models, particularly language models, have increased to the point where even the creators have difficulty comprehending why their models make specific predictions. This lack of interpretability is a major concern, particularly in situations where individuals want to understand the reasoning behind a model’s output”
"ASI" is still limited by physics and logic. Or if not, we're just more screwed, maybe. I suppose we're in agreement that in the long run (maybe longer than we think with some really cool stuff like Ems in between...), the likely scenario is we're screwed one way or another? I'm just glad people are working on trying to discover the chains of logic that let us not be so screwed, hard as that problem may be. I don't see any paradox there that makes the problem impossible, logic is logic independent of intelligence. Similarly, I'm very sure that no one else and no superintelligent AGI can ever reproduce the original string I used to produce the sha512 hash of a5136df53e73c4fdcde80d9ba41f2526c6e144e344c85a355f518b161f5b5171a6de831908ec32f6a6e503ae055219027d8e9890517977bbcd16a6cd9f36c858 (perhaps they'll find a collision). (Incidentally there are many secure systems and schemes that have not fallen. But you'd probably write them off as too narrow for some reason.)
Yes, there's "difficulty comprehending" current deep learning models, put modestly. There's also interesting work in interpretability/explainability going on to work on that. Nothing there says "impossible". However, even granting that it is actually impossible to explain and reason about current AI methods to the degree we need to in order to create chains of logic for alignment purposes, that just says that it's impossible to reason about an AGI produced using such methods, if it's even possible to produce an AGI that way. It says nothing about it being impossible to reason about an AGI produced using different methods.
> I'm just glad people are working on trying to discover the chains of logic that let us not be so screwed, hard as that problem may be. I don't see any paradox there that makes the problem impossible,
I understand you don't perceive the paradox. Nonetheless, my point is rather that when a paradox presents an impossible problem, then something about our premise is not understood. Or we need to find an entirely different concept to solve our problems. For example, an entirely different concept of intelligence not modelled strictly on how we perceive our brains. In summary, I think we are trying to solve the wrong problem. I might write something more depth at some point on that to explain my views.
> Incidentally there are many secure systems and schemes that have not fallen. But you'd probably write them off as too narrow for some reason.
This is precisely my point. We can build what is logically infallible to only be subverted by different means. If you can't hack the system, trick a human to give you access. This is how every such system has been defeated.
I think you’re a bit confused. Alignment researchers are the ones saying it will likely be impossible to box a superintelligent AI and that this is one small part as to why this whole AI alignment thing is hard in the first place. They are precisely working on the problem because there seems to be no easy solution like boxing the AI. And so they need to find a way to make sure the AI is aligned with our values even if the AI has the capability of escaping any box we put it in. In reality, companies will not even try to box the AI so it’s probably irrelevant.
If have seen it stated that containment is the insurance for alignment since there is no way to guarantee alignment will either remain in the form we set or will deviate from the path we perceive.
Which is part of the same argument, in that if the goal is AI safety that does not have free agency to do harm either directly or as an indirect unforeseen behavior, then it is an impossibility without containment and if containment is impossible so is alignment.
How could we expect it to be otherwise? At least if you are going to model the intelligence based on human experience and how we process the world then we have to assume just like a human mind can become unaligned so could ASI.
Containment is generally just a thing that people say we should at least try to do to buy time, but there’s very little discussion beyond that. No alignment researcher is arguing that containment solves our problems. The solution to alignment needs to work even if we don’t box the AI. The point is that, yes, alignment is indeed hard.
> The point is that, yes, alignment is indeed hard.
An aligned AI is essentially a soft boxed AI. It is not that different in concept. The end result is we want a set of behaviors as outcomes and other behaviors to not be present.
Complex problems are hard. Paradoxes are not solvable. That is the distinct difference I'm trying to make.
82 comments
[ 3.2 ms ] story [ 121 ms ] threadThe point is "aligning the AI's goals with human interests", i.e. make the AI non-evil.
You're somewhat lucky to never have gone down that particular rabbit hole. Some strange, but also seemingly very intelligent people on the internet argue that it's the most significant extinction risk humanity faces. My impression is that actual AI/ML researchers believe this to be mostly bullshit. (I'm personally not qualified to decide, just explaining the positions.)
The author of this curriculum seems to define it as:
> Within the coming decades, artificial general intelligence (AGI) may surpass human capabilities at a wide range of important tasks. We outline a case for expecting that, without substantial effort to prevent it, AGIs could learn to pursue goals which are undesirable (i.e. misaligned) from a human perspective. We argue that if AGIs are trained in ways similar to today’s most capable models, they could learn to act deceptively to receive higher reward, learn internally-represented goals which generalize beyond their training distributions, and pursue those goals using power-seeking strategies. We outline how the deployment of misaligned AGIs might irreversibly undermine human control over the world, and briefly review research directions aimed at preventing this outcome.
This is from here, which I haven't read, but found three links deep on the posted site: https://arxiv.org/abs/2209.00626
And no peer-reviewed research, apart from the dozens of widely-cited academic papers on this? All of these are just from one lab, not even including groups like CHAI: https://www.anthropic.com/#papers
Which of these papers track AGI Safety and are also published in peer reviewed journals like SIGKDD, NeurIPS, AAAI etc?
https://openai.com/blog/our-approach-to-alignment-research
Open AI published a number of papers, essentially and implicitly supporting the position of the AI inexperienced founders. They try to highlight AI safety aspects quite frequently. But I would not take it too seriously, just like you wouldn't take a Facebook employees endorsement of Zucks vision.
If you are familiar with ML, you would know that L2 norms are a key optimization metric in ML. L2 norms are basically Pythagorean hypotenuse computations in high dimensions. I will take AI safety research seriously when they start discussing the risk factors associated with Pythagoras theorem and how we as a society must deal with it before we become a slave to L2 norm driven super intelligence.
If you want to really talk about human risk from AI, Tesla's self driving killing machines are already here along with ML guided drone bombers. It's no different from Therac 25. If we are not discussing these and creating a course about super intelligence, it's just mental masturbation.
What I have seen in LessWrong is lots extrapolations upon extrapolations with worse case scenarios just to say there is nothing we can do. That seems to take away from any serious research that might exist since they are probably the best known community associated with AI Alignment.
It seems like most people I see billing themselves as an “AGI researcher” or discussing alignment on Twitter (or irl in SF) have no actual background in AI/ML.
“The creator of the course was a research engineer at DeepMind and now works at OpenAI. Several people at those orgs have acted as facilitators for the course since its creation.”
The course is an intro to AI risk. Nobody acts like an AGI safety researcher when they finish the course with no ML experience. The majority keep skilling up for a while, which includes replicating ML papers.
Me personally, I started doing ML in 2015 and got into AI alignment in early 2022.
So I'm always a little surprised how seriously people take AI safety and alignment concerns about systems we don't yet know how to build, and how uninterested many of those same people are about 'alignment' issues with organizations and institutions we have today. And given our poor success in addressing our current issues, even where good policy proposals exist, why should we believe that technical solutions for AI alignment would be used even if they're researched and well-described?
The problem is that the toolset available to someone who is much more intellectually capable is beyond what we can think of.
People are not afraid of AGI because it will behave like a very smart human, people are afraid of AGI because the capability gap will be more like the one we experience between humans and other animals.
The toolset available to humans when dealing with monkeys is literally incomprehensible to the monkeys.
The toolset available to AGI is similarly incomprehensible to humans.
For a nuclear approach, the AI would only have to hijack the least-hackproof of US, Russian, or Chinese arsenals in order to trigger an exchange from all sides. But it would probably opt for a different method that would do less collateral damage to its own resources.
This has been an issue raised since at least the early 2010s if not before, and so (arguably) predates the most recent round of US political polarization. The core arguments are unchanged, but became more urgent as AIs broke through several milestones thought to be decades out, such as defeating top human Go players, cracking the protein folding problem, and passing the Turing test with flying colors.
Is it possible that a new technology could destroy the world? Of course. It could've turned out that nuclear weapons would incinerate the atmosphere upon detonation, as some were worried they would. It could be that the next technological innovation will kill us, there's nothing prevent it in the laws of physics.
AGI is a specific technology we are worried about, because the whole premise is "once we build something that is extremely capable at a variety of things, one thing it will be capable of is destroying the world. Even by accident."
We're already using AI techniques to help with problems in biology like protein folding. Take it a few dozen iterations forward, and these systems will be helping design medicines and vaccines that no human can do by themselves. At that point, what's to stop the system from creating a super-flu that kills everyone? Forget about intent here, how about a bug?
ChatGPT often misunderstands queries, take something like ChatGPT but 100x more capable, do you really think people won't be using it to do things? And given that they will, it could easily have a bug that "oops, incinerates the atmosphere" as a side effect.
AI "alignment" is a made up field ride with crackpots and charlatans.
The purpose of the course is to give people an introduction to AI risk so that they can start getting into the field. Some who take it have been ML engineers for a while and some are just learning about the field.
The expectation was never to consider yourself a full-fledged alignment researcher once you finish this course, but instead to get people started. Many people will spend a bunch of time doing things like implementing GPT from scratch and replicate papers after this to try to get an internship at one of the big orgs. And then do other things to slowly make their way to producing great research. Sometimes ya’ll just want to hate on this site.
https://www.linkedin.com/mwlite/profile/in/jamie-bernardi/
The researcher who has apparently created the syllabus is also a fresh PhD with 3 years of work experience.
It's not taught by James solely, but there's weekly discussion groups on readings chosen by a mix of alums from OpenAI and Oxford's Future Of Humanity Institute.
"None of the material in this nuclear disarmament course even covers how to build a modern hypersonic missile!"
—
There is exactly one correct analogue to AGI risk in our history, and that is the risk of an all-out nuclear exchange. And surprise surprise, we as a species took that risk pretty damn seriously once we realized what the stakes were. Do you think that the considerable efforts to avoid nuclear proliferation, indeed even the use of a single low-yield tactical nuclear warhead in anger, are misguided?
AI is the opposite of all these characteristics. Its access is nearly everyone. The technology is not understood and not predictable. It is a nuclear warhead that thinks for which we don't know what might make it explode.
AI is created and run on computers so AGI risk must by definition be a subset of computer proliferation risk
If there was no manhattan project and work towards that goal had been done in the open over a long period of time, I don't think people with safety concerns would be laughed out of the building.
In fact I think we got kinda lucky that it turns out to be so expensive to build nukes that only nation states can do it, and that the barriers are still prohibitive even then such that we can tell if a country is trying to gain that ability. I'm not sure we could have ruled out the possibility of individuals or small groups being in control of nukes.
This is something I've also realized. Nuclear weapons are undeniably an existential threat but they're in part mitigated because it's very very difficult to get the materials and assemble them. That was complete luck. We got lucky. That's all that saved(es) us.
Imagine if making a nuke was as easy as making a steel sword. Or perhaps, as easy as running Make-Me-Global-Dictator.exe
In 1900 there were massive rail stations and harbors. Just say it has to safely receive new train/ship every 10 minutes, and that very well describes the problem.
> better directed at making trains or cars safer.
In Greece there was train collision recently, somehow passenger and cargo train were on the same track. I doubt any advanced "lessons" were applied compared to 100 years earlier.
Most people are ignorant.
1. AI systems respond to incentives, not commands, including incentives you did not intend to be there.
2. The technology will eventually outsmart its creators, making the imposition of reactionary controls difficult or impossible.
#1 was a good bet, and has already been proven with existing ML systems. In fact, it's one of the most common training blunders you can do and there are countless examples of artificial stupidity coming about from poorly thought-out incentives. AI are ruthless optimizers that will find the loopholes in your systems and maximally exploit them[0].
#2 implies significant harm if we get it wrong (and according to #1, we will always get it wrong). It is very plausible that AI will exceed human capability in at least some tasks, since there are already problem domains where AI dominates (e.g. chess, Go, etc).
My main objection to #2 is that the idea of a new structure or machine that exceeds human intelligence is not novel. Humans evolved to live in small tribes of no more than 100 people and we have not exceeded that limitation. So we're already several nested singularities deep[1] and humanity has not yet been destroyed.
On the other hand, humanity itself is a singularity: the system that created us (genetic evolution) is not capable of effectively controlling us anymore. We control it, and have been doing so since the dawn of agriculture. And we can see the harms that we have done to nature in pursuit of our own goals. So it's not irrational to extrapolate and see how an AI with more intelligence than us can do more harm to us.
[0] So are human children, but they do not have the I/O capability of a data center full of A100s.
[1] Agriculture beget civilization, which beget nation-states, which beget corporations, which beget computers. Each one is a singularity unto it's creators.
This is a point I've made. We are playing with paradoxes that proponents tend to ignore. They won't be solvable. The more sophisticated AI becomes, the more non-deterministic behavior arises. The emergent deception has been very interesting.
https://bounded-regret.ghost.io/emergent-deception-optimizat...
I've elaborated on #2 much further here as well - https://dakara.substack.com/p/ai-singularity-the-hubris-trap
We like to perscribe malice to the people spreading false news, but we forget that ultimately its often the recommendation engine that decides which things to highlight and which not to.
If you give an AI a value function that measures engagement perfectly well, and you get a recommendation algorithm that spreads false news and outrage inducing posts, the algorithm did exactly what you asked from it. The problem is that the optimizer didn't optimize for other implicit values you had in mind. (This is one of the key problems discussed in AI alignment)
Of course, this would be strongly contested by proponents of free speech who would say the AI gave us exactly what we wanted. But... did it? Or did the optimizer find a behavior that successfully manipulates people without taking into account the side-effect of radicalization of online discourse? Is that what we really want?
So, I'd say misaligned primitive AI might be already causing serious trouble in the world.
- unpenalized externalities, including carbon emissions, and other environmental effects
- clearly broken tax law which allows some enormously profitable companies to pay very low tax rates
- political donations protected as "speech"
- partisan gerrymandering
These are cases where we made systems with rules, in which agents have incentives, and where the combination of the design of the system and the incentives on participants can cause meaningful harm to the public. In many cases, transparency is meaningfully hindered, and "explanations" of particular outcomes are obscured. Several of the structural issues that AI alignment people care about are present, but the harms are already happening, and yet we seem unable (and AI alignment folk seem comparatively uninterested) in addressing them.
Aside from Basilisk cultists and people play-acting concern for safety and alignment as a PR tactic to push centralized control of AI in support of their own financial self interests (groups which have more overlap than one might expect), the main (non-academic) safety and alignment concerns are more about the kind of systems we do know how to build, and which are seeing increasing deployment deployment demonstrable severe biases that have significant impacts in application and other issues.
Hopf algebras (which are really just tensors with recurrence relations built in) subsume convnets, transformers and diffusion model and also provide a theoretically better autodiff that operates within single layers as opposed to across entire graphs.
Furthermore, there is a correspondence between Hopf algebra and linear logic and Hopf algebras are related to zonotopes, which are polyhedra that have been used in verified numerical computation. I'm strongly convinced the LL connection can provide proofs over zonotopes which paves the way towards interpretable AI and will be central for alignment.
I'm working on a next gen Hopf algebra based machine learning framework.
Join my discord if you want to discuss this further https://discord.cofunctional.ai.
That being we must accept that a low IQ entity could design an unescapable containment for a high IQ entity which was built for the very purpose of solving imperceptible problems of the low IQ entity.
I have put a lot of thought into what appears from my perspective a logical contradiction. I expand on that in great detail here - https://dakara.substack.com/p/ai-singularity-the-hubris-trap
Has anyone seen this argument proposed and answered in a convincing manner?
(I read the last paragraphs at your link to see if I'm misunderstanding you.)
Their position is that it is a hard problem to solve, but not impossible. They are fully committed to bring about AGI/ASI.
- Creating an AGI whose goals are compatible with ours.
- Creating an AGI with inscrutable goals and safely confining it.
They're not the same!
In whoever you've been talking to I can't recognize the rationalist scene I've known since it grew out of the 90s extropians. "AI boxing" was a live argument around the early 2000s. Not now.
Online you often see people in that community saying that slower progress in AI capabilities would be a very good thing. At the same time, yes, they do want us to achieve AGI/ASI as soon as possible, provided we don't screw it up.
That's the point, it is not possible. The premise is based on a paradox and you can't throwaway containment and have alignment that remains defined by original intent.
But I think you gave a false impression of others at the start of this thread.
That is precisely what I have stated.
> But I think you gave a false impression of others at the start of this thread.
How so?
1. Congrats on recognizing that it's a hard (perhaps impossibly so, at least with the time left) problem! Too many (not any of the alignment people) have thought it's a trivial matter, or that Friendliness (the old word) will just "emerge", or that somehow being "superintelligent" will cause it to just smoke weed and play games all day like some super smart dorm room friend so there's nothing to worry about, or...
2. The idea of Friendliness or Aligned values is a difficult one purely from definitions alone, yes, let alone implementation, so that's part of what makes it such a hard problem. The idea of an AI tiling the solar system with smiley faces was once a response to a serious suggestion that one can just train the AI to optimize "human happiness". Imagine trying to formally model something like "If AI-Gandhi was offered a pill that made AI-Gandhi 2x smarter but also want to destroy the world, we know AI-Gandhi would not take such a pill because [...]" What's the [...]? A proof about its values? For actual Gandhi, we might as humans say "because he's Gandhi, and Gandhi does not want to destroy the world, and does not want to want to destroy the world". On the topic of the submission, I hope that any who take the course can by the end demonstrate many issues with things like the old and obsolete Coherent Extrapolated Volition proposal.
3. Containment is of course folly, you can look up the old AI Box arguments if you wanted more on that from back in the day. Hence the focus on Friendly (and more modernly, Aligned) AI.
4. I, an approximately +1 std dev IQ human, fully expect that it'd be no challenge for a -1 std dev IQ human to dig a pitfall trap that could not only contain me but kill me. Because gravity still applies even if I'm smarter than my adversary. Sure, an AGI will figure things out about physics that we don't know, and math too, but physics will still apply, I would bet that even a superintelligent AGI will never find a way to break the speed of light and travel the universe. But perhaps we're very wrong about special relativity. We'd have to be very wrong about everything if our proofs from first order logic are somehow awry. But that too highlights the level of challenge to the problem: we have to prove the AI is Friendly, with the reliability of first order logic. As a matter of fact humans can do things perfectly, your CPU is executing billions of instructions perfectly every second. (Or to be fair, there are errors, but those are modeled accurately and through correction and redundancy (with multiple systems for things that go into space) we can drive the probability of a total failure as low as we please.)
If that was your takeaway, then you missed the point. It was not an argument that it was a hard problem. It is precisely the argument that it is impossible as paradoxes are not solvable.
> As a matter of fact humans can do things perfectly, your CPU is executing billions of instructions perfectly every second
Yes, if you constrain the context small enough you get get the result you want. However, the concept of a ASI with free agency expands the context to a scope that we can't even reason about.
Similarly I'm not convinced that the scope is broad enough that it really cannot be reasoned about at all, either in its entirety or by breaking it up into smaller components (as we do for large problems already). I think you'd have to give some sort of impossibility proof on what about the problem is possible to reason about and what is not possible to reason about to convince me. (Note: chaos theory won't necessarily help you here, because probabilistic reasoning exists as well as do() operations by us, and if it must be used the argument must explain why certain catastrophic errors cannot be modeled and driven to arbitrarily low values.)
You might have some confused notions about free agency that if cleared up would resolve your other questions. This might be useful: https://wiki.lesswrong.com/wiki/Free_will_%28solution%29
It is an isolated narrow context. The problem with ASI is there is no such limitations. I addressed this point in the post, but another example would be the Kobayashi Maru Test.
Or rather, there has been no system devised that humans have not found a vulnerability. Why would AGI be any different? Alignment/containment must also address the possibility of human exploit of the system as well. So we are back at the beginning in either case. Which means a human could still devise a trap for the AI to leverage its ability or modify its alignment.
> Similarly I'm not convinced that the scope is broad enough that it really cannot be reasoned about at all, either in its entirety or by breaking it up into smaller components
We are already having challenges in this area. Example...
“The size and complexity of deep learning models, particularly language models, have increased to the point where even the creators have difficulty comprehending why their models make specific predictions. This lack of interpretability is a major concern, particularly in situations where individuals want to understand the reasoning behind a model’s output”
from - https://arxiv.org/pdf/2302.03494.pdf
Yes, there's "difficulty comprehending" current deep learning models, put modestly. There's also interesting work in interpretability/explainability going on to work on that. Nothing there says "impossible". However, even granting that it is actually impossible to explain and reason about current AI methods to the degree we need to in order to create chains of logic for alignment purposes, that just says that it's impossible to reason about an AGI produced using such methods, if it's even possible to produce an AGI that way. It says nothing about it being impossible to reason about an AGI produced using different methods.
I understand you don't perceive the paradox. Nonetheless, my point is rather that when a paradox presents an impossible problem, then something about our premise is not understood. Or we need to find an entirely different concept to solve our problems. For example, an entirely different concept of intelligence not modelled strictly on how we perceive our brains. In summary, I think we are trying to solve the wrong problem. I might write something more depth at some point on that to explain my views.
> Incidentally there are many secure systems and schemes that have not fallen. But you'd probably write them off as too narrow for some reason.
This is precisely my point. We can build what is logically infallible to only be subverted by different means. If you can't hack the system, trick a human to give you access. This is how every such system has been defeated.
Which is part of the same argument, in that if the goal is AI safety that does not have free agency to do harm either directly or as an indirect unforeseen behavior, then it is an impossibility without containment and if containment is impossible so is alignment.
How could we expect it to be otherwise? At least if you are going to model the intelligence based on human experience and how we process the world then we have to assume just like a human mind can become unaligned so could ASI.
An aligned AI is essentially a soft boxed AI. It is not that different in concept. The end result is we want a set of behaviors as outcomes and other behaviors to not be present.
Complex problems are hard. Paradoxes are not solvable. That is the distinct difference I'm trying to make.