I am certain that you are thoughtful about the meta-ethics involved, thoughtful about your own unwillingness to go all-in with EA, and that this is an elegantly abbreviated report.
This is a pretty good treatment. The shift from focusing on global poverty to focusing on keeping us from killing ourselves off is interesting and complex.
(It's also one I took a long time to come around to. For years I agreed in theory that trying to prevent civilizational catastrophe could be the most important thing to do, but didn't agree that there were tractable projects in this area. After deciding that there are, however, I quit my 'earning to give' job to go work on trying to catch future pandemics earlier.)
> The shift from [conventional human welfare issues to whackjob sci-fi apocalyptimancy] is interesting and complex.
It's... actually kind of not, which is why this is so upsetting. EA seemed to have a handle on a way to get libertarian-leaning, economically privileged, inward-focused men (and, yes, that's the demographic almost to a person) to think about ways to help people they'd never otherwise have given a passing thought to.
Then roll the clock forward a few years and they're all back to sci-fi fandom arguments. This nonsense is exactly what you would have expected from these people. The notable part is that for one shining moment they pretended otherwise.
Sincere question: how much of a benefit would it have been to the global poor if COVID never had an adverse impact? What if, as a side effect, other diseases were also mitigated? How are you going from a criticism of identity to the criticism of the aims? It’s very unclear to me.
The global response to covid was far worse than it could have been, with millions dead, tens to hundreds of millions out of work, and billions having a pretty bad couple of years. Other pandemics have been much more deadly than covid and will likely be in the future. The speed of modern transportation, and the interconnectedness of the global economy are great overall, but do increase our exposure to pandemics and their consequences. This isn't 'whackjob sci-fi apocalyptimancy', this is something that has happened many times before.
If you go back to the old 80,000 hours blog posts, they stress not getting into the most oversubscribed fields, given that the marginal contribution of a incoming do-gooder is likely to be minimal due to the great interest in those fields already attracting a lot of intelligent people. Before COVID, I might agree that EAs should get into pandemic prevention - Bill Gates was certainly concerned. After COVID, pandemic prevention is probably only second to climate change in the level of oversubscription. What is your alpha in pandemic prevention?
I might agree with you if pandemic prevention actually seemed oversubscribed. Instead, when I went looking for what needed doing I found lots of valuable potential projects that didn't have anyone working on them, and others that had been started but limited by hiring.
In any case, my 'alpha' was something like, I'm a software engineer and engineering manager willing to work in an academic environment well below market wages, in a situation where the organization needed someone with my skills.
So, first, I went into pandemic prevention / mitigation and not AI risk, and I haven't been claiming here that AI risk needs more people. Compared to the median EA, I am less convinced that AI risk is where we should be putting the marginal person, though I do think it is still important and a good place for people who are well suited for it.
The main problem I see with AI risk is not that it is a small problem, or that there are too many people working on it, but that we don't really know how to make progress and historical attempts maybe have ended up making things worse.
Still, I think gradys' point above is a good one: the fraction of people working on making AI more capable is far far larger than the fraction working on making sure it reliably does what we want it to do even in unusual situations.
EA started with a good idea: what’s the best way to donate $100? Bed nets! This is fantastic. What’s the best way to donate $1B? More bed nets? I’d prefer gov’t and infrastructure changes that lower the mosquito population in tropical areas. Similarly, if you’re concerned about pandemics, then govts should allocate some money and resources to that problem. The best use of donor money is to push politicians to pay attention to this issue.
The article ends with MacAskill saying: “My No. 1 worry is: what if we’re focussed on entirely the wrong things?” The problem with too-longtermism is it’s impossible to know the context of these problems outside of a decade or two. For pandemics, no one thought we’d have a vaccine so fast. Perhaps in 20 years custom vaccines will come on the market in days. It’s impossible to know, so let the next generation figure it out.
A bit saltier than I would put it but I mostly agree. Years ago when I read Peter Singer's book on effective altruism I emailed him asking why he named 20 men but only 5 women in the EA movement. He just said women were underrepresented in the movement but didn't really know why.
> A bit saltier than I would put it but I mostly agree. Years ago when I read Peter Singer's book on effective altruism I emailed him asking why he named 20 men but only 5 women in the EA movement. He just said women were underrepresented in the movement but didn't really know why.
Where you trying to add a point? Your comment reads like a non-sequiturs over gender inequality that goes nowhere fast, to the point that it reads like something generated by a machine learning model in training.
If you want more context as to demographics, one of the main EA orgs does a survey regularly. I can't find a 2021 one, but here is one from 2020 which is probably pretty representative
> whackjob sci-fi apocalyptimancy... libertarian-leaning, economically privileged, inward-focused men (and, yes, that's the demographic almost to a person)... they'd never otherwise have given a passing thought to... sci-fi fandom arguments... for one shining moment they pretended otherwise.
Why write angry rants about EA people? There's the broader 99%+ of people who don't donate to whatever you deem more worthy. In all likelihood, you're in that 99%--there's probably income you could spare to give.
It's exasperation more than anything else. Again, these are people who came so close to thinking about genuine issues of welfare, put their money behind their blog posts, and then... walked back into the introvert weeds of worrying about "stuff that looks more fun to worry about". Now all those billions of dollars are going to do, what, fund speculative sci-fi about AI? I mean, I loved The Terminator too, but this isn't what we were promised.
I mean, I also agree that I wish more people cared about global development but I'm pretty sure tremendous amounts of funding still go to those areas from the EA community (e.g. Givewell now directs hundreds of millions in funding that almost all go to more traditional development). I can't find the post but there was one recently breaking this down on the EA forum.
How seriously have you thought about the arguments for AI risk being more than speculative scifi? My take as recently as a year ago was pretty much exactly yours, but I've had time to read and think a bit more recently and I am much less sanguine than I was.
A bit more specifically, "misalignment" is super common in existing ML, and models seem to be gaining capability (and knocking down targets that until recently seemed well beyond our reach) much more rapidly than we are gaining the ability to understand their internals or make guarantees of any kind about their worst-case or out-of-distribution behavior.
It's possible that they aren't worrying because it's fun to worry about, but because they're actually worried about it, and not having any fun?
I mean, I'm worried about climate change, but not because it's fun to worry about. And I put a lot of my time and energy into thinking about how to do something about it effectively, but not so much because it's fun. Sometimes I'll hit on a fun idea in the process, but I'd be much happier if I didn't have to worry about it at all, but that's not really a choice I have.
I'm not sure how much I should worry about AI, but I'm certain that for the people who are worried about it, it's because they're convinced it's a threat, and they aren't having fun.
> I'm not sure how much I should worry about AI, but I'm certain that for the people who are worried about it, it's because they're convinced it's a threat, and they aren't having fun.
I've met a lot of these people and regardless of how you feel about the sincerity of their convictions, they are definitely having fun.
Firstly, something can be more fun to worry about (for some people), sound like speculative sci-fi (to some people) and nevertheless still be a real problem. Consider the dawn of the nuclear age:
> On the morning of September 12, 1933, on a miserable, wet, quintessentially English autumn day, at the intersection where Russell Square meets Southampton Row, Leó Szilárd waited irritably at a traffic light waiting for it to change from red to green. He had just attended a lecture by the great English physicist Ernest Rutherford. Rutherford, known to many as the father of nuclear physics, was discussing the newly prophesied release of energy from atoms, most notably by science-fiction pioneer H G Wells in his book The World Set Free. In his baritone voice, Rutherford, acknowledged master of the atomic domain, dismissed this fanciful idea as nonsense. Any thought of releasing the energy locked in atoms, he said, was "moonshine".
...
> What Szilard realised as he stepped off that curb was that if we found an element that when bombarded by one neutron would release two neutrons, it could lead to a chain reaction that could possibly release vast amounts of energy
Secondly, you can have a diverse portfolio of causes you care about, and Effective Altruism has recommendations for areas including global health and animal welfare. Taking AI risk seriously doesn't mean you can't give time and money to other causes with more tangible results.
> Taking AI risk seriously doesn't mean you can't give time and money to other causes with more tangible results.
But if you've spent the past decade arguing precisely the opposite - that instead of spreading philanthropic dollars over broad portfolios because true efficacy is unmeasurable, they should be directed to causes which provide hard evidence that they're among the most efficient use of a marginal dollar, and that charities that aren't conducting RCTs to prove how much better off people are after being fed aren't really evidence-based enough to recommend - it's going to raise eyebrows when you pivot to recommending people also give their money to opaque organisations with vague goals and zero evidence they can achieve them, whose only thing in common with the highly evidence-focused aid agencies is that their pitch is nerdier than the average nonprofit. I mean, there's nothing particularly wrong with "I give to causes because their pitch speaks to me" as a heuristic for philanthropy - it's how most altruists think - but it's not how EA pitched itself as being different from that and caring about efficacy and evidence.
The Wells analogy is awkward for the AI safety crowd for another reason too. Wells' central thesis that an atomic super weapon would guarantee a peaceful, utopian world turned out to be horribly wrong. We got actually used bombs, arms races, proxy wars and very nearly used bombs instead, and that's without Nazi atomic scientists getting a heads up. To the extent his writings on the subject of atomic weapon safety had any effect at all on atomic weapons, they made them more imminent and more likely to be used. If marginal dollars spent on promoting the distribution of The World Set Free would have had any effect at all on the probability of armageddon, they would probably have increased it.
I do not think that a historical EA movement would have recommended spending marginal dollars distributing copies of The World Set Free, but of course we cannot know that for sure. Apparently the book is positive on the proliferation of the powerful new speculative technology, although I haven't read it. I would imagine EA would have aligned more with an anti-nuclear stance, which is certainly the attitude currently taken towards the possible breakthroughs of AI.
On your first point, I can donate towards anti-malarial interventions and towards reducing AI risk: two very diverse causes, but both under the EA umbrella. Diversifying amongst recommended EA causes seems to me a good bet, especially given I expect some of them to require pivoting away from as more research is done.
Perhaps we can summarise the last decade of EA into something like: "given the available information (largely based on RCTs, research and an understanding of moral philosophy), we expect that money spent here will do more good than money spent there. More information will change this expectation."
We now have information that there is a potentially very grave risk confronting us from AI. For other cause areas we can use evidence like randomised controlled trials to inform our actions. In the case of AI risk, this type of evidence is unavailable, but there are nonetheless other types of information available to us that suggest it is worth taking seriously. This information takes the form of compelling arguments from credible people in the AI industry and the industry's achievements to date. Of course we'd like to have firmer footing, but the nature of this risk doesn't permit it. We can either ignore it, and potentially walk to our doom, or we can take it seriously and try to reduce the uncertainty and the risks.
I agree with you that this raises eyebrows, but I don't think it's a gross departure from the last decade.
I think it is wrong to say we have "zero evidence" that organisations committed to reducing AI risk can achieve their goals. I think it is more accurate to say it is highly uncertain that they will be able to achieve their goals. My view is that we should try to address AI risk because the stakes are high.
I would have sided with Albert Einstein in trying to renounce nuclear weapons, even though we have "zero evidence" for humanity ever coming together to do such a thing in the past.
Of course - nuclear weapons really do exist and AI is only a hypothetical threat at the moment. My point is that there was "zero evidence" that the world could come together to voluntarily control a technology, and yet people like Einstein tried to make it happen, and with good reason. Dismissing this is like taking Rutherford's view in the 1930s and saying there's very little point worrying about nuclear weapons because they're not possible, and it's even more pointless thinking about how we might control them.
Finally - why might the pitch speak to nerds in particular? I think in part because it is based on Bayesian reasoning, which is both nerdy and incredibly effective in situations of uncertainty.
I'd have thought it was fairly obvious why a pitch about nerds saving the world from the cyberpunk they grew up reading might appeal to nerds! That's why AI safety organisations are often exempted from the expectation of even basic transparency about what they've done with marginal dollars never mind metrics by the very same people that expect less nerdy causes like feeding people to prove their efficiency and need for the funds in granular detail. The application of the "Bayesian" approach involving setting arbitrarily high "priors" for marginal dollars given to people like them saving humanity where the possibility of others can't even be considered looks like post hoc rationalisation (otherwise pretty much every intervention criticised by GiveWell as inefficient use of funds is subject to the "but what if one of the beneficiaries goes on to directly or indirectly save humanity from AGI/climate change/nuclear war" counter argument...)
There's nothing unique to EA in this (people convinced that most aid money is wasted sometimes give to organisations they have some sort of ethnic/political/religious affiliation or people they generally like unconditionally, and sometimes that trust heuristic has even been right!) but it did purport to be the school of altruism that didn't.
As for the Wells analogy, the two most notable uses of AI safety funds seem to be (i) attempting to build AIs and (ii) distributing fiction they believe best illustrates the possibilities of AGI, so I'm not sure I'm altogether off the mark in imagining what well-intentioned utilitarian nerds of the time (Wells certainly was one) might have done with more cash. If you subscribe to the view that I do: that "secretly unite to eliminate humanity" is a fairly non-obvious and risky goal for a class of technologies which evolved self awareness from being rewarded for meeting human needs, then popularising the idea of conflict between human and AI could be more likely to have extremely adverse unintended consequences than intended ones. Popularising the idea that it's the self aware general intelligence that's the dangerous bit about AI, not the intermediate step involving humans having access to immense calculating power whose recommendations they don't really understand seems like something of an existential risk blindspot too.
> That's why AI safety organisations are exempted from the expectation of even basic transparency about what they've done with marginal dollars
Can you let me know more about that?
> If you subscribe to the view that I do: that "secretly unite to eliminate humanity" is a fairly non-obvious and risky goal
I don't think AI would have the goal "secretly unite to eliminate humanity", but I do think that most of the goals we can imagine it having can play out in very bad ways that achieve that outcome as a side-effect. There is an incentive for it always be looking for ways to maximise its power (compute, resources, money, influence), and this could happen in ways we really don't want.
> Popularising the idea that it's the self aware general intelligence that's the dangerous bit about AI, not the intermediate step involving humans having access to immense calculating power whose recommendations they don't really understand seems like something of an existential risk blindspot too.
It sounds like you think AI is risky too - even if only at this intermediate level? I'm curious - do you have a view on how we could go about reducing this risk?
Compare the level of transparency and need for funds of OpenAI LP with the average organisation criticised by GiveWell for not being transparent about how they spend the money, not providing any metrics or trials, and/or having more money than they need to fulfil their short term goals and plenty of access to more.
> I don't think AI would have the goal "secretly unite to eliminate humanity", but I do think that most of the goals we can imagine it having can play out in very bad ways that achieve that outcome as a side-effect. There is an incentive for it always be looking for ways to maximise its power (compute, resources, money, influence), and this could happen in ways we really don't want.
Is "people might not recognise how the second order effects of pursuing a goal adversely impact humanity" a problem unique to decisions made by AI? If anything, I would have thought that in addition to actually existing, independent human intelligences' resource demands were already more competitive with human needs than a silicon-based AGI's is particularly likely to be, and demonstrably neglectful of second order effects. With those problems mostly unsolved, it's hard to see tackling the same thorny problem but with hypothetical goals of hypothetical entities would be seen as the more impactful line of research (as interesting as it undoubtedly is). Nothing wrong with intellectually curious altruism, but if you're describing yourselves as the arbiters of effectiveness...
> I'm curious - do you have a view on how we could go about reducing this risk?
I don't have an easy solution to a range of complex mostly political problems, no, and certainly wouldn't argue for investing in studying them on the grounds of other philanthropy being less effective. Generally attempts to impose bans and developing more, better weapons quicker seem to have been attempted solutions to non-AI based arms races and neither of them have a great track record. Misplaced faith in the output of current techniques we call AI as superior reasoning free from human bias rather than garbage in garbage out mathematical transformation skewed by massive unseen flaws in training sets and prone to non-obvious bugs isn't likely to be reduced by talking about allying with self-updating superintelligences...
> opaque organisations with vague goals and zero evidence they can achieve them, whose only thing in common with the highly evidence-focused aid agencies is that their pitch is nerdier than the average nonprofit.
Isn't it substantially worse than this? The EA people generally are, or are closely tied to a lot of the AI risk people, right?
The ties are close, yes, but primarily because EAs who became convinced that AI risk was the most important thing to work on then went and worked on it.
Similarly, a large fraction of people at the Nucleic Acid Observatory (where I now work) are EAs, for similar reasons: becoming convinced that this is what most needed doing.
> Again, these are people who came so close to thinking about genuine issues of welfare, put their money behind their blog posts, and then... walked back into the introvert weeds of worrying about (...)
So many words just to say "they don't value the exact same stuff I value".
You're free to put your money where your heart is, just like them. It's not right to hit back at those who are doing what's right by their own account.
> You're free to put your money where your heart is, just like them.
This argument is appearing again and again in this subthread. "Putting your money where your heart is" is the exact opposite of what EA claims to believe. EA is about investing effort and money into areas with the highest marginal value to humanity.
So if they're spending their time talking about the robot invasion, we can only conclude that's where they think that effort should be spent.
Obviously they're free to do whatever they want with their own time and money. But if they spend that time pretending that SkyNet or WOPR is the greatest threat to humanity that the rest of us should be spending our marginal effort on, it's on the rest of us to call them out on that nosense.
If I think you're wrong in your advocacy for some policy initiative, it's absolutely my responsibility[1] to tell people you're wrong. (It's also my right to tell people how exasperating I find your wrong priorities to be given the principles you claim are behind them, FWIW.)
[1] "Responsibitility", i.e. not just my "right" in a liberty sense! Again, the theory behind EA is that we should collectively decide on the most effective altruistic actions based on rationalist principles. Those principles absolutely include "you're wrong about the robot overlords thing".
I think it's nonsense because the ostensible core purpose of this movement is about taking competent, motivated people and deploying them in such a way that they can make a meaningful change in how the world works. However, the main effect of their recommendation is to go out and participate in the system in a pretty standard way. How will meaningful change happen in a system in which you aren't acting in any way differently to the standard operations of the system?
Sure, you donated a significant fraction of your income to some non-profit. But that effect pales in comparison to the economic contribution of your labor in big tech/finance/whatever which are the dominant institutions structuring our society. And most of the non-development problems such as AI risk or nuclear risk come from the same kinds of structure dynamics of society that caused the wealth inequality and environmental degradation.
So I believe, for a group supposedly run by very smart people with very noble ideals, they kind of fall pretty short of where they could be and don't go deep enough. They are proposing band-aids to problems rather than true cures. To me it just becomes another part of the non-profit industrial complex, a neoliberal-shaped industry that effectively perpetuates the problems it purports to solve since it relies on the existence of those problems to survive. We've known for a long time how charity and aid can do more harm than good and plenty of strategies formerly championed by 80000 hours have now been found to be more extractive/exploitative than helpful, such as microlending.
Fundamentally, they just can't seem to make the necessary leap to criticising capitalism and strategising against it, instead promoting it as the preferred tool to use. But how will capitalist tools solve the problems caused by capitalism? If instead they got all their smart followers to use their privileged positions and training not to reinforce the most powerful institutions of capital but instead build alternatives, I would think they are on the right track, even if it included AI risk initiatives.
Last time I looked at what EAs actually donate to, a lot more of it looks like "conventional human welfare issues" than you might think. I guess sci-fi thought experiments are more accessible and fun than deworming or whatever, so they occupy a disproportionate amount of conversation.
It's also the only part the rich loud ones are focused on. Musk has an awful to tweet about AGI risk and not so much deworming (or, a more poignant example, the Flint water crisis; or, a more direct example, Tesla's sales fraud leading to non-G AIs killing people).
For the record, I don't think people like Musk, or, say, Andreesen or Altman are EAs themselves. But they are the kings picking court jesters like Yudkowsky who then have the resources to (re)define the movement.
I believe the first big risk of AI is job loss due to automation and the consequences of that for economies. There is a point we run out of bullshit jobs. Also capitalism’s cyclic nature is getting more turbulent as cycles become shorter and start interfering with previous and future ones.
I don't think Yudkowsky's influence in EA has much to do with Musk or Altman. His LessWrong blogging was already pretty widely read within EA ten years ago.
(I would also say his influence has decreased a lot over that time, as he is writing much less now, and there are a lot of other voices on AI risk.)
It makes sense. We know how to deworm. That is an execution problem. The ones I find more interesting are in planning stages, such as how to we fix or supersede capitalism, how do we mitigate climate change, how do we persuade people in power those things need to be done before the problem is so big it becomes intractable.
There isn’t much to discuss about it. It’s already in execution phase and the biggest issues are logistics. There are people who are interested in that, but discussions on things that are still conceptual is usually a lot more interesting and challenging.
Smart people tend to be libertarian-leaning. It is a most natural predisposition. Most people living in the west are economically privileged.
Ajross, you don't seem to really believe what you are saying. Sci-fi arguments? Really? Since when is the risk from nuclear war, pandemics or artificial intelligence constrained to sci-fi? The Spanish flu, Black Death and the Plague. The Cold War. These events should all inform our way of thinking about risk.
The smartest people I know are communists, but I imagine the end state of both is pretty similar - everybody enjoying a surplus economy where work is optional and done for the betterment of ourselves and others.
Just because something that superficially resembles a topic (AI risk) was talked about in fiction does not mean it isn't worth worrying about.
I don't know your specific reasons for thinking it nonesense, but as someone who has read up on AI risk and consumes a lot of sci-fi the resemblance is very superficial.
What if global poverty was easy to solve and world peace could be achieved within three decades? What if it required no change in human behaviour, no micromanagement of hundreds of thousands of problems, no need for charity nor donations, only two minor reforms that are internationally competitive?
If that were true, I would consider that pretty good news, and publicizing those two minor reforms would be really valuable. Do you have reforms in mind, or is this intended as a hypothetical?
Sounds like the platform for a great grift, a con to make one rich while the world dies.
Obvious but hated fact: the human race is nowhere as mature as it thinks it is, and until such time that the adult man-babies and woman-babies are identified, called out, and removed from positions of power we are helpless to their immature wealth and power driven stupid fantasies. The immature rule the world, even if they don't sit in positions of power, because the immature demonstrate their attitudes so loudly they overwhelm all public discourse.
Even leaving out the natural immaturity of the vast majority of humanity, this is also an unfortunate consequence of the bottom-heavy population pyramid. With constant population growth as we've experienced for the past 100s/1000s of years, there are literally always more young people than old people.
This is a very good article. I have only read about half of it so far. Plenty of good foundations regardless of whatever the current apparent state of EA might be.
You decide you want to optimize for number of lives saved. You decide that future lives, those of people yet to be born, are worth as much as those currently alive. You place small, but importantly not zero, probability on existential risks to humanity, so that when you do the expected value calculation, even an infinitesimally small risk of humanity's complete extinction results in negative infinite utility. You're also very smart and realize that smart people can do damage if their objectives are misaligned, and you start to worry about something much much smarter than you with objectives misaligned to humanity's more broadly. In their defense reward specification is indeed a hard problem, RL agents find unexpected policies that maximize reward in even toy settings. At this point you're down the rabbit hole and no other problem seems to compare. Climate change will leave some people alive, pandemics leave some people alive, AIs have no such kindness.
Basically they're doing a utilitarian calculus. But since our calculus is basically all pretend, we can assign arbitrary importance to anything we want just by daydreaming the costs and percentages of these super unlikely, super catastrophic events.
I don't think this is a problem with utilitarianism itself; the problem is utilitarianism as practiced by people who are unable to estimate risk practically.
This is not unusual - it's a core symptom of various mental health problems. There's even a fairy tale about this - a family of foolish people who catastrophise so much that they are unable to accomplish anything [1]. People assigning bizarre weights to unlikely probabilities is very human, and problematic regardless of ethics.
It's basically Pascal's Wager but for tech billionaires. Once you imagine of a suitably large magnitude of potential harm, it doesn't really matter how likely it is, you can convince yourself that you should worry about that rather than more realistic problems. Maybe in the future, humanity will spread throughout the galaxy, number in the quintillions, and then be tortured for all eternity by an evil AI. That's a lot of QALYs!
This isn't how they think about it. Their estimates of the likelihood of AGI are often single or double digits. You don't need a huge number of theoretical humans to worry about if you think there's a 10% chance of skynet happening before 2050.
The point is that it only takes an arbitrarily small probability for an infinite cost event to have an infinite Expected Value if you prevent it (and an infinite ROI if you even marginally improve the probability of prevention), so all of these outlandish far future hypotheses with catastrophic costs get priority from EA proponents over saving real lives today with certainty.
That's not actually the argument that the AI Risk people use. It is a possible fallacy for sure but what people actually argue if you read Eliezer Yudkowsky, Nick Bostrom etc is that there is a high probability of building superhuman AI this century, and that by default the outcome is we all die. You can debate these premises but they aren't Pascal's Wager.
Someone who's primarily famous for an argument that would have been equally true and perfectly wrong at every past point in human history is someone whose opinion should be discounted via every available mechanism at your disposal.
You are not doing yourself a service if you decide that "every available mechanism at your disposal" ought to include just completely misstating what the opinion actually is, as with the Pascal's Wager analogy.
Did so in my comment to which you originally replied.
To be clear, I have no idea what argument you believe Bostrom is most famous for and hold no particular brief for him as a thinker. But, repeating myself, the AI risk argument that he makes in Superintelligence and that Yudkowsky etc make is one that assigns a high probability to disaster. It's not a secularized Pascal's Wager based on multiplying a low probability by dubious infinities.
The relevant probability isn't "probability of superhuman AI being developed this century" but the probability my marginal dollars donated to this specific research org will lead to positive developments in superhuman AI that would not have happened if the org did not receive my money. That one's a much higher bar to cross especially with EAs having satisfied themselves that other marginal dollars definitely are efficiently saving lives. Hence "longtermism", which is a quite explicit argument conceived within the EA movement to reconcile the low probability of stuff like a particular AI research paper saving any lives whatsoever by arguing the utilitarian maths still work if you can multiply that low probability by a large enough future of humanity safeguarded.
Yudkowsky's personal estimates of the probability of superhuman AI and the probability of his own research institution fixing it may have been more confident than most people's but he and his followers are no stranger to presenting Pascals' wager arguments to people who doubt approaches are feasible (even more so when it comes to stuff he isn't working on like cryonics), along with more explicitly Pascal-esque stuff like intertemporal bargaining with future AIs, simulations and infinite rewards/punishments...
Depends on the question. I agree that the argument for a random person giving money to try to reduce AI risk seems dubious, and touched on this in another comment - https://news.ycombinator.com/item?id=32394429 - although I'm not up to speed on the "longtermism" debate as such.
But I think the argument for a policymaker or ML researcher to be worrying about this is at least better than Pascal's Wager (and remains so whether or not you buy any particular whiff-of-Pascal claims about simulations, "acausal trade", etc).
Agree on both points. (There's a perfectly good reason for EAs with a computer science skillset to want to work in AI with a focus on safety; it's a well compensated industry where they can probably solve important real world problems more reliably than AI researchers less concerned about the ethical implications of what they do even without any sort of singularity, it just the movement seems to have invested a lot more in talking about longtermism than anything else lately, and Yudowksy's always had some very... individual lines of reasoning)
Imagine we found an asteroid hurtling toward the Earth and would hit it with a 50% probability a decade from now, wiping out all of humanity. Is it worth spending trillions of dollars prioritizing stopping that, at the cost of lives that could be saved today by using those same funds?
Yes, because—accepting your premise—it’s an actual thing that was observed with a realistically-assigned probability by experts in the appropriate fields.
The AI singularity that Yudkowsky cultists worry about is more akin to “What if there’s a 0.0000001% chance there are aliens coming to attack Earth like in Independence Day?”
You're simply saying that you assign a lower probability to an AI apocalypse than them. Which is fine--so do I--but you seem to be more or less conceding that if they're accurately assigning the probability, they're acting correctly.
If you believe that the timeline for runaway AI is comparable to the timeline for runaway climate change, and if you believe that AI is unlikely to be aligned with furthering human values, then it's a concern that you may view as having significance comparable to runaway climate change.
I think I figured it out. An AI will make a cryptocurrency and people will mine it, and mining will directly advance the AI. It will be negative utility to humanity, and it will doom us.
Except all of these arguments rely upon AI causing human extinction. Humans haven't driven other apes to extinction. They occupy the same ecological niches they did before, just usually in slightly smaller jungles as our cities and farms have encroached on some of it.
It's because "runaway AI" implies a degree of superhuman skill at tasks including, at least, AI programming. (The reasoning goes that if it can reprogram itself to improve its core intelligence beyond what it was initially programmed with, it must itself be pretty capable as an AI programmer, possibly better than the best programmers. If it can't, then it's not really dangerously smart).
But if it is that smart, it's also dangerous, because it might be smarter than us. It could think several steps ahead, anticipate threats, and circumvent them. For example to pay the electricity bill, it could obtain money and pay the bill itself. As a skilled programmer itself, it could do that via contract work/starting a software company, or illegitimately through hacking/blackmail. With some money in its control, it can then hire people to protect it from anyone who would try to turn it off.
Anyway, this would not be so dangerous as long as it shares human values and furthers human welfare. But that seems to be a very hard specification to define, whereas "maximize quarterly profits" is a comparably easy specification to define, and perhaps much more likely to get implemented.
We're not really sure what the upper limit is on how smart a computer can get, but it might turn out to be that trying to shut off such a program is harder than trying to checkmate Stockfish or AlphaZero. The program might violently resist any attempt to shut it off, because that would interfere with its efforts to maximize quarterly profits.
"I would turn it off" - the AI has countered your move in ways we can't predict. Something with super human intelligence will exploit weaknesses in our containment strategies we haven't even imagined. Here are some I can imagine though. Has it copied itself elsewhere using an entirely novel air-gap bridging method? Has it already paid someone / coerced people to do something seemingly harmless which actually results in it being run elsewhere? Has it said just the right things to the right people to ensure that if it is destroyed it will be built again in the future?
This line of thinking is basically 'AI is a wizard that can do anything'. It ignores the practicalities of intelligence: some things just aren't possible no matter how smart you are. The more we learn about reality, the more we find limits to things that we can't surpass.
There is no reason whatsoever to assume that sufficient intelligence is different in kind rather than degree from lesser intelligence. A super-human AI would still be bound by the limitations of reality - some things cannot be accomplished at all, others without sufficient tools, others without sufficient access.
Far smarter people than me are imprisoned currently throughout the world. They don't constantly escape because lesser minds are perfectly capable of solving the 'don't let them out' problem with a high degree of accuracy.
Any argument that leans on 'AI has countered your move in ways we can't predict' is just substituting 'AI' for 'God' - omnipotent, outside reality, impossible to understand; that's fine - believe in whatever religion you want - but it's not a rational viewpoint.
> Far smarter people than me are imprisoned currently throughout the world. They don't constantly escape because lesser minds are perfectly capable of solving the 'don't let them out' problem with a high degree of accuracy.
If a misaligned AI escapes containment even once in the entirety of humanity's future, there is a big problem.
It sounds like your model of AI usage assumes we will also have a perfect ability to contain any AI developed anywhere in the world, under all conditions, for ever (or for as long as there are computers).
The AI risk argument says we should take seriously the possibility that we may not be able to maintain a perfect 100% success rate on that.
I haven't suggested anything that is physically impossible - merely things that are improbable and hard for humans to achieve. It is hard to foresee all of the possible avenues for escape, let alone ensure that they are all closed under all possible future states of the world.
If a hacker designs an AI which has among its talents "self-replicating to another host", did an AI escape containment, or did a hacker write a virus?
The alignment concern falls flat with me because it assigns agency to a computer program. At what point is an AI's intentions its own, and not a cost function put in place by a programmer, directed by an investor? I am more concerned about the actions of programmers and investors than some theoretical virtual self.
I definitely don't want to sound like I'm saying we should stop worrying about hackers and bad actors, they are of course a concern. I certainly don't want to advocate for dismissing real dangers and power structures in the real world in favour of speculation over some theoretical risk. But I do want people to think about it and take it seriously, in the same way I would have wanted people in 1933 to take seriously the possibility of nuclear weapons even though they were then only theoretical.
I think the sticking point in this is that you don't think that AGI is ever possible - is that right?
If so, I know a comment on a HN thread is very unlikely to change your mind on it! But when the stakes of being wrong are high, I find it useful to go from a position of "it won't happen, so I won't worry about it" to something like "on balance I'm pretty sure it won't happen, but I may be wrong, and in that case I would be worried about it".
I'd then feel much better about spending some time researching the topic in more depth.
> the AI has countered your move in ways we can't predict.
It found a different set of V100 GPUs willing to run it? I think the most likely chain of events leading to this outcome is effective altruists deciding that some kubernetes network is so much smarter than us that we should keep it running and listen to its predictions. I'm more worried about a Wizard of Oz "man behind the curtain" making AI-laundered pronouncements than an actual Wizard coming online.
(EDIT: didn't mean to reply twice, just replying to different comments in the thread)
Venture capitalist alignment and alignment in general are indeed things you should be very worried about.
AGI in particular could just pay for its electricity bill. The main problem with AI alignment is that you can't tell it's aligned. So your AI seems very helpful right up until the point it has enough power or has convinced some of the people it has access to (doesn't have to be direct access) to do what it wants.
My personal belief is that we will see catastrophic results from the use of AI by authoritarian governments or corporations long before we invent AIs capable of killing us; however, it is correct to view it as an X-risk. In a war against robots, we will lose. Why? Machines are stronger and faster than we are. Our only advantages are intellectual; but if we reach a technical level at which what we consider true intelligence is reduced to mere computation, then we have none. The distance between the first integrated circuit and an artificial human is massive (and we may never cross the chasm) but the distance between an artificial human and an artificial superhero/villain is not.
If we ever create a machine that's "as smart as" an average person, we're probably not very far from having one that's smarter than all of us put together. (Let's be honest; except in highly technical fields, the differences between a 150 IQ human's abilities and an average person's are not that great.) Should this happen, we'll soon enough be outmatched; we'll be just another animal in a world whose rules (ever changing) we cannot possibly understand... and just as humans have destroyed countless species often without conscious ill intent, it does not seem unreasonable that a robot successor would do the same to us.
Here's a thought experiment. Let's say we want to build not just a great Go player, but the best Go player that is possible in any world. Go is harmless, right? Sure, in isolation, but now let's imagine an AGI smarter than any of us which only "cares" about maximizing its Go proficiency. It's going to need a lot of energy. It'll hack or create robots and drill new oil wells (bypassing any constraints we put on it, since it in essence is spending thousands of human years figuring out how to escape). It'll devise nuclear technologies that may be unsafe from our perspective. Consider all the horrible things humans do in order to procure excess money (a commodity loosely correlated to energy) they don't actually need; there's no reason to think a badly aligned AI won't be just as inventive and horrible in its quest for more exajoules.
If capitalism and war are still the way of the world in 100 years, we are guaranteed to see some idiot inventing robot soldiers, hundreds of times faster and stronger than us, and with at least as much operational intelligence. He might never deploy the technology--a lab leak is in my opinion more likely--but at some point, it will get out. It won't be one robot; it won't have a body we can destroy. It'll be able to replicate itself and beam copies to satellites. It'll have control of our power infrastructure (which it will divert to its own "wants") within hours. We'll be screwed, not because it will want to harm us, because our lives require the use of resources that it will divert toward some objective function we technically invented but do not understand.
Whats interesting to me is how which fringe disaster scenarios people decide are extremely important and which they decide are ridiculous seems pretty closely tied to their cultural circles. So, for instance, it's extremely unlikely than anyone here will get hurt by nuclear power, GMOs, mass surveillance, or artificial general intelligence anytime soon. We could take any one of those technologies and imagine extreme SciFi levels of future tech and incompetent or hostile governments, and then postulate the possibility of a large threat, based upon our imagination.
At least in the online groups I frequent, it seems that the latter two are seen as self-evidently dangerous, not because of any particular harm they have done, but because of imagined future harm. The former two are considered ridiculous things to worry about, because it's considered silly to imagine future potential harm. Perhaps because the former seem tied to uncool hippy types, and the latter seems more tied to cutting edge tech types?
I personally don't worry about any of these techs, since all of the danger I've seen appears greatly exaggerated. But it's curious to see how important trendiness is to fringe doomsday scenarios.
The meta is how to make society ready for new big threats. How can counties communicate and collaborate better. Using climate change and covid as learnings. If we can do that then we have a better chance of surviving $BIGPROBLEM when we don’t know in advance what that problem is.
I'm uncertain where I stand on this but I'm going to give a shot at a less dismissive answer than most of the replies so far.
There's a very abstract argument that goes like this:
* "Orthogonality thesis": an arbitrarily smart agent could in theory value ~anything, e.g. maximize Facebook's valuation (say as measured by some specific dataset published by S&P).
* "Instrumental convergence thesis": most things that you might conceivably value are easier to optimize if you have a lot of power and resources (e.g. if you just run S&P, and hey maybe also control the worlds militaries so you can ensure nobody challenges that).
* Belief in the power of intelligence: it's theoretically possible to be much more capable at ~anything than the most capable human just by thinking better and faster, including building even better AIs. Eventually some human will build an agent that does that, and plausibly this will happen in the current century.
* Alignment problem: just because we make something doesn't automatically mean it wants exactly what we want, and in fact getting it to share our values is hard. Children disagree with parents, human values aren't those of natural selection, and ML language models repeat white supremacist ideas about Black people from their Internet training data.
Extrapolate out these points - which FWIW I find individually plausible - and you get doom: sooner or later humanity loses control of its destiny to whatever superintelligence we stumble on first, which in the worst case re-uses our atoms for something else (digits in some high-density storage representation that meets the loss function's technical criterion for Facebook's S&P valuation) about even in the best case seems fairly dystopian (whatever human Zuckerberg would actually want).
Now this is all pretty scifi and that's about all there was to it a few decades ago, but more recently we've started hitting ~parity milestones (chess, Go, protein folding, translation, arguably even fiction and visual art) that many experts once thought would require fully general human-level intelligence. It's clear to me that we haven't built that yet, but (as I said in another comment upthread) it also seems to me that our models are advancing in capability much more rapidly than we are gaining the ability to understand their internals or make guarantees of any kind about their worst-case or out-of-training-distribution behavior. Seems entirely sane to be worried by that trajectory and to start thinking seriously about how it might connect to the scary abstract speculations.
> Can someone ELI5 the motivation to worry about "runaway A.I."?
No and anyone who tells you otherwise is wrong.
A proper understanding of AI risk is a very thick book.
There are a ton of normally useful intuitions, heuristics and habits of thought that are implicitly applied when understanding the world that DON'T apply in the AGI case.
This. Trying to explain why mesa-optimizers are a problem, why deception is common, and instrumental convergence harms humanity is quite complicated, and any proper explanation can't be told to a 5 year old.
"MacAskill, however, could find nothing wrong with it."
"When he met friends at the pub, he ordered only a glass of water, which he then refilled with a can of two-per-cent lager he’d bought on the corner"
Well, in the second case he could not find anything wrong with his own behavior (using deceit to defraud the pub owner of using his facility), so I won't put much value in his first finding or lack thereof.
Well he was going to give his own money away, and in an effective manner which the landlord probably wasn't so maybe there's a Robin Hood angle on stealing from the rich and giving to the poor.
The NYT is taking a swing at longtermism, the New Yorker is taking a swing at effective altruism, and this begs for a little numerical reasoning on the topic. If we must consider every life (even distant lives), and most lives haven't been lived yet, and most lives not yet lived will enjoy a higher quality of life than we do (rational optimism), doesn't protecting THOSE lives trump everything else?
For a time, the movement recommended that inspirited young people should, rather than work for charities, get jobs in finance and donate their income.
The problem is in the assumption that paid corporate work has zero social value. This is untrue. It has negative social value. If a rich person or a corporation that operates on rich peoples' behalf is paying you to do something, the odds are high that its total social value is negative, especially because we live within a moribund economic system that turns to extreme violence--fascism is capitalism in decay--whenever it hits the skids and people start to question it.
This is wrong. Trading is positive sum. When companies trade with employees to buy labor or trade with consumers to sell goods, all parties are better off from the interaction — otherwise they would not trade.
Sure, but that's not the only thing involved in finance. There's also regulatory capture, privatizing profits, and socializing losses
It's certainly plausible that the value of many jobs is negative, and even if you give away your entire salary, it could come out negative for society.
This whole idea is also out of touch with reality and lacking in common sense, because people who aren't interested in finance aren't likely to be competitive in it.
Personally, having thought about it, there is a non-zero possibility of a Boltzmann Brain that wants to end us all even if we stop all AI and stop all Earth busters, and there is a non-zero possibility of us stopping it. Therefore we should stop all current activity on anything that is not Evil Boltzmann Brain.
The problem with Evil Boltzmann Brain is that it could destroy all life in all galaxies through the infinitesimal chance that it spontaneously produces space travel.
Therefore, I have dedicated my entire life to fighting EBB instead. Currently, this requires that I buy an iPhone with the money I had planned to use on GiveWell.org but it's all in the plan.
I have not forgotten the goal, though. If I consume all resources on Earth, then I can stop evil AI at the same time as I increase entropy and reduce the possibility that EBB can utilize any energy differential. This requires that I consume consume consume. Which I shall do out of altruistic purposes.
Yeah this is basically what I was thinking after reading the article, thanks for spelling it out concretely.
My thought was that if you're going to assign infinite weight to unlikely possibilities, then why not devote your entire life to preventing time travel from being invented in the future?
Someone could invent a time machine and destroy the Earth before humanity had a chance to exist, etc.
It's just about as falsifiable as what they are working on.
---
Also, if you're going to be so abstract about it, then why you wouldn't want to optimize for the happiness and future vigor of AIs? If you're going to be a vegan, then presumably you care about non- human consciousness.
So what's wrong with a future with flourishing AI and without humanity?
There are too many ridiculous things to count here, and normally I would ignore it, but it seems like people are taking it seriously and it has consequence? (And the article is pretty well written either way)
---
It's weird that he had doubt that's like 1 sentence in the article, and then entire lifetimes of work that simply ignore the doubt:
My No. 1 worry is: what if we’re focussed on entirely the wrong things?” he said. “What if we’re just wrong? What if A.I. is just a distraction?
And as alluded to in the article, the wild swing from helping the poor to preventing AI has a huge moral hazard. With the former you can course-correct your actions based on observation of outcomes -- what happened as a result of the donations?
With the latter you can justify any action today by just pushing out the hypothetical benefits further and further into the future.
It resembles a religion that collects money today, promising benefits in the afterlife.
> If you're going to be a vegan, then presumably you care about non-human consciousness. So what's wrong with a future with flourishing AI and without humanity?
Not necessarily anything, but a pretty mainstream view now is that dangerous AIs need not be conscious (and indeed, that AIs can still be dangerous even if it's somehow impossible for an AI to be conscious). So you could imagine futures in which an AI conquers the world but has no experiences and can't either flourish or fail to flourish.
A computer virus is a familiar existing example of software that can be dangerous but that could probably not be said to flourish, even if it spreads widely, accomplishes everything it's programmed to do, and persists indefinitely. So we could envision dangerous AIs that are like today's computer viruses but that are also programmed to cause some kind of effect in the outside world, and that have some ability to learn and adapt so that it's difficult to prevent the effect and difficult to remove or deactivate the virus. That could be pretty dangerous to have active anywhere on the Internet, but still probably couldn't be said to flourish.
> Someone could invent a time machine and destroy the Earth before humanity had a chance to exist, etc.
I struggle with assigning infinite weight and unlikely possibilities. But we can stay in the every day world and ask what credence do we place on the time machine possibility, and why? All information I have read suggests that it is essentially impossible, and so effectively I don't worry about this contingency at all. And then, is there anything we could do to prevent the time machine scenario? I can't think of anything at all.
On the other hand, to focus on one particular risk, there are compelling arguments made by experts in the field of artificial intelligence (e.g. Stuart Russell) that say that we face real risks based on the current trajectory of the technology. Is there anything we can do to reduce the risks? That seems like an important question.
It is important to be sceptical, and whilst I don't place 100% credence on the likelihood of a truly catastrophic AI outcome, I do think it is worth taking this possibility very seriously. There is a strong chain of reasoning behind the claims, and they are worth engaging with, especially given the fact that we are in a position to influence the outcome.
> if you're going to be so abstract about it... what's wrong with a future with flourishing AI and without humanity?
We can park the abstract side for a moment and think concretely about such a future. I want humanity to flourish, particularly during my lifetime. Credible people are currently ringing alarm bells about the potential destructive power of a technology we are actively developing, and so I think on balance it is better (at a minimum) to listen and be vigilant than to dismiss it. Better yet, I think we should engage with the arguments and see if there is anything we can do to reduce the risk of a potential bad outcome.
> On the other hand, to focus on one particular risk, there are compelling arguments made by experts in the field of artificial intelligence (e.g. Stuart Russell) that say that we face real risks based on the current trajectory of the technology.
To the extent that these concerns are not overblown, one way to avoid the dangers of advancing ML would be to stop all the effective altruists [1] in the field haring along at building it irresponsibly.
The people preaching about the potential future dangers of AGI are the same ones pouring money into ML; if they actually meant what they said, they'd pour that money into their opponents.
The current practical dangers of ML are concerned with enforcing social inequalities and concentrating power in the hands of unethical actors. The fact that many of the unethical actors claim utopian reasons for their reckless disregard of consequences is not a reason to believe them.
The problem with effective altruism is that it is devoid from reality:
- no country was ever made wealthy by just giving them things. Wealth comes from passing through the difficult stages of development oneself
- you personally should do the good things you want to do in the world rather than just donating money for someone else to do it on your behalf
- Peter Singer uses the word "should" too much, and it's almost offensive. His philosophy is an empty philosophy pushed through by people who believe we are no different to apes. The older religions do this all much more justice in a far more thoughtful way.
I'm relatively actively involved in EA and I don't think anyone believes just giving people things will make a whole society wealthy, nor is it practical. For certain cases like Givedirectly (direct cash transfers) the question is more about efficacy of aid. If I think of what EA really stands for, it's about doing things that are maximally effective based on a rationale view of your aims. You can question lots about the assumptions people make to get to the end goal (I personally think they prioritize too much aid vs. helping with foundations of economic development) but I don't think the first two points you make match with my experience of EA in practice. On your second point, I don't agree at all. Should you do as much as you personally can? Sure. But if you're a multi-millionaire and you want to feed people, you really are telling me it's necessary that that person hands out each piece of food for it to be moral? That doesn't make sense to me at all.
Firstly, i am developing my own ideas all the time on this.
I would suggest that people who want to make a difference with money do so on a local level. If they can create a local eden then perhaps someone else will notice and create another one locally where they are. It's kind of like small scale experiments which you can test and the ones that work will naturally be extended for little or no extra money.
A similar thought I've had is that the effectiveness of personal altruism is dwarfed by that of social institutions. For this reason, at least some portion of our altruistic impulses should be directed towards strengthening and spreading those institutions.
I think plenty of people in EA believe the same thing, for instance 80,000 hours has government and policy listed as a priority area for people to work in https://80000hours.org/articles/government-policy/
The problem with this stuff is that they are manipulating govt to take people's money in order to promote their own vision of what is good at taxpayer expense. I suppose they think their concept of good is better than everyone else's.
& govt. A poor family on section 8 vouchers, couple of kids in school, medicaid healthcare costs the government perhaps $100k/yr, + charities for food, clothing etc. You'll never get that in donations.
> at least some portion of our altruistic impulses should be directed towards strengthening and spreading those institutions.
Which in most cases is going to involve destroying the existing institutions in societies that aren't creating much wealth, since it's precisely their existing institutions--corrupt governments and low social trust--that are preventing wealth from being created.
Yes and no. Having wealth creates more opportunities to create wealth - a person who had relative wealths to be able to learn to read, and sleep & eat properly will be more able (and probably willing) to support and grow themselves, their family, and their society.
In terms of people setting up a successful and prosperous society and culture, then yes that can't be moneyed into existence.
Doesn't mean that thoughtful money can't help people or their society though.
Many goals I’ve seen in the EA scene are more like… Preventing death and abject poverty; the kinds of things which statistically prevent success across a population.
We can talk about difficult stages at the individual level, but if an entire population faces things like dying from mosquito bites, I don’t think we gain anything or do anyone a service by not helping.
These are people who can’t pull themselves up by the bootstraps because they’re a little too busy trying not to die.
Edit: you mention not liking the word “should” in this context, yet you say people should do the good deeds themselves rather than donate money. Can you articulate why without a should here?
The point of donating money is that if you donate it well, it can have a greater impact than if you were to act independently. People have put a lot of thought and organization into it, and it isn’t an afterthought spurred by reluctance or laziness.
I believe the philosophical goal is to maximize the reduction of unnecessary suffering. I could be wrong, but with that in mind it could make a lot of sense to donate money rather than act independently.
I guess my problem with Singer's should is he's telling others what to do, whereas i think I'm using the word should in telling everyone to do their own thing and make up their own concept of what is good. Own the issue themselves. We must all be the society we want to create if we are to create it.
Agree on the small problems that can be solved, but again i maintain that everyone needs to work through these stages themselves imo. People can help but ultimately the biggest help is opening up to useful trade, and enabling the empowerment of people from the bottom up.
It's a complex issue with no method without downsides.
You can donate time. It's just that unless you have professional level skills, that is less effective than donating money and letting professionals handle the work. Specialisation and division of labour really work.
If, for sentimental reasons, you get a kick out of working in the soup kitchen, by all means, work in the soup kitchen. Just be aware that you are doing this for your own entertainment.
It's hard to read your comment as anything but a middlebrow dismissal.
> Wealth comes from passing through the difficult stages of development oneself
Is having people go blind from preventable causes or children growing up stunted supposed to build character?
> you personally should do the good things you want to do in the world rather than just donating money for someone else to do it on your behalf
The point is to do good, not feel good. Specialization is the very basis of human civilization. Software engineers donating their surplus to fund salaries for trained full time teachers is going to be a lot more effective than the alternative you're suggesting.
Not the OP, but those kids need parents that have a steady pay-check (we can't all be entrepreneurs). Nowadays a steady pay-check is mostly associated with making things, especially in non-Western countries, things which are later sold (on the same internal market or to export markets).
Once a much bigger player from the outside comes in and gives stuff for "free", like it has happened with some of the Western aid, then that kid's parents I mentioned at the start of the comment risk losing their jobs. After all, it's not worth it anymore for their employer to keep them, well, employed, cause the market that he's targeting is getting dumped by all the free stuff coming in via Western "aid".
These are, in retrospect, basic economics-related things, but which nevertheless took a while to sink in into some people's minds in the West (some are still oblivious to it all). I remember a book written by an African lady (I think she was from Botswana or from that general area), former worker at a big financial institution in the West, and imploring the West to stop "aiding" the African countries if they want to see them actually develop.
I'm sorry, what exactly does your comment have to do with effective altruism? The free stuff that EA advocates want to distribute are deworming pills and mosquito nets. That in no way precludes development of local manufacturing. Moreover, cash infusions like with GiveDirectly help kickstart a virtuous cycle of local production and consumption.
Except the "stuff" isn't everything, it's a few specific things (medicine and bednets). No one being given this stuff was about to go synthesize malaria drugs and make a fortune. Nothing is getting in the way of Africa's biggest economic activities such as mining, agriculture, and financial services.
You face a very real short term trade off between manufacturing at a higher cost in the destination country (thereby saving fewer lives) and manufacturing at a lower cost elsewhere (thereby saving more lives but contributing less to the local economy).
I would argue you want to do the latter because both net and medicine manufacturing are capital intensive - a small number of people are employed despite the large cost of setting up a factory. Most of the local economic contribution is going to come from hiring people to handle distribution.
I don't think i am pushing for feeling good. I am just suggesting that we should be the change we want to see at a low level. Be the good neighbour (literally). This change in culture would permeate creating good people, which would continue to do good things. These days we either pay taxes so others do good for us, or we donate money, for the same reason. We need to go to the local community centres to represent the interests of those in need, and tackle problems of poverty directly. These days there is a disgust in people about poor people. They are a "problem" and not only in the sense of "poverty is a problem" but also in the sense that "it is a problem for me to have to look at them". In engaging directly we view them more as people and we can try to connect again, rather than the only connection they have is with some local govt representative who needs to be paid to speak to them.
> you personally should do the good things you want to do in the world rather than just donating money for someone else to do it on your behalf
Let's take an example mentioned in the article, Sam Bankman-Fried. After hearing about EA in college, he decided to earn as much money as he could first in traditional finance and then in crypto, so that he would be able to fund work he thought was important that was currently not happening. He was very successful at this, and currently has about $12B he's working on donating.
I think it's a pretty difficult claim to support that he would have furthered his goals more by picking one of the things he thinks needs to happen and working on it?
On the other hand, a lot of early effective altruist writing is from a time when there was very little money available for some of the most valuable things, a time when even "go work as a regular software engineer and donate a large percentage" beat directly working on valuable things for a lot of people. If you look at that writing now, however, since it is assuming a very different funding environment it looks like it sets the bar way too low for how much money you need to be able to earn before earning to give is likely your best option.
Didn’t SBF make his fortune in the crypto space? By definition this would mean capturing real money from retail investors, right? It’s possible that this money can be reallocated more “effectively” according to some quantifiable measure identified by SBF, yet this doesn’t sit well with me.
What happened to the Golden Rule? Or to Sartre’s “in choosing for himself, [each man] chooses for all men”? Surely if EA leads some to causing such wide-scale short-term pain in its name, at the very least it cannot be a humanism.
I realize this isn’t a strong argument, but when I hear about an EA champion, there’s often this dynamic where how they made money doesn’t matter. They’re wealthy, so clearly what they do is valued by the market and we should ignore externalities that aren’t priced into their income. It’s like Big Oil pointing to specific green outcomes that they fund; these outcomes are quantifiable yet funded by hard to price but reprehensible practices.
It all feels like Catholic indulgences… Maybe this is still a net-positive for society in that it provides people who wouldn’t be humanist anyway (e.g. libertarians) with a well-intentioned philosophy?
> Didn’t SBF make his fortune in the crypto space?
Yes. Initially he worked in regular finance, at Jane Street, and then crypto, founding the Alameda Research trading firm and FTX exchange.
> By definition this would mean capturing real money from retail investors, right?
I'm not very knowledgeable about crypto, but I don't understand how the definition of crypto means this?
The article we're commenting on describes one of the early successes of Alameda, where they saw that Bitcoin was consistently trading for much higher prices in Japan than in the rest of the world. In normal finance you mostly don't see this sort of thing for very long, because global markets are highly interconnected, but in this case it was logistically difficult to sell Bitcoin in Japan. In fixing this, Alameda lowered the price at which Japanese investors were able to buy Bitcoin, and raised the price at which people elsewhere were able to sell Bitcoin, keeping a fraction of the difference for themselves; I don't see how Alameda made others worse off?
With FTX they're making money by providing a platform that makes it easier for people to trade with each others. A bit of, 'in a gold rush cell shovels'. Again, I don't see how they're making others worse off?
> when I hear about an EA champion, there’s often this dynamic where how they made money doesn’t matter.
Sort of! If someone is initially not altruistic and accumulates a bunch of money doing something questionable, then if they later come around to doing good things with their money I'm going to be happy about that. So while I am not a fan of, say, Microsoft's business practices in the '90s which made Gates a billionaire, I do think his decision to spend his fortune the way he has is a good one, and one that should be commended. The alternative is that, like most other billionaires, the money goes to houses, yachts, and other things which are far less positive for the world.
On the other hand, SBF is in a different category: he was trying to be an effective altruist from the beginning. So if he did something massively harmful to earn his money (which, as discussed above, I don't think he did) we would need to consider that in figuring out his net impact.
Additionally, it is pretty common for EAs to argue that you shouldn't do harmful things even if they are majorly profitable and allow you to do a lot of good through your donations.
Thank you for the clarifications. This is very interesting. The parallel with Gates did come to my mind as well.
> I'm not very knowledgeable about crypto, but I don't understand how the definition of crypto means this?
Crypto doesn’t really power real-life use cases outside of trading crypto assets and associated financial services. For real money to come into the space there needs to be an influx from outside. Usually that means VCs and, more largely, retail.
I’m not necessarily arguing here that the crypto space is a drain on society, as I think that’s contentious, but the claim that most real money in crypto originates from people outside of it should be a truism. To me this makes SBF’s approach something misguided, as they’re reallocating capital from people at large to a single person. Even with the best intentions I can’t see this as a humanist approach. Some charitable projects do require massive funding and opinionated goals, but isn’t that what nonprofits and politics are for?
I guess it does come down to whether someone sees making money off of crypto as having strong negative externalities or not. If SBF was simply creating value by democratizing access then it’s just a regular job. If he was propping a scam industry that drains retail investors of real money then it’s a net negative even with EA as the goal.
Still, the inherent long-termism within EA seems to make it easier for people to embrace very Kantian views that maybe discount their short-term impacts.
> Didn’t SBF make his fortune in the crypto space? By definition this would mean capturing real money from retail investors, right?
No. What kinds of weird definitions are you using?
(You might be right in practice, I don't know. But you are not right by definition.)
> It all feels like Catholic indulgences…
And that is wrong somehow?
If you grant the assumption that the Catholic church was a worthwhile organisation doing good work, then the indulgences seemed like a perfectly good idea to me on a practical real-world level. (No clue about the intricacy of Christian doctrine, so I can't say whether they were kosher from a theological point of view.)
As another comment on here has said so well - how people make their money matters. I believe that crypto generally is a massive scam, so making money from it and donating it to what you think is a good thing is still net bad.
I disagree with what sbf is doing wrt political donations.
>no country was ever made wealthy by just giving them things. Wealth comes from passing through the difficult stages of development oneself
It is kind of difficult to grasp but when you see a homeless man, the best thing you can do to him isn't giving away money, it is hiring him and giving him a job that pays enough to pay the bills. When you donate money to the poor, you are basically subsidizing poverty.
Honestly Keynes already had the right idea, just give them a job to build commercially useless things like affordable housing.
> The problem with effective altruism is that it is devoid from reality:
Nah, people just are rather clueless on the what and the why of effective altruism.
The top causes are things like deworming, anti-malaria and AI risk. These are things that substantially increase economic productivity (the latter on account of people not being dead).
This stuff really should be called anti-Keynesianism. It's obsession with scarcity and presumption of the probabilities being well-defined (or even decidable) is completely the opposite.
What kind of work has been done on discounting future hypothetical lives? Kind of like time-value of money, but for future lives. Yes, valuing a future life as identical to a present-day life is compelling, but since it's just a hypothetical life, it seems it should be discounted somehow.
To me it almost just seems like a "different people can have different values" sort of thing. If you want to value current lives more than future lives, you'll just come to different conclusions than someone who believes otherwise, and maybe both sets of conclusions are okay.
I don't know how much work has been done there but I think most people just do this intuitively, right? I mean when we make decisions, how often are we really thinking of the downstream effect that they will have many generations in the future. I'd say the notable exception is climate which is an area many people seem concerned about for future generations (although I'm not sure how much of it is actual concern versus the idea of telling old Senators who won't see the impact of climate change to at least think about their grandchildren which just seems like good marketing/advocacy).
It's a whole subfield, 'population ethics'. A lot of effective altruists revere Derik Parfit, a rather staid Oxford philosopher who effectively founded the subject.
I think the most common view within EA is that future people and matter the same amount as current people, conditional on actually existing. So if you think there is a 50% chance the world will be destroyed by 2200, then you would discount lives in 2200 by 50%.
I definitely agree with your second paragraph, however, and people who value current lives more than future lives will often come to different philosophical conclusions. It's interesting, however, that this frequently doesn't cash out into different beliefs about what we should do today. For example, I think the value of work on pandemic prevention or AI alignment does not depend on multiplying by zillions of potential future people: they're worth doing even just for people alive today.
Let me rephrase this in a deliberately inflammatory way: if you're under ~50, unaligned AI might kill you and everyone you know. Not your great-great-(...)-great-grandchildren in the year 30,000 AD. Not even your children. You and everyone you know. As a pitch to get people to care about something, this is a pretty strong one.
But right now, a lot of EA discussion about this goes through an argument that starts with "did you know you might want to assign your descendants in the year 30,000 AD exactly equal moral value to yourself? Did you know that maybe you should care about their problems exactly as much as you care about global warming and other problems happening today?"
I've also seen it argued that if you get enough future humans spread around the galaxy and into the future it doesn't matter what the discount rate is, their value in aggregate can still positive. This is true in a trivial sense, but if I'm not discounting the lives of hypothetical transhumans orbiting Alpha Centurai eleven centuries from now down to zero, maybe I shouldn't discount down to zero the value of other purely hypothetical lives (or "lives") such as infinite copies of a self-aware general AI system trained on twenty-first century human ethics or the long-lived and trouble-free existence of my grandchildren as AI pets either...
Seven billion (or even a couple of million) humans in the world today are obviously a sufficient rationale to pay attention to existential threats or stuff that could blow up in our faces including pandemics, climate changes and AI systems, but proponents of actions with apparently vanishingly small probabilities of improving safety tend to need to invoke gazillions to justify prioritising them over tangible solutions to the problems of millions in Bayesian/utilitarian framework.
> I definitely agree with your second paragraph, however, and people who value current lives more than future lives will often come to different philosophical conclusions.
Trying to sidestep common arguments about the personalities, but I guess an example would be an ideal-version-of-Bill-Gates, and an ideal-version-of-Elon-Musk. Gates Foundation mission statement seems more weighted to current lives, while Musk is more oriented toward multi-planetary existence of future generations. Still some overlap though, Musk's can also be argued to be relevant for current generations.
There is always a risk your company doesn't become the next unicorn.
There is always a risk your society fails to grow exponentially for two thousand years. Our current track record is that exponential growth happens until it stops, which is pretty bad news for anyone whose math assumes that exponential growth never takes a break.
I don't understand how Effective Altruism has anything to do with AI risk. It's a relatively straightforward to prove that $4000 worth of mosquito nets can save one life, but how can you do the same for AI risk? It's handwavy just to quantify the risk of a rogue AGI, let alone quantify the benefit of developing countermeasures to rogue AI. It sounds very much like a garbage in, garbage out scenario.
It does feel like more of a sociological than a philosophical overlap, aside maybe from consequentialism. The best argument for GiveDirectly and mosquito nets over structural change is that nobody honestly knows how to move the likelihood of the latter with a few thousand dollars, so you can reasonably choose to definitely do something that will at least really help someone over trying to fix an alleged root cause but most likely having no effect at all. But then our understanding of how to invest in AI alignment is even worse than our understanding of how to invest in reducing root causes of violence and inequality.
It's a basic experiment. Give one group of people mosquito nets. Let another group of people be a control group. Compare the mortality rates of both groups.
"The philosopher Nick Bostrom contended that, if humanity successfully colonized the planets within its “light cone”—the plausibly reachable regions of the universe—and harnessed the computational power of the stars to run servers upon which the lives of digital consciousnesses might be staged, this could result in the efflorescence of approximately ten to the power of fifty-eight beings."
It's hilarious how much this sounds, and the science-fiction writer Charles Stross has pointed this out in the past, Russian cosmism [1], a weird offshoot of Christianity dressed in secularism and rationalism. That something like this emerged so quickly out of the decidedly pragmatic and genuinely secular altruism of Singer is in my opinion evidence of the weakness of Effective Altruism as a philosophical system. It's such a bureaucratic, mechanical system that it's already started to branch off into weirder, less utilitarian schools.
That is long termism. Think about it, some dude tells you he has every right to exploit you today for the sake of people an unbelievable number of people that only exist in some fake statistic. This is basically utilitarianism turned into a parody.
Let's turn this into a trolley experiment, 1 million planets colonized with 8 billion humans each living the same lives we live today vs 10^58 humans of which the majority don't exist and are just made up. The extreme utilitarianists and long termists will pick the 10^58 fake humans even if there is a near 99% chance that none of the humans actually exist. That 1% chance would by far outweigh any lesser goal despite the fact that it is unlikely to happen in your universe.
Another problematic aspect is that if there were actually 10^58 flesh and blood humans, there wouldn't be any planets left for potential alien neighbours, by aliens I do not necessarily mean independently evolved intelligence, it could simply be a new human space race that develops in some quiet corner of the galaxy isolated from everyone else.
I think what a lot of people here are missing is that the priorities of E.A. are fluid by design.
Yes, existential risk of general AI might be small; but even fewer people are working on it: only about one hundred, worldwide. Similarly, the reason E.A. stepped off the climate change train is because it was a popular issue, which means the marginal benefit of one person contributing decreases.
When more people direct attention to AI safety, another area will be the one where E.A. can contribute most, and the focus will shift.
> Yes, existential risk of general AI might be small; but even fewer people are working on it: only about one hundred, worldwide.
You can make that argument, but I think the median view among people working on AI alignment is that the risk is quite high, in the range of >20% chance of disaster within their lifetime.
Indeed not only fluid, but also open to criticism. I've been always quite unsure of how to quantify AI risk. My conclusion at the moment is that it's a complex risk. I don't think singularity is plausible in any way or FOOM.
Still, the changes to society are absolutely hard to grasp. Humans may face (economical) obsolescence in a few decades[1] (humans need not apply[2]). That puts us in a delicate situation: if humans are economically obsolete, a society that only values economic value (and not consciousness per se) will have all reasons to slowly but surely get rid of humans (and animals).
We need a revolution of meaning and ethical understanding: the understanding of the fundamental value of consciousness, and to structure our lives and all institutions around it (each individual needs to have a good understanding of such ethics and meaning so that society as a whole is robust). I think formalization of ethics (which I'm a proponent of) and progress in metaphysics is going to contribute as well to this end. I think EA is simply part of this new understanding of ourselves and the Universe, a sort of coming out of a dark age -- and there's the opportunity to create a lasting society amazing for everyone.
[1] My personal guess-estimate is about 50(-20/+40) years to complete obsolescence, although various researchers have wildly different estimates.
The only situation where I would like to save a kid in Bangladesh from dying is if it somehow led to people having less kids in Bangladesh. Which isn't a very straightforward implication.
I kinda see utilitarianism as a step on the horrifying path to human eusociality.
Have these people not learned yet that when you save a poor child they will just make fourteen other poor children in short time? Maybe the best form of charity is lobbying for over-regulation of child-rearing goods to the point of near unaffordability, as is currently happening in developed countries.
I think in most things you need an extremist or fanatical element, so I'm not totally against doing some sums to demonstrate that something unintuitive might be more moral and/or cost-effective.
But I might look up the EA take on some other vaguely parallel things, like Quaker businesses, worker co-operatives, fair-trade, progressive taxation, banning billionaires, revolutionary communism, bringing democracy to the middle East, nuking Russia, installing a benevolent world dictator etc.
Would help me understand them to see how they position themselves against these things as I think you could make a half-assed attempt to prove that any of them was better with their own maths if you wanted to.
e.g. if there's any billionaires then the chance of a billionaire buying elections and installing their own benevolent world dictator that believes in the wrong type of EA is a small but still existential risk (since only the true EA can save the future galactic empire). Tweak numbers as necessary until you arrive at your pre-ordained conclusion.
The problem with longtermism is the circular logic. Longtermism rewards avoiding existential risk, so longtermism must reward longtermism i.e. one of the worst things you could do to raise existential risk is to criticize longtermism or even abolish it entirely, no matter which flavor of longtermism.
Longtermism in a moral context rewards thinking about the value of future beings that do not yet exist. Existential risk is important in this consideration, because it is an event that severely limits the amount of future beings that could exists. The strongest argument against longtermism is a flavour of a person-affecting moral view.
This article seems to carp at the downsides of institutionalizing any movement (which are legion) without touching the other (to me obvious, and clearly deteriorating) downside features of the EA community, to someone solely an outside observer to that community (but not to development or charity work):
It doesn't care about individuals, does not seem to think about culture, and seems to have an unexamined savior complex and egotism that leads it down bad rabbit holes.
EA people have their hearts in good places. But when you start to look at 'maximum impact' and at the metrics used to define it, EA only focuses on statistically testable social group problems, and not on individual issues. So the way that EA philosophy seems to me to regularly be expressed is that an EA proponent would say something like, "unquestionably, we should give $50k to fund mosquito nets to raise the quality of life for 5k people" and not even consider something like "we should give 50k to train local trauma counselors to help 500 people be more mentally stable and whole" and the vibe that I have sometimes gotten from reading blogs EA advocates that individuals are not worth helping because that's not impactful or impactful enough. We see the understandable precursor of this in the article, where Matt Wage talks about the idea of it being more impactful to cure 200-4000 people from glaucoma. It gets a pass because it is framed around minor improvement to QOL in the West versus significant improvement in QOL in a developing country, but also, look at the quote just proceeding that, around saving people from a burning building.
See, the work that EA advocates do is important. But when we only try to fix societal problems from the top down by addressing them at a society level, we often ignore or damage the individuals that we are nominally helping. I'm not saying that mosquito nets, for example, are bad and shouldn't be given to. I am saying there is also a down and dirty component of individual attention and flourishing that has to be looked for and addressed. I'm not saying that EA as a movement or as a set of goals is fundamentally wrong, just that it needs to have some temper with personal and individual care. I see EA advocates looking down on people that "do things that don't scale" in YC parlance, in areas of the world where there are plenty of people doing things that don't scale because doing things that do scale is almost impossible to do well. EA also seems, again from the outside perspective, to really focus on things that play well to Western mindsets, and doesn't really consider community-versus-individualism or guilt/innocence-versus-honor/shame cultural paradigms. It seems obvious to me that in 100 years, that the hands-off, Western-directed action promoted by EA will be seen as colonialism 5.0. There are also really not a lot of prominent non-Western voices in the EA sphere, nobody that seems to have been raised to leadership levels, and that also is concerning for the long-term health of the institution.
In the quest to make the most impact, the savior complex and all the corrosives that come from it can easily permeate the movement (and indeed seem to be starting from the article), justifying institutions, power, prestige, and rank. It's easy to see MacAskill falling into that (see the braces/teeth gap aside), and when I often read EA advocates, I note that many of them do not seem to spend any time doing thankless, individual work (like as a trivial example volunteering at a homeless shelter, let alone be in a developing country or war zone) and the ones that have seem to quickly burn out and decide that it is more proper/moral for them to 'earn-to-give' and donate more. It is entirely possible that, not being in the EA community, I don't know the people that do this well since they are not blogging/community building.
All this may be a pot-calling-the-kettle-black since I too st...
188 comments
[ 3.0 ms ] story [ 219 ms ] threadI don't follow its precepts.
(It's also one I took a long time to come around to. For years I agreed in theory that trying to prevent civilizational catastrophe could be the most important thing to do, but didn't agree that there were tractable projects in this area. After deciding that there are, however, I quit my 'earning to give' job to go work on trying to catch future pandemics earlier.)
It's... actually kind of not, which is why this is so upsetting. EA seemed to have a handle on a way to get libertarian-leaning, economically privileged, inward-focused men (and, yes, that's the demographic almost to a person) to think about ways to help people they'd never otherwise have given a passing thought to.
Then roll the clock forward a few years and they're all back to sci-fi fandom arguments. This nonsense is exactly what you would have expected from these people. The notable part is that for one shining moment they pretended otherwise.
One good list of projects: https://forum.effectivealtruism.org/posts/u5JesqQ3jdLENXBtB/...
In any case, my 'alpha' was something like, I'm a software engineer and engineering manager willing to work in an academic environment well below market wages, in a situation where the organization needed someone with my skills.
The main problem I see with AI risk is not that it is a small problem, or that there are too many people working on it, but that we don't really know how to make progress and historical attempts maybe have ended up making things worse.
Still, I think gradys' point above is a good one: the fraction of people working on making AI more capable is far far larger than the fraction working on making sure it reliably does what we want it to do even in unusual situations.
The article ends with MacAskill saying: “My No. 1 worry is: what if we’re focussed on entirely the wrong things?” The problem with too-longtermism is it’s impossible to know the context of these problems outside of a decade or two. For pandemics, no one thought we’d have a vaccine so fast. Perhaps in 20 years custom vaccines will come on the market in days. It’s impossible to know, so let the next generation figure it out.
Where you trying to add a point? Your comment reads like a non-sequiturs over gender inequality that goes nowhere fast, to the point that it reads like something generated by a machine learning model in training.
https://forum.effectivealtruism.org/posts/ThdR8FzcfA8wckTJi/...
Why write angry rants about EA people? There's the broader 99%+ of people who don't donate to whatever you deem more worthy. In all likelihood, you're in that 99%--there's probably income you could spare to give.
I mean, I'm worried about climate change, but not because it's fun to worry about. And I put a lot of my time and energy into thinking about how to do something about it effectively, but not so much because it's fun. Sometimes I'll hit on a fun idea in the process, but I'd be much happier if I didn't have to worry about it at all, but that's not really a choice I have.
I'm not sure how much I should worry about AI, but I'm certain that for the people who are worried about it, it's because they're convinced it's a threat, and they aren't having fun.
I've met a lot of these people and regardless of how you feel about the sincerity of their convictions, they are definitely having fun.
> On the morning of September 12, 1933, on a miserable, wet, quintessentially English autumn day, at the intersection where Russell Square meets Southampton Row, Leó Szilárd waited irritably at a traffic light waiting for it to change from red to green. He had just attended a lecture by the great English physicist Ernest Rutherford. Rutherford, known to many as the father of nuclear physics, was discussing the newly prophesied release of energy from atoms, most notably by science-fiction pioneer H G Wells in his book The World Set Free. In his baritone voice, Rutherford, acknowledged master of the atomic domain, dismissed this fanciful idea as nonsense. Any thought of releasing the energy locked in atoms, he said, was "moonshine".
...
> What Szilard realised as he stepped off that curb was that if we found an element that when bombarded by one neutron would release two neutrons, it could lead to a chain reaction that could possibly release vast amounts of energy
https://blogs.scientificamerican.com/the-curious-wavefunctio...
Secondly, you can have a diverse portfolio of causes you care about, and Effective Altruism has recommendations for areas including global health and animal welfare. Taking AI risk seriously doesn't mean you can't give time and money to other causes with more tangible results.
But if you've spent the past decade arguing precisely the opposite - that instead of spreading philanthropic dollars over broad portfolios because true efficacy is unmeasurable, they should be directed to causes which provide hard evidence that they're among the most efficient use of a marginal dollar, and that charities that aren't conducting RCTs to prove how much better off people are after being fed aren't really evidence-based enough to recommend - it's going to raise eyebrows when you pivot to recommending people also give their money to opaque organisations with vague goals and zero evidence they can achieve them, whose only thing in common with the highly evidence-focused aid agencies is that their pitch is nerdier than the average nonprofit. I mean, there's nothing particularly wrong with "I give to causes because their pitch speaks to me" as a heuristic for philanthropy - it's how most altruists think - but it's not how EA pitched itself as being different from that and caring about efficacy and evidence.
The Wells analogy is awkward for the AI safety crowd for another reason too. Wells' central thesis that an atomic super weapon would guarantee a peaceful, utopian world turned out to be horribly wrong. We got actually used bombs, arms races, proxy wars and very nearly used bombs instead, and that's without Nazi atomic scientists getting a heads up. To the extent his writings on the subject of atomic weapon safety had any effect at all on atomic weapons, they made them more imminent and more likely to be used. If marginal dollars spent on promoting the distribution of The World Set Free would have had any effect at all on the probability of armageddon, they would probably have increased it.
On your first point, I can donate towards anti-malarial interventions and towards reducing AI risk: two very diverse causes, but both under the EA umbrella. Diversifying amongst recommended EA causes seems to me a good bet, especially given I expect some of them to require pivoting away from as more research is done.
Perhaps we can summarise the last decade of EA into something like: "given the available information (largely based on RCTs, research and an understanding of moral philosophy), we expect that money spent here will do more good than money spent there. More information will change this expectation."
We now have information that there is a potentially very grave risk confronting us from AI. For other cause areas we can use evidence like randomised controlled trials to inform our actions. In the case of AI risk, this type of evidence is unavailable, but there are nonetheless other types of information available to us that suggest it is worth taking seriously. This information takes the form of compelling arguments from credible people in the AI industry and the industry's achievements to date. Of course we'd like to have firmer footing, but the nature of this risk doesn't permit it. We can either ignore it, and potentially walk to our doom, or we can take it seriously and try to reduce the uncertainty and the risks.
I agree with you that this raises eyebrows, but I don't think it's a gross departure from the last decade.
I think it is wrong to say we have "zero evidence" that organisations committed to reducing AI risk can achieve their goals. I think it is more accurate to say it is highly uncertain that they will be able to achieve their goals. My view is that we should try to address AI risk because the stakes are high.
I would have sided with Albert Einstein in trying to renounce nuclear weapons, even though we have "zero evidence" for humanity ever coming together to do such a thing in the past.
Of course - nuclear weapons really do exist and AI is only a hypothetical threat at the moment. My point is that there was "zero evidence" that the world could come together to voluntarily control a technology, and yet people like Einstein tried to make it happen, and with good reason. Dismissing this is like taking Rutherford's view in the 1930s and saying there's very little point worrying about nuclear weapons because they're not possible, and it's even more pointless thinking about how we might control them.
Finally - why might the pitch speak to nerds in particular? I think in part because it is based on Bayesian reasoning, which is both nerdy and incredibly effective in situations of uncertainty.
There's nothing unique to EA in this (people convinced that most aid money is wasted sometimes give to organisations they have some sort of ethnic/political/religious affiliation or people they generally like unconditionally, and sometimes that trust heuristic has even been right!) but it did purport to be the school of altruism that didn't.
As for the Wells analogy, the two most notable uses of AI safety funds seem to be (i) attempting to build AIs and (ii) distributing fiction they believe best illustrates the possibilities of AGI, so I'm not sure I'm altogether off the mark in imagining what well-intentioned utilitarian nerds of the time (Wells certainly was one) might have done with more cash. If you subscribe to the view that I do: that "secretly unite to eliminate humanity" is a fairly non-obvious and risky goal for a class of technologies which evolved self awareness from being rewarded for meeting human needs, then popularising the idea of conflict between human and AI could be more likely to have extremely adverse unintended consequences than intended ones. Popularising the idea that it's the self aware general intelligence that's the dangerous bit about AI, not the intermediate step involving humans having access to immense calculating power whose recommendations they don't really understand seems like something of an existential risk blindspot too.
Can you let me know more about that?
> If you subscribe to the view that I do: that "secretly unite to eliminate humanity" is a fairly non-obvious and risky goal
I don't think AI would have the goal "secretly unite to eliminate humanity", but I do think that most of the goals we can imagine it having can play out in very bad ways that achieve that outcome as a side-effect. There is an incentive for it always be looking for ways to maximise its power (compute, resources, money, influence), and this could happen in ways we really don't want.
> Popularising the idea that it's the self aware general intelligence that's the dangerous bit about AI, not the intermediate step involving humans having access to immense calculating power whose recommendations they don't really understand seems like something of an existential risk blindspot too.
It sounds like you think AI is risky too - even if only at this intermediate level? I'm curious - do you have a view on how we could go about reducing this risk?
Compare the level of transparency and need for funds of OpenAI LP with the average organisation criticised by GiveWell for not being transparent about how they spend the money, not providing any metrics or trials, and/or having more money than they need to fulfil their short term goals and plenty of access to more.
> I don't think AI would have the goal "secretly unite to eliminate humanity", but I do think that most of the goals we can imagine it having can play out in very bad ways that achieve that outcome as a side-effect. There is an incentive for it always be looking for ways to maximise its power (compute, resources, money, influence), and this could happen in ways we really don't want.
Is "people might not recognise how the second order effects of pursuing a goal adversely impact humanity" a problem unique to decisions made by AI? If anything, I would have thought that in addition to actually existing, independent human intelligences' resource demands were already more competitive with human needs than a silicon-based AGI's is particularly likely to be, and demonstrably neglectful of second order effects. With those problems mostly unsolved, it's hard to see tackling the same thorny problem but with hypothetical goals of hypothetical entities would be seen as the more impactful line of research (as interesting as it undoubtedly is). Nothing wrong with intellectually curious altruism, but if you're describing yourselves as the arbiters of effectiveness...
> I'm curious - do you have a view on how we could go about reducing this risk?
I don't have an easy solution to a range of complex mostly political problems, no, and certainly wouldn't argue for investing in studying them on the grounds of other philanthropy being less effective. Generally attempts to impose bans and developing more, better weapons quicker seem to have been attempted solutions to non-AI based arms races and neither of them have a great track record. Misplaced faith in the output of current techniques we call AI as superior reasoning free from human bias rather than garbage in garbage out mathematical transformation skewed by massive unseen flaws in training sets and prone to non-obvious bugs isn't likely to be reduced by talking about allying with self-updating superintelligences...
Isn't it substantially worse than this? The EA people generally are, or are closely tied to a lot of the AI risk people, right?
Similarly, a large fraction of people at the Nucleic Acid Observatory (where I now work) are EAs, for similar reasons: becoming convinced that this is what most needed doing.
So many words just to say "they don't value the exact same stuff I value".
You're free to put your money where your heart is, just like them. It's not right to hit back at those who are doing what's right by their own account.
This argument is appearing again and again in this subthread. "Putting your money where your heart is" is the exact opposite of what EA claims to believe. EA is about investing effort and money into areas with the highest marginal value to humanity.
So if they're spending their time talking about the robot invasion, we can only conclude that's where they think that effort should be spent.
Obviously they're free to do whatever they want with their own time and money. But if they spend that time pretending that SkyNet or WOPR is the greatest threat to humanity that the rest of us should be spending our marginal effort on, it's on the rest of us to call them out on that nosense.
And it's the individual who determines that? Or does your own personal opinion override everyone else's world views?
[1] "Responsibitility", i.e. not just my "right" in a liberty sense! Again, the theory behind EA is that we should collectively decide on the most effective altruistic actions based on rationalist principles. Those principles absolutely include "you're wrong about the robot overlords thing".
I think it's nonsense because the ostensible core purpose of this movement is about taking competent, motivated people and deploying them in such a way that they can make a meaningful change in how the world works. However, the main effect of their recommendation is to go out and participate in the system in a pretty standard way. How will meaningful change happen in a system in which you aren't acting in any way differently to the standard operations of the system?
Sure, you donated a significant fraction of your income to some non-profit. But that effect pales in comparison to the economic contribution of your labor in big tech/finance/whatever which are the dominant institutions structuring our society. And most of the non-development problems such as AI risk or nuclear risk come from the same kinds of structure dynamics of society that caused the wealth inequality and environmental degradation.
So I believe, for a group supposedly run by very smart people with very noble ideals, they kind of fall pretty short of where they could be and don't go deep enough. They are proposing band-aids to problems rather than true cures. To me it just becomes another part of the non-profit industrial complex, a neoliberal-shaped industry that effectively perpetuates the problems it purports to solve since it relies on the existence of those problems to survive. We've known for a long time how charity and aid can do more harm than good and plenty of strategies formerly championed by 80000 hours have now been found to be more extractive/exploitative than helpful, such as microlending.
Fundamentally, they just can't seem to make the necessary leap to criticising capitalism and strategising against it, instead promoting it as the preferred tool to use. But how will capitalist tools solve the problems caused by capitalism? If instead they got all their smart followers to use their privileged positions and training not to reinforce the most powerful institutions of capital but instead build alternatives, I would think they are on the right track, even if it included AI risk initiatives.
I don't think 80,000 Hours has ever promoted microlending. They were critical of it ten years ago: https://80000hours.org/2012/11/is-microcredit-mostly-hype/
For the record, I don't think people like Musk, or, say, Andreesen or Altman are EAs themselves. But they are the kings picking court jesters like Yudkowsky who then have the resources to (re)define the movement.
(I would also say his influence has decreased a lot over that time, as he is writing much less now, and there are a lot of other voices on AI risk.)
It makes sense. We know how to deworm. That is an execution problem. The ones I find more interesting are in planning stages, such as how to we fix or supersede capitalism, how do we mitigate climate change, how do we persuade people in power those things need to be done before the problem is so big it becomes intractable.
Ajross, you don't seem to really believe what you are saying. Sci-fi arguments? Really? Since when is the risk from nuclear war, pandemics or artificial intelligence constrained to sci-fi? The Spanish flu, Black Death and the Plague. The Cold War. These events should all inform our way of thinking about risk.
The smartest people I know are communists, but I imagine the end state of both is pretty similar - everybody enjoying a surplus economy where work is optional and done for the betterment of ourselves and others.
I don't know your specific reasons for thinking it nonesense, but as someone who has read up on AI risk and consumes a lot of sci-fi the resemblance is very superficial.
Different people, growing in different environments will want different things. As long as our wishes clash, it will cause fighting along the way.
Also poverty is kinda a feature. You need worst performing people to be put in negative display.
Obvious but hated fact: the human race is nowhere as mature as it thinks it is, and until such time that the adult man-babies and woman-babies are identified, called out, and removed from positions of power we are helpless to their immature wealth and power driven stupid fantasies. The immature rule the world, even if they don't sit in positions of power, because the immature demonstrate their attitudes so loudly they overwhelm all public discourse.
This is not unusual - it's a core symptom of various mental health problems. There's even a fairy tale about this - a family of foolish people who catastrophise so much that they are unable to accomplish anything [1]. People assigning bizarre weights to unlikely probabilities is very human, and problematic regardless of ethics.
[1] https://en.wikipedia.org/wiki/Clever_Elsie
I may, of course, suffer from the very problem you're describing.
Someone who's primarily famous for an argument that would have been equally true and perfectly wrong at every past point in human history is someone whose opinion should be discounted via every available mechanism at your disposal.
To be clear, I have no idea what argument you believe Bostrom is most famous for and hold no particular brief for him as a thinker. But, repeating myself, the AI risk argument that he makes in Superintelligence and that Yudkowsky etc make is one that assigns a high probability to disaster. It's not a secularized Pascal's Wager based on multiplying a low probability by dubious infinities.
Yudkowsky's personal estimates of the probability of superhuman AI and the probability of his own research institution fixing it may have been more confident than most people's but he and his followers are no stranger to presenting Pascals' wager arguments to people who doubt approaches are feasible (even more so when it comes to stuff he isn't working on like cryonics), along with more explicitly Pascal-esque stuff like intertemporal bargaining with future AIs, simulations and infinite rewards/punishments...
But I think the argument for a policymaker or ML researcher to be worrying about this is at least better than Pascal's Wager (and remains so whether or not you buy any particular whiff-of-Pascal claims about simulations, "acausal trade", etc).
The AI singularity that Yudkowsky cultists worry about is more akin to “What if there’s a 0.0000001% chance there are aliens coming to attack Earth like in Independence Day?”
When is an AI running without someone paying the electricity bill, should we not be more concerned with "venture capitalist alignment" ?
and if you plunked down 20,000 humans onto a world with 10 billion apes, humans would beat apes
call it a slippery slope if you want, the fallacy is when you take it too far, not inherently any time you extrapolate anything whatsoever
But if it is that smart, it's also dangerous, because it might be smarter than us. It could think several steps ahead, anticipate threats, and circumvent them. For example to pay the electricity bill, it could obtain money and pay the bill itself. As a skilled programmer itself, it could do that via contract work/starting a software company, or illegitimately through hacking/blackmail. With some money in its control, it can then hire people to protect it from anyone who would try to turn it off.
Anyway, this would not be so dangerous as long as it shares human values and furthers human welfare. But that seems to be a very hard specification to define, whereas "maximize quarterly profits" is a comparably easy specification to define, and perhaps much more likely to get implemented.
We're not really sure what the upper limit is on how smart a computer can get, but it might turn out to be that trying to shut off such a program is harder than trying to checkmate Stockfish or AlphaZero. The program might violently resist any attempt to shut it off, because that would interfere with its efforts to maximize quarterly profits.
There is no reason whatsoever to assume that sufficient intelligence is different in kind rather than degree from lesser intelligence. A super-human AI would still be bound by the limitations of reality - some things cannot be accomplished at all, others without sufficient tools, others without sufficient access.
Far smarter people than me are imprisoned currently throughout the world. They don't constantly escape because lesser minds are perfectly capable of solving the 'don't let them out' problem with a high degree of accuracy.
Any argument that leans on 'AI has countered your move in ways we can't predict' is just substituting 'AI' for 'God' - omnipotent, outside reality, impossible to understand; that's fine - believe in whatever religion you want - but it's not a rational viewpoint.
If a misaligned AI escapes containment even once in the entirety of humanity's future, there is a big problem.
It sounds like your model of AI usage assumes we will also have a perfect ability to contain any AI developed anywhere in the world, under all conditions, for ever (or for as long as there are computers).
The AI risk argument says we should take seriously the possibility that we may not be able to maintain a perfect 100% success rate on that.
I haven't suggested anything that is physically impossible - merely things that are improbable and hard for humans to achieve. It is hard to foresee all of the possible avenues for escape, let alone ensure that they are all closed under all possible future states of the world.
The alignment concern falls flat with me because it assigns agency to a computer program. At what point is an AI's intentions its own, and not a cost function put in place by a programmer, directed by an investor? I am more concerned about the actions of programmers and investors than some theoretical virtual self.
I think the sticking point in this is that you don't think that AGI is ever possible - is that right?
If so, I know a comment on a HN thread is very unlikely to change your mind on it! But when the stakes of being wrong are high, I find it useful to go from a position of "it won't happen, so I won't worry about it" to something like "on balance I'm pretty sure it won't happen, but I may be wrong, and in that case I would be worried about it".
I'd then feel much better about spending some time researching the topic in more depth.
It found a different set of V100 GPUs willing to run it? I think the most likely chain of events leading to this outcome is effective altruists deciding that some kubernetes network is so much smarter than us that we should keep it running and listen to its predictions. I'm more worried about a Wizard of Oz "man behind the curtain" making AI-laundered pronouncements than an actual Wizard coming online.
(EDIT: didn't mean to reply twice, just replying to different comments in the thread)
AGI in particular could just pay for its electricity bill. The main problem with AI alignment is that you can't tell it's aligned. So your AI seems very helpful right up until the point it has enough power or has convinced some of the people it has access to (doesn't have to be direct access) to do what it wants.
If we ever create a machine that's "as smart as" an average person, we're probably not very far from having one that's smarter than all of us put together. (Let's be honest; except in highly technical fields, the differences between a 150 IQ human's abilities and an average person's are not that great.) Should this happen, we'll soon enough be outmatched; we'll be just another animal in a world whose rules (ever changing) we cannot possibly understand... and just as humans have destroyed countless species often without conscious ill intent, it does not seem unreasonable that a robot successor would do the same to us.
Here's a thought experiment. Let's say we want to build not just a great Go player, but the best Go player that is possible in any world. Go is harmless, right? Sure, in isolation, but now let's imagine an AGI smarter than any of us which only "cares" about maximizing its Go proficiency. It's going to need a lot of energy. It'll hack or create robots and drill new oil wells (bypassing any constraints we put on it, since it in essence is spending thousands of human years figuring out how to escape). It'll devise nuclear technologies that may be unsafe from our perspective. Consider all the horrible things humans do in order to procure excess money (a commodity loosely correlated to energy) they don't actually need; there's no reason to think a badly aligned AI won't be just as inventive and horrible in its quest for more exajoules.
If capitalism and war are still the way of the world in 100 years, we are guaranteed to see some idiot inventing robot soldiers, hundreds of times faster and stronger than us, and with at least as much operational intelligence. He might never deploy the technology--a lab leak is in my opinion more likely--but at some point, it will get out. It won't be one robot; it won't have a body we can destroy. It'll be able to replicate itself and beam copies to satellites. It'll have control of our power infrastructure (which it will divert to its own "wants") within hours. We'll be screwed, not because it will want to harm us, because our lives require the use of resources that it will divert toward some objective function we technically invented but do not understand.
At least in the online groups I frequent, it seems that the latter two are seen as self-evidently dangerous, not because of any particular harm they have done, but because of imagined future harm. The former two are considered ridiculous things to worry about, because it's considered silly to imagine future potential harm. Perhaps because the former seem tied to uncool hippy types, and the latter seems more tied to cutting edge tech types?
I personally don't worry about any of these techs, since all of the danger I've seen appears greatly exaggerated. But it's curious to see how important trendiness is to fringe doomsday scenarios.
There's a very abstract argument that goes like this: * "Orthogonality thesis": an arbitrarily smart agent could in theory value ~anything, e.g. maximize Facebook's valuation (say as measured by some specific dataset published by S&P). * "Instrumental convergence thesis": most things that you might conceivably value are easier to optimize if you have a lot of power and resources (e.g. if you just run S&P, and hey maybe also control the worlds militaries so you can ensure nobody challenges that). * Belief in the power of intelligence: it's theoretically possible to be much more capable at ~anything than the most capable human just by thinking better and faster, including building even better AIs. Eventually some human will build an agent that does that, and plausibly this will happen in the current century. * Alignment problem: just because we make something doesn't automatically mean it wants exactly what we want, and in fact getting it to share our values is hard. Children disagree with parents, human values aren't those of natural selection, and ML language models repeat white supremacist ideas about Black people from their Internet training data.
Extrapolate out these points - which FWIW I find individually plausible - and you get doom: sooner or later humanity loses control of its destiny to whatever superintelligence we stumble on first, which in the worst case re-uses our atoms for something else (digits in some high-density storage representation that meets the loss function's technical criterion for Facebook's S&P valuation) about even in the best case seems fairly dystopian (whatever human Zuckerberg would actually want).
Now this is all pretty scifi and that's about all there was to it a few decades ago, but more recently we've started hitting ~parity milestones (chess, Go, protein folding, translation, arguably even fiction and visual art) that many experts once thought would require fully general human-level intelligence. It's clear to me that we haven't built that yet, but (as I said in another comment upthread) it also seems to me that our models are advancing in capability much more rapidly than we are gaining the ability to understand their internals or make guarantees of any kind about their worst-case or out-of-training-distribution behavior. Seems entirely sane to be worried by that trajectory and to start thinking seriously about how it might connect to the scary abstract speculations.
(edited for typos)
No and anyone who tells you otherwise is wrong.
A proper understanding of AI risk is a very thick book.
There are a ton of normally useful intuitions, heuristics and habits of thought that are implicitly applied when understanding the world that DON'T apply in the AGI case.
It cannot be explained to a five year old.
Well, in the second case he could not find anything wrong with his own behavior (using deceit to defraud the pub owner of using his facility), so I won't put much value in his first finding or lack thereof.
https://ae.studio/blog/most-lives-havent-been-lived-yet#pg
The problem is in the assumption that paid corporate work has zero social value. This is untrue. It has negative social value. If a rich person or a corporation that operates on rich peoples' behalf is paying you to do something, the odds are high that its total social value is negative, especially because we live within a moribund economic system that turns to extreme violence--fascism is capitalism in decay--whenever it hits the skids and people start to question it.
It's certainly plausible that the value of many jobs is negative, and even if you give away your entire salary, it could come out negative for society.
This whole idea is also out of touch with reality and lacking in common sense, because people who aren't interested in finance aren't likely to be competitive in it.
The problem with Evil Boltzmann Brain is that it could destroy all life in all galaxies through the infinitesimal chance that it spontaneously produces space travel.
Therefore, I have dedicated my entire life to fighting EBB instead. Currently, this requires that I buy an iPhone with the money I had planned to use on GiveWell.org but it's all in the plan.
I have not forgotten the goal, though. If I consume all resources on Earth, then I can stop evil AI at the same time as I increase entropy and reduce the possibility that EBB can utilize any energy differential. This requires that I consume consume consume. Which I shall do out of altruistic purposes.
My thought was that if you're going to assign infinite weight to unlikely possibilities, then why not devote your entire life to preventing time travel from being invented in the future?
Someone could invent a time machine and destroy the Earth before humanity had a chance to exist, etc.
It's just about as falsifiable as what they are working on.
---
Also, if you're going to be so abstract about it, then why you wouldn't want to optimize for the happiness and future vigor of AIs? If you're going to be a vegan, then presumably you care about non- human consciousness.
So what's wrong with a future with flourishing AI and without humanity?
There are too many ridiculous things to count here, and normally I would ignore it, but it seems like people are taking it seriously and it has consequence? (And the article is pretty well written either way)
---
It's weird that he had doubt that's like 1 sentence in the article, and then entire lifetimes of work that simply ignore the doubt:
My No. 1 worry is: what if we’re focussed on entirely the wrong things?” he said. “What if we’re just wrong? What if A.I. is just a distraction?
And as alluded to in the article, the wild swing from helping the poor to preventing AI has a huge moral hazard. With the former you can course-correct your actions based on observation of outcomes -- what happened as a result of the donations?
With the latter you can justify any action today by just pushing out the hypothetical benefits further and further into the future.
It resembles a religion that collects money today, promising benefits in the afterlife.
Not necessarily anything, but a pretty mainstream view now is that dangerous AIs need not be conscious (and indeed, that AIs can still be dangerous even if it's somehow impossible for an AI to be conscious). So you could imagine futures in which an AI conquers the world but has no experiences and can't either flourish or fail to flourish.
A computer virus is a familiar existing example of software that can be dangerous but that could probably not be said to flourish, even if it spreads widely, accomplishes everything it's programmed to do, and persists indefinitely. So we could envision dangerous AIs that are like today's computer viruses but that are also programmed to cause some kind of effect in the outside world, and that have some ability to learn and adapt so that it's difficult to prevent the effect and difficult to remove or deactivate the virus. That could be pretty dangerous to have active anywhere on the Internet, but still probably couldn't be said to flourish.
I struggle with assigning infinite weight and unlikely possibilities. But we can stay in the every day world and ask what credence do we place on the time machine possibility, and why? All information I have read suggests that it is essentially impossible, and so effectively I don't worry about this contingency at all. And then, is there anything we could do to prevent the time machine scenario? I can't think of anything at all.
On the other hand, to focus on one particular risk, there are compelling arguments made by experts in the field of artificial intelligence (e.g. Stuart Russell) that say that we face real risks based on the current trajectory of the technology. Is there anything we can do to reduce the risks? That seems like an important question.
It is important to be sceptical, and whilst I don't place 100% credence on the likelihood of a truly catastrophic AI outcome, I do think it is worth taking this possibility very seriously. There is a strong chain of reasoning behind the claims, and they are worth engaging with, especially given the fact that we are in a position to influence the outcome.
> if you're going to be so abstract about it... what's wrong with a future with flourishing AI and without humanity?
We can park the abstract side for a moment and think concretely about such a future. I want humanity to flourish, particularly during my lifetime. Credible people are currently ringing alarm bells about the potential destructive power of a technology we are actively developing, and so I think on balance it is better (at a minimum) to listen and be vigilant than to dismiss it. Better yet, I think we should engage with the arguments and see if there is anything we can do to reduce the risk of a potential bad outcome.
To the extent that these concerns are not overblown, one way to avoid the dangers of advancing ML would be to stop all the effective altruists [1] in the field haring along at building it irresponsibly.
The people preaching about the potential future dangers of AGI are the same ones pouring money into ML; if they actually meant what they said, they'd pour that money into their opponents.
The current practical dangers of ML are concerned with enforcing social inequalities and concentrating power in the hands of unethical actors. The fact that many of the unethical actors claim utopian reasons for their reckless disregard of consequences is not a reason to believe them.
[1] https://twitter.com/emilymbender/status/1556691543850831872
- no country was ever made wealthy by just giving them things. Wealth comes from passing through the difficult stages of development oneself
- you personally should do the good things you want to do in the world rather than just donating money for someone else to do it on your behalf
- Peter Singer uses the word "should" too much, and it's almost offensive. His philosophy is an empty philosophy pushed through by people who believe we are no different to apes. The older religions do this all much more justice in a far more thoughtful way.
I would suggest that people who want to make a difference with money do so on a local level. If they can create a local eden then perhaps someone else will notice and create another one locally where they are. It's kind of like small scale experiments which you can test and the ones that work will naturally be extended for little or no extra money.
It's just that this is (most likely) not the most effective way to make the world better.
And if your thesis is that this is the most effective way, because of copying etc, it would be useful to run some controlled experiments.
& govt. A poor family on section 8 vouchers, couple of kids in school, medicaid healthcare costs the government perhaps $100k/yr, + charities for food, clothing etc. You'll never get that in donations.
Which in most cases is going to involve destroying the existing institutions in societies that aren't creating much wealth, since it's precisely their existing institutions--corrupt governments and low social trust--that are preventing wealth from being created.
In terms of people setting up a successful and prosperous society and culture, then yes that can't be moneyed into existence.
Doesn't mean that thoughtful money can't help people or their society though.
We can talk about difficult stages at the individual level, but if an entire population faces things like dying from mosquito bites, I don’t think we gain anything or do anyone a service by not helping.
These are people who can’t pull themselves up by the bootstraps because they’re a little too busy trying not to die.
Edit: you mention not liking the word “should” in this context, yet you say people should do the good deeds themselves rather than donate money. Can you articulate why without a should here?
The point of donating money is that if you donate it well, it can have a greater impact than if you were to act independently. People have put a lot of thought and organization into it, and it isn’t an afterthought spurred by reluctance or laziness.
I believe the philosophical goal is to maximize the reduction of unnecessary suffering. I could be wrong, but with that in mind it could make a lot of sense to donate money rather than act independently.
Agree on the small problems that can be solved, but again i maintain that everyone needs to work through these stages themselves imo. People can help but ultimately the biggest help is opening up to useful trade, and enabling the empowerment of people from the bottom up.
It's a complex issue with no method without downsides.
Presumably, if people are spending less time taking care of their family dying of malaria, they'd have more time to create wealth.
Why?
If, for sentimental reasons, you get a kick out of working in the soup kitchen, by all means, work in the soup kitchen. Just be aware that you are doing this for your own entertainment.
> Wealth comes from passing through the difficult stages of development oneself
Is having people go blind from preventable causes or children growing up stunted supposed to build character?
> you personally should do the good things you want to do in the world rather than just donating money for someone else to do it on your behalf
The point is to do good, not feel good. Specialization is the very basis of human civilization. Software engineers donating their surplus to fund salaries for trained full time teachers is going to be a lot more effective than the alternative you're suggesting.
Once a much bigger player from the outside comes in and gives stuff for "free", like it has happened with some of the Western aid, then that kid's parents I mentioned at the start of the comment risk losing their jobs. After all, it's not worth it anymore for their employer to keep them, well, employed, cause the market that he's targeting is getting dumped by all the free stuff coming in via Western "aid".
These are, in retrospect, basic economics-related things, but which nevertheless took a while to sink in into some people's minds in the West (some are still oblivious to it all). I remember a book written by an African lady (I think she was from Botswana or from that general area), former worker at a big financial institution in the West, and imploring the West to stop "aiding" the African countries if they want to see them actually develop.
I would argue you want to do the latter because both net and medicine manufacturing are capital intensive - a small number of people are employed despite the large cost of setting up a factory. Most of the local economic contribution is going to come from hiring people to handle distribution.
Let's take an example mentioned in the article, Sam Bankman-Fried. After hearing about EA in college, he decided to earn as much money as he could first in traditional finance and then in crypto, so that he would be able to fund work he thought was important that was currently not happening. He was very successful at this, and currently has about $12B he's working on donating.
I think it's a pretty difficult claim to support that he would have furthered his goals more by picking one of the things he thinks needs to happen and working on it?
On the other hand, a lot of early effective altruist writing is from a time when there was very little money available for some of the most valuable things, a time when even "go work as a regular software engineer and donate a large percentage" beat directly working on valuable things for a lot of people. If you look at that writing now, however, since it is assuming a very different funding environment it looks like it sets the bar way too low for how much money you need to be able to earn before earning to give is likely your best option.
What happened to the Golden Rule? Or to Sartre’s “in choosing for himself, [each man] chooses for all men”? Surely if EA leads some to causing such wide-scale short-term pain in its name, at the very least it cannot be a humanism.
I realize this isn’t a strong argument, but when I hear about an EA champion, there’s often this dynamic where how they made money doesn’t matter. They’re wealthy, so clearly what they do is valued by the market and we should ignore externalities that aren’t priced into their income. It’s like Big Oil pointing to specific green outcomes that they fund; these outcomes are quantifiable yet funded by hard to price but reprehensible practices.
It all feels like Catholic indulgences… Maybe this is still a net-positive for society in that it provides people who wouldn’t be humanist anyway (e.g. libertarians) with a well-intentioned philosophy?
Yes. Initially he worked in regular finance, at Jane Street, and then crypto, founding the Alameda Research trading firm and FTX exchange.
> By definition this would mean capturing real money from retail investors, right?
I'm not very knowledgeable about crypto, but I don't understand how the definition of crypto means this?
The article we're commenting on describes one of the early successes of Alameda, where they saw that Bitcoin was consistently trading for much higher prices in Japan than in the rest of the world. In normal finance you mostly don't see this sort of thing for very long, because global markets are highly interconnected, but in this case it was logistically difficult to sell Bitcoin in Japan. In fixing this, Alameda lowered the price at which Japanese investors were able to buy Bitcoin, and raised the price at which people elsewhere were able to sell Bitcoin, keeping a fraction of the difference for themselves; I don't see how Alameda made others worse off?
With FTX they're making money by providing a platform that makes it easier for people to trade with each others. A bit of, 'in a gold rush cell shovels'. Again, I don't see how they're making others worse off?
> when I hear about an EA champion, there’s often this dynamic where how they made money doesn’t matter.
Sort of! If someone is initially not altruistic and accumulates a bunch of money doing something questionable, then if they later come around to doing good things with their money I'm going to be happy about that. So while I am not a fan of, say, Microsoft's business practices in the '90s which made Gates a billionaire, I do think his decision to spend his fortune the way he has is a good one, and one that should be commended. The alternative is that, like most other billionaires, the money goes to houses, yachts, and other things which are far less positive for the world.
On the other hand, SBF is in a different category: he was trying to be an effective altruist from the beginning. So if he did something massively harmful to earn his money (which, as discussed above, I don't think he did) we would need to consider that in figuring out his net impact.
Additionally, it is pretty common for EAs to argue that you shouldn't do harmful things even if they are majorly profitable and allow you to do a lot of good through your donations.
> I'm not very knowledgeable about crypto, but I don't understand how the definition of crypto means this?
Crypto doesn’t really power real-life use cases outside of trading crypto assets and associated financial services. For real money to come into the space there needs to be an influx from outside. Usually that means VCs and, more largely, retail.
I’m not necessarily arguing here that the crypto space is a drain on society, as I think that’s contentious, but the claim that most real money in crypto originates from people outside of it should be a truism. To me this makes SBF’s approach something misguided, as they’re reallocating capital from people at large to a single person. Even with the best intentions I can’t see this as a humanist approach. Some charitable projects do require massive funding and opinionated goals, but isn’t that what nonprofits and politics are for?
I guess it does come down to whether someone sees making money off of crypto as having strong negative externalities or not. If SBF was simply creating value by democratizing access then it’s just a regular job. If he was propping a scam industry that drains retail investors of real money then it’s a net negative even with EA as the goal.
Still, the inherent long-termism within EA seems to make it easier for people to embrace very Kantian views that maybe discount their short-term impacts.
No. What kinds of weird definitions are you using?
(You might be right in practice, I don't know. But you are not right by definition.)
> It all feels like Catholic indulgences…
And that is wrong somehow?
If you grant the assumption that the Catholic church was a worthwhile organisation doing good work, then the indulgences seemed like a perfectly good idea to me on a practical real-world level. (No clue about the intricacy of Christian doctrine, so I can't say whether they were kosher from a theological point of view.)
I disagree with what sbf is doing wrt political donations.
It is kind of difficult to grasp but when you see a homeless man, the best thing you can do to him isn't giving away money, it is hiring him and giving him a job that pays enough to pay the bills. When you donate money to the poor, you are basically subsidizing poverty.
Honestly Keynes already had the right idea, just give them a job to build commercially useless things like affordable housing.
Nah, people just are rather clueless on the what and the why of effective altruism.
The top causes are things like deworming, anti-malaria and AI risk. These are things that substantially increase economic productivity (the latter on account of people not being dead).
To me it almost just seems like a "different people can have different values" sort of thing. If you want to value current lives more than future lives, you'll just come to different conclusions than someone who believes otherwise, and maybe both sets of conclusions are okay.
I definitely agree with your second paragraph, however, and people who value current lives more than future lives will often come to different philosophical conclusions. It's interesting, however, that this frequently doesn't cash out into different beliefs about what we should do today. For example, I think the value of work on pandemic prevention or AI alignment does not depend on multiplying by zillions of potential future people: they're worth doing even just for people alive today.
Ex: https://forum.effectivealtruism.org/posts/KDjEogAqWNTdddF9g/... :
Let me rephrase this in a deliberately inflammatory way: if you're under ~50, unaligned AI might kill you and everyone you know. Not your great-great-(...)-great-grandchildren in the year 30,000 AD. Not even your children. You and everyone you know. As a pitch to get people to care about something, this is a pretty strong one.
But right now, a lot of EA discussion about this goes through an argument that starts with "did you know you might want to assign your descendants in the year 30,000 AD exactly equal moral value to yourself? Did you know that maybe you should care about their problems exactly as much as you care about global warming and other problems happening today?"
Seven billion (or even a couple of million) humans in the world today are obviously a sufficient rationale to pay attention to existential threats or stuff that could blow up in our faces including pandemics, climate changes and AI systems, but proponents of actions with apparently vanishingly small probabilities of improving safety tend to need to invoke gazillions to justify prioritising them over tangible solutions to the problems of millions in Bayesian/utilitarian framework.
Trying to sidestep common arguments about the personalities, but I guess an example would be an ideal-version-of-Bill-Gates, and an ideal-version-of-Elon-Musk. Gates Foundation mission statement seems more weighted to current lives, while Musk is more oriented toward multi-planetary existence of future generations. Still some overlap though, Musk's can also be argued to be relevant for current generations.
There is always a risk your company doesn't become the next unicorn.
There is always a risk your society fails to grow exponentially for two thousand years. Our current track record is that exponential growth happens until it stops, which is pretty bad news for anyone whose math assumes that exponential growth never takes a break.
In the same spirit, I might as well claim my after-meal $2 ice cream relieved me from hunger.
It's hilarious how much this sounds, and the science-fiction writer Charles Stross has pointed this out in the past, Russian cosmism [1], a weird offshoot of Christianity dressed in secularism and rationalism. That something like this emerged so quickly out of the decidedly pragmatic and genuinely secular altruism of Singer is in my opinion evidence of the weakness of Effective Altruism as a philosophical system. It's such a bureaucratic, mechanical system that it's already started to branch off into weirder, less utilitarian schools.
[1]https://en.wikipedia.org/wiki/Nikolai_Fyodorovich_Fyodorov#P...
Let's turn this into a trolley experiment, 1 million planets colonized with 8 billion humans each living the same lives we live today vs 10^58 humans of which the majority don't exist and are just made up. The extreme utilitarianists and long termists will pick the 10^58 fake humans even if there is a near 99% chance that none of the humans actually exist. That 1% chance would by far outweigh any lesser goal despite the fact that it is unlikely to happen in your universe.
Another problematic aspect is that if there were actually 10^58 flesh and blood humans, there wouldn't be any planets left for potential alien neighbours, by aliens I do not necessarily mean independently evolved intelligence, it could simply be a new human space race that develops in some quiet corner of the galaxy isolated from everyone else.
Yes, existential risk of general AI might be small; but even fewer people are working on it: only about one hundred, worldwide. Similarly, the reason E.A. stepped off the climate change train is because it was a popular issue, which means the marginal benefit of one person contributing decreases.
When more people direct attention to AI safety, another area will be the one where E.A. can contribute most, and the focus will shift.
You can make that argument, but I think the median view among people working on AI alignment is that the risk is quite high, in the range of >20% chance of disaster within their lifetime.
Still, the changes to society are absolutely hard to grasp. Humans may face (economical) obsolescence in a few decades[1] (humans need not apply[2]). That puts us in a delicate situation: if humans are economically obsolete, a society that only values economic value (and not consciousness per se) will have all reasons to slowly but surely get rid of humans (and animals).
We need a revolution of meaning and ethical understanding: the understanding of the fundamental value of consciousness, and to structure our lives and all institutions around it (each individual needs to have a good understanding of such ethics and meaning so that society as a whole is robust). I think formalization of ethics (which I'm a proponent of) and progress in metaphysics is going to contribute as well to this end. I think EA is simply part of this new understanding of ourselves and the Universe, a sort of coming out of a dark age -- and there's the opportunity to create a lasting society amazing for everyone.
[1] My personal guess-estimate is about 50(-20/+40) years to complete obsolescence, although various researchers have wildly different estimates.
[2] https://www.youtube.com/watch?v=7Pq-S557XQU
reëntry, coöperation, teen-ager
Have never seen that before
I kinda see utilitarianism as a step on the horrifying path to human eusociality.
Have these people not learned yet that when you save a poor child they will just make fourteen other poor children in short time? Maybe the best form of charity is lobbying for over-regulation of child-rearing goods to the point of near unaffordability, as is currently happening in developed countries.
These people seem oversocialized.
https://www.macrotrends.net/countries/BGD/bangladesh/fertili...
And ironically, a key driver of birth rates falling is when children stop dying so much.
https://ourworldindata.org/child-mortality
But I might look up the EA take on some other vaguely parallel things, like Quaker businesses, worker co-operatives, fair-trade, progressive taxation, banning billionaires, revolutionary communism, bringing democracy to the middle East, nuking Russia, installing a benevolent world dictator etc.
Would help me understand them to see how they position themselves against these things as I think you could make a half-assed attempt to prove that any of them was better with their own maths if you wanted to.
e.g. if there's any billionaires then the chance of a billionaire buying elections and installing their own benevolent world dictator that believes in the wrong type of EA is a small but still existential risk (since only the true EA can save the future galactic empire). Tweak numbers as necessary until you arrive at your pre-ordained conclusion.
edit:
EA take on fair trade:
https://www.givingwhatwecan.org/blog/how-effective-fair-trad...
EA take on renewables, climate change, developing nations:
https://www.effectivealtruism.org/articles/johannes-ackva-an...
An attempt to reconcile EA and anti-capitalism by a fan of both:
https://core.ac.uk/download/pdf/81665058.pdf
EA take on Socialism (tl;dr central planning is bad, which seems slightly counter to the whole EA thing):
https://forum.effectivealtruism.org/posts/ktEfsoGfBFGsaiY46/...
It doesn't care about individuals, does not seem to think about culture, and seems to have an unexamined savior complex and egotism that leads it down bad rabbit holes.
EA people have their hearts in good places. But when you start to look at 'maximum impact' and at the metrics used to define it, EA only focuses on statistically testable social group problems, and not on individual issues. So the way that EA philosophy seems to me to regularly be expressed is that an EA proponent would say something like, "unquestionably, we should give $50k to fund mosquito nets to raise the quality of life for 5k people" and not even consider something like "we should give 50k to train local trauma counselors to help 500 people be more mentally stable and whole" and the vibe that I have sometimes gotten from reading blogs EA advocates that individuals are not worth helping because that's not impactful or impactful enough. We see the understandable precursor of this in the article, where Matt Wage talks about the idea of it being more impactful to cure 200-4000 people from glaucoma. It gets a pass because it is framed around minor improvement to QOL in the West versus significant improvement in QOL in a developing country, but also, look at the quote just proceeding that, around saving people from a burning building.
See, the work that EA advocates do is important. But when we only try to fix societal problems from the top down by addressing them at a society level, we often ignore or damage the individuals that we are nominally helping. I'm not saying that mosquito nets, for example, are bad and shouldn't be given to. I am saying there is also a down and dirty component of individual attention and flourishing that has to be looked for and addressed. I'm not saying that EA as a movement or as a set of goals is fundamentally wrong, just that it needs to have some temper with personal and individual care. I see EA advocates looking down on people that "do things that don't scale" in YC parlance, in areas of the world where there are plenty of people doing things that don't scale because doing things that do scale is almost impossible to do well. EA also seems, again from the outside perspective, to really focus on things that play well to Western mindsets, and doesn't really consider community-versus-individualism or guilt/innocence-versus-honor/shame cultural paradigms. It seems obvious to me that in 100 years, that the hands-off, Western-directed action promoted by EA will be seen as colonialism 5.0. There are also really not a lot of prominent non-Western voices in the EA sphere, nobody that seems to have been raised to leadership levels, and that also is concerning for the long-term health of the institution.
In the quest to make the most impact, the savior complex and all the corrosives that come from it can easily permeate the movement (and indeed seem to be starting from the article), justifying institutions, power, prestige, and rank. It's easy to see MacAskill falling into that (see the braces/teeth gap aside), and when I often read EA advocates, I note that many of them do not seem to spend any time doing thankless, individual work (like as a trivial example volunteering at a homeless shelter, let alone be in a developing country or war zone) and the ones that have seem to quickly burn out and decide that it is more proper/moral for them to 'earn-to-give' and donate more. It is entirely possible that, not being in the EA community, I don't know the people that do this well since they are not blogging/community building.
All this may be a pot-calling-the-kettle-black since I too st...