It’s interesting that she was threatened with being held responsible for violating a fiduciary duty for the for-profit entity while she was sitting on the non-profit board.
Going fwd, I wonder why they cannot convert the structure to a traditional c-corp. Supposedly: “tax issues”.
Whatever problems OpenAI dealt with last week, the current non-profit structure will continue to cause future problems IMO.
Great learning opportunity for those that have not taken time to learn and understand non-profit structures in the USA.
this is clearly an opportunity for the uninformed to study the US non-profit system, unique in the world. For example, I recall a delegation from Japan touring the HQ of a non-profit educational group in San Francisco. Like the p-poster here, they apparently had no clue why or how capable people could come together and form stable work culture aside from "hierarchical, profit motive in USD$" (.. and people did actual work, unlike many management level stock manipulators).
I'm from southeast Europe and I graduated from IB high school programme[0], it is an international Swiss based educational programme that prepares you to study abroad and it is a non-profit. As a non-profit they are efficient af at what they do but in the world of technological innovations, incentives are so much higher and so much more impactful when an organization is for-profit.
Mark Surman writes: As I wrote in Fast Company earlier this week, we risk missing the real lesson from the OpenAI governance debacle. ...
"In the aftermath of the OpenAI governance battle, the narrative has been dominated by the idea that “the capitalists won.” While this conclusion is catchy, it misses the point.
The real (and bigger) story is: The founders of OpenAI set out to create a nonprofit public institution to ensure that AI’s evolution aligns with the interests of humanity at large. Last month, it became clear that they failed.
What is also clear: Public institutions that prioritize the public good over short-term profits are desperately needed for the AI era."
I read it as both. If OpenAI collapses operationally then the non profit fails at its mission too. That’s the board she is on and that’s the fiduciary duty relevant
>If OpenAI collapses operationally then the non profit fails at its mission too.
Not necessarily true.
"We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome [...] if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project."
> Going fwd, I wonder why they cannot convert the structure to a traditional c-corp. Supposedly: “tax issues”.
Because, IIRC, when a charity nonprofit like OpenAI, Inc.-- a 501(c)(3), other nonprofits are different -- converts to a for-profit (C-corp or otherwise doesn't matter), all the assets acquired while it was a charity must either remain dedicated to and used exclusively for the charitable purpose, be returned, or be donated to charity (or sold, and then the proceeds treated the same way.) For the charity OpenAI, Inc., this would include its interest in OpenAI, LLC (the wholly owned subsidiary through which it exercises control of OpenAI, LP and OpenAI Global, LLC -- the latter being the for profit entity that actually sells product, etc.), and OpenAI, LP (the holding company which in turn is majority owner of OpenAI Global, LLC).
Just curious: can they legally sell all their assets to themselves (i.e. OpenAI nonprofit sells it to for-profit OpenAI Inc. for market price real $$$) and then use that $$$ for charitable purposes. Would this satisfy IRS by letting them retain the intellectual property rights?
also OpenAI Inc. for profit doesnt exist, a wholly owned LLC subsidiary does. so they would need to divest of that LLC first (by selling the shares)
the way around the self dealing regulations are to reduce their ownership below 20% - 35% (yes, its that range, for multiple reasons)
good thing here is that most venture backed companies subvert this already by simply share dilution and youre breaking into one the best secrets of Silicon Valley. Non-profits pump the value of startups and launder the reputation of the founders because their ideas get so much “validation” so quickly. (non profits they partially control act as lead investors for startups they control. they both sell shares at higher price, ideally: founders get QSBS tax free treatment, non profit retains tax free capital gains too. nothing wrong with that, you just probably cant do it too)
so open ai for profit llc needs to get rid of the PPU’s and bring in outside investors to dilute ownership enough, but two birds one stone: the outside investors have to get a higher valuation
> also OpenAI Inc. for profit doesnt exist, a wholly owned LLC subsidiary does.
The for-profit "OpenAI" (OpenAI Global LLC) is not a wholly-owned subsidiary of anything, it is majority owned by a holding company (OpenAI, LP), which in turn is majority owned by the OpenAI charity (OpenAI, Inc.)
There is a wholly owned LLC subsidiary of the charity (OpenAI GP, LLC), but that is the entity through which OpenAI, Inc., exercises control of OpenAI, LP and OpenAI Global, LLC (why it doesn't do so directly for OpenAI, LP, and either directly or via OpenAI, LP, for OpenAI Global, LLC, don't ask me), it is not the for-profit.
Both OpenAI, LP, and OpenAI Global, LLC have other investors (Microsoft being a big one for OpenAI Globla, LLC) already.
I thought you were being sarcastic at first, but thanks for breaking that down
one explanation is “blockers”. basically regulations imposed on one kind of entity such as the non profit, fail when there is one degree of separation via a “blocker” entity.
LP’s can have undisclosed silent limited partners, intentionally less flexible than an LLC which can do the same
It’s interesting that she was threatened with being held responsible for violating a fiduciary duty for the for-profit entity while she was sitting on the non-profit board.
Fiduciary doesn't mean financial or monetary duty, it means holding something in trust. Violating a fiduciary duty could be financial, social, reputational, or in this case, literally destroying the operation of the company you were entrusted with.
She's the leading academic on the issue of AI safety. It's really ridiculous people don't even know her name and say random things about her, not realizing she's a rock star in her field.
Her leaving the board is a tremendous loss for OpenAI, about as terrible as Ilya leaving the board. These are two giants of AI currently.
As an academic from a different field, this is not a publication record that would be anywhere close to someone being "the leading academic on the issue of AI safety". It indicates that she has had been one of very many participants at two impactful papers (with zero indication of what her individual contribution was, with so many authors it might as well be being part of the funding organization), has done a few collaborations with strong researchers, and that her personal first-author or sole-author research is barely relevant.
Like, it's not a bad publication record, especially for a junior researcher (heck, her first participation in other people's papers is just in 2018 and her own work starts in 2021-2022, so she's effectively just getting started and may have many opportunities to prove herself in the future), it does indicate doing a couple years of research and might qualify for an okayish faculty job when (if?) she gets her PhD, but it's not also something indicating a senior researcher that's the pillar of anything and has had some impact - a grad student who's lucky to study in a strong leading researcher's lab and does some work on two of the advisor's papers might have a similar publication record already at their graduation before their career has even seriously started.
Many of her co-authors have significantly larger research impact and personal research (https://scholar.google.com/citations?hl=en&user=VclFrJ8AAAAJ or https://scholar.google.com/citations?hl=en&user=MbBntPgAAAAJ for example or https://scholar.google.com/citations?hl=en&user=0-G2eiEAAAAJ) - which also raises a question of how much each co-author did in a two-author paper where the leading author is a strong researcher and the second author is a "Director of Strategy and Foundational Research Grants" i.e. getting money for that research, so labeling her as a leading researcher (much less the leading researcher) of some field seems misleading as the publication record would indicate that she's not even on the same level as they are.
And what happens when they read the articles and come to realize that you could give an average college freshman any of those subjects and they could produce a comparable paper in a few weeks' time with a full course load and a part time job? Seriously, I've never seen a more underwhelming set of articles on google scholar from a supposed "leading" expert in a field... and what's worse is that she wasn't even the lead writer on most of them.
> What impact has she made? What has she built? Did she actually make AI safer?
She was on the board that fired Sam Altman, that had the courage to fire Sam Altman, presumably (according to the article at least) because he is a liar and she and three other board members did not trust him to be loyal to the non-profit's mission of working to keep AI safe.
That particular impact was undone in a single weekend. She may be an absolute rockstar otherwise but "firing Altman" is not impact when he is still in control of OpenAI.
The parent post question presumably was about the impact she has had as an academic or researcher (as "the leading academic on the issue of AI safety"), where the political actions on some board are not relevant.
from the article, the paragraph right before it explained how she got on the board:
"In 2019, she spent nine months in Beijing studying its AI ecosystem. When she returned, Toner helped establish a research organization at Georgetown University, called the Center for Security and Emerging Technology, where she continues to work."
It's interesting that on her LinkedIn profile she instead states that in Beijing she spent those 9 months in "intensive Mandarin Chinese language training" at
a Chinese language program in Tsinghua University, not studying its AI ecosystem.
I don't think "rock star" is quite the right term.
Unless you think David Geffen or Lester Bangs are rock stars. Normally rock stars sing or play an instrument. Someone who got a nice title in a nonprofit by writing about rock is not a rock star.
She comes across as a childish idealist, typical of many uni graduates who've yet to encounter the real world, also typical of the EA cult. Giving people like her real power before they've had the uni nonsense knocked out of their heads is a ludicrous idea.
I'm embarrassed she comes from my country, but not surprised about the particular uni she attended within my country.
I'm doubtful that the "childish idealist" description is accurate. But even if it was, given the choice between having existentially dangerous AI developed by a childish idealist vs a slippery machiavellian like Sam, I would choose the childish idealist.
The big risk with a childish idealist is that they become the pawn of a slippery machiavellian. Choosing the slippery machiavellian just seems like speedrunning failure.
I’m quite happy with the childish idealist description though. Putting childish idealists in charge of $billions of other people’s money might sound like a great idea to you, but would you put them in charge of your money?
"Cult" makes it sound like Scientology. Scientology doesn't run criticism contests of itself. As another datapoint, many of the most-upvoted posts on the EA Forum are EA critiques of one sort or another: https://forum.effectivealtruism.org/allPosts?timeframe=allTi...
Wikipedia says you have to apply to attend some EA events. That's true for lots of things. TED isn't open to just anyone either. If you want to go to an EA event without applying, most group events are pretty casual and don't require any application: https://forum.effectivealtruism.org/groups (Note: I heard a rumor that EA events in Germany are less likely to be open to the public)
>Putting childish idealists in charge of $billions of other people’s money might sound like a great idea to you, but would you put them in charge of your money?
Well, I have in fact donated significant sums to some EA organizations, and I have a recurring donation to the Good Food Institute (working on meat alternatives). I don't agree with the "childish idealist" description of EA though. From my perspective, it's remarkable how easily Sam Altman seems to manipulate people, and he seems more successful at manipulating people outside of the EA community.
You're welcome to say EA is a cult if you include the caveat that "by cult, I mean a thing that gives lots of money and upvotes to people who criticize it". That way it will be clear that your definition of "cult" is broad enough to include things like that. Not everyone defines the word so broadly.
The post is from May. The tweet is from November. I'm not sure why you'd expect the post to mention the tweet, because (a) the tweet was 6 months in the future, relative to the time of the post, and (b) the tweet doesn't appear very relevant to the topic of the post.
Peter Singer is more of an EA popularizer than an EA leader. The post you linked is the only post he has made in the past ~9 years: https://forum.effectivealtruism.org/users/peter_singer The tweet you linked is related to his Journal of Controversial Ideas, not EA.
It's certainly not the case that Singer should be taken as representative of the average EA, you'd have to look at surveys for that: https://forum.effectivealtruism.org/s/FxFwhFG227F6FgnKk However EA tends to be friendly to contrarian thought experiments (not usually taking it to Peter Singer levels). And EAs usually prefer to focus on getting facts right over condemning each other in an inflammatory way.
So the high priest / philosophical originator (guardian.com) of the cult should not "be taken as a representative of the average EA," and nobody saw fit to mention his...unusual views (due to his naive utilitarian philosophy) later on in that EA forum thread, the first thread he's posted in 9 years, the thread who's topic is closely related to the topic of the Twitter thread I linked?
Is EA distancing itself from Singer (who's views are, by all accounts, extremely influential in the EA community), as they did with SBF (whose hedge fund's initial $170m seed capital was raised from EA devotees)? When multiple high-profile people associated with EA are proven to be....problematic, should we pretend the common factor linking them is insignificant?
Inflammatory? I thought I was quite restrained compared to some of the commenters on that Twitter thread.
What are you talking about? She has no qualifications and her publications are juvenile. Her h-index is 8. She has no background in CS and has a few publication in Foreign Affairs. If she is the leading academic, then AI safety is not a serious academic field. She is in no way comparable to Ilya. This is the worst take I've seen on HN in awhile.
>"She's the leading academic on the issue of AI safety."
From article, this is what I know about her:
>"Toner graduated from the University of Melbourne, Australia, in 2014 with a degree in chemical engineering and subsequently worked as a research analyst at a series of firms, including Open Philanthropy, a foundation that makes grants based on the effective-altruism philosophy."
>"In 2019, she spent nine months in Beijing studying its AI ecosystem. When she returned, Toner helped establish a research organization at Georgetown University, called the Center for Security and Emerging Technology, where she continues to work."
She's been working in the field of AI safety for what 3 years at most? I find it very hard to believe that she's "thee" leading academic on the issue of safety. Stuart Russell's group, the Center for Human Compatible AI at UC Berkeley, MIT's The Algorithmic Alignment Group and David Kruegger's group at Cambridge are all academic leaders and groups with great expertise in this subdomain.
Further it's not even clear what "studying Beijing's AI ecosystem" actually means. Does this refer to the city? The university? The CCPs' surveillance state(not likely)? It's very hand wavy.
I don't have any problem with what she did or her views. I also think it's a loss for the company. I do question her bona fides for being called the leading academic on the issue though.
> Does she see herself as more trustworthy? She can't even be bothered to give an excuse for the firing.
I agree that outsiders can't weigh in on this topic without statements from the board.
But as the (former) board member of a private organization, she's not accountable to the public. Making statements about why she fired Altman can carry substantial legal risk.
> “Fancy titles like ‘Director of Strategy at Georgetown’s Center for Security and Emerging Technology’ can lead to a false sense of understanding of the complex process of entrepreneurial innovation,” Khosla wrote in an essay
Exactly. Being a rent-seeking grifter doesn't make you an expert. In anything.
Fancy titles like Founder and Managing Director of Khosla Ventures can lead to a false sense of understanding about what a researcher like Helen Toner understands.
I'm amazed anyone can side with Khosla after he blatantly showed his $ > ethics side when it came to the Martin's Beach scandal[0]. In a battle of $ and ethics, finding yourself agreeing with Khosla is a smoking gun where someone stands.
That we have VC's character assassinating someone in contrast to all the people actually around her who can vouch, is quite a sad legacy.
And yet now it’s Helen Toner who is “there,” as a chemical engineer BSc, involved in one of the most impactful boards because she’s respected by the important people around her, and you’re writing about it.
She’s one of the only people to bridge the AI communities of the US and China, which will be a crucial factor in preventing a state level arms race of AGI. BSc Chemical Engineering, MA from Georgetown, spent the rest doing research on what the future looks like.
She replaced Holden Karnofsky as a representative of the "effective altruist" group, Open Philanthropy, who donated money early on to OpenAI.
From the article:
"She succeeded her former manager from Open Philanthropy, Holden Karnofsky, on the OpenAI board in 2021 after he stepped down. His wife co-founded OpenAI rival Anthropic."
I have been wondering the same about Altman. Dude had a failed startup, and somehow wormed his way into being the president of YC, a job he f'd up and got fired from, and then managed to parachute into being the CEO of OpenAI.
She specifically declines in the article to provide the additional context that would probably clarify everything. This is just more non-answers and only a tad more information than we had before (threats of violating a fiduciary duty).
Nobody involved is being transparent about what went down here. Not sure we'll find out the full story here...
It's frustrating, but she's exposed to potentially massive personal liability for the board's actions. I'm surprised that she agreed to an interview at all - frankly, I probably wouldn't if I were in her position - and I'm sure her answers were carefully scripted and run by her attorney(s).
Article does give context as to what "inconsistently candid" meant: Altman allegedly went to each board member to say some negative things about Toner, but when board members got together to verify consistency they found out Altman said a false or misleading thing to at least one board member about Toner.
That's not a good look on Altman at all, it looks like Altman was using divide-and-conquer in a skeezy way, spreading around negative impression, even rumors, about his bosses, as a means dispose of a problem.
I think that's a technicality if the expressed concern was this piece was just non-answers. It's a regurgitated answer but still new information to many of us.
Declining to give more details is what professionals typically do. In addition to being more risky one's self-interests and the interests of the organization, there are also often very good legal/confidentiality reasons not to run one's mouth off.
"Altman approached other board members, trying to convince each to fire Toner. Later, some board members swapped notes on their individual discussions with Altman. The group concluded that in one discussion with a board member, Altman left a misleading perception that another member thought Toner should leave, the people said.
By this point, several of OpenAI’s then-directors already had concerns about Altman’s honesty, people familiar with their thinking said."
Wow, there is no way around it. Sam Altman is basically getting his way by misleading people. The board did the right thing. Honesty is crucial as a leader. I think for Sam Altman his goal is more important than ethics and he will do whatever to get his way.
Honesty is great, in moderation, but it's not the be-all and end-all of leadership.
Leadership is fundamentally about manipulating people. If the people already agreed, they wouldn't need leadership. Your job as leader is to get them all working in more or less the same direction, even though they don't really want to.
Honesty is a great way to do that, when it works. Honesty has the virtue of not having to keep track of your lies, and you don't get caught in an inconsistency.
But it doesn't always work, and then you have to consider alternatives. Sometimes people are dumb. Sometimes people are wrong. Sometimes people are hostile. Sometimes people are right, but in a way that can't actually be implemented, and you need to go with a sub-optimal path despite that.
At core, you can't lead if you don't believe that you've listened to all of the concerns and reached the solution that is best for the group as a whole, according to whatever definition of "best" and "whole" you were hired for. Sometimes that's going to force you to use other means that just telling people the truth and hoping they reach the same conclusion.
Leadership is fundamentally a people problem, not a technical one. And sometimes that's going to mean being dishonest. A good leader does it as little as possible, but simply dismissing it as a tactic is a recipe for failure because it hides a fundamental truth about why and how people behave in groups.
First of all, it's the CEO's job to take direction from the board and implement their vision, not "lead them". Second of all, your argument seems to be that ethics (specifically honestly) sometimes gets in the way of goal. I think everyone knows that, but the reason we choose to behave ethically is in spite of this, not because we are deluding into thinking ethical behavior always maximizes some objective. Thirdly, I would argue that in the long run ethical behavior is the optimal path in leadership or any other endeavor which involves cooperation.
One who doesn't want to be replaced, especially in companies with a few very large shareholders who have their representatives on the board (e.g. major investors in startups) where if a CEO isn't proactive in ensuring that their vision is aligned with the board's vision will not be a CEO for long, even if they were the founder or something.
"Managing the board" is one of the major jobs of the CEO, most of that work done outside those key moments when the board meets to formally vote on what they usually already decided based on earlier discussions between themselves, CEO, shareholders and others.
leadership for openai is supposedly accountable to the board.
lying to board is dick move, manoeuvring to get someone fired is a dick move. If what we read is mostly true, then I can see why the board fired him, after all, how can they trust him?
The thing about the "celebrity CEO" cult that has taken over the Valley-- and is in full force at OpenAI-- is that people like the parent have completely forgotten that the way it's supposed to work is that the CEO serves at the discretion of the board of directors. The board is not some speedbump or hassle to be maneuvered around, they choose the CEO and can revoke their choice.
Not that that's actually ever going to be the case at OpenAI again, but the parent has the chain of command here totally bass-ackwards.
> the way it's supposed to work is that the CEO serves at the discretion of the board of directors. The board is not some speedbump or hassle to be maneuvered around, they choose the CEO and can revoke their choice.
> CEO is an instrument of the board. The CEO is not there to lead the board.
In truth it's both. That said, leadership doesn't mean lying. Particularly not about what another person has said. Very particularly not when it's trivial for them to compare notes.
> Leadership is fundamentally about manipulating people.
This is wrong. Manipulating people is not leadership. It might be management in some circumstances, but management is not leadership.
> If the people already agreed, they wouldn't need leadership.
This is wrong too. People can agree on a goal without seeing how to get there. Leadership is not just about seeing the goal. It's about leading people there. But if you have to manipulate the people, you're not leading them. You're just using them for your own ends.
> Leadership is fundamentally about manipulating people.
This is true...if you are the leader of a cult, which exist to deify you more than benefit the group.
> If the people already agreed, they wouldn't need leadership. Your job as leader is to get them all working in more or less the same direction, even though they don't really want to.
Yes, but if you're having to puppet people to do it, it's not leadership. It's just manipulation. You should be able to do this through inspiration, and transparently in group settings, not by compartmentalizing lies and half-truths privately. That's just fucking embarrassing. Altman's leadership skills are on par with a 14-year old influencer.
I feel like you're taking the argument for why "we let Pearl Harbor happen" and using it to excuse the whims of a psychopath.
Obviously it wasn’t the right thing, as he’s back. You don’t fire someone so important for that and extrapolate it into equating human extinction. He has flaws but he seem to have the right balance between acceleration and safety, even though I’m not a fan of him gunning for regulatory capture
>Obviously it wasn’t the right thing, as he’s back.
The dude essentially threatened to destroy the company if he didn't get his way. Might does not make right. Sam turned out to have more leverage here; that doesn't make him a good person.
>You don’t fire someone so important for that and extrapolate it into equating human extinction.
On the other hand, would you really want a company developing existentially dangerous technology to be lead by someone who's known as a slippery operator?
My hunch is that part of the board's motivation was a feeling that they should have unusually high standards for the CEO of OpenAI.
You are assuming Altman is out to destroy humanity, he values his life and others just as much as you do, if not more. He has as much to lose as anybody if he blows himself up. Also who is to say deceleration in the age of AI isn’t worse than acceleration? Every govt entity is developing it, and it could be a national sec issue to speed it up faster than an adversary
>In AI, like in any field, most of the people who hold power are people who have been very good at winning a bunch of races. It's hard for these people to not want to race and they privately think they should win the race.
>...
>Pretty much everyone who works on AI thinks that they're 'one of the good people'. Statistically, this is unlikely to be the case.
>...
>Proactively giving up power is one of the hardest things for people to do. Giving up power and money is even harder. AI orgs are rapidly gathering power and money and it's not clear they have right incentives to willfully shed their own power. This sets us up for racing.
>Also who is to say deceleration in the age of AI isn’t worse than acceleration? Every govt entity is developing it, and it could be a national sec issue to speed it up faster than an adversary
Anyone else come to the realization that when spokespeople talk about "AI Safety" they aren't concerned with the skynet-esque enslavement of mankind or paperclip maximizing, but that controls be in place that prevent people from using the technology in a way that is misaligned with the maximum extraction of profit?
Your "realization" is counter to everyone on the anti-safety side being monetarily incentivized (VC's, OpenAI employees with $1m+ pay packages, startup founders seeking to get rich) compared to the people that never aimed to profit at all like Helen Toner, Yudkowsky, Bengio, and Hinton.
I've mentioned in past posts that the terms "safety" when applied to the AI discussion are never quantified or qualified in any way.
It's more of a smear. If you're the one arguing from an AI safety standpoint, it means your opponent is not being safe and is dangerous.
You should be expected to articulate why your position is "safer" and how "safe" is defined.
In my opinion most AI safety arguments are about vaguely defined variables like speed: "it's going too fast" - to who? who defines what fast is? do you slow down 2x or 10x? do competitors and/or rival nations less concerned zip by and the end result is still the same?
There is some validity to the discussion if AI is being "racist" or "biased" (again however you define these) but again competitors and/or rival nations may be less concerned and the end result may still be the same.
If anything the "safest" option is to define what your concerns are, then race ahead to try to be the canonical solution or defined standard in order to set standards for the rest to follow or be bound by.
Its like this with a lot of industries. E.g. Meta is probably interested in social media “safety” to the extend it moats them legal protections and allows them to define the regulatory environment they operate within through lobbying. Likewise for automakers or any other business with a chance for harm.
At this point, I suspect that "AI Safety" has taken on the practical meaning of "regulations to make sure the big guys are the ones who benefit"... which is probably true anyways because of the capex needed to run these massive LLMs, but I am sure they would still like a moat or two against, e.g. the Chinese.
Otherwise I don't see what this means now; that the LLMs e.g. dont use racist terms? ok, great, nice, but how is that anything more than what you need to do on the web now anyway? how's that related to "AI" at all?
What I'd love to see is that this gambit backfires and instead we start talking about "tech safety" and that creates actual regulations with teeth that cut down the techzillas a bit (or a lot).
The most embarassing thing about this isn't that the board decided to fire the CEO. That happens all the time. Many startups replace their CEO, it isn't always clear whether it's the right decision, but that's the responsibility of the board.
The embarassing part is that the board decided to fire the CEO, announced their decision, refused to say why, attempted to put in place a new CEO but had to immediately demote the new CEO (Mira) after she rejected their plan, upset and alienated their core partners, along with almost all of their employees, and then publicly backtracked to undo the firing that led to this all happening.
Once you screw this up to such an incredible degree, how can anyone really trust that you were doing the rest of your job well?
Interesting how an intellectual movement claiming to know better than anyone else how development of an unpredictable technology might pan out over the course of years failed to predict how one decision would pan out over the course of a single weekend.
Perhaps there's some level of overconfidence at play from systems thinkers who overintellectualize their ability to conceptualize and extrapolate forward an impossibly complex system.
> Perhaps there's some level of overconfidence at play from systems thinkers who overintellectualize their ability to conceptualize and extrapolate forward an impossibly complex system.
You may well be on to something. I'd trust a cabal of science fiction writers more than that I would trust these self appointed governors of our collective future. They lack imagination, for starters.
I never said I didn't like them. I said they have asterisks behind their names indicating that putting them on a board of ethics may not be the best idea.
Sorry that Heinlein one is paywalled I'll look further.
I've read all his stuff, even the horrible mess of the last books and posthumous thing.
He was completely into turning into a woman even early on much less the book where he had his brain put into a secretary who died in an accident and went into the ramifications.
I say he as Lazarus Long and the rest were him.
Got it. Had to re-read thread, you were responding to someone else that put these authors on a pedestal, to be trusted more than corporate leaders. And you were just saying, as good as the books were, they aren't saints either.
I'm not sure we want Orson Scott Card deciding the future of humanity given his outspoken racist and homophobic beliefs, not to mention his warnings about Obama raising a secret army to become the next Hitler.
No, I didn't have him in mind. But he did have imagination. How many people do you know that can say that they founded a church that is even crazier than the ones that were already out there?
Wasn't that a bet with Heinlein? His entry was "Stranger in a Strange Land" which having a human raised by Martians as the main person was a bit better
Oh and the fosterites were poking fun at Scientology, Mormons, and proto megachurch People.
Considering they get most of their doomsday predictions from science fiction I’d say that would be a smart bet. Why get it second hand, just go right to the source.
I mean, that's essentially the thesis of the notkilleveryoneist position: We don't know how to control powerful agents, and we need to pause AI development in order to figure out.
>Perhaps there's some level of overconfidence at play from systems thinkers who overintellectualize their ability to conceptualize and extrapolate forward an impossibly complex system.
Actually this is more or less the point that Eliezer Yudkowsky makes in this essay about the need for caution in AI development:
I doubt overconfidence is a problem specific to effective altruism. In any case, any good machine learning engineer knows that a dataset with only a single data point is essentially worthless -- even if we grant the premise that the board took the wrong action given the information they had available to them at the time.
It's ironic how they'd have been more successful if they had followed recommendations from their own LLM [1] (or Bard [2]).
Of course the recommendations are not that novel; that's CEO succession planning 101. But I guess none of those four have done any large succession planning, and were clearly out of their depth.
> how development of an unpredictable technology might pan out
> how one decision would pan out
I'm not sure what point you are making here. Are you trying to say "see, the AI not-kill-everyone-ists couldn't predict the future even in the short term, therefore we shouldn't put much credence into the the idea that the specific examples of AI doom they have given will happen"?
Or are you trying to imply that the idea of AI doom as a whole is bunk, because we can't predict the future... therefore everything will be fine...?
> but had to immediately demote the new CEO (Mira) after she rejected their plan
According to reports, Mira was informed of the decision to fire Sam the night before Sam was fired. There's no indication she was against it and she also didn't inform Sam. So, I don't get this "she rejected their plan".
If the board informed her they were going to fire Sam, it seems logical that they also informed her they were going to give her the interim CEO position. I have no knowledge of this (it just kind of makes sense to me).
That may have revolved around the fact that there was no valid reason that they could put up to defend the firing. After all, if they had then she may well have stayed on board, the same goes for CEO #3.
> Once you screw this up to such an incredible degree, how can anyone really trust that you were doing the rest of your job well?
As someone who pretty harshly described the board's actions, I agree with the first part of what you wrote, but I don't agree with this.
People can be good at critical aspects of their job while lacking the maturity to understand large group dynamics. I thought this quote from the article said a lot:
> The board members weren’t prepared for the fallout from their decision. The members, including Toner, were taken aback by staffers’ apparent willingness to abandon the company without Altman at the helm and the extent to which the management team sided with the ousted CEO, according to people familiar with the matter.
It's pretty hard to me to think, based on the board's initial press release, that there wouldn't be a huge fallout from this decision. The release also made me think that Altman had done something truly egregious/malevolent, and that wasn't the case.
What I saw this as was a group of people who had real concerns about Altman's leadership, and some of those concerns ended up sounding pretty valid, but who didn't have the experience/maturity to understand how to go about the next step. OpenAI's "frankenstein" org structure, with a board beholden to the nonprofit's charter but also responsible for billions in investor capital, also helped contribute to this cluster-f.
>What I saw this as was a group of people who had real concerns about Altman's leadership, and some of those concerns ended up sounding pretty valid,
What did you hear that sounded valid? Toner had her chance to make her case here, and it was simply that Altman tried to get her fired, so she retaliated. It wasn't about safety or leadership. It was petty politics over board control.
The article alleges that the board compared notes after speaking individually with Altman and determined that he had misrepresented statements/opinions from the other board members when speaking to them one on one. This caused the board to believe Altman is/was not being genuine and honest with them.
Is an attempt to manipulate the other board members into firing one board member reason enough to be fired?
> Is an attempt to manipulate the other board members into firing another board member reason enough to be fired?
No, but lying to Board members about what other Board members have said is. The employee equivalent would be lying to one member of the C-suite about what another had said.
The board should have shared those specific details with their initial announcement. Might not have been enough to win over the employees/stakeholders, but they'd have a much better chance that way than how things actually played out
But there were plenty of lawyers on all sides with some threatening to sue anyone who said a bad word about Altman regardless of the truth. A threat can be exactly worded or implied.
The rack up debt and fail on mediocre idea and then sell your only valuable resource, your human employees Plan gave him a lot of money to hire a lot of law school grads who have since made partner.
> Altman tried to get her fired, so she retaliated
I don't think it's that simple.
First, after confronting Toner directly and getting a response he didn't like, Altman went behind her back to the other board members trying to get her fired. That's not what a responsible CEO does. A responsible CEO, if he does not agree with the position taken by one Board member, brings that concern openly to the rest of the Board in a formal setting. He doesn't go behind the one Board member's back. If Altman really believed Toner's actions had harmed the company, his responsibility as a CEO was to bring that concern openly to the Board and let them make the call openly. And that's especially true for a company that, according to Altman's own public claims, has stewardship of a technology that could be an existential threat to humans. That makes any behind-the-scenes maneuvering on his part even more of a problem.
Second, according to the article, other Board members had concerns about Altman's honesty, and not just due to whatever transpired during Altman's behind-the-back attempts to get Toner fired. So it's not just that particular incident that's relevant here.
That's not to say that the Board didn't make mistakes. I agree with other posters that they should have seen the consequences coming for firing Altman. It also appears, from the fact that the lawyer's advice about fiduciary duty came after they fired Altman, that they didn't seek legal advice before doing that, which seems like an obvious mistake. (It's possible, though, that they did ask the lawyer beforehand and he didn't raise the possibility of a backlash and consequent damage to the company, which would put at least some of the blame on him.) I just don't think that because the Board messed up, that makes Altman the good guy. I don't think any of these people come out of this looking good. Certainly I don't have confidence in any of them as stewards of AI if AI is really as big a deal as they think it is.
> brings that concern openly to the rest of the Board in a formal setting
Yes and no.
Meetings are a terrible way to drive toward decisions but a great way to signal positions to others. Or "decision meetings" bad; "consensus meetings" good.
If you're driving one of those, you should know how everyone involved feels before they meet. You should lobby any of them beforehand if necessary. The idea is to try to make sure you have buy-in before everyone shows up - no drag-out arguments, no rabbit trails, just statements of their positions and a thumbs up.
And Altman seems to know that, given his behavior.
> Meetings are a terrible way to drive toward decisions
The decision Altman was driving for was to fire a Board member he disagreed with. That's fundamentally different from an ordinary Board decision, and should not be lobbied for in advance the way you would organize support in advance for a normal Board decision. When the CEO of the company has a fundamental disagreement with a Board member, and wants to get that Board member fired, lobbying beforehand behind the one Board member's back is fundamentally dishonest. He should put the disagreement before the entire Board openly.
Btw, if I were Toner, as soon as I knew I had a fundamental disagreement with the CEO, I would have told the other Board members and requested a Board meeting to discuss the matter (and told the CEO I was doing those things). The meeting would not have to be an actual decision meeting with the Board formally in session. But I would want any discussion to take place with everyone present and with everything openly out on the table, not in behind-the-back individual sessions. I'm somewhat surprised that Toner didn't do that.
The scariest two words: 'unconsciously incompetent'. That's where the biggest accidents come from. At least you can recognize malice and do something about it.
> People can be good at critical aspects of their job while lacking the maturity to understand large group dynamics.
I agree with this as a general principle, but in this specific case, the 'understanding group dynamics' part is the 'critical aspect of the job' part. By the same token, you don't just promote the best engineer to CEO if he's bad at managing, because the managing is the critical part of the job.
>What I saw this as was a group of people who had real concerns about Altman's leadership, and some of those concerns ended up sounding pretty valid, but who didn't have the experience/maturity to understand how to go about the next step.
That and that alone is enough to question their competence, though. The impression I get is that Toner has been in the academic/nonprofit world for so long that she doesn't understand how the real world works. In those places, the sad truth is nothing you really do is of much consequence to anybody in the grand scheme of things, other than you collecting prestige and a paycheck.
Then she tried making a consequential decision in an organization valued at around $90 billion, and lo and behold, people started caring about her actions in a way she has never experienced before.
"Once you screw this up to such an incredible degree, how can anyone really trust that you were doing the rest of your job well."
How can anyone take any "AI" company seriously other than as a get rich quick scheme. Doubtful a successful investor focused on management would bother with a company like OpenAI.
"AI" is an inferior form of autocomplete. Autocomplete normally the user to choose the most appropriate completion, or choose none of they all suck, while "AI" does not even allow the user to select from multiple choice, it forces one on the user, no matter how dumb. It's a classic tactic used by so-called "tech" companies: remove user choice.
Embarrassing? It’s a complex system and it’s impossible to predict the future when so many parties are involved. Altman started the chaos, but somehow avoids the blame.
OpenAI's oversight structure is plainly stupid by any measure. Their CEO and core "partners" liquidated the board and reformed it within a few days of his firing. The buck stops with Sam. The only thing Helen Toner screwed up was being an associate professor with net worth < $500,000,000.
I've made this point before, but the ratio of "people criticizing OpenAI's legal structure" to "people proposing a better legal structure" in these discussions is remarkably high.
Not one critic I've seen has stated what the best legal structure for OpenAI should actually have been, in a world where OpenAI really is stewarding a technology that could be an existential threat to humans. I really wish they would do so, because then we'd have a shot at improving OpenAI's governance. But so far it's been crickets.
BTW, I think one of the big lessons of this episode is that legal structure matters much less than you'd think. On paper, OpenAI is supposed to follow its charter https://openai.com/charter and the board had the power to fire Sam. In practice, Sam was kinda sorta violating the charter, but he's a charismatic leader who was about to bring in a big payday for the employees (https://www.cnbc.com/2023/11/30/openai-tender-offer-on-track...), and that turned out to matter more than the org's legal structure.
I'm not advocating anything be broken. But if they're really doing what they say, and what I assumed you referred to by existential risk -- developing technologies "akin to the hydrogen bomb", that stand to "capture the light cone of all future value in the universe", etc. -- then of course that has to happen.
Okay, I'll take a shot. I think the legal structure is very important, but the important part is not the vague wording of the charter. "Violating the charter" is always fuzzy and debatable. "Getting the votes of most board members" is not. So the important part is who controls the board.
Instead of a structure where the board members get to choose other board members, I would suggest a structure where different outside groups get fixed numbers of board members. So, for example, it could have been (remembering that the initial board was just Sam Altman and Elon Musk):
* One board member chosen by Sam Altman
* One board member chosen by Elon Musk
* One board member chosen by OpenPhil
* One board member chosen by Microsoft
* One board member chosen by some VC investor
The precise composition is something they would argue about during funding, maybe some of these groups would get more or less seats.
The root "governance" problem is that the board churned a lot because so many people had conflicts, or other reasons that I don't know as an outsider. The remaining board was unrepresentative of stakeholders (ie, no Elon Musk, nobody independently trusted by Microsoft) and probably not as experienced as you'd like. The problem with this sort of structure is that it gives everyone a lot of incentives to play politics with the composition of the board itself. Which happened a lot here - Reid Hoffman exiting and not really getting replaced by a similar person, the debate between Sam and Helen about her presence on the board, probably other cases.
I bet they end up with something similar, with different stakeholders represented. Give Bret Taylor some time to cook.
>The embarassing part is that the board decided to fire the CEO, announced their decision, refused to say why, attempted to put in place a new CEO but had to immediately demote the new CEO (Mira) after she rejected their plan, upset and alienated their core partners, along with almost all of their employees, and then publicly backtracked to undo the firing that led to this all happening.
I'm reminded of these great PG tweets:
>When people criticize an action on the grounds of the "optics," they're almost always full of shit. All they're really saying is "What you did looks bad." But if they phrased it that way, they'd have to answer the question "Was it actually bad, or not?"
>If someone did something bad, you don't need to talk about "optics." And if they did something that seems bad but that you know isn't, why are you criticizing it at all? You should instead be explaining why it's not as bad as it seems.
The situation is similar if you criticize an action on the grounds that "it upset people". If the action was actually bad, you should be able to explain why directly.
Here are some quotes from the OpenAI charter:
>We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome.
>...
>Our primary fiduciary duty is to humanity.
>...
>We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project.
>In October, Toner, who is director of strategy at a think tank in Washington, D.C., co-wrote a paper on AI safety. The paper said OpenAI’s launch of ChatGPT sparked a “sense of urgency inside major tech companies” that led them to fast-track AI products to keep up. It also said Anthropic, an OpenAI competitor, avoided “stoking the flames of AI hype” by waiting to release its chatbot.
>After publication, Altman confronted Toner, saying she had harmed OpenAI by criticizing the company so publicly. Then he went behind her back, people familiar with the situation said.
Who is doing a better job of upholding the charter in this situation: Altman, or Toner?
The "people are upset" criticism essentially states that people became upset when Toner attempted to actually uphold the charter. To me, that says more about those people than it says about Toner.
I read it pretty much the same way. I do have some ideas on how you should structure the board of something like OpenAI, but it is more along the lines of a broadly represented structure than five industry experts and a customer or two.
For this to work you need all layers of society to be represented and since this is intended to be global you'd be looking at the UN. Of course that's 100% anathema to anybody involved with OpenAI or even any cross section of HN because 'bureaucrats don't get technology and the UN can't agree on anything'. But if it really is what it seems to be that to me seems to be the only venue where this can be addressed.
Of course the race is on and the first party to cross the threshold will get the gold and never mind the consequences so I expect any attempt at oversight to be murdered the second they have it working.
If you have ideas, I encourage you to write a blog post and submit it to HN.
Somewhere I got the notion that the UN is generally considered to be dysfunctional these days, and that's part of the reason there's been a shift in diplomacy to other venues like the G20. Representativeness is important, but it's just one of several important desiderata in AI governance.
It seems to me that the current important decisions facing OpenAI are for the most part fairly technical, so it's good for their oversight to be technical. Representativeness becomes more important later. I'd say if OpenAI actually does succeed in developing AGI, that's the point at which the board should be replacing itself with some other, more representative institution.
>Of course the race is on and the first party to cross the threshold will get the gold and never mind the consequences so I expect any attempt at oversight to be murdered the second they have it working.
Maybe a good question from a governance perspective would be, how can we configure the institutional incentives to discourage this, or at least select for leaders with strong character and ethics who won't ruthlessly seek power at all costs?
I've been criticizing Sam Altman a fair amount lately, but it sounds like you take it much farther in essentially viewing him as some sort of crime boss, or else you're assuming that anyone with significant power will automatically adopt crime boss ethics as a result. I doubt the latter statement is true -- for example, I don't believe that Obama adopted crime boss ethics when he became POTUS.
I think the main reason the UN is dysfunctional is because the largest countries have been effectively in the driving seat and the rest is along for show. As long as there is veto power for a select few and the big military entities ignore any restrictions laid in their way to do what they want to do (by way of veto or otherwise) it will continue that way. But that breeds a ton of resentment elsewhere in the world and the people in those countries effectively have no representation, even if they have the vote. It's apartheid writ large, based on weapons and money.
> It seems to me that the current important decisions facing OpenAI are for the most part fairly technical
For OpenAI, yes. But for society the stakes are much higher and non-technical in nature. For instance: mass disruption of the labour market. Note that when we had the industrial revolution the world population was a fraction of what it is today.
> so it's good for their oversight to be technical
But that could change in a heartbeat and by then it is too late. Winner takes all.
> Representativeness becomes more important later.
There won't be a 'later', whoever gets this first will instantly weaponize it and seize power, though you won't know about it until you realize that it's a done deal. Stable dictatorships are one of the more likely outcomes of AI.
> I'd say if OpenAI actually does succeed in developing AGI, that's the point at which the board should be replacing itself with some other, more representative institution.
Right now our best hope is that they can't. If not, all bets are off and you may want to brush up on your Charles Stross.
> Maybe a good question from a governance perspective would be, how can we configure the institutional incentives to discourage this, or at least select for leaders with strong character and ethics who won't ruthlessly seek power at all costs?
Leaders with strong power and those who ruthlessly seek power at all costs have strong overlap, assuming they are not congruent.
> I've been criticizing Sam Altman a fair amount lately, but it sounds like you take it much farther in essentially viewing him as some sort of crime boss, or else you're assuming that anyone with significant power will automatically adopt crime boss ethics as a result.
I see him just like I see Elon Musk: brilliant but dangerous, not necessarily a crime boss, more like someone who doesn't realize that they don't have the right to make the decisions they make and too little in terms of reflective capabilities and humility.
> I doubt the latter statement is true -- for example, I don't believe that Obama adopted crime boss ethics when he became POTUS.
Obama was a huge disappointment, so no argument there, but the flipside is that Trump was exactly what he promised to be and more and the world came out for the worse anyway. It makes you wonder how much worse that can get. Probably a lot worse. I probably don't want to know.
> When people criticize an action on the grounds of the "optics," they're almost always full of shit.
You're characterizing this like it was just a bunch of Twitter armchair quarterbacks.
Which is was.... Plus 90%+ of the boots on the ground who voted no confidence in their leaders.
Only the most obstenate can cling to the belief they are a good leader/communicator with that statistic.
EDIT: There is a feeling that companies are full of unremarkable, out-of-touch, over-compensated leaders who ignore the opinions of their subordinates.
And yet when that situation actually materializes, or mostly materializes, you read opinions like this.
I'm concerned that boots on the ground may be susceptible to motivated cognition due to their financial interest. I suspect that their finances come before upholding the charter for many of them. Why else did they get so upset when Helen said that shutting down OpenAI could be the right thing to do? That's pretty much implied by the charter.
If you're still young and idealistic, it's maybe ok to think like this, as you've likely had nonsense poured into your head for years & little real-world experience. If you're as old as me and you still think this way, there's little hope for you.
I have worked for many CEOs, and (surprise!) fewer than 100% of them are sociopaths. Good grief.
I think the OpenAI shenanigans are far from over and the next act is likely to play out. A couple of my personal opinions are:
- As PG pointed out Sam would find his way to the top of an island of cannibals. If you're generous to Toner's side in this article she likely _was_ alarmed by how Sam tried to get other board members to turn against her. When she walked into the arena with Altman to try and remove him however he had much more experience and orchestrated an incredible counter-coup.
- There must be an army of attorneys representing the board members, and investors because there are huge potential legal pitfalls to destroying value, or failing fiduciary duties.
- We don't know a lot because anything put out in public has to be spotless legally. If you're a board member and you know the truth, the only upside to leaking more info is clout, but the downsides are being sued and tied up in court losing a lot of time and money.
- People seem dissatisfied with what was revealed in this article but Toner has likely been advised by lots of counsel on exactly what can be said, and the list of those things is probably miniscule. I'd say she's doing the best she can to give the public more info and I'm happy she's taken the effort to do so.
> When she walked into the arena with Altman to try and remove him however he had much more experience and orchestrated an incredible counter-coup.
Nit: Altman may have orchestrated a coup, but the board certainly did not. The board is the legitimate authority, and in no way is a board firing its subordinate a "coup." Rather, it's executing its legitimate power.
Coups triggered when subordinate feels threatened is a pretty common pattern. IIRC, the recent coup in Niger was triggered when the leader of the presidential guard thought the president was about to fire him, too power in a coup rather than lose his job.
Presumably, Toner et al agreed to this compromise because they believed it was a meaningful improvement compared to a board composed solely of Sam and friends.
The "fat lady hasn't sung" point works both ways -- for all we know, the new board's investigation into this incident will be incredibly damning for Sam.
That's true, it may well be that. Or it may be a whitewash because he has them by the short and curlies now that it is known what throwing him out will do.
The board's most important job is to figure out if the CEO should be fired. Your comment seems to imply that the board should never fire a CEO if that CEO is on the board. If that's the policy, then having a board with the CEO on it seems fairly pointless.
The board met but didn't include the board chair, and voted without him being present (they could form a quorum). I am not so sure it's exercising your legitimate power if you don't include the board chair on purpose.
Of course it is: otherwise, how would you ever remove the board chair? Quorum is normally all that matters as long as the rest of the bylaws of the non-profit board are followed. Whether it is elegant or not is another matter.
Regarding the narrative about Sama's behaviour during this fiasco, I believe the perception of him acting strategically is mistaken. I think he was just highly loved by the employees and that alone is what facilitated his return.
This strikes me as a very odd take. All the things that seem to "hasty" and non-strategic to you look to me like deft PR moves that each had exactly their intended effect.
> lawyers [...] huge potential legal pitfalls to destroying value, or failing fiduciary duties.
> advised by lots of counsel on exactly what can be said, and the list of those things is probably miniscule.
Hmmm, in the US yes and no. First, US lawyers point out risk in every direction. Because nothing is risk-less and the law is unclear or unsettled (and anyone can sue for just about anything). Once they have duly warned to their satisfaction however, their job becomes to come up with a theory on how to do exactly what it is you want to do. The two are different tasks, and are not incompatible.
That's basically the definition of a good business lawyer.
She didn’t think there would be employee backlash when they fired the guy who was almost finished closing a financing that would make most senior employees millionaires?
I respect that she and other board members didn’t like that Sam was trying to manipulate them into ousting her but their counterattack was poorly thought out.
No, and I don't think it particularly matters. In practice, the people who signal they'll make their friends rich will have the most backers. The board is just there to give the appearance of checks and balances. If these two claims sound outrageous consider that this is exactly how it's playing out.
OpenAI was formed as a nonprofit with a specific charter ("OpenAI’s mission is to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work—benefits all of humanity. We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome.") and the capped-profit entity under which daily operations occur formed years later with the claim that it was instrumentally useful for pursuing that charter. The capped-profit entity remains a subsidiary of the nonprofit. The board in the dispute is the board of the overseeing nonprofit.
So there are many particulars that mean pattern matching to a standard board dispute will lose something. I think it's likely many of the primary actors have, at various times, had non-strictly-pecuniary motives. That one side won doesn't mean that the other side was always a farce.
Ok thank you, but I haven't changed my mind. In practice people with more power will get their way, regardless of what's written on the charter of some company.
Any board member who believes it is her duty to destroy the organization she pledged to serve should simply step down.
The folks who argue that it is a CEO's duty to serve the board do not understand governance or power.
Open AI would have wallowed in irrelevance had Altman not raised the billions from Microsoft to fund its research. Because he was able to do that, and built the relationship with Nadella, he had and has power. Toner's behavior seems naive in that context. But naiveté among academics does not surprise me.
No, my argument is: if you do not know how to navigate power, you have no business on a board that purports to govern a fast-growing, prominent and critical organization. This board does not exist to serve itself, its abstractions or its ungrounded anxieties. It exists to practically guide a real organization employing many hundreds of people and serving many millions. There is no room for naiveté.
My reading: They weren't picked for their ethics, they were picked for being malleable, to serve as a fig-leaf and to be able to mislead regulators. To Sam's surprise some of them took their job serious, so he tried to get rid of them because he saw the confrontation as inevitable. That backfired, he got terminated, and then that backfired so the boardmembers got the choice to jump on command or vacate the premises.
You can be fairly certain that the board will be stocked with 'yes-men' (and women) beholden to Altman. He will not make that same mistake twice.
I mean, it's totally fine for a non-profit to "wallow in irrelevance"? I think the fundamental issue here is that a tax-exempt non-profit organization like these board members are leading _shouldn't_ necessarily be chasing fame/fortune. By definition they've put mission above profits (with tax writeoffs as a benefit). The naivety seems to be that the company is trying to be altruistically mission driven while also acting like a typical ambitious/profit seeking startup. Startups are great! Just weird to wrap one in a non-profit which brings different incentives
He might have been the one to build the relationship but this hardly gives him “power.” He did a sales job basically. Sales people get replaced all the time.
"Pournelle's Iron Law of Bureaucracy states that in any bureaucratic organization there will be two kinds of people: those who work to further the actual goals of the organization, and those who work for the organization itself. Examples in education would be teachers who work and sacrifice to teach children, vs. union representative who work to protect any teacher including the most incompetent. The Iron Law states that in all cases, the second type of person will always gain control of the organization, and will always write the rules under which the organization functions."
> organization she pledged to serve
First there is no pledge, second I think she served the Open AI Charter.
openai is where it's at because of sama & greg (& ofc contributions of employees, but funding & solid engineering leadership is a big part in its success), to think that she could just push them away without it backfiring is mindblowing.
> Helen Toner was a relatively unknown 31-year-old academic from Australia—until she became one of the four board members who fired Sam Altman
> Toner graduated from the University of Melbourne, Australia, in 2014 with a degree in chemical engineering [...] In 2019, she spent nine months in Beijing studying its AI ecosystem. When she returned, Toner helped establish a research organization at Georgetown University, called the Center for Security and Emerging Technology, [...] She succeeded her former manager from Open Philanthropy, Holden Karnofsky, on the OpenAI board in 2021 after he stepped down.
Honest question, how do people find themselves in the board of one of the hottest startups at the tender age of 31? Are they geniuses or is it all about connections?
These people took chances on their belief in the future, that AGI was coming and monumentally impactful, back in 2015 when the entirety of people on HN mocked and derided it. Even in 2023, the majority do.
Right place, right time. She was one of very few people "working" on AI Safety. I made another comment about this elsewhere on this post, but to me this debacle is a function of the fact that there is no there there when it comes to AI Safety "research".
I encourage anyone who thinks there is "no there there" to, for example, glance over this preprint and some of its references: https://arxiv.org/pdf/2310.17688.pdf
Her profile - as you state - checks lots of boxes: concerned highly credentialed young woman interested in an emerging field that is part of china-us battleground. Would look great on any corporate brochure
I’m glad she’s off the board. You cannot have a trigger happy fire an important ceo over some to me is a small trust issue, infact it seems like she’s over using her powers and “hold principle” and destroy everything even to achieve it. It proves she cannot think inbetween, saying it’s “protect humanity at all costs”, you really need a brain to execute that and she has proven it was a massive f-up
> You cannot have a trigger happy fire an important ceo over some to me is a small trust issue
Well, if that was the case it actually was fine but there is no hard proof that it was like that and if it was they messed it up. So failure on all counts.
This doesn't sound like a small trust issue. It sounds like either Altman had to go or she had to go, and Altman made that decision himself, he really only gave her two options: get Altman fired or resign.
Allowing the board to control AI safety is BS. Is like ethics, the engineer is usually imbued with ethics when they work at any firm. It’s that simple, you work at an airline manufacturer, your duty is to not cause harm, test properly, it’s no different with AI. The engineer understands this more than any board member getting paid to do nothing.
>the engineer is usually imbued with ethics when they work at any firm. It’s that simple, you work at an airline manufacturer, your duty is to not cause harm, test properly, it’s no different with AI.
One point here is that there's no licensing requirement for AI engineers like you see in some other fields of engineering. As someone recently quipped on Twitter:
> did you know that there are currently no licensing requirements in the US for training potentially world-threatening AI models, but there are in the state of Louisiana for selling flower arrangements
In practice, AI engineers self-select into different places of employment depending on how worried they are. If you're not very worried, you work at Meta. If you're significantly worried, you work at Anthropic or even someplace like MIRI. Most AI labs have a safety culture that's somewhere in between those extremes. [Obviously, from the point of view of preventing extinction, this situation seems unsatisfactory.]
It seems to me as if Toner was just in the right place at the right time in order to get a seat on the board of Open AI. She has an undergrad degree in chemical engineering and a master's in "security studies." Her work on AI Safety opened this door for her but seems, at least to me, to be ... superficial. I'm no expert but I have worked in tech for awhile and at a public policy think tank. So I guess my question is there a real scientific field of AI Safety? Are there any real experts? Are there any real insights? I dislike the idea of trusting giant tech companies with breakthrough technologies with minimal oversight / regulation. But it just seems like there is no real science regarding AI x public policy. Like the policy experts have no clue what they are talking about. And after this debacle they probably won't be lucky enough to find themselves on the boards of organizations like Open AI.
A group from Cambridge has been running an online course you can take if you're interested in getting up to speed: https://aisafetyfundamentals.com/ Application deadline is early February. I took the course a couple years ago; happy to answer any questions.
AI Safety or lack thereof, to the extent that it emerges as a real threat, will be a matter for law enforcement.
Just like “terrorism” and “drugs” and all the other vaguely menacing things that ended up being worth a pile of money to some incumbent interest, the real story will be boring: some crooks used a fancy-ass LLM to take a bank apart or something. The bank calls the cops, this happens about twice before the cops have a division for this sort of thing, and sooner or later someone is in a courtroom.
It would be nauseatingly boring to get into how many times this has happened.
Also predictably depressing, the real story, which is yet another brick in the wall of “we’re handing the reigns to people who make movie Zuck look flat fucking normal”, goes largely unreported.
We used to complain about dual-class share structures.
Sam just eviscerated a board of an ostensible charity with the full backing of MSFT and got himself installed as Ungovernable God Emperor of Arrakis for life. Still doing the eyeball scanner thing.
Still has never shown the public he can handle numpy to the tune of MNIST.
This is all in broad daylight and the Gemini launch gets clobbered because the 4-series comparable one comes out in January.
Sam’s a smart guy, I’m quite sure he’d pay attention if the braintrust (this site) complained loudly.
Helen Toner has proven herself to be a poor communicator. This is what will happen if you give inexperienced people, that have not earned it, board seats.
> Some of Altman’s backers, including OpenAI investor Vinod Khosla, publicly expressed derision specifically toward Toner and Tasha McCauley, another former OpenAI board member who voted to fire Altman and is connected to organizations that promote effective altruism.
> “Fancy titles like ‘Director of Strategy at Georgetown’s Center for Security and Emerging Technology’ can lead to a false sense of understanding of the complex process of entrepreneurial innovation,” Khosla wrote in an essay in tech-news publication the Information, referring to Toner and her current position.
Strong words from the infamous beach villain and resident asshole, Vinod Khosla...
253 comments
[ 3.5 ms ] story [ 302 ms ] threadGoing fwd, I wonder why they cannot convert the structure to a traditional c-corp. Supposedly: “tax issues”.
Whatever problems OpenAI dealt with last week, the current non-profit structure will continue to cause future problems IMO.
Great learning opportunity for those that have not taken time to learn and understand non-profit structures in the USA.
this is clearly an opportunity for the uninformed to study the US non-profit system, unique in the world. For example, I recall a delegation from Japan touring the HQ of a non-profit educational group in San Francisco. Like the p-poster here, they apparently had no clue why or how capable people could come together and form stable work culture aside from "hierarchical, profit motive in USD$" (.. and people did actual work, unlike many management level stock manipulators).
[0] https://en.wikipedia.org/wiki/International_Baccalaureate
"In the aftermath of the OpenAI governance battle, the narrative has been dominated by the idea that “the capitalists won.” While this conclusion is catchy, it misses the point.
The real (and bigger) story is: The founders of OpenAI set out to create a nonprofit public institution to ensure that AI’s evolution aligns with the interests of humanity at large. Last month, it became clear that they failed.
What is also clear: Public institutions that prioritize the public good over short-term profits are desperately needed for the AI era."
I read it as both. If OpenAI collapses operationally then the non profit fails at its mission too. That’s the board she is on and that’s the fiduciary duty relevant
(She may also be on the for profit board idk)
Not necessarily true.
"We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome [...] if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project."
https://openai.com/charter
Because, IIRC, when a charity nonprofit like OpenAI, Inc.-- a 501(c)(3), other nonprofits are different -- converts to a for-profit (C-corp or otherwise doesn't matter), all the assets acquired while it was a charity must either remain dedicated to and used exclusively for the charitable purpose, be returned, or be donated to charity (or sold, and then the proceeds treated the same way.) For the charity OpenAI, Inc., this would include its interest in OpenAI, LLC (the wholly owned subsidiary through which it exercises control of OpenAI, LP and OpenAI Global, LLC -- the latter being the for profit entity that actually sells product, etc.), and OpenAI, LP (the holding company which in turn is majority owner of OpenAI Global, LLC).
also OpenAI Inc. for profit doesnt exist, a wholly owned LLC subsidiary does. so they would need to divest of that LLC first (by selling the shares)
the way around the self dealing regulations are to reduce their ownership below 20% - 35% (yes, its that range, for multiple reasons)
good thing here is that most venture backed companies subvert this already by simply share dilution and youre breaking into one the best secrets of Silicon Valley. Non-profits pump the value of startups and launder the reputation of the founders because their ideas get so much “validation” so quickly. (non profits they partially control act as lead investors for startups they control. they both sell shares at higher price, ideally: founders get QSBS tax free treatment, non profit retains tax free capital gains too. nothing wrong with that, you just probably cant do it too)
so open ai for profit llc needs to get rid of the PPU’s and bring in outside investors to dilute ownership enough, but two birds one stone: the outside investors have to get a higher valuation
The for-profit "OpenAI" (OpenAI Global LLC) is not a wholly-owned subsidiary of anything, it is majority owned by a holding company (OpenAI, LP), which in turn is majority owned by the OpenAI charity (OpenAI, Inc.)
There is a wholly owned LLC subsidiary of the charity (OpenAI GP, LLC), but that is the entity through which OpenAI, Inc., exercises control of OpenAI, LP and OpenAI Global, LLC (why it doesn't do so directly for OpenAI, LP, and either directly or via OpenAI, LP, for OpenAI Global, LLC, don't ask me), it is not the for-profit.
Both OpenAI, LP, and OpenAI Global, LLC have other investors (Microsoft being a big one for OpenAI Globla, LLC) already.
one explanation is “blockers”. basically regulations imposed on one kind of entity such as the non profit, fail when there is one degree of separation via a “blocker” entity.
LP’s can have undisclosed silent limited partners, intentionally less flexible than an LLC which can do the same
Fiduciary doesn't mean financial or monetary duty, it means holding something in trust. Violating a fiduciary duty could be financial, social, reputational, or in this case, literally destroying the operation of the company you were entrusted with.
Yeah, that smells like a bullshit threat. Anyone would do well to ignore it.
> Toner declined to provide specific details on why she and the three others voted to fire Altman from OpenAI
Does she see herself as more trustworthy? She can't even be bothered to give an excuse for the firing.
Someone like this couldn't be trusted in literally any function, how did she get a board seat on OpenAI?
She's the leading academic on the issue of AI safety. It's really ridiculous people don't even know her name and say random things about her, not realizing she's a rock star in her field.
Her leaving the board is a tremendous loss for OpenAI, about as terrible as Ilya leaving the board. These are two giants of AI currently.
Sounds like a made up position to me. Just another bureaucrat with a cool sounding name
https://scholar.google.com/citations?user=NNnQg0MAAAAJ&hl=en
Like, it's not a bad publication record, especially for a junior researcher (heck, her first participation in other people's papers is just in 2018 and her own work starts in 2021-2022, so she's effectively just getting started and may have many opportunities to prove herself in the future), it does indicate doing a couple years of research and might qualify for an okayish faculty job when (if?) she gets her PhD, but it's not also something indicating a senior researcher that's the pillar of anything and has had some impact - a grad student who's lucky to study in a strong leading researcher's lab and does some work on two of the advisor's papers might have a similar publication record already at their graduation before their career has even seriously started.
Many of her co-authors have significantly larger research impact and personal research (https://scholar.google.com/citations?hl=en&user=VclFrJ8AAAAJ or https://scholar.google.com/citations?hl=en&user=MbBntPgAAAAJ for example or https://scholar.google.com/citations?hl=en&user=0-G2eiEAAAAJ) - which also raises a question of how much each co-author did in a two-author paper where the leading author is a strong researcher and the second author is a "Director of Strategy and Foundational Research Grants" i.e. getting money for that research, so labeling her as a leading researcher (much less the leading researcher) of some field seems misleading as the publication record would indicate that she's not even on the same level as they are.
https://scholar.google.com/citations?hl=en&user=6-e-ZBEAAAAJ
It sounds like someone who talked a lot but didn’t actually create anything, change anything, or make anyone’s lives better.
Happy to revise my perspective if someone can provide any concrete data on her impact.
She was on the board that fired Sam Altman, that had the courage to fire Sam Altman, presumably (according to the article at least) because he is a liar and she and three other board members did not trust him to be loyal to the non-profit's mission of working to keep AI safe.
"In 2019, she spent nine months in Beijing studying its AI ecosystem. When she returned, Toner helped establish a research organization at Georgetown University, called the Center for Security and Emerging Technology, where she continues to work."
I don't think "rock star" is quite the right term.
Unless you think David Geffen or Lester Bangs are rock stars. Normally rock stars sing or play an instrument. Someone who got a nice title in a nonprofit by writing about rock is not a rock star.
Yann LeCun is a leading academic...
I'm embarrassed she comes from my country, but not surprised about the particular uni she attended within my country.
I'm doubtful that the "childish idealist" description is accurate. But even if it was, given the choice between having existentially dangerous AI developed by a childish idealist vs a slippery machiavellian like Sam, I would choose the childish idealist.
The big risk with a childish idealist is that they become the pawn of a slippery machiavellian. Choosing the slippery machiavellian just seems like speedrunning failure.
I’m quite happy with the childish idealist description though. Putting childish idealists in charge of $billions of other people’s money might sound like a great idea to you, but would you put them in charge of your money?
Wikipedia says you have to apply to attend some EA events. That's true for lots of things. TED isn't open to just anyone either. If you want to go to an EA event without applying, most group events are pretty casual and don't require any application: https://forum.effectivealtruism.org/groups (Note: I heard a rumor that EA events in Germany are less likely to be open to the public)
>Putting childish idealists in charge of $billions of other people’s money might sound like a great idea to you, but would you put them in charge of your money?
Well, I have in fact donated significant sums to some EA organizations, and I have a recurring donation to the Good Food Institute (working on meat alternatives). I don't agree with the "childish idealist" description of EA though. From my perspective, it's remarkable how easily Sam Altman seems to manipulate people, and he seems more successful at manipulating people outside of the EA community.
Other than telling me that you have a narrow understanding of a "cult", I'm not sure what this is supposed to communicate.
BTW, for people reading this discussion, here's an alternative explanation of why EA tends to be controversial: https://www.richardhanania.com/p/effective-altruism-thinks-y...
Peter Singer is more of an EA popularizer than an EA leader. The post you linked is the only post he has made in the past ~9 years: https://forum.effectivealtruism.org/users/peter_singer The tweet you linked is related to his Journal of Controversial Ideas, not EA.
It's certainly not the case that Singer should be taken as representative of the average EA, you'd have to look at surveys for that: https://forum.effectivealtruism.org/s/FxFwhFG227F6FgnKk However EA tends to be friendly to contrarian thought experiments (not usually taking it to Peter Singer levels). And EAs usually prefer to focus on getting facts right over condemning each other in an inflammatory way.
Is EA distancing itself from Singer (who's views are, by all accounts, extremely influential in the EA community), as they did with SBF (whose hedge fund's initial $170m seed capital was raised from EA devotees)? When multiple high-profile people associated with EA are proven to be....problematic, should we pretend the common factor linking them is insignificant?
Inflammatory? I thought I was quite restrained compared to some of the commenters on that Twitter thread.
From article, this is what I know about her:
>"Toner graduated from the University of Melbourne, Australia, in 2014 with a degree in chemical engineering and subsequently worked as a research analyst at a series of firms, including Open Philanthropy, a foundation that makes grants based on the effective-altruism philosophy."
>"In 2019, she spent nine months in Beijing studying its AI ecosystem. When she returned, Toner helped establish a research organization at Georgetown University, called the Center for Security and Emerging Technology, where she continues to work."
She's been working in the field of AI safety for what 3 years at most? I find it very hard to believe that she's "thee" leading academic on the issue of safety. Stuart Russell's group, the Center for Human Compatible AI at UC Berkeley, MIT's The Algorithmic Alignment Group and David Kruegger's group at Cambridge are all academic leaders and groups with great expertise in this subdomain.
Further it's not even clear what "studying Beijing's AI ecosystem" actually means. Does this refer to the city? The university? The CCPs' surveillance state(not likely)? It's very hand wavy.
I don't have any problem with what she did or her views. I also think it's a loss for the company. I do question her bona fides for being called the leading academic on the issue though.
I agree that outsiders can't weigh in on this topic without statements from the board.
But as the (former) board member of a private organization, she's not accountable to the public. Making statements about why she fired Altman can carry substantial legal risk.
Exactly. Being a rent-seeking grifter doesn't make you an expert. In anything.
I'm amazed anyone can side with Khosla after he blatantly showed his $ > ethics side when it came to the Martin's Beach scandal[0]. In a battle of $ and ethics, finding yourself agreeing with Khosla is a smoking gun where someone stands.
That we have VC's character assassinating someone in contrast to all the people actually around her who can vouch, is quite a sad legacy.
[0] https://www.washingtonpost.com/technology/2020/01/08/califor...
Those who can, do. Those who can't, write about it.
I was there, as an engineer. I'm a character in the novel.
https://scholar.google.com/citations?user=NNnQg0MAAAAJ&hl=en
None of them are about chemical engineering, which has nothing much to do with AI, anyway, so I don't know why you mention that.
when did she go over to the Dark Side?
https://scholar.google.com/citations?user=NNnQg0MAAAAJ&hl=en
"bridge the AI communities of the US and China": what does that mean? She organized some meetings where platitudes were intoned?
She replaced Holden Karnofsky as a representative of the "effective altruist" group, Open Philanthropy, who donated money early on to OpenAI.
From the article:
"She succeeded her former manager from Open Philanthropy, Holden Karnofsky, on the OpenAI board in 2021 after he stepped down. His wife co-founded OpenAI rival Anthropic."
I can almost promise you there is a lawyer telling her not to talk about the firing.
Some people just fail upwards. Mind boggling.
Was he fired or he stepped down to be the CEO of Open AI?
> Some people just fail upwards. Mind boggling.
Didnt he hire the initial team and get investor money, that is a pretty good skill.
Brockman hired the top researchers with help from Yoshua Bengio.
Elon Musk was the biggest initial donor, and he is on record saying that Ilya was the key hire.
https://en.wikipedia.org/wiki/OpenAI#History
Nobody involved is being transparent about what went down here. Not sure we'll find out the full story here...
I think this Reddit comment captures the most plausible story at this point: https://www.reddit.com/r/MachineLearning/comments/1812w04/co...
That's not a good look on Altman at all, it looks like Altman was using divide-and-conquer in a skeezy way, spreading around negative impression, even rumors, about his bosses, as a means dispose of a problem.
"Altman approached other board members, trying to convince each to fire Toner. Later, some board members swapped notes on their individual discussions with Altman. The group concluded that in one discussion with a board member, Altman left a misleading perception that another member thought Toner should leave, the people said. By this point, several of OpenAI’s then-directors already had concerns about Altman’s honesty, people familiar with their thinking said."
Leadership is fundamentally about manipulating people. If the people already agreed, they wouldn't need leadership. Your job as leader is to get them all working in more or less the same direction, even though they don't really want to.
Honesty is a great way to do that, when it works. Honesty has the virtue of not having to keep track of your lies, and you don't get caught in an inconsistency.
But it doesn't always work, and then you have to consider alternatives. Sometimes people are dumb. Sometimes people are wrong. Sometimes people are hostile. Sometimes people are right, but in a way that can't actually be implemented, and you need to go with a sub-optimal path despite that.
At core, you can't lead if you don't believe that you've listened to all of the concerns and reached the solution that is best for the group as a whole, according to whatever definition of "best" and "whole" you were hired for. Sometimes that's going to force you to use other means that just telling people the truth and hoping they reach the same conclusion.
Leadership is fundamentally a people problem, not a technical one. And sometimes that's going to mean being dishonest. A good leader does it as little as possible, but simply dismissing it as a tactic is a recipe for failure because it hides a fundamental truth about why and how people behave in groups.
Which successful CEO takes direction from a board that meets a few times a year?
"Managing the board" is one of the major jobs of the CEO, most of that work done outside those key moments when the board meets to formally vote on what they usually already decided based on earlier discussions between themselves, CEO, shareholders and others.
Honestly? Nearly all of them. It's how the whole thing is supposed to work, and for the most part it works ok. There are exceptions.
Hard disagree. A leader cultivates agreement and cooperation through clear articulation of a vision and a path to get there.
lying to board is dick move, manoeuvring to get someone fired is a dick move. If what we read is mostly true, then I can see why the board fired him, after all, how can they trust him?
Not that that's actually ever going to be the case at OpenAI again, but the parent has the chain of command here totally bass-ackwards.
In theory, yes. In practice: Not always.
In truth it's both. That said, leadership doesn't mean lying. Particularly not about what another person has said. Very particularly not when it's trivial for them to compare notes.
Your line of reasoning fulfills the stereotype that tech people are sorely lacking in a well-rounded education.
This is wrong. Manipulating people is not leadership. It might be management in some circumstances, but management is not leadership.
> If the people already agreed, they wouldn't need leadership.
This is wrong too. People can agree on a goal without seeing how to get there. Leadership is not just about seeing the goal. It's about leading people there. But if you have to manipulate the people, you're not leading them. You're just using them for your own ends.
This is true...if you are the leader of a cult, which exist to deify you more than benefit the group.
> If the people already agreed, they wouldn't need leadership. Your job as leader is to get them all working in more or less the same direction, even though they don't really want to.
Yes, but if you're having to puppet people to do it, it's not leadership. It's just manipulation. You should be able to do this through inspiration, and transparently in group settings, not by compartmentalizing lies and half-truths privately. That's just fucking embarrassing. Altman's leadership skills are on par with a 14-year old influencer.
I feel like you're taking the argument for why "we let Pearl Harbor happen" and using it to excuse the whims of a psychopath.
The dude essentially threatened to destroy the company if he didn't get his way. Might does not make right. Sam turned out to have more leverage here; that doesn't make him a good person.
>You don’t fire someone so important for that and extrapolate it into equating human extinction.
On the other hand, would you really want a company developing existentially dangerous technology to be lead by someone who's known as a slippery operator?
My hunch is that part of the board's motivation was a feeling that they should have unusually high standards for the CEO of OpenAI.
>...
>Pretty much everyone who works on AI thinks that they're 'one of the good people'. Statistically, this is unlikely to be the case.
>...
>Proactively giving up power is one of the hardest things for people to do. Giving up power and money is even harder. AI orgs are rapidly gathering power and money and it's not clear they have right incentives to willfully shed their own power. This sets us up for racing.
https://threadreaderapp.com/thread/1555980412333133824
>Also who is to say deceleration in the age of AI isn’t worse than acceleration? Every govt entity is developing it, and it could be a national sec issue to speed it up faster than an adversary
See e.g. https://pauseai.info/faq#if-we-pause-how-about-china
Silicon Valley seems to have a fairly strong monopoly on AI talent.
This and the FTX saga just shows EA as nerdy movement in arrested development.
This is the biggest news of it all So all the Q* thing and Ilya's safety position and other theories were incorrect?
It's more of a smear. If you're the one arguing from an AI safety standpoint, it means your opponent is not being safe and is dangerous.
You should be expected to articulate why your position is "safer" and how "safe" is defined.
In my opinion most AI safety arguments are about vaguely defined variables like speed: "it's going too fast" - to who? who defines what fast is? do you slow down 2x or 10x? do competitors and/or rival nations less concerned zip by and the end result is still the same?
There is some validity to the discussion if AI is being "racist" or "biased" (again however you define these) but again competitors and/or rival nations may be less concerned and the end result may still be the same.
If anything the "safest" option is to define what your concerns are, then race ahead to try to be the canonical solution or defined standard in order to set standards for the rest to follow or be bound by.
Otherwise I don't see what this means now; that the LLMs e.g. dont use racist terms? ok, great, nice, but how is that anything more than what you need to do on the web now anyway? how's that related to "AI" at all?
What I'd love to see is that this gambit backfires and instead we start talking about "tech safety" and that creates actual regulations with teeth that cut down the techzillas a bit (or a lot).
The embarassing part is that the board decided to fire the CEO, announced their decision, refused to say why, attempted to put in place a new CEO but had to immediately demote the new CEO (Mira) after she rejected their plan, upset and alienated their core partners, along with almost all of their employees, and then publicly backtracked to undo the firing that led to this all happening.
Once you screw this up to such an incredible degree, how can anyone really trust that you were doing the rest of your job well?
Perhaps there's some level of overconfidence at play from systems thinkers who overintellectualize their ability to conceptualize and extrapolate forward an impossibly complex system.
You may well be on to something. I'd trust a cabal of science fiction writers more than that I would trust these self appointed governors of our collective future. They lack imagination, for starters.
- Heinlein
- Asimov - Clarke - Orson Scott Card Sorry if that toppled any of your heroes.I have book by all of them on my shelves but they're not necessarily saints.
More importantly, if you don't like somebody, don't just hint at it- give us a link.
He was completely into turning into a woman even early on much less the book where he had his brain put into a secretary who died in an accident and went into the ramifications. I say he as Lazarus Long and the rest were him.
archive.xxx is prblematic at best if you use Cloudflare 1.1.1.1 DNS, fine with other providers, Quad9, Google, etc.
What authors are approved for reading?
I want to make sure I fit into the perfect cookie cutter world views.
Thank you for that, it had completely passed me by.
Oh and the fosterites were poking fun at Scientology, Mormons, and proto megachurch People.
And his intent wasn't to rip off or lie to anyone
Actually this is more or less the point that Eliezer Yudkowsky makes in this essay about the need for caution in AI development:
https://www.econlib.org/archives/2016/03/so_far_unfriend.htm... (see especially points A/B/C/D)
I doubt overconfidence is a problem specific to effective altruism. In any case, any good machine learning engineer knows that a dataset with only a single data point is essentially worthless -- even if we grant the premise that the board took the wrong action given the information they had available to them at the time.
Of course the recommendations are not that novel; that's CEO succession planning 101. But I guess none of those four have done any large succession planning, and were clearly out of their depth.
[1] https://chat.openai.com/share/e77bc868-fe27-4346-9c13-0908e8...
[2] https://g.co/bard/share/549edfddc624
> how one decision would pan out
I'm not sure what point you are making here. Are you trying to say "see, the AI not-kill-everyone-ists couldn't predict the future even in the short term, therefore we shouldn't put much credence into the the idea that the specific examples of AI doom they have given will happen"?
Or are you trying to imply that the idea of AI doom as a whole is bunk, because we can't predict the future... therefore everything will be fine...?
According to reports, Mira was informed of the decision to fire Sam the night before Sam was fired. There's no indication she was against it and she also didn't inform Sam. So, I don't get this "she rejected their plan".
If the board informed her they were going to fire Sam, it seems logical that they also informed her they were going to give her the interim CEO position. I have no knowledge of this (it just kind of makes sense to me).
Are you saying they told Mira they were going to fire Sam and they didn't give her a reason but she was still okay with it?
Mira immediately started trying Sam back. [1]
If that's not rejecting the board's plans, I don't know what is.
[1] https://economictimes.indiatimes.com/tech/technology/openais...
As someone who pretty harshly described the board's actions, I agree with the first part of what you wrote, but I don't agree with this.
People can be good at critical aspects of their job while lacking the maturity to understand large group dynamics. I thought this quote from the article said a lot:
> The board members weren’t prepared for the fallout from their decision. The members, including Toner, were taken aback by staffers’ apparent willingness to abandon the company without Altman at the helm and the extent to which the management team sided with the ousted CEO, according to people familiar with the matter.
It's pretty hard to me to think, based on the board's initial press release, that there wouldn't be a huge fallout from this decision. The release also made me think that Altman had done something truly egregious/malevolent, and that wasn't the case.
What I saw this as was a group of people who had real concerns about Altman's leadership, and some of those concerns ended up sounding pretty valid, but who didn't have the experience/maturity to understand how to go about the next step. OpenAI's "frankenstein" org structure, with a board beholden to the nonprofit's charter but also responsible for billions in investor capital, also helped contribute to this cluster-f.
What did you hear that sounded valid? Toner had her chance to make her case here, and it was simply that Altman tried to get her fired, so she retaliated. It wasn't about safety or leadership. It was petty politics over board control.
No, but lying to Board members about what other Board members have said is. The employee equivalent would be lying to one member of the C-suite about what another had said.
Yes. He's not a teenage girl, he's the CEO of a nonprofit. At his level this is fraudulent misrepresentation.
The rack up debt and fail on mediocre idea and then sell your only valuable resource, your human employees Plan gave him a lot of money to hire a lot of law school grads who have since made partner.
I don't think it's that simple.
First, after confronting Toner directly and getting a response he didn't like, Altman went behind her back to the other board members trying to get her fired. That's not what a responsible CEO does. A responsible CEO, if he does not agree with the position taken by one Board member, brings that concern openly to the rest of the Board in a formal setting. He doesn't go behind the one Board member's back. If Altman really believed Toner's actions had harmed the company, his responsibility as a CEO was to bring that concern openly to the Board and let them make the call openly. And that's especially true for a company that, according to Altman's own public claims, has stewardship of a technology that could be an existential threat to humans. That makes any behind-the-scenes maneuvering on his part even more of a problem.
Second, according to the article, other Board members had concerns about Altman's honesty, and not just due to whatever transpired during Altman's behind-the-back attempts to get Toner fired. So it's not just that particular incident that's relevant here.
That's not to say that the Board didn't make mistakes. I agree with other posters that they should have seen the consequences coming for firing Altman. It also appears, from the fact that the lawyer's advice about fiduciary duty came after they fired Altman, that they didn't seek legal advice before doing that, which seems like an obvious mistake. (It's possible, though, that they did ask the lawyer beforehand and he didn't raise the possibility of a backlash and consequent damage to the company, which would put at least some of the blame on him.) I just don't think that because the Board messed up, that makes Altman the good guy. I don't think any of these people come out of this looking good. Certainly I don't have confidence in any of them as stewards of AI if AI is really as big a deal as they think it is.
Yes and no.
Meetings are a terrible way to drive toward decisions but a great way to signal positions to others. Or "decision meetings" bad; "consensus meetings" good.
If you're driving one of those, you should know how everyone involved feels before they meet. You should lobby any of them beforehand if necessary. The idea is to try to make sure you have buy-in before everyone shows up - no drag-out arguments, no rabbit trails, just statements of their positions and a thumbs up.
And Altman seems to know that, given his behavior.
The decision Altman was driving for was to fire a Board member he disagreed with. That's fundamentally different from an ordinary Board decision, and should not be lobbied for in advance the way you would organize support in advance for a normal Board decision. When the CEO of the company has a fundamental disagreement with a Board member, and wants to get that Board member fired, lobbying beforehand behind the one Board member's back is fundamentally dishonest. He should put the disagreement before the entire Board openly.
Btw, if I were Toner, as soon as I knew I had a fundamental disagreement with the CEO, I would have told the other Board members and requested a Board meeting to discuss the matter (and told the CEO I was doing those things). The meeting would not have to be an actual decision meeting with the Board formally in session. But I would want any discussion to take place with everyone present and with everything openly out on the table, not in behind-the-back individual sessions. I'm somewhat surprised that Toner didn't do that.
I agree with this as a general principle, but in this specific case, the 'understanding group dynamics' part is the 'critical aspect of the job' part. By the same token, you don't just promote the best engineer to CEO if he's bad at managing, because the managing is the critical part of the job.
That and that alone is enough to question their competence, though. The impression I get is that Toner has been in the academic/nonprofit world for so long that she doesn't understand how the real world works. In those places, the sad truth is nothing you really do is of much consequence to anybody in the grand scheme of things, other than you collecting prestige and a paycheck.
Then she tried making a consequential decision in an organization valued at around $90 billion, and lo and behold, people started caring about her actions in a way she has never experienced before.
How can anyone take any "AI" company seriously other than as a get rich quick scheme. Doubtful a successful investor focused on management would bother with a company like OpenAI.
https://fortune.com/2023/05/06/ai-warren-buffett-charlie-mun...
The probability of OpenAI employing wackos, like the guy at Google who thought automplete was sentient, is very high.
Not one critic I've seen has stated what the best legal structure for OpenAI should actually have been, in a world where OpenAI really is stewarding a technology that could be an existential threat to humans. I really wish they would do so, because then we'd have a shot at improving OpenAI's governance. But so far it's been crickets.
BTW, I think one of the big lessons of this episode is that legal structure matters much less than you'd think. On paper, OpenAI is supposed to follow its charter https://openai.com/charter and the board had the power to fire Sam. In practice, Sam was kinda sorta violating the charter, but he's a charismatic leader who was about to bring in a big payday for the employees (https://www.cnbc.com/2023/11/30/openai-tender-offer-on-track...), and that turned out to matter more than the org's legal structure.
Here's a group pushing for regulation and looking for volunteers: https://pauseai.info/
I would hesitate to physically break stuff -- sounds potentially unethical and could harm momentum for regulation
However I make an exception for existentially dangerous technology
Instead of a structure where the board members get to choose other board members, I would suggest a structure where different outside groups get fixed numbers of board members. So, for example, it could have been (remembering that the initial board was just Sam Altman and Elon Musk):
* One board member chosen by Sam Altman
* One board member chosen by Elon Musk
* One board member chosen by OpenPhil
* One board member chosen by Microsoft
* One board member chosen by some VC investor
The precise composition is something they would argue about during funding, maybe some of these groups would get more or less seats.
The root "governance" problem is that the board churned a lot because so many people had conflicts, or other reasons that I don't know as an outsider. The remaining board was unrepresentative of stakeholders (ie, no Elon Musk, nobody independently trusted by Microsoft) and probably not as experienced as you'd like. The problem with this sort of structure is that it gives everyone a lot of incentives to play politics with the composition of the board itself. Which happened a lot here - Reid Hoffman exiting and not really getting replaced by a similar person, the debate between Sam and Helen about her presence on the board, probably other cases.
I bet they end up with something similar, with different stakeholders represented. Give Bret Taylor some time to cook.
I'm reminded of these great PG tweets:
>When people criticize an action on the grounds of the "optics," they're almost always full of shit. All they're really saying is "What you did looks bad." But if they phrased it that way, they'd have to answer the question "Was it actually bad, or not?"
>If someone did something bad, you don't need to talk about "optics." And if they did something that seems bad but that you know isn't, why are you criticizing it at all? You should instead be explaining why it's not as bad as it seems.
https://nitter.net/paulg/status/1728015636771504300#m
The situation is similar if you criticize an action on the grounds that "it upset people". If the action was actually bad, you should be able to explain why directly.
Here are some quotes from the OpenAI charter:
>We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome.
>...
>Our primary fiduciary duty is to humanity.
>...
>We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project.
https://openai.com/charter
And here's a quote from the OP:
>In October, Toner, who is director of strategy at a think tank in Washington, D.C., co-wrote a paper on AI safety. The paper said OpenAI’s launch of ChatGPT sparked a “sense of urgency inside major tech companies” that led them to fast-track AI products to keep up. It also said Anthropic, an OpenAI competitor, avoided “stoking the flames of AI hype” by waiting to release its chatbot.
>After publication, Altman confronted Toner, saying she had harmed OpenAI by criticizing the company so publicly. Then he went behind her back, people familiar with the situation said.
Who is doing a better job of upholding the charter in this situation: Altman, or Toner?
The "people are upset" criticism essentially states that people became upset when Toner attempted to actually uphold the charter. To me, that says more about those people than it says about Toner.
For this to work you need all layers of society to be represented and since this is intended to be global you'd be looking at the UN. Of course that's 100% anathema to anybody involved with OpenAI or even any cross section of HN because 'bureaucrats don't get technology and the UN can't agree on anything'. But if it really is what it seems to be that to me seems to be the only venue where this can be addressed.
Of course the race is on and the first party to cross the threshold will get the gold and never mind the consequences so I expect any attempt at oversight to be murdered the second they have it working.
Somewhere I got the notion that the UN is generally considered to be dysfunctional these days, and that's part of the reason there's been a shift in diplomacy to other venues like the G20. Representativeness is important, but it's just one of several important desiderata in AI governance.
It seems to me that the current important decisions facing OpenAI are for the most part fairly technical, so it's good for their oversight to be technical. Representativeness becomes more important later. I'd say if OpenAI actually does succeed in developing AGI, that's the point at which the board should be replacing itself with some other, more representative institution.
>Of course the race is on and the first party to cross the threshold will get the gold and never mind the consequences so I expect any attempt at oversight to be murdered the second they have it working.
Maybe a good question from a governance perspective would be, how can we configure the institutional incentives to discourage this, or at least select for leaders with strong character and ethics who won't ruthlessly seek power at all costs?
I've been criticizing Sam Altman a fair amount lately, but it sounds like you take it much farther in essentially viewing him as some sort of crime boss, or else you're assuming that anyone with significant power will automatically adopt crime boss ethics as a result. I doubt the latter statement is true -- for example, I don't believe that Obama adopted crime boss ethics when he became POTUS.
> It seems to me that the current important decisions facing OpenAI are for the most part fairly technical
For OpenAI, yes. But for society the stakes are much higher and non-technical in nature. For instance: mass disruption of the labour market. Note that when we had the industrial revolution the world population was a fraction of what it is today.
> so it's good for their oversight to be technical
But that could change in a heartbeat and by then it is too late. Winner takes all.
> Representativeness becomes more important later.
There won't be a 'later', whoever gets this first will instantly weaponize it and seize power, though you won't know about it until you realize that it's a done deal. Stable dictatorships are one of the more likely outcomes of AI.
> I'd say if OpenAI actually does succeed in developing AGI, that's the point at which the board should be replacing itself with some other, more representative institution.
Right now our best hope is that they can't. If not, all bets are off and you may want to brush up on your Charles Stross.
> Maybe a good question from a governance perspective would be, how can we configure the institutional incentives to discourage this, or at least select for leaders with strong character and ethics who won't ruthlessly seek power at all costs?
Leaders with strong power and those who ruthlessly seek power at all costs have strong overlap, assuming they are not congruent.
> I've been criticizing Sam Altman a fair amount lately, but it sounds like you take it much farther in essentially viewing him as some sort of crime boss, or else you're assuming that anyone with significant power will automatically adopt crime boss ethics as a result.
I see him just like I see Elon Musk: brilliant but dangerous, not necessarily a crime boss, more like someone who doesn't realize that they don't have the right to make the decisions they make and too little in terms of reflective capabilities and humility.
> I doubt the latter statement is true -- for example, I don't believe that Obama adopted crime boss ethics when he became POTUS.
Obama was a huge disappointment, so no argument there, but the flipside is that Trump was exactly what he promised to be and more and the world came out for the worse anyway. It makes you wonder how much worse that can get. Probably a lot worse. I probably don't want to know.
You're characterizing this like it was just a bunch of Twitter armchair quarterbacks.
Which is was.... Plus 90%+ of the boots on the ground who voted no confidence in their leaders.
Only the most obstenate can cling to the belief they are a good leader/communicator with that statistic.
EDIT: There is a feeling that companies are full of unremarkable, out-of-touch, over-compensated leaders who ignore the opinions of their subordinates.
And yet when that situation actually materializes, or mostly materializes, you read opinions like this.
I'm concerned that boots on the ground may be susceptible to motivated cognition due to their financial interest. I suspect that their finances come before upholding the charter for many of them. Why else did they get so upset when Helen said that shutting down OpenAI could be the right thing to do? That's pretty much implied by the charter.
Disqualifying their opinion on OpenAPI leadership seems unjust.
https://www.ribbonfarm.com/2009/10/07/the-gervais-principle-...
I have worked for many CEOs, and (surprise!) fewer than 100% of them are sociopaths. Good grief.
I am not saying this necessarily as a criticism, but self-delusion appears to be an important part of many founder and executive stories.
Insightful.
- As PG pointed out Sam would find his way to the top of an island of cannibals. If you're generous to Toner's side in this article she likely _was_ alarmed by how Sam tried to get other board members to turn against her. When she walked into the arena with Altman to try and remove him however he had much more experience and orchestrated an incredible counter-coup.
- There must be an army of attorneys representing the board members, and investors because there are huge potential legal pitfalls to destroying value, or failing fiduciary duties.
- We don't know a lot because anything put out in public has to be spotless legally. If you're a board member and you know the truth, the only upside to leaking more info is clout, but the downsides are being sued and tied up in court losing a lot of time and money.
- People seem dissatisfied with what was revealed in this article but Toner has likely been advised by lots of counsel on exactly what can be said, and the list of those things is probably miniscule. I'd say she's doing the best she can to give the public more info and I'm happy she's taken the effort to do so.
Nit: Altman may have orchestrated a coup, but the board certainly did not. The board is the legitimate authority, and in no way is a board firing its subordinate a "coup." Rather, it's executing its legitimate power.
Coups triggered when subordinate feels threatened is a pretty common pattern. IIRC, the recent coup in Niger was triggered when the leader of the presidential guard thought the president was about to fire him, too power in a coup rather than lose his job.
The "fat lady hasn't sung" point works both ways -- for all we know, the new board's investigation into this incident will be incredibly damning for Sam.
Sam was on the board. They didn't fire a subordinate they kicked a peer out of the group.
Why did they take out the chairman of the board.
> advised by lots of counsel on exactly what can be said, and the list of those things is probably miniscule.
Hmmm, in the US yes and no. First, US lawyers point out risk in every direction. Because nothing is risk-less and the law is unclear or unsettled (and anyone can sue for just about anything). Once they have duly warned to their satisfaction however, their job becomes to come up with a theory on how to do exactly what it is you want to do. The two are different tasks, and are not incompatible.
That's basically the definition of a good business lawyer.
Companies implode all the time. Sure you might get a lawsuit but the company would pay it or the directors insurance would.
HN is always scared about lawsuits. If Trump taught us anything is that there’s a lot of leeway in that area.
I respect that she and other board members didn’t like that Sam was trying to manipulate them into ousting her but their counterattack was poorly thought out.
Somebody put her there, told her "you know, say it's for humanity or something" and she actually believed it.
No, it's for people to get rich.
But you seem to have something in mind. Tell me.
So there are many particulars that mean pattern matching to a standard board dispute will lose something. I think it's likely many of the primary actors have, at various times, had non-strictly-pecuniary motives. That one side won doesn't mean that the other side was always a farce.
The folks who argue that it is a CEO's duty to serve the board do not understand governance or power.
Open AI would have wallowed in irrelevance had Altman not raised the billions from Microsoft to fund its research. Because he was able to do that, and built the relationship with Nadella, he had and has power. Toner's behavior seems naive in that context. But naiveté among academics does not surprise me.
You can be fairly certain that the board will be stocked with 'yes-men' (and women) beholden to Altman. He will not make that same mistake twice.
"Pournelle's Iron Law of Bureaucracy states that in any bureaucratic organization there will be two kinds of people: those who work to further the actual goals of the organization, and those who work for the organization itself. Examples in education would be teachers who work and sacrifice to teach children, vs. union representative who work to protect any teacher including the most incompetent. The Iron Law states that in all cases, the second type of person will always gain control of the organization, and will always write the rules under which the organization functions."
> organization she pledged to serve
First there is no pledge, second I think she served the Open AI Charter.
> Toner graduated from the University of Melbourne, Australia, in 2014 with a degree in chemical engineering [...] In 2019, she spent nine months in Beijing studying its AI ecosystem. When she returned, Toner helped establish a research organization at Georgetown University, called the Center for Security and Emerging Technology, [...] She succeeded her former manager from Open Philanthropy, Holden Karnofsky, on the OpenAI board in 2021 after he stepped down.
Honest question, how do people find themselves in the board of one of the hottest startups at the tender age of 31? Are they geniuses or is it all about connections?
Her brothers: https://www.vesparum.com/origins
So the short answer is wealth and connections, or daddy.
Well, if that was the case it actually was fine but there is no hard proof that it was like that and if it was they messed it up. So failure on all counts.
One point here is that there's no licensing requirement for AI engineers like you see in some other fields of engineering. As someone recently quipped on Twitter:
> did you know that there are currently no licensing requirements in the US for training potentially world-threatening AI models, but there are in the state of Louisiana for selling flower arrangements
https://nitter.net/littIeramblings/status/173092523732656542...
In any case, what's been happening in AI engineering is significantly different than in other fields of engineering.
There is no consensus as to the magnitude of AI risk, although extinction probability estimates are generally high: https://nitter.net/AISafetyMemes/status/1729892336782524676#...
In practice, AI engineers self-select into different places of employment depending on how worried they are. If you're not very worried, you work at Meta. If you're significantly worried, you work at Anthropic or even someplace like MIRI. Most AI labs have a safety culture that's somewhere in between those extremes. [Obviously, from the point of view of preventing extinction, this situation seems unsatisfactory.]
With regard to the point about the board -- their job was to supervise the CEO, and they say Sam was making that difficult. See https://nitter.net/hlntnr/status/1730034020140912901#m
I'd say so, and in fact it's older than you might think.
Here's a lit review from 2018 for example: https://arxiv.org/pdf/1805.01109.pdf
This post is more recent, but understanding it might require some context: https://www.alignmentforum.org/posts/zaaGsFBeDTpCsYHef/shall...
A group from Cambridge has been running an online course you can take if you're interested in getting up to speed: https://aisafetyfundamentals.com/ Application deadline is early February. I took the course a couple years ago; happy to answer any questions.
Just like “terrorism” and “drugs” and all the other vaguely menacing things that ended up being worth a pile of money to some incumbent interest, the real story will be boring: some crooks used a fancy-ass LLM to take a bank apart or something. The bank calls the cops, this happens about twice before the cops have a division for this sort of thing, and sooner or later someone is in a courtroom.
It would be nauseatingly boring to get into how many times this has happened.
Also predictably depressing, the real story, which is yet another brick in the wall of “we’re handing the reigns to people who make movie Zuck look flat fucking normal”, goes largely unreported.
We used to complain about dual-class share structures.
Sam just eviscerated a board of an ostensible charity with the full backing of MSFT and got himself installed as Ungovernable God Emperor of Arrakis for life. Still doing the eyeball scanner thing.
Still has never shown the public he can handle numpy to the tune of MNIST.
This is all in broad daylight and the Gemini launch gets clobbered because the 4-series comparable one comes out in January.
Sam’s a smart guy, I’m quite sure he’d pay attention if the braintrust (this site) complained loudly.
But the message is: “more, not less”.
> “Fancy titles like ‘Director of Strategy at Georgetown’s Center for Security and Emerging Technology’ can lead to a false sense of understanding of the complex process of entrepreneurial innovation,” Khosla wrote in an essay in tech-news publication the Information, referring to Toner and her current position.
Strong words from the infamous beach villain and resident asshole, Vinod Khosla...
Every Generation Gets the Beach Villain It Deserves (NYT) https://archive.is/6mss9