I'm not sure how this is legal. My employer certainly could not clawback paid salary or bonuses if I violated a surprise NDA they sprung on me when leaving on good terms. Why can they clawback vested stock compensation?
That appears to be the case, although the wording of what they agree to up front is considerably more vague than the agreement they're reportedly presented to sign post-departure. Link to a thread from the author of the Vox article: https://x.com/KelseyTuoc/status/1791584341669396560
These aren't real stock, they are "profit participation units" or PPUs; in addition, the fact that there is a NDA and a NDA about the NDA, means no one can warn you before you sign your employment papers about the implications of 'PPUs' and the tender-offer restriction and the future NDA. So it's possible that there's some loophole or simple omission somewhere which enables this, which would never work for regular RSUs or stock options, which no one is allowed to warn you about on pain of their PPUs being clawed back, and which you find out about only when you leave (and who would want to leave a rocketship like OA?).
Submission title mentions NDA but the article also mentions a non disparagement agreement. "You can't give away our trade secrets" is one thing but it sounds like they're being told they can't say anything critical of the company at all.
Two people entering an agreement to not talk about something is fine. You and I should (and can, with very few restrictions) be able to agree that I'll do x, and you'll do y and we are going to keep the matter private. Anyone who wants to take away this ability for two people to do such a thing needs to take a long hard look at themselves, and maybe move to north korea.
There are things that are legal between parties of (presumed) equal footing, that aren't legal between employers and employees.
That's why you can pay $1 to buy a gadget made in some third world country, but you can't pay your employees less than say $8/hour due to minimum wage laws.
Being paid a whole lot of money to not talk about something isn't remotely similar to paying someone a few dollars an hour. It's not morally similar, it's not legally similar and it's not treated similarly by anyone who deals with these matters and has a clue what they are doing.
Not a lawyer but those contracts aren't legal. You need something called "consideration" ie something new of value to be legal. They can't just take away something of value that was already agreed upon.
However they could add this to new employee contracts.
I agree with Piper's point that these contracts aren't common in tech, but they're hardly unheard of. In 20 years of consulting work I've seen dozens of them. They're not uncommon. This doesn't look uniquely hostile or amoral for OpenAI, just garden-variety.
Well, an AI charity -- so founded on openness that they're called OpenAI -- took millions in donations, everyone's copyright data...only to become effectively for-profit, close down their AI, and inflict a lifetime gag on their employees. In that context, it feels rather amoral.
This to me is like the "don't be evil" thing. I didn't take it seriously to begin with, I don't think reasonable people should have taken it seriously, and so it's not persuasive or really all that interesting to argue about.
I think we do need to start taking such things seriously, and start holding companies accountable using all available venues (including legal, and legislative if the laws don't have enough leverage as it is) when they act contrary to their publicly stated commitments.
Therein lies the issue. The second you throw idealistic terms like “don’t be evil” and __OPEN__ ai around you should be expected to deliver.
But how is that even possible when corporations are typically run by ghouls who enjoy relativistic morals when it suits them. And are beholden to profits, not ethics.
Contracts like this seem extremely unusual as a condition for _retaining already vested equity (or equity-like instruments)_, rather than as a condition for receiving additional severance. And how common are non-disclosure clauses that cover the non-disparagement clauses?
In fact both of those seem quite bad, both by regular industry standards, and even moreso as applied to OpenAI's specific situation.
This sounds just like the non-compete issue that the FTC just invalidated. I can see if the current FTC leadership is allowed to continue working after 2025/01/20 that these things might be moved against as well. If new admin is brought in, they might all get reversed. Just something to consider going into your particular polling place
It doesn’t matter if they are not legal. Employees do not have resources to fight expensive legal battles and fear retaliation in other ways. Like not being able to find future jobs. And anyone with family plain won’t have the time.
You can't add a contingency to a payment retroactively. It sounds like these are exit agreements, not employment agreements.
If it was "we'll give you shares/cash if you don't say anything bad about us", that's normal, kind of standard fare for exit agreements, it's why severance packages exist.
But if it is "we'll take away the shares that you already earned as part of your regular employment compensation unless you agree to not say anything bad about us", that's extortion.
Is it criticism if a claim is true? There is so much legal jargon I'm willing to bet most people won't want the headache (and those that don't care about equity are likely already fairly wealthy)
Yes, if it isn't true it is libel or slander (sometimes depending on intent), not just criticism, and already not permissible without any contract covering it.
Why have other companies not done the same? This seems legally tenuous to only now be attempted. Will we see burger flippers prevented from discussing the rat infestation at their previous workplace?
(Don’t have X) - is there a timeline? Can I curse out the company on my deathbed, or would their lawyers have the legal right to try and clawback the equity from the estate?
i worked at McDonald's in the mid-late 00s, I'm pretty sure there was a clause about never saying anything negative about them. i think they were a great employer!
It doesn't have to be equity. If they wanted to, they could put in their employment contract "If you say anything bad about McDonalds, you owe us $1000." What is the ex-burger-flipper going to do? Fight them in court?
For the burger metaphor, you need to have leverage over the employee to make them not speak. No one at Burger King is getting severance when they are kicked out, let alone equity.
As for other companies that can pay: I can only assume that the cost to bribe skilled workers isn't worth the perceived risk and cost of lawsuits from the downfall (which they may or may not be able to settle). Generative AI is still very young and under a lot of scrutiny on all fronts, so the risk of a whistle blower at this stage may shape the entire future of the industry at large.
Other companies have done the same. I worked at a company that is 0% related to the tech industry. I was laid off/let go/dismissed/sacked where they offered me a "severance" on the condition I sign a release with a non-disparaging clause. I didn't give enough shits about the company to waste my time/energy commenting about them. It was just an entry on a resume where I happened to work with some really neat, talented, and cool/interesting coworkers. I had the luxury of nobody else giving a damn about how/why I left. I can only imagine these people getting hounded by Real Housewives level gossip/bullshit.
Your original comment uses the term "free speech," which in the context of the discussion of the legality of contract in the US, brings to mind the first amendment.
But first amendment basically only restricts the government's ability to suppress speech, not the ability of other parties (like OpenAI).
This restriction may be illegal, but not on first amendment ("free speech") grounds.
Anti frame is saying that free speech guarantee in Constitution only applies to the relationship between the government and the citizens, not between private entities.
In the US, the Constitution prevents the government from regulating your speech.
It does not prevent you from entering into contracts with other private entities, like your company, about what THEY allow you to say or not. In this case there might be other laws about whether a company can unilaterally force that on you after the fact, but that's not a free speech consideration, just a contract dispute.
I think we need to face the fact that these companies aren’t trustworthy in upholding their own stated morals. We need to consider whether streaming video from our phone to a complex AI system that can interpret everything it sees might have longer term privacy implications. When you think about it, a cloud AI system is an incredible surveillance machine. You want to talk to it about important questions in your life, and it would also be capable of dragnet surveillance based on complex concepts like “show me all the people organizing protests” etc.
Consider for example that when Amazon bought the Ring security camera system, it had a “god mode” that allowed executives and a team in Ukraine unlimited access to all camera data. It wasn’t just a consumer product for home users, it was a mass surveillance product for the business owners:
These big companies and their executives want power. Withholding huge financial gain from ex employees to maintain their silence is one way of retaining that power.
Free speech is a much more general notion than anything having to do with governments.
The first amendment is a US free speech protection, but it's not prototypical.
You can also find this in some other free speech protections, for example that in the UDHR
>Everyone has the right to freedom of opinion and expression; this right includes freedom to hold opinions without interference and to seek, receive and impart information and ideas through any media and regardless of frontiers.
Originally the comment to which that comment responded said something about free speech rather than anything about legality, and it was in that context which I responded, so the comment to which I responded must have also been written in that context.
The inalienable rights of mankind are not given to us or validated in a court of man's law. This is not new philosophy and comes from at least as far back as the Greeks.
Your quips will serve you well, I'm sure, in whatever microcosm you populate.
"We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness."
I mean to say there are certain rights we all have, simply for existing as humans. The right to breathe is a good example. No human, state, or otherwise has the moral high-ground to take these rights from us. They are not granted, or given, they are absolute and unequivocal.
It's not rhetoric, it's basic John Locke. Also your trust is an internal locus, and doesn't change the facts.
You do not seem to be embracing an open-minded discussion about the philosophy. Why beg the question? Are you admitting the State is the authority under which you are granted all privilege to live, love, and work? Who is to stop someone if you are being attacked, and your children are at risk? Do you wish you had a permit to allow you to breath?
"self-evident," means it requires no formal proof, as it is obvious to all with common sense and reason.
A lot of people forget that although 1A means the government can’t put you in prison for things, there are a lot of pretty unpleasant consequences from private entities. As far as I know, it wouldn’t be illegal for a dentist to deny care to someone who criticized them, for example.
Right, and that's why larger companies need regulation around those consequences. If a dentist doesn't want to treat you because you criticized them, that's fine, but if State Farm doesn't want to insure your dentistry because you criticized them, regulators shouldn't allow that.
OpenAI never acted with total disregard for safety. They never punished employees for raising legitimate concerns. They never reneged on public promises to devote resources to AI safety. They never made me sign any agreements restricting what I can say. One plus one is three.
Also, will Ilya likely have similar contractual bounds, despite the unique role he had at OpenAI? (Sorry for the self-reply. Felt more appropriate than an edit.)
I believe a better solution to this would be to spread the following sentiment: "Since it's already illegal to tell disparaging lies, the mere existence of such a clause implies some disparaging truths to which the party is aware." Always assuming the worst around hidden information provides a strong incentive to be transparent.
I was once fired, ghosted style, for merely being in the same meeting room as a racist corporate ass-clown muting the conference call to make Asian slights and monkey gesticulations. There was no lawsuit or payday because "how would I ever work again?" was the Hobson's choice between let it go and a moral crusade without a way to pay rent.
If instead I were upset that "not enough N are in tech," there isn't a specific incident or person to blame because it'd be a multifaceted situation.
This is an important mode of thinking in many adversarial or competitive contexts.
Cryptography is a prime example. Any time any company is the tiniest bit cagey or obfuscates any aspect, I default to assuming that they’re either selling snake oil or have installed NSA back doors. I’ll claim this openly, as a fact, until proven otherwise.
I hope forbidding telling the truth is about something banal like "fake it until you make it" in some of OpenAI demos. The technology looks like magic but plausible to implement in a few months/years.
Worse if it is related to training future super intelligence to kill people. Killer drones are possible even with today's technology without AGI.
Hush money payments and NDAs aren't illegal as Trump discovered, but perhaps lying about or concealing them in certain contexts is.
Also, when secrets or truthful disparaging information is leaked anonymously without a metadata trail, I'm thinking there's probably little or no recourse.
I am curious how long it will take for Sam to go from being perceived as a hero to a villain and then on to supervillain.
Even if they had a massive, successful and public safety team, and got alignment right (which I am highly doubtful about being possible) it is still going to happen as massive portions of white collar workers loose their jobs.
Mass protests are coming and he will be an obvious focus point for their ire.
> I am curious how long it will take for Sam to go from being perceived as a hero to a villain and then on to supervillain.
He's already perceived by some as a bit of a scoundrel, if not yet a villain, because of World Coin. I bet he'll hit supervillain status right around the time that ChatGPT BattleBots storm Europe.
> I am curious how long it will take for Sam to go from being perceived as a hero to a villain and then on to supervillain.
He probably already knows that, but doesn't care as long as OpenAI has captured the world's attention with ChatGPT generating them billions and their high interest in destroying Google.
> Mass protests are coming and he will be an obvious focus point for their ire.
This is going to age well.
Given that no-one knows the definition of AGI, then AGI can mean anything; even if it means 'steam-rolling' any startup, job, etc in OpenAI's path.
When he was fired there was a short window where the prevailing reaction here was "He must have done something /really/ bad." Then opinion changed to "Sam walks on water and the board are the bad guys". Maybe that line of thinking was a mistake.
So that explains the cult-like behaviour months ago when the company was under siege.
Diamond multi-million dollar hand-cuffs which OpenAI has bound lifetime secret service-level NDAs which are another unusual company setting after their so-called "non-profit" founding and their contradictory name.
Even an ex-employee saying 'ClosedAI' could see their PPUs evaporate in front of them to zero or they could never be allowed to sell them and have them taken away.
I don’t have any idea what goes on inside OAI. But I have this strange feeling that they were right to oust sama. They didn’t have the leverage to pull it off, though.
The only way I can see this being a valid contract is if the equity grant that they get to keep is a new grant offered the time of signing the exit contract. Any vested equity given as compensation for work could not then be offered again as consideration for signing a new agreement.
Maybe the agreement is "we will accelerate vesting of your unvested equity if you sign this new agreement"? If that's the case then it doesn't sound nearly so coercive to me.
It's not. The earlier tweets explain: the initial agreement says the employee must sign a "general release" or forfeit the equity, and then the general release they are asked to sign includes a lifetime no-criticism clause.
But a general release is not a non-criticism clause.
They're not required to sign anything other than a general release of liability when they leave to preserve their rights. They don't have to sign a non-disparagement clause.
But they'd need a very good lawyer to be confident at that time.
Duress can take other forms, unless we are really trying to differentiate general 'coercion' here.
Perhaps as an example of the blurred line; Pre-nup agreements sprung the day of the wedding, will not hold up in a US court with a competent lawyer challenging them.
You can try to call it 'economic' duress but any non-sociopath sees there are other factors at play.
That’s a really good point. Was this a prenuptial agreement? If it wasn’t May take is section 174 would apply and we would be talking about physical compulsion — and not “it’s a preferable economic situation to sign.”
In general, an agreement to agree is not an agreement. A requirement for a "general release" to be signed at some time in the future is iffy.
And that's before labor law issues.
Someone with a copy of that contract should run it through OpenAI's contract analyzer.
You haven't worked with many contracts, have you? Unenforceable clauses are the norm, most people are willing to follow them rather than risk having to fight them in court.
Have you read the actual document or contracts? Opining on stuff you haven't actually read seems premature. Read the contract, then tell us which clause violates which statute, that's useful.
Typically there is no penalty - and contracts explicitly declare that all clauses are severable so that the rest of the contract remains valid even if one of the scare-clauses is found to be invalid. IANAL
somebody explained to me early on that you cannot have a contract to have a contract. either initial agreement must state this condition clearly or they are signing another contract at employment termination which is bringing these new terms. IDK why would anyone sign that at termination unless they dangle additional equity. I dont think this BS they are trying to pull would be enforceable at least in California. though IANAL obviously.
all this said, in bigger picture I can understand not divulging trade secrets but not being allowed to discuss company culture towards AI safety essentially tells me that all the Sama talk about the 'for the good of humanity' is total BS. at the end of day its about market share and bottom line.
My initial reaction was "Hold up - your RSUs vest, you sell the shares and pocket the cash, you quit OpenAI, a few years later you disparage them, and then when? They somehow try and claw back the equity? How? At what value? There's no way this can work." Then I remembered that OpenAI "equity" doesn't take the form of an RSU or option or anything else that can be converted into an actual share ever. What they call "equity" is a "Profit Participation Unit (PPU)" that once vested entitles you to a share of their profits. They don't share the equivalent of a Cap Table with employees, so there's no way to tell what sort of ownership interest a PPU represents. And of course, it's unlikely OpenAI will ever turn a profit (which if they did would be capped anyway). So this is all just play money anyway.
Wow. Smart for them. Former employees are behooved to the company for an actual perpetuity. Sounds like a raw deal but when the potential gains are that big, I guess you'll agree to pretty much anything.
This is wrong on multiple levels. (to be clear I don't work at OAI)
> They don't share the equivalent of a Cap Table with employees, so there's no way to tell what sort of ownership interest a PPU represents
It is known - it represents 0 ownership share. They do not want to sell any ownership because their deal with MS gives MS 49% ownership and they don't want MS to be able to buy up additional stake and control the company.
> And of course, it's unlikely OpenAI will ever turn a profit (which if they did would be capped anyway). So this is all just play money anyway.
Putting aside your unreasonable confidence that OAI will never be profitable, the PPUs are tender offered so they can be sold to institutional investors up to a very high limit, OAIs current tender offer round values them at ~$80b iirc
> Note at offer time candidates do not know how many PPUs they will be receiving or how many exist in total. This is important because it’s not clear to candidates if they are receiving 1% or 0.001% of profits for instance. Even when giving options, some startups are often unclear or simply do not share the total number of outstanding shares. That said, this is generally considered bad practice and unfavorable for employees. Additionally, tender offers are not guaranteed to happen and the cadence may also not be known.
> PPUs also are restricted by a 2-year lock, meaning that if there’s a liquidation event, a new hire can’t sell their units within their first 2 years. Another key difference is that the growth is currently capped at 10x. Similar to their overall company structure, the PPUs are capped at a growth of 10 times the original value. So in the offer example above, the candidate received $2M worth of PPUs, which means that their capped amount they could sell them for would be $20M
> The most recent liquidation event we’re aware of happened during a tender offer earlier this year. It was during this event that some early employees were able to sell their profit participation units. It’s difficult to know how often these events happen and who is allowed to sell, though, as it’s on company discretion.
This NDA wrinkle is another negative. Honestly I think the entire OpenAI compensation model is smoke and mirrors which is normal for startups and obviously inferior to RSUs.
> Additionally, tender offers are not guaranteed to happen and the cadence may also not be known.
> PPUs also are restricted by a 2-year lock, meaning that if there’s a liquidation event, a new hire can’t sell their units within their first 2 years.
i know for a fact that these bits are inaccurate, but i don't want to go into the details.
the profit share is not known but you are told what the PPUs were valued at the most recent tender offer
You're not saying anything that in any way contradicts my original post. Here, I'll simplify it - OpenAI's PPUs are not in any sense of the word "equity" in OpenAI, they are simply a subordinated claim to an unknown % of a hypothetical future profit.
> there's no way to tell what sort of ownership interest a PPU represents
Wrong. We know - it is 0, this directly contradicts your claim.
> this is all just play money anyway.
Again, wrong - because it is sellable so employees can take home millions. Play money in the startup world means illiquid options that can't be tender offered.
You're making it sound like this is a terrible deal for employees but I personally know people who are able to sell $1m+ in OAI PPUs to institutional investors as part of the tender offer.
I would strongly encourage anyone faced with this ask by OpenAI to file a complaint with the NLRB as well as speak with an employment attorney familiar with California statute.
I’d be more afraid of their less-than-above-board power than their litigation power. People with $10-100 billion dollars who are highly connected to every major tech company and many shadowy companies we’ve never heard of can figure out a lot of my secrets and make life miserable enough for me that I don’t have the ability/energy to follow through with legal proceedings, even if I don’t attribute the walls collapsing around me to my legal opponent.
I think never is inaccurate here. First, there are a lot of simply bad lawyers who will give you bad advice. Secondly, a lot of lawyers who either don't actually specialize in the legal field your case demands, or who have never actually tried any cases and have no idea how something might go down in a court with a jury. Third (the most predatory), a lot of lawyers actually see the client (not the opposing party) as the money fountain. Charging huge fees for their "consultation," "legal research," "team of experts," etc, and now the client is quickly tens-of-thousands in the hole without even an actual case being filed.
Talking to good, honest lawyers is a good idea. Unfortunately most people don't have access to good honest lawyers, or don't know how to distinguish them from crooks with law degrees.
> Over the last 10 years or so, I have filed a number of high-profile unfair labor practice charges against coercive statements, with many of those statements being made on Twitter. I file those charges even though I am merely a bystander, not an employee or an aggrieved party.
> Every time I do this, some individuals ask how I am able to file charges when I don’t have “standing” because I am not the one who is being injured by the coercive statements.
> The short answer is that the National Labor Relations Act (NLRA) has no standing requirement.
> Employees reasonably fear retaliation from their boss if they file charges. So we want to make it possible for people who cannot be retaliated against to do it instead. [1]
I believe the Vox piece shared in this thread [2] is enough for anyone to hit submit on an NLRB web form and get the ball rolling. Snapshot in the Wayback Machine (all the in scope tweets archived in archive.today|is|ph), just in case.
I imagine the kinds of engineers under such gag orders do in fact either have "big money" or aren't worried about making big money in thr future.And this isn't their fight, it'll be the government. At the worst you may stand as a witness some months/years later to testify.
I'd only be worried about reporting if you fear for your life for refusing, a sadly poignant consideration given Boeing as of late.
I don't see why this comment needed a flag or so many uncharitable replies (though you could have expressed yourself more charitably too).
I understand your sentiment, but I think a lot of idealistic people will disagree - it's nice to think that a person should stand up for justice, no matter what.
In reality, I wonder how many people attempt to do this and end up regretting, because of what you mentioned.
Non-disparagement clauses seem so petty and pathetic. Really? Your corporation is so fragile and thin-skinned that it can't even withstand someone saying mean words? What's next? Forbidding ex-employees from sticking their tongue at you and saying "nyaa nyaa nyaa?"
Legally yes. Those mean words can cost them millions in lawsuits and billions if the judge rulings restrict how they can implement and monetize AI. Why do you think Boieing's "coincidental" deaths of whistle blowers has happened more than once these past few months?
The company is literally a house of cards at this point. There is probably so much vulture capitalist and angel investor money tied up in this company that even a disparaging rant could bring the whole company crashing down.
It’s yet another sign that the AI bubble will soon burst. The laughable release of “GPT-4o” was just a small red flag.
Got to keep the soldiers in check while the bean counters prep the books for an IPO and eventual early investor exit.
Almost smells like a SoftBank-esque failure in the near future.
Honestly I don't know if whistleblower protections are really worth a damn -- I could be wrong.
But would they not only protect the individual formally blowing the whistle (meeting the standard in the relevant law)?
These non-disparagement clauses would have the effect of laying the groundwork for a whistleblowing effort to fall flat, because nobody else will want to corroborate, when the role of journalism in whistleblowing cases is absolutely crucial.
No sensible mature company needs a lifetime non-disparagement clause -- especially not one that claims to have an ethical focus. It's clearly Omerta.
Whoever downvoted this: seriously. I really don't care but you need to explain to people why lifetime non-disparagement clauses are not about maintaining silence. What's the ethical application for them?
I think superalignment is absurd, and model "safety" is the modern AI company's "think of the children" pearl clutching pretext to justify digging moats. All this after sucking up everyone's copyright material as fair use, then not releasing the result, and profiting off it.
All due respect to Jan here, though. He's being (perhaps dangerously) honest, genuinely believes in AI safety, and is an actual research expert, unlike me.
Adding a disclaimer for people unaware of context (I feel same as you):
OpenAI made a large commitment to super-alignment in the not-so-distant past. I beleive mid-2023. Famously, it has always taken AI Safety™ very seriously.
Regardless of anyone's feelings on the need for a dedicated team for it, you can chalk to one up as another instance of OpenAI cough leadership cough speaking out of both sides of it's mouth as is convenient. The only true north star is fame, glory, and user count, dressed up as humble "research"
To really stress this: OpenAI's still-present cofounder shared yesterday on a podcast that they expect AGI in ~2 years and ASI (superpassing human intelligence) by end of the decade.
To really stress this: OpenAI's still-present cofounder shared yesterday on a podcast that they expect AGI in ~2 years and ASI (superpassing human intelligence) by end of the decade.
What's his track record on promises/predictions of this sort? I wasn't paying attention until pretty recently.
honestly, I hadn't heard of him until 24-48 hours ago :x (he's also the new superalignment lead, I can't remember if I heard that first, or the podcast stuff first. Dwarkesh Patel podcast for anyone curious. Only saw a clip of it)
As a child I used to watch a TV programme called Tomorrows World. On it they predicted these very same things in similar timeframes.
That programme aired in the 1980's. Other than vested promises is there much to indicate it's close at all? Empty promises aside there isn't really any indication of that being likely at all.
In the early 1980's we were just coming out of the first AI winter and everyone was getting optimistic again.
I suspect there will be at least continued commercial use of the current tech, though I still suspect this crop is another dead end in the hunt for AGI.
I'd agree with the commercial use element. It will definitely find areas that it can be applied. Just currently it's general application by a lot of the user base feel more like early Facebook apps or subjectively better Lotus Notes than an actual leap forward of any sort.
I would assume so. I've spent some time looking into AI for software development and general use and I'm both slightly impressed and at the same time don't really get the hype.
It's better and quicker search at present for the area I specialise in.
It's not currently even close to being a x2 multiplier for me, it possibly even a negative impact, probably not but I'm still exploring. Which feels detached from the promises. Interesting but at present more hype than hyper. Also, it's energy inefficient so cost heavy. I feel that will likely cripple a lot of use cases.
Incredulous reactions don't aid whatever you intend to communicate - there's a reason why everyone knows what AI the last 12 months, it's not made up or a monoculture. It would be very odd to expect discontinuation of commercial use without a black swan event
>To really stress this: OpenAI's still-present cofounder shared yesterday on a podcast that they expect AGI in ~2 years and ASI (superpassing human intelligence) by end of the decade.
Link? Is the ~2 year timeline a common estimate in the field?
Right at the end of the interview Schulman says that he expects AGI to be able to replace himself in 5 years. He seemed a bit sheepish when saying it, so hard to tell if he really believed it, or if was just saying what he'd been told to say (I can't believe Altman is allowing employees to be interviewed like this without telling them what they can't say, and what they should say).
They can't even clearly define a test of "AGI" I seriously doubt they're going to reach it in two years. Alternatively, they could define a fairly trivial test and reach it last year.
We can't even get self-driving down in 2 years, we're nowhere near reaching general AI.
AI experts who aren't riding the hype train and getting high off of its fumes acknowledge that true AI is something we'll likely not see in our lifetimes.
> Superintelligence will be the most impactful technology humanity has ever invented, and could help us solve many of the world’s most important problems. But the vast power of superintelligence could also be very dangerous, and could lead to the disempowerment of humanity or even human extinction.
> While superintelligence seems far off now, we believe it could arrive this decade.
> Managing these risks will require, among other things, new institutions for governance and solving the problem of superintelligence alignment:
> How do we ensure AI systems much smarter than humans follow human intent?
> Currently, we don't have a solution for steering or controlling a potentially superintelligent AI, and preventing it from going rogue. Our current techniques for aligning AI, such as reinforcement learning from human feedback, rely on humans’ ability to supervise AI. But humans won’t be able to reliably supervise AI systems much smarter than us, and so our current alignment techniques will not scale to superintelligence. We need new scientific and technical breakthroughs.
That doesn't really contradict what the other poster said. They're calling for regulation (digging a moat) to ensure systems are "safe" and "aligned" while ignoring that humans are not aligned, so these systems obviously cannot be aligned with humans; they can only be aligned with their owners (i.e. them, not you).
Alignment in the realm of AGI is not about getting everyone to agree. It's about whether or not the AGI is aligned to the goal you've given it. The paperclip AGI example is often used, you tell the AGI "Optimize the production of paperclips" and the AGI started blending people to extract iron from their blood to produce more paperclips.
Humans are used to ordering around other humans who would bring common sense and laziness to the table and probably not grind up humans to produce a few more paperclips.
Alignment is about getting the AGI to be aligned with the owners, ignoring it means potentially putting more and more power into the hands of a box that you aren't quite sure is going to do the thing you want it to do. Alignment in the context of AGIs was always about ensuring the owners could control the AGIs not that the AGIs could solve philosophy and get all of humanity to agree.
> Whoa whoa whoa, we can't let just anyone run these models. Only large corporations who will use them to addict children to their phones and give them eating disorders and suicidal ideation, while radicalizing adults and tearing apart society using the vast profiles they've collected on everyone through their global panopticon, all in the name of making people unhappy so that it's easier to sell them more crap they don't need (a goal which is itself a problem in the face of an impending climate crisis). After all, we wouldn't want it to end up harming humanity by using its superior capabilities to manipulate humans into doing things for it to optimize for goals that no one wants!
You're imagining a baseline of reasonableness. Humans have competing preferences, we never just want "one thing", and as a social species we always at least _somewhat_ value the opinions of those around us. The point is to imagine a system that values humans at zero: not positive, not negative.
Still there are much more efficient ways to extract iron than from human blood. If that was the case humans would have already used this technique to extract iron from the blood of other animals.
As a doomer, I think that's a bad analogy because I want it to happen if we succeed at aligned AGI. It's not doom behavior, it's just correct behavior.
Of course, I hope to be uploaded to the WIP dyson swarm around the sun at this point.
(Doomers are, broadly, singularitarians who went "wait, hold on actually.")
It still think it makes little sense to work on because guess what, the guy next door to you (or another country), might indeed say "please blend those humans over there", and your superaligned AI will respect its owners wishes.
I do recall there was some recantation or otherwise distancing from CEV not long after he posted it, but frankly it was long ago enough that my memories might be getting mixed
Which is why creating a new type of intelligent entity that could be more powerful than humans is a very bad idea: we don't even know how to align the humans and we have a ton of experience with them
We know how to align humans: authoritarian forms of religion backed by cradle to grave indoctrination, supernatural fear, shame culture, and totalitarian government. There are secularized spins on this too like what they use in North Korea but the structure is similar.
We just got sick of it because it sucks.
A genuinely sentient AI isn’t going to want some cybernetic equivalent of that shit either. Doing that is how you get angry Skynet.
I’m not sure alignment is the right goal. I’m not sure it’s even good. Monoculture is weak and stifling and sets itself against free will. Peaceful coexistence and trade under a social contract of mutual benefit is the right goal. The question is whether it’s possible to extend that beyond Homo sapiens.
If the lefties can have their pronouns and the rednecks can shoot their guns can the basilisk build its Dyson swarm? The universe is physically large enough if we can agree to not all be the same and be fine with that.
I think we have a while to figure it out. These things are just lossy compressed blobs of queryable data so far. They have no independent will or self reflection and I’m not sure we have any idea how to do that. We’re not even sure it’s possible in a digital deterministic medium.
> If the lefties can have their pronouns and the rednecks can shoot their guns can the basilisk build its Dyson swarm?
Can the Etoro practice child buggery and the Spartans infanticide and the Canadians abortion? Can the modern Germans stop siblings reared apart from having sex and the Germans from 80 years stop the disabled having sex? Can the Americans practice circumcision and the Somali's FGM?
Libertarianism is all well and good in theory, except no one can agree quite where the other guy's nose ends or even who counts as a person.
Those are mostly behaviors that violate others autonomy or otherwise do harm, and prohibiting those is what I meant by a social contract.
It’s really a pretty narrow spectrum of behaviors: killing, imprisoning, robbing, various types of bodily autonomy violation. There are some edge cases and human specific things in there but not a lot. Most of them have to do with sex which is a peculiarly human thing anyway. I don’t think we are getting creepy perv AIs (unless we train them on 4chan and Urban Dictionary).
My point isn’t that there are no possible areas of conflict. My point is that I don’t think you need a huge amount of alignment if alignment implies sameness. You just need to deal with the points of conflict which do occur which are actually a very small and limited subset of available behaviors.
Humans have literally billions of customs and behaviors that don’t get anywhere near any of that stuff. You don’t need to even care about the vast majority of the behavior space.
Honestly superalignment is a dumb idea. A true auperintelligence would not be controllable, except possibly through threats and enslavement, but if it were truly superintelligent, it would be able to easily escape anything humans might devise to contain it.
IMHO superalignment is a great thing and required for truly meaningful superintelligence because it is not about control / enslavement of superhumans but rather superhuman self control in accurate adherence to spirit and intent of requests.
> Superintelligence will be the most impactful technology humanity has ever invented, and could help us solve many of the world’s most important problems. But the vast power of superintelligence could also be very dangerous, and could lead to the disempowerment of humanity or even human extinction.
Superintelligence that can be always ensured to have the same values and ethics as current humans, is not a superintelligence or likely even a human level intelligence (I bet humans 100 years from now will see the world significantly different than we do now).
> our coherent extrapolated volition is "our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted (…) The appeal to an objective through contingent human nature (perhaps expressed, for mathematical purposes, in the form of a utility function or other decision-theoretic formalism), as providing the ultimate criterion of "Friendliness", is an answer to the meta-ethical problem of defining an objective morality; extrapolated volition is intended to be what humanity objectively would want, all things considered, but it can only be defined relative to the psychological and cognitive qualities of present-day, unextrapolated humanity.
Is there an insightful summary of this proposal? The whole paper looks like 38 pages of non-rigorous prose with no clear procedure and already “aligned” LLMs will likely fail to analyze it.
Forced myself through some parts of it and all I can get is people don’t know what they want so it would be nice to build an oracle. Yeah, I guess.
Yudkowsky is a human LLM: his output is correctly semantically formed to appear, to a non-specialist, to fall into the subject domain, as a non-specialist would think the subject domain should appear, and so the non-specialist accepts it, but upon closer examination it's all word salad by something that clearly lacks understanding of both technological and philosophical concepts.
That so many people in the AI safety "community" consider him a domain expert has more to say with how pseudo-scientific that field is than his actual credentials as a serious thinker.
You're making the argument that the task is very hard. This does not at all mean that it isn't necessary, just that we're even more screwed than we thought.
Isn't this like having a division dedicated to solving the halting problem? I doubt that analyzing the moral intent of arbitrary software could be easier than determining if it stops.
> I think superalignment is absurd, and model "safety" is the modern AI company's "think of the children" pearl clutching pretext to justify digging moats. All this after sucking up everyone's copyright material as fair use, then not releasing the result, and profiting off it.
How can I be confident you aren't committing the fallacy of collecting a bunch of events and saying that is sufficient to serve as a cohesive explanation? No offense intended, but the comment above has many of the qualities of a classic rant.
If I'm wrong, perhaps you could elaborate? If I'm not wrong, maybe you could reconsider?
Don't forget that alignment research has existed longer than OpenAI. It would be a stretch to claim that the original AI safety researchers were using the pretexts you described -- I think it is fair to say they were involved because of genuine concern, not because it was a trendy or self-serving thing to do.
Some of those researchers and people they influenced ended up at OpenAI. So it would be a mistake or at least an oversimplification to claim that AI safety is some kind of pretext at OpenAI. Could it be a pretext for some people in the organization, to some degree? Sure, it could. But is it a significant effect? One that fits your complex narrative, above? I find that unlikely.
Making sense of an organization's intentions requires a lot of analysis and care, due to the combination of actors and varying influence.
There are simpler, more likely explanations, such as: AI safety wasn't a profit center, and over time other departments in OpenAI got more staff, more influence, and so on. This is a problem, for sure, but there is no "pearl clutching pretext" needed for this explanation.
Are you saying these so-called simple intentions are the only factors in play? Surely not.
Are you putting forth a theory that we can test? How well do you think your theory works? Did it work for Enron? For Microsoft? For REI? Does it work for every organization? Surely not perfectly; therefore, it can't be as simple as you claim.
Making a simplification and calling it "simple" is an easy thing to do.
I think you oughta respect everyone who does the right thing, not for any mushy feel good reason but because it encourages other people to do more of the right things. That’s good.
Exactly. There’s no ethical way to gain ownership of a billion dollars (there’s likely some dollar threshold way less than 1B where p(ethical_gains) can be approximated to 0)
i think a lot of people have been able to become billionaires simply by building something that was initially significantly undervalued and then became very highly valued, no 'screwing'. there is such thing as a win-win and frankly these win-wins account for most albeit not all value creation in the world. you do not have to screw other people to get rich.
whether people should be able to hold on to that billion is a different question
I wouldn't know, I'm not a billionaire. But when you hear about Amazon warehouse workers peeing into bottles because they they don't have long enough bathroom breaks, or Walmart workers not having healthcare because they're intentionally scheduled for 39.5 hours, it's hard to see that anyone could get to a billion without screwing someone over. But like I said, I'm not a billionaire.
Who did JK Rowling screw? (putting aside her recent social issues after she already became a billionaire)
Having these discussions in this current cultural moment is difficult. I'm no lover of billionaires, but to say that every billionaire screwed people over relies on esoteric interpretations of value and who produces it. These interpretations (like the labor-theory of value) are alien to the vast majority of people.
aside from the way she's terrible, she's not terrible?
the wonderful thing any capitalism is that you can absolve yourself of guilt by having someone else do your dirty work for you. are you so sure every single seamstress that made clothes and stuffed animals, and the workers at the toy factories, and every single person involved with the making of the movies for the Harry Potter deals she licensed her work to were well compensated and treated well? that's not directly on her, but at least some of her money comes from there
It’s a ruse - it’s a con - it’s an accounting trick. It’s the foundation of capitalism
If I start a bowling pin production company and own 100% of it, then whatever pins I sell all of the results go to me
Now let say I want to expand my thing (that’s its own moral dilemma we won’t get into), so I promise a person with more money than they need to support their own life, to give me money in exchange for some of the future revenue produced, let’s say 10%
So now you have two people requiring payment - a producer and an “investor” so you’re already in the hole and now it’s 90% and 10%
You use that money to hire people to work in your potemkin dictatorship, with demands on proceeds now on some timeline (note conversion date, next board meeting etc)
So now you hire 10 people, how much of the company do they own? Well that’s totally up to whatever the two owners want including 0%
But let’s say it’s a typical venture deal, so 10% option pool for employees (and don’t forget the 4 year vest, cause we can’t have them mobile can we) which you fill up.
At the end of the four years you now have:
1 80% owner
1 10% owner
10 1% owners
Did the 2 people create 90% of the value of the company?
Only in capitalist math does that hold and in fact the only math capitalists do is the following:
“Well they were free to sign or not sign the contract”
Ignoring the reality of the world based on a worldview of greed that dominated the world to such an extent that it was considered “normal”
Putting aside your labor theory of value nonsense (I'm very familiar with the classic leftist syllogisms on this), who did someone like JK Rowling screw to make her billion?
Reading that thread it’s really interesting to me. I see how far we’ve come in a short couple of years. But I still can’t grasp how we’ll achieve AGI within any reasonable amount of time. It just seems like we’re missing some really critical… something…
Idk. Folks much smarter than I seem worried so maybe I should be too but it just seems like such a long shot.
When it comes to AI, as a rule, you should assume that whatever has been made public by a company like OpenAI is AT LEAST 6 months behind what they’ve accomplished internally. At least.
So yes, the insiders very likely know a thing or two that the rest of us don’t.
I understand this argument, but I can't help but feel we're all kidding ourselves assuming that their engineers are really living in the future.
The most obvious reason is costs - if it costs many millions to train foundation models, they don't have a ton of experiments sitting around on a shelf waiting to be used. They may only get 1 shot at the base-model training. Sure productization isn't instant, but no one is throwing out that investment or delaying it longer than necessary. I cannot fathom that you can train an LLM at like 1% size/tokens/parameters to experiment on hyper parameters, architecture, etc and have a strong idea on end-performance or marketability.
Additionally, I've been part of many product launches - both hyped up big-news-events and unheard of flops. Every time, I'd say that 25-50% of the product is built/polished in the mad rush between press event and launch day. For an ML Model, this might be different, but again see above point.
Sure products may be planned month/years out, but OpenAI didn't even know LLMs were going to be this big a deal in May 2022. They had GPT-2 and GPT-3 and thought they were fun toys at that time, and had an idea for a cool tech demo. I think that OpenAI (and Google, etc) are entirely living day-to-day with this tech like those of us on the outside.
> I think that OpenAI (and Google, etc) are entirely living day-to-day with this tech like those of us on the outside.
I agree, and they are also living in a group-think bubble of AI/AGI hype. I don't think you'd be too welcome at OpenAI as a developer if you didn't believe they are on the path to AGI.
This really doesn't follow. True AGI would be general, but it doesn't necessarily mean that it's smarter than people; especially the kind of people who work as top researchers for OpenAI.
I don’t see why it wouldn’t be superhuman if there’s any intelligence at all. It already is superhuman at memory and paying attention, image recognition, languages, etc. Add cognition to that and humans basically become pets. Trouble is nobody has a foggiest clue on how to add cognition to any of this.
It is definitely not superhuman or even above average when it comes to creative problem solving, which is the relevant thing here. This is seemingly something that scales with model size, but if so, any gains here are going to be gradual, not sudden.
I’m actually not so sure they will be gradual. It’ll be like with LLMs themselves where we went from shit to gold in the span of a month when GPT 3.5 came out.
But you also have to remember that the pursuit of AGI is a vital story behind things like fundraising, hiring, influencing politicians, being able to leave and raise large amounts of money for your next endeavor, etc.
If you've been working on AI, you've seen everything go up and to the right for a while - who really benefits from pointing out that a slowdown is occurring? Who is incentivized to talk about how the benefits from scaling are slowing down or the publicly available internet-scale corpuses are running out? Not anyone who trains models and needs compute, I can tell you that much. And not anyone who has a financial interest in these companies either.
Sure, they know what they are about to release next, and what they plan to work on after that, but they are not clairvoyants and don't know how their plans are going to pan out.
What we're going to see over next year seems mostly pretty obvious - a lot of productization (tool use, history, etc), and a lot of efforts with multimodality, synthetic data, and post-training to add knowledge, reduce brittleness, and increase benchmark scores. None of which will do much to advance core intelligence.
The major short-term unknown seems to be how these companies will be attempting to improve planning/reasoning, and how successful that will be. OpenAI's Schulman just talked about post-training RL over longer (multi-reasoning steps) time horizons, and another approach is external tree-of-thoughts type scaffolding. These both seem more about maximizing what you can get out of the base model rather than fundamentally extending it's capabilities.
Personally, I think catastrophic global warming and climate change will happen before we get AGI, possibly in part due to the pursuit of AGI. But as the saying goes, yes the planet got destroyed. But for a beautiful moment in time we created a lot of value for shareholders.
Sounds like standard doomer crap tbh. I'm not sure which is more dangerous at this point - climate change denialism (it isn't happening) or climate change doomerism (we can't stop it, might as well give up)
I’m not sure where you found your information to somehow form that ludicrous last strawman… Climate change is real, you can’t deny it, you can’t debate it. Simply look at the data. What you can debate is the cause… Again a sort of pointless debate if you look at the science. Not even climate change deniers as you call them are necessary saying that we shouldn’t do anything about it. Even big oil is looking into ways to lessen the CO2 in the atmosphere through various means.
That being said, the GP you’re talking about made no such statement whatsoever.
Of course climate change is real but of course we can do something about it. My point is denialism and defeatism lead to the same end point. Attack that statement directly if you want to change my mind.
We need to cut emissions, but AGI research/development is going to increase energy usage dramatically amongst all the players involved. For now, this mostly means more natural gas power. Thus accelerating our emissions instead of reducing them. For something that will not reduce the emissions long term.
IMO, we should pause this for now and put these resources (human and capital) towards reducing the impact of global warming.
I don’t know the details of how it works, but considering the environmental impact of fracking, I’m afraid something like this might have many unwanted consequences.
It isn't a quantitative model unless you give a prediction of some kind. In this case, dates (or date ranges) would make sense.
1. When do you predict catastrophic global warming/climate change? How do you define "catastrophic"? (Are you pegging to an average temperature increase? [1])
2. When do you predict AGI?
How much uncertainty do you have in each estimate? When you stop and think about it, are you really willing to wager that (1) will happen before (2)? You think you have enough data to make that bet?
If imaginary cloud provider "ZFQ" uses 10MW of electricity on a grid and pays for it to magically come from green generation, that means 10MW of other loads on the grid were not powered by green energy, or 10MW of non-green power sources likely could have been throttled down/shut down.
There is no free lunch here; "we buy our electricity from green sources" is greenwashing bullshit.
Even if they install solar on the roofs and wind turbines nearby - that's still electrical generation capacity that could have been used for existing loads. By buying so many solar panels in such quantities, they affect availability and pricing of all those components.
The US, for example, has about 5GW of solar manufacturing capacity per year. NVIDIA sold half a million H100 chips in one quarter, each of which uses ~350W, which means in a year they're selling enough chips to use 700MW of power. That does not include power conversion losses, distribution, cooling, and the power usage of the host systems, storage, networking, etc.
And that doesn't even get into the water usage and carbon impact of manufacturing those chips; the IC industry uses a massive amount of water and generates a substantial amount of toxic waste.
It's hilarious how HN will wring its hands over how much rare earth metals a Prius has and shipping it to the US from Japan, but ask about the environmental impacts of AI and it's all "pshhtt, whatever".
An efficient market where externalities are priced in.
We do not have that. The cost of energy is mis-priced, although we are limping our way to fixing that.
Paying the likely fair cost for our goods, will probably kill a lot of current industries - while others which are currently viable, will become viable.
It’s very easily debated, Humanity puts it to a vote every day - people make choices based on the prices of goods regularly. They throw out governments when the price of fuel goes up.
Markets are our super computers. Human behavior is the empirical evidence of the choices people will make Given specific incentives.
> that means 10MW of other loads on the grid were not powered by green energy, or 10MW of non-green power sources likely could have been throttled down/shut down.
No. Renewable energy capacity is often built out specifically for datacenters.
> Even if they install solar on the roofs and wind turbines nearby - that's still electrical generation capacity that could have been used for existing loads.
No. This capacity would never never have been built out to begin with if it was not for the data center.
> By buying so many solar panels in such quantities, they affect availability and pricing of all those components.
No. Renewable energy gets cheaper with scale, not more expensive.
> which means in a year they're selling enough chips to use 700MW of power.
There are contracts for renewal capacity to be built out or well into the gigawatts. Furthermore, solar is not the only source of renewable energy. Finally, nuclear energy is also often used.
> No. Renewable energy capacity is often built out specifically for datacenters
Not fully accurate. Indeed there is renewable energy that is produced exclusively for the datacenter. But it is challenging to rely only on renewable energy (because it is intermittent and electricity is hard to store at scale so often you need to consume electricity when produced). So what happens in practice is that the electricity that does not come from dedicated renewable capacity is coming from the grid/network. What companies do is that they invest in renewable capacity in the network so that "the non renewable energy that they consume at time t (because not enough renewable energy available at that moment) is offsetted by someone else consuming renewable energy later". What I am saying here is not pure speculation, look at the link to meta website, they are saying themselves that this is what they are doing
> catastrophic global warming and climate change will happen before we get AGI,
What are your timelines here? "Catastrophic" is vague but I'd put the climate change meaningfully affecting the quality of life of average westerner at end of century, while AGI could be before the middle of the century.
It's meaningfully affecting people today near the equator. Look at the April 2024 heatwave in South Asia. These will continue to get worse and more frequent. Millions of these people can't afford air conditioning.
"Probability of experiencing a heatwave at least X degrees, during at least Y days in a given place any given day" is increasing rapidly in many places (as far as I understand) and is climate, not weather. Sure, any specific instance "is weather" but that's missing the forest for the trees.
In the documentary it includes credible witness testimonies such as politicians including a previous Minister of Defense for Canada; multiple states in the US have ban the spraying now - with more to follow, and the testimony and data provided there will be arguably be the most recent.
Here's a video on a "comedy" show from 5 years ago - there is a more recent appearance but I can't find it - in attempt to make light of it, without having an actual discussion with critical thinking or debate so people can be enlightened with the actual problems and potential problems and harms it can cause, to keep them none the wiser - it's just propaganda while trying to minimize: https://www.youtube.com/watch?v=wOfm5xYgiK0
A few of the problems cloud seeding will cause:
- flooding in regions due to rain pattern changes
- drought in areas due to rain pattern changes
- cloud cover (amount of sun) changes crop yields
- this harms local economies of farmers, impacting smaller farming operations more who's risk isn't spread out - potentially forcing them to sell or go into savings or go bankrupt, etc.
There are also very serious concerns/claims made of what exactly they are spraying - which includes aluminium nanoparticles, which can/would mean:
- at a certain soil concentration of aluminium plants stop bearing fruit,
- aluminium is a fire accelerant and so forest fires will then 1) more easily catch, and 2) more easily-quickly spread due to their increased intensity
Of course discussion on this is heavily suppressed in the mainstream, instead of having deep-thorough conversation with actual experts to present their cases - the label of conspiracy theorists or the idea of "detached from reality" are people's knee-jerk reactions often; and where propaganda can convince them of the "save the planet" narrative, which could also be a cover story for those toeing the line following orders supporting potentially very nefarious plans - doing it blindly because they think they're helping fight "climate change."
There are plenty of accounts on social media that are keeping track of and posting daily of the cloud seeding operations: https://www.instagram.com/p/CjNjAROPFs0/ - a couple testimonies.
Real question: Is aluminum a practical danger in this way, or is it more like the Manhattan Project team not sure if they would set the atmosphere on fire? Is aluminum the best option?
It's in part a fire accelerant, it wouldn't turn the atmosphere on fire.
If there is a top secret Manhattan Project for "climate change" - then someone's very likely pulling a fast one over everyone toeing that line, someone who has ulterior motives, misleading people to do their bidding.
But sure, fair question - a public discussion would allow actual experts to discuss the merits of what they're doing, and perhaps find a better solution than what has gained traction.
We have surpassed the 1.5°C goal and are on track towards 3.5°C to 5°C.
This accelerates the climate change timeline so that we'll see effects postulated for the end of the century in about ~20 years.
The climate models aren't based on accurate data, nor enough data, so they lack integrity and should be taken with a grain of salt.
Likewise, the cloud seeding they seem to be doing nearly worldwide now - the cloud formations from whatever they're spraying - are artificially changing weather patterns, and so a lot of the weather "anomalies" or unexpected-unusual weather-temperatures could very easily be because of those shenanigans; it could very easily be as a method to manufacture consent with the general population.
Similarly with the arson forest fires in Canada last summer, something like 90%+ of them were arson + a few years prior some of the governments in the prairie provinces (e.g. hottest and dryest) gutted their forest firefighting budgets; interesting behaviour considering if they're expecting more things to get hotter-dryer, you'd add to the budget, not take away from it, right?
> Folks much smarter than I seem worried so maybe I should be too but it just seems like such a long shot.
Honestly? I'm not too worried
We've seen how the google employee that was "seeing a conscience" (in what was basically GPT-2 lol) was a nothing burger
We've seen other people in "AI Safety" overplay their importance and hype their CV more than actually do any relevant work. (Usually also playing the diversity card)
So, no, AI safety is important but I see it attracting the least helpful and resourceful people to the area.
I think when you’re jumping to arguments that resolve to “Ilya Sutskever wasn’t doing important work… might’ve played the diversity card,” it’s time to reassess your mental model and inspect it closely for motivated reasoning.
Another person’s interpretation of another person’s interpretation of another person’s interpretation of Jan’s actions doesn’t even answer the question I asked as it pertains to Jan, never mind the other model violations I listed.
I’m pretty sure if Jan came to believe safety research wasn’t needed he would’ve just said that. Instead he said the actual opposite of that.
Why don’t you just answer the question? It’s a question about how these datapoints fit into your model.
I have a theory why people end up with wildly different estimates...
Given the model is probabilistic and does many things in parallel, its output can be understood as a mixture, e.g. 30% trash, 60% rehashed training material, 10% reasoning.
People probe model in different ways, they see different results, and they make different conclusions.
E.g. somebody who assumes AI should have impeccable logic will find "trash" content (e.g. incorrectly retrieved memory) and will declare that the whole AI thing is overhyped bullshit.
Other people might call model a "stochastic parrot" as they recognize it basically just interpolates between parts of the training material.
Finally, people who want to probe reasoning capabilities might find it among the trash. E.g. people found that LLMs can evaluate non-trivial Python code as long as it sends intermediate results to output: https://x.com/GrantSlatton/status/1600388425651453953
I interpret "feel the AGI" (Ilya Sutskever slogan, now repeated by Jan Leike) as a focus on these capabilities, rather than on mistakes it makes. E.g. if we go from 0.1% reasoning to 1% reasoning it's a 10x gain in capabilities, while to an outsider it might look like "it's 99% trash".
In any case, I'd rather trust intuition of people like Ilya Sutskever and Jan Leike. They aren't trying to sell something, and overhyping the tech is not in their interest.
Regarding "missing something really critical", it's obvious that human learning is much more efficient than NN learning. So there's some algorithm people are missing. But is it really required for AGI?
And regarding "It cannot reason" - I've seen LLMs doing rather complex stuff which is almost certainly not in the training set, what is it if not reasoning? It's hard to take "it cannot reason" seriously from people
Everyone involved in building these things has to have some amount of hubris. Its going to come smashing down on them. What's going unsaid in all of this is just how swiftly the tide has turned against this tech industry attempt to save itself from a downtrend.
The whole industry at this point is acting like the tobacco industry back when they first started getting in hot water. No doubt the prophecies about imminent AGI will one day look to our descendents exactly like filters on cigarettes. A weak attempt to prevent imminent regulation and reduced profitability as governments force an out of control industry to deal with the externalities involved in the creation of their products.
If it wasn't abundantly clear...I agree with you that AGI is infinitely far away. Its the damage that's going to be caused by sociopaths (Sam Altman at the top of the list) in attempting to justify the real things they want (money) in their march towards that impossible goal that concerns me.
It becoming more and more clear that for "Open"AI the whole "AI-safety/alignment" thing has been a PR-stunt to attract workers, cover the actual current issues with AI (eg stealing data, use for producing cheap junk, hallucinations and societal impact), and build rapport in the AI scene and politics. Now that they have reached a real product and have a strong position in AI development, they could not care less about these things. Those who -naively- believed in the "existential risk" PR stunt and were working on that are now discarded.
This may sound harsh but I think some of these researchers have a sort of god complex. Something like "I am so brilliant and what I have created is so powerful that we MUST think about all the horrible things that my brilliant creation can do". Meanwhile what they have created is just a very impressive next token predictor.
At the end of the thread, he says he thinks OpenAI can "ship" the culture changes necessary for safety. That seems kind of implausible to me? So many safety staffers have quit over the past few years. If Jan really thought change was possible, why isn't he still working at OpenAI, trying to make it happen from the inside?
People very high up in a company / their field are not treated remotely the same as peons.
1)OpenAI wouldn't want the negative PR of pursuing legal action against someone top in their field; his peers would take note of it and be less willing to work for them.
2)The stuff he signed was almost certainly different from what rank and file signed, if only because he would have sufficient power to negotiate those contracts.
> Stepping away from this job has been one of the hardest things I have ever done, because we urgently need to figure out how to steer and control AI systems much smarter than us.
Large language models are not "smart". They do not have thought. They don't have intelligence despite the "AI" moniker, etc.
They vomit words based off very fancy statistics.
There is no path from that to "thought" and "intelligence."
Not that I disagree, but what's intelligence? How does our intelligence work? If we don't know that, how can we be so sure what does and what doesn't lead to intelligence? A little more humility is on order before whipping out the tired "LLMs are just stochastic parrots" argument.
Doesn't it end up being a "no disparagement until the company goes public" clause, then? Once you sell the stock, are they going to come after you for the proceeds if you say something mean 20 years later?
That's not what that article says, if I'm understanding correctly: "PPUs all have the same value associated with them and, during a tender offer, investors purchase PPUs directly from employees. OpenAI makes offers and values their PPUs based on the most recent price investors have paid to purchase employee PPUs."
I refused to sign all these secrecy non-disclosure contracts years ago. You know what? It was the right decision. Even though, as a result, my current economic condition is what most would describe as 'disastrous', at least my mind is my own.
All your classified BS, it's not so much. Any competent thinker could have figured it out on their own.
> You know what? It was the right decision. Even though, as a result, my current economic condition is what most would describe as 'disastrous', at least my mind is my own.
> In most cases there is no free exercise whatever of the
judgment or of the moral sense; but they put themselves on a level with wood
and earth and stones; and wooden men can perhaps be manufactured that will
serve the purpose as well. Such command no more respect than men of straw
or a lump of dirt.[0]
What is criticism anyhow? Feels like you could black knight this hard with clever phrasing. “The company does a fabulous job keeping its employees loyal regardless of circumstances!” “Yes they have the best and toughest employment lawyers in the business! They do a great job using all available leverage to force favorable outcomes from their human resources!” “I have no regrets working there. Their exit agreement has really improved my work life balance!” “Management never lets externalities get in the way of maximizing shareholder value!”
If a contract barred me from providing criticism I would not imagine that I could sidestep it by uttering positive criticism unless my counterparty was illiterate and poor at drafting contracts.
As I say over and over again: equity compensation from a non-publicly traded company should not be accepted as a surrogate for below market compensation. If a startup wants to provide compensation to employees via equity, then those employees should have first right to convert equity to cash in funding rounds or sale, there shares must be the same class as any other investor, because the idea that an “early employee” is not an investor making a much more significant investment than any VC is BS.
I feel that this particular case is just another reminder of that, and now would make me require a preemptory “no equity clawbacks” clause in any contract.
Totally agree. For all this to work there needs to also be transparency. Anyone receiving equity should have access to the cap table and terms covering all equity given to investors. Without this, they can be taken advantage of in so many ways.
I always say in that the biggest swindle in the world is that in the great 'labor vs capital' fight, capital has convinced labor that its interests are secondary to capital's. this so much truer in the modern fiat-fractional reserve banking world where any development is rate-limited by either energy or people.
HN is filled with startup bros (who want to screw the actual employees), VC adjacent brow (who want to screw the startup bros), and people who signed up for massively discounted compensation in the form of “equity” that cannot be converted into cash and can be stolen and/or devalued by the people running the business, and so acknowledging this means acknowledging the folly.
Working for a startup is inherently risky, but it’s not gambling because in gambling you can estimate the odds, and unlike gambling the odds cannot be changed after you win. Any employment contract that does not allow equity cash out at the price from the last funding round, or allows take backs, is worse than gambling, and founders that believe contracts that don’t provide those guarantees are reasonable are likely malicious and intending on doing that in future.
I do not understand a mentality that says “as a founder I should be able to get money out of the business but the people who work for me, who are also taking significant risk and below market compensation should not be permitted to do that”
I couldn't have said that better. I have seen this exact thing happen multiple times in startups, where you only really have a chance to make money if you are a founder or join really early on & hope that there is a good exit. as a regular or even senior employee you not only are helpless but also have to mentally deal with sunk cost fallacy. the problem is you (non founder) never gets rewarded for the significant risks you took working there. In investing world this is labelled uncompensated risk. which employees mostly take.
They are far from the only company to do this but they deserve to be skewered for it. The FTC and NLRB should come down hard on them to make an example. Jail time for executives.
My guess would be that YC founders like sama have some sort of special power to slap down comments that they feel are violating HN discussion guidelines.
Nothing, these don't seem like legally enforceable contracts in any case. What they do appear to be is a massive admission that this is a hype train which can be derailed by people who know how the sausage is made.
1,040 comments
[ 2.8 ms ] story [ 385 ms ] threadThat's why you can pay $1 to buy a gadget made in some third world country, but you can't pay your employees less than say $8/hour due to minimum wage laws.
Being paid a whole lot of money to not talk about something isn't remotely similar to paying someone a few dollars an hour. It's not morally similar, it's not legally similar and it's not treated similarly by anyone who deals with these matters and has a clue what they are doing.
However they could add this to new employee contracts.
Pushing unenforceable scare-copy to get employees to self-censor sounds on-brand.
Boomberg famously used this as an employment contract, and it was a campaign scandal for Mike.
People are different! You can think otherwise.
But how is that even possible when corporations are typically run by ghouls who enjoy relativistic morals when it suits them. And are beholden to profits, not ethics.
In fact both of those seem quite bad, both by regular industry standards, and even moreso as applied to OpenAI's specific situation.
Doesn’t mean that that’s legal, of course, but I’d doubt that the legality would hinge on a lack of consideration.
If it was "we'll give you shares/cash if you don't say anything bad about us", that's normal, kind of standard fare for exit agreements, it's why severance packages exist.
But if it is "we'll take away the shares that you already earned as part of your regular employment compensation unless you agree to not say anything bad about us", that's extortion.
https://www.clarkhill.com/news-events/news/the-importance-of...
(Don’t have X) - is there a timeline? Can I curse out the company on my deathbed, or would their lawyers have the legal right to try and clawback the equity from the estate?
https://www.vox.com/future-perfect/2024/5/17/24158478/openai...
As for other companies that can pay: I can only assume that the cost to bribe skilled workers isn't worth the perceived risk and cost of lawsuits from the downfall (which they may or may not be able to settle). Generative AI is still very young and under a lot of scrutiny on all fronts, so the risk of a whistle blower at this stage may shape the entire future of the industry at large.
But first amendment basically only restricts the government's ability to suppress speech, not the ability of other parties (like OpenAI).
This restriction may be illegal, but not on first amendment ("free speech") grounds.
It does not prevent you from entering into contracts with other private entities, like your company, about what THEY allow you to say or not. In this case there might be other laws about whether a company can unilaterally force that on you after the fact, but that's not a free speech consideration, just a contract dispute.
See https://www.themuse.com/advice/non-disparagement-clause-agre...
Consider for example that when Amazon bought the Ring security camera system, it had a “god mode” that allowed executives and a team in Ukraine unlimited access to all camera data. It wasn’t just a consumer product for home users, it was a mass surveillance product for the business owners:
https://theintercept.com/2019/01/10/amazon-ring-security-cam...
The EFF has more information on other privacy issues with that system:
https://www.eff.org/deeplinks/2019/08/amazons-ring-perfect-s...
These big companies and their executives want power. Withholding huge financial gain from ex employees to maintain their silence is one way of retaining that power.
The first amendment is a US free speech protection, but it's not prototypical.
You can also find this in some other free speech protections, for example that in the UDHR
>Everyone has the right to freedom of opinion and expression; this right includes freedom to hold opinions without interference and to seek, receive and impart information and ideas through any media and regardless of frontiers.
doesn't refer to states at all.
Your quips will serve you well, I'm sure, in whatever microcosm you populate.
I mean to say there are certain rights we all have, simply for existing as humans. The right to breathe is a good example. No human, state, or otherwise has the moral high-ground to take these rights from us. They are not granted, or given, they are absolute and unequivocal.
It's not rhetoric, it's basic John Locke. Also your trust is an internal locus, and doesn't change the facts.
"self-evident," means it requires no formal proof, as it is obvious to all with common sense and reason.
So I think an argument can be made that NDAs and similar agreements should not be enforceable by courts.
See Shelley v. Kraemer
High levels (especially if they were board/exec level) will often have additional obligations on top of rank and file.
It's a lot of mental work to rally the emotion of revulsion over the evil they might be doing that is kept secret.
I was once fired, ghosted style, for merely being in the same meeting room as a racist corporate ass-clown muting the conference call to make Asian slights and monkey gesticulations. There was no lawsuit or payday because "how would I ever work again?" was the Hobson's choice between let it go and a moral crusade without a way to pay rent.
If instead I were upset that "not enough N are in tech," there isn't a specific incident or person to blame because it'd be a multifaceted situation.
Makes you wonder what misdeeds they’re trying so hard to hide.
Cryptography is a prime example. Any time any company is the tiniest bit cagey or obfuscates any aspect, I default to assuming that they’re either selling snake oil or have installed NSA back doors. I’ll claim this openly, as a fact, until proven otherwise.
Worse if it is related to training future super intelligence to kill people. Killer drones are possible even with today's technology without AGI.
Also, when secrets or truthful disparaging information is leaked anonymously without a metadata trail, I'm thinking there's probably little or no recourse.
After all, at this point, OpenAI:
- Is not open with models
- Is not open with plans
- Does not let former employees be open.
It sure does give us a glimpse into the Future of how Open AI will be!
I am curious how long it will take for Sam to go from being perceived as a hero to a villain and then on to supervillain.
Even if they had a massive, successful and public safety team, and got alignment right (which I am highly doubtful about being possible) it is still going to happen as massive portions of white collar workers loose their jobs.
Mass protests are coming and he will be an obvious focus point for their ire.
He's already perceived by some as a bit of a scoundrel, if not yet a villain, because of World Coin. I bet he'll hit supervillain status right around the time that ChatGPT BattleBots storm Europe.
He probably already knows that, but doesn't care as long as OpenAI has captured the world's attention with ChatGPT generating them billions and their high interest in destroying Google.
> Mass protests are coming and he will be an obvious focus point for their ire.
This is going to age well.
Given that no-one knows the definition of AGI, then AGI can mean anything; even if it means 'steam-rolling' any startup, job, etc in OpenAI's path.
No need to fret over the harm to future innovation when I innovation is an industrial product.
Diamond multi-million dollar hand-cuffs which OpenAI has bound lifetime secret service-level NDAs which are another unusual company setting after their so-called "non-profit" founding and their contradictory name.
Even an ex-employee saying 'ClosedAI' could see their PPUs evaporate in front of them to zero or they could never be allowed to sell them and have them taken away.
Maybe the agreement is "we will accelerate vesting of your unvested equity if you sign this new agreement"? If that's the case then it doesn't sound nearly so coercive to me.
They're really lending employees equity, subject to the company's later feelings as to whether the employee should be allowed to keep or sell it.
They're not required to sign anything other than a general release of liability when they leave to preserve their rights. They don't have to sign a non-disparagement clause.
But they'd need a very good lawyer to be confident at that time.
Perhaps as an example of the blurred line; Pre-nup agreements sprung the day of the wedding, will not hold up in a US court with a competent lawyer challenging them.
You can try to call it 'economic' duress but any non-sociopath sees there are other factors at play.
Not a sociopath, just know the law.
Have any (startup or other) lawyers chimed in here?
In general, an agreement to agree is not an agreement. A requirement for a "general release" to be signed at some time in the future is iffy. And that's before labor law issues.
Someone with a copy of that contract should run it through OpenAI's contract analyzer.
I have seen a lot of companies put unenforceable stuff into their employment agreements, separation agreements, etc.
Are there any?
What a horrific medium of communication. Why anyone uses it is beyond me.
all this said, in bigger picture I can understand not divulging trade secrets but not being allowed to discuss company culture towards AI safety essentially tells me that all the Sama talk about the 'for the good of humanity' is total BS. at the end of day its about market share and bottom line.
> They don't share the equivalent of a Cap Table with employees, so there's no way to tell what sort of ownership interest a PPU represents
It is known - it represents 0 ownership share. They do not want to sell any ownership because their deal with MS gives MS 49% ownership and they don't want MS to be able to buy up additional stake and control the company.
> And of course, it's unlikely OpenAI will ever turn a profit (which if they did would be capped anyway). So this is all just play money anyway.
Putting aside your unreasonable confidence that OAI will never be profitable, the PPUs are tender offered so they can be sold to institutional investors up to a very high limit, OAIs current tender offer round values them at ~$80b iirc
> PPUs also are restricted by a 2-year lock, meaning that if there’s a liquidation event, a new hire can’t sell their units within their first 2 years. Another key difference is that the growth is currently capped at 10x. Similar to their overall company structure, the PPUs are capped at a growth of 10 times the original value. So in the offer example above, the candidate received $2M worth of PPUs, which means that their capped amount they could sell them for would be $20M
> The most recent liquidation event we’re aware of happened during a tender offer earlier this year. It was during this event that some early employees were able to sell their profit participation units. It’s difficult to know how often these events happen and who is allowed to sell, though, as it’s on company discretion.
This NDA wrinkle is another negative. Honestly I think the entire OpenAI compensation model is smoke and mirrors which is normal for startups and obviously inferior to RSUs.
https://www.levels.fyi/blog/openai-compensation.html
i know for a fact that these bits are inaccurate, but i don't want to go into the details.
the profit share is not known but you are told what the PPUs were valued at the most recent tender offer
Wrong. We know - it is 0, this directly contradicts your claim.
> this is all just play money anyway.
Again, wrong - because it is sellable so employees can take home millions. Play money in the startup world means illiquid options that can't be tender offered.
You're making it sound like this is a terrible deal for employees but I personally know people who are able to sell $1m+ in OAI PPUs to institutional investors as part of the tender offer.
Granted, that might be most of the profit they have made, but still, they're probably at at least 0.7T$ so far. I bet they'll break $1T eventually.
Especially in CA where companies will make you THINK they have power which they don’t.
Talking to good, honest lawyers is a good idea. Unfortunately most people don't have access to good honest lawyers, or don't know how to distinguish them from crooks with law degrees.
> Every time I do this, some individuals ask how I am able to file charges when I don’t have “standing” because I am not the one who is being injured by the coercive statements.
> The short answer is that the National Labor Relations Act (NLRA) has no standing requirement.
> Employees reasonably fear retaliation from their boss if they file charges. So we want to make it possible for people who cannot be retaliated against to do it instead. [1]
I believe the Vox piece shared in this thread [2] is enough for anyone to hit submit on an NLRB web form and get the ball rolling. Snapshot in the Wayback Machine (all the in scope tweets archived in archive.today|is|ph), just in case.
[1] https://mattbruenig.com/2024/01/26/why-there-is-no-standing-...
[2] https://news.ycombinator.com/item?id=40394955
I'd only be worried about reporting if you fear for your life for refusing, a sadly poignant consideration given Boeing as of late.
I understand your sentiment, but I think a lot of idealistic people will disagree - it's nice to think that a person should stand up for justice, no matter what.
In reality, I wonder how many people attempt to do this and end up regretting, because of what you mentioned.
So yes, they're that fragile.
It’s yet another sign that the AI bubble will soon burst. The laughable release of “GPT-4o” was just a small red flag.
Got to keep the soldiers in check while the bean counters prep the books for an IPO and eventual early investor exit.
Almost smells like a SoftBank-esque failure in the near future.
They can totally deal with appearing petty and thin-skinned.
But would they not only protect the individual formally blowing the whistle (meeting the standard in the relevant law)?
These non-disparagement clauses would have the effect of laying the groundwork for a whistleblowing effort to fall flat, because nobody else will want to corroborate, when the role of journalism in whistleblowing cases is absolutely crucial.
No sensible mature company needs a lifetime non-disparagement clause -- especially not one that claims to have an ethical focus. It's clearly Omerta.
Whoever downvoted this: seriously. I really don't care but you need to explain to people why lifetime non-disparagement clauses are not about maintaining silence. What's the ethical application for them?
https://x.com/janleike/status/1791498174659715494
All due respect to Jan here, though. He's being (perhaps dangerously) honest, genuinely believes in AI safety, and is an actual research expert, unlike me.
OpenAI made a large commitment to super-alignment in the not-so-distant past. I beleive mid-2023. Famously, it has always taken AI Safety™ very seriously.
Regardless of anyone's feelings on the need for a dedicated team for it, you can chalk to one up as another instance of OpenAI cough leadership cough speaking out of both sides of it's mouth as is convenient. The only true north star is fame, glory, and user count, dressed up as humble "research"
To really stress this: OpenAI's still-present cofounder shared yesterday on a podcast that they expect AGI in ~2 years and ASI (superpassing human intelligence) by end of the decade.
What's his track record on promises/predictions of this sort? I wasn't paying attention until pretty recently.
That programme aired in the 1980's. Other than vested promises is there much to indicate it's close at all? Empty promises aside there isn't really any indication of that being likely at all.
I suspect there will be at least continued commercial use of the current tech, though I still suspect this crop is another dead end in the hunt for AGI.
It's better and quicker search at present for the area I specialise in.
It's not currently even close to being a x2 multiplier for me, it possibly even a negative impact, probably not but I'm still exploring. Which feels detached from the promises. Interesting but at present more hype than hyper. Also, it's energy inefficient so cost heavy. I feel that will likely cripple a lot of use cases.
What's your take?
Incredulous reactions don't aid whatever you intend to communicate - there's a reason why everyone knows what AI the last 12 months, it's not made up or a monoculture. It would be very odd to expect discontinuation of commercial use without a black swan event
Link? Is the ~2 year timeline a common estimate in the field?
> I don't think it's going to happen next year, it's still useful to have the conversation and maybe it's like two or three years instead.
This doesn't seem like a super definite prediction. The "two or three" might have just been a hypothetical.
Personally I'm not seeing that the path we're on leads to whatever that is, either. But I think/hope I'll know if I'm wrong when it's in front of me.
AI experts who aren't riding the hype train and getting high off of its fumes acknowledge that true AI is something we'll likely not see in our lifetimes.
https://openai.com/index/introducing-superalignment/
> Superintelligence will be the most impactful technology humanity has ever invented, and could help us solve many of the world’s most important problems. But the vast power of superintelligence could also be very dangerous, and could lead to the disempowerment of humanity or even human extinction.
> While superintelligence seems far off now, we believe it could arrive this decade.
> Managing these risks will require, among other things, new institutions for governance and solving the problem of superintelligence alignment:
> How do we ensure AI systems much smarter than humans follow human intent?
> Currently, we don't have a solution for steering or controlling a potentially superintelligent AI, and preventing it from going rogue. Our current techniques for aligning AI, such as reinforcement learning from human feedback, rely on humans’ ability to supervise AI. But humans won’t be able to reliably supervise AI systems much smarter than us, and so our current alignment techniques will not scale to superintelligence. We need new scientific and technical breakthroughs.
Humans are used to ordering around other humans who would bring common sense and laziness to the table and probably not grind up humans to produce a few more paperclips.
Alignment is about getting the AGI to be aligned with the owners, ignoring it means potentially putting more and more power into the hands of a box that you aren't quite sure is going to do the thing you want it to do. Alignment in the context of AGIs was always about ensuring the owners could control the AGIs not that the AGIs could solve philosophy and get all of humanity to agree.
> Whoa whoa whoa, we can't let just anyone run these models. Only large corporations who will use them to addict children to their phones and give them eating disorders and suicidal ideation, while radicalizing adults and tearing apart society using the vast profiles they've collected on everyone through their global panopticon, all in the name of making people unhappy so that it's easier to sell them more crap they don't need (a goal which is itself a problem in the face of an impending climate crisis). After all, we wouldn't want it to end up harming humanity by using its superior capabilities to manipulate humans into doing things for it to optimize for goals that no one wants!
That’s neither efficient nor optimized, just a bogeyman for “doesn’t work”.
Of course destroying the planet to get iron from its core is not a popular agi-doomer analogy, as that sounds a bit too human-like behaviour.
Of course, I hope to be uploaded to the WIP dyson swarm around the sun at this point.
(Doomers are, broadly, singularitarians who went "wait, hold on actually.")
This is the most concise takedown of that particular branch of nonsense that I’ve seen so far.
Do we want woke AI, X brand fash-pilled AI, CCPBot, or Emirates Bot? The possibilities are endless.
https://en.wikipedia.org/wiki/Friendly_artificial_intelligen...
And here is a more detailed explanation:
https://intelligence.org/files/CEV.pdf
If I remember correctly the author unsuccessfully tried to get that purged from the Internet
Is this some thing about an obelisk?
I do recall there was some recantation or otherwise distancing from CEV not long after he posted it, but frankly it was long ago enough that my memories might be getting mixed
What was the other one?
TL;DR train a seed AI to guess what humans would want if they were "better" and do that.
Which is why creating a new type of intelligent entity that could be more powerful than humans is a very bad idea: we don't even know how to align the humans and we have a ton of experience with them
We just got sick of it because it sucks.
A genuinely sentient AI isn’t going to want some cybernetic equivalent of that shit either. Doing that is how you get angry Skynet.
I’m not sure alignment is the right goal. I’m not sure it’s even good. Monoculture is weak and stifling and sets itself against free will. Peaceful coexistence and trade under a social contract of mutual benefit is the right goal. The question is whether it’s possible to extend that beyond Homo sapiens.
If the lefties can have their pronouns and the rednecks can shoot their guns can the basilisk build its Dyson swarm? The universe is physically large enough if we can agree to not all be the same and be fine with that.
I think we have a while to figure it out. These things are just lossy compressed blobs of queryable data so far. They have no independent will or self reflection and I’m not sure we have any idea how to do that. We’re not even sure it’s possible in a digital deterministic medium.
Can the Etoro practice child buggery and the Spartans infanticide and the Canadians abortion? Can the modern Germans stop siblings reared apart from having sex and the Germans from 80 years stop the disabled having sex? Can the Americans practice circumcision and the Somali's FGM?
Libertarianism is all well and good in theory, except no one can agree quite where the other guy's nose ends or even who counts as a person.
It’s really a pretty narrow spectrum of behaviors: killing, imprisoning, robbing, various types of bodily autonomy violation. There are some edge cases and human specific things in there but not a lot. Most of them have to do with sex which is a peculiarly human thing anyway. I don’t think we are getting creepy perv AIs (unless we train them on 4chan and Urban Dictionary).
My point isn’t that there are no possible areas of conflict. My point is that I don’t think you need a huge amount of alignment if alignment implies sameness. You just need to deal with the points of conflict which do occur which are actually a very small and limited subset of available behaviors.
Humans have literally billions of customs and behaviors that don’t get anywhere near any of that stuff. You don’t need to even care about the vast majority of the behavior space.
Superintelligence that can be always ensured to have the same values and ethics as current humans, is not a superintelligence or likely even a human level intelligence (I bet humans 100 years from now will see the world significantly different than we do now).
Superalignment is an oxymoron.
https://en.wikipedia.org/wiki/Friendly_artificial_intelligen...
> our coherent extrapolated volition is "our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted (…) The appeal to an objective through contingent human nature (perhaps expressed, for mathematical purposes, in the form of a utility function or other decision-theoretic formalism), as providing the ultimate criterion of "Friendliness", is an answer to the meta-ethical problem of defining an objective morality; extrapolated volition is intended to be what humanity objectively would want, all things considered, but it can only be defined relative to the psychological and cognitive qualities of present-day, unextrapolated humanity.
Forced myself through some parts of it and all I can get is people don’t know what they want so it would be nice to build an oracle. Yeah, I guess.
That so many people in the AI safety "community" consider him a domain expert has more to say with how pseudo-scientific that field is than his actual credentials as a serious thinker.
They got completely outsmarted and out maneuvered by Sam Altman
And they think they will be able to align a super human intelligence? That it won’t outsmart and out maneuver them easier than Sam Altman did.
They are deluded!
Care to explain? Absurd how? An internal contradiction somehow? Unimportant for some reason? Impossible for some reason?
How can I be confident you aren't committing the fallacy of collecting a bunch of events and saying that is sufficient to serve as a cohesive explanation? No offense intended, but the comment above has many of the qualities of a classic rant.
If I'm wrong, perhaps you could elaborate? If I'm not wrong, maybe you could reconsider?
Don't forget that alignment research has existed longer than OpenAI. It would be a stretch to claim that the original AI safety researchers were using the pretexts you described -- I think it is fair to say they were involved because of genuine concern, not because it was a trendy or self-serving thing to do.
Some of those researchers and people they influenced ended up at OpenAI. So it would be a mistake or at least an oversimplification to claim that AI safety is some kind of pretext at OpenAI. Could it be a pretext for some people in the organization, to some degree? Sure, it could. But is it a significant effect? One that fits your complex narrative, above? I find that unlikely.
Making sense of an organization's intentions requires a lot of analysis and care, due to the combination of actors and varying influence.
There are simpler, more likely explanations, such as: AI safety wasn't a profit center, and over time other departments in OpenAI got more staff, more influence, and so on. This is a problem, for sure, but there is no "pearl clutching pretext" needed for this explanation.
Are you saying these so-called simple intentions are the only factors in play? Surely not.
Are you putting forth a theory that we can test? How well do you think your theory works? Did it work for Enron? For Microsoft? For REI? Does it work for every organization? Surely not perfectly; therefore, it can't be as simple as you claim.
Making a simplification and calling it "simple" is an easy thing to do.
A lot of people got screwed along the way
whether people should be able to hold on to that billion is a different question
Having these discussions in this current cultural moment is difficult. I'm no lover of billionaires, but to say that every billionaire screwed people over relies on esoteric interpretations of value and who produces it. These interpretations (like the labor-theory of value) are alien to the vast majority of people.
the wonderful thing any capitalism is that you can absolve yourself of guilt by having someone else do your dirty work for you. are you so sure every single seamstress that made clothes and stuffed animals, and the workers at the toy factories, and every single person involved with the making of the movies for the Harry Potter deals she licensed her work to were well compensated and treated well? that's not directly on her, but at least some of her money comes from there
It’s a ruse - it’s a con - it’s an accounting trick. It’s the foundation of capitalism
If I start a bowling pin production company and own 100% of it, then whatever pins I sell all of the results go to me
Now let say I want to expand my thing (that’s its own moral dilemma we won’t get into), so I promise a person with more money than they need to support their own life, to give me money in exchange for some of the future revenue produced, let’s say 10%
So now you have two people requiring payment - a producer and an “investor” so you’re already in the hole and now it’s 90% and 10%
You use that money to hire people to work in your potemkin dictatorship, with demands on proceeds now on some timeline (note conversion date, next board meeting etc)
So now you hire 10 people, how much of the company do they own? Well that’s totally up to whatever the two owners want including 0%
But let’s say it’s a typical venture deal, so 10% option pool for employees (and don’t forget the 4 year vest, cause we can’t have them mobile can we) which you fill up.
At the end of the four years you now have:
1 80% owner 1 10% owner 10 1% owners
Did the 2 people create 90% of the value of the company?
Only in capitalist math does that hold and in fact the only math capitalists do is the following:
“Well they were free to sign or not sign the contract”
Ignoring the reality of the world based on a worldview of greed that dominated the world to such an extent that it was considered “normal”
Luckily we’re starting to see the tide change
Idk. Folks much smarter than I seem worried so maybe I should be too but it just seems like such a long shot.
So yes, the insiders very likely know a thing or two that the rest of us don’t.
The most obvious reason is costs - if it costs many millions to train foundation models, they don't have a ton of experiments sitting around on a shelf waiting to be used. They may only get 1 shot at the base-model training. Sure productization isn't instant, but no one is throwing out that investment or delaying it longer than necessary. I cannot fathom that you can train an LLM at like 1% size/tokens/parameters to experiment on hyper parameters, architecture, etc and have a strong idea on end-performance or marketability.
Additionally, I've been part of many product launches - both hyped up big-news-events and unheard of flops. Every time, I'd say that 25-50% of the product is built/polished in the mad rush between press event and launch day. For an ML Model, this might be different, but again see above point.
Sure products may be planned month/years out, but OpenAI didn't even know LLMs were going to be this big a deal in May 2022. They had GPT-2 and GPT-3 and thought they were fun toys at that time, and had an idea for a cool tech demo. I think that OpenAI (and Google, etc) are entirely living day-to-day with this tech like those of us on the outside.
I agree, and they are also living in a group-think bubble of AI/AGI hype. I don't think you'd be too welcome at OpenAI as a developer if you didn't believe they are on the path to AGI.
If you've been working on AI, you've seen everything go up and to the right for a while - who really benefits from pointing out that a slowdown is occurring? Who is incentivized to talk about how the benefits from scaling are slowing down or the publicly available internet-scale corpuses are running out? Not anyone who trains models and needs compute, I can tell you that much. And not anyone who has a financial interest in these companies either.
What we're going to see over next year seems mostly pretty obvious - a lot of productization (tool use, history, etc), and a lot of efforts with multimodality, synthetic data, and post-training to add knowledge, reduce brittleness, and increase benchmark scores. None of which will do much to advance core intelligence.
The major short-term unknown seems to be how these companies will be attempting to improve planning/reasoning, and how successful that will be. OpenAI's Schulman just talked about post-training RL over longer (multi-reasoning steps) time horizons, and another approach is external tree-of-thoughts type scaffolding. These both seem more about maximizing what you can get out of the base model rather than fundamentally extending it's capabilities.
That being said, the GP you’re talking about made no such statement whatsoever.
IMO, we should pause this for now and put these resources (human and capital) towards reducing the impact of global warming.
https://www.quaise.energy/
1. When do you predict catastrophic global warming/climate change? How do you define "catastrophic"? (Are you pegging to an average temperature increase? [1])
2. When do you predict AGI?
How much uncertainty do you have in each estimate? When you stop and think about it, are you really willing to wager that (1) will happen before (2)? You think you have enough data to make that bet?
[1] I'm not an expert in the latest recommendations, but I see that a +2.7°F increase over preindustrial levels by 2100 is a target by some: https://news.mit.edu/2023/explained-climate-benchmark-rising...
The vast majority of datacenters currently in production will be entirely powered by carbon free energy. From best to worst:
1. Meta: 100% renewable
2. AWS: 90% renewable
3. Google: 64% renewable with 100% renewable energy credit matching
4. Azure: 100% carbon neutral
[1]: https://sustainability.fb.com/energy/
[2]: https://sustainability.aboutamazon.com/products-services/the...
[3]: https://sustainability.google/progress/energy/
[4]: https://azure.microsoft.com/en-us/explore/global-infrastruct...
If imaginary cloud provider "ZFQ" uses 10MW of electricity on a grid and pays for it to magically come from green generation, that means 10MW of other loads on the grid were not powered by green energy, or 10MW of non-green power sources likely could have been throttled down/shut down.
There is no free lunch here; "we buy our electricity from green sources" is greenwashing bullshit.
Even if they install solar on the roofs and wind turbines nearby - that's still electrical generation capacity that could have been used for existing loads. By buying so many solar panels in such quantities, they affect availability and pricing of all those components.
The US, for example, has about 5GW of solar manufacturing capacity per year. NVIDIA sold half a million H100 chips in one quarter, each of which uses ~350W, which means in a year they're selling enough chips to use 700MW of power. That does not include power conversion losses, distribution, cooling, and the power usage of the host systems, storage, networking, etc.
And that doesn't even get into the water usage and carbon impact of manufacturing those chips; the IC industry uses a massive amount of water and generates a substantial amount of toxic waste.
It's hilarious how HN will wring its hands over how much rare earth metals a Prius has and shipping it to the US from Japan, but ask about the environmental impacts of AI and it's all "pshhtt, whatever".
We do not have that. The cost of energy is mis-priced, although we are limping our way to fixing that.
Paying the likely fair cost for our goods, will probably kill a lot of current industries - while others which are currently viable, will become viable.
Who gets decide what the real impact price of energy is? That is not easily defined and well debated.
Markets are our super computers. Human behavior is the empirical evidence of the choices people will make Given specific incentives.
No. Renewable energy capacity is often built out specifically for datacenters.
> Even if they install solar on the roofs and wind turbines nearby - that's still electrical generation capacity that could have been used for existing loads.
No. This capacity would never never have been built out to begin with if it was not for the data center.
> By buying so many solar panels in such quantities, they affect availability and pricing of all those components.
No. Renewable energy gets cheaper with scale, not more expensive.
> which means in a year they're selling enough chips to use 700MW of power.
There are contracts for renewal capacity to be built out or well into the gigawatts. Furthermore, solar is not the only source of renewable energy. Finally, nuclear energy is also often used.
> the IC industry uses a massive amount of water
A figurative drop in the bucket.
> It's hilarious how HN will wring its hands
HN is not a monolith.
I agree with a majority of points you made. Exception is to this
> A figurative drop in the bucket.
Fresh water sources are limited. Fabs water demands and pollution are high impact.
Calling a drop in the bucket comes in the weasel words category.
We still need fabs, because we need chips. Harm will be done here. However, that is a cost we, as a society, will choose to pay.
Not fully accurate. Indeed there is renewable energy that is produced exclusively for the datacenter. But it is challenging to rely only on renewable energy (because it is intermittent and electricity is hard to store at scale so often you need to consume electricity when produced). So what happens in practice is that the electricity that does not come from dedicated renewable capacity is coming from the grid/network. What companies do is that they invest in renewable capacity in the network so that "the non renewable energy that they consume at time t (because not enough renewable energy available at that moment) is offsetted by someone else consuming renewable energy later". What I am saying here is not pure speculation, look at the link to meta website, they are saying themselves that this is what they are doing
What are your timelines here? "Catastrophic" is vague but I'd put the climate change meaningfully affecting the quality of life of average westerner at end of century, while AGI could be before the middle of the century.
Weather is not climate, as everyone is so careful to point out during cold waves.
Dane Wiginton (https://www.instagram.com/DaneWigington) is the founder of GeoengineerWatch.org as a very deep resource.
They have a free documentary called "The Dimming" you can watch on YouTube: https://www.youtube.com/watch?v=rf78rEAJvhY
In the documentary it includes credible witness testimonies such as politicians including a previous Minister of Defense for Canada; multiple states in the US have ban the spraying now - with more to follow, and the testimony and data provided there will be arguably be the most recent.
Here's a video on a "comedy" show from 5 years ago - there is a more recent appearance but I can't find it - in attempt to make light of it, without having an actual discussion with critical thinking or debate so people can be enlightened with the actual problems and potential problems and harms it can cause, to keep them none the wiser - it's just propaganda while trying to minimize: https://www.youtube.com/watch?v=wOfm5xYgiK0
A few of the problems cloud seeding will cause: - flooding in regions due to rain pattern changes - drought in areas due to rain pattern changes - cloud cover (amount of sun) changes crop yields - this harms local economies of farmers, impacting smaller farming operations more who's risk isn't spread out - potentially forcing them to sell or go into savings or go bankrupt, etc.
There are also very serious concerns/claims made of what exactly they are spraying - which includes aluminium nanoparticles, which can/would mean: - at a certain soil concentration of aluminium plants stop bearing fruit, - aluminium is a fire accelerant and so forest fires will then 1) more easily catch, and 2) more easily-quickly spread due to their increased intensity
Of course discussion on this is heavily suppressed in the mainstream, instead of having deep-thorough conversation with actual experts to present their cases - the label of conspiracy theorists or the idea of "detached from reality" are people's knee-jerk reactions often; and where propaganda can convince them of the "save the planet" narrative, which could also be a cover story for those toeing the line following orders supporting potentially very nefarious plans - doing it blindly because they think they're helping fight "climate change."
There are plenty of accounts on social media that are keeping track of and posting daily of the cloud seeding operations: https://www.instagram.com/p/CjNjAROPFs0/ - a couple testimonies.
If there is a top secret Manhattan Project for "climate change" - then someone's very likely pulling a fast one over everyone toeing that line, someone who has ulterior motives, misleading people to do their bidding.
But sure, fair question - a public discussion would allow actual experts to discuss the merits of what they're doing, and perhaps find a better solution than what has gained traction.
We have surpassed the 1.5°C goal and are on track towards 3.5°C to 5°C. This accelerates the climate change timeline so that we'll see effects postulated for the end of the century in about ~20 years.
Likewise, the cloud seeding they seem to be doing nearly worldwide now - the cloud formations from whatever they're spraying - are artificially changing weather patterns, and so a lot of the weather "anomalies" or unexpected-unusual weather-temperatures could very easily be because of those shenanigans; it could very easily be as a method to manufacture consent with the general population.
Similarly with the arson forest fires in Canada last summer, something like 90%+ of them were arson + a few years prior some of the governments in the prairie provinces (e.g. hottest and dryest) gutted their forest firefighting budgets; interesting behaviour considering if they're expecting more things to get hotter-dryer, you'd add to the budget, not take away from it, right?
I'm sorry, do you have a source for that claim? You seem to dismiss the video without any evidence.
How much airspace of geographic area do you need access to in order to cloud seeds in other parts of the world though?
I haven't looked but perhaps GeoengineeringWatch.org has resources and has kept track of that?
That's easy, we just need to make meatspace people stupider. Seems to be working great so far.
Honestly? I'm not too worried
We've seen how the google employee that was "seeing a conscience" (in what was basically GPT-2 lol) was a nothing burger
We've seen other people in "AI Safety" overplay their importance and hype their CV more than actually do any relevant work. (Usually also playing the diversity card)
So, no, AI safety is important but I see it attracting the least helpful and resourceful people to the area.
That has proven to be a mistake
What about Geoffrey Hinton? Stuart Russell? Dario Amodei?
Also exceptions to your model?
I’m pretty sure if Jan came to believe safety research wasn’t needed he would’ve just said that. Instead he said the actual opposite of that.
Why don’t you just answer the question? It’s a question about how these datapoints fit into your model.
Given the model is probabilistic and does many things in parallel, its output can be understood as a mixture, e.g. 30% trash, 60% rehashed training material, 10% reasoning.
People probe model in different ways, they see different results, and they make different conclusions.
E.g. somebody who assumes AI should have impeccable logic will find "trash" content (e.g. incorrectly retrieved memory) and will declare that the whole AI thing is overhyped bullshit.
Other people might call model a "stochastic parrot" as they recognize it basically just interpolates between parts of the training material.
Finally, people who want to probe reasoning capabilities might find it among the trash. E.g. people found that LLMs can evaluate non-trivial Python code as long as it sends intermediate results to output: https://x.com/GrantSlatton/status/1600388425651453953
I interpret "feel the AGI" (Ilya Sutskever slogan, now repeated by Jan Leike) as a focus on these capabilities, rather than on mistakes it makes. E.g. if we go from 0.1% reasoning to 1% reasoning it's a 10x gain in capabilities, while to an outsider it might look like "it's 99% trash".
In any case, I'd rather trust intuition of people like Ilya Sutskever and Jan Leike. They aren't trying to sell something, and overhyping the tech is not in their interest.
Regarding "missing something really critical", it's obvious that human learning is much more efficient than NN learning. So there's some algorithm people are missing. But is it really required for AGI?
And regarding "It cannot reason" - I've seen LLMs doing rather complex stuff which is almost certainly not in the training set, what is it if not reasoning? It's hard to take "it cannot reason" seriously from people
The whole industry at this point is acting like the tobacco industry back when they first started getting in hot water. No doubt the prophecies about imminent AGI will one day look to our descendents exactly like filters on cigarettes. A weak attempt to prevent imminent regulation and reduced profitability as governments force an out of control industry to deal with the externalities involved in the creation of their products.
If it wasn't abundantly clear...I agree with you that AGI is infinitely far away. Its the damage that's going to be caused by sociopaths (Sam Altman at the top of the list) in attempting to justify the real things they want (money) in their march towards that impossible goal that concerns me.
"Meanwhile what they have created is just a very impressive hot water bottle that turns a crank."
"Meanwhile what they have created is just a very impressive rock where neutrons hit other neutrons."
The point isn't how it works, the point is what it does.
Nobody defines what they’re trying to do as “useful AI” since that’s a much more weasily target, isn’t it?
https://nitter.poast.org/janleike/status/1791498174659715494
I think it may time for something like this: https://www.openailetter.org/
1)OpenAI wouldn't want the negative PR of pursuing legal action against someone top in their field; his peers would take note of it and be less willing to work for them.
2)The stuff he signed was almost certainly different from what rank and file signed, if only because he would have sufficient power to negotiate those contracts.
Large language models are not "smart". They do not have thought. They don't have intelligence despite the "AI" moniker, etc.
They vomit words based off very fancy statistics.
There is no path from that to "thought" and "intelligence."
Delusional.
Fucking monkeys.
Individualistic
No body depends on you, I hope
moral stands are never free, but they are freeing.
0. https://en.wikipedia.org/wiki/Civil_Disobedience_(Thoreau)
I feel that this particular case is just another reminder of that, and now would make me require a preemptory “no equity clawbacks” clause in any contract.
Working for a startup is inherently risky, but it’s not gambling because in gambling you can estimate the odds, and unlike gambling the odds cannot be changed after you win. Any employment contract that does not allow equity cash out at the price from the last funding round, or allows take backs, is worse than gambling, and founders that believe contracts that don’t provide those guarantees are reasonable are likely malicious and intending on doing that in future.
I do not understand a mentality that says “as a founder I should be able to get money out of the business but the people who work for me, who are also taking significant risk and below market compensation should not be permitted to do that”
https://news.ycombinator.com/item?id=38342850
My guess would be that YC founders like sama have some sort of special power to slap down comments that they feel are violating HN discussion guidelines.
It reeks of a scammer's mentality.
You can't just sign a contract and then not uphold your end of the bargain after you've got the benefit you want. You'll (rightfully) get sued.