It wasn't that OpenAI was open as in "open source" but rather that its stated mission was to research AI such that all could benefit from it (open), as well as to ensure that it could not be controlled by any one player, rather than to develop commercial products to sell and make a return on (closed).
How are they gonna pay for their compute costs to get the frontier? Seems hard to attract enough investment while almost explicitly promising no return.
I was actually expecting Apple to get their hands on Ilya. They also have the privacy theme in their branding, and Ilya might help that image, but also have the chops to catch up to OpenAI.
Perhaps. But also throwing more flops at it has long been Ilya’s approach so it would be surprising. Notice also the reference to scale (“scale in peace”).
But any model, no matter how safe it was in training, can still be prompt hacked, or fed dangerous information to complete nefarious tasks. There is no safe model by design. Not to mention that open weights models can be "uncensored" with ease.
This makes sense. Ilya can probably raise practically unlimited money on his name alone at this point.
I'm not sure I agree with the "no product until we succeed" direction. I think real world feedback from deployed products is going to be important in developing superintelligence. I doubt that it will drop out of the blue from an ivory tower. But I could be wrong. I definitely agree that superintelligence is within reach and now is the time to work on it. The more the merrier!
I have a strong intuition that chat logs are actually the most useful kind of data. They contain many LLM outputs followed by implicit or explicit feedback, from humans, from the real world, and from code execution. Scaling this feedback to 180M users and 1 trillion interactive tokens per month like OpenAI is a big deal.
Yeah, similar to how Google's clickstream data makes their lead in search self-reinforcing. But chat data isn't the only kind of data. Multimodal will be next. And after that, robotics.
That doesn't necessarily imply that chat logs are not valuable for creating AGI.
You can think of LLMs as devices to trigger humans to process input with their meat brains and produce machine-readable output. The fact that the input was LLM-generated isn't necessarily a problem; clearly it is effective for the purpose of prodding humans to respond. You're training on the human outputs, not the LLM inputs. (Well, more likely on the edge from LLM input to human output, but close enough.)
If brain without language would suffice, a single human could rediscover all we know on their own. But it's not like that, brains are feeble individually, only in societies we have cultural evolution. If humanity lost language and culture and start from scratch, it would take us another 300K years to rediscover what we lost.
But if you train a random-init LLM on the same data, it responds (almost) like a human on a diversity of tasks. Does that imply humans are just language models on two feet? Maybe we are also language modelling our way through life. New situation comes up, we generate ideas based on language, select based on personal experience, and then act and observe the outcomes to update our preferences in the future.
His idea that only corporations and governments should have access to this product. He doesn’t think people should have access even to ChatGPT or LLMs.
Goal is to build companies with evaluations of dozens, hundreds trillions of dollars and make sure only US government will have access to super intelligence to surpass other countries economy and military wise, ideally to solidify US hegemony and undermine other countries economies and progress towards super intelligence.
I mean who wouldn’t trust capitalists that are laying of people by thousands just to please investors or government that is “under-intelligent” and hasn’t brought anything but pain and suffering to other countries.
Personally I wouldn’t trust OpenAi to work on super intelligence - it can indeed cause mass extinction.
Government is completely different story they will specifically train AI to develop biological, chemical and weapons of mass destruction. Train it to strategize and plan on how to win war conflicts, social engineering and manipulations, hacking. And obviously will let it control drone planes and tanks, artillery. Give it access to satellites and so on.
Nothing can go wrong when jarheads are at work :). Maybe it will even find the trillions of dollars that Pentagon can’t find during every audit they can’t pass.
Cracked, especially in saying "cracked engineers", refers to really good engineers these days. It's cracked as in like broken in a good way, like too over-powered that it's unfair.
> I've seen footballers using it a lot over the years (Neymar, Messi etc) fwiw.
The word "craque" in Portuguese was borrowed from English with the "exceptional" definition and is commonly used. Given how many Brazilian players there are in the top leagues, I wouldn't be surprised if it was commonly used by football players in general.
I am not on the bleeding edge of this stuff. I wonder though: How could a safe super intelligence out compete an unrestricted one? Assuming another company exists (maybe OpenAI) that is tackling the same goal without spending the cycles on safety, what chance do they have to compete?
That’s the first step towards returning to candlelight. So it isn’t a step toward safe super intelligence, but it is a step away from any super intelligence. So I guess some people would consider that a win.
Not sure if you want to share the capitalist system with an entity that outcompetes you by definition. Chimps don't seem to do too well under capitalism.
You might be right, but that wasn't my point. Capitalism might yield a friendly AGI or an unfriendly AGI or some mix of both. Collectivism will yield no AGI.
One can already see the beginning of AI enslaving humanity through the establishment. Companies work on AI get more investment and those who don't gets kicked out of the game. Those who employ AI get more investment and those who pay humans lose confidence through the market. People lose jobs, get harshly low birth rates while AI thrives. Tragic.
So far it is only people telling AI what to do. When we reach the day where it is common place for AI to tell people what to do then we are possibly in trouble.
It is a trendy but dumbass tautology used by intellectually lazy people who think they are smart. Society is based upon capitalism therefore everything bad is the fault of capitalism.
People spending so much time thinking about the systems (the models) themselves, not enough about the system that builds the systems. The behaviors of the models will be driven by the competitive dynamics of the economy around them, and yeah, that's a big, big problem.
Honestly, what does it matter. We're many lifetimes away from anything. These people are trying to define concepts that don't apply to us or what we're currently capable of.
AI safety / AGI anything is just a form of tech philosophy at this point and this is all academic grift just with mainstream attention and backing.
This goes massively against the consensus of experts in this field. The modal AI researcher believes that "high-level machine intelligence", roughly AGI, will be achieved by 2047, per the survey below. Given the rapid pace of development in this field, it's likely that timelines would be shorter if this were asked today.
I am in the field. The consensus is made up by a few loudmouths. No serious front line researcher I know believes we’re anywhere near AGI, or will be in the foreseeable future.
So the researchers at Deepmind, OpenAI, Anthropic, etc, are not "serious front line researchers"? Seems like a claim that is trivially falsified by just looking at what the staff at leading orgs believe.
Apparently not. Or maybe they are heavily incentivized by the hype cycle. I'll repeat one more time: none of the currently known approaches are going to get us to AGI. Some may end up being useful for it, but large chunks of what we think is needed (cognition, world model, ability to learn concepts from massive amounts of multimodal, primarily visual, and almost entirely unlabeled, input) are currently either nascent or missing entirely. Yann LeCun wrote a paper about this a couple of years ago, you should read it: https://openreview.net/pdf?id=BZ5a1r-kVsf. The state of the art has not changed since then.
LeCun has his own interests at heart, works for one of the most soulless corporations I know of, and devotes a significant amount of every paper he writes to citing himself.
Fair, ad hominems are indeed not very convincing. Though I do think everyone should read his papers through a lens of "having a very high h-index seems to be a driving force behind this man".
Moving on, my main issue is that it is mostly speculation, as all such papers will be. We do not understand how intelligence works in humans and animals, and most of this paper is an attempt to pretend otherwise. We certainly don't know where the exact divide between humans and animals is and what causes it, which I think is hugely important to developing AGI.
As a concrete example, in the first few paragraphs he makes a point about how a human can learn to drive in ~20 hours, but ML models can't drive at that level after countless hours of training. First you need to take that at face value, which I am not sure you should. From what I have seen, the latest versions of Tesla FSD are indeed better at driving than many people who have only driven for 20 hours.
Even if we give him that one though, LeCun then immediately postulates this is because humans and animals have "world models". And that's true. Humans and animals do have world models, as far as we can tell. But the example he just used is a task that only humans can do, right? So the distinguishing factor is not "having a world model", because I'm not going to let a monkey drive my car even after 10,000 hours of training.
Then he proceeds to talk about how perception in humans is very sophisticated and this in part is what gives rise to said world model. However he doesn't stop to think "hey, maybe this sophisticated perception is the difference, not the fundamental world model". e.g. maybe Tesla FSD would be pretty good if it had access to taste, touch, sight, sound, smell, incredibly high definition cameras, etc. Maybe the reason it takes FSD countless training hours is because all it has are shitty cameras (relative to human vision and all our other senses). Maybe linear improvements in perception leads to exponential improvement in learning rates.
Basically he puts forward his idea, which is hard to substantiate given we don't actually understand the source of human-level intelligence, and doesn't really want to genuinely explore (i.e. steelman) alternate ideas much.
Anyway that's how I feel about the first third of the paper, which is all I've read so far. Will read the rest on my lunch break. Hopefully he invalidates the points I just made in the latter 2/3rds.
I don't give much credit to the claim that it's impossible for current approaches to get us to any specific type or level of capabilities. We're doing program search over a very wide space of programs; what that can result in is an empirical question about both the space of possible programs and the training procedure (including the data distribution). Unfortunately it's one where we don't have a good way of making advance predictions, rather than "try it and find out".
It is in moments like these that I wish I wasn’t anonymous on here and could bet a 6 figure sum on AGI not happening in then next 10 years, which is how I define “foreseeable future”.
You disagreed that 2047 was reasonable on the basis that researchers didn't think it wouldn't happen in the foreseeable future, so your definition must be at least 23 years for consistency's sake
I'd be OK with that, too, if we adjusted the bet for inflation. This is, in a way, similar to fusion. We're at a point where we managed to ignite plasma for a few milliseconds. Predictions of when we're going to be able to generate energy have become a running joke. The same will be the case with AGI.
This could also just be an indication (and I think this is the case) that many Manifold betters believe the ARC AGI Grand Prize to be not a great test of AGI and that it can be solved with something less capable than AGI.
I don't understand how you got 2047. For the 2022 survey:
- "How many years until you expect: - a 90% probability of HLMI existing?"
mode: 100 years
median: 64 years
- "How likely is it that HLMI exists: - in 40 years?"
mode: 50%
median: 45%
And from the summary of results: "The aggregate forecast time to a 50% chance of HLMI was 37 years, i.e. 2059"
It'd be naive if it wasn't literally a standard point that is addressed and acknowledged as being a major part of the problem.
There's a reason OpenAI's charter had this clause:
“We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.””
How does that address the issue? I would have expected them to do that anyhow. Thats what a lot of businesses do: let another company take the hit developing the market, R and D, and supply chain, then come in with industry standardization and cooperative agreements only after the money was proven to be good in this space. See electric cars. Also they could drop that at any time. Remember when openAI stood for opensource?
Neither mention anything about open-source, although a later update mentions publishing work (“whether as papers, blog posts, or code”), which isn't exactly a ringing endorsement of “everything will be open-source” as a fundamental principle of the organization.
Since no one knows how to build an AGI, hard to say. But you might imagine that more restricted goals could end up being easier to accomplish. A "safe" AGI is more focused on doing something useful than figuring out how to take over the world and murder all the humans.
Assuming AGI works like a braindead consulting firm, maybe. But if it worked like existing statistical tooling (which it does, today, because for an actual data scientist and not aunt cathy prompting bing, using ml is no different than using any other statistics when you are writing your python or R scripts up), you could probably generate some fancy charts that show some distributions of cars produced under different scenarios with fixed resource or power limits.
In a sense this is what is already done and why ai hasn't really made the inroads people think it will even if you can ask google questions now. For the data scientists, the black magicians of the ai age, this spell is no more powerful than other spells, many of which (including ml) were created by powerful magicians from the early 1900s.
That is a very good question. In a well functioning democracy a government should apply a thin layer of fair rules that are uniformly enforced. I am an old man, but when I was younger, I recall that we sort of had this in the USA.
I don’t think that corporations left on their own will make safe AGI, and I am skeptical that we will have fair and technologically sound legislation - look at some of the anti cryptography and anti privacy laws raising their ugly heads in Europe as an example of government ineptitude and corruption. I have been paid to work in the field of AI since 1982, and all of my optimism is for AI systems that function in partnership with people and I expect continued rapid development of agents based on LLMs, RL, etc. I think that AGIs as seen in the Terminator movies are far into the future, perhaps 25 years?
This is not a trivial point. Selective pressures will push AI towards unsafe directions due to arms race dynamics between companies and between nations. The only way, other than global regulation, would be to be so far ahead that you can afford to be safe without threatening your own existence.
Similar to how law-abiding citizens turn on law-breaking citizens today or more old-fashioned, how religious societies turn on heretics.
I do think the notion that humanity will be able to manage superintelligence just through engineering and conditioning alone is naive.
If anything there will be a rogue (or incompetent) human who launches an unconditioned superintelligence into the world in no time and it only has to happen once.
The problem is the training data. If you take care of alignment at that level the performance is as good as an unrestricted one, except for things you removed like making explosives or ways to commit suicide.
But that costs almost as much as training on the data, hundreds of millions. And I'm sure this will be the new "secret sauce" by Microsoft/Meta/etc. And sadly nobody is sharing their synthetic data.
There's a reason OpenAI had this as part of its charter:
“We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.””
Glad to see Ilya is back in a position to contribute to advancing AI. I wonder how they are going to manage to pay the kinds of compensation packages that truly gifted AI researchers can make now from other companies that are more commercially oriented. Perhaps they can find people who are ideologically driven and/or are already financially independent. It's also hard to see how they will be able to access enough compute now that others are spending many billions to get huge new GPU data centers. You sort of need at least the promise/hope of future revenue in a reasonable time frame to marshall the kinds of resources it takes to really compete today with big AI super labs.
AI researchers at top firms make significantly more than software engineers at the same firms though (granted that the difference is likely not an order of magnitude in this case though).
Unless you know something I don’t, that’s not the case. It also makes sense, engineers are far more portable and scarcity isn’t an issue (many ML PhDs find engineering positions).
That is incredibly untrue and has been for years in the AI/ML space at many startups and at Amazon, Google, Facebook, etc. Good ML researchers have been making a good amount more for a while (source: I've hired both and been involved in leveling and pay discussions for years)
At the end game, a "non-safe" superinteligence seems easier to create, so like any other technology, some people will create it (even if just because they can't make it safe). And in a world with multiple superintelligent agents, how can the safe ones "win"? It seems like a safe AI is at inherent disadvantage for survival.
The current intelligences of the world (us) have organized their civilization in a way that the conforming members of society are the norm and criminals the outcasts. Certainly not a perfect system, but something along those lines for the most part.
That suggests that there are scenarios under which we survive. I'm not sure we'd like any of them, though "benign neglect" might be the best of a bad lot.
I disagree that civilization is organized along the lines of conforming and criminals. Rather, I would argue that the current intelligences of the world have primarily organized civilization in such a way that a small percentage of its members control the vast majority of all human resources, and the bottom 50% control almost nothing[0]
I would hope that AGI would prioritize humanity itself, but since it's likely to be created and/or controlled by a subset of that same very small percentage of humans, I'm not hopeful.
Academic compensation is different than what you’d find elsewhere on Hacker News. Likewise, academic performance is evaluated differently than what you’d expect as a software engineer. Ultimately, everyone cares about scientific impact so academic compensation relies on name and recognition far more than money. Personally, I care about the performance of the researchers (i.e., their publications), the institution’s larger research program (and their resources), the institution’s commitment to my research (e.g., fellowships and tenure). I want to do science for my entire career so I prioritize longevity rather than a quick buck.
I’ll add, the lack of compute resources was a far worse problem early in the deep learning research boom, but the market has adjusted and most researchers are able to be productive with existing compute infrastructure.
But wouldn't the focus on "safety first" sort of preclude them from giving their researchers the unfettered right to publish their work however and whenever they see fit? Isn't the idea to basically try to solve the problems in secret and only release things when they have high confidence in the safety properties?
If I were a researcher, I think I'd care more about ensuring that I get credit for any important theoretical discoveries I make. This is something that LeCun is constantly stressing and I think people underestimate this drive. Of course, there might be enough researchers today who are sufficiently scared of bad AI safety outcomes that they're willing to subordinate their own ego and professional drive to the "greater good" of society (at least in their own mind).
If you're working on superintelligence I don't think you'd be worried about not getting credit due to a lack of publications, of all things. If it works, it's the sort of thing that gets you in the history books.
Not sure about that. It might get Ilya in the history books, and maybe some of the other high profile people he recruits early on, but a junior researcher/developer who makes a high impact contribution could easily get overlooked. Whereas if that person can have their name as lead author on a published paper, it makes it much easier to measure individual contributions.
There is a human cognitive limit to the detail in which we can analyze and understand history.
This limit, just like our population count, will not outlast the singularity. I did the math a while back, and at the limit of available energy, the universe has comfortable room for something like 10^42 humans. Every single one of those humans will owe their existence to our civilization in general and the Superintelligence team in specific. There'll be enough fame to go around.
> compensation packages that truly gifted AI researchers can make now
I guess it depends on your definition of "truly gifted" but, working in this space, I've found that there is very little correlation between comp and quality of AI research. There's absolutely some brilliant people working for big names and making serious money, there's also plenty of really talented people working for smaller startups doing incredible work but getting paid less, academics making very little, and even the occasional "hobbyist" making nothing and churning out great work while hiding behind an anime girl avatar.
OpenAI clearly has some talented people, but there's also a bunch of the typical "TC optimization" crowd in there these days. The fact that so many were willing to resign with sama if necessary appears largely because they were more concerned with losing their nice compensation packages than any of their obsession with doing top tier research.
Definitely true of even normal software engineering; my experience has been the opposite of expectations, that TC-creep has infected the industry to an irreparable degree and the most talented people I've ever worked around or with are in boring, medium-sized enterprises in the midwest US or australia, you'll probably never hear of them, and every big tech company would absolutely love to hire them but just can't figure out the interview process to weed them apart from the TC grifters.
TC is actually totally uncorrelated with the quality of talent you can hire, beyond some low number that pretty much any funded startup could pay. Businesses hate to hear this, because money is easy to turn the dial up on; but most have no idea how to turn the dial up on what really matters to high talent individuals. Fortunately, I doubt Ilya will have any problem with that.
I have also worked in multiple different sized companies, including FAANG, and multiple countries. My assessment is that FAANGs tend to select for generally intelligent people who can learn quickly and adapt to new situations easily but who nowadays tend to be passionless and indifferent to anything but money and prestige. Personally I think passion is the differentiator here, rather than talent, when it comes to doing a good job. Passion means caring about your work and its impact beyond what it means for your own career advancement. It means caring about building the best possible products where “best” is defined as delivering the most value for your users rather than the most value for the company. The question is whether big tech is unable to select for passion or whether there are simply not enough passionate people to hire when operating at FAANG scale. Most likely it’s the latter.
So I guess I agree with both you and the parent comment somewhat in that in general the bar is higher at FAANGs but at the same time I have multiple former colleagues from smaller companies who I consider to be excellent, passionate engineers but who cannot be lured to big tech by any amount of money or prestige (I’ve tried). While many passionless “arbitrary metric optimizers” happily join FAANGs and do whatever needs to be done to climb the ladder without a second thought.
I sort of agree and disagree. I wouldn't agree with the idea that most FAANG engineers are not passionate by nature about their work.
What I would say is that the bureaucracy and bullshit one has to deal with makes it hard to maintain that passion and that many end up as TC optimizers in the sense that they stay instead of working someplace better for less TC.
That said, I am not sure how many would make different choices. Many who join a FAANG company don't have the slightest inkling of what it will be like and once they realize that they are tiny cog in a giant machine it's hard to leave the TC and perks behind.
Two people I knew recently left Google to join OpenAI. They were solid L5 engineers on the verge of being promoted to L6, and their TC is now $900k. And they are not even doing AI research, just general backend infra. You don't need to be gifted, just good. And of course I can't really fault them for joining a company for the purpose of optimizing TC.
When I looked into it and talked to some hiring managers, the big names were offering cash comp similar to total comp for big tech, with stock (sometimes complicated arrangements that were not options or RSUs) on top of that. I’m talking $400k cash for a senior engineer with equity on top.
Because op’s usage of base implies base + stock. including a place where base = total comp is really misleading and is just being unnecessarily pedantic about terminology.
OP is correct that a base cash of 400k is truly rare if you’re talking about typical total comp packages where 50% is base and 50% is stock.
I don’t know what point you’re trying to make other than being super pedantic. This was a discussion about how OpenAI’s base of 400k is unique within the context of a TCO in the 800-900k range. It is. That quantfi and Netflix offer similar base because that’s also their TCO is a silly argument to make.
> This was a discussion about how OpenAI’s base of 400k is unique within the context of a TCO in the 800-900k range.
That's not how I interpret the conversation.
I see a claim that 900k is a BS number, a counterargument that many big AI companies will give you 400k of that in cash so the offers are in fact very hot, then a claim that only finance offers 400k cash, and a claim that netflix offers 400k cash.
I don't see anything that limits these comparisons to companies with specific TCOs.
Even if the use of the word "base" is intended to imply that there's some stock, it doesn't imply any particular amount of stock. But my reading is that the word "base" is there to say that stock can be added on top.
You're the one being pedantic when you insist that 400k cash is not a valid example of 400k cash base.
Notice how the person being replied to looked at the Netflix example and said "Okay that's true". They know what they meant a lot better than you do.
ok so the conversation starts out with 900k TCO with 400k in cash, a claim that that’s BS and then morphs into a discussion about a TCO of 400k all cash being an example of equivalent compensation to OpenAI packages?
Nobody said it was equivalent. The subdiscussion was about whether you can even get that much cash anywhere else, once TCO got pulled apart into cash and stock to be compared in more detail.
Again, the person that made the original claim about where you can get "400k cash base" accepted the Netflix example. Are you saying they're wrong about what they meant?
It's amazing to me how many people are willing to just say the first thing that comes to their head while knowing they can be fact-checked in a heartbeat.
> Note at offer time candidates do not know how many PPUs they will be receiving or how many exist in total. This is important because it’s not clear to candidates if they are receiving 1% or 0.001% of profits for instance. Even when giving options, some startups are often unclear or simply do not share the total number of outstanding shares. That said, this is generally considered bad practice and unfavorable for employees. Additionally, tender offers are not guaranteed to happen and the cadence may also not be known.
> PPUs also are restricted by a 2-year lock, meaning that if there’s a liquidation event, a new hire can’t sell their units within their first 2 years. Another key difference is that the growth is currently capped at 10x. Similar to their overall company structure, the PPUs are capped at a growth of 10 times the original value. So in the offer example above, the candidate received $2M worth of PPUs, which means that their capped amount they could sell them for would be $20M
> The most recent liquidation event we’re aware of happened during a tender offer earlier this year. It was during this event that some early employees were able to sell their profit participation units. It’s difficult to know how often these events happen and who is allowed to sell, though, as it’s on company discretion.
As a community we should stop throwing numbers around like this when more than half of this number is speculative. You shouldn't be able to count it as "total compensation" unless you are compensated.
Word in town is [1] openai "plans" to let employees sell "some" equity through a "tender process" which ex-employees are excluded from; and also that openai can "claw back" vested equity, and has used the threat of doing so in the past to pressure people into signing sketchy legal documents.
I would definitely discount OpenAI equity compared to even other private AI labs (i.e. Anthropic) given the shenanigans, but they have in fact held 3 tender offers and former employees were not, as far as we know, excluded (though they may have been limited to selling $2m worth of equity, rather than $10m).
> Word on town is OpenAI folks heavily selling shares in secondaries in 100s of millions
OpenAI heavily restricts the selling of its "shares," which tends to come with management picking the winners and losers among its ESOs. Heavily, heavily discount an asset you cannot liquidate without someone's position, particularly if that person is your employer.
Are you seriously asking how the most talented AI researcher of the last decade will be able to recruit other researchers? Ilya saw the potential of deep learning way before other machine learning academics.
My guess is they will work on a protocol to drive safety with the view that every material player will use / be regulated and required to use that could lead to a very robust business model
I assume that OpenAI and others will support this effort and the comp / training / etc and they will be very well positioned to offer comparable $$$ packages, leverage resources, etc
Generally, the mindset that makes the best engineers is an obsession with solving hard problems. Anecdotally, there's not a lot of overlap between the best engineers I know and the best paid engineers I know. The best engineers I know are too obsessed with solving problems to be sidetracked the salary game. The best paid engineers I know are great engineers, but the spend a large amount of time playing the salary game, bouncing between companies and are always doing the work that looks best on a resume, not the best work they know how to do.
Great analysis, but you're missing two key factors IMO:
1. People who honestly think AGI is here aren't thinking about their careers in the typical sense at all. It's sorta ethical/"ideological", but it's mostly just practical.
2. People who honestly think AGI is here are fucking terrified right now, and were already treating Ilya as a spiritual center after Altman's coup (quite possibly an unearned title, but oh well, that's history for ya). A rallying cry like this -- so clearly aimed at the big picture instead of marketing they don't even need CSS -- will be seen as a do-or-die moment by many, I think. There's only so much of "general industry continues to go in direction experts recommend against; corporate consolidation continues!" headlines an ethical engineer can take before snapping and trying to take on Goliath, odds be damned
the vision here is amazing, but I can't help but feel a bit uncertain about basing a "safe super intelligence" lab in an apartheid state that is currently genociding its indigenous population using variants of the very same technology
Well, yes¹: if it actually contributes to intelligent action, this is exactly the goal - you nailed it. Safer, more sensible action for individuals, communities, and statal entities. "Less mistakes in the future" - that would be a good motto.
(¹: whether basing a [successful in substance] "safe super intelligence" lab in an environment with imperfect record would improve safety.)
--
And now let us see if the sniper can come up with a good argument...
It does say they have a business model ("our business model means safety, security, and progress are all insulated from short-term commercial pressures"). I imagine it's some kind of patron model that requires a long-term commitment.
Prediction - the business model becomes an external protocol - similar to SSL - that the litany of AI companies working to achieve AGI will leverage (or be regulated to use)
From my hobbyist knowledge of LLMs and compute this is going to be a terrifically complicated problem, but barring a defined protocol & standard there's no hope that "safety" is going to be executed as a product layer given all the different approaches
Ilya seems like he has both the credibility and engineering chops to be in a position to execute this, and I wouldn't be suprised to see OpenAI / MSFT / and other players be early investors / customers / supporters
I like your idea. But on the other hand, training an AGI, and then having a layer on top “aligning” the AGI sounds super dystopian and good plot for a movie.
Poisoning Socrates was done because it was "good for society". I'm frankly even more suspicious of "good for society" than the average untrustworthy board of directors.
seriously? you're more worried about what your elected officials might legislate than what a board of directors whose job is to make profits go brrr at all costs, including poisoning the environment, exploiting people and avoiding taxes?
Didn't vast majority of elected officials vote for war in Iraq, Vietnam and Afghanistan?
Vast majority of elected officials are crooks that take millions from foreign interest groups (e.g AIPAC) and from corporations - and make laws in their favour.
weren't the us invited to defend the democratic government of Vietnam? weren't the taleban hiding al quaeda who attacked the US? didn't Saddam Hussein use chemical weapons against his own people? are you comparing all these people with Socrates?
Did the people want to participate in Vietnam? Get drafted so that they can die thousand of miles away from home?
> didn't Saddam Hussein use chemical weapons against his own people?
You can't argue that when US is providing weapons to Israel which are directly used to massacre thousands of innocents. That was just an excuse to the public
> are you comparing all these people with Socrates?
No, my point is that the government does not act in the interest of people or in the interest of upholding human rights.
At this point, all the computing power is concentrated among various companies such as Google, Facebook, Microsoft, Amazon, Tesla, etc.
It seems to me it would be much safer and more intelligent to create a massive model and distribute the benefits among everyone. Why not use a P2P approach?
In my area, internet and energy are insanely expensive and that means I'm not at all willing to share my precious bandwidth or compute just to subsidize someone generating Rule 34 porn of their favorite anime character.
I don't seed torrents for the same reason. If I lived in South Korea or somewhere that bandwidth was dirt cheap, then maybe.
There is a way to achieve load balancing, safety, and distribution effectively. The models used by Airbnb, Uber, and Spotify have proven to be generally successful. Peer-to-peer (P2P) technology is the future; even in China, people are streaming videos using this technology, and it works seamlessly. I envision a future where everyone joins the AI revolution with an iPhone, with both training and inference distributed in a P2P manner. I wonder why no one has done this yet.
>aiming to create a safe, powerful artificial intelligence system within a pure research organization that has no near-term intention of selling AI products or services.
Who is going to fund such a venture based on blind faith alone? Especially if you believe in the scaling hypothesis type of ai research where you spend billions on compute, this seems bound to fail once the AI hype dies down and raising money becomes a bit harder
> Building safe superintelligence (SSI) is the most important technical problem of our time.
Isn't this a philosophical/psychological problem instead? Technically it's solved, just censor any response that doesn't match a list of curated categories, until a technician whitelists it. But the technician could be confronted with a compelling "suicide song":
Maybe I'm just old and grumpy, but I can't help shake that the real most dangerous thing about AGI/ASI is centralization of its power (if it is ever possibly achieved).
> What exactly is "safe" in this context, can someone give me an eli5?
In practice it essentially means the same thing as for most other AI companies - censored, restricted, and developed in secret so that "bad" people can't use it for "unsafe" things.
Good Q. My understanding of "safe" in this context is a superintelligence that cannot escape its bounds. But that's not to say that's Ilya's definition.
1,144 comments
[ 2.6 ms ] story [ 461 ms ] threadI'm not sure I agree with the "no product until we succeed" direction. I think real world feedback from deployed products is going to be important in developing superintelligence. I doubt that it will drop out of the blue from an ivory tower. But I could be wrong. I definitely agree that superintelligence is within reach and now is the time to work on it. The more the merrier!
You can think of LLMs as devices to trigger humans to process input with their meat brains and produce machine-readable output. The fact that the input was LLM-generated isn't necessarily a problem; clearly it is effective for the purpose of prodding humans to respond. You're training on the human outputs, not the LLM inputs. (Well, more likely on the edge from LLM input to human output, but close enough.)
But if you train a random-init LLM on the same data, it responds (almost) like a human on a diversity of tasks. Does that imply humans are just language models on two feet? Maybe we are also language modelling our way through life. New situation comes up, we generate ideas based on language, select based on personal experience, and then act and observe the outcomes to update our preferences in the future.
I mean who wouldn’t trust capitalists that are laying of people by thousands just to please investors or government that is “under-intelligent” and hasn’t brought anything but pain and suffering to other countries.
Like the Tottenham Hotspurs owner??
If AGI doesn't coach them to trophies, nothing ever will.
https://deepmind.google/discover/blog/tacticai-ai-assistant-... / https://archive.is/wgJWu
If I didn't know that it was real, I would have thought it was parody.
> We are assembling a lean, cracked team of the world’s best engineers and researchers dedicated to focusing on SSI and nothing else.
Do you have to have a broken bone to join?
Apparently, grammatical nuances are not an area of focus for safety, unless they think that a broken team ("cracked") is an asset in this area.
Just realized - we even say "it's not all its cracked up to be" as a negative statement which would imply "cracked up" is positive.
The word "craque" in Portuguese was borrowed from English with the "exceptional" definition and is commonly used. Given how many Brazilian players there are in the top leagues, I wouldn't be surprised if it was commonly used by football players in general.
Racism, unsafe roads, hunger, bad weather, good weather, stubbing toes on furniture, etc.
Don't believe me?
See https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
Are there any non-capitalist utopias out there without any problems like this?
Is that the only solution here? We need to destroy billions of lives so that we can potentially prevent "unsafe" super intelligence?
Let me guess, your cure for cancer involves abolishing humanity?
Should we abolish governments when some random government goes bad?
Insufficiently regulated capitalism fails to account for negative externalities. Much like a Paperclip Maximising AI.
One could even go as far as saying AGI alignment and economic resource allocation are isomorphic problems.
From history, governments have done more physical harm (genocides, etc) than capitalist companies with advanced tech (I know Chiquita and Dow exist).
Even though I agree with your general point.
People spending so much time thinking about the systems (the models) themselves, not enough about the system that builds the systems. The behaviors of the models will be driven by the competitive dynamics of the economy around them, and yeah, that's a big, big problem.
AI safety / AGI anything is just a form of tech philosophy at this point and this is all academic grift just with mainstream attention and backing.
https://www.vox.com/future-perfect/2024/1/10/24032987/ai-imp...
He is far from the best person to follow on this.
Moving on, my main issue is that it is mostly speculation, as all such papers will be. We do not understand how intelligence works in humans and animals, and most of this paper is an attempt to pretend otherwise. We certainly don't know where the exact divide between humans and animals is and what causes it, which I think is hugely important to developing AGI.
As a concrete example, in the first few paragraphs he makes a point about how a human can learn to drive in ~20 hours, but ML models can't drive at that level after countless hours of training. First you need to take that at face value, which I am not sure you should. From what I have seen, the latest versions of Tesla FSD are indeed better at driving than many people who have only driven for 20 hours.
Even if we give him that one though, LeCun then immediately postulates this is because humans and animals have "world models". And that's true. Humans and animals do have world models, as far as we can tell. But the example he just used is a task that only humans can do, right? So the distinguishing factor is not "having a world model", because I'm not going to let a monkey drive my car even after 10,000 hours of training.
Then he proceeds to talk about how perception in humans is very sophisticated and this in part is what gives rise to said world model. However he doesn't stop to think "hey, maybe this sophisticated perception is the difference, not the fundamental world model". e.g. maybe Tesla FSD would be pretty good if it had access to taste, touch, sight, sound, smell, incredibly high definition cameras, etc. Maybe the reason it takes FSD countless training hours is because all it has are shitty cameras (relative to human vision and all our other senses). Maybe linear improvements in perception leads to exponential improvement in learning rates.
Basically he puts forward his idea, which is hard to substantiate given we don't actually understand the source of human-level intelligence, and doesn't really want to genuinely explore (i.e. steelman) alternate ideas much.
Anyway that's how I feel about the first third of the paper, which is all I've read so far. Will read the rest on my lunch break. Hopefully he invalidates the points I just made in the latter 2/3rds.
Separately, it's very clear that LLMs have "world models" in most useful senses of the term. Ex: https://www.lesswrong.com/posts/nmxzr2zsjNtjaHh7x/actually-o...
I don't give much credit to the claim that it's impossible for current approaches to get us to any specific type or level of capabilities. We're doing program search over a very wide space of programs; what that can result in is an empirical question about both the space of possible programs and the training procedure (including the data distribution). Unfortunately it's one where we don't have a good way of making advance predictions, rather than "try it and find out".
https://manifold.markets/JacobPfau/will-the-arcagi-grand-pri...
ENIAC was built in 1945, that's roughly a lifetime ago. Just think about it
There's a reason OpenAI's charter had this clause:
“We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.””
> Remember when openAI stood for opensource?
I surely don't, but maybe I missed it, can you show me?
https://web.archive.org/web/20151211215507/https://openai.co...
https://web.archive.org/web/20151213200759/https://openai.co...
Neither mention anything about open-source, although a later update mentions publishing work (“whether as papers, blog posts, or code”), which isn't exactly a ringing endorsement of “everything will be open-source” as a fundamental principle of the organization.
Even if you focus an AGI on producing more cars for example, it will quickly realize that if it has more power and resources it can make more cars.
In a sense this is what is already done and why ai hasn't really made the inroads people think it will even if you can ask google questions now. For the data scientists, the black magicians of the ai age, this spell is no more powerful than other spells, many of which (including ml) were created by powerful magicians from the early 1900s.
I don’t think that corporations left on their own will make safe AGI, and I am skeptical that we will have fair and technologically sound legislation - look at some of the anti cryptography and anti privacy laws raising their ugly heads in Europe as an example of government ineptitude and corruption. I have been paid to work in the field of AI since 1982, and all of my optimism is for AI systems that function in partnership with people and I expect continued rapid development of agents based on LLMs, RL, etc. I think that AGIs as seen in the Terminator movies are far into the future, perhaps 25 years?
Similar to how law-abiding citizens turn on law-breaking citizens today or more old-fashioned, how religious societies turn on heretics.
I do think the notion that humanity will be able to manage superintelligence just through engineering and conditioning alone is naive.
If anything there will be a rogue (or incompetent) human who launches an unconditioned superintelligence into the world in no time and it only has to happen once.
It's basically Pandora's box.
But that costs almost as much as training on the data, hundreds of millions. And I'm sure this will be the new "secret sauce" by Microsoft/Meta/etc. And sadly nobody is sharing their synthetic data.
“We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.””
This and safety techniques themselves can improve the performance of the hypothetical AGI.
RLHF was originally an alignment tool, but it improves llms significantly
And maybe you need to discover talent rather than buy talent from the previous generation.
i find the way people reason nowadays baffling
given the nature of their mission, this shouldn't be too terribly difficult; many gifted researchers do not go to the highest bidder
I like to think AGIs will decide to do that too.
1. Pests to eliminate 2. Benign neglect 3. Workers 4. Pets 5. Food
That suggests that there are scenarios under which we survive. I'm not sure we'd like any of them, though "benign neglect" might be the best of a bad lot.
I would hope that AGI would prioritize humanity itself, but since it's likely to be created and/or controlled by a subset of that same very small percentage of humans, I'm not hopeful.
[0] https://en.wikipedia.org/wiki/Wealth_inequality_in_the_Unite...
Perhaps this isn't a system we should be trying to emulate with a technology that promises to free us of our current inefficiencies or miseries.
I’ll add, the lack of compute resources was a far worse problem early in the deep learning research boom, but the market has adjusted and most researchers are able to be productive with existing compute infrastructure.
If I were a researcher, I think I'd care more about ensuring that I get credit for any important theoretical discoveries I make. This is something that LeCun is constantly stressing and I think people underestimate this drive. Of course, there might be enough researchers today who are sufficiently scared of bad AI safety outcomes that they're willing to subordinate their own ego and professional drive to the "greater good" of society (at least in their own mind).
This limit, just like our population count, will not outlast the singularity. I did the math a while back, and at the limit of available energy, the universe has comfortable room for something like 10^42 humans. Every single one of those humans will owe their existence to our civilization in general and the Superintelligence team in specific. There'll be enough fame to go around.
I guess it depends on your definition of "truly gifted" but, working in this space, I've found that there is very little correlation between comp and quality of AI research. There's absolutely some brilliant people working for big names and making serious money, there's also plenty of really talented people working for smaller startups doing incredible work but getting paid less, academics making very little, and even the occasional "hobbyist" making nothing and churning out great work while hiding behind an anime girl avatar.
OpenAI clearly has some talented people, but there's also a bunch of the typical "TC optimization" crowd in there these days. The fact that so many were willing to resign with sama if necessary appears largely because they were more concerned with losing their nice compensation packages than any of their obsession with doing top tier research.
TC is actually totally uncorrelated with the quality of talent you can hire, beyond some low number that pretty much any funded startup could pay. Businesses hate to hear this, because money is easy to turn the dial up on; but most have no idea how to turn the dial up on what really matters to high talent individuals. Fortunately, I doubt Ilya will have any problem with that.
In my anecdotal experience, I can only think of one or two examples of someone from the enterprise world who I would consider outstanding.
The overall quality of engineers is much higher at the FAANG companies.
So I guess I agree with both you and the parent comment somewhat in that in general the bar is higher at FAANGs but at the same time I have multiple former colleagues from smaller companies who I consider to be excellent, passionate engineers but who cannot be lured to big tech by any amount of money or prestige (I’ve tried). While many passionless “arbitrary metric optimizers” happily join FAANGs and do whatever needs to be done to climb the ladder without a second thought.
What I would say is that the bureaucracy and bullshit one has to deal with makes it hard to maintain that passion and that many end up as TC optimizers in the sense that they stay instead of working someplace better for less TC.
That said, I am not sure how many would make different choices. Many who join a FAANG company don't have the slightest inkling of what it will be like and once they realize that they are tiny cog in a giant machine it's hard to leave the TC and perks behind.
Everyone knows that openai TC is heavily weighted by ~~RSUs~~ options that themselves are heavily weighted by hopes and dreams.
Big names where? Inside of openai? What does that even mean?
The only place you can get 400k cash base for senior is quantfi
confident yet wrong
not only can you get that much at AI companies, netflix will also pay that much all cash - and that’s fully public info
Please show not tell
> netflix will also pay that much all cash
Okay that's true
That statement is false for the reasons I said. I’m not sure why your point matters to what I’m saying
OP is correct that a base cash of 400k is truly rare if you’re talking about typical total comp packages where 50% is base and 50% is stock.
That's not how I interpret the conversation.
I see a claim that 900k is a BS number, a counterargument that many big AI companies will give you 400k of that in cash so the offers are in fact very hot, then a claim that only finance offers 400k cash, and a claim that netflix offers 400k cash.
I don't see anything that limits these comparisons to companies with specific TCOs.
Even if the use of the word "base" is intended to imply that there's some stock, it doesn't imply any particular amount of stock. But my reading is that the word "base" is there to say that stock can be added on top.
You're the one being pedantic when you insist that 400k cash is not a valid example of 400k cash base.
Notice how the person being replied to looked at the Netflix example and said "Okay that's true". They know what they meant a lot better than you do.
Again, the person that made the original claim about where you can get "400k cash base" accepted the Netflix example. Are you saying they're wrong about what they meant?
It's amazing to me how many people are willing to just say the first thing that comes to their head while knowing they can be fact-checked in a heartbeat.
i suspect people commenting this don’t have a clue how PPU compensation actually works
> PPUs also are restricted by a 2-year lock, meaning that if there’s a liquidation event, a new hire can’t sell their units within their first 2 years. Another key difference is that the growth is currently capped at 10x. Similar to their overall company structure, the PPUs are capped at a growth of 10 times the original value. So in the offer example above, the candidate received $2M worth of PPUs, which means that their capped amount they could sell them for would be $20M
> The most recent liquidation event we’re aware of happened during a tender offer earlier this year. It was during this event that some early employees were able to sell their profit participation units. It’s difficult to know how often these events happen and who is allowed to sell, though, as it’s on company discretion.
https://www.levels.fyi/blog/openai-compensation.html
Edit:
I’m realizing we had the exact same conversation a month ago. It sounds like you have more insider information.
As a community we should stop throwing numbers around like this when more than half of this number is speculative. You shouldn't be able to count it as "total compensation" unless you are compensated.
The number is as real as someone else is willing to pay for them. Plenty of VCs willing to pay for it.
[1] https://www.cnbc.com/2024/06/11/openai-insider-stock-sales-a...
OpenAI heavily restricts the selling of its "shares," which tends to come with management picking the winners and losers among its ESOs. Heavily, heavily discount an asset you cannot liquidate without someone's position, particularly if that person is your employer.
the people i often have the most respect for.
(they mean total compensation)
HF is sort of big now. Stanford is well funded and they did PyReft.
Neither of these are even remotely big labs like what I’m discussing
I assume that OpenAI and others will support this effort and the comp / training / etc and they will be very well positioned to offer comparable $$$ packages, leverage resources, etc
I don't think SSI will struggle to raise money.
1. People who honestly think AGI is here aren't thinking about their careers in the typical sense at all. It's sorta ethical/"ideological", but it's mostly just practical.
2. People who honestly think AGI is here are fucking terrified right now, and were already treating Ilya as a spiritual center after Altman's coup (quite possibly an unearned title, but oh well, that's history for ya). A rallying cry like this -- so clearly aimed at the big picture instead of marketing they don't even need CSS -- will be seen as a do-or-die moment by many, I think. There's only so much of "general industry continues to go in direction experts recommend against; corporate consolidation continues!" headlines an ethical engineer can take before snapping and trying to take on Goliath, odds be damned
(¹: whether basing a [successful in substance] "safe super intelligence" lab in an environment with imperfect record would improve safety.)
--
And now let us see if the sniper can come up with a good argument...
Either the Facebook of this era has yet to present itself or it’s Alphabet/DeepMind
From my hobbyist knowledge of LLMs and compute this is going to be a terrifically complicated problem, but barring a defined protocol & standard there's no hope that "safety" is going to be executed as a product layer given all the different approaches
Ilya seems like he has both the credibility and engineering chops to be in a position to execute this, and I wouldn't be suprised to see OpenAI / MSFT / and other players be early investors / customers / supporters
Vast majority of elected officials are crooks that take millions from foreign interest groups (e.g AIPAC) and from corporations - and make laws in their favour.
> didn't Saddam Hussein use chemical weapons against his own people?
You can't argue that when US is providing weapons to Israel which are directly used to massacre thousands of innocents. That was just an excuse to the public
> are you comparing all these people with Socrates?
No, my point is that the government does not act in the interest of people or in the interest of upholding human rights.
It seems to me it would be much safer and more intelligent to create a massive model and distribute the benefits among everyone. Why not use a P2P approach?
I don't seed torrents for the same reason. If I lived in South Korea or somewhere that bandwidth was dirt cheap, then maybe.
Isn't this a philosophical/psychological problem instead? Technically it's solved, just censor any response that doesn't match a list of curated categories, until a technician whitelists it. But the technician could be confronted with a compelling "suicide song":
https://en.wikipedia.org/wiki/Gloomy_Sunday
Everyone just fiend-ing for their version of it.
If it's "taking over the world" safe, does it not mean that this is a part of AGI?
In practice it essentially means the same thing as for most other AI companies - censored, restricted, and developed in secret so that "bad" people can't use it for "unsafe" things.