I expect some amount of brain drain as researchers interested in publishing go elsewhere. That assumes other companies maintain an open publishing policy. I expect at least some will as it serves their interests (Meta for example).
I also think that these companies are realizing that at some stage, if AI keeps becoming more powerful, it may actually jeopardize their business, it's quite a conundrum for everyone.
Just as employees are feeling afraid for their jobs, I'm sure some of these companies, probably even Open AI themselves feel similar.
Like it's fun to release these models and keep the secret sauce secret, but at some stage, Open AI will probably be breached or someone will leak information about the architecture and weights, or both, and then it's kind of game over from there.
Sounds like this is the end of DeepMind as we know it. The move will only push the last remnants of DeepMind out of Google, as there are no leverage or incentive to stay.
Makes a lot of sense. Given that AI has gone commercial and might be the dominant technology of this/next decade, this makes a lot of sense.
I doubt there will be a big exodus. Where can researchers go who has both deep pockets for salary and infra and will let them publish the same way google was allowing.
OpenAI is anyways not publishing anything (see the GPT-4 pdf which is so void of any details).
This seems like it will result in a much more costly on-boarding process new employees.
Currently, if you hire some freshly minted phd to work on ML stuff, chances are they have read most the specialty's relevant cutting-edge research. Moving forward these same phds may have to spend months learning your fancy model architectures just to be somewhat useful. Hopefully at the end of that extended on-boarding, google doesn't find out their new hire is a complete dud. I guess if you have money, burn it.
Plus, if I spend years working on research at Google only to be unable to take those skills with me to my next job... well I wouldn't say yes to that job in the first place. Why not put a big sign on the door that proudly reads "lifers only"?
This is not surprising but at the same time, sad news for the community.
My thought is the following:
I think this is partly thanks to closedAI (openAI) but also partly Googles fault.
Google underestimated the Transformer model, closedAI did not and made something revolutionary with it, with little contribution back to the community (for example how little details they have given out regarding GPT-4).
Google realized this mistake and is now trying to catch up. They (Google) learned their lesson now.
My issue with this going forward is that Google might release some other powerful models / techniques as well and might underestimate them too, but this time, they will just collect dust somewhere in the dark.
Yea agreed. Thanks to OpenAI. Since OpenAI commercialized the technique but did not give back to the community, why should Google continue to publish their cutting edge research?
Sadly, that's the new reality now. Since the cat is out of the box, AI openness is no more. ClosedAI killed it the moment they decided to go chase the bag. The scary part is everything from here will be ruled by money before anything else.
Oddly enough, it may also have the reverse effect.
If the academic crowd stops relying on corporations and promote truly open/scalable tools, then this can start a virtuous cycle that promotes truly open solutions.
Having said that, I wouldn't hold my breath with the current state of academia though.
This is really sad. End of an golden era of AI research.
Within few years all major companies will stop producing fundamental research. We are going to get lot of marketing material masquerading as research paper ( looking at you GPT4). Thanks ClosedAI ( OpenAI ) for brining so much openness in the field /s
Odd to see this the day after the appearance of what's supposed to be a "leaked" memo from inside Google advocating the opposite course -- claiming that the outsider crowd playing with (mostly) LLaMa derivatives is getting good results, and that the best thing for Google might be to try to engage. It claims, for example, that the open-source crowd, trying to find cheaper training methods using smaller, more targeted data sets -- because they have to -- has found ones that work as well, by some metrics, as anything that the groups with huge compute clusters. And that because they work as well, with less resources, they can subsequently explore alternatives a whole lot quicker than anything involving training new base models from scratch.
So the lesson they could only take from their failings is to limit publishing instead of capitalizing on new techs? Almost every if not all authors of "Attention is all you need" left Google before ChatGPT broke the world. If the paper was internal I suspect they would have probably left even earlier, taken their ideas elsewhere and still beaten Google to it. If everyone stayed hush hush, Google might not even know what gave other's an edge. There's no secret to AI as we know it today, deep learning, RL, transformers, LOTS and LOTS of data, plus lots and lots of training.
This is ending soon. Even according to sam altman from ai we have reached a limit on large model. We need new techniques like attention, transformer, layernorms
28 comments
[ 0.17 ms ] story [ 58.9 ms ] threadJust as employees are feeling afraid for their jobs, I'm sure some of these companies, probably even Open AI themselves feel similar.
Like it's fun to release these models and keep the secret sauce secret, but at some stage, Open AI will probably be breached or someone will leak information about the architecture and weights, or both, and then it's kind of game over from there.
This will be like many other industries:
Make 100k a year at this noble job and better the knowledge of mankind
vs
Make 500k a year at a soulsucking corp job, doing research to better the rate at which we get people to click adds by 1%
Guess which jobs many great minds would take? I would take #2. At the end of the day, I can't feed my family with goodwill and back pats.
https://news.ycombinator.com/item?id=35813322
I doubt there will be a big exodus. Where can researchers go who has both deep pockets for salary and infra and will let them publish the same way google was allowing.
OpenAI is anyways not publishing anything (see the GPT-4 pdf which is so void of any details).
Currently, if you hire some freshly minted phd to work on ML stuff, chances are they have read most the specialty's relevant cutting-edge research. Moving forward these same phds may have to spend months learning your fancy model architectures just to be somewhat useful. Hopefully at the end of that extended on-boarding, google doesn't find out their new hire is a complete dud. I guess if you have money, burn it.
Plus, if I spend years working on research at Google only to be unable to take those skills with me to my next job... well I wouldn't say yes to that job in the first place. Why not put a big sign on the door that proudly reads "lifers only"?
My thought is the following:
I think this is partly thanks to closedAI (openAI) but also partly Googles fault.
Google underestimated the Transformer model, closedAI did not and made something revolutionary with it, with little contribution back to the community (for example how little details they have given out regarding GPT-4).
Google realized this mistake and is now trying to catch up. They (Google) learned their lesson now.
My issue with this going forward is that Google might release some other powerful models / techniques as well and might underestimate them too, but this time, they will just collect dust somewhere in the dark.
If the academic crowd stops relying on corporations and promote truly open/scalable tools, then this can start a virtuous cycle that promotes truly open solutions.
Having said that, I wouldn't hold my breath with the current state of academia though.
The (supposed, unsourced) leak: https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...