"in which Microsoft is implanting machine intelligence"
I thought it was extremely adept change of referring to it as machine intelligence instead of artificial. There's nothing artificial about this intelligence, merely different. I think this will probably be increasingly more correct moving forward with additional advances.
They will not dominate anything cuz of the number of useless teams within all fighting each other about who should own it. Big tech is going the way of IBM and good riddance.
This. Some folks will spend their lives trying to find faster horses and build lighter carriages, and all is good and well until someone shows up with a car.
It will be interesting to see how long anyone can dominate. AI might actually result in actually the opposite. Meaning that the era of years and decades of dominance is not sustainable any longer.
Why would this be? Because AI allows you to copy your competitor in ways that was never possible before. It essentially becomes a skill and technology replicator. For example, look how Alpaca was able to somewhat replicate the multi million dollar models with $600 dollar cost.
I don't have the link, but someone estimated that training a Llama clone from scratch would cost around ~$80k. That's pocket change for most corporations and some individuals.
Yes, but you don't own that data, it's scraped off the Internet. Either anyone can scrape data and train an AI with it, or nobody can do that: there is no special privilege or advantage for OpenAI.
The Pile and Common Crawl are where everyone gets their data from.
I did lie a bit; OpenAI does have private datasets. For example, GPT-3 was trained on Internet data, but InstructGPT/ChatGPT was further finetuned through their own dataset of question/answer pairs. This was augmented in a really clever way: the initial set of valid completions was written by humans, but then OpenAI made more AI models to augment their training data before finetuning GPT-3 on it.
Even then, there's techniques for taking the output of one large language model and using it to finetune another. So just having a publicly-available AI chatbot is enough to get it to generate a dataset that you can then train another AI chatbot on.
I personally have been scraping Wikimedia Commons for public-domain imagery to train an art generator bot on; but this is mostly because I want an art generator that isn't full of pending copyright infringement lawsuits.
Interesting. I think the raw original source data would be good to have a copy of if possible and over time tools to curate and sample it should get better.
What I can’t get really pinned down is how much storage one needs to have your own copy of all of the most valuable raw data. At initial estimate it seemed like less than 1PB was enough for sure if focusing on text and not images/video. It even looked like maybe 200-400 TB might have been more than enough. I need to investigate more. I need numbers like 900GB for pile, then 2.5TB for a meta curated text only common crawl. So I’m assuming there is at least 10 of these in same ballpark that are important to have a copy of.
Any thoughts on the size requirements? Focusing on text.
I'm excited for home assistant's voice efforts these years. Packaging speech and an assistant into open source hobby electronics could be incredibly powerful.
The prospect of not having your own LLM would be frightening to any big tech CEO. LLMs might be a massive productivity boost very soon (IMHO GPT4 is already there), and if you don't have one, but your competitor does, you will either have to use their product or perish.
Big Tech hasnt dominated anything. They got trounced by a 400 person startup. Alexa, Google Search, FB research all spend untold money on ML systems. Neither is releasing game changing products anywhere close to OpenAI.
OpenAI was backed by probably the biggest tech company around, Microsoft. Without the computing afforded by Azure, they would have problems training and running the LLM.
It is true that big companies have brands and existing revenue stream to protect (Google), so maybe only a little company with nothing to lose could do what OpenAI did.
And maybe MS found the perfect arrangement giving them an ability to adjust how closely to associate themself with the tech according to need.
But is this all necessarily a good thing? OpenAI gave MS access to GPT and MS clearly wants to use GPT to destroy Google, so Google now has respond aggressively, and now we have exactly the AI arms race that many predicted would lead to the worst outcomes.
I disagree. All of the big players have been involved in some way, creating their own systems (which are quite competitive) as well as laying much of the foundation for existing models with research they published.
I wouldn't say big tech was "trounced" by OpenAI. Advances in LLMs are driven by the research community. OpenAI is a small, well-resourced organization with the sole purpose of quickly deploying and hyping state of the art AI models. Big tech naturally has a slower product development cycle - and will make full use of LLMs in due time.
> I wouldn't say big tech was "trounced" by OpenAI.
I mean not a single “big tech” company is offering any LLM products not powered by Open AI. It’s been almost 3 years since GPT-3 was released. I’m not sure they can catch up at this point. And I think the GPs point was a good one: big tech threw billions and billions at personal assistants. Nothing innovative was delivered.
Probably, but using a transformer isn’t Open AI’s advantage. A widely available language model with “emergent abilities” is Open AI’s advantage. The fact that it’s a transformer is more of an implementation detail.
The Transformer architecture that powers GPT models is based on a now-famous Google Brain paper, Attention is All You Need. You're right that OpenAI has productized this research much faster than Big Tech, but the research itself came from established R&D in Academia and Big Tech.
The diffusion papers most analogous to the transformer paper are from Berkeley, Stanford, and Freiburg. Imagen wasn’t foundational in the way you’re suggesting.
Yes and tech was so path breaking and innovative that the company sold its 49% share for 10 billion dollar /s. Sam Altman is extremely experienced in the VC world and could have easily raised billions for potential value. He wouldn't have sold half of his company within 6 months of releasing chatGPT if there was any thing remotely defensible in the product
I already have (before GPT 3 or 4) a plug in for Zoom which summarizes zoom calls, makes transcripts, and does all sorts of other things I don't care about (e.g. tries to gauge engagement and sentiment). It's pretty handy.
Of course like everything these days it's a cloud silo: the transcripts, videos etc are in their cloud. There's no way to have them simply posted to slack channels so they can be found later in searches etc.
The economic models and thinking of 2023 are seemingly still stuck in the PC era rather than being focused on the needs of the customer.
What do you mean by the “PC” era? At least with PCs as the dominant computing platform we had files and programs separate from each other. Smartphones began the trend of app-bound data.
The PC era was marked by app-specific file formats and little interchange. To me, when the PC was released it was a shock as I already used to (somewhat) interchangeable file formats on the networked machines we used back then.
By the late 90s there were some interchangable formats for photos and audio at least, though often you still had to install plug ins, drivers, codecs etc for various proprietary formats especially video.
It is like reading the news or a study paper but you know it is correct because a robot is telling you. This is the closest thing to a prophet people will have. Go downvote me.
We should care about collaboration and people’s well being.
Barring a catastrophe, AI is not going to stop.
At the current rate, by next month we will have some previously-never-seen model/algorithm/chip/tech that will run locally on our phones and computers. At that point is game over for the big server-model companies.
After that… well, pretty much all companies will be at stake.
There’s a much bigger revolution happening right now than just the tech industry.
We will be forced to ask ourselves what it means to be human.
No company is going to dominate AI, AI is going to dominate AI.
All indications are that you can scale LLMs up & down pretty smoothly, I bet there will be lots of workflows in future using AI on pc-or mobile scale hardware which can call out to cloud AI for "deep thinking" or to double check stuff.
It seems likely there will be AIs specialised in taxes or medicine, people/systems will consult multiple AIs to smooth out errors and look for consensus etc. In short, I agree with OP that we will probably end up with something that looks more like an ecosystem than a hegemony.
Doesn't the recent development (Stable Diffusion, LLaMA) tell us that these things actually run surprisingly well locally and that the models can be compressed much further?
And aren't we finding out that the amount of parameters seems to matter less than we used to believe?
> by next month we will have some previously-never-seen model/algorithm/chip/tech that will run locally on our phones and computers. At that point is game over
This comment is so wildly inaccurate and hype, and yet so common in the GPT craze on HN
It is not happening on your phone and computers in the near term for 2 reasons:
1. Entropy, the models are LARGER, NOT SMALLER. The amazing performance come from the fact that they are BIG. This is opposite of Moore's law, which itself is ending.
2. Companies want to make money on their models, vending a client is clearly not the direction that companies are going. In fact greedy companies wanted everything to happen on the server end even without any real customer need, e.g. Jira.
Check it out. They are claiming 100x faster training and inference with smaller models, performing almost with the same accuracy as GPT.
People are already running models on their phones. Some people have been posting here and on Twitter how they can run Alpaca (Llama + GPT fine tuning), on their Pixel phones.
2. The cat is out of the bag. People are creating and releasing stuff faster than companies can keep up with.
Have you seen what ChatGPT is already able to do with plug-ins? You can give it control of a Linux machine and tell it “create a basic CRUD app and run it on a docker container”, and it will do it.
That means soon AI is going to be training itself, or at least with minimum input/direction from humans.
If you don’t have access to plug-ins yet, you could do it yourself by using the GPT api, writing a prompt that tells GPT they are interacting with a terminal and to only output terminal commands, then pass the output to a terminal, and passing back the output to GPT in a loop.
I can’t find the exact link to the docker app video/article.
Lately it’s really hard to keep up. The search feature here on HN kinda sucks on mobile and Google is hopelessly behind with how fast things are coming out.
That project isn’t close to the SoTA models. It performs well, but not close to GPT-4 for example. “Almost the same performance as gpt” always ends up meaning it performs like one of the smaller GPT models.
It is not about GPT, it’s a whole revolution that is happening.
New models are coming out almost every day. Including some that have almost same performance as GPT but are 100x faster to train and are smaller (look at this:https://news.ycombinator.com/item?id=35281026).
You can believe or deny whatever you want.
The technology is moving way faster than we can keep up with and the speed is accelerating.
I'm lazy, so this isn't exhaustive. The first benchmark in the RWKV README is LAMBADA, where it got about 39% accuracy. SOTA is 90%, so that's about 6x more errors than SOTA. 4.5x the error rate of the first versions of GPT-3 in 2020. I didn't find LAMBADA benchmarks for GPT3.5 or GPT4, but from the GPT4 technical report, it generally makes about 2-3x fewer mistakes than GPT3.5 or prior SOTA on various benchmarks, so the difference is even more extreme. It's safe to say RWKV is not close to SOTA.
Meanwhile GPT4 still really sucks at a lot of things. So by the time open source commodity hardware models do reach GPT4 level, the SOTA will be that much further ahead.
The tech will be ubiquitous some day, but it's not right around the corner.
> The $11bn that Microsoft has reportedly put into Openai would, at the startup’s latest rumoured valuation of $29bn, give the software giant a stake of 38%.
That's not quite accurate, Microsoft first invested $1 billion in OpenAI in 2019 (four years ago) and another $10 billion a few months ago when it was valued at $29B.
So it's latest $10B investment gives it a 35% ownership, but it's unclear how much the $1B 2019 investment gave Microsoft. Some news reports peg Microsoft's current ownership at 49% of OpenAI, which means Microsoft's 2019 investment gave it 14% of the company (which would further indicate that OpenAI was valued at $7-8 billion at that time).
68 comments
[ 2.7 ms ] story [ 118 ms ] threadI thought it was extremely adept change of referring to it as machine intelligence instead of artificial. There's nothing artificial about this intelligence, merely different. I think this will probably be increasingly more correct moving forward with additional advances.
But then again, I don’t think humans are either.
Why would this be? Because AI allows you to copy your competitor in ways that was never possible before. It essentially becomes a skill and technology replicator. For example, look how Alpaca was able to somewhat replicate the multi million dollar models with $600 dollar cost.
I’d really like to start acquiring them. I know about The Pile and Common Crawl.
Are their any resources for individuals, that happen to have large amounts of storage, in regards to acquiring the best datasets being used?
I did lie a bit; OpenAI does have private datasets. For example, GPT-3 was trained on Internet data, but InstructGPT/ChatGPT was further finetuned through their own dataset of question/answer pairs. This was augmented in a really clever way: the initial set of valid completions was written by humans, but then OpenAI made more AI models to augment their training data before finetuning GPT-3 on it.
Even then, there's techniques for taking the output of one large language model and using it to finetune another. So just having a publicly-available AI chatbot is enough to get it to generate a dataset that you can then train another AI chatbot on.
I personally have been scraping Wikimedia Commons for public-domain imagery to train an art generator bot on; but this is mostly because I want an art generator that isn't full of pending copyright infringement lawsuits.
What I can’t get really pinned down is how much storage one needs to have your own copy of all of the most valuable raw data. At initial estimate it seemed like less than 1PB was enough for sure if focusing on text and not images/video. It even looked like maybe 200-400 TB might have been more than enough. I need to investigate more. I need numbers like 900GB for pile, then 2.5TB for a meta curated text only common crawl. So I’m assuming there is at least 10 of these in same ballpark that are important to have a copy of.
Any thoughts on the size requirements? Focusing on text.
https://www.home-assistant.io/blog/2022/12/20/year-of-voice/
And maybe MS found the perfect arrangement giving them an ability to adjust how closely to associate themself with the tech according to need.
But is this all necessarily a good thing? OpenAI gave MS access to GPT and MS clearly wants to use GPT to destroy Google, so Google now has respond aggressively, and now we have exactly the AI arms race that many predicted would lead to the worst outcomes.
I mean not a single “big tech” company is offering any LLM products not powered by Open AI. It’s been almost 3 years since GPT-3 was released. I’m not sure they can catch up at this point. And I think the GPs point was a good one: big tech threw billions and billions at personal assistants. Nothing innovative was delivered.
Anything that can be provided by a micro or small sized company can be provided by billion dollar corporations too.
Maybe size brings a drawback, in the same way a mega ship finds it difficult to turn around without a day's notice — but these companies can and will.
Microsoft once betted all chips on the internet being fad; look at them now.
Of course like everything these days it's a cloud silo: the transcripts, videos etc are in their cloud. There's no way to have them simply posted to slack channels so they can be found later in searches etc.
The economic models and thinking of 2023 are seemingly still stuck in the PC era rather than being focused on the needs of the customer.
By the late 90s there were some interchangable formats for photos and audio at least, though often you still had to install plug ins, drivers, codecs etc for various proprietary formats especially video.
Basically a silo situation as bad as today.
We have been corrupted by words like that.
Why should we care about dominance?
We should care about collaboration and people’s well being.
Barring a catastrophe, AI is not going to stop.
At the current rate, by next month we will have some previously-never-seen model/algorithm/chip/tech that will run locally on our phones and computers. At that point is game over for the big server-model companies.
After that… well, pretty much all companies will be at stake.
There’s a much bigger revolution happening right now than just the tech industry.
We will be forced to ask ourselves what it means to be human.
No company is going to dominate AI, AI is going to dominate AI.
It seems likely there will be AIs specialised in taxes or medicine, people/systems will consult multiple AIs to smooth out errors and look for consensus etc. In short, I agree with OP that we will probably end up with something that looks more like an ecosystem than a hegemony.
And aren't we finding out that the amount of parameters seems to matter less than we used to believe?
This comment is so wildly inaccurate and hype, and yet so common in the GPT craze on HN
It is not happening on your phone and computers in the near term for 2 reasons:
1. Entropy, the models are LARGER, NOT SMALLER. The amazing performance come from the fact that they are BIG. This is opposite of Moore's law, which itself is ending.
2. Companies want to make money on their models, vending a client is clearly not the direction that companies are going. In fact greedy companies wanted everything to happen on the server end even without any real customer need, e.g. Jira.
Check it out. They are claiming 100x faster training and inference with smaller models, performing almost with the same accuracy as GPT.
People are already running models on their phones. Some people have been posting here and on Twitter how they can run Alpaca (Llama + GPT fine tuning), on their Pixel phones.
2. The cat is out of the bag. People are creating and releasing stuff faster than companies can keep up with.
Have you seen what ChatGPT is already able to do with plug-ins? You can give it control of a Linux machine and tell it “create a basic CRUD app and run it on a docker container”, and it will do it.
That means soon AI is going to be training itself, or at least with minimum input/direction from humans.
Here’s a good write up about that plug-in and an overview of a lot of the things you can do: https://andrewmayneblog.wordpress.com/2023/03/23/chatgpt-cod...
If you don’t have access to plug-ins yet, you could do it yourself by using the GPT api, writing a prompt that tells GPT they are interacting with a terminal and to only output terminal commands, then pass the output to a terminal, and passing back the output to GPT in a loop.
I can’t find the exact link to the docker app video/article.
Lately it’s really hard to keep up. The search feature here on HN kinda sucks on mobile and Google is hopelessly behind with how fast things are coming out.
Also what is “close”? The benchmarks I saw showed around 80% accuracy compared to GPT.
It just came out, but in 4 months of development this vehicle should be able to do 2000 miles per gallon instead of 20 miles per gallon.
GPT is not magic, it's performance is linear with its size.
New models are coming out almost every day. Including some that have almost same performance as GPT but are 100x faster to train and are smaller (look at this:https://news.ycombinator.com/item?id=35281026).
You can believe or deny whatever you want.
The technology is moving way faster than we can keep up with and the speed is accelerating.
Meanwhile GPT4 still really sucks at a lot of things. So by the time open source commodity hardware models do reach GPT4 level, the SOTA will be that much further ahead.
The tech will be ubiquitous some day, but it's not right around the corner.
That's not quite accurate, Microsoft first invested $1 billion in OpenAI in 2019 (four years ago) and another $10 billion a few months ago when it was valued at $29B.
So it's latest $10B investment gives it a 35% ownership, but it's unclear how much the $1B 2019 investment gave Microsoft. Some news reports peg Microsoft's current ownership at 49% of OpenAI, which means Microsoft's 2019 investment gave it 14% of the company (which would further indicate that OpenAI was valued at $7-8 billion at that time).