> Kenn explained what multimodal AI is about, which can translate text not only accordingly into images, but also into music and video.
Interesting that they are broadening instead of deepening the text side of GPT - maybe they are running into problems on that side and are building the feature set with low hanging fruit on adjacent areas.
A paper from a week ago found that models trained on multiple data modes perform an order of magnitude better than text-only models of the same or even larger size.
Some of these large models are able to do zero shot learning and perform tasks they weren't explicitly trained on since the training objective is very general.
Being able to perform more advanced types of zero shot learning tasks would be comparable and further the accuracy on those tasks can be evaluated
The next big step for coding LLMs will be context window increases, leaked docs have OpenAI pricing for up to 16K I believe, 4x the current maximum. Now you're talking "write a class" instead of this line and maybe sometimes a method
Are you referring to PALM-E? It didn't have any positive transfer for NLP tasks, in fact the unfrozen model performed slightly worse after the finetune.
That being said, PALM-E wasn't really a multimodal model from the start, it's still basically just a text model with a visual one glued on top. Whether a truly multimodal model will be better at reasoning and data efficiency is still an open question though.
There is substantial evidence that cross training on multiple tasks improves performance.
ChatGPT struggles with intuition about the world, if it has image/video knowledge than it can potentially reason about what happens when you flip a plate upside down.
I wonder if there’s any reason this wouldn’t also apply in reverse: could a corpus of textual training data from which the role of a hand can be derived (including the fact that it naturally has five fingers) help DALL-E reliably render hands correctly?
Almost certainly. There is a vast corpus of text which references hands and what they are used for. Likewise there is a vast corpus of images which include hands. A sufficiently advanced statistical model can infer that any image with 6 fingers is a fabrication provided that it has learned how to classify fingers and has enough linguistic knowledge. Contrast this situation with a primarily image based model which will struggle to differentiate what situations detailed hands appear in compared with a patterned rug.
There is an interesting “takeoff” point where models are able to generate enough “interesting” data to train themselves. Asking chatGPT for ml training data on various tasks makes me think we are close to that inflection point.
One of the biggest improvements to text models like Image/Parti which are much better than Stable Diffusion is that they typically either train a much larger text model from scratch, or plug in a very good pre-trained text model like T5. This instantly upgrades their images.
It turns out that the image models already understand the low-level details of images well, and it's high-level reasoning/composition that is screwing their samples up - which is due to the poor text embedding they receive as a summary. (You can't generate a good image of 'a blue ball left of a red ball' if your text embedding is confused what 'left' is, no matter how high quality you are able to generate 'blue ball' or 'red ball' individually.) So, I would not be surprised if hands are improved by simply using a much better text model. (And why not plug in GPT-3 or PaLM for even better results...)
That said, hands are also like text-inside-images in being small, highly variable, and intrinsically difficult. And we know that the larger models solve text rendering simply by scaling up an OOM or 2 past SD. (You can see this from the text samples in the papers, and also more recently from Stability's Deep Floyd samples they've been teasing on Twitter.) So either way, scaling will probably solve hands soon.
Cross-training doesn't always help, I think the architecture is still being worked out. I can't remember the name of it but Google had some multimodal model they released a paper for, maybe six months or a year ago, and it did slightly worse than SOTA ("negative transfer" is the term they used for doing worse from additional modalities), and I recall the paper talking about how reaching near-parity with single mode models was an improvement.
One of their new models, PaLM-E, does show some positive transfer, so it's definitely a potential path for increased capabilities.
i think it's likely that they hired a diversified AI team with expertise in multiple areas and they're all working on their own areas. parallel efforts.
Also, amazingly, transfer learning works better than anyone could have hoped! As an analogy, imagine that you could show that football players learned musical instruments 50% quicker than non-athletes. That would be amazing!
But LLMs can write music and training them on music makes them better writers. It's almost like intelligence & skills are completely general!
That's just not true. There's no more super cheap data, but there's still probably about 5x the data we are currently feeding these models just waiting to be scanned and OCRd, and most of it is higher quality (if less topical).
The fact that AI requires so much data does make me think that we are somehow on the wrong track. Evolution didn't scan every atom on earth in order to produce a brain. FSD is a good example of this. It is ridiculous to me that we need to show the car videos of every possible situation in order for it to know how to drive.
It’s not the wrong track, it’s just very early and specialized. As you get more multi-modal models with better ability to learn/remember on the fly, they will require less and less data to train.
I'm pretty sure we will look back and laugh about how this was done in 50 years. No disrespect to anyone working in the area now of course, but I think better, more natural approaches will emerge.
> The fact that AI requires so much data does make me think that we are somehow on the wrong track.
I know the entire AI industry on the wrong track, but there's nothing that can be done about it. Sometimes you just have to wait for the crash and burn, and then things can change. It's like being at the ocean and trying to stop a wave from crashing after it has already formed, you just have to let it resolve itself of its own volition. As Einstein said, stupidity is infinite.
Not just that. Language acquisition. Teacher never stands in front of class and says "here's what the rules of grammar aren't. You today learn want do not how." There is a paucity of negative examples in the "training". Same goes for so many other tasks we learn quicker than can be explained with a Hebbian punishment/reward model. Most people can play a board game somewhat competently after just being told the rules. Some are even good at it. You don't need to crash your car dozens of time to figure out how to drive. All over we see something fundamental missing, and the industry is trying to hammer the screw into the wall while pretending that's the best way to do it.
OpenAI is doing an amazing job fueling hype and intrigue. It is hard to separate what is real and what is imagined.
Previously, I heard that Bing Chat was based on GPT4. Is this true or not? Is it publicly known, or not? Certainly the model is far more powerful than what is in ChatGPT.
If Bing is GPT4, did they remove its multimodal capabilities?
If Bing is not GPT4, what is it? Does there exist a GPT3.5-mega with vastly improved reasoning capabilities and larger context windows?
I find the current moment in tech quite confusing.
chatgpt is what they call got3.5-turbo-0301 (current month checkpoint)
bingGPT is a allegedly finetuned version of gpt3.5 turbo that is better with search results.... microsoft internally calls it prometheus. You may also hear it referred to as Sydney.
There are many multimodal models that take an existing text only llm and do some visual training. Blip-2, Fromage, prismer and palm-e are examples. Multimodal isn't necessarily multimodality from scratch
Have I missed something? Last I heard Bing chat was off the rails, "vastly improving" the ability to rationalize contradictions rather than to reason. There was that screenshot going around where, facing a logical inconsistency, it resolved it by insisting to the user the current year is 2021.
Watching the Bing Chat escapades is both hilarious and mildly unsettling. It reminds me _so strongly_ of the sarcastic evil AI that runs the levels in the _Portal_ games.
If even half of what we're hearing about GPT 4 comes to fruition then we're going to know in the next year or two tops if the scaling hypothesis is true or not.
If it's true, then we're going to have something very similar to the Star Trek computer by the end of the decade at the latest.
If it's false, then we're probably going to enter another AI winter this decade.
I am curious if AI is successful if an AI backlash is coming. Perhaps one of the existing political parties or a new political party taking up the mantle of regulating or outright banning of AI. Or even just attacks on AI researchers and institutions.
it's already happening. Stable diffusion forums regularly get brigaded, and there are lawsuits in the works against stability ai and midjourney. Those lawsuits are nominally about copyright, but one gets the sense that it's really about putting the breaks on developing the tech.
I kind of get it, too. If I made my money making art this would scare me. What I wonder is what's gonna happen when it's [ artists, accountants, copywriters, bloggers, designers, programmers, etc, etc, etc].
The resistance if futile. The amount of effort currently going into making generative models like Stable Diffusion or ChatGPT is insane. It's the fastest changing subfield in ML. These models are getting better and better every month, with no sign of plateauing in the near future. I expect decent music and video generation models appearing within the next 2 years. There's no way to stop the progress.
"The actual internet" allowed people to get information 20x times faster than before (e.g. vs physically going to a library, or writing a letter to an expert). If we now have a technology that lets us get answers 20x times faster than what we had before it, then yes - I see this technology as a huge advancement over the "old" internet.
1) I wanted to make a github action to auto build, run tests and publish a package. I don't do a lot of this type of work so quite unfamiliar with it. ChatGPT gave me a very good start with some example yaml which I was able to tweak and get started with. When something wasn't working with my tweaked yaml file, I asked it what the problem was and it was able to pin-point the issue right away. I don't use it at work for anything but it is very useful for side projects and things like that.
2) I had some example json text from a service for which I wanted to create some serialization classes. This is easy to do but it is tedious work. ChatGPT was able to write most of this for me. There are likely tools, websites, services that can create this, but the experience in ChatGPT is pretty great - copy and paste in some json text and ask it to create the code. If it isn't quite right, give it some additionally instructions. When 80% OK, manually tweak it.
3) I use it to get ideas for things or to provide answers about things. For example, one of the commenters in this thread discussed the "AI scaling hypothesis". Google would have had a good answer for this but if you happen to have any questions(after the initial response is provided) this is a completely new search. ChatGPT, though not always correct, feels a lot more calming to use. Google seems very "noisy" to me now. Google is still very necessary but it feels like only 20% of queries need to go there at this point.
I see. I’m skeptical all of this (and similar use cases) could add up to “as much or more” than the regular internet. The average internet usage is something like 6-7 hours per day. I don’t see how even extremely heavy usage of GPT gets anywhere close to that, unless much of that time is “playing” (which is itself extremely cool!)
The comparison I draw is to google search, not general internet usage like Netflix and YouTube. I think it makes sense to use ChatGPT more often than search for most of my use cases. Fully agree that it does not replace the entire internet.
Okay yes that I find much more believable. Information retrieval is pretty awesome, though I’d suggest a lot of that is due to the inferiority of the search engine business model than superiority of LLM technology. Especially if hallucinations and fake citations are solvable, this will be a killer app for sure.
Obviously generative usecases are awesome (though IMO will be practically fewer and further between than people seem to think), and summarization usecases are awesome.
Today I used chatGPT to workshop title ideas for a research paper I’m finalizing. (It wasn’t able to produce good backronym titles, sadly, but it was a pretty good source of feedback/ideas)
Some graduate students I collaborate use it to help them revise and polish their writing (English is not their first language)
Look outside of dev work, which needs a huge amount of context in a specialized language domain, and instead look at things that are human to human communication in order to find areas ChatGPT where already an indispensable tool.
Right but most internet usage is read, not write, which is why I was specifically curious about the claim that they use GPT as much or more than the internet.
Also these types of use cases I think will not actually last very long. They’ll be adversarial’d out of value and then probably out of existence after that.
In a world where every listing is written by a bot, a reader’s first objective is to distill the human-written parts out of the unreliable narration and filler words of the bot.
What kind of a "winter" would this be when literally the world already runs on AI?
Search engines run on AI, stock markets trade on AI, social networks distribute content by AI, your photos and content are sorted on your phone by AI, vaccines and medicines are produced by AI, precedent law is implemented increasingly via AI, our weapons are AI (drones) and so on and so on and so on.
Technological singularity is the point no one is able to predict what happens after the point from what comes before. We have not arrived that yet. And yes, strong AI will pave road to technological singularity.
If the definition of singularity has such low requirements, I think we've probably reached singularity somewhere back at the dawn of human civilization when we started making lots of predictions, for sure, and the overwhelming majority of them terrible. We carry this old tradition with ourselves to this day.
You're using the term AI pretty generously there. Let's not delude ourselves into believing that deterministic algorithms are general artificial intelligence.
This has always made me curious about multi modality and especially Google’s Palm. At least how google presents Palm in diagrams.
The way I’ve interpreted the scaling hypothesis is that we will see emergent intelligence with larger nets through automated training alone. If we want a model to learn images we throw a larger net at it with training data.
The way I’ve interpreted some of these newer techniques to multi modality is that they are stitched together models. If we want a model to learn images we decide this and teach one model images and then connect it to the core model. There’s not a lot of emergent behavior due to scaling in this scenario.
With that perception I don’t see how gpt4 says anything about the scaling hypothesis. However I am not in this field and would be grateful to learn more.
The brain is composed of distinct regions that specialise in specific tasks. It's reasonable to assume AGI would be the same.
So the goal should be: we've created a "language module" (LLMs) and a "visual perception module" (computer vision), but we also need to add a "logic module", a "reasoning module", an "empathy module", etc, while continuing to improve each.
I just don't see how you could get an LLM, no matter how advanced, to recognize a car. Even if it can describe cars (wheels, windshield, doors) it doesn't know what any of those components look like. It's like that old joke about philosophers being unable to define a chair beyond "I'll know it when I see it".
it's even more foundamental than that, the brain is constantly learning and retaining new knowledge and solve problems with a mix of old knowledge and knowledge learned in the context of the problem, either by experimentation or research
these net can replicate at most the first step for now, even tuning by reingofrcement learning is more of a set up batch than an ongoing thing, and certainly not something they will be able to do in the context of a single problem but as part of a retraining
agi is still a fair bit away, I'm unsure if these super large architecture will ever get to replicate the second part of our brain, the flexibility while on the job, because of their intrinsic training mechanism.
I recall watching a video where Sam Altman was downplaying expectations for GPT4 so I'm not expecting Star Trek. However, I don't think we are due for another AI winter anytime soon. Even if technology were to plateau at GPT3.5 for several years it would not constitute a typical AI winter as I don't think we've found the edges of the current technology yet.
We can have an AI "winter" without actually having a solid freeze in terms of development if expectations fail to match the reality. Right now, the hype around ChatGPT is unwarranted in my opinion, and if nothing else comes from it we will end up in at least a cooling off period as people realize that.
A lot of people find Siri (and other AI assistants) helpful, and use it every day. Some would even say Siri is good enough for the limited set of tasks it currently performs. ChatGPT is like Siri on steroids - it's much better in every way, with the important caveat that it - rarely - can provide a wrong/bad answer. If Siri is level 2 autonomous driving, then ChatGPT is full level 3. People hope that GPT-4 will be equivalent to level 4 driving, but I don't think anyone has promised that. Most of the hype I hear today is about what ChatGPT can already do (more or less). I actually think most people haven't even caught up with what it can do today.
I've looked for papers, but can't figure this out: do our current neural networks (LLM, vision) qualify as complex systems?
Are they composed of autonomous agents that exhibit:
- diversity (in behavior)
- high connectedness
- interaction among agents
- adaptation (agents changing their behavior based on the behavior of other agents)?
Because if current NNs don't fit that definition of complexity, they it seems unreasonable to expect them to be intelligent, no matter their scale. Intelligence will likely be an emergent property of the above four criteria.
If anyone knows of relevant books or papers, would love to hear it.
BTW: "AI winter" refers to a collapse in funding; but I think we're well past that point. Previous winters only happened because academia led AI research, and there were no commercial opportunities. Now, the economic benefits are far more tangible. And if funding doesn't dry up, there's no reason to think innovation will dry up either.
they're gonna. I've seen a lot of material on wiring these models together to have them do more interesting cross-functional work. Microsoft is about to put out a model you can talk to that makes and recognizes images - and there's already multi-model frameworks that do things like speech-to-text to LLM to text-to-speech, which means you can hold a verbal conversation.
Once the continuous learning problem is solved, it'll be pretty trivial to just tell your compound model to "go learn everything you can about x" and have it autonomously do so.
If it were me, the first thing I'd ask it to do was learn how to make other AI, then give it some hardware and the directive to build the best thing it could and test it until it can build a model better than itself.
Artificial intelligence will always be artificial.
If science can neither have a better definition of intelligence, or analyze and understand the resulting black box of a neural net, I don't see how AI would improve or get nearer general intelligence.
I have much more curiosity about neurology and how psychology define intelligence, than how programmers are trying to build things that digest data.
Proper science requires understanding. ML is not about understanding, it's only "sophisticated statistical methods". It is still artificial, thus but genuine.
I don't really get the obsession with if the MS models is some variant of GPT-3.5 vs GPT-4.
It's just a version number, and there aren't really meaningful reasons why something is a 0.5 increment vs a 1.0 release increment, other than "there are lots of changes" where lots ~= something.
The gap in quality between gpt2 and gpt3 was much much larger than the gap between gpt3 and 3.5. If they could get the gap between 2 -> 3 again when moving to 4 it would be crazy
It is a big deal because the major version numbers usually represent a new model trained from scratch, vs just a base model that has been given extra training (which is what happened from GPT 3.0 -> 3.5)
118 comments
[ 0.11 ms ] story [ 247 ms ] threadInteresting that they are broadening instead of deepening the text side of GPT - maybe they are running into problems on that side and are building the feature set with low hanging fruit on adjacent areas.
Being able to perform more advanced types of zero shot learning tasks would be comparable and further the accuracy on those tasks can be evaluated
With 16k and some other techniques, I’m guessing it could write a custom CMS database backed web application.
more literally and correctly, it’s the maximum number of tokens in the input and output, combined, where a token is 4/3 of a word
So we’re shifting from 5K words maximum to 40K (per sibling comment, who pointed out 32K context leaked as well)
ChatGPT struggles with intuition about the world, if it has image/video knowledge than it can potentially reason about what happens when you flip a plate upside down.
There is an interesting “takeoff” point where models are able to generate enough “interesting” data to train themselves. Asking chatGPT for ml training data on various tasks makes me think we are close to that inflection point.
Dalle/GPT: that's not right, there should be 5 fingers on this clock.
It turns out that the image models already understand the low-level details of images well, and it's high-level reasoning/composition that is screwing their samples up - which is due to the poor text embedding they receive as a summary. (You can't generate a good image of 'a blue ball left of a red ball' if your text embedding is confused what 'left' is, no matter how high quality you are able to generate 'blue ball' or 'red ball' individually.) So, I would not be surprised if hands are improved by simply using a much better text model. (And why not plug in GPT-3 or PaLM for even better results...)
That said, hands are also like text-inside-images in being small, highly variable, and intrinsically difficult. And we know that the larger models solve text rendering simply by scaling up an OOM or 2 past SD. (You can see this from the text samples in the papers, and also more recently from Stability's Deep Floyd samples they've been teasing on Twitter.) So either way, scaling will probably solve hands soon.
One of their new models, PaLM-E, does show some positive transfer, so it's definitely a potential path for increased capabilities.
Also, amazingly, transfer learning works better than anyone could have hoped! As an analogy, imagine that you could show that football players learned musical instruments 50% quicker than non-athletes. That would be amazing!
But LLMs can write music and training them on music makes them better writers. It's almost like intelligence & skills are completely general!
I know the entire AI industry on the wrong track, but there's nothing that can be done about it. Sometimes you just have to wait for the crash and burn, and then things can change. It's like being at the ocean and trying to stop a wave from crashing after it has already formed, you just have to let it resolve itself of its own volition. As Einstein said, stupidity is infinite.
Previously, I heard that Bing Chat was based on GPT4. Is this true or not? Is it publicly known, or not? Certainly the model is far more powerful than what is in ChatGPT.
If Bing is GPT4, did they remove its multimodal capabilities?
If Bing is not GPT4, what is it? Does there exist a GPT3.5-mega with vastly improved reasoning capabilities and larger context windows?
I find the current moment in tech quite confusing.
chatgpt is what they call got3.5-turbo-0301 (current month checkpoint)
bingGPT is a allegedly finetuned version of gpt3.5 turbo that is better with search results.... microsoft internally calls it prometheus. You may also hear it referred to as Sydney.
i suspect GPT4 is just a bad translation
Then why does the chatgpt endpoint show as text-davinci-002-render-sha? It's very confusing.
> ChatGPT is powered by gpt-3.5-turbo, OpenAI’s most advanced language model.
https://platform.openai.com/docs/guides/chat
Watching the Bing Chat escapades is both hilarious and mildly unsettling. It reminds me _so strongly_ of the sarcastic evil AI that runs the levels in the _Portal_ games.
Isn't GLaDOS a brain scan, not an AI?
There's even people who thinks brain scans/emulations will be how AGI is created, which I personally find very unlikely.
That was at least 2 weeks ago....
They've taken steps to limit it which have been mostly effective. There are some jailbreaks but they are getting harder.
It's a beta product. Give it 6 months and then judge it.
If it's true, then we're going to have something very similar to the Star Trek computer by the end of the decade at the latest.
If it's false, then we're probably going to enter another AI winter this decade.
I suspect there will be a lot of resistance as it “succeeds.”
I kind of get it, too. If I made my money making art this would scare me. What I wonder is what's gonna happen when it's [ artists, accountants, copywriters, bloggers, designers, programmers, etc, etc, etc].
I see problems coming.
Even if there are a lot of great applications for these things (I think there are), it’s not clear they’re going to be truly transformative?
For me, it is already pretty transformative. I use ChatGPT at least as much as the regular internet - probably more.
Not just as a hobby/toy?
> "Computer, locate Commander Riker's repo"
> "Commander Riker's repo is not currently on the ship"
For example. I could have figured it out elseways, but would have taken >20x the time.
1) I wanted to make a github action to auto build, run tests and publish a package. I don't do a lot of this type of work so quite unfamiliar with it. ChatGPT gave me a very good start with some example yaml which I was able to tweak and get started with. When something wasn't working with my tweaked yaml file, I asked it what the problem was and it was able to pin-point the issue right away. I don't use it at work for anything but it is very useful for side projects and things like that.
2) I had some example json text from a service for which I wanted to create some serialization classes. This is easy to do but it is tedious work. ChatGPT was able to write most of this for me. There are likely tools, websites, services that can create this, but the experience in ChatGPT is pretty great - copy and paste in some json text and ask it to create the code. If it isn't quite right, give it some additionally instructions. When 80% OK, manually tweak it.
3) I use it to get ideas for things or to provide answers about things. For example, one of the commenters in this thread discussed the "AI scaling hypothesis". Google would have had a good answer for this but if you happen to have any questions(after the initial response is provided) this is a completely new search. ChatGPT, though not always correct, feels a lot more calming to use. Google seems very "noisy" to me now. Google is still very necessary but it feels like only 20% of queries need to go there at this point.
Obviously generative usecases are awesome (though IMO will be practically fewer and further between than people seem to think), and summarization usecases are awesome.
Some graduate students I collaborate use it to help them revise and polish their writing (English is not their first language)
Eg https://www.cnn.com/2023/01/28/tech/chatgpt-real-estate/inde...
Also these types of use cases I think will not actually last very long. They’ll be adversarial’d out of value and then probably out of existence after that.
In a world where every listing is written by a bot, a reader’s first objective is to distill the human-written parts out of the unreliable narration and filler words of the bot.
https://www.youtube.com/watch?v=DbvAaQYtgYM
1) Short scripts in languages im unfamiliar with
2) Generating short, specific, well defined functions
3) Curing the tyranny of the blank page effect re emails, explanations, etc
4) Rephrasing that long email last thing on a friday when English just wont click
https://en.m.wikipedia.org/wiki/Gartner_hype_cycle
Search engines run on AI, stock markets trade on AI, social networks distribute content by AI, your photos and content are sorted on your phone by AI, vaccines and medicines are produced by AI, precedent law is implemented increasingly via AI, our weapons are AI (drones) and so on and so on and so on.
And all this up there was BEFORE GPT and DALL-E.
No, no winter. We're in the singularity.
The way I’ve interpreted the scaling hypothesis is that we will see emergent intelligence with larger nets through automated training alone. If we want a model to learn images we throw a larger net at it with training data.
The way I’ve interpreted some of these newer techniques to multi modality is that they are stitched together models. If we want a model to learn images we decide this and teach one model images and then connect it to the core model. There’s not a lot of emergent behavior due to scaling in this scenario.
With that perception I don’t see how gpt4 says anything about the scaling hypothesis. However I am not in this field and would be grateful to learn more.
So the goal should be: we've created a "language module" (LLMs) and a "visual perception module" (computer vision), but we also need to add a "logic module", a "reasoning module", an "empathy module", etc, while continuing to improve each.
I just don't see how you could get an LLM, no matter how advanced, to recognize a car. Even if it can describe cars (wheels, windshield, doors) it doesn't know what any of those components look like. It's like that old joke about philosophers being unable to define a chair beyond "I'll know it when I see it".
these net can replicate at most the first step for now, even tuning by reingofrcement learning is more of a set up batch than an ongoing thing, and certainly not something they will be able to do in the context of a single problem but as part of a retraining
agi is still a fair bit away, I'm unsure if these super large architecture will ever get to replicate the second part of our brain, the flexibility while on the job, because of their intrinsic training mechanism.
followed by GPT95, 98, XP, etc
Are they composed of autonomous agents that exhibit: - diversity (in behavior) - high connectedness - interaction among agents - adaptation (agents changing their behavior based on the behavior of other agents)?
Because if current NNs don't fit that definition of complexity, they it seems unreasonable to expect them to be intelligent, no matter their scale. Intelligence will likely be an emergent property of the above four criteria.
If anyone knows of relevant books or papers, would love to hear it.
BTW: "AI winter" refers to a collapse in funding; but I think we're well past that point. Previous winters only happened because academia led AI research, and there were no commercial opportunities. Now, the economic benefits are far more tangible. And if funding doesn't dry up, there's no reason to think innovation will dry up either.
Once the continuous learning problem is solved, it'll be pretty trivial to just tell your compound model to "go learn everything you can about x" and have it autonomously do so.
If it were me, the first thing I'd ask it to do was learn how to make other AI, then give it some hardware and the directive to build the best thing it could and test it until it can build a model better than itself.
If science can neither have a better definition of intelligence, or analyze and understand the resulting black box of a neural net, I don't see how AI would improve or get nearer general intelligence.
I have much more curiosity about neurology and how psychology define intelligence, than how programmers are trying to build things that digest data.
Proper science requires understanding. ML is not about understanding, it's only "sophisticated statistical methods". It is still artificial, thus but genuine.
It's just a version number, and there aren't really meaningful reasons why something is a 0.5 increment vs a 1.0 release increment, other than "there are lots of changes" where lots ~= something.
> We train three model sizes (1.3B, 6B, and 175B parameters)[1]
It does use the same model architecture as GPT-3 though so I guess there is a bit of reasoning there maybe.
[1] https://arxiv.org/pdf/2203.02155.pdf