> a living room with two white armchairs and a painting of the colosseum. the painting is mounted above a modern fireplace.
With the ability to construct complex 3D scenes, surely the next step would be for it to ingest YouTube videos or TV/movies and be able to render entire scenes based on a written narration and dialogue.
The results would likely be uncanny or absurd without careful human editorial control, but it could lead to some interesting short films, or fan-recreations of existing films.
I'd love to see what this does with item/person/artwork/monster
descriptions from Dwarf Fortress. Considering the game has
creatures like were-zebras, beasts in random shapes and
materials, undead hair, procedurally generated instruments and
all kinds of artefacts menacing with spikes I imagine it could
make the whole thing even more entertaining.
I think in a way that's the next step but they may have to wait a little bit before they have the processing power.
If you are talking about 24 frames per second, then theoretically one second of video could require 24 times as much processing power. And 100 seconds 2400 X. Obviously that's just a random guess but surely it is much more than for individual images.
I really do think AI is going to replace millions of workers very quickly, but just not in the order that we used to think of. We will replace jobs that require creativity and talent before we will replace most manual factor workers, as hardware is significantly more difficult to scale up and invent than software.
At this point I have replaced a significant amount of creative workers with AI for personal usage, for example:
- I use desktop backgrounds generated by VAEs (VD-VAE)
- I use avatars generated by GANs (StyleGAN, BigGAN)
- I use and have fun with written content generated by transformers (GPT3)
- I listen to and enjoy music and audio generated by autoencoders (Jukebox, Magenta project, many others)
- I don't purchase stock images or commission artists for many previous things I would have when a GAN exists that already makes the class of image I want
All of this has happened in that last year or so for me, and I expect that within a few more years this will be the case for vastly more people and in a growing number of domains.
I believe that AI will accelerate creativity. This will have a side effect of devaluing some people's work (like you mentioned), but it will also increase the value of some types of art and, more importantly, make it possible to do things that were impossible before, or allow for small teams and individuals to produce content that were prohibitively expensive.
You are probably right. Still, there is hope that this just a prelude to getting closer to a Transmetropolitan box ( assuming we can ever figure out how to make AI box that can make physical items based purely on information given by the user ).
>Models in general are generally considered “transformative works” and the copyright owners of whatever data the model was trained on have no copyright on the model. (The fact that the datasets or inputs are copyrighted is irrelevant, as training on them is universally considered fair use and transformative, similar to artists or search engines; see the further reading.) The model is copyrighted to whomever created it.
"Models in general are generally considered “transformative works” and the copyright owners of whatever data the model was trained on have no copyright on the model. (The fact that the datasets or inputs are copyrighted is irrelevant, as training on them is universally considered fair use and transformative, similar to artists or search engines; see the further reading.) The model is copyrighted to whomever created it. Hence, Nvidia has copyright on the models it created but I have copyright under the models I trained (which I release under CC-0)."
I bet they can claim copyright up to the gradients generated on their media, but in the end the gradients get summed up, so their contribution is lost in the cocktail.
If I write a copyrighted text on a book, then I print a million other texts on top of it, in both white an black, mixing it all up to be like white noise, would the original authors have a claim?
But does that still hold when the model memorized a chunk of the training data? Or can a network plagiarize output while being a transformative work itself?
I agree, and due to the amount of compute that is required for those types of works I think those are still quite awhile away.
But the profession for creative individuals consists of much more than highly-paid well-credentialed individuals working at well-known US corporations. There are millions of artists that just do quick illustrations, logos, sketches, and so on, on a variety of services, and they will be replaced far before Pixar is.
Not entirely, no, I don't hope I implied that. I listen to human-created music every day. I just mean to say that I've also listened to AI-created music that I've enjoyed, so it's gone from being 0% of what I listen to to 5%, and presumably may increase much more later.
You should try Aiva (http://aiva.ai). At some point I was mostly listening to compositions I generated through that platform. Now I'm back to Spotify, but AI music is definitely on my radar.
There still needs to be some sort of human curation, lest bad/rogue output risks sinking the entire AI-generated industry. (in the case of DALL-E, OpenAI's new CLIP system is intended to mitigate the need for cherry-picking, although from the final demo it's still qualitative)
The demo inputs here for DALL-E are curated and utilize a few GPT-3 prompt engineering tricks. I suspect that for typical unoptimized human requests, DALL-E will go off the rails.
Yes, but there's no reason we can't partially solve this by throwing more data at the models, since we have vast amounts of data we can use for that (ratings, reviews, comments, etc), and we can always generate more en masse whenever we need it.
This isn't a problem that can be solved with more data. It's a function of model architecture, and as OpenAI has demonstrated, larger models generally perform better even if normal people can't run them on consumer hardware.
But there is still a lot of room for more clever architectures to get around that limitation. (e.g. Shortformer)
I think it's both - we have a lot of architectural improvements that we can try now and in the future, but I don't see why you can't take the output of generative art models, have humans rate them, and then use those ratings to improve the model such that its future art is likely to get a higher rating.
Frankly, I think the "AI will replace jobs that require X" angle of automation is borderline apocalyptic conspiracy porn. It's always phrased as if the automation simply stops at making certain jobs redundant. It's never phrased as if the automation lowers the bar to entry from X to Y for /everyone/, which floods the market with crap and makes people crave the good stuff made by the top 20%. Why isn't it considered as likely that this kind of technology will simply make the best 20% of creators exponentially more creatively prolific in quantity and quality?
> Why isn't it considered as likely that this kind of technology will simply make the best 20% of creators exponentially more creatively prolific in quantity and quality?
I think that's well within the space of reasonable conclusions. For as much as we are getting good at generating content/art, we are also therefore getting good at assisting humans at generating it, so it's possible that pathway ends up becoming much more common.
Not to undermine this development, but so far, no surprise, AI depends on vast quantities of human-generated data. This leads us to a loop: if AI replaces human creativity, who will create novel content for new generation of AI? Will AI also learn to break through conventions, to shock and rewrite the rules of the game?
It’s like efficient market hypothesis: markets are efficient because arbitrage, which is highly profitable, makes them so. But if they are efficient, how can arbitrageurs afford to stay in business? In practice, we are stuck in a half-way house, where markets are very, but not perfectly, efficient.
I guess in practice, the pie for humans will keep on shrinking, but won’t disappear too soon. Same as horse maintenance industry, farming and manufacturing, domestic work etc. Humans are still needed there, just a lot less of them.
> Will AI also learn to break through conventions, to shock and rewrite the rules of the game?
I think AlphaGo was a great in-domain example of this. I definitely see things I'd refer to colloquially as 'creativity' in this DALL-E post, but you can decide for yourself, but that still isn't claiming it matches what some humans can do.
True, but AlphaGo exists in a world where everything is absolute. There are new ways of playing Go, but the same rules.
If I train an AI on classical paintings, can it ever invent Impressionism, Cubism, Surrealism? Can it do irony? Can it come up with something altogether new? Can it do meta? “AlphaPaint, a recursive self-portrait”?
Maybe. I’m just not sure we have seen anything in this dimension yet.
>If I train an AI on classical paintings, can it ever invent Impressionism, Cubism, Surrealism?
I see your point, but it's an unfair comparison: if you put a human in a room and never showed them anything except classical paintings, it's unlikely they would quickly invent cubism either. The humans that invented new art styles had seen so many things throughout their life that they had a lot of data to go off of. Regardless, I think we can do enough neural style transfer already to invent new styles of art though.
if AI replaces human creativity, who will create novel content for new generation of AI?
Vast majority of human generated content is not very novel or creative. I'm guessing less than 1% of professional human writers or composers create something original. Those people are not in any danger to be replaced by AI, and will probably be earning more money as a result of more value being placed on originality of content. Humans will strive (or be forced) to be more creative, because all non-original content creation will be automated. It's a win-win situation.
Most arbitrageurs cannot stay in the business, it's the law of diminishing returns. Economies of scale eventually prevent small individual players to profit from the market, only a few big-ass hedge funds can stay, because due to their investments they can get preference from exchanges (significantly lower / zero / negative fees, co-located hardware, etc.) which makes the operation reasonable to them. With enough money you can even build your own physical cables between exchanges to outperform the competitors in latency games. I'm a former arbitrageur, by the way :)
Same with AI-generated content. You would have to be absolutely brilliant to compete with AI. Only a few select individuals would be "allowed" to enter the market. Not even sure that it has something to do with the quality of the content, maybe it's more about prestige.
You see, there already are gazillions of decent human artists, but only a few of them are really popular. So the top-tier artists would probably remain human, because we need someone real to worship to. Their producers would surely use AI as a production tool, depicting it as a human work. But all the low-tier artists would be totally pushed out of the market. There will be simply no job for a session musician or a freelance designer.
It's too hard to say I think. Big players will definitely benefit a lot, so it probably isn't a bad idea, but if you could find the right startups or funds, you might be able to get significantly more of a return.
I won't say many of those things are creativity driven. There are more like auto assets generation.
One use case of such model would be in gaming industry, to generate large amount of assets quickly. This process along takes years, and more and more expensive as gamers are demanding higher and higher resolution.
AI can make this process much more tenable, bring down the overall cost.
The way this model operates is the equivalent of machine learning shitposting.
Broke: Use a text encoder to feed text data to an image generator, like a GAN.
Woke: Use a text and image encoder as the same input to decode text and images as the same output
And yet, due to the magic of Transformers, it works.
From the technical description, this seems feasible to clone given a sufficiently robust dataset of images, although the scope of the demo output implies a much more robust dataset than the ones Microsoft has offered publicly.
It's not really surprising given what we now know about autoregressive modeling with transformers. It's essentially a game of predict hidden information given visible information. As long as the relationship between the visible and hidden information is non-random you can train the model to understand an amazing amount about the world by literally just predicting the next token in a sequence given all the previous ones.
I'm curious if they do a backward pass here, would probably have value. They seem to describe sticking the text tokens first meaning that once you start generating image tokens all the text tokens are visible. That would have the model learning to generate an image with respect to a prompt but you could also literally just reverse the order of the sequence to have the model also learn to generate prompts with respect to the image. It's not clear if this is happening.
That approach wouldn't work out of the box; it sees text for the first 256 tokens and images for the following 1024 tokens, and tries to predict the same. It likely would not have much to go on if you gave it the 1024 tokens for the image and then 256 for the text later since it doesn't have much of a basis.
A network optimizing for both use cases (e.g. the training set is half 256 + 1024, half 1024 + 256) would likely be worse than a model optimizing for one of the use cases, but then again models like T5 argue against it.
Is this kind of happening with the CLIP classifier [1] to rank the generated images?
> Similar to the rejection sampling used in VQVAE-2, we use CLIP to rerank the top 32 of 512 samples for each caption in all of the interactive visuals. This procedure can also be seen as a kind of language-guided search16, and can have a dramatic impact on sample quality.
> CLIP pre-trains an image encoder and a text encoder to predict which images were paired with which texts in our dataset. We then use this behavior to turn CLIP into a zero-shot classifier. We convert all of a dataset’s classes into captions such as “a photo of a dog” and predict the class of the caption CLIP estimates best pairs with a given image.
Shows you where the role of a meme and a shit poster may exist in a cosmological technological hierarchy. Humans are just rendering notes replicating memes, man. /s in the dude voice from big Lebowski.
It's actually a bit more complicated. Since DALL-E uses CLIP for training, and CLIP is itself trained using separate text and image encoders: https://openai.com/blog/clip/
At some point we'll have so many models based on so many other models it will longer longer be possible to tell which techniques are really involved.
Now we just have to wait for huggingface to create an open source implementation. So much for openness I guess if you go on Microsoft azure you can use closed ai.
In various Episodes of Star Trek The Next Geneneration, the crew asks the computer to generate some environment or object with relatively little description. It’s a story telling tool of course, but looking at this, I can begin to imagine how we might get there from here.
I looked around a bit and couldn't find Dall-e in there. A higher post in this thread said they don't usually release their models. It's a shame, this would have been fun to play with.
Possibly, but in software the realm of errors is wider and more detrimental. Imagery, the human mind will fill in the gaps and allow interpretation. Software, not so much.
True, but the human mind needs an expensive, singleton body in the real world, while a code writing GPT-3 only needs a compiler and a CPU to run its creations. Of course they would put a million cores to work at 'learning to code' so it would go much faster. Compare that with robotics, where it's so expensive to run your agents. I think this direction really has a shot.
What you'll get is the same thing as GPT-3: the equivalent of googling the prompt. You can google "implement a scientific calculator" and get multiple tutorials right now.
You'll still need humans to make anything novel or interesting, and companies will still need to hire engineers to work on valuable problems that are unique to their business.
All of these transformers are essentially trained on "what's visible to google", which also defines the upper bound of their utility
> You'll still need humans to make anything novel or interesting, and companies will still need to hire engineers to work on valuable problems that are unique to their business.
Give it 10 years :) GPT-10 will probably be able to replace a sizeable proportion of today's programmers. What will GPT-20 be able to do?
After their new CEO came in, former president of YC, they closed off everything and took a lot of investment. Only thing that's open about them is the name.
"Teapot in the shape of brain coral" yields the opposite. The topology is teapot-esque. The texture composed of coral-like appendages. Sorry if this is overly semantic, I just happen to be in a deep dive in Shape Analysis at the moment ;)
>>> DALL·E appears to relate the shape of a half avocado to the back of the chair, and the pit of the avocado to the cushion.
That could be human bias recognizing features the generator yields implicitly. Most of the images appear as "masking" or "decal" operations. Rather than a full style transfer. In other words the expected outcome of "soap dispenser in the shape of hibiscus" would resemble a true hybridized design. Like an haute couture bottle of eau du toilette made to resemble rose petals.
I was thinking "no way it can draw a pile of glasses, the lighting alone is difficult and it's so obscure" and then it just drew a pile of eyeglasses. I...
Context Free is incredibly fun but it's not machine learning or even AI. And it's very easy to understand how things go from defintions of shape, rotations, translations, etc, to the finished image.
Who knows, maybe in a few years you will be amazed at the new universal transformer chip that runs on 20W of power and can do almost any task. No need for retraining, just speak to it, show it what you need. Even prompt engineering has been automated (https://arxiv.org/abs/2101.00121) so no more mystery. So much for the new hot job of GPT prompt engineer that would replace software dev.
I think that they're correct saying that GPT-3 isn't revolutionary, since it just demonstrates the power of scaling. However I would argue that the underlying architecture, the Transformer (GP(T)), is/was/will be revolutionary.
Maybe I'm missing something but does it say what library of images was used to train this model? I couldn't quite understand the process of building DALL-E. Did they have a large database of labeled images and they combined this database with GPT-3?
Maybe I'm cynical, but I'm really skeptical. What if this is just some poor image generation code and a few hundred minimum wage workers manually assembling the examples? Unless I can feed it arbitrary sentences we can never know.
I would be disappointed, but not surprized if OpenAI turns out to be the Theranos of AI...
Does this address NLP skeptics' concerns that Transformer models don't "understand" language?
If the AI can actually draw an image of a green block on a red block, and vice versa, then it clearly understands something about the concepts "red", "green", "block", and "on".
I think it is safe to say that learning a joint distribution of vision + language, is fully possible at this stage, demonstrating by this work.
But 'understanding' itself needs to be further specified, in order to be tested even.
What strikes me most is the fidelity of those generated images, matching the SOTA from GAN literature with much more variety, without using the GAN objective.
It seems Transformer model might be the best neural construct we have right now, to learn any distribution, assuming more than enough data.
Try a large block on a small block. As the authors also have noted in their comments the success rate is nearly zero. One may wonder why. Maybe because that's something you see rarely in photos? At the end, it doesn't "understand" the meaning of the words.
There are examples on twitter showing it doesn't really understand spatial relations very well. Stuff like "red block on top of blue block on top of green block" will generate red, green, and blue blocks, but not in the desired order.
The root-case of skepticism has always been that while Transformers do exceptionally well on finite-sized tasks, they lack any fully recursive understanding of the concepts.[0]
A human can learn basic arithmetic, then generalize those principles to bigger number arithmetic, then go from there to algebra, then calculus, then so. Successively building on previously learned concepts in a fully recursive manner. Transformers are limited by the exponential size of their network. So GPT-3 does very well with 2-digit addition and okay with 2-digit multiplication, but can't abstract to 6-digit arithmetic.
DALL-E is an incredible achievement, but doesn't really do anything to change this fact. GPT-3 can have an excellent understanding of a finite sized concept space, yet it's still architecturally limited at building recursive abstractions. So maybe it can understand "green block on a red block". But try to give it something like "a 32x16 checkerboard of green and red blocks surrounded by a gold border frame studded with blue triangles". I guarantee the architecture can't get that exactly correct.
The point is that, in some sense, GPT-3 is a technical dead-end. We've had to exponentially scale up the size of the network (12B parameters) to make the same complexity gains that humans make with linear training. The fact that we've managed to push it this far is an incredible technical achievement, but it's pretty clear that we're still missing something fundamental.
> So GPT-3 does very well with 2-digit addition and okay with 2-digit multiplication, but can't abstract to 6-digit arithmetic.
That sounds disappointing but what if instead of trying to teach it to do addition one would teach it to write source code for making addition and other maths operations instead?
Then you can ask it to solve a problem but instead of it giving you the answer it would give you source code for finding the answer.
So for example you ask it “what is the square root of five?” then it responds:
The accuracy largely comes from the fact that addition rarely requires carrying more than a single digit. So it's easy to pattern match from single digit problems that it was previously trained on.
With multiplication, which requires much more extensive cross-column interaction, accuracy falls off a cliff with anything more than a few digits.
> The accuracy largely comes from the fact that addition rarely requires carrying more than a single digit. So it's easy to pattern match from single digit problems that it was previously trained on.
(note that GPT-3 is never told that the BPE [580] is composed of the digits [5], [8], and [0]. It has to guess this from the contexts [580] occurs in.)
> With multiplication, which requires much more extensive cross-column interaction, accuracy falls off a cliff with anything more than a few digits.
You couldn't learn long multiplication if you had to use BPEs, were never told how BPEs worked or corresponded to sane encodings, were basically never shown how to do multiplication, and were forced to do it without any working out.
Quick, what's 542983 * 39486? No writing anything down, you have to output the numbers in order, and a single wrong digit is a fail. (That's easy mode, I won't even bother asking you to do BPEs.)
ML models can learn multiplication, obviously they can learn multiplication, they just can't do it in this absurd adversarial context. GPT-f[1] was doing Metamath proofs on 9-digit division (again, a vastly harder context, they involve ~10k proof steps) with 50-50 accuracy, and we have a toy proof of concept[2] for straight multiplication.
In retrospect this was a pretty poorly framed point of mine. The point about addition being harder than stated because BPEs break at arbitrary points doesn't make much sense given the previous comment of mine pointed out that GPT-3 can only do addition with comma separators, that mostly (if not quite entirely) defeat that problem. GPT-3 does have to still know the various constructions and additions of 3-digit spans, but that's within its capabilities.
According to your definition of understanding, this program understands something about the concept RED. But the code is just dealing with arbitrary values in memory (e.g. RED = 0xFF0000)
I'm not sure how to feel, because I had this exact same thought. The evolution of porn from 320x200 EGA on a BBS, to usenet (alt.binaries,pictures.erotica, etc.) on XVGA (on an AIX Term), to the huge pool of categories on today's porn sites, which eventually became video and bespoke cam performers... Is this going to be some new weird kind of porn that Gen Alpha normalizes?
286 comments
[ 4.5 ms ] story [ 242 ms ] threadWith the ability to construct complex 3D scenes, surely the next step would be for it to ingest YouTube videos or TV/movies and be able to render entire scenes based on a written narration and dialogue.
The results would likely be uncanny or absurd without careful human editorial control, but it could lead to some interesting short films, or fan-recreations of existing films.
If you are talking about 24 frames per second, then theoretically one second of video could require 24 times as much processing power. And 100 seconds 2400 X. Obviously that's just a random guess but surely it is much more than for individual images.
But I'm sure we'll get there.
At this point I have replaced a significant amount of creative workers with AI for personal usage, for example:
- I use desktop backgrounds generated by VAEs (VD-VAE)
- I use avatars generated by GANs (StyleGAN, BigGAN)
- I use and have fun with written content generated by transformers (GPT3)
- I listen to and enjoy music and audio generated by autoencoders (Jukebox, Magenta project, many others)
- I don't purchase stock images or commission artists for many previous things I would have when a GAN exists that already makes the class of image I want
All of this has happened in that last year or so for me, and I expect that within a few more years this will be the case for vastly more people and in a growing number of domains.
Couldn't any creator of images that a model was trained on sue for copyright infringement?
Or do great artists really just steal (just at a massive scale)?
>Models in general are generally considered “transformative works” and the copyright owners of whatever data the model was trained on have no copyright on the model. (The fact that the datasets or inputs are copyrighted is irrelevant, as training on them is universally considered fair use and transformative, similar to artists or search engines; see the further reading.) The model is copyrighted to whomever created it.
Source (scroll up slightly past where it takes you): https://www.gwern.net/Faces#copyright
"Models in general are generally considered “transformative works” and the copyright owners of whatever data the model was trained on have no copyright on the model. (The fact that the datasets or inputs are copyrighted is irrelevant, as training on them is universally considered fair use and transformative, similar to artists or search engines; see the further reading.) The model is copyrighted to whomever created it. Hence, Nvidia has copyright on the models it created but I have copyright under the models I trained (which I release under CC-0)."
If I write a copyrighted text on a book, then I print a million other texts on top of it, in both white an black, mixing it all up to be like white noise, would the original authors have a claim?
https://arxiv.org/abs/1802.08232
Worse, sometimes the input data is illegal to distribute for other reasons than copyright.
But the profession for creative individuals consists of much more than highly-paid well-credentialed individuals working at well-known US corporations. There are millions of artists that just do quick illustrations, logos, sketches, and so on, on a variety of services, and they will be replaced far before Pixar is.
> - I listen to and enjoy music and audio generated by autoencoders (Jukebox, Magenta project, many others)
Really, you've "replaced" normal music and books with these? Somehow I doubt that.
The demo inputs here for DALL-E are curated and utilize a few GPT-3 prompt engineering tricks. I suspect that for typical unoptimized human requests, DALL-E will go off the rails.
But there is still a lot of room for more clever architectures to get around that limitation. (e.g. Shortformer)
I want the stuff that no human being could have made - not the things that could pass for genuine works by real people.
Unfortunately many generations fail to hit that.
Frankly, I think the "AI will replace jobs that require X" angle of automation is borderline apocalyptic conspiracy porn. It's always phrased as if the automation simply stops at making certain jobs redundant. It's never phrased as if the automation lowers the bar to entry from X to Y for /everyone/, which floods the market with crap and makes people crave the good stuff made by the top 20%. Why isn't it considered as likely that this kind of technology will simply make the best 20% of creators exponentially more creatively prolific in quantity and quality?
I think that's well within the space of reasonable conclusions. For as much as we are getting good at generating content/art, we are also therefore getting good at assisting humans at generating it, so it's possible that pathway ends up becoming much more common.
It’s like efficient market hypothesis: markets are efficient because arbitrage, which is highly profitable, makes them so. But if they are efficient, how can arbitrageurs afford to stay in business? In practice, we are stuck in a half-way house, where markets are very, but not perfectly, efficient.
I guess in practice, the pie for humans will keep on shrinking, but won’t disappear too soon. Same as horse maintenance industry, farming and manufacturing, domestic work etc. Humans are still needed there, just a lot less of them.
I think AlphaGo was a great in-domain example of this. I definitely see things I'd refer to colloquially as 'creativity' in this DALL-E post, but you can decide for yourself, but that still isn't claiming it matches what some humans can do.
If I train an AI on classical paintings, can it ever invent Impressionism, Cubism, Surrealism? Can it do irony? Can it come up with something altogether new? Can it do meta? “AlphaPaint, a recursive self-portrait”?
Maybe. I’m just not sure we have seen anything in this dimension yet.
I see your point, but it's an unfair comparison: if you put a human in a room and never showed them anything except classical paintings, it's unlikely they would quickly invent cubism either. The humans that invented new art styles had seen so many things throughout their life that they had a lot of data to go off of. Regardless, I think we can do enough neural style transfer already to invent new styles of art though.
Vast majority of human generated content is not very novel or creative. I'm guessing less than 1% of professional human writers or composers create something original. Those people are not in any danger to be replaced by AI, and will probably be earning more money as a result of more value being placed on originality of content. Humans will strive (or be forced) to be more creative, because all non-original content creation will be automated. It's a win-win situation.
Most arbitrageurs cannot stay in the business, it's the law of diminishing returns. Economies of scale eventually prevent small individual players to profit from the market, only a few big-ass hedge funds can stay, because due to their investments they can get preference from exchanges (significantly lower / zero / negative fees, co-located hardware, etc.) which makes the operation reasonable to them. With enough money you can even build your own physical cables between exchanges to outperform the competitors in latency games. I'm a former arbitrageur, by the way :)
Same with AI-generated content. You would have to be absolutely brilliant to compete with AI. Only a few select individuals would be "allowed" to enter the market. Not even sure that it has something to do with the quality of the content, maybe it's more about prestige.
You see, there already are gazillions of decent human artists, but only a few of them are really popular. So the top-tier artists would probably remain human, because we need someone real to worship to. Their producers would surely use AI as a production tool, depicting it as a human work. But all the low-tier artists would be totally pushed out of the market. There will be simply no job for a session musician or a freelance designer.
I won't say many of those things are creativity driven. There are more like auto assets generation.
One use case of such model would be in gaming industry, to generate large amount of assets quickly. This process along takes years, and more and more expensive as gamers are demanding higher and higher resolution.
AI can make this process much more tenable, bring down the overall cost.
Do you have a GPT-3 key?
Broke: Use a text encoder to feed text data to an image generator, like a GAN.
Woke: Use a text and image encoder as the same input to decode text and images as the same output
And yet, due to the magic of Transformers, it works.
From the technical description, this seems feasible to clone given a sufficiently robust dataset of images, although the scope of the demo output implies a much more robust dataset than the ones Microsoft has offered publicly.
"A photo of a iPhone from the stone age."
"Adolf Hitler pissing against the wind and enjoying it."
"Painting: Captain Jean-Luc Picard crossing of the Delaware River in a Porsche 911".
I'm curious if they do a backward pass here, would probably have value. They seem to describe sticking the text tokens first meaning that once you start generating image tokens all the text tokens are visible. That would have the model learning to generate an image with respect to a prompt but you could also literally just reverse the order of the sequence to have the model also learn to generate prompts with respect to the image. It's not clear if this is happening.
A network optimizing for both use cases (e.g. the training set is half 256 + 1024, half 1024 + 256) would likely be worse than a model optimizing for one of the use cases, but then again models like T5 argue against it.
> Similar to the rejection sampling used in VQVAE-2, we use CLIP to rerank the top 32 of 512 samples for each caption in all of the interactive visuals. This procedure can also be seen as a kind of language-guided search16, and can have a dramatic impact on sample quality.
> CLIP pre-trains an image encoder and a text encoder to predict which images were paired with which texts in our dataset. We then use this behavior to turn CLIP into a zero-shot classifier. We convert all of a dataset’s classes into captions such as “a photo of a dog” and predict the class of the caption CLIP estimates best pairs with a given image.
[1] https://openai.com/blog/clip/
At some point we'll have so many models based on so many other models it will longer longer be possible to tell which techniques are really involved.
https://github.com/openai/
Some pics are of drinking glasses and some are of eye glasses, and one has both.
Prompt: a Windows GUI executable that implements a scientific calculator.
You'll still need humans to make anything novel or interesting, and companies will still need to hire engineers to work on valuable problems that are unique to their business.
All of these transformers are essentially trained on "what's visible to google", which also defines the upper bound of their utility
Give it 10 years :) GPT-10 will probably be able to replace a sizeable proportion of today's programmers. What will GPT-20 be able to do?
Strongly recommend watching the whole video!
This is impressive.
Is OpenAI going to offer this as a closed paywalled service? Once again wondering how the “open” comes into play.
>>> DALL·E appears to relate the shape of a half avocado to the back of the chair, and the pit of the avocado to the cushion.
That could be human bias recognizing features the generator yields implicitly. Most of the images appear as "masking" or "decal" operations. Rather than a full style transfer. In other words the expected outcome of "soap dispenser in the shape of hibiscus" would resemble a true hybridized design. Like an haute couture bottle of eau du toilette made to resemble rose petals.
The name DALL-E is terrific though!
Another good example is the "collection of glasses" on the table. It makes both glassware and eyeglasses!
I can't remember its name.
EDIT:
https://www.contextfreeart.org/
I wrote a blog post on that a few months ago after playing a bit with GPT-3, and it holds up. https://news.ycombinator.com/item?id=23891226
To be honest, it's not where I'd like to see efforts in the field go.
Not because I'm afraid of AI taking over, but because I'd rather have humans recreate something comparable to a human brain (functionality wise).
I would be disappointed, but not surprized if OpenAI turns out to be the Theranos of AI...
If the AI can actually draw an image of a green block on a red block, and vice versa, then it clearly understands something about the concepts "red", "green", "block", and "on".
But 'understanding' itself needs to be further specified, in order to be tested even.
What strikes me most is the fidelity of those generated images, matching the SOTA from GAN literature with much more variety, without using the GAN objective.
It seems Transformer model might be the best neural construct we have right now, to learn any distribution, assuming more than enough data.
https://twitter.com/peabody124/status/1346565268538089483
A human can learn basic arithmetic, then generalize those principles to bigger number arithmetic, then go from there to algebra, then calculus, then so. Successively building on previously learned concepts in a fully recursive manner. Transformers are limited by the exponential size of their network. So GPT-3 does very well with 2-digit addition and okay with 2-digit multiplication, but can't abstract to 6-digit arithmetic.
DALL-E is an incredible achievement, but doesn't really do anything to change this fact. GPT-3 can have an excellent understanding of a finite sized concept space, yet it's still architecturally limited at building recursive abstractions. So maybe it can understand "green block on a red block". But try to give it something like "a 32x16 checkerboard of green and red blocks surrounded by a gold border frame studded with blue triangles". I guarantee the architecture can't get that exactly correct.
The point is that, in some sense, GPT-3 is a technical dead-end. We've had to exponentially scale up the size of the network (12B parameters) to make the same complexity gains that humans make with linear training. The fact that we've managed to push it this far is an incredible technical achievement, but it's pretty clear that we're still missing something fundamental.
[0] https://arxiv.org/pdf/1906.06755.pdf
That sounds disappointing but what if instead of trying to teach it to do addition one would teach it to write source code for making addition and other maths operations instead?
Then you can ask it to solve a problem but instead of it giving you the answer it would give you source code for finding the answer.
So for example you ask it “what is the square root of five?” then it responds:
This is false, GPT-3 can do 10-digit addition with ~60% accuracy, with comma separators. Without BPEs it would doubtlessly manage much better.
With multiplication, which requires much more extensive cross-column interaction, accuracy falls off a cliff with anything more than a few digits.
Again, not at all true due to BPEs.
(note that GPT-3 is never told that the BPE [580] is composed of the digits [5], [8], and [0]. It has to guess this from the contexts [580] occurs in.)> With multiplication, which requires much more extensive cross-column interaction, accuracy falls off a cliff with anything more than a few digits.
You couldn't learn long multiplication if you had to use BPEs, were never told how BPEs worked or corresponded to sane encodings, were basically never shown how to do multiplication, and were forced to do it without any working out.
Quick, what's 542983 * 39486? No writing anything down, you have to output the numbers in order, and a single wrong digit is a fail. (That's easy mode, I won't even bother asking you to do BPEs.)
ML models can learn multiplication, obviously they can learn multiplication, they just can't do it in this absurd adversarial context. GPT-f[1] was doing Metamath proofs on 9-digit division (again, a vastly harder context, they involve ~10k proof steps) with 50-50 accuracy, and we have a toy proof of concept[2] for straight multiplication.
[1] https://arxiv.org/abs/2009.03393
[2] https://github.com/Thopliterce/transformer-arithmetic
According to your definition of understanding, this program understands something about the concept RED. But the code is just dealing with arbitrary values in memory (e.g. RED = 0xFF0000)
It looks like a variation on plain old image search engine, unreliable at that, as compared to exact matching.
But it has obvious application in design as it can create these interesting combinations of objects & styles. And I loved the snail-harp.
Donald Trump is Nancy Pelosi's and AOC's step-brother in a three-way in the Lincoln Bedroom.
Wow!