directly training a NeRF on a single image is a terribly unconstrained problem that would lead to a volume that looks bad when the viewpoint changes. the gist of render + use diffusion model to refine is a great idea though and core to our method! the details of how to use the diffusion model for this refinement was the challenge, but once we figured that out it... just worked :)
Silly replying to myself I know, but I had more thoughts. I'm an architect for 3D worlds and I am desperate, lol, for this kind of tool. I use both blender and grasshopper, but I use midjourney to think and prototype all the time. Obvious but it would be astonishing to have something like this for game worlds. I used another version of this to create "a forest emerging from an aircraft carrier" https://www.instagram.com/p/CiRfXKzpnLC/ but the technique didn't have good resolution yet (high fidelity).
Hi @sirianth, this is Ajay. Are you talking about the https://ajayj.com/dreamfields colab? Feel free to dm me on Twitter (@ajayj_) or email me if you're facing bugs. Dependencies keep shifting, really should have pinned versions originally...
Someone did implement DreamBooth for stable diffusion so I was imagining, like you say, this (implemented by someone else) in a couple months with DreamBooth + stable diffusion
Did we hit some sort of technical inflection point in the last couple of weeks or is this just coincidence that all of these ML papers around high quality procedural generation are just dropping every other day?
From the abstract: “We introduce a loss based on probability density distillation that enables the use of a 2D diffusion model as a prior for optimization of a parametric image generator. Using this loss in a DeepDream-like procedure, we optimize a randomly-initialized 3D model (a Neural Radiance Field, or NeRF) via gradient descent such that its 2D renderings from random angles achieve a low loss.”
This seems like basically plugging a couple of techniques together that already existed, allowing to turn 2D text-to-image into 3D text-to-image.
Time and time again these ML techniques are proving to be wildly modular and pluggable. Maybe sooner or later someone will build a framework for end to end text-to-effective-ML-architecture that will just plug different things together and optimize them.
True (I made such a proposal myself a few hours ago, albeit in vaguer terms). The thing is deployment infrastructure is good enough now that we can just treat it as modular signal flows and experiment a lot without having to engineer a whole pile of custom infrastructure for each impulsive experiment.
Same as it ever was, scientific revolutions arrive all at once, punctuating otherwise uneventful periods. As I understand, the present one is the product of the paper "Attention is all you need": https://arxiv.org/pdf/1706.03762.pdf.
> This seems like basically plugging a couple of techniques together that already existed [...]
In his Lex Fridman interview, John Carmack makes similar assertions about this prospect for AGI: That it will likely be the clever combination of existing primitives (plus maybe a couple novel new ones) that make the first AGI feasible in just a couple thousand lines of code.
That's a great example that reminds me of another one: there was nothing new about Bitcoin conceptually, it was all concepts we already had just in a new combination. IRC, Hashing, Proof of Work, Distributed Consensus, Difficulty algorithms, you name it. Aside from Base58 there wasn't much original other than the combination of those elements.
Hello Stavros, I agree. When I look at the goals that base58 sought to achieve, (eliminating visually similar characters) I couldn't help but wonder why more characters were not eliminated. There is quite a bit of typeface androgyny when you consider case and face.
That was a great interview. I really liked his perspective on how close we are to having AGI. His point is that there's only a few more things we need to figure out and then it will basically happen.
I also liked the analogy he made with his earlier work on 2D and 3D graphics engines where taking a few short cuts basically got him on a path to success. For a while we had this "almost" 3D capability long before the hardware was ready to do 3D properly. It's the same with AGIs. A few short cuts will get us AI that is pretty decent and can do some impressive things already - as witnessed by the recent improvements in image generation. It's not a general AI but it has enough intelligence that it can still do photo realistic images that make sense to us. There's a lot of that happening right now and just scaling that up is going to be interesting by itself.
Billions of creatures with stronger neutral networks, more parameters, better input have lived on earth for millions of years, but only now something like humans showed up. I fully expect AI to do everything animals can do pretty soon, but since whatever it is that differentiates humans didn't happen for million of years, there's good chance AGI research will get stuck at a similar point.
Nature has the advantage of self organisation and (partially because of that) parallelism, that's proved hard to mimic in man made devices. But on the other hand, nature also has obstacles such as energy consumption, procreation & development, and survival, that AI doesn't have to worry about.
I think finding a niche for humans has proved difficult especially because of those reasons, and AI can take those hurdles much easier.
It takes nature thousands of years to create a rock that looks like a face, just by using geology. A human can do that in a couple hours. And then this AI can generate 50 3d human faces per second (assuming enough CPU).
It could be that an AGI is around the corner, as they say. We might not be machines, but are way faster than nature at reaching places. We don't have the option of waiting for thousands of years.
"You're posting too fast" is a limit that's manually applied to accounts that have a history of "posting too many low-quality comments too quickly and/or getting into flame wars". You can email dang (hn@ycombinator.com) if you think it was applied in error, but if it's been applied to you more than once... you probably have a continuing problem with proliferating flame wars or posting otherwise combative comments.
Whose full paper submission deadline was also 2 days ago.
This should be further up than all the speculation about AI accelerationism. There's a very simple explanation why a lot of awesome papers come out right now, it's prestigious conference paper deadlines.
This isn't what is usually meant by "technological singularity". It is an inflection point where technology growth becomes incontrollable and unpredictable, usually theorized to be cause by a self improving agent (/AI) that becomes smarter with each of its iterations. This is still standard technological progress, human control, even if very fast
It's not really "human controlled". It's an evolutionary process, researchers are scanning the space of possibilities, each with a limited view, but in aggregate it has an emergent positive trend.
It’s basically when AI starts self-improving. I think this started with large language models. They are central to these developments. Complete autonomy is not required for AGI-nor therefore for the singularity.
My hot take is that we're merely catching up on until recently unutilized hardware improvements. There's nothing 'self-improving', it's largely "just" scaled up methods or new, clever applications of scaled up methods.
The pace at which methods scale up is currently a lot faster than hardware improvements, so unless these scaled up methods become incredibly lucrative (not impossible), I think it's quite likely we'll soon-ish (a couple years from now) see a slowdown.
It has become clear since alphaGo that intelligence is an emergent property of neural networks. Since then the time and cost requirements to create a useful intelligence have been coming down. The big change was in August when Stable Diffusion was able to run on consumer hardware. Things were already accelerating before August, but that has really kicked up the speed because millions of people can play around with it and discover intelligence applications, especially in the latent space.
The numbers of researchers and research labs scaled up that there is now many well funded teams with experience.
Public tooling and collaboration has reached a point where research happen across the open internet between researchers at a pace that wasn't before possible. Common Crawl, stable diffusion, hugging-face, etc...)
All the techniques that took years in small labs to prove as viable are now getting scaled up across data and people in front of our eyes.
This is crazy good - most prior text-to-3d models produced weird amorphous blobs that would kind of look like the prompt from some angles, but had no actual spatial consistency.
Blown away by how quickly this stuff is advancing, even as someone who's relatively cynical about AI art.
Why the downvote? I wasn't being sarcastic, it was a honest question, I'm really impressed how far this technology has come since GPT-3 2 years ago to DALl-E and Stable Diffusion ro Meta's text to video to this...
I was wondering the same thing in the other thread about Text to Video. Someone asked about 3D Blender models, which made me think about animating blender models. Bang, now on this thread we see animated images… it does feel like we can get to asking for a 3D environment, put on a VR glass and experience it. And with outpainting, that we can even change it in real time.
It’s totally sci-fi, and at the same time seems to be possible? I am amazed how even image generation evolved over the last year, but that’s just me daydreaming.
3D game and VR game (i.e. stereoscopic 3D game) is not the same experience.
But I agree that particular company is creating a bad marketing around the term Metaverse. (But pretty decent HW.) NVIDIA has much better footing with their Omniverse. But in the end, we all know the thing will be build as web browsers are today. USD + JS + WebRTC + WebXR can go a long way.
This is par for the course - there have been other instances where an 'anonymous' paper mentioned training on a cluster of TPUs that weren't publicly available yet - dead giveaway it was Google.
Lots of reasons to stay anonymous besides for hiding what org is behind the paper. Maybe they don't want to be kidnapped by the North Koreans and forced to produce new paens with "lost footage" to Kim Il-Sung.
A large portion of the ML community (rightly) discredits Google papers because:
- they rarely provide the data or code used so it's basically "i swear it works bro" research
- what they achieve is usually through having the most pristine dataset on the planet and is often unusable by other researchers
- other times they publish papers that are basically "we slightly modified this excellent open source paper, slapped an internal name on it and trained it on our proprietary dataset"
- sometimes they achieve remarkably little but their papers still get a shiny spot because they're a big name and sponsor all the conferences
- they've also been caught trying to patent/copyright ML techniques; disregarding that this is the same as privatizing math, these are often techniques they plainly didn't come up with
Also ever since OpenAI did their "we have to go closed-source for-profit to save humanity" PR campaign, every company that releases models that can achieve a large amount in NLP/CV gets dragged by the media and equated to Skynet.
IMO the biggest algorithmic advances made by Google such as the transformer have greatly pushed the field forward. The giant model's that will have similar variations released in the next 3 months actually aren't that important on a conceptual level except as a PoC.
While the transformer model is important that was 1 paper in an ocean they put out every year.
Also it’s one of the only papers they put out that falls completely outside what I put above. They released everything about it including the model code, pretrained weights, the techniques; and it took quite a while for the model to “catch on” while it was peer reviewed and reproduced by others.
Unclear to me what is going on, but there’s another URL that lists the authors names. Given it’s possible this change was done for reason, not linking to it, but strikes me as odd it’s still up. Anyone know what’s going on without causing problems for the authors?
Aware of the link, though you have not provided any clarification for why there are two links; strikes me as odd if authors are trying to post it anonymously that simple Google finds authors names.
I believe ICLR guidelines require the authors to submit papers and any supplementary materials (including links to webpages, videos, etc) without identifying information, but authors are not barred from public announcements on other forums. IIUC, the idea behind this policy was originally to accommodate author freedom to engage in common practices such as simultaneous submission to arxiv (which identifies the authors). To respect the double blind spirit of review, reviewers are asked not to actively search the web in attempt identify the authors. In the past, when social media promotion was less common, it was reasonably likely that reviewers would follow this guidance and would not have seen the arxiv submission, preserving the double blind nature of review in most cases. However, the use of social media in academia has radically changed in recent years, as more researchers use social media to keep up with the latest advancements, so promotion of papers in submission on platforms like Twitter can offer significant advantages to authors.
So, authors today often submit anonymously following the conference guidelines, but simultaneously post publicly elsewhere, walking a fine line as not to overstep the conference policies. This appears to be the case for this submission. Note, recently, some conferences, such as CVPR, have started to institute new policies forbidding social media promotion until acceptance, as they adapt to the changing landscape of social media promotion. If this were a CVPR submission, the authors would not be allowed to tweet publicly about their work yet, nor have the version of the webpage with their names visible.
Thanks, appreciate explanation - maybe it’s me, but found it odd, didn’t know what to make of it. Maybe it’s me, but if there not against authors posting with their identity public elsewhere, they should just have a button you’re able to click, a call warning, and then get to see the authors.
The thing that frightens me is that we are rapidly reaching broad humanity disrupting ML technologies without any of the social or societal frameworks to cope with it.
There were bigger disruptions in the past. The telegraph, railroads, explosives. "The Devils" by Dostoevsky is a great fictional account of what all these technological disruptions do to the fragile social order in the late 19th century Russia countryside. All of a sudden all these foreign people, ideas , technology and commerce start streaming in to these once isolated communities.
I'm usually not a fan of this general hand wringing / fear mongering around ML that a lot of people with too much time and not enough STEM background constantly bring up.
Stable diffusion has been made available to the public for quite a while now and if anything has disproved a lot of the ungrounded nonsense that made companies like OpenAI censor their generative models.
>I'm usually not a fan of this general hand wringing / fear mongering around ML that a lot of people with too much time and not enough STEM background constantly bring up.
What's with the STEM reference? Are you implying that STEM is related to intelligence and that people without a STEM background are not intelligent?
It is well known among academics (AKA STEM MAJORS) that human society is a chaotic system and that ML can change society for the better or for the worse... the outcome is basically unknown... It is therefore the intelligent choice to consider the negative consequences of this technology.
To not consider the other side indicates a lack of something.
SD has been out for about a month and it is first gen technology. If "quite a while" was 5 years, then sure, I'll agree.
But we are one month into the experiment, and the technology still struggles to get wholly authentic looking media. I'd say it would be wise to give it a few years before claiming victory.
I watched this humor group when I was little (think 30 years ago). Usually they are inoffensive, but they have this particular sketch were a (man crossdressed as a) woman complains about "his husband hitting her" ... and you can see that one of her eyes is black. With canned laughs during the whole thing.
This used to make people laugh. 30 years ago. Then we had social changes and gradually our perspective as a society shifted, and this humor piece looks ... grotesque.
That's how I interpret the "societal changes" the OP is referring to. We have shiny new things every week and there's no time to adapt to them.
I think what's happening here is that the flat-looking table is actually raised up in the center, in the shape of something like a smooth pyramid. There's dough painted on both sides of the rolling pin, but because of the curvature of the "table" you only see each side's dough when the camera is on that side of the pyramid.
The most incredible thing here is that this demonstrates a level of 3D understanding that I didn't believe existed in 2D image models yet. All of the 3D information in the output was inferred from the training set, which is exclusively uncurated and unsorted 2D still images. No 3D models, no camera parameters, no depth maps. No information about picture content other than a text label (scraped from the web and often incorrect!).
From a pile of random undifferentiated images the model has learned the detailed 3D structure and plausible poses and variants of thousands (millions?) of everyday objects. And all we needed to get that 3D information out of the model was the right sampling procedure.
They reapply noise to the potentially distorted image and then predict the de-noised version like the originally rendered first frame. So the image is at least internally consistent for the frame (to the extend the the system generates consistency whatsoever).
The example with a squirrel wearing a hoodie demonstrates an interesting edge case, the "front" of the squirrel (with hoodie over the head) show a normal hooded face as expected, but when you rotate to the "back" you get another face where the hoodie is low over the eyes. Each looks fine in isolation, but in aggregate it seems like we have a two-faced squirrel.
Yes, this is often a problem. We use view-dependent prompts (e.g. "cat wearing sunglasses, back view") but the pretrained 2D model often does not do a good job of interpreting non-canonical views and will put sunglasses on the back of the cats head (as well as the front).
Bad prompt, missing implied antecedent/ambiguous subject...
You may want:
Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either.
>Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either.
In short: natural language is not good enough and you need a DSL. If only the last 60 years of language research had warned us of this.
Up next: English sentences are ambiguous and need context information to parse correctly. Machine learning community in shambles.
The trouble is all those darn uninitiated and trying to create a generalized oracle to map their inspecific ramblings to what they mean to free them of having to actually communicate properly...
Actually, funnily enough, this has cross section with philosophy in a way most programmers scoff at; but communication is frigging hard, and worse, detecting when someone is trying to get something across, but just needs a nudge in the right direction to be able to find the language to explain it is really damn hard.
I run into it every time I get a haircut. I have no idea how to speak their language, so it's always "Uh... A little off the top and rounded at the back, I guess?"
Some of the "mesh exports" used as examples on the page actually show this, to some extent. Look specifically at the plush corgi's belly and the weird geometry on the underside of the lemur's book, and to a lesser extent the underside of the bedsheet-ghost.
Co-author here - we were also surprised :) The breadth of knowledge of the visual world embedded in these 2D models and what they unlock is astounding.
As far as I understand from a quick read of the paper, the 2D diffusion doesn't have a 3D understanding. It probably have some sort of local neighborhood understanding, aka small geometric transformation of objects map close to each other in the diffusion space (That's why like with latent spaces you can "interpolate" (https://replicate.com/andreasjansson/stable-diffusion-animat...) in the diffusion space).
But that's not really surprising because when you have enough data, even simple clustering methods group objects like faces by the direction they are looking to. With enough views even a simple L2 distance in pixel space allow t-SNE to do that.
They are injecting the 3D constraints via the NERF and an optimization process to add the consistency between the frames.
It's a deep dream process that optimize by alternating updates for 3D consistency, and updates for text-to-2Dimage correspondence. It's searching for a solution that satisfy these two constraints at the same time.
Even though they only need to run a single diffusion step to get the update direction, this optimization process is quite long : 1h30 (but they are not using things like instant Nerf (or even simple voxel grids) ).
But this will allow for creation of a dataset of 3D objects with corresponding text, which will then allow to train a diffusion model that will have a 3D understanding and will be able to generate 3D objects directly with a single diffusion process.
The model clearly has an understanding of the 3D structure of objects. If it didn't, using it to generate 3D models wouldn't work. The knowledge that the leg bone is connected to the knee bone, etc, isn't coming from NeRF, it's all in the "2D" model. Sure, maybe you could distill that knowledge into a different model architecture that is somehow natively 3D in order to improve the efficiency of sampling. But that's more or less just an optimization. What's interesting is the fact that the knowledge is in there already, learned without 3D input data.
Humans don't get natively 3D training data either. We don't have depth sensors or any other natively 3D senses. Our eyes only ever see 2D images and we learn 3D structure from that, so I guess these 2D image models are doing something analagous. Except they don't even have the benefit of stereo images!
its not an over simplification. The extra information from convergence is negligible... our eyes derive virtually identical information when looking at flat 2D pictures of 3D scenes. Evidence of this is everywhere in pictures.
I think you're both right! It is incredible that the 2D model knows enough about the visual world to produce many objects from all angles, but the 3D model is essential for gluing these views together, and in some ways can fill in the gaps the 2D model doesn't know about. Imagine just taking a huge collection of photographs of an object. While there is enough information in those photos to reconstruct the 3D object, I wouldn't personally call that collection of images "an understanding of 3D." In our case, the diffusion model is the collection of photos and the NeRF model + optimization procedure is what figures out how all those photos can be related to a shared underlying 3D representation. - ben p (author)
I think this is a misunderstanding of how these models work. The model does not understand anything at all. It's computing correlations. I could spend all day computing correlation without ever understanding what the correlations correspond to in the physical world. The correlations could still amount to a useful description of some physical phenomenon, but their interpretation requires much more than just the ability to compute correlations in some limited space.
If we elevate "correlations" to mean "understanding", we're quickly going to run out of words.
>The model clearly has an understanding of the 3D structure of objects.
And the submarine clearly can swim! :D
The model encompasses 3D information, but a model is not a thinking entity able to understand anything. Well, it might be argued that saying that the "model understands" is a metaphor, and used as such OK, fine. But that’s it. Analogies rarely scale well.
As for humans, have you ever heard of proprioception? If that is not "native 3D sensors", I have no idea of what a 3D sense might be.
Yes, this is amazing. The reason I am a bit less surprised is that I've been playing with textual inversion in the last few days. Just from six photos of me, it can create portraits showing me from any direction. That means it has a very good 3D model of my face, and that information is fully contained in the model plus the CLIP embedding obtained by textual inversion.
207 comments
[ 2.7 ms ] story [ 198 ms ] thread[1]: https://github.com/nv-tlabs/GET3D
Although they start with random initialization and a text prompt. It seems to work well. I now see no reason we can't start with image initialization!
> In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.
> “What are you doing?”, asked Minsky.
> “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied.
> “Why is the net wired randomly?”, asked Minsky.
> “I do not want it to have any preconceptions of how to play”, Sussman said.
> Minsky then shut his eyes.
> “Why do you close your eyes?”, Sussman asked his teacher.
> “So that the room will be empty.”
> At that moment, Sussman was enlightened.
http://www.catb.org/jargon/html/koans.html
You'll have to wait a few months for someone else to replicate it.
I'm also not sure someone couldn't cleverly figure out a way to use stable diffusion to write code based on a text prompt.
This seems like basically plugging a couple of techniques together that already existed, allowing to turn 2D text-to-image into 3D text-to-image.
They have others as well.
as with a majority of ML research
Plus "we did the same thing, but with 10x the compute resources".
But yeah.
https://arxiv.org/abs/2003.08934
Do this enough times and eventually the thing you have looks indistinguishable from something completely novel.
In his Lex Fridman interview, John Carmack makes similar assertions about this prospect for AGI: That it will likely be the clever combination of existing primitives (plus maybe a couple novel new ones) that make the first AGI feasible in just a couple thousand lines of code.
I also liked the analogy he made with his earlier work on 2D and 3D graphics engines where taking a few short cuts basically got him on a path to success. For a while we had this "almost" 3D capability long before the hardware was ready to do 3D properly. It's the same with AGIs. A few short cuts will get us AI that is pretty decent and can do some impressive things already - as witnessed by the recent improvements in image generation. It's not a general AI but it has enough intelligence that it can still do photo realistic images that make sense to us. There's a lot of that happening right now and just scaling that up is going to be interesting by itself.
I think finding a niche for humans has proved difficult especially because of those reasons, and AI can take those hurdles much easier.
It takes nature thousands of years to create a rock that looks like a face, just by using geology. A human can do that in a couple hours. And then this AI can generate 50 3d human faces per second (assuming enough CPU).
It could be that an AGI is around the corner, as they say. We might not be machines, but are way faster than nature at reaching places. We don't have the option of waiting for thousands of years.
and I agree with you!
And the OP comment its by the magnanimous/infamous AnigBrowl
You need to start doing AI legal admin ( I dont have the terms, but you may - we need legal language to control how we deal with AI)
and @dang - kill the gosh darn "posting too fast" thing
Jiminey Crickets I have talked to you abt this so many times...
I'm a kotlin programmer so it's a scary fun thought for me.
This should be further up than all the speculation about AI accelerationism. There's a very simple explanation why a lot of awesome papers come out right now, it's prestigious conference paper deadlines.
WTF - the singularity is closer than we thought!!!
Whatever it is, this is a massive phase shift.
The pace at which methods scale up is currently a lot faster than hardware improvements, so unless these scaled up methods become incredibly lucrative (not impossible), I think it's quite likely we'll soon-ish (a couple years from now) see a slowdown.
The numbers of researchers and research labs scaled up that there is now many well funded teams with experience.
Public tooling and collaboration has reached a point where research happen across the open internet between researchers at a pace that wasn't before possible. Common Crawl, stable diffusion, hugging-face, etc...)
All the techniques that took years in small labs to prove as viable are now getting scaled up across data and people in front of our eyes.
Blown away by how quickly this stuff is advancing, even as someone who's relatively cynical about AI art.
It’s totally sci-fi, and at the same time seems to be possible? I am amazed how even image generation evolved over the last year, but that’s just me daydreaming.
How long then until we get photorealistic AI generated 3D games and experiences?
But I agree that particular company is creating a bad marketing around the term Metaverse. (But pretty decent HW.) NVIDIA has much better footing with their Omniverse. But in the end, we all know the thing will be build as web browsers are today. USD + JS + WebRTC + WebXR can go a long way.
- they rarely provide the data or code used so it's basically "i swear it works bro" research
- what they achieve is usually through having the most pristine dataset on the planet and is often unusable by other researchers
- other times they publish papers that are basically "we slightly modified this excellent open source paper, slapped an internal name on it and trained it on our proprietary dataset"
- sometimes they achieve remarkably little but their papers still get a shiny spot because they're a big name and sponsor all the conferences
- they've also been caught trying to patent/copyright ML techniques; disregarding that this is the same as privatizing math, these are often techniques they plainly didn't come up with
Also ever since OpenAI did their "we have to go closed-source for-profit to save humanity" PR campaign, every company that releases models that can achieve a large amount in NLP/CV gets dragged by the media and equated to Skynet.
Also it’s one of the only papers they put out that falls completely outside what I put above. They released everything about it including the model code, pretrained weights, the techniques; and it took quite a while for the model to “catch on” while it was peer reviewed and reproduced by others.
Something something broken clock
Once the paper is accepted (or rejected) the names may be revealed.
Though, in reality, the reviewers can often easily tell who wrote the paper.
So, authors today often submit anonymously following the conference guidelines, but simultaneously post publicly elsewhere, walking a fine line as not to overstep the conference policies. This appears to be the case for this submission. Note, recently, some conferences, such as CVPR, have started to institute new policies forbidding social media promotion until acceptance, as they adapt to the changing landscape of social media promotion. If this were a CVPR submission, the authors would not be allowed to tweet publicly about their work yet, nor have the version of the webpage with their names visible.
Stable diffusion has been made available to the public for quite a while now and if anything has disproved a lot of the ungrounded nonsense that made companies like OpenAI censor their generative models.
What's with the STEM reference? Are you implying that STEM is related to intelligence and that people without a STEM background are not intelligent?
It is well known among academics (AKA STEM MAJORS) that human society is a chaotic system and that ML can change society for the better or for the worse... the outcome is basically unknown... It is therefore the intelligent choice to consider the negative consequences of this technology.
To not consider the other side indicates a lack of something.
But we are one month into the experiment, and the technology still struggles to get wholly authentic looking media. I'd say it would be wise to give it a few years before claiming victory.
I watched this humor group when I was little (think 30 years ago). Usually they are inoffensive, but they have this particular sketch were a (man crossdressed as a) woman complains about "his husband hitting her" ... and you can see that one of her eyes is black. With canned laughs during the whole thing.
This used to make people laugh. 30 years ago. Then we had social changes and gradually our perspective as a society shifted, and this humor piece looks ... grotesque.
That's how I interpret the "societal changes" the OP is referring to. We have shiny new things every week and there's no time to adapt to them.
https://dreamfusion-cdn.ajayj.com/gallery_sept28/crf20/a_DSL...
But if you look closely, the pin looks like it's actually rolling across the dough as the camera orbits.
From a pile of random undifferentiated images the model has learned the detailed 3D structure and plausible poses and variants of thousands (millions?) of everyday objects. And all we needed to get that 3D information out of the model was the right sampling procedure.
The example with a squirrel wearing a hoodie demonstrates an interesting edge case, the "front" of the squirrel (with hoodie over the head) show a normal hooded face as expected, but when you rotate to the "back" you get another face where the hoodie is low over the eyes. Each looks fine in isolation, but in aggregate it seems like we have a two-faced squirrel.
It'll just make up some colours and geometries that don't contradict anything it already knows from the defined perspectives.
Or leave it empty.
Bad prompt, missing implied antecedent/ambiguous subject...
You may want:
Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either.
In short: natural language is not good enough and you need a DSL. If only the last 60 years of language research had warned us of this.
Up next: English sentences are ambiguous and need context information to parse correctly. Machine learning community in shambles.
The trouble is all those darn uninitiated and trying to create a generalized oracle to map their inspecific ramblings to what they mean to free them of having to actually communicate properly...
Actually, funnily enough, this has cross section with philosophy in a way most programmers scoff at; but communication is frigging hard, and worse, detecting when someone is trying to get something across, but just needs a nudge in the right direction to be able to find the language to explain it is really damn hard.
I run into it every time I get a haircut. I have no idea how to speak their language, so it's always "Uh... A little off the top and rounded at the back, I guess?"
-the point
But that's not really surprising because when you have enough data, even simple clustering methods group objects like faces by the direction they are looking to. With enough views even a simple L2 distance in pixel space allow t-SNE to do that.
They are injecting the 3D constraints via the NERF and an optimization process to add the consistency between the frames.
It's a deep dream process that optimize by alternating updates for 3D consistency, and updates for text-to-2Dimage correspondence. It's searching for a solution that satisfy these two constraints at the same time.
Even though they only need to run a single diffusion step to get the update direction, this optimization process is quite long : 1h30 (but they are not using things like instant Nerf (or even simple voxel grids) ).
But this will allow for creation of a dataset of 3D objects with corresponding text, which will then allow to train a diffusion model that will have a 3D understanding and will be able to generate 3D objects directly with a single diffusion process.
Humans don't get natively 3D training data either. We don't have depth sensors or any other natively 3D senses. Our eyes only ever see 2D images and we learn 3D structure from that, so I guess these 2D image models are doing something analagous. Except they don't even have the benefit of stereo images!
I think that would be an oversimplification. We do have some 3D information from focus and eye convergence.
If we elevate "correlations" to mean "understanding", we're quickly going to run out of words.
At some point we have to accept that when you layer simple ops on top of simple ops enough times you get complex behavior.
And the submarine clearly can swim! :D
The model encompasses 3D information, but a model is not a thinking entity able to understand anything. Well, it might be argued that saying that the "model understands" is a metaphor, and used as such OK, fine. But that’s it. Analogies rarely scale well.
As for humans, have you ever heard of proprioception? If that is not "native 3D sensors", I have no idea of what a 3D sense might be.
A machine with 3d understanding from 2d samples predates computers
"For he spoke, and it came to be;
he commanded, and it stood firm."
Psalm 33:9, NIV
:)
The samples are lacking definition, but they're otherwise spatially stable across perspectives.
That's something that's been struggled with for years.