We're probably lucky that they chose to model fully clothed subjects. Based on the smearing we see in some translated details like the fingers and examples of holes in skirts we could have some comically bad output.
Same as many other instances of specialized ML models, their results look good generally, but completely fall apart under scrutiny. Remember GPT-3 blogpost writing?
Although, they are getting better years after years, so maybe we'll be out of the uncanny valley at some point ?
I have been recently looking at several "3d scanning" from monocular images/videos solutions and although they make for impressive demos I'm afraid they are a few years away from robust results.
On one hand you have traditional photogrammetry based on classical image descriptors (from the beginning of the 2000s). A nice open source solution is Meshroom [1]. I have very little experience as I have just tinkered a little but I would say they work OK with geometry of medium complexity and detailed textures. They fail horribly with untextured objects, like for example a candle. They are semi-automatic: you upload some photos and run the pipepiline but for best results it's easy to tweak as there are a lot of knobs to try.
On the other hand you have these deep learning research papers. A lot of them have Colab notebooks you can try yourself (which is awesome, thank you). They can give you these very nice demos for some kind of objects (the ones we have a lot of training data, like human bodies), but they are not truly general and sometimes will generate artifacts. There is some opportunity here if they are integrated in a semi-supervised workflow, like this [2].
What is the timeline between these papers and an app that can take me from a photo to doing an animated dance flip to impress my kids as in their output example?
Pretty sure that's not what is intended. They used to sell supposed x-ray googles to gullible people, the ads implying you'd be able to see people naked.
> They used to sell supposed x-ray googles to gullible people, the ads implying you'd be able to see people naked.
This is actually technology we already have today. You wouldn't want X-ray goggles, but infrared "goggles" would do the job just fine. Like X-rays, infrared light will pass right through clothes; unlike X-rays, it won't pass through skin.
Hook up an IR camera to a VR headset and you're there.
If an ai can reconstruct a view of the far side of something from data like the near side view and the shadows & other effects on the surrounding environment, then it can do the same thing to reconstruct a view of what's underneath clothing, theoretically, which may be reasonably restated as eventually.
> If an ai can reconstruct a view of the far side of something from data like the near side view and the shadows & other effects on the surrounding environment, then it can do the same thing to reconstruct a view of what's underneath clothing
This is actually not something that people want. The body within clothes is generally constrained by them in ways that look odd if you can't see the clothes; what people really want is to see what someone would look like if they were naked, not what they would look like if their clothes were invisible.
Well, with the right training data I imagine you could make something that produce an imaginary model of what the person might look like naked in the same pose but unconstrained by clothing. Obviously though, like the system in the link, for everything it can't see it's just constructing it based on what it's been trained with.
I would assume the main use would be to see people you know, not to enhance yourself, but, I guess there would be a market for that too. Especially for content producers.
I'd love to see indie game developers use these tools in combination with DALL-E to recreate the style of the PlayStation-era Final Fantasy games. 3d characters in 2d pre-rendered worlds. Presumably with the right tooling -- creating imagined geometry in DALL-E and generating boundaries programmatically or by hand, character models from friends / cosplayers found via craigslist and open animation sets -- you'd be able to generate a game of Final Fantasy VIII's caliber with team as small as a lone developer in only a few years, maybe shorter.
I would propose that the reason you are nostalgic for the FF games of past is because of the unique creative vision of those games, which is not something that DALL-E can help you with much yet, though it's absolutely astonishing at mimicking other's creative styles. (there may still be a way it could help, if an artist created part of the style and used DALL-E to augment the visual production pipeline)
DALL-E can't help in art direction but (somewhat surprisingly) GPT-3 plus a human scanning Pinterest can be an art director. I've spent hundreds of OpenAI dollars on this because it works.
To me, that means a version DALL-E which can do art direction without me in the loop at all is not too far off.
As an lone indie dev making an AI-generated FOSS Final Fantasy, I can confirm it's possible :D. That era of games really fits well with the kind of content that AI is capable of generating today.
I've been primarily tackling the synthesis from the GPT-3 + 3D animation side but the 2d prerender idea is genius. I'll need to try to hook it up one of these weekends. The only hard part, it seems, is finding the navmesh.
How is this reconstruction and not just construction? There is no data there to reconstruct. It's just made up based on some hinted guessing and a lot of assumptions.
There’s no accepted threshold between the two. The input image is obviously highly constraining the output model - the pose and clothing color are consistent - so this is within the domain norm to call this a reconstruction.
Not exactly the same thing but I have been thinking about the idea of 3D construction of cloth/jeans you want to purchase onto your own body, so that you have better idea before placing order, which could be really useful for online shopping.
I was thinking in game graphic way (bottom-up), such as building 3D model of myself, shopping website providing 3D models of their clothes, and then it would be a "fitting" problem to properly place clothes onto the body.
From this paper it seems that it is not necessary to have low-level data to achieve the objective.
Unfortunately the human body deforms from clothing...
I have most faith in energy-dispersive CT or MRT to provide the required information about bone shapes and tissue types to extrapolate how a garment works while moving.
Even much more so for shoes that I think need shape customization the most.
In the early 2010s, I did a security evaluation of Styku's Kinect-based fitting room. Back then, it was in a similar vein as what you described - the sensors would build a 3D model of your body, and then recommend the jeans that would fit best.
Their original idea was even better, IMO. It was supposed to send the model to an on-demand automated clothing factory that would custom-sew entire pieces of clothing to your measurements.
I think the technology wasn't quite there yet at the time. Even with four first-gen Kinects, the body model was probably too inaccurate to compare to an actual tailor. They seem to have moved away from clothing in general since then.
Is it just me or does it look like the AI is just pasting a blurry version of the front to fill in the back? You can see hints of necklines on backs, or the red from a dress showing where a black overcoat should be.
It knows the neck area can’t be the same between front and back, but it doesn’t know if it should sample the skin or cloth textures to fill the gap, so it just takes a blend of both.
I guess, by hedging this way, it managed to improve its evaluation score without having to learn how clothes actually work.
It's definitely not perfect, but I'd argue that it is better than a "blurry front paste". For example, it doesn't place breasts on the back. Also, for the polo shirt example the model is doing significantly better than copying the v-line of the front of the shirt and shows a round collar. I think it is pretty impressive and one could argue the model has somewhat learned what a human looks like from the back given the clothing of the front, even though it is quite blurry.
Yeah, it's photorealistic if your photo is about 200px high and you present the 3d models at the same size. As soon as you zoom in, it loses some of the wow factor, they look like models from The Sims.
This will eventually link up with MetaHuman, DeepMotion and ZivaDynamics to get the high res end result. There a lot of money to be made. It is will happen in the next few years. We're very close.
Speaking as a former employee of a 3D startup that successfully exit - those technologies are never a game changer, they need a great business foundation to operate , such as a successful retail bussiness. It limits the business value of such startup. To make things worse, we have to always expecting the next big thing will come, like meta verse today, Kinect 10 years ago.
They could be a great application for niches, though. Prototyping 3d games or movies can be helped greatly by such techniques.
If you can take a photo (effort required: seconds) and turn it into a textures low- to mid-poly model within seconds, that's a huge time saver right there.
Combined with other techniques [1], [2] this could indeed present a viable business model - again for certain niches. It's the same old routine: identify potential, present solution, profit (if only for a limited time). On its own this might not look too useful/impressive, but I'm certain there's people out there who could already benefit greatly from this work.
It's not always about "the next big thing" or "turning an entire industry on its head". I could see this work its way into various products without much fanfare and gazillion dollar valuations.
Combine this with a webcam pointed at a busy street, some pose estimation and an auto rigger and you can steal people's identities with this in an automated way.
Example? A fashion billboard that doesn't use model photos but renders passers-by in fancy new clothes. Of course, the gathered data gets syphoned of into a database somewhere.
The example actually sounds quite amazing. There's much simpler ways (that are already in use, btw.) to "steal" people's identities.
Smartphones and CCTV cameras are much more effective sources of information than a smart billboard anyway. Smart billboards, however, sound quite awesome.
You have to consider who the collected data is available to. CCTV in public spaces is usually the domain of law enforcement and thus the related data is available to law enforcement and potentially other government agencies, but not normally to private companies. A smart billboard and related data would be owned by a private company who may go on and process and sell it behind your back. It's complex.
Computing and related tech is an enabler of so many new possible applications. It has become really tough to develop something useful for consumers that can be commercially successful, but doesn't at least have the potential to be turned into something creepy.
It is unnerving how many people are OK with governments using surveillance tech but draw the line with private corporations. At the end of the day they are just organizations made up of fallible humans. Either using advanced surveillance technology worries me deeply.
Corporations have an increased ability to do things w/o oversight, but the government can literally do whatever it wants when it wants as long as those in power allow it. Government has the monopoly on power and violence. This concern is increased even more in democracies where there is great division. One side can abuse government to enforce their aims on an unwilling opposing side.
Additionally, corporations generally just want to make money so they are fairly predictable and easy to contain (i.e take your business somewhere else if you don't want to deal with them).
Government operates at the beck and call of political whims which, let's face it, are more often than not emotional and knee-jerk reactions. Lastly, only government can make it illegal to avoid surveillance and put you in jail for it. The worst a corp can do is close your account or something.
You're not wrong about whims of governments. But that doesn't mean that companies are more predictable or controllable. Both types of organizations have their own ways in which they can screw you.
Anyway, I think this is getting too far from the actual topic here. I'll leave it at that.
What would much cooler is if you had a drawing and you could convert that to a 3d Model.
For gamedev you often have a drawing of a Character made up which then gets scuplted, retopo, texture ...etc. This is very time consuming especially for characters that are in the background. If you could create a bunch of characters from drawings that would make large crowd scenes much easier to create.
If fills in the backs, but it cannot do buts. Most of these pictures are professional models. They are attractive people. But their backsides are all flat as pancakes and thier cloaths obviously looser in the back then the fronts, which does a disservice to all the time they likely put in at the gym.
Check out https://zivadynamics.com/. It has the realism but the models need to be manually posed. Using the low-poly from efforts like this one, Unity will make it easier to pose the high res Ziva model. There are several papers on how to capture/simulate the clothing which should result in high res outputs. Also see DeepMotion.
69 comments
[ 3.2 ms ] story [ 153 ms ] threadAlthough, they are getting better years after years, so maybe we'll be out of the uncanny valley at some point ?
On one hand you have traditional photogrammetry based on classical image descriptors (from the beginning of the 2000s). A nice open source solution is Meshroom [1]. I have very little experience as I have just tinkered a little but I would say they work OK with geometry of medium complexity and detailed textures. They fail horribly with untextured objects, like for example a candle. They are semi-automatic: you upload some photos and run the pipepiline but for best results it's easy to tweak as there are a lot of knobs to try.
On the other hand you have these deep learning research papers. A lot of them have Colab notebooks you can try yourself (which is awesome, thank you). They can give you these very nice demos for some kind of objects (the ones we have a lot of training data, like human bodies), but they are not truly general and sometimes will generate artifacts. There is some opportunity here if they are integrated in a semi-supervised workflow, like this [2].
Anyway, just curious and not an expert.
[1] https://alicevision.org/ [2] https://keentools.io/products/facebuilder-for-blender
A image-to-skeleton pipeline might even be useful for 2D sprite rigging.
This is actually technology we already have today. You wouldn't want X-ray goggles, but infrared "goggles" would do the job just fine. Like X-rays, infrared light will pass right through clothes; unlike X-rays, it won't pass through skin.
Hook up an IR camera to a VR headset and you're there.
If an ai can reconstruct a view of the far side of something from data like the near side view and the shadows & other effects on the surrounding environment, then it can do the same thing to reconstruct a view of what's underneath clothing, theoretically, which may be reasonably restated as eventually.
This is actually not something that people want. The body within clothes is generally constrained by them in ways that look odd if you can't see the clothes; what people really want is to see what someone would look like if they were naked, not what they would look like if their clothes were invisible.
Would be super creepy though.
You're not thinking evil enough. Provide app for free, then use face recognition to allow people to pay to look better naked through the app.
To me, that means a version DALL-E which can do art direction without me in the loop at all is not too far off.
I've been primarily tackling the synthesis from the GPT-3 + 3D animation side but the 2d prerender idea is genius. I'll need to try to hook it up one of these weekends. The only hard part, it seems, is finding the navmesh.
If you're not recreating something from actual data elsewhere, it's just being made up on intuition level guessing.
My concern is what the purpose of this "tech" is being used for, and more specifically, how this "tech" will be mis-used for other purposes.
For archviz you don't need an accurate 3D model but one that looks good and casts shadows.
I'd imagine smart mirrors in clothes stores as a much more intriguing application.
Not exactly the same thing but I have been thinking about the idea of 3D construction of cloth/jeans you want to purchase onto your own body, so that you have better idea before placing order, which could be really useful for online shopping.
I was thinking in game graphic way (bottom-up), such as building 3D model of myself, shopping website providing 3D models of their clothes, and then it would be a "fitting" problem to properly place clothes onto the body.
From this paper it seems that it is not necessary to have low-level data to achieve the objective.
https://github.com/minar09/awesome-virtual-try-on
Their original idea was even better, IMO. It was supposed to send the model to an on-demand automated clothing factory that would custom-sew entire pieces of clothing to your measurements.
I think the technology wasn't quite there yet at the time. Even with four first-gen Kinects, the body model was probably too inaccurate to compare to an actual tailor. They seem to have moved away from clothing in general since then.
I guess, by hedging this way, it managed to improve its evaluation score without having to learn how clothes actually work.
If you can take a photo (effort required: seconds) and turn it into a textures low- to mid-poly model within seconds, that's a huge time saver right there.
Combined with other techniques [1], [2] this could indeed present a viable business model - again for certain niches. It's the same old routine: identify potential, present solution, profit (if only for a limited time). On its own this might not look too useful/impressive, but I'm certain there's people out there who could already benefit greatly from this work.
It's not always about "the next big thing" or "turning an entire industry on its head". I could see this work its way into various products without much fanfare and gazillion dollar valuations.
[1] https://github.com/Shimingyi/MotioNet
[2] https://deepai.org/publication/end-to-end-learning-for-3d-fa...
Example? A fashion billboard that doesn't use model photos but renders passers-by in fancy new clothes. Of course, the gathered data gets syphoned of into a database somewhere.
Smartphones and CCTV cameras are much more effective sources of information than a smart billboard anyway. Smart billboards, however, sound quite awesome.
Computing and related tech is an enabler of so many new possible applications. It has become really tough to develop something useful for consumers that can be commercially successful, but doesn't at least have the potential to be turned into something creepy.
Corporations have an increased ability to do things w/o oversight, but the government can literally do whatever it wants when it wants as long as those in power allow it. Government has the monopoly on power and violence. This concern is increased even more in democracies where there is great division. One side can abuse government to enforce their aims on an unwilling opposing side.
Additionally, corporations generally just want to make money so they are fairly predictable and easy to contain (i.e take your business somewhere else if you don't want to deal with them).
Government operates at the beck and call of political whims which, let's face it, are more often than not emotional and knee-jerk reactions. Lastly, only government can make it illegal to avoid surveillance and put you in jail for it. The worst a corp can do is close your account or something.
Anyway, I think this is getting too far from the actual topic here. I'll leave it at that.
For gamedev you often have a drawing of a Character made up which then gets scuplted, retopo, texture ...etc. This is very time consuming especially for characters that are in the background. If you could create a bunch of characters from drawings that would make large crowd scenes much easier to create.