Show HN: Generative Fill with AI and 3D (github.com)
You've probably seen projects that add objects to an image from a style or text prompt, like InteriorAI (levelsio) and Adobe Firefly. The prevalent issue with these diffusion-based inpainting approaches is that they don't yet have great conditioning on lighting, perspective, and structure. You'll often get incorrect or generic shadows; warped-looking objects; and distorted backgrounds.
What is Fill 3D? Fill 3D is an exploration on doing generative fill in 3D to render ultra-realistic results that harmonize with the background image, using industry-standard path tracing, akin to compositing in Hollywood movies.
How does it work? 1. Deproject: First, deproject an image to a 3D shell using both geometric and photometric cues from the input image. 2. Place: Draw rectangles and describe what you want in them, akin to Photoshop's Generative Fill feature. 3. Render: Use good ol' path tracing to render ultra-realistic results.
Why Fill 3D? + The results are insanely realistic (see video in the github repo, or on the website). + Fast enough: Currently, generations take 40-80 seconds. Diffusion takes ~10seconds, so we're slower, but for the level of realism, it's pretty good. + Potential applications: I'm thinking of virtual staging in real estate media, what do you think?
Check it out at https://fill3d.ai + There's API access! :D + Right now, you need an image of an empty room. Will loosen this restriction over time.
Fill 3D is built on Function (https://fxn.ai). With Function, I can run the Python functions that do the steps above on powerful GPUs with only code (no Dockerfile, YAML, k8s, etc), and invoke them from just about anywhere. I'm the founder of fxn.
Tell me what you think!!
PS: This is my first Show HN, so please be nice :)
109 comments
[ 4.6 ms ] story [ 451 ms ] threadToday, as in right now, with less than 5 relatively not-horrible photographs you can create a realistic AI version of anyone to do anything you'd like them to, or wear. Animation included. From your home computer.
Or just inpaint the clothes away from any image.
https://www.bbc.co.uk/news/world-europe-66877718
This has nothing to do with social media. The images have circulated through WhatsApp and Telegram channels. And if they hadn’t, they would have through email or MMSes.
That's what was going on in 1992.
I'm a little alarmed at the general laziness / lack of initiative of these kids today, TBH. but whatever.
How well does it work on non-room images?
For different kinds of images, it's a question of using other cues to build a 3D structure that's very close to the original image. And no, monocular depth estimation isn't enough (happy to nerd out about why) ;)
Tried some on fill3D from a dataset we had before (happy to share more), and yup: https://imgur.com/a/Ut2GwZ0
Tough Tough Tough!
fxn.ai looks super cool too, I might try it out!
Also in that regards: I'm curious about what it can't handle. Any situations where it borks?
the bug I saw was after uploading a background image, on the right side, I only saw a generate and a reset button, nothing else. I clicked "generate", expecting it to ask me to input a prompt, but it started to render and the result was the same background I uploaded.
Imagine adding "yourself" into a scene like this, moving around as you were/are from a video you just created of yourself. As in: film yourself walking around your bedroom with your phone. Then use an app like this to add you and your movement (cropped from the video) to a different background scene.
Goodbye, Hollywood elites!
https://twitter.com/GabRoXR/status/1706691466460836333?t=3z7...
It's like Joan is Awful, but for AirBnB.
The goal is for devs like you to bring your original Python code, and we'll generate a library that is cached and runs on-device. See this demo: https://demos.natml.ai/@natml/blazepalm-landmark (wave your hands)
> Imagine adding "yourself" into a scene like this, moving around as you > were/are from a video you just created of yourself. As in: film yourself > walking around your bedroom with your phone. Then use an app like this to > add you and your movement (cropped from the video) to a different > background scene.
> Goodbye, Hollywood elites!
As someone working in this arena, statements like this make me chuckle.
Don't get me wrong-- I think this is really cool. I get that people are excited about new tech, and technical people always overestimate the value of technical advancements in creative workflows, but no: people being able to place a perfect hyper-realistic replica of themselves in a film wouldn't kill the film industry any more than RPG Maker + generative AI would kill the games industry. I'd wager it probably would not even leave a dent.
Firstly, there would need to be a film to begin with, and that requires a lot. A whole hell of a lot.
Secondly, characters matter. A lot. Especially main characters. Do you replace the name of the main character with your name in stories you tell? How about doing a global search-and-replace in the ebooks you read? It's not like we don't have the technical capability. I could see this being a novelty feature in some action movies, especially superhero movies, and more likely in games and porn, but one of the biggest draws a movie has is who stars in it-- and if you look at the rest of the human population, you'll notice that we're not choosing representative samples. The fact that it's someone else with a personality and back story and motives and strengths and flaws-- a character-- is a pretty important part of stories. Their appearance matters, too. Most people don't even like staring at themselves for a few minutes, let alone for an entire feature length film. In most situations, I think it would be distracting as hell. Sure, people might find it amusing to see themselves in Top Gun Maverick, but would they want to see themselves getting bullied by an IRS agent in Everything Everywhere All at Once? Getting a box cutter held to their throat in Emily the Criminal? Would replacing Jon Hamm's appearance with their own really make watching Madmen better? Do most people want to see themselves beat to a pulp in Fight Club? I'd wager that few would.
Thirdly, most people aren't particularly interested in putting in the thought and effort to customize their phones: I'm pretty sure they're less interested in putting thought and effort into customizing their passive entertainment. They just want to hit play and have a nice little escape.
So, no. As long as people will continue to seek entertainment for the reasons they've always sought it, this is not going to fundamentally change the art of storytelling anytime soon.
Not sure what you mean by that. Like, there needs to be a written script?
But I don't think you're following the "vision" here :) I wouldn't want my own face in a movie and i'd bet most people don't. I'm sure you've seen the demos from Wonder Dynamics and others like them. So that tech already exists and will get "smaller" and in the hands of non-studios soon.
In regards to the rest of your comment, what i'm getting at is this:
First, just let a human continue to write the script as usual (no AI really needed for this). There's countless unemployed screenwriters in Hollywood (and aspiring all over the world i'm sure) that could provide this. Does it need to be a "top tier" screenwriter already connected to Hollywood? That's rubbish. Some stories will be crap, others could be phenomenal. Good and bad YouTube content is a great example of this. Also podcasting. Also music production. All of which is on a much smaller production scale than what i'm talking about here, but the leap in technology is rapidly unlocking insane capabilities.
Second, create an app/platform that hooks in the script and allows a "director" or whatever to define scenes using visual templates, pre-baked scenes, AI-generated, whatever and link to a timeline (like a soup'd up Adobe Premier). The current and upcoming games-industry use of AI could help here and i'd imagine will get substantially better over the next year.
They've already been using game assets and 3d game scenes in current production. Look at the scenes created by "The Volume" from ILM. Much of it is Unreal engine 3d game scenes. There's plenty of 3d artists outside of the Hollywood/Disney sphere that could provide assets and talent. AI could help polish it or completely build it at some point in the near future. No need for camera-men, lighting experts, grips, etc.
Third, all you need as of "filming" of an actor is to point a phone at them, crop out their body and facial features/movement, and generate 3d space around them to allow the director to point the virtual camera from any angle. You could have multiple cameras/phones at different angles just to help the 3d space generation and probably a max of 4 to create a suitable 3d space all around an actor. Note: these are not director's "film angles", this is just to drive the AI scene generation. Although i've seen tools come out recently: look at Google research's AI generative fill technique that came out recently. Or Adobe's generative fill feature. This will only get better and more adaptable to video rather than still pictures.
Example: film me at my dinner table talking to my wife. Then, employ the platform to inpaint a 3d scene around me, replace my face/body with whatever I choose, and let me point a virtual camera a different angles throughout a timeline.
Example 2: film me running down my street as if i'm being chased by a [dinosaur, a car, a mob of zombies]. Crop me out and build a scene around me.
Obviously, each of these would be independent scenes but multiple shots/angles could be captured during editing. But of course it would take quite a bit of time and talent to build up an entire movie or show. The platform i'm talking about here wouldn't be for most people. But it wouldn't just be for big-budget filmmakers either :)
#2 and 3 obviously take a lot of tech to build, but we're already seeing pieces of this with current AI tools. Heck, i've seen several within the last 2 weeks that could be a good starting point for this, including the topic of this HN post.
I don't agree at all with "one of the biggest draws a movie has is who stars in it". Can you name any great films that don't star an A-lister over the last 30 ...
The reason i'm convinced this can happen is because I think the [very fundamental - read early] building blocks seem to be already there. Will it happen? Depends if a well-enough funded group with the right talent and motivation choose to aim their sights at the consumer market rather than enterprise/studios. Meaning, it will probably not be Adobe, or any of the Streamers. Maybe could be Meta or Snap.
You only know what the building blocks are for the required software. You don't know what the other far more consequential building blocks are. You can't just assume they don't exist or aren't consequential because you don't know they're there or how they work.
You're looking at a straight, mile-long, 10 foot thick wall from the side and assuming you're looking at a 10 foot wide wall that's a few inches thick. And then, despite someone who builds walls like that telling you otherwise, you insist you could knock it flat with a wrecking ball in a matter of seconds. It doesn't matter if the wrecking ball is exponentially better than the hammer you're used to, it's still not going to do what you think it does.
This sort of hubris which is endemic to current dev culture was a major factor in my deciding to leave the software industry.
I’ve got a big empty studio with a bed and couch I’ve already purchased but trying to figure out what to fill in for all the other gaps. Coffee table, media console, tv or UST projector, bar or bookshelf or desk.
Would be nice if there was a way to populate it with items/products that can be purchased and aren’t purely conceptual.
Exciting times ahead.
(No affiliation!)
So it works by trying to estimate a 3D 'room' that matches your image. Everything from the geometry, to the light fixtures, to the windows. It's heavily inspired by how humans (weird to contrast 'human' vs. AI work) do image/video compositing.
TL;DR: Image in, 3D scene out.
The tech itself looks amazing though, well done.
Don't worry, there's more options. But thank you!
https://investor.wayfair.com/news/news-details/2023/Wayfair-...
Yes, they'll break a software license and use garbage that uses garbage that uses garbage. Way to draw the line.
Your comment sounds like a criticism of the project in general rather than the pointlessness of adding a clause to the license. Personally, I think this is pretty novel, better than the 100's of stable-diffusion-as-a-service things that have popped up lately.
> while riding on the back of things the owner has no idea how they would actually create if they were asked to do so
I mean, everyone builds on top of things they couldn't recreate. If you're a software developer, chances are you couldn't recreate your favorite languages runtime/compiler/whatever, you couldn't recreate your OS, you couldn't recreate the hardware that's running your software. I don't get this criticism at all.
FWIW this isn't my problem with this project. It's that the writer doesn't know what they're doing and represents a new type of post-code/post-crypto monkey that just links together APIs in clever ways and tries to charge maximum $ for it by selling it to people (monkeys?) who think it's magic.
People like this will make a lot of money, and eventually do something that injures you and your family personally. So it's best to attack them and slander them early and often.
Today watched a video about gravelbike touring where some young whippersnapper was getting mad excited about the idea of putting a rack and panniers on the back of their bike - just like in the good old days. What a world we live in. I'm 100% old af
I needed an image of an empty room recently. I just took a photo of my very not empty room, ran it through a canny algorithm, painted out the objects with black, and then used stable diffusion with canny controlnet to generate an empty room. Worked pretty well. Did not look that much like the original room, but it was certainly good enough to check furniture placement etc.