Tried a few random images and scenes, overall wasn't that impressive. Maybe I'm using the wrong kinds of input images or something, but for the most part once I moved more than a small amount, the rendering was mostly noise. To be fair, I didn't really expect much more.
Neat demo, but feels like things need to come quite a ways to make this interesting.
My understanding of JavaScript is cursory, but my reading of that webpage is the UI is just smoke and mirrors, and it is just waiting for the whole thing to be processed in a single remote API call to some back-end system. If the back-end is down, it will always stop at 90%. The crawling progress bar is fake with canned messages updated with Math.Random() delays. Gives you something to look at, I guess, but seems a little misleading. Might be wrong ...
Yeah I think you're right. It calls that out (in really tiny footer text) that it's leveraging ml-sharp.
It's pretty trivial to get running locally and generating the PLY files. Spark's a pretty good renderer for it after you've generated the gaussian splats.
If this model is so good at estimating depth from single image, shouldn't it also be able to take multiple images as input and estimate even better? But searching a bit it looks like this is supposed to be a single image to 3D only. I don't understand why it does not (can not?) work with multiple images.
I also feel like an heavily multimodal model could be very nice for this: allow multiple images from various angles, optionally some true depth data even if imperfect (like what a basic phone LIDAR would output), why not even photos of the same place even if it comes from other sources at other times (just to gather more data), and based on that generate a 3D scene you can explore, using generative AI for filling with plausible content what is missing.
Its funny, always stucks on 90% till it fails with the error that another big image may be keeping the server busy.
I mean ok its a "demo" tho the funny thing is if you actually check the cli and requests, you clearly can see that the 3 stages the images walks through on "processing" are fake, its just doing 1 post request in the backend that runs while it traverses through the states, and at 90% it stops until (in theory) the request ends.
27 comments
[ 3.5 ms ] story [ 47.2 ms ] threadNeat demo, but feels like things need to come quite a ways to make this interesting.
This is the heavy lifting: https://github.com/apple/ml-sharp
Previous discussion: https://news.ycombinator.com/item?id=46284658
It's pretty trivial to get running locally and generating the PLY files. Spark's a pretty good renderer for it after you've generated the gaussian splats.
https://github.com/sparkjsdev/spark
I mean ok its a "demo" tho the funny thing is if you actually check the cli and requests, you clearly can see that the 3 stages the images walks through on "processing" are fake, its just doing 1 post request in the backend that runs while it traverses through the states, and at 90% it stops until (in theory) the request ends.