This looks very interesting. Lately I've been playing a lot with the new photogrammetry API that Apple provides in Monterey on M1 (cloud instance) and it's working exceptionally well already compared to desktop based solutions to scan cars and humans. It fits my workflow of just using a phone, sending a video and getting a 3D model back (of course with some plumbing).
I wonder how this compares and most importantly how the licensing model will look like? Is the processing done on the device or on a remote server farm?
However, my congratulations on the release of the beta! I'm downloading it as we speak.
Also people need to share their 3D models for colleagues, marketing, fans, etc. to look at. Quixel was using them internally anyway, so Epic bought Sketchfab:
Scannings apps like this already exist though, like Scaniverse (recently acquired by Niantic) and Polycam. Both seem to output higher quality scans than RealityScan:
Hi, I'm currently working on a project in that area. The main workflow in the system is take a bunch of pictures, SIFT & match them, then generate a sparse model of 2d features (image coords), 3d features (point cloud in relative coords) and camera positions. We then shard and re-combine different models to synthesize larger or more precise reconstructions.
I'm building with clojurescript & Three.js[0] in the frontend, and can recommend the stack
Back to your comment I'd love to hear what you're looking to do and what you'd need, sounds like I may have explored the same problem space.
And I hope someone has finally found a way to enhance photogrammetry with depth sensor data because i'm pretty sure both combined has the potential to produce better results than either alone.
I'm a little torn on this. On the one hand, this is really impressive for on-the-fly photogrammetry. On the other hand though, this asset kinda just looks... bad? From a distance it's pretty passable, and in a relative vacuum of visual information it looks fine. However, the material itself is really sketchy. Whatever their default shader settings are makes the leather look toy-ish, caught somewhere between a reflective and diffuse material. The texturing is pretty low-quality (as expected), but the really annoying part is how it handles the ambient occlusion on the surface: it just looks like self-shadowing is baked into the texture.
It's a cool proof-of-concept, but I'm not quite sure who could use it in it's current state. Even by video game standards, these assets would probably look really out-of-place...
As a 3D artist who uses a lot of photoscans -- the scan quality looks fantastic to me! Photoscanned assets aren't usually meant to hold up up close, they're more intended for quickly building out a larger scene. Additionally, the material you see straight out of the app can be enhanced significantly with some common shader tricks like faking a bump/roughness map. Lastly, good lighting makes a world of difference when using photoscanned assets. Personally, I've just joined the beta and expect I'll be taking advantage of it heavily for several upcoming CGI projects.
All good! I'm not trying to detract from anyone's enjoyment/use of the product if they have one :p
I'm just a 3D hobbyist, and I don't really do a lot of photo scans. My thoughts on the matter are coming purely from a CG/asset generation perspective, if you're an artist with a little more agency with how you represent 3D objects then I bet you could work wonders with this (especially the geometry portion). Perhaps it was silly for me to expect these scans to rival hand-made assets, but my larger point was that we simply have a long ways to go before we reach parity with the two.
In the time that's passed, I actually did come up with a use for it though. It might be fun to use Blender's layer functionality to "trace" a 3D scan into a higher-res production model, just to get proportions right. I'll probably still hold off until this reaches general availability though.
This looks bad to you? Admittedly I have little experience with 3D models. As a layperson this model looks incredible to me. If the real life results are anything close to that then photogrammetry is as good as solved, as far as my eyes are concerned.
Well, that's why I say I'm torn. I really do think this is cool, but when you compare it to hand-made assets the two would look starkly different. My biggest gripe is with the lighting: if you look at the creases of the couch, you can see that the lighting is literally "baked" in to the model. Shadows are rendered by the texture instead of the lighting engine, which means the models will appear static if you use them in a dynamically lit environment. Worse yet is the fact that these "shadows" can actually reflect light since the model has no reflection mapping or light mapping to tell the renderer that the shadows are rough as opposed to glossy. That gives the model the aforementioned "toy look" that I brought up earlier.
It's still quite cool, but asset capture has a long ways to go (even if everyone's results are this good).
29 comments
[ 1.9 ms ] story [ 73.0 ms ] threadAlgorithmic (SIFT) or ML based?
Many people developing UE games need 3D assets, so Epic bought Quixel, a 3D asset store:
http://www.cgchannel.com/2019/11/epic-games-acquires-quixel/
Also people need to share their 3D models for colleagues, marketing, fans, etc. to look at. Quixel was using them internally anyway, so Epic bought Sketchfab:
https://www.epicgames.com/site/en-US/news/sketchfab-is-now-p...
And people want to create their own 3D assets, so Epic bought the company that Quixel was using for 3D scanning, called Capturing Reality:
https://www.epicgames.com/site/en-US/news/capturing-reality-...
And now you can go from reality to virtual twin to realtime 3D visualization without ever leaving Epic Games' servers.
That's probably a major driving point to the strategy after the Xbox Underground hacks.
Fun read of the tale: https://www.blackhatethicalhacking.com/articles/free-access/...
They got into Epic's network and that gave them access to many other companies.
Funny enough it wasn't the Apache simulator stuff that broke the camel's back, but the Gears of War 3 leak and Xbox One prototypes.
I remember the Gears 3 leak and at the time the conspiracies that Microsoft leaked it themselves:
https://kotaku.com/is-this-where-gears-of-war-3-was-leaked-f...
https://poly.cam/
https://scaniverse.com/
Or at least a path to accomplishing this with the lowest barriers to entry?
There is Regard3D, if by barriers to entry you mean sensor hardware: http://www.regard3d.org/index.php/documentation/introduction
If by barriers to entry you mean technical skills, there are SaaS solutions like Beholder: https://beholder.vision/
About embedding the demo in a web page, I have no idea.
Back to your comment I'd love to hear what you're looking to do and what you'd need, sounds like I may have explored the same problem space.
[0] https://threejs.org/
Can't wait :)
I digitized my flat with an m1 ipad and that looks already okay.
It's funny to see the flat layout after doing the 3d scan. An interesting way to persistent the flat memory.
It's a cool proof-of-concept, but I'm not quite sure who could use it in it's current state. Even by video game standards, these assets would probably look really out-of-place...
I'm just a 3D hobbyist, and I don't really do a lot of photo scans. My thoughts on the matter are coming purely from a CG/asset generation perspective, if you're an artist with a little more agency with how you represent 3D objects then I bet you could work wonders with this (especially the geometry portion). Perhaps it was silly for me to expect these scans to rival hand-made assets, but my larger point was that we simply have a long ways to go before we reach parity with the two.
In the time that's passed, I actually did come up with a use for it though. It might be fun to use Blender's layer functionality to "trace" a 3D scan into a higher-res production model, just to get proportions right. I'll probably still hold off until this reaches general availability though.
It's still quite cool, but asset capture has a long ways to go (even if everyone's results are this good).