Launch HN: Vimmerse (YC W22) – Platform and SDKs to create and play 3D video
We have been watching video in two dimensions for too long! Most 3D content people experience today is computer generated, such as VR games. Diverse use cases can benefit from real-world 3D video, such as music performances, training, family memories, and the metaverse. Why isn’t there more real-world 3D video? Because it has been difficult and expensive to create, stream, and playback real-world, camera-captured 3D video content.
I am an IEEE Fellow and inventor of over 200 patents. Basel has a PhD in electrical engineering with deep experience in VR/AR/3D video. While at Intel (as Chief Media Architect and Intel Fellow), I led an MPEG standards workgroup on 360/VR video. I found that 360/VR video’s limitation to 3 Degrees of Freedom (DoF) caused discomfort or even nausea in some viewers, because we experience the real world in 6 DoF (controlling both position + orientation), not in 3DoF (just orientation). I initiated an activity in MPEG to develop the MPEG Immersive Video (MIV) standard, which provides 6DoF. I became the lead editor of the MIV standard, and Basel was the lead editor of the test model.
While at Intel, we developed a MIV 3D video player for Intel GPUs and observed the greater engagement that 3D video provides to viewers. However there was no content available for the new MIV standard, and creation of 3D video content was a very difficult and expensive process. We realized that if 3D video were to become widely used, the creation and distribution processes needed to be simplified. We founded Vimmerse with a mission to greatly expand access to 3D video.
Businesses can build their own services using our APIs to upload captured content and prepare 3D video on our platform. Our platform is capture agnostic, meaning it can work with any video device suitable for 3D capture, such as iPhones or Microsoft Azure Kinect depth sensors. More than 60% of iPhone 12 and 13 models sold (Pro and Pro Max) have LiDAR depth sensors, which can be used for capturing 3D video content.
The Vimmerse platform prepares 3D content from the uploaded capture files. Our approach is built on top of industry standard video streaming protocols and codecs, so existing video streaming servers and hardware video decoders can be utilized. The content preparation platform creates two types of output bitstreams from the uploaded captures: bullet video and 3D video. Bullet video (named after the Matrix movie’s bullet effect) is a 2D video representation of the 3D video, following a predetermined navigation path selected by the content creator. 3D video gives viewers the ability to control navigation with 6 Degrees of Freedom (6DoF), where they can pan around or step into the scene. Bullet video may be streamed (HLS) or downloaded (MP4) for playback on any device. 3D video playback may be streamed (HLS) to the Vimmerse 3D video player.
Services may use the Vimmerse 3D video player app, or developers can use our player SDK inside their own apps. Viewers have the ability to control navigation using any viewer input method: device motion, mouse/keyboard, touch controls, head/gesture tracking. The player SDK renders views for the selected 6DoF position and orientation.
We haven’t published pricing yet, but our plan is to charge for our content preparation APIs based on usage (e.g. minutes of video processed and streamed) and player SDKs based on number of units.
The Vimmerse website https://vimmerse.net provides a no code way to test out our platform or view featured content. We invite the community to upload their own test content. Instructions for prepa...
34 comments
[ 4.7 ms ] story [ 85.8 ms ] threadLots of documentation available on the public Joint Video Experts Team (joint group between MPEG and ITU-T) website https://jvet-experts.org/ about the activities I led in 360 video.
Here is an example: https://jvet-experts.org/doc_end_user/current_document.php?i...
But that's a very biased population for sure!
But more seriously (but still on the spirit of Emacs), how will your 3D video player be scriptable and extensible at runtime?
I have some positive experience with extensible 3D panoramic stereo videos players, which I'd love to share with you:
I developed a system for JauntVR that enables cross-platform scripting of Unity3D applications in JavaScript, for Jaunt's 3D panoramic video player. It also greatly eases and accelerates development and debugging, which is otherwise extremely slow and tedious with Unity.
Unfortunately JauntVR ran out of money for that project and pivoted to other stuff before publishing it, but Arthur van Hoff generously let me open source the general purpose Unity/JavaScript scripting system I'd developed for it, which I call "UnityJS".
It's useful for all kinds of other things beyond scripting 3D videos, and it works quite nicely not just on mobile devices but also with WebGL, but scripting interactive 3D video was the problem I originally created it to solve.
I've used for several applications and integrated it with several libraries, including financial data visualization on WebGL, TextMeshPro in Unity, ARCore on Android, and ARKit on iOS.
UnityJS also makes developing and debugging scriptable Unity apps much easier and quicker (by orders of magnitude) because you can use the standard JavaScript debuggers (even on mobile devices) and just restart the app to reload new code in seconds, instead of waiting the 10-60 minutes it takes to totally rebuild the app in Unity.
By using the same standard built-in JavaScript engine of the web browser on each platform, you can implement cross-platform interfaces in JavaScript instead of re-tooling the interface for each platform in different languages, and incorporate standard up-to-date off-the-shelf JavaScript libraries.
I believe the 3D video player should be much more like an extensible scriptable browser (and like Emacs and NeWS!), instead of just a dumb fixed-function play/pause/rewind/fast-forward video player (like piping a text file through "more" on a VT100).
So you can download dynamic content and scripts that implement custom interfaces for 3D videos, like interactive titles and credits, custom menus, theater lobbies, even networked multi player environments for watching 3D videos together.
It's great fun to overlay live 3D graphics, text, physics simulation, interactive kinetic typography, particle systems, etc, into 3D videos.
It's extremely useful for selecting and navigating content, displaying interactive titles and credits, or anything else you might imagine, like games and applications embedded in and orchestrating 3D video.
And of course you can publish new content, hot updates, and extend the experience by downloading new code and data over the internet, instead of pushing out a new version of the app through the app store every time you want to improve the experience or publish a new video or series.
Some obvious and useful application for 3D videos is designing compelling opening and closing credits, captions, subtitles, pop-up notifications, pie menus, player controls, navigation, bookmarking, tagging, and any other kinds of user interfaces, that can adapt to the content and environment in whatever direction you're currently looking, so you don't miss them and they're not unusable, just because you were looking in the wrong direction at the wrong time.
3D videos suffer practically and creatively from hard-coded static title sequences and credits, because you might not be looking in the right direction at the right time to see them. It's like pointing a movie camera at a stage to film a play, instead of inventing a whole new expressive cinematic language appropriate for the new medium. And I'm not just talking 3D scrolling Star Wars credits!
Of course you could just build a dumb non-extensible 3D video player with just one hard-coded style of menus and ...
We hadn't thought about making the 3D video player be scriptable and extensible at runtime, and will give it some thought.
Being able to overlay 3D graphics ( including titles) onto the 3D video is on our roadmap. Glad to hear confirmation that it will be a useful feature to add.
Once you can overlay 3D graphics on 3D video, you'll definitely want runtime scriptability!
Because of its name "UnityJS", sometimes people misinterpret that it's something like Unity's old and now thankfully deprecated "UnityScript", a compiled (not runtime interpreted or JITted) ersatz JavaScript-ish language, that was really just a thin wrapper around the CLR C# APIs, without most of the standard JavaScript APIs, plus it's own weird syntax and object system to make it even more uncanny.
But UnityJS is a totally different approach for a much different purpose, and I wish I could think of a better less confusing and loaded name for it.
Each platform has its own APIs and quirks for efficiently integrating and exchanging JSON messages with its own browser and Unity, in Objective C for iOS, Java for Android, and JavaScript for WebGL.
UnityJS abstracts those platform differences like (and by using) a web browser, so you can write cross-platform JavaScript code, and communicate with Unity via JSON messages, which uses JSON.net to convert back and forth between JSON and C# and Unity objects.
It's better to rely on the build-in JavaScript engine in each platform's standard web browser, than trying to roll your own scripting language from scratch, bundle your own copy of Chrome, or use a less ubiquitous languages than JavaScript (as much as I love Lua and Python).
What's great about that approach is that it lets you use standard development tools: you can live code and debug WkWebView based iOS apps with the desktop Safari developer tools, and Android Chrome component based apps with the desktop Chrome developer tools.
And it works seamlessly with the entire ecosystem of browser based JavaScript libraries. (Which is a relief if you've ever tried to get a Windows C# library to work with Unity's version of C#, let alone trying to port any JavaScript library to UnityScript, which was practically futile).
On iOS, using the standard WkWebView browser as a scripting engine also avoids the infuriating non-technical Dumb Apple App Store Rules Problem, because they prohibit apps that dynamically download and update and interpret any kind of dynamic code, UNLESS you use their browser and JavaScript engine.
Consequently, WkWebKit's JavaScript is the only runtime scripting language you're allowed to use on iOS (it's much better than the old UIWebView because it has a great JIT compiler). Fortunately it works quite well! So be it.
JavaScriptCore runs in-process and consequentially the JIT is disabled, because Apple doesn't allow non-Apple processes to write to executable code memory.
UIWebView's JavaScript engine is JavaScriptCore, so it runs in-process (in YOUR process, not Apple's), but it's deprecated in favor of WkWebView, which runs in another sandboxed process (Safari, blessed by Apple) and communicates with by sending fewer JSON messages instead of requiring many direct two-way procedure calls.
Since WkWebView runs in a separate process fully controlled and trusted by Apple, Apple marks it as being allowed to write to instruction memory, which is necessary in order for the JIT to work.
There is one advantage to UIWebView/JavaScriptCore has that WkWebView doesn't have, and that's that you can use Apple's lovely Objective C / JavaScript bridge stuff to directly and efficiently extend the JavaScript interpreter with your own code, and call back and forth directly between JavaScript and your own code without performing remote procedure calls by sending messages.
That is what NativeScript does, so you can call native iOS APIs directly from JavaScript. (Or at least last I checked, it's been years since I took a close look into studied how it works, but that's pretty fundamental to how it works.) But that means NativeScript's JavaScript code is never going to run as blazingly fast as it would if the JIT were enabled.
https://nativescript.org/
While NativeScript is designed to expose and call native platform APIs like Cocoa directly from JavaScript, UnityJS is designed around asynchronous JSON messaging, with techniques to reduce the number and size of messages, using the same interface across all platforms instead of exposing native APIs.
UnityJS is based on sending fewer larger JSON messages, instead of lots of small procedure calls. For example, with NativeScript you would make a whole bunch of fine granularity calls between JavaScript and native code just to make a window and configure it and put some buttons in it and set each of their colors and positions and constraints and callbacks, while UnityJS would just send one big blob of JSON with multiple nested higher level messages to create and configure many objects at once (Unity prefabs and C# objects instead of Cocoa objects), describing the whole tree of objects you want to create and their callbacks at once, amortizing the cost crossing the JavaScript/native code barrier.
UnityJS also has some clever techniques for reducing the number and size of callback messages: when JavaScript express interest in an event or callback from Unity, you can declare (with a JSON query template with accessor path traversal expressions, kind of like XPath for JSON+C#+Unity objects) exactly which parameters you want sent back with the event when it happens, using "path expressions" that can drill down and pick out just the parameters you need, which get automatically converted to JSON and sent back in the message. So the callback messages you receive contain exactly the parameters you're interested in, and no more. For example, why send the screen coordinates and timestamp and other random properties of a touch event, when you're only interested in the 3d coordinates of a ray projected into the scene and the object id of whatever it hits? Then you don't have to ping-pong back and forth to derive the world coordinates and target object id from the screen coordinates, which would be painfully slow!
These documents describe the issues and APIs and alternatives in more detail:
https://github.com/SimHacker/UnityJS/blob/mast...
Javascriptcore-on-device is debuggable too!
I personally use native javasriptcore on ios&mac, (which lets me get my engine+app down to under 10mb) javascriptcore (the open source one) for linux (and at one point windows, but I use chakra there). At one point I used v8 on windows & mac, but it's such a PITA to build into the form (dll/static) I want, I just had to give up in favour of native runtimes
There are some annoyances with the difference between apple jscore & open source jscore, but all fixable! (not sure it took me 3 months :)
https://github.com/NewChromantics/PopEngine
hackernews isnt the great for discussion, but if you ever want to chat more, just find my username on twitter/instagram/etc (I don't get to talk about this low level js stuff much, I think the number of people using it outside chromium is under 2 figures :)
https://github.com/SimHacker/UnityJS/blob/master/notes/Oculu...
https://github.com/SimHacker/UnityJS/blob/master/notes/PInvo...
https://github.com/SimHacker/UnityJS/blob/master/notes/Skybo...
https://github.com/SimHacker/UnityJS/blob/master/notes/Unity...
https://github.com/SimHacker/UnityJS/blob/master/notes/Surfa...
https://github.com/SimHacker/UnityJS/blob/master/notes/ZeroC...
Now I recall that we even had an email discussion about how to render an Android WebView into a Unity texture and efficiently get GPU textures into Unity, relating to your UnityAndroidVideoTexture and popmovie plugin, back in December 2016 -- thanks for the great advice, and for open sourcing and sharing that code! That was some very tricky stuff to have to figure out on my own.
And of course every platform has its own weird tricky plumbing like that that (which changes over time). So UnityJS is intended to abstract that from the casual programmer, who only needs to deal with good old JavaScript, without mixing in C++, Java, C, and Objective C.
Another great source to learning the best way of doing tricky low level platform dependent plumbing things like that is the Chrome and Firefox browser source code. Much better than reading through old out-of-date blog postings that don't even apply any more. Those huge projects have extremely talented people from the respective companies working directly on them, who deeply know (or even implemented) all the latest poorly- or un-documented APIs.
Reading that code is like drinking from a firehose, and it can be overwhelming finding the actual four lines of magic code that does the tricky stuff, but you know it has to be in there somewhere, so don't give up!
A good way of figuring out where to start and finding a pointer into the right part of the monolithic source code is to look for related bugs and issues and pr's that discuss and point into the code, and search the bug database for names of important and unique API functions and symbols.
For example, here are some notes I took on Apple's "IOSurface", which is kind of like the Apple equivalent of Android GL_TEXTURE_EXTERNAL_OES textures:
https://github.com/SimHacker/UnityJS/blob/master/notes/IOSur...
I'm curious why you're recommending UnityJS instead of just a straight WebGL implementation.
Unity is an amazing tool, but my understanding is that the assets need to be packaged with the player, and that isn't done in real-time. So adding new functionality would mean either some of your videos don't have the full capabilities, or you're recompiling them.
We've been mostly out of this space for the last two years, so it's also possible that I'm entirely wrong about that.
It was a change of mindset to apply for my first patent for Vimmerse.
If you are in the US, the USPTO has a good overview here: https://www.uspto.gov/patents/basics/patent-process-overview
Application fees are lower for small entities or micro entities. https://www.uspto.gov/learning-and-resources/fees-and-paymen...
But the biggest cost is paying for a patent attorney to prepare the application. If you want to try to do it on your own, I found this book very helpful: https://www.amazon.com/Invention-Analysis-Claiming-Patent-La...
Good luck!
I've been very interested in this field since the original Kinect came out and it was discovered you can easily get the raw RGB+D data over USB. I made a bunch of prototypes and had ambitious plans to make a depth-enabled short movie for my film school MA degree project, with a player on the original iPad and head tracking for a parallax effect. (A project much too ambitious; I never finished the player or the movie or the degree.)
IMO you should hire an in-house artist right away, assuming you have funding. Right now your Featured Content section looks like demo footage from academic papers. For someone uninitiated, this content doesn't really present the potential of immersive video.
You should have someone with enough artistic vision and technical competence whose primary job is to come up with new exciting demos of your technology. That person will need patience and a start-up mentality because the content production pipeline for something this cutting-edge is probably very foreign to most artists. But it would make a huge difference in how people can understand your project.
(If I were twelve years younger I'd send you a job application right away. Working on this would have been my dream job.)
We encourage you to shoot some content with a Kinect DK (or iPhone) and upload it to our platform to test it out yourself.
Can their software measure depth and height? As in, if I take a 3D picture of an object, can it tell me the dimensions. As in, can this thing give me a computer model of a 3d object on my screen?
No, our system doesn't measure object dimensions. It provides the ability to capture a real-world scene and stream it for remote playback in which the viewer has the ability to control navigation around the scene.
Also, just an FYI but Meta (nee Facebook) is doing the hard-press on LinkedIn for engineers with streaming experience specifically looking for AR/VR video streaming. How do you fit into that eco-system? Video is powerful because it is everywhere ... on Reddit, in my Facebook feed, on Instagram not to mention YouTube. It took a while for that to happen - but is there an on-ramp to similar services in your roadmap?
I've played a bit with 3d video, even bought a VR 180 camera. You mention one deficiency of VR 360 you sought to solve was that it lacked in degrees of freedom. My own experience was that VR 180 was significantly more interesting. I honestly don't want to swivel my head around much when consuming content and VR 180 at least keeps the majority of the action right in front of me. It feels more appropriate in many ways for many purposes, including a Zoom like VR conferencing setup. I would assume it uses less data as well which seems like a win.
Finally, AR feels like such an interesting area in this space. Do your formats include any kind of objects/meshes/shaders/etc? Now that I think about it, I'm even more interested in 3d spacial audio. Do either of these figure into your offering?
Because our approach is built on existing video streaming protocols/servers and video codecs, we think it is a straightforward step to add 3D video streaming to existing 2D video streaming services. As you say, 2D video is everywhere now. We envision a future where 3D video also becomes ubiquitous.
With our system, 180 or 360 cameras can be used. It is up the creator to decide what range of volume to capture, what type of cameras, how many cameras, etc., which determines the range of motion is supported for the viewer.
It is on our roadmap to allow augmentation of real-world 3D video with objects/meshes like in AR, except instead of augmenting your current local scene, you can augment a remote scene (or remote in time scene).
Spatial audio would also be a very useful feature. We are video experts, not audio experts, so would plan to work with a partner to offer support for spatial audio in the future.
Thanks for your comments. It's great to hear what features people are most interested in.
Do you have encumbering patents for depth+colour video playback? Why not give away the player to aid/standardise adoption of the format?
A lot of people are using this format (including myself [0]) and it seems nobody is really pushing a standard/shared player for web, unity, unreal, etc still; it feels to me we could all move along a bit faster bu charging for capture and freeing playback.
[0] https://panopo.ly