Show HN: I turned my face rec system into a video codec (vertigo.ai)
When the real world shut down, there was suddenly nothing to monitor on streets and in restaurants, so I set out to try and evolve our real time face recognition system into a video codec for high quality face-to-face online interactions, as I was not satisfied with the quality of Zoom and friends. I got it to work, and the first release for IOS was just approved on Apple's app store, link: https://apps.apple.com/app/vertigo-focus/id1540073203
The way it works is that you create a meeting URL, which you can share out-of-band, for instance via slack or text message. You can also share as a QR code which the app can scan to join a call. You then place your device on a surface in front of you so that the front camera can see you, and it will recognize you face and assign you to your own session, which is broadcast to the meeting channel. If more than one person is in view, both of you will be broadcast but with separate session ids, like if you were on separate cameras. Other meeting participants will show up on your screen and you can start talking. It is optimized for eye contact, meaning that the eyes will actually make it through to the other side as more than just dark pixel clouds, so thinks should feel a bit more personal than the standard Zoom/Teams/or Google Meet call.
Because it uses face rec, you can ONLY show your face, and if you disappear from view your audio will stop after a while, to avoid situations like when you need to go the the restroom but forget to mute. This also solves dick-pics etc.
The CODEC is not based on H26[45], but is pure AI that runs on the GPU. There is a neural network that compresses the video in real time, and another one decompressing on the receiving end. Finding a tight network architecture that would do this in real time with acceptable quality was a major part of the effort. There are several quality settings possible, but right now it is set fairly high and for 20FPS maxes out around 700kbit/s, though typically uses about half. I've demonstrated good results down to around 200kbit/s, so in theory it should work over satellite links or even Bluetooth. The protocol is UDP with no congestion control but with (Wirehair) FEC to protect against mild packet loss, future versions will detect packet loss and adapt to available bandwidth.
The audio just uses OPUS and may click a little bit, I blame AudioEngine or the fact that the last time I wrote audio code was for the game I published for the Amiga in 1994.
If you don't have a friend around or multiple devices to play with, there is an "echo test" server mode that allows you to be in a meeting with yourself. Traffic will be peer-to-peer if possible, but otherwise you will be relaying through my tiny Raspberry PI server, so YMMV. I plan to try to switch to something like fly.io soon to improve scalability.
There is also a MacOS version coming very soon, and the underlying AI engine also runs on Windows & Linux. Android support is planned.
Please take a look and let me know what you think.
149 comments
[ 3.5 ms ] story [ 48.0 ms ] thread* I applaud the work to have it run on tiny-bandwidths, how hard would it be to up the frame-rate to 60?
* How well does "framing" work? Are you able to add flexible amounts of padding around the head or is very focussed on a face taking up the whole canvas?
* How much does it "cheat". Is it firing only feature-maps so if I have a spot on my chin does it loose it in translation?
* How did you build the face-recogniser? Is it bespoke or a library?
* Is there a testing framework? Does it work on diverse faces?
Wrt upping the frame rate the main problem is that the phone may run a bit hot, newer iPhones/iPads should be able to handle it just fine, but the older ones based on, say, the A10, might have trouble keeping up, especially with multiple remote parties connected.
* The framing depends on a transformation derived from the face landmarks, and the amount of padding is somewhat flexible. Distance from the camera seems to impact this, so it could be that my landmarks model needs some tweaking to work better when you are sitting very close to the camera.
* This is closer to being a general video codec than a face-generating GAN, so there is not a lot of "cheating" in that respect. It is optimized for transmission of faces, but other images will pass through if you let them (which I currently don't).
* I built the AI engine and the face recognizer etc from scratch, though with the help of a former co-founder who was originally the one training our models (in pytorch). The vertigo.ai home page has some demo videos. We initially targeted raspberry-pi style devices, NVIDIA Jetsons, etc., but have since ported to IOS and MacOS. Our initial customers were startups, mostly in the US, and a large Danish university that uses us for auditorium head counting.
* It empirically does seem to work on diverse faces, both in real life and when testing on for example the "coded bias" trailer. Ideally I would like to test more systematically on something like Facebook/Meta's "casual conversations" dataset.
Just wait until you find out the Chinese have the same, but they train theirs for Uygher locating...
Yeah, these technologies are amazing, but also terrifying when viewed through the OBEY lens.
Also, where can I learn about your edge AI smart camera system?
https://developer.nvidia.com/ai-video-compression
edit: oh the "recognize your face" and compression I see (referring to Nvidia link someone else posted) wow
The actual video works well. kudos!
Eventually two of us left to start Vertigo.ai, and found a customer who would fund a fast object detector to run on a $18 Nano-PI. That was a fun challenge and forces me to think about how to make the AI run fast and with relatively low footprint.
Today fast.ai might be a good starting point, definitely recommend going with pytorch, cloning cool projects from github, and going from there.
There’s some weird banding/rainbow effects around my glasses and the background (not on my face), but that’s the only major artifact that stood out to me.
Glasses are sometimes a little bit of a problem, I don't have enough of those in my training sets.
Likewise, many cameras do not properly pick up the skin tone of black, indigenous, people of colour. This is partly because of technological limits with respect to the number of F-stops available in commercial cameras. But also because there is a wide variety of human skin tones existing in the world, and camera manufacturers do not test for a majority of them.
Perhaps these papers speak louder for me, given that Hacker News only accepts neoliberal anecdotes :)
"Until recently, due to a light-skin bias embedded in colour film stock emulsions and digital camera design, the rendering of non-Caucasian skin tones was highly deficient and required the development of compensatory practices and technology improvements to redress its shortcomings"
https://cjc-online.ca/index.php/journal/article/view/2196/30...
"For a fixed decision threshold, the African-American image cohort has a higher false match rate and a lower false non-match rate. "
https://arxiv.org/abs/1904.07325
" The Gender Shades project revealed discrepancies in the classification accuracy of face recognition technologies for different skin tones and sexes. These algorithms consistently demonstrated the poorest accuracy for darker-skinned females and the highest for lighter-skinned males."
https://sitn.hms.harvard.edu/flash/2020/racial-discriminatio...
https://sitn.hms.harvard.edu/flash/2020/racial-discriminatio...
To respond in sincere mode: while I don’t think it’s terribly important whether black people are rendered correctly by some Zoom feature, nevertheless including a reasonable number of black people in one’s training data sounds like it shouldn’t require any extra effort, and so I think it’s a reasonable enough point to make.
These kinds of failure modes for "AI" are the most interesting to me.
It seems extremely smart - but don’t you think that to have success in a mass market product - think like MS Teams - it would need to be a combined solution. Where it both can do this for faces, efficiently, but also continues to work in a predictable way if I want to show an item/page from a book/my cat/my kid to other people in the call?
Yes I agree I have to work more on the business side of things, definitely on the lookup for for a business-savvy co-founder, hints at potential companies to partner with, etc.!
This is just a phenomenal idea - I hope your patent is approved too!
>> This also solves dick-pics etc.
Is this a problem on zoom meetings, for people other than Jeffrey Toobin?
What is the invention? The models are just complex mathematical formulas and these cannot be patented.
I am not a super fan of patents, but for background please consider that Asger Jensen and I could have patented VM live migration in 2002 and chose not to for idealistic reasons, just to see VMware do it.
Well, in my country they actually don't. YMMV as per one's location.
But the story has a deeper meaning for me (because of the book I am writing). You switched from street face surveillance (an arguably highly unethical application) to more intimate videoconferencing (a more humanistic and socially beneficial end).
May I ask you in all sincerity, what if any ethical considerations played a part in your change of direction?
I suspected from the title to read at least some sub-text that you turned your back on mass-surveillance to find a "better" use for your work. But you express no value judgements and only really mention that the pandemic took away potential targets.
That said, we build embeddable/edge face rec because we could, and I believe our partners who used it in the real pre-pandemic world found some very innocuous uses for it. In one case we replaced a system running all the faces through Rekognition with one running purely on devices and not storing any long-term data, which I think was an ethics win overall.
Or that makes it easier to identify individuals as they give consent to have their face ‘fingerprinted’ as part of the app’s EULA.
If I were going to sell a mass-surveillance solution I’d certainly want to have the ability to identify individuals without having to scrape all of Facebook or whatever. As much as people hate on apple they do make it so carrying around one of their phones doesn’t make it easy for someone to identify you.
I, for one, would think twice about installing an app from someone who “pivoted away” from their Orwellian surveillance unicorn dreams.
Ideally I would like to collect faces to train the compression on, in which case we would have to consult with lawyers to come up with an EULA allowing us to do this. The advantage compared to using broadly available datasets to train on would be more realistic shot noise, low light images, and so on. I don't see any other valid business purpose of collecting people's faces.
We've been sitting on the face recognition tech since 2018, so if we'd wanted to become Clearview.ai we probably would have a long time ago.
It says right at the beginning of the post you were doing quite well until the pandemic shut down businesses.
I try not to be overly critical (I really do) but this is one of those cases I just can’t help myself, I see no reason individual businesses should be running facial recognition on their customers and am kind of wary of someone who would enable that. And cities adding it to their collection of public cameras is beyond wrong.
IDK, somewhere we, as a society, decided 1984 was an instruction manual and not a warning…
The one face rec system we actually sold was used to measure waiting times in a retail setting, and replaced an existing system that was using AWS Rekognition in the same manner, except with all the video footage going to the cloud. That license has long expired and the system is no longer running.
In any case, what is stated in the app's privacy statement is what we are doing. At the moment I don't even collect the IP addresses of users connecting.
Wrt encoding/decoding complexity this is the major bottleneck, because you have the GPU competing with custom ASICs. I have a version of the codec that works in combination with H265, but still gets largish bandwidth gains, so if all you wanted was an insane hi-res and hi-bitrate transmissions that might be the way to go near-term.
Have you considered entering into the YC Program? I think it could be an awesome match. There are many startups I know they may want to take use of your service, and even fly.io is part of YC family!
Also, have you thought about open-sourcing it? (perhaps using a dual license could work great for an enterprise offering)
I tried entering YC in the fall 2021 batch, and got to the top 10%. I believe my main problem wrt YC is that I currently lack a co-founder, so I did not apply in the Spring as this was still the case.
I am seriously thinking about open source, I believe for instance WebRTC found a good model with dual-licensing, where you have something like AGPL with the option of buying exceptions.
I have had multiple advisors telling me not to though, they fear it would scare away any potential investors ;)
In any case I'd love to help you on both aspects (YC application and OSS), I believe your idea has really great potential. Please ping me to syrus@wasmer.io and we can schedule some time to chat!
The face boxing seems very aggressive, I feel like I’m trapped in the virtual prison of some 90s dr who episode.
> The rooms are hosted on a Raspberry Pi 3 lying on the floor of my office.
Hah, probably :D
One thing that struck me as odd on the page is that the H265 still looks considerably worse than H264, despite being the better codec and being larger. What's up with that?
Who would have thought anti dick pic technology would become a product feature but here we are in 2022.
It should do peer-to-peer for the majority of connections though, the server just does the initial hand-shake.
A thought: now that neural compression is becoming widespread, it could be a good idea to put some kind of indicator or watermark stating the compression is neural (learned/function approximation in general). I think this would avoid liabilities and criticism around the fact that some weird things may appear (incorrect detail generation), maybe giving a wrong semantic idea. It may also be a good idea to put a mean squared error term in your objective function to help preserve general meaning.
Absolutely. Reminds me of Xerox number mangling:
https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres...
For example, you could train a network to give semantic image descriptions of significant features in the image, and maybe also transcribe text. Then you can include semantic preservation in the objective, or some kind of graceful degradation when semantic preservation isn't achieved.
https://petapixel.com/2020/08/17/gigapixel-ai-accidentally-a...
That ship has sailed.
Smartphone cameras and laptop webcams already use machine learning algos to improve low light performance and noise. The result is that the images already contain details that are generated.
And it's impossible to turn off.