You could integrate this with adtech to help make advertisements more interactive and to analyze how the viewer is receiving the ad by their pose and facial expressions. Based on that you could have an algorithm serve only ads that improve facial expressions and poses to positive ones
I can't speak about this specific project but pose tracking in the browser can actually be really useful to get it on mobile without the need for dedicated apps.
We are currently experimenting with using phones (or tablets or basically anything with a browser and a camera) as a tracking source for full body tracking on the Oculus Quest ([1]) and being able to just use a Tensorflow model in the browser has cut down the time of development on the mobile device side to almost nothing.
There are of course a whole lot of other issues one buys into with that approach (browser security policies, the whole thing not working in certain countries due to problems with loading the model) but that's manageable
I made a slouching detector a while back. It calibrated the distance of your face from your laptop and if you start leaning in it’ll give you a notification to sit back. It’s a bit dated but still works on Chrome and FF, I believe:
Wow this is amazing!
I remember a few years ago in my thesis I was trying to use something similar in ROS to detect hand gestures and make a robot drive toward you, it's incredible to see this in the browser!
I'll use this post / opportunity to ask the HN community
(I understand that this is a technically complicated, and potentially sensitive subject)
I've been approached by a few people who are doing what they can to lower the cost of prescription glasses. It's a mix of (licensed) ophthalmologists, people interested in offering community health services, etc.
What is the current state of the art for pupillary distance (PD) measurement based on face detection / pupil tracking using laptop webcams or mobile (front) cameras?
I imagine (obviously?) that there must be scholar research on the acceptable error margins for a PD measurement (depending on the type of vision condition, i.e. farsightedness, etc.)?
Would using something like Handsfree.js or https://github.com/esimov/pigo (which has pupil detection) be a good start, or would these be an ~order of magnitude off in terms of the necessary margins?
It comes down to physics. Cameras and their lenses come in all shapes and sizes, most of webcams don't expose their geometry in a programmatically accessible way, even if they did, most of them don's have means of determining distance to the subject.
We can detect faces, eyes and pupils, but with no reference points we have no idea how big or far apart they are.
What about sites (such as Warby Parker) that make you hold up a credit card (which has a standard size) and use that to determine (via what I assume is a rule of three) the PD? [0]
I can see of course problems with the card not being “perpendicular” and other 3d distortions.
And Google's MediaPipe Iris used iris diameter for distance estimation.[1]
EyeQue is perhaps a cautionary tale. "Change the world" "few dollars of optics + phone = D+CYL" become $45 one-person-per-subscription "vision tracker".
There are a lot of problems with measuring pd accurately with a photo. Ideally you need to ensure the pupils are aligned (i.e. the head can't be turned) which is really hard to tell from a photo (there's no depth).
The credit card trick somewhat works since you can guage depth given the size and positioning of the card, but the errors still can end up in the 2-3mm range fairly easily.
The best mechanism would be to create something like a Google cardboard to guarantee the head is properly positioned and then perform the measurements.
This probably isn't very helpful, but I'd imagine underestimating the PD is going to lead to happier patients even if not all that accurate. We can accommodate with crossing our eyes, but the opposite is not usually true.
If my memory serves me correctly, I once a used a digital try-on app from an online glasses retailer and they had me hold a common item with a known width (something like a credit card or drivers license) in front of the cam within a bounding box. It could guess the location of my pupils but allowed me to drag the pupil location markers to the exact centers. Calculating pupillary distance was then just a matter of using the distance in pixels between the pupils and converting them to mm using the ratio of pixels to mm of the card in the previous photo.
I have tried the underlying library, MediaPipe/TensorFlowJS, and it is only okay. Once you try to use it for visual augmentation, I found it doesn't track that well and it has problems with scale.
So, while it may look good with the skeletons, notice they do not have useful demos, mostly just proof of concept demos that do almost nothing.
Have you tried the newest MoveNet from google?
That thing works really well. I only started experimenting with the Lighting model (not Thunder) for our VR game but even that is pretty robust.
Just the 2D positions plus a confidence level. In the video we are using only one view so it's no true full body tracking yet. We are currently trying to use two views to get a rough 3D tracking.
The good thing is that the positions of the detected points are relatively stable so with a big enough baseline that should work out for a rough 3D pose estimation
I'm very sorry you don't think Handsfree.js has useful demos. That kind of hurts because I put a great deal of effort into making them useful as starting points (I started all this while I was homeless to help someone). I work on Handsfree.js full time and my goal is really to create simple demos so that others like yourself can run with it!
35 comments
[ 2.9 ms ] story [ 87.8 ms ] threadWe are currently experimenting with using phones (or tablets or basically anything with a browser and a camera) as a tracking source for full body tracking on the Oculus Quest ([1]) and being able to just use a Tensorflow model in the browser has cut down the time of development on the mobile device side to almost nothing.
There are of course a whole lot of other issues one buys into with that approach (browser security policies, the whole thing not working in certain countries due to problems with loading the model) but that's manageable
[1] https://www.youtube.com/watch?v=zz9S5hgrWpM (relevant part starts at 0:29)
https://slouchcam.com
(I understand that this is a technically complicated, and potentially sensitive subject)
I've been approached by a few people who are doing what they can to lower the cost of prescription glasses. It's a mix of (licensed) ophthalmologists, people interested in offering community health services, etc.
What is the current state of the art for pupillary distance (PD) measurement based on face detection / pupil tracking using laptop webcams or mobile (front) cameras?
I imagine (obviously?) that there must be scholar research on the acceptable error margins for a PD measurement (depending on the type of vision condition, i.e. farsightedness, etc.)?
Would using something like Handsfree.js or https://github.com/esimov/pigo (which has pupil detection) be a good start, or would these be an ~order of magnitude off in terms of the necessary margins?
Thanks a lot.
We can detect faces, eyes and pupils, but with no reference points we have no idea how big or far apart they are.
What about sites (such as Warby Parker) that make you hold up a credit card (which has a standard size) and use that to determine (via what I assume is a rule of three) the PD? [0]
I can see of course problems with the card not being “perpendicular” and other 3d distortions.
[0] - https://www.warbyparker.com/pd/instructions
And Google's MediaPipe Iris used iris diameter for distance estimation.[1]
EyeQue is perhaps a cautionary tale. "Change the world" "few dollars of optics + phone = D+CYL" become $45 one-person-per-subscription "vision tracker".
[1] https://ai.googleblog.com/2020/08/mediapipe-iris-real-time-i...
The credit card trick somewhat works since you can guage depth given the size and positioning of the card, but the errors still can end up in the 2-3mm range fairly easily.
The best mechanism would be to create something like a Google cardboard to guarantee the head is properly positioned and then perform the measurements.
So, while it may look good with the skeletons, notice they do not have useful demos, mostly just proof of concept demos that do almost nothing.
If you want to see how stable it is, have a look at that: https://youtu.be/zz9S5hgrWpM?t=147
It's tracking me with the Oculus Quest on the head, so it does not even have facial features to go on
Maybe you'll find some of these more useful?
- Face Pointer, and accessibility tool to help people who can't use a mouse/keyboard: https://handsfree.js.org/ref/plugin/facePointer.html
- Palm Pointers, scroll the page and even multiple scroll areas at once: https://handsfree.js.org/ref/plugin/palmPointers.html
- Gesture Mapper, map static poses in seconds: https://handsfree.dev/tools/gesture-mapper/
- Face Coding (not ready), to help people with disabilities code by snapping blocks together: https://twitter.com/GoingHandsfree/status/140056015231972966...
- The library itself doubles as a Chrome Extension, here are some examples of what you can do with it: https://handsfree.dev/sites/
I have dozens and dozens of more experiments. Most of them are pretty basic, but the point is to show you what's possible :)
...
I did do another whacky one today that is a lot easier to do. It's called "Flintstone Mode": https://twitter.com/GoingHandsfree/status/140136648492059033...
Pretty fun to watch!