We hated using the keyboard and mouse for controlling music in the background or watching movies on our computers from a few feet away. So we decided to solve this problem by using hand gestures to control music and videos via the built-in webcam in your computer.
Please download and try it. Do let us know what you think in your comments below or by emailing us at feedback [at] flutter.io.
after typing in my email and name i got this:
The link that you have given is not valid. Please verify that you have copied the link completely. This incident will be reported.
I hardly ever get excited after downloading something, maybe a, wow that's cool, but when I installed this (which was very easy) and it just worked, I actually clapped and said 'this is fucking awesome.'
I had the exact same feeling. You see these gesture controls in movies and demos and they seem great, but it's truly exciting when you get to use it yourself.
Thought that this was great - how about a next/previous track gesture, and being able to change the frequency that it checks for the hand? (it looks like it's time based)
Surveillance in the name of convenience has gone too far. I can see a future where companies offer services as wrappers around visually analyzing people's behavior in their own homes. They will sell that data just as they do now. That's not a future I encourage.
I so hope this is an art project and in a few days we'll have a website of people making funny gestures to the camera to pause and play Justin Bieber songs ;)
Though I know there's a chance you'd not be willing to share the information, I'd love to learn a little bit about how you built this product.
It appears as though you're using ML to recognize the gestures—anything else you could share would be fascinating. I'll be sending in a job application, as well!
You're right. We use image processing, computer vision & machine learning to detect Flutters. Our entire tech stack is custom written as we are solving some quite challenging & difficult AI problems.
"The music swirled and dived for a moment. Another voice broke in, presumably Halfrunt. He said "Vell, Zapbod's just zis guy, you know?" but got no further because an electric pencil flew across the cabin and through the radio's on/off-sensitive airspace. Zaphod turned and glared at Trillian -- she had thrown the pencil. "
This is very cool. It worked perfectly even in the dimly lit room I am sitting in.
What I've been wanting and have thought about developing myself is a universal play/pause control that pauses whatever you happen to be listening to at the moment, whether it be a desktop client like iTunes, Spotify, Rdio, etc, or if it is something running in a web browser like Pandora, Soundcloud, Youtube, etc.
This app takes a step towards this in the way it handles iTunes and Spotify at the same time, by controlling whichever app had focus last.
I imagine for web browser control a browser extension would necessary. And ideally a plugin architecture would exist to allow 3rd parties to add support for additional apps.
A different but functionally similar idea would be an automatic pausing of all other services when you start playing one (i.e. when I start playing a Youtube video, automatically pause iTunes).
There needs to be a way to override those behaviours though, as there are situations where the audio of a YouTube video (for example) isn't very interesting, and you'd rather listen to music. Lack of background-playing in the iOS YouTube-app is still an annoyance to me.
However, music control isn't a pain point for me. What I would pay for is a start/pause gesture for Netflix/Hulu/Amazon Video. Worst thing about TV on a computer.
My friend and I looked into controlling those apps for a program we made to control media players with your voice called Shoevox - http://shoevox.org
The hard part was that we used sending keyboard shortcuts as our method of sending commands (which works for programs we support like WMP or VLC), but we couldn't do that reliably to Flash. We thought about switching to UI Automation but haven't done so yet.
In case anyone was wondering, I can confirm "mirror stops" work: try facing your laptop towards a mirror and doing your gestures from behind the computer- wow!
I assume it is looking for 5 fingers. I noticed if i folded my thumb in, it did not recognize the gesture so maybe an edge case where people missing fingers can't use this.
Also when I scratched my face briefly it did recognize it... funny.
I would strongly recommend trying to build a web demo somewhere down the line. Even if it requires flash to get the webcam working, I'm sure your conversions would go through the roof if you could show how well it worked instantly on your landing page.
Small note -- the blurring of the image is distracting rather than focusing. Down the line maybe think about a better landing page image.
This works incredibly well. I wasn't sure how accurate this could be without the kinect-style depth sensor but it's surprisingly accurate. Great work guys!
What's nice about this is you don't need to switch screens to pause the music if you find yourself writing/coding and need some quiet when you're in the zone. Granted you can do this with the mac play/pause physical buttons but that would require your eyes to look down and pause your thought process.
This happened to me just now after I installed Flutter, and it seemed to me that physical gestures didn't interrupt my higher brain functions. Of course when I noticed this it made me interrupt my work and come to HN to leave this comment. The irony.
Glad you noticed it. We want you to never think about controlling your music at all. This is one of the things I love too... we listen to music in a background, so controlling it should be in the background too.
That's what I like most about it too. Love not having to switch screens to stop the music when a call comes in. One thing: I didn't find the installation super intuitive. Probably some room to improve that experience
By the way I see great application in this for education, particularly with smaller kids
I was blown away when I saw a demo of Flutter a couple of months ago. Their Spotify and iTunes integration only scratches the surface of all the gestures that they supported in that demo. I can't wait for them to add some additional gestures, such as skipping to the next track, and PC support so that I can get rid of my keyboard with the dedicated music player controls.
Really cool. I was just playing with it and my girlfriend walked over to ask me a question. I held up my hand to my computer as if to say "Hold on, computer."
It was a bit surreal. Very cool. Can't wait for more gestures or a generalized way to program my own gestures. I would love to be able to gesture "Switch to iTerm.app" by doing an OK sign or something. Sign language letters maybe?
Anyway, thanks. Really enjoy it already, can't wait for more. Also, I would pay money for this. If it had all of the play/pause/forward/back functionality I think I'd give you 4.99 if it was accurate. Just sayin'.
My thoughts almost exactly on the programmable front... like Better Touch Tool but rather No touch Tool, although the latter name does retain the locker-room hilarity of the former.
Perhaps a gesture to bring up the app switching ribbon and a gesture to reference the item (numbered from left to right maybe.)
I'd also like to have the option to calibrate it, my eyes are painfully photosensitive and I tend to work in very dimly-lit conditions.
127 comments
[ 8.6 ms ] story [ 879 ms ] threadPlease download and try it. Do let us know what you think in your comments below or by emailing us at feedback [at] flutter.io.
Currently, it works for Mac OSX 10.6 and up.
Are you on Mac? Can you tell us what happened?
Love it!
which kept switching your music off and on?
Expected reaction, having a video demonstration of the product on your landing page: "THIS IS FRICKIN' AWESOME!".
Please, please, but a video on your landing page. You will thank yourself after checking your conversion rate.
It appears as though you're using ML to recognize the gestures—anything else you could share would be fascinating. I'll be sending in a job application, as well!
HHGTTG - Douglas Adams
I... wow.
Take my money.
What I've been wanting and have thought about developing myself is a universal play/pause control that pauses whatever you happen to be listening to at the moment, whether it be a desktop client like iTunes, Spotify, Rdio, etc, or if it is something running in a web browser like Pandora, Soundcloud, Youtube, etc.
This app takes a step towards this in the way it handles iTunes and Spotify at the same time, by controlling whichever app had focus last.
I imagine for web browser control a browser extension would necessary. And ideally a plugin architecture would exist to allow 3rd parties to add support for additional apps.
However, music control isn't a pain point for me. What I would pay for is a start/pause gesture for Netflix/Hulu/Amazon Video. Worst thing about TV on a computer.
The hard part was that we used sending keyboard shortcuts as our method of sending commands (which works for programs we support like WMP or VLC), but we couldn't do that reliably to Flash. We thought about switching to UI Automation but haven't done so yet.
I assume it is looking for 5 fingers. I noticed if i folded my thumb in, it did not recognize the gesture so maybe an edge case where people missing fingers can't use this.
Also when I scratched my face briefly it did recognize it... funny.
I would strongly recommend trying to build a web demo somewhere down the line. Even if it requires flash to get the webcam working, I'm sure your conversions would go through the roof if you could show how well it worked instantly on your landing page.
Small note -- the blurring of the image is distracting rather than focusing. Down the line maybe think about a better landing page image.
Implement it for Android and you can charge me 30€ so i can easily control the mp3 player in my car.
This happened to me just now after I installed Flutter, and it seemed to me that physical gestures didn't interrupt my higher brain functions. Of course when I noticed this it made me interrupt my work and come to HN to leave this comment. The irony.
By the way I see great application in this for education, particularly with smaller kids
It was a bit surreal. Very cool. Can't wait for more gestures or a generalized way to program my own gestures. I would love to be able to gesture "Switch to iTerm.app" by doing an OK sign or something. Sign language letters maybe?
Anyway, thanks. Really enjoy it already, can't wait for more. Also, I would pay money for this. If it had all of the play/pause/forward/back functionality I think I'd give you 4.99 if it was accurate. Just sayin'.
Perhaps a gesture to bring up the app switching ribbon and a gesture to reference the item (numbered from left to right maybe.)
I'd also like to have the option to calibrate it, my eyes are painfully photosensitive and I tend to work in very dimly-lit conditions.