They manage to do it (just sound, not keypresses) with a 60fps DSLR camera at the end by examining the rows of the video - does this mean that for any videos already in existence, sound can be decoded from the images?
For this particular open-source hobbyist project, not likely; those who're most likely to attempt to use such a tactic maliciously are likely to keep it to themselves rather than demonstrate that the security concern exists.
Attempted to train it by typing for ~2 minutes. Basically typed everything on the GitHub page and tried getting predictions.
Results were disappointing. I didn't see any accurate predictions. Even with the default p/q program I see mostly random results.
I tested on a 15" Macbook Pro 2018 (latest version of keyboard that is softer to type of / less noisy)
The p/q test fails on my Macbook Pro 2013 as well. Plugging an external mechanical keyboard makes it work, so it's definitely due to the soft type of keyboard and not the mic.
I've tested this on my Filco Majestouch 2 with MX Browns but it barely managed to detect anything, and even then it didn't do very well. I think this is also highly dependent on the microphone used (I used the one in my Logitech webcam).
Could you maybe do away with training beforehand on a single source and instead use multiple, very sensitive microphones and triangulate the locations of the keys being pressed? The estimations might not be accurate, but you could put the results through a smartphone typo correction algorithm.
I've always thought Twitch streamers were opening up an attack vector through this exact method.
Cool to see someone follow through with it. Any streamers out there should figure out a way of avoiding keypress bleed or muting their mics when typing sensitive info (e.g. passwords).
Yes, countermeasures, such as playing other sounds around those frequencies and/or filtering the microphone at those frequencies would be interesting to explore. Filtering the mic seems the less annoying from a user perspective!
Most streamers, myself included, use a filter called a noise gate, which requires the volume of the mic reach a certain threshold before being broadcast. This filters out the majority of background noise on the stream.
This is super cool. Very similar to timing attacks. I wonder if there is a way to tune the model. Essentially cater the probability of each key to a person with maybe a sentence or something.
In keytap2 I'm trying to make use of the statistical distribution of n-grams in the language. The idea is to first group the unknown keys into clusters based on how similar they sound. The prediction then is performed by breaking the obtained substitution cypher (assuming each cluster corresponds to a letter).
Can you combine this with or does it use relative positioning of the microphone? Seems like a good way to map where keys are (measure lower decibels which would be keys further from the mic).
In what scene? I don't think he actually did... I'm pretty sure when they're trying to crack the guy's password they're using video to record and zoom in to watch him and he's blocking it.
There is more research on this than what you have cited here, unfortunately the literature goes under the catchy keyword "acoustic keyboard emanations", try that in Google scholar.
35 comments
[ 2.1 ms ] story [ 98.6 ms ] thread[0]http://news.mit.edu/2014/algorithm-recovers-speech-from-vibr...
Its not a complicated paper, give it a go.
For this particular open-source hobbyist project, not likely; those who're most likely to attempt to use such a tactic maliciously are likely to keep it to themselves rather than demonstrate that the security concern exists.
I tested on a 15" Macbook Pro 2018 (latest version of keyboard that is softer to type of / less noisy)
https://github.com/ggerganov/kbd-audio/issues/3
https://www.youtube.com/watch?v=2OjzI9m7W10
Cool to see someone follow through with it. Any streamers out there should figure out a way of avoiding keypress bleed or muting their mics when typing sensitive info (e.g. passwords).
https://www.math.unipd.it/~dlain/papers/2017-skype.pdf