TL;DR it's possible to detect where you tapped on your phone screen somewhat reliably. Theoretically, someone could develop an app to associate screen location taps to keys and could pick out a password. Authors suggest phone manufacturers should limit access to accelerometers to prevent the attack.
This is extremely contrived and would take a shitload of skilled work to get right. It's way easier to make a phishing page coupled with social engineering to get what you want.
I think step counters depend on being able to read accelerometer data. You might be able to add enough noise to the signal to make tap localization impossible while still keeping steps detectable, but that requires careful tuning.
On iOS at least, most ‘step counter’ apps don’t acually do the step counting - they just pull the data from HealthKit, as the phone is already tracking them.
Oh my mistake, it was actually I who misread the first comment.
Regardless, I don't think a screen tap logger would be necessary in many cases, since you could just open the keyboard on the phone and manually check the bounding boxes for each key.
Making this work on N phones would be more difficult, but not unfeasible. You'd probably just need to know a few things about the phone (resolution, screen size), and it wouldn't be hard to find that information.
Would be especially easy to just target the limited iPhone line.
Is this why most password entry forms in Android do not allow use of the Swype-type keyboard? I always thought that was strange. Now it makes more sense....backdooring
Wouldn't that have more to do with passwords not being proper words? Swiping is like autocomplete, f7_4sl+lsS$@q.. can't imagine swiping something like that.
I was a bit disappointed by the lack of a real scenario implementation. While the data suggests it might work, I wanted to see how precise it would be in a simple implementation like the author described towards the end, where a simple game of tapping trains the NN and then uses the information to try and capture a typed password.
If I got that close to making something as cool as this, I would not stop experimenting until I found out whether it worked or not.
That is, unless I tested it, found that the imprecisions made it impossible to obtain reliable data, and decided to not write about it to not discredit all the work I had done.
I really hope someone takes the idea and tries it out to assess how viable it is, then I can be paranoid about 2 more sensors spying on me.
Also this is mindblowing, logging keystrokes from a standard keyboard using wifi.
"we show for the first time that WiFi signals can also be exploited to recognize keystrokes. The intuition
is that while typing a certain key, the hands and fingers of a user move in a unique formation and direction and thus generate a unique pattern in the time-series of Channel State Information (CSI) values"
Wouldn't the model be different for each user? and different for different typing scenarios? Like typing one handed vs two, sitting on a bus, in a car, while walking, versus just sitting on the couch.
It seems like it could be possible, but it would require a lot of training. Get it into a popular note-taking app to record keystroke + accelerometer data.
Also a simple fix for iOS as a platform would just be blocking out or filtering motion data when the secure keyboard is showing. I assume a similar thing could be done on android.
To allow for training of the ML to differences of each users the authors describe have the end-user first play a game
Unsuspecting user downloads “Evil Flappy”, an app where they have to tap on the screen mindlessly to advance some objective. During this tapping, the app uses transfer learning to tailor the model to the user and test its own predictive capacity.
Attacks like these can be easily dwarfed by employing a nonstandard virtual keyboard which displays a randomly permutated key layout during PIN entry. Some online banking web apps did this already ten years ago as a defense against "mouseloggers". Using a good RNG is advised though.
Neat, though i'd call it key-estimation rather then logging, as i consider logging a very accurate, 1:1 log of what actually happened, this could have some errors in it, not making it less 'dangerous', lack of better word.
But for now i'll have to randomly rotate my device at each input... And or get one of those keyboards that shuffles the letters around the keyboard at each input.
27 comments
[ 4.3 ms ] story [ 14.9 ms ] threadThis is extremely contrived and would take a shitload of skilled work to get right. It's way easier to make a phishing page coupled with social engineering to get what you want.
On iPhone it restricts background activity after 10 minutes, but on Android you'd have free reign until they manually close the app.
Sorry, looks like I misunderstood them (didn't notice they were referring to the paper).
Regardless, I don't think a screen tap logger would be necessary in many cases, since you could just open the keyboard on the phone and manually check the bounding boxes for each key.
Making this work on N phones would be more difficult, but not unfeasible. You'd probably just need to know a few things about the phone (resolution, screen size), and it wouldn't be hard to find that information.
Would be especially easy to just target the limited iPhone line.
If I got that close to making something as cool as this, I would not stop experimenting until I found out whether it worked or not.
That is, unless I tested it, found that the imprecisions made it impossible to obtain reliable data, and decided to not write about it to not discredit all the work I had done.
I really hope someone takes the idea and tries it out to assess how viable it is, then I can be paranoid about 2 more sensors spying on me.
https://www.usenix.org/legacy/event/hotsec11/tech/final_file...
They focused purely on the number keyboard to try and extract PIN numbers and were able to achieve 70% accuracy using only a basic classifier.
I wonder how much newer smartphones with bigger screens and more precise sensors could improve these results.
It's scary to think of how many different sensors could be used to gather even more information to make it more precise.
Also this is mindblowing, logging keystrokes from a standard keyboard using wifi.
"we show for the first time that WiFi signals can also be exploited to recognize keystrokes. The intuition is that while typing a certain key, the hands and fingers of a user move in a unique formation and direction and thus generate a unique pattern in the time-series of Channel State Information (CSI) values"
https://www.sigmobile.org/mobicom/2015/papers/p90-aliA.pdf
Edit: I wonder as an aside how precisely you could potentially use the accelerometer etc. for dead-reckoning.
Edit: Wow, I'm just skimming it at the moment, I'd never heard of 'Magnetic Particle Filtering' before, that is really amazing!
Also semi-related, I just found this https://en.wikipedia.org/wiki/Magnetic_anomaly_detector. Apparently they degauss submarines precisely to avoid that :)
It seems like it could be possible, but it would require a lot of training. Get it into a popular note-taking app to record keystroke + accelerometer data.
Also a simple fix for iOS as a platform would just be blocking out or filtering motion data when the secure keyboard is showing. I assume a similar thing could be done on android.
Unsuspecting user downloads “Evil Flappy”, an app where they have to tap on the screen mindlessly to advance some objective. During this tapping, the app uses transfer learning to tailor the model to the user and test its own predictive capacity.
Very sneaky indeed...and I loved Evil Flappy :-)
But for now i'll have to randomly rotate my device at each input... And or get one of those keyboards that shuffles the letters around the keyboard at each input.