Wish they had waited to publish for when he had posted the plans. I am also curious to see the BOM, because in the article it said he spent thousands on equipment, so I hope that was V1, and a working model can be had for a more modest budget.
Supposedly the White House uses vibrating windows to avoid this class of attacks.
I'm also super excited to see more on this, it's been living rent free in my head for a long time. I played around with keypress extraction from sound recordings a couple decades ago in college, and it was remarkably straightforward to extract keypresses based on sound signals, even with zero training - based on the time between keypresses and the unique signature of space/return/backspace you can build a predictor pretty quickly. We never made it to the "bounce a pair of lasers off two window panes a known distance apart and triangulate everyone's keyboard", but it was one of those things that's definitely doable with enough time and brains.
I remember making a demo in 6th or 7th grade of console login dialog and disclosing password asking friends to log in.. they couldn't, because during typing my program would analyze time between pressing different keys.. I don't remember if I hard coded it or it was trained first... But yeah, essentially it was checking not just the password, but also signature of it..
Weighted curtains don't really help with sound dampening that much unless it's somewhat air tight. I shopped around a bit for window sound dampening products, them need to form seal around the window hole.
I've worked in a few places that use vibrating windows to defeat parabolic directional microphones; they've been an imoportant counter espionage measure for at least 25 years - probably a lot longer!
I'm assuming this cannot be either a stable signal or a well known source like a local Radio station because then you could "subtract" the noise and .. well get the in-room sounds just like you used to.
Or it has sufficient energy to make any transmission effects through vibration useless because broad-spectrum the glass is shaking to shakira too much anyway.
I wonder what stopping three letter agencies to secretly push for keyboard manufacturers in such way that each keystroke has unique sound and it becomes easier to detect keystrokes.
even if its not, enough decent fidelity data with letter/word frequency analysis
paired with small Neural net will quickly disambiguate keystrokes after the first paragraph.
The problem is, as stuff becomes easier and cheaper, usage becomes more widespread.
DNA tests used to be in the many thousands of $ range, so they were used for heavy crimes like murders and rape only. Nowadays, it's routine for police to use them for petty crimes like graffiti [1].
It's just the same for camera surveillance, mass exfiltration and analysis of just about the whole Internet's worth of traffic... even searching phones and datamining them is cheap enough these days that US CBP does it for travellers, and German authorities for refugees (until a court order stopped that crap [2]).
This kind of conspiracy theory mindset really just doesn’t stand up to any scrutiny. Every keyboard manufacturer is going to build this in without anyone ever giving away the secret? How many people have to be in on the secret for this to work?
the problem with these rebuttals is that these types of exploits are regularly leaked (well on average: rather occasionally but in large batches, like Snowden leaks).
its like saying, humans haven't reached the moon, even though humanity witnessed it in the sixties.
Related to this Applied Science on YouTube has a pretty cool video where he demonstrates how a single laser diode can be used to measure miniscule vibrations. I wonder if this would be sufficient for a laser microphone?
Thinking in product terms instead of surveillance terms, imagine wireless keyboards that don't need batteries, bluetooth, or even any internal electronics!
I can also see a future home assistant using a laser microphone to pick up on users' speech from anywhere in a room without needing to raise one's voice above a mutter.
Is the amount of power needed for the laser lower than the amount of power a USB-keyboard, or a Bluetooth connected keyboard needs?
Either way, I’m all for laser monitored keyboard. If nothing else then just for the reactions of others when I bring my laptop and a USB-corded keyboard and start typing on the external keyboard without plugging it in :)
(Although I guess, there’s a pretty high chance they will just assume that the keyboard simply has Bluetooth and that that is how it’s working even though not plugged in.)
You just gave me a cool idea. You could do this by fitting small tuning forks (or similar) to a mechanical keyboard. Bonus points if all of the forks are tuned to a pleasant scale.
Then all you need is a microphone running on the destination device. Obviously this is insecure but could be fun anyway.
It would be difficult to manage the ~90 different tuning forks such a keyboard would require. Far more sensible for each of the keys to be connected to some sort of tensioned wire, the tension of which could be adjusted to achieve the required tuning.
We can only await the revival of keyboard tuners, and the interviewees of the future puzzling over how many of them are required for a city the size of Chicago.
It's also not clear how you would handle keys that are held down. Or rather, how you would distinguish between keydown and keyup events. But perhaps with your tensioned wire you could use a pick on the key (such as in a harpsichord) which has a different texture on each side, affecting the timbre but not the pitch.
I don't think the keyboard would be accurate enough with current ones, but if they were modified to have unique tones or audio characteristics it might work.
The room mic would be awesome, as long as laser safety concerns were addressed.
Every key stroke has a unique sound signature. You just need to record it with a microphone that isn't terrible. For this reason a bugged room can be a bug in every keyboard in that room if the attacker is assumed to be sophisticated. There have been some projects which attempted this[1].
It may be hard to do a non-terrible DIY laser microphone that would allow to achieve this while targeting a non-laptop surface.
After you have the recording the biggest challenge is a problem solving one, you need to isolate the waveform for every keystroke, convert it to something that can be compared with other waveforms (and cluster them) and then solve the resultant substitution cipher (waveform -> key identity). If the keyboard was used for any length of time, you can assume the entry of common words and phrases occurred at some point and the solution to the cipher is trivial.
If you have the resources to collect the data, it should be possible to train a transformer model to convert an audio recording to the output of sound signatures belonging to key strokes of a keyboard or multiple keyboards. The trick would be to not try to train it to do the entire problem which will almost certainly fail or not generalize.
Re. "every key stroke has a unique sound signature": is this because each key in a keyboard has subtle differences/defects unique to that particular device, or because of where the key sits within a QWERTY keyboard, geometry-wise?
I'm trying to understand whether this requires a "decoder ring" recording ahead of time (in the case of the former—okay, on this keyboard an A sounds like this, a B sounds like this...), or whether you'd be able to pull this off with no prep work because an A always sounds like an A and a B always sounds like a B.
Your question is off a bit. It isn’t exactly either of those.
Each combination of room, mic, keyboard, and letter is going to have a unique sound signature. But, you don’t need prep work.
If you can just record enough keystrokes, it’s possible to figure out what sounds go with what letters. This is the “substitution cipher” the other comment mentioned.
For example, the sound that comes every 2-10 letters or so but rarely twice is spacebar. The one that sometimes comes between two spaces is I. It’s obviously more advanced, but just to give you an idea.
Why does it need to be perfectly accurate? These are examples intended to communicate the principle, not instructions on how to do frequency analysis. This description may be incomplete, but it does make it pretty clear how frequency analysis works in general terms. This criticism seems misplaced to me.
It would be trickier however if looking at raw keystrokes, as one has to consider several confounding factors:
* The caret can be moved both by keystrokes and mouse
* You don't know what is being typed. If someone is typing people's names all day, or writing software then the analysis of letter/key frequency will be shifted
Yeah, lots of different uses for a keyboard with wildly different frequency distributions. Writing an English letter != text chat in Spanish != data entry != league of legends. It's also not quite a substitution cipher due to modifiers (shift), toggles (caps lock), non-printable characters (escape, volume up/down), and deletion (backspace/delete).
It's an interesting problem you'd probably try to solve with Markov chains back in the day, but now you'd just throw machine learning at.
As other commenters have said, you're just going to learn the decoder ring from context, but I'm also curious. If I take off the keycaps to clean the keyboard and then put them back, do I need to relearn the defects? If I rip out all the switches and rearrange them, do the defects follow the switches or is it more about the position of the key? Is there some resonance that could change if I move the keyboard onto a desk pad. If I switch to Dvorak in software, obviously the key presses mean something different, but also I'll type completely differently. How much does user cadence/timing matter? To what extent is user cadence/timing identifying?
Anyway, I assume the answers aren't known because it would be an ethics/privacy nightmare to run these kinds of experiments on anyone but yourself.
You would presumably need todo this for the exact keyboard and acoustic properties of your target. And , what if they move the keyboard to their lap? Or put the keyboard legs down?
Also many keyboards are very quiet clicks these days. And what if I’m playing music?
I imagine a surreptitiously planted keystroke logger is far simpler. Or perhaps camera(s).
If you have access to the target computer you would just hack it directly to send you the information like an evil maid attack. No reason to do this keystroke sound hooey..
Perhaps a speaker in the keyboard that emits a random click-sound to mask the real sound could be helpful here. But it might confuse the typist when it emits the spacebar sound after you typed a regular letter...
All you need to do is obscure the original sound of the key press so that it can't be distinguished from presses of other keys. You don't need to create an authentic key press sound from scratch
Not really just visualize 2 rows of alphabet: the top row normal alphabetic, the bottom row the character you'd have to type to select the upper row character.
This obviously assume the screen is not simultaneously snooped, which is feasible..
How "not terrible" are we talking here. I can totally believe this works with a proper microphone next to the keyboard in a room with no other sounds. A phone mic in an office environment though? I would be very surprised.
I just priced laser projection keyboards. 40.00 or so. I think that’s the ”safest” option.
Presuming you have sufficient levels of assurance that no software exploits are present and no keystroke loggers etc. The only way to attack and the value of the attack being high enough (root CA compromise?) that an adversary would deploy (laser) mics.
I figure that anyone this concerned would program a coupe of rubber duckies or yubikey static mode ? Then the emitted strings would be unknown.
Also what about collecting from the screen via emissions?
Tapping on different points on a desk also produces sound characteristics that are unique to those points.
One thing that could help to randomize that is a surface with tightly-coupled [truly] randomly-moving masses underneath, so that the resonances of the surface also change randomly.
(And if done very right or very wrong, you get a free Van De Graaf generator!)
---
Or: A laser projected keyboard where the keys are always shifting to different locations.
thats probably even worse since a laser is a coherent beam, and fingers interrupting a beam closer or further will change the spatial and temporal coherence of light emitted from the scene, thats a lot of information leakage...
It probably doesn’t matter. The Oval Office is a formal office, it is for ceremonial work, not sensitive work. But I don't think it's vulnerable anyway, because I've never seen a picture where there's even a computer in it.
Sensitive government information is handled in SCIFs.
That's hilarious! Could work but I'd need the keys correspondence shown on the pic while typing: if physical 'f' key (on a QWERTY keyboard) is now 'p', I need to see "p/[f]" at the physical location of key 'f'.
Then a nitpick:
> "For example, if you type only "the" all day, then when you press the letter 't' the letter 'h' will be on the home row."
God bless Unix with composable tools via stdin/stdout.
Once you run a script batching nearly all your jobs, good luck trying to guess the user it's doing.
A plus if you run tools like ii/jj or similar with bitlbee as your chat client. If you can bind vi/vim to it by using a file as the input, vi has the 'ab' (abbrevebiate) ex command (use nvi if you like unicode, in some systems it's called nvi2) which can autoexpand abbreviated input.
Such as:
ab obsd openbsd
ab nbsd netbsd
ab fbsd freebsd
ab hnews news.ycombinator.com
ab kbd keyboard
ab spc space
ab cmd command
I have a question: what measures are taken with touch systems (smartphones) to disable the accelerometer during password input? And would it be ludicrous to randomize a soft keyboard for password entry?
88 comments
[ 4.6 ms ] story [ 174 ms ] threadSupposedly the White House uses vibrating windows to avoid this class of attacks.
(I am willing to die on this hill.)
Or it has sufficient energy to make any transmission effects through vibration useless because broad-spectrum the glass is shaking to shakira too much anyway.
Declining is futile.
And he's got an awesome project using helium weather balloons to fly networked RGB leds synchronised to the DJ tunes.
https://github.com/ggerganov/kbd-audio
He's subsequently well known for llama.cpp.
even if its not, enough decent fidelity data with letter/word frequency analysis paired with small Neural net will quickly disambiguate keystrokes after the first paragraph.
DNA tests used to be in the many thousands of $ range, so they were used for heavy crimes like murders and rape only. Nowadays, it's routine for police to use them for petty crimes like graffiti [1].
It's just the same for camera surveillance, mass exfiltration and analysis of just about the whole Internet's worth of traffic... even searching phones and datamining them is cheap enough these days that US CBP does it for travellers, and German authorities for refugees (until a court order stopped that crap [2]).
[1] https://www.krone.at/2718383
[2] https://www.tagesschau.de/inland/innenpolitik/bamf-handydate...
its like saying, humans haven't reached the moon, even though humanity witnessed it in the sixties.
sometimes peoples memories are short.
https://www.youtube.com/watch?v=MUdro-6u2Zg
I can also see a future home assistant using a laser microphone to pick up on users' speech from anywhere in a room without needing to raise one's voice above a mutter.
Either way, I’m all for laser monitored keyboard. If nothing else then just for the reactions of others when I bring my laptop and a USB-corded keyboard and start typing on the external keyboard without plugging it in :)
(Although I guess, there’s a pretty high chance they will just assume that the keyboard simply has Bluetooth and that that is how it’s working even though not plugged in.)
take out a blank sheet of paper, draw a game pad with controls, start playing a game on screen by tapping those buttons you just drew
edit:
combine ..
Gestures from Apple Vision Pro
GenAI capabilities that can convert a sketch into a working web page
Smart Home automation
https://makeymakey.com/
https://www.bbc.com/news/technology-18303012
I think the point is that the keyboard itself needs 0 power, so it doesn't ever need to be tethered, recharged, have batteries replaced, etc.
Then all you need is a microphone running on the destination device. Obviously this is insecure but could be fun anyway.
The room mic would be awesome, as long as laser safety concerns were addressed.
It may be hard to do a non-terrible DIY laser microphone that would allow to achieve this while targeting a non-laptop surface.
After you have the recording the biggest challenge is a problem solving one, you need to isolate the waveform for every keystroke, convert it to something that can be compared with other waveforms (and cluster them) and then solve the resultant substitution cipher (waveform -> key identity). If the keyboard was used for any length of time, you can assume the entry of common words and phrases occurred at some point and the solution to the cipher is trivial.
If you have the resources to collect the data, it should be possible to train a transformer model to convert an audio recording to the output of sound signatures belonging to key strokes of a keyboard or multiple keyboards. The trick would be to not try to train it to do the entire problem which will almost certainly fail or not generalize.
[1] <https://github.com/shoyo/acoustic-keylogger>
I'm trying to understand whether this requires a "decoder ring" recording ahead of time (in the case of the former—okay, on this keyboard an A sounds like this, a B sounds like this...), or whether you'd be able to pull this off with no prep work because an A always sounds like an A and a B always sounds like a B.
Each combination of room, mic, keyboard, and letter is going to have a unique sound signature. But, you don’t need prep work.
If you can just record enough keystrokes, it’s possible to figure out what sounds go with what letters. This is the “substitution cipher” the other comment mentioned.
For example, the sound that comes every 2-10 letters or so but rarely twice is spacebar. The one that sometimes comes between two spaces is I. It’s obviously more advanced, but just to give you an idea.
A convenient theory but not a fact.
Thank you. A lot of people don’t get these are two different things. I’m not going for perfect, I’m going for “oh I get it.”
It would be trickier however if looking at raw keystrokes, as one has to consider several confounding factors:
* The caret can be moved both by keystrokes and mouse * You don't know what is being typed. If someone is typing people's names all day, or writing software then the analysis of letter/key frequency will be shifted
Not impossible, but makes the process harder.
It's an interesting problem you'd probably try to solve with Markov chains back in the day, but now you'd just throw machine learning at.
Anyway, I assume the answers aren't known because it would be an ethics/privacy nightmare to run these kinds of experiments on anyone but yourself.
Also many keyboards are very quiet clicks these days. And what if I’m playing music?
I imagine a surreptitiously planted keystroke logger is far simpler. Or perhaps camera(s).
Perhaps a speaker in the keyboard that emits a random click-sound to mask the real sound could be helpful here. But it might confuse the typist when it emits the spacebar sound after you typed a regular letter...
This obviously assume the screen is not simultaneously snooped, which is feasible..
The sample from that link seems to be the former:
https://github.com/shoyo/acoustic-keylogger/blob/master/data...
Doesn't really tell me anything about how realistic this attack is.
Presuming you have sufficient levels of assurance that no software exploits are present and no keystroke loggers etc. The only way to attack and the value of the attack being high enough (root CA compromise?) that an adversary would deploy (laser) mics.
I figure that anyone this concerned would program a coupe of rubber duckies or yubikey static mode ? Then the emitted strings would be unknown.
Also what about collecting from the screen via emissions?
One thing that could help to randomize that is a surface with tightly-coupled [truly] randomly-moving masses underneath, so that the resonances of the surface also change randomly.
(And if done very right or very wrong, you get a free Van De Graaf generator!)
---
Or: A laser projected keyboard where the keys are always shifting to different locations.
(Bonus: Free psychosis!)
Sensitive government information is handled in SCIFs.
https://www.nbcnews.com/politics/politics-news/what-scif-who...
He uh, is more than an 'attractive youtuber'.
:-)
Then a nitpick:
> "For example, if you type only "the" all day, then when you press the letter 't' the letter 'h' will be on the home row."
'h' is already on the home row on most keyboards.
> 'h' is already on the home row on most keyboards.
For some reason this made me think about our unit tests at work :)
Once you run a script batching nearly all your jobs, good luck trying to guess the user it's doing. A plus if you run tools like ii/jj or similar with bitlbee as your chat client. If you can bind vi/vim to it by using a file as the input, vi has the 'ab' (abbrevebiate) ex command (use nvi if you like unicode, in some systems it's called nvi2) which can autoexpand abbreviated input. Such as:
and so on.Every article of theirs tells you it's the end of the world and you have no chance against those evil dirty hackers around the corner.