Because of a quirk in the design of most cameras’ sensors,
the researchers were able to infer information about
high-frequency vibrations even from video recorded at a
standard 60 frames per second.
It's explained further on in the press release. The "quirk" is the same one that causes rolling shutter artifacts in videos: reading the camera's sensor one row at a time, instead of all at once.
Cameras basically read their sensors one row of pixels at a time. By measuring the distortion of each row, they can detect vibrations higher than the camera's frame rate.
So it's like if 960-row video at 60fps were actually a 57600 rows-per-second video, right? Which they can extract info from because having more rows in a still frame doesn't mean having more information (at least not linearly), i.e. in still frames with no rolling shutter, rows contain redundant vibration already extracted from previous rows.
So having a rolling shutter is good for this specific application because it trades off resolution (most of which is redundant or insignificant information) for sampling rate.
Between the time the first and last row are read, the object might've moved a little bit. So if you take a picture with your phone from the side window of a moving car, the picture will appear stretched.
What's at play is the sampling theorem. Using instantaneous shutter, considering the speed of sound is very high*, every part of the image is resonating essentially the same sound; the sampling theorem says you can only perfectly reconstruct up to F/2 hertz if you sample at F frames per second. The rolling shutter introduces regular variations at at much higher frequency, allowing much better sampling.
- Actually, doing a quick calculation shows that at 1khz a 1/2 wavelength is just 17cm. I wonder how precise spatial scene/source information has to be to allow this diversity to contribute significantly to the sampling. If you had a planar source and precisely spaced two objects it shouldn't be too hard to increase spectral resolution. The complementary possibilities are also be interesting -- with precisely laid out N objects and a good spectral resolution for each afforded by the shutter you could perhaps resolve the sound into N distinct sources, allowing to determine the origin of the sound; with precisely known source locations you may be able to extract some object location information.
And now think of how much high definition video, CCTV, and other forms of recordings already exist.. and then think about running large collections of pre-existing video through such algorithms, along with the best speech-to-text in the business. You could have a whole new Wikileaks on your hands :-)
Yeah, I've heard that's what the spies use, although that does require specific effort. What I find more intriguing about this development is how pre-existing footage could be used. While HD CCTV is certainly not popular, I suspect enough has been said in the presence of existing HD video to incriminate a few people :-)
I think it unlikely that pre-existing footage can be used, because HD video is almost always compressed, thus masking the minute vibrations in the pixels. The algorithm described in the article work best for uncompressed video directly from the image sensor, and they can run it in real-time without needing to store the video.
Am I the only one who isn't getting any audio from the video at all? I tried in two different browsers, downloaded the video with youtube-dl and tried to play it with mpv, everything to no avail, it's just a video with no sound.
It doesn't seem to mention what kind of camera you'd need to do this at the mentioned distance (15 feet). I'm assuming it's at least a couple thousand bucks, or maybe even so expensive that most professional photographers won't have it, but does anyone actually know? Or did I miss it in the article?
The paper says they used a Phantom V10. I'm not knowledgeable about such devices, but various Google results suggest such a camera might top $100,000 depending on the mounts, lenses, and accessories (e.g. data storage and transfer equipment).
It's reminiscent of the laser microphone, which measures sound-induced vibrations in objects by bouncing a laser off them, and reconstructs the original sound waves
This is amazing. Imagine taking high definition video of a crowd of people. You could pick out objects nearby and hear what individuals are saying. You could nearly produce a 3D auditorium by sampling different points in a video. Couple this with a 3D camera and an Oculus Rift, you could have something incredible.
They also did something like this in the recent remake of "Robocop". Some thugs are in a diner having a discussion, and it's caught on CCTV. Later, the main character goes back and analyzes the video and extracts the audio from it, presumably based on some vibrations somewhere in the scene. They don't say, but I got the impression it was from the window vibrations, similar to the technique of shooting a laser at it and watching the vibrations of the reflected light.
lol that is the first thing that I thought as well. I just passed it off as a typical Hollywood trope, but now I'm quite delighted to see it become a reality.
"Any sufficiently advanced technology is indistinguishable from magic." Those who view it as magic will express it as a trope, even if it is merely advanced technology.
Just yesterday I chuckled when recalling a James Bond movie involving Bond driving a car in reverse by viewing a back-up camera. At the time, it formed an instant "trope" because it was so cool and novel. Yesterday, I was doing exactly that with the backup camera on my car - obviously realistic and doable technology.
Not very well, at least for the initial approach with a high-speed camera. With 240 fps you're limited to 240 samples per second. If I'm not terribly mistaken that would limit you to frequencies up to 120 Hz in the reconstruction.
But the GoPro has a rolling shutter as well, so their second approach would be applicable. However, that effectively relies on rows per second and while you have a higher frame rate you have a lower resolution. In the end they could cancel each other out.
Incredible and terrifying at the same time. If they're doing this kind of stuff right now with consumer cameras, imagine how effective this technology will be in just a few decades. Privacy is fading quickly with the advent of exciting technology like this.
> Because of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second.
The audio from the 60fps video sounds pretty bad though, which I suspect is mostly because of inherent maths/physics limitations rather than anything that software can improve.
Edit: They mention capturing frequencies up to five times higher than the 60Hz frame rate, which would mean a maximum frequency of 300Hz, which would suggest the equivalent of 0.6kHz audio, which is a 73.5th of the audio rate of a CD. I doubt you'd get intelligible speech from current consumer hardware using this technique.
There is some small possibility of improvement through software techniques, such as maybe data assimilation, which can use information from surrounding time-frames to improve the measurement. This is assuming that the magnitude of vibrations changes a lot slower than the vibrations themselves, which is usually true, and how most audio compression works. It may be able to clean up the sound a little. However, I would say that the results they have obtained so far are very impressive.
The data comes in faster than 60 fps. A camera sensor doesn't capture the entire frame instantly every 1/60 second. It progressively scans through the frame over some measurable fraction of that 1/60 second. This is that quirk.
Suppose the camera scans 720 lines in HD every 1/60 second. Each row is offset in time by 1/43200 second. A rigid object could be slightly offset in space on each line of pixels, indicating that sound waves perturbed it in the time gap between when the camera captured each line. So that subframe video data can be turned back into audio at a much higher frequency than that apparent 60 Hz video sampling rate.
In other words, we're not just talking about 60 frames-per-second from a camera. It's really perhaps 43,200 rows per second, an enormously higher sampling frequency.
Let's say that it would read the entire image in 1/120 second, then it is waiting and does nothing another 1/120 second before it starts reading next frame.
The real number would be significantly smaller. Therefore they can not bump the sample rate more then five or six times. And I imagine they are using some intelligent algorithm to evenly space out the captured samples already.
Yes, yes, that was completely obvious from the article. We are getting thousands of "measurements" per second.
However, each of those measurements is incredibly inaccurate. Each one is trying to detect the change of colour of 1/200 of the colour range in a single pixel. You may be getting less than a single bit of entropy per measurement.
An advanced signal processing technique will look at the longer-term picture. Sound vibrations are not a random walk - they tend to be a combination of sine wave vibrations, where the rate of change of magnitude of each wavelength is significantly lower than the vibrations themselves. Therefore they are to a certain extent predictable, and this predictability is used by audio compression algorithms. The signal processing algorithm will have to make use of the extremely limited information coming from the measurements, and match up possible sets of varying sine waves that could be causing those measurements. This may be sufficient to reject some of the noise that we could hear on that video, and clean up the sound a bit, but it is quite a hard (and CPU-intensive) processing task.
"intelligible speech", are you sure that speech recognition really requires whole frequency range? Often it seems that data can be extracted after all, even if most of it is missing.
A CD has a lot of overkill for basic speech. Doing a test here with some speech samples, 2kHz is ugly but intelligible, 1kHz is a mess but mostly understandable with effort, and 600Hz is almost useless for trying to find words (without any practice or computer assistance, of course).
From watching the video, I get the impression that there is a very large amplitude of the input audio -- taking advantage of the "loud" in loudspeaker.
A good question. I too got that impression from the first example. In the second example (chips bag through glass), the "control" audio had a lot of reverb, which might have been introduced by the phone they acquired it through, but may also suggest that it wasn't just a person talking, but a reproduction through some kind of amplification equipment.
I am just worried that they are picking up part of information from camera mic. Maybe camera mic output is not totally independent from the video sensor, and is encoded in the final video.
I know, but I still wonder about level of soundproof/mechanical vibration and the level of mic and camera sensor isolation VS level of optical vibration. In addition sound can travel many different paths since it is mechanical vibration, and traveling through sound proof glass is not the only path. Anw cool project.
I'm sure it's such a tiny amount that it isn't really relevant. Easy enough to test. Remove the bag of chips and just film the floor. See how much signal you can derive from that. Not exactly perfect science but gives a quick indication of "signal" that's reaching the system via other means.
Thanks. However, there is still a possibility that mechanical vibration is coming to the video sensor through some path (floor+camera tripod), and affecting the video. After all mechanical vibration is affecting the potato chips. Why is it so impossible to affect video sensor? Especially in the sensitive high speed camera? Let the downvotes begin :)
If my evil :) speculation is true, the camera sensor is just picking up mechanical vibrations from environment (coming through air, tripod etc.) and encoding them into video. There is no proof that the bag of chips is vibrating (they even say that the vibrations are not visible on footage). They are extracting something from video, but that something may just be spuriuos pickups by equipment, and not related image/video of bag of chips. Thus it would be just a silly way to measure spurious mechanical vibrations.
Well, the chip bag weighs a few grams at most. The camera and gear weigh something like 50kg. The higher the video frame rate, the greater the damping you need to prevent unwanted vibration. Notice the tests were performed with a speaker playing back inside the room rather than a person, in order to minimize mechanical vibrations. The sensors are very firmly mounted inside the camera, trust me.
Reminds me a little of "Dual Photography" as presented at Siggraph in 2005. All the data you need to construct a new view is available if you know where and how to look: https://www.youtube.com/watch?v=p5_tpq5ejFQ
Wouldn't it be really neat to apply this to HD movie sequences, and hear what the sounds on the set and the voices of actors were like pre-production? And how unreal some of the sounds must have turned out with all the visual tweaking that happens in production?
I only down vote when something is demonstrably wrong or offensive. I thought that's how everyone is doing it, so I wondered what could I have written that was wrong or offensive?
For this particular film the rather characteristic look of digital video is appropriate, whereas most productions would rather shoot with much fancier gear. But you'll find plenty DSLRs working on film sets these days, and you will see more and more convergence as high-quality sensors become a commodity. Black Magic have a pocket camera that delivers 13 stops of dynamic range.
It definitely has to be 2x the desired maximum frequency if you don't want aliasing. They probably capture higher frequencies by making assumptions about the input signal, but this will inevitably lead to aliasing when these assumptions are violated. The article doesn't really go into depth about precisely how they go about this.
They have lots of pixels, though, which means you should be able to go higher than nyquist with some clever work.
With two time samples you shouldn't be able to learn anything about the state of waves in a pool, but if each sample is a photograph with lots of pixels you can actually tell a lot.
You missed the part where 60 frames per second only allowed to identify the speaker and the number of people in the room.
> Because of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second. While this audio reconstruction wasn’t as faithful as it was with the high-speed camera, it may still be good enough to identify the gender of a speaker in a room; the number of speakers; and even, given accurate enough information about the acoustic properties of speakers’ voices, their identities.
Some cheap video cameras don't shutter the frames (grabbing all pixels at once), they progress thru pixels - leading to a usually-annoying stretchy visual effect. If the sound affects much of the frame with suitable uniformity, you've turned a 60fps camera into a 2MHz "visual" audio sampler (lots of noise, artifacts, & other difficulties aside).
I record the dialog for your movies. Aprt from obvious sound effects like Darth Vader voices or so, actors' dialog is changed as little as possible during post production. It's not heavily EQed or comrpessed, because that would mean a corresponding change to the room tone and background noise, which would then fluctuate unnaturally as you went back and forth between the participants in a scene. In scenes with a lot of movement actors' voices sometimes sound a little deeper than in real life, because they are fitted with a tiny wireless microphone, and there's usually some resonance from the chest cavity.
But generally what you hear is very close to how the person actually sounds - although their accent or inflections may be adopted for the purposes of their role. This can be a bit jarring; I've worked with method actors who maintain their screen accent at all times during production until the film is done, so when they switch back to their regular accent after a month or so it's extremely disorienting, since I've been listening to them in my headphones day in day out for weeks, and am paid to pay as much attention to their voices as the cinematographer pays to their faces.
Some actors go even farther in support of their public image. Rock Hudson had a somewhat high voice that producers deemed incompatible with his looks, so during production he would warm up every day by shouting for 20 minutes and gargling with orange juice to inflame his vocal cords, and of course he smoked a lot too. What actors will do to themselves in pursuit of screen presence far exceeds anything I've ever been asked to do in post. Editing dialog is more than enough work without trying to sculpt people's voices.
Thanks a lot for that insight. I will assume you work primarily for Hollywood with that description. It is interesting to hear about the behind-the-scenes from film industries in different parts of the world.
In the Indian film industry (maybe not so much Bollywood, but more of the local ones like Tamil, Malayalam, etc) there's a LOT more post being done. Some entire Malayalam films have dubbed voices (for the original movie). Not to mention nearly every song track does not have the singing recorded during the shooting of the dance sequences. So those sets and actor voices I would bet would be completely different.
Yeah, here in California we have the luxury to do good production sound almost all the time. Dubbing the voices later all sounds very different, because the recordings are not in the same space as the original production and so on. I forgot to think about that aspect, because here we always record production sound unless there is an impossibly high level of noise (eg from a wind machine or something). With modern cameras it's been very good for the sound department because they are so quiet, whereas older 16 and 35mm film cameras were quite noisy, like a sewing machine, and it required some work to prevent this messing up the dialog recording.
While it's cool to see this technique applied to consumer video, the general idea has been used in international espionage for a while now. In fact, Léon Theremin (yes, of the musical instrument) invented an early device capable of eavesdropping based on window vibrations: http://en.wikipedia.org/wiki/L%C3%A9on_Theremin#Espionage . I also recall that this is why all of the windows in the White House are fitted with tiny devices that vibrate the windows randomly as a countermeasure.
As a kid (in the early 80's!) I accidentally figured this out while "hacking" a sound based "walkie talkie" with a photo cell and a LED pointed at each other through a little telescope, from my house to my neighbors across the bock.
The device worked, but had a bug. a hummmm on the receiver.
For the longest time I could not figure out what was the "hummm", till I noticed it was there even with the transmitter off.. and then, I heard strange voices even with the transmitter being off!
Then I realized the humm was the reflection of incandescent light on the window, and the voices where the people in the room making the window vibrate too..
Ahh.. those where great fun days early on "hacker" life as a hacker. :)
>>till I noticed it was there even with the transmitter off.. and then, I heard strange voices even with the transmitter being off!
...and that is when little Johnny knew he had to lay off the shrooms.
:P
I love hearing stories like yours though. I, too, miss the child-like wonder of making discoveries. We take the physical world we live in for granted, but there's just so much out there to see and learn from.
131 comments
[ 3.1 ms ] story [ 202 ms ] threadSo having a rolling shutter is good for this specific application because it trades off resolution (most of which is redundant or insignificant information) for sampling rate.
- Actually, doing a quick calculation shows that at 1khz a 1/2 wavelength is just 17cm. I wonder how precise spatial scene/source information has to be to allow this diversity to contribute significantly to the sampling. If you had a planar source and precisely spaced two objects it shouldn't be too hard to increase spectral resolution. The complementary possibilities are also be interesting -- with precisely laid out N objects and a good spectral resolution for each afforded by the shutter you could perhaps resolve the sound into N distinct sources, allowing to determine the origin of the sound; with precisely known source locations you may be able to extract some object location information.
If you want to record a particular person through a window, for instance, you can get a laser microphone that catches the vibrations of the glass.
$ youtube-dl -f 18 http://newsoffice.mit.edu/2014/algorithm-recovers-speech-fro...
You can use the -F option to list the available formats.
There are also some consumer cameras that can capture almost 1,000 fps from a small sensor window. I wonder if those would work.
http://en.wikipedia.org/wiki/Laser_microphone
http://en.wikipedia.org/wiki/Cocktail_party_effect
http://en.wikipedia.org/wiki/Independent_component_analysis
Edit: Spacing.
I often wonder what makes people to pass off things like that as "typical tropes", where they are obviously realistic and doable.
Just yesterday I chuckled when recalling a James Bond movie involving Bond driving a car in reverse by viewing a back-up camera. At the time, it formed an instant "trope" because it was so cool and novel. Yesterday, I was doing exactly that with the backup camera on my car - obviously realistic and doable technology.
But the GoPro has a rolling shutter as well, so their second approach would be applicable. However, that effectively relies on rows per second and while you have a higher frame rate you have a lower resolution. In the end they could cancel each other out.
Edit: They mention capturing frequencies up to five times higher than the 60Hz frame rate, which would mean a maximum frequency of 300Hz, which would suggest the equivalent of 0.6kHz audio, which is a 73.5th of the audio rate of a CD. I doubt you'd get intelligible speech from current consumer hardware using this technique.
Suppose the camera scans 720 lines in HD every 1/60 second. Each row is offset in time by 1/43200 second. A rigid object could be slightly offset in space on each line of pixels, indicating that sound waves perturbed it in the time gap between when the camera captured each line. So that subframe video data can be turned back into audio at a much higher frequency than that apparent 60 Hz video sampling rate.
In other words, we're not just talking about 60 frames-per-second from a camera. It's really perhaps 43,200 rows per second, an enormously higher sampling frequency.
Let's say that it would read the entire image in 1/120 second, then it is waiting and does nothing another 1/120 second before it starts reading next frame.
The real number would be significantly smaller. Therefore they can not bump the sample rate more then five or six times. And I imagine they are using some intelligent algorithm to evenly space out the captured samples already.
Yes, yes, that was completely obvious from the article. We are getting thousands of "measurements" per second.
However, each of those measurements is incredibly inaccurate. Each one is trying to detect the change of colour of 1/200 of the colour range in a single pixel. You may be getting less than a single bit of entropy per measurement.
An advanced signal processing technique will look at the longer-term picture. Sound vibrations are not a random walk - they tend to be a combination of sine wave vibrations, where the rate of change of magnitude of each wavelength is significantly lower than the vibrations themselves. Therefore they are to a certain extent predictable, and this predictability is used by audio compression algorithms. The signal processing algorithm will have to make use of the extremely limited information coming from the measurements, and match up possible sets of varying sine waves that could be causing those measurements. This may be sufficient to reject some of the noise that we could hear on that video, and clean up the sound a bit, but it is quite a hard (and CPU-intensive) processing task.
BTW: not everyone uses exactly your own personal guidelines.
BTW: HN Guidelines include "Resist complaining about being downmodded. It never does any good, and it makes boring reading."
For this particular film the rather characteristic look of digital video is appropriate, whereas most productions would rather shoot with much fancier gear. But you'll find plenty DSLRs working on film sets these days, and you will see more and more convergence as high-quality sensors become a commodity. Black Magic have a pocket camera that delivers 13 stops of dynamic range.
With two time samples you shouldn't be able to learn anything about the state of waves in a pool, but if each sample is a photograph with lots of pixels you can actually tell a lot.
> Because of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second. While this audio reconstruction wasn’t as faithful as it was with the high-speed camera, it may still be good enough to identify the gender of a speaker in a room; the number of speakers; and even, given accurate enough information about the acoustic properties of speakers’ voices, their identities.
http://youtu.be/TKF6nFzpHBU?t=10s
But generally what you hear is very close to how the person actually sounds - although their accent or inflections may be adopted for the purposes of their role. This can be a bit jarring; I've worked with method actors who maintain their screen accent at all times during production until the film is done, so when they switch back to their regular accent after a month or so it's extremely disorienting, since I've been listening to them in my headphones day in day out for weeks, and am paid to pay as much attention to their voices as the cinematographer pays to their faces.
Some actors go even farther in support of their public image. Rock Hudson had a somewhat high voice that producers deemed incompatible with his looks, so during production he would warm up every day by shouting for 20 minutes and gargling with orange juice to inflame his vocal cords, and of course he smoked a lot too. What actors will do to themselves in pursuit of screen presence far exceeds anything I've ever been asked to do in post. Editing dialog is more than enough work without trying to sculpt people's voices.
In the Indian film industry (maybe not so much Bollywood, but more of the local ones like Tamil, Malayalam, etc) there's a LOT more post being done. Some entire Malayalam films have dubbed voices (for the original movie). Not to mention nearly every song track does not have the singing recorded during the shooting of the dance sequences. So those sets and actor voices I would bet would be completely different.
The device worked, but had a bug. a hummmm on the receiver.
For the longest time I could not figure out what was the "hummm", till I noticed it was there even with the transmitter off.. and then, I heard strange voices even with the transmitter being off!
Then I realized the humm was the reflection of incandescent light on the window, and the voices where the people in the room making the window vibrate too..
Ahh.. those where great fun days early on "hacker" life as a hacker. :)
...and that is when little Johnny knew he had to lay off the shrooms.
:P
I love hearing stories like yours though. I, too, miss the child-like wonder of making discoveries. We take the physical world we live in for granted, but there's just so much out there to see and learn from.
Or maybe it's time to think how to adapt to a world without privacy?