This is insanely cool. I've often vaguely fantasized about stuff like this but this guy takes it all the way. He deserves a huge budget. Can we fund this somehow?
I think the value of what he does is that it's driven by curiosity. A pile of money donated by people might put him under pressure to deliver things to his audience which might ruin that.
A fund that hackers can apply to for building cool projects is an interesting idea though (to the extend that I'm sure one already exists).
It's a simultaneously awesome and saddening feeling. Awesome because you know that there are unexpected amazing things that are going to happen. Sad because a bunch of them might give up from lack of exposure, feedback, etc...
Yeah. From the hacker's point of view many such projects seem trivial, not worth bothering to talk about.
You really need a spotter to find out which of your projects are cool and which aren't.
Then again, after trying to post a dozen times and getting nothing, one would just stop trying most of the time, and would work for the attic, so to speak.
The timing issue between the microphones doesn't surprise me. Multi channel soundcards require a single ADC that processes all of the channels in unison (same for the DAC).
This is also why you can't just plug four stereo cards into a PC and expect it to make an 8-channel system.
You can't? I'm sure I tried multi-channel via separate sound cards and it sounded fine to me. Maybe I was lucky with the timing, or the audio subsystem somehow accounted for it?
Depending upon what you're using it for, you might well get away with it, but for some purposes the accumulated sample rate clock drift eventually gets too bad.
You could make a long multitrack recording this way, and so long as you played it back from the start on the same hardware (i.e. so the soundcards would drift relative to one another more-or-less consistently with how they drifted during the recording), the playback might be ok. But if you begin playback half way through the recording, then the timing between channels might be out, as your software has no idea how much drift each channel ought to have accumulated by that point in the recording.
Mind, it's been at least 15 years since I actually put any of this stuff to the test. Tolerances on modern soundcards might be really tight.
Strictly, multiple ADCs are fine, but what matters is that they are all locked to the same sample rate clock, connected to each card either via a digital audio input (e.g. SPDIF) or a dedicated clock input (e.g. word clock).
Yay, a sound-field camera! I've been wondering if such things are made. Should I take the existence of this project as an indication that they aren't?
The use case I imagine is finding what part of a machine is emitting a noise, much like how you would use a thermal camera for finding what part of it is running hot.
Not exactly unheard of, but quite rare. I found a company that made large arrays like that, but they seem to operate in near field, measuring only the sound volume.
Imagine a wall of microphones next to a truck - you can tell which part of the wheel makes the sound. That sort of stuff.
As for an actual "camera" that turns sound into an image - i've never heard of one.
Have you seen light field photography [1]? It would seem that your array is quite similar in concept. I'd imagine that getting images similar to those produced by various SONAR imaging technologies is a similar sort of deconvolution problem. If you're looking for textbooks or scientific papers to review, I think the field of tomography is also ripe with methods to explore here.
I'd challenge the notion however that this isn't 'just' a passive SONAR device. Keeping the camera analogy, active SONAR use spread spectrum pings, much like the white light of a camera's flash. Your camera lacks a flash function, and its lens is maybe not yet as in focus as it could be.
If you did want to add a flash, you're likely a $0.02 piezo device and maybe $20 in other various parts (and heaps of numerical processing on a PC) away from having the ability to produce a nice 3D reconstruction of the physical scene.
If you get direct data capture going, please do post some raw data online. I'm sure others would have heaps of fun playing around with it.
No. SONAR sends out an ultrasonic ping and times the response of its echo to get a distance.
This project is similar in that it involves visualization from audio, but apart from that, it's quite difference. This is the ability to see a sound. To derive the positions of arbitrary sounds in a viewing plane, and render it like you would a picture or video.
> Most sonar is active (sends out pings) and has only one input (one microphone)
While I believe the US Navy doesn't prefer the term "Passive Sonar", submarines and some surface ships are equipped with hydrophone arrays (either on the hull, or towed arrays, or both) as well as active sonar; this isn't an uncommon thing. (I was under the impression that the active sonar didn't use a single hydrophone, but instead used the same hydrophone array used for passive detection.)
Yes it is. Despite all the other replies asserting SONAR is active, passive SONAR systems are very much a thing.
For my undergraduate project years ago I built a microphone array that could be used to locate noisy objects. I demonstrated it by tracking aircraft taking off at a local airport.
Whilst I didn't make 'acoustic photos' with it, I did plot the flight track of the aircraft in real time, and could have a decent crack at identifying it based on frequency content.
Congrats, beautiful history and nice hacking.
I hope you get a really big bag of money one of these days so you can keep doing this kind of things!!! kudos!
He gives a pretty decent explanation of what he is seeing in the Catalyst transcript. It's fascinating, as from his description it's a low resolution (due to the longer wavelength) of what you or I would see. Presumably he also sees things such as colours, but based on audio reflectivity rather than optical reflectivity?
Your point about only him appreciating his own experience is interesting, as the same question can be asked of all of us [1]. In audio, maybe apples and oranges are the same colour and strawberries are different? It would be an interesting application of a sonic camera, to render the image so that equivalent audio colours are consistently mapped to the same light colours.
I commented it elsewhere in this thread, but I a company called Sorama has been doing sound camera stuff for quite a while. Afaicr it's based on the phd dissertation of their CEO Rick Scholte so that name might yield some articles you might like.
There is a Swiss startup called Distran (http://www.distran.ch/) which develops ultrasound cameras. They also have a spherical version which looks really cool and allows 360deg sight, unfortunately I cannot find anything about that one on the website anymore.
We all have a bit of a different way of approaching the problem and our target market differs somewhat. You can use this technology for a lot of different things, not just the obvious use cases. We're still looking at a lot of applications, so the clearer product differentiation is something that will form as time goes by.
Not quite the same thing though. These systems are active. They have sound being produces most likely at a very specific frequency and then listened for so that depth information can be gathered. This instead images the sound that's already out there in the light. It's the difference between lidar and a camera
Nope, theirs is the sound-camera, detector array and sound-lens, but with an acoustic floodlight. I think they do use ranging though, so the 2D image is "sound camera" in one raster dimension, and "sonar range" in the other.
No, not common, but as the OP noted the evolution of MEMs based microphones might make them a lot more common. As the author notes, they are more like holographic cameras than what many people think of as "regular" cameras. By converting the imaging into the FFT domain the resolution gets better "everywhere" the more sensors you add. (If you have ever cut up a hologram into smaller and smaller pieces you can see this in action, as the full image is still there, just lower resolution). It would also be interesting to illuminate a non-active source with sound beam. By using two, slightly out of phase, ultrasonic sources he might be able to reveal depth detail of the surface being imaged.
It's used my military & events for drone tracking by sound, and concert venues for seeing where sound leaks out to the surrounding neighborhood. (Among other uses)
Right on. Reminds me of Richard Feynman or Steve Wozniak, the way they just casually talk about taking things apart and putting things together while following a curiosity on an adventure. This is representative of the sort of ethos that drew everyone here in the first place, I think.
You seriously need to consider starting a Patreon[0] campaign. I'm sure many people would be interested to see more from you. There are individuals on there who are making significant salaries based purely on interest from the crowd.
Don't that require making regular content? My posts are fairly random by time and quality, and most are just rants on various philosophical nonsense [0] or game design [1].
Wouldn't be fair to people who signed up for something like this camera.
Perhaps not as a regular income; but if funding for these projects is a limitation I bet crowdfunding would bring in funds to keep these posts coming :)
Could there be a version of Patreon, where not the author themself but instead the fans start the pledges. The author could choose to take the money or not, and could prove their identity by putting their bitcoin address in a <meta> tag on their website.
So we could create a project on the site say "support sound camera posts from ribbonfarm.com" and money would be gathered for it, then when a new qualifying post would be made the site owner would be contacted "hey, you have $500 waiting for you, put this meta tag in your <head> to accept it".
This way modest authors wouldn't need to think "am I good enough for Patreon?" and people could support them. People could provide wallet addresses to which the money would be returned if not accepted.
This would almost certainly not work due to how easily most sites can be hacked by experienced and motivated hackers. That website would essentially just be a bounty list for hackers. The better option would be requiring an extra DNS entry meaning the hackers would be trying to hack into a registrar which is probably a fair bit harder on average.
The cynic in me wants to say that bosses tell their talented workers that pursuing personal projects is delusional simply because it keeps the labor cheap!
"Daredevil" as in vision through sound. Very neat project! I wonder how much people can see in the dark through listening to sound sources by moving their heads for motion parallax.
This would be pretty powerful if you combine it with speech recognition. If you put this array in the middle of a crowded room, can you better tease apart different audio sources?
Ironically, this thing can't record sound as we know it - it only have enough buffer space for a few hundreds of samples per mic, 30 times per second. They are to be FFT-ed rather than continuously recorded.
While such a contraption should, in theory, be able to distinguish between dozens of voices in a room, it would take much, much heavier signal processing to achieve.
If you have more than one mic you can begin to triangulate sources (you might be able to use a single mic if you know the geometry of the room and/or could send out noises), so it would definitely be possible.
Consider that most of us are carrying around multiple, networked mics. Some may already do it, but I doubt it will be too long before everyone can virtually tour real-life spaces and visualize the sounds there in real-time.
Not this specific hardware, but in general, yes. Amazon Echo does pretty much this, except it focuses on one source instead of all possible sources (from what I know). PulseAudio also recently gained a feature to use beamforming (see a HN submission from either today or yesterday), but for now it needs the location of the object you're trying to record as an input. I guess you could kind of brute-force that and let it run many times over the same audio stream to find all possible audio sources.
You can use the phase/timing information between different mics to separate sources, yes.
Interestingly it's also doable with a single mic - previously with algorithms like NMF and ICA, more recently with neural networks. Our brains do it very well so there is probably more to discover.
Google blind source separation or cocktail party problem if you want to find out more.
You can use the phase/timing information between different mics to separate sources, yes.
Interestingly it's also doable with a single mic - previously with algorithms like NMF and ICA, more recently with neural networks. Our brains do it very well so there is probably more to discover.
I think the whole thing could be done pretty much all in software. Setup a bunch of phone numbers in Twilio, using a bunch of your friends iPhones, strategically located, call the numbers and process the data using Twilio API. Whatsapp or other real-time voice communication apps with an API may work too.
You'll never be able to collect phase shift info from such a setup. Not only will each phone's DSP algos delay the data, but the jitter in transmission and reception will just kill any chance of it working properly.
Hey Artlav, when did you start posting on ribbonfarm? I used to subscribe to it, but unsubscribed a couple years back after Rao wrote a dumb post on information security.
About a year ago. He invited me in after seeing the "How do you make batteries that work on Venus" article ( http://orbides.org/page.php?id=1005 ) i wrote on my own semi-dead site.
>Unfortunately, we are still talking about a total of 64 Mbps of data. That needs a USB 2.0 sampling board that would pull the data out of the cells over the bulk interface
not 16, but 8 '40-ish MHz' should be no problem as long as you sync them. FX2LP gives you raw 8/16bit wide datapump in GPIF Mode mode, or a nice FPGA friendly fifo in Slave FIFO mode, you can sort your 'spi like' data streams out on the pc side later/in real time. and since modern PCs have at least 2 usb hosts you could use two FX2LPs for all 16 streams.
Modern i7 cpu has surprising amount of flops and could handle this no problem (not to mention modern gpus). Consider modifying one of your boards, instead, or in addition to doing fft on fpga try pumping raw data over usb to a PC, 64 3MHz PDM streams should be 25MB/s? comfortably within usb 2.0 limits. FX2LP breaks no sweat while pumping ~40MB/s
hence 'modifying one of your boards', as an experiment. if it works to your liking then you could move to FX3, that one can pump 400MB/s of raw bytes over usb 3.0. and that is for processing raw stream all on PC side.
If all you want is the current setup then above is valid with 2x $6 modules.
Why is the resolution so much fixed to the number of mics?
Shouldn't 3 mics be somewhat sufficient for intermediate values as well? More mics => more redundancy => clearer image of the waves, but why do the pixels/blobs "snap" to a microphone?
edit: ah, he does not use timing (at 10fps obviously not) but needs continuous sound to see the wave pattern across the array...
edit 2: so do these gunfire locators¹ only time stamp impulses? Or do they do continuous analysis? With natural, irregular sound, it should be possible to match and measure the delay between mics, no?
edit 3: the gunfire locators would need to analyze for gunfire characteristics anyways, to weed out false positives, but that could be separated from the locating.
Almost, kinda related is this Invisbilia episode that's pretty interesting about a blind man that uses directed clicking in his mouth as a way of "seeing via echolocation". They talk about how it activates similar parts of the brain that vision does which I think is a cool parallel to what's been created here.
This is pretty cool, and reminded me of the far-field microphone array in an Amazon Echo (which uses 7 spaced out microphones). Which also makes me wonder, taking something like this (which has way more than just 7 mics) and applying it to the problem of far field speech recognition, are there large gains to be had? Just thinking of what new applications become possible, when you have the ability to print hundreds or thousands of miniature mics on a single board that will fit in a small-ish device.
I have an idea for you. You mention that you want to use a webcam so that you can "overlay" this information on top of an image.
What if you could send it directly to your brain?
There is a technology called Brainport[0], which is effectively a 16x16 grid of electrodes small enough to place on your tongue. By feeding electrical impulses through these electrodes, you stimulate the nerves on your tongue. After two minutes of a "champagn bubbling" sensation, your brain rewires the senses from your tongue to your visual cortex and you can "sense" any input.
This technology has been used to give sight to the blind, balance to those with inner ear problems, and sonar to NAVY seals.
It would be amazing if you were the first person in the world to truly "see" sound.
I'm pretty sure that would be mechanically impossible - the brain is very plastic, but touch-into-sight seems a little far for adult learning. Instead, you could expect this to feel more like acquiring a new sense - the experience reported by those who have used pager vibrators to give their body a compass[1].
one of those projects is feelSpace[1] which is a belt with vibrators giving you either a new compass sense, or tells you where to go (as a navigation system).
Whilst a commercial offering is probably quite expensive, it strikes me as the sort of thing you could do quite cheaply and easily with an arduino.
I've actually been collecting the vibration motors from disposable electric toothbrushes to build my own belt; as they were designed to run form a single AA battery powering them should be easy.
I find it quite expensive too. I guess the price is fair if you include the research they did, plus the prototyping, manufacturing and company overhead.
I've tried a prototype version of this belt once (for a couple of hours), it worked surprisingly well. But integrating the vibrations as a "new sense" into your brain (so that your subconscious can use it) takes at least some days.
In 1996 I was prototyping a belt with piezo transducers spaced around it with each doubling as emitters and receivers for a crude ultrasonic distance, and then each would vibrate at a higher frequency the closer the object came.
Nicknamed 'batbelt' in-house.
I dropped it after looking into insurance requirements and ADA compliance.
Yes, and I used a Basic Stamp, and replaced it with a PIC chip later on. I had several normal issues: ignoring very close objects like your arms swinging when you walk; short postives; timing issues with getting the whole array working, since I was trying to triple use piezo transducers as emitter/receiver/vibrator. Other than those issues, it was not too refined due to the limitations of the transducers, speed of the 8-bit chips I was using, and you had to wear it within proximity of your body, but then not cover it up.
Before I ditched the idea, I was going to put the vibrator belt as a second belt worn under the clothing, but there was no Bluetooth or WiFi or easily accessible way of going wireless between the belts in a practical fashion.
I had my neighbor, blind man who had a guide dog, trial it. He was a great guy, and helped me quite a bit to 'see' past my vision biases. Unfortunately, he passed away. He used to work for the IRS, and we would joke about me getting my deductions past him! He did like what I had accomplished, but many blind people do not have the money for something that is not subsidized, and then you get caught up in the compliance insurance issues. Not sure if that has changed much nowadays.
It was a great maker project before the 'maker' movement. Too bad there was no startup, or crowd-funding web sites back then!
In 1996 I was prototyping a belt with piezo transducers spaced around it with each doubling as emitters and receivers for a crude ultrasonic distance, and then each would vibrate at a higher frequency the closer the object came.
Nicknamed 'batbelt' in-house.
I dropped it after looking into insurance requirements and ADA compliance.
It's no surprise to me that one would adapt over time. The article you link didn't mention any long-term study. I'd be curious to know how long it would take to replace one's vision to that of vertically or horizontally mirrored vision, in full. It's one thing to walk or drive a motorcycle, it would be another thing to handle situations requiring sub-second reflexes.
Once you've managed to fully adapt to the new mapping, is it just as jarring to revert and go back to normal, or have you learned how to handle both?
Say you rotated for an entire year - or longer - on a weekly schedule between normal vision, vertically mirrored vision, and horizontally mirrored vision. Do you gain full training in all three to the point that you could transition between normal/vertical/horizontal at any time without any disorientation or readjustment period? Or can the brain really only handle one mapping at a time? This would be the ultimate question to answer.
I suspect the results would be similar to how we handle multiple languages. Your primary vision (ie: the one you learned first, or the one you have used exclusively for a decade or more) would remain the easiest to work with, while acquired alternatives may approach the "fluency" of the primary, while not being perfect.
Later on in the video, he mentions that having gotten used to the backwards bike, a regular bike was nearly impossible to ride until he adjusted to it again.
A human brain is very plastic. There is an app that can turn a picture from the phone's camera into sound real-time, and after a few minutes of training you can start to "see" the picture from listening to that sound.
The thing basically scanned a line across the image, and produced sound based on the brightness. So, with your eyes closed you could quite easily imagine a basic shape of the image from the sound.
It's a process called Sensory Substitution[0]. It's technically the perception of that sense.
I understand that the rest of your brain 'trains' this pathway to occur. So because you know that there is a sound source and you can see it move in space, and you 'feel' this on your tongue as well, you are capable of correlating the two senses together and associating them with each other.
And it's about time someone did it! Yours is the first hobbyist project I've encountered since my own extremely crude ten-dollar version in 1981. No square array or processing, mine was a mechanical scanner-disk: eight LEDs glued in a spiral to an old vinyl record album, each with an electret microphone and and op amp. Spin the disk with a motor at about 6Hz. Wind noise, so add foam to the mics. It worked. A mechanical acoustic-phosphor raster sensor and display screen, where sound patterns caused light patterns. The best part was to hold some headphones near it while playing 10KHz sine wave. This produced interference stripes on the scanning disk! I hoped to build a huge version someday, but somebody beat me to it: Tom Zimmerman, an MIT student who patented the dataglove, then used the money to work at Exploratorium Museum for a year (as Artist in Residence), building a scanning-microphone LED disk about 6ft wide.
Since then we have companies selling DIDSON full-blown video for underwater (with sonic illumination,) also a euro company with a 3D phase-array microphone imager positioned on a ring or a wire sphere. See:
157 comments
[ 2.9 ms ] story [ 141 ms ] threadThe address is on my Orbital Designs site - http://orbides.org/
A fund that hackers can apply to for building cool projects is an interesting idea though (to the extend that I'm sure one already exists).
https://www.flickr.com/photos/fotoopa_hs/sets/72157611107153...
https://www.flickr.com/photos/fotoopa_hs/4082464167/in/album...
You really need a spotter to find out which of your projects are cool and which aren't.
Then again, after trying to post a dozen times and getting nothing, one would just stop trying most of the time, and would work for the attic, so to speak.
This is also why you can't just plug four stereo cards into a PC and expect it to make an 8-channel system.
Huh, I always sorta wondered about this. Thanks for clearing it up!
You could make a long multitrack recording this way, and so long as you played it back from the start on the same hardware (i.e. so the soundcards would drift relative to one another more-or-less consistently with how they drifted during the recording), the playback might be ok. But if you begin playback half way through the recording, then the timing between channels might be out, as your software has no idea how much drift each channel ought to have accumulated by that point in the recording.
Mind, it's been at least 15 years since I actually put any of this stuff to the test. Tolerances on modern soundcards might be really tight.
The use case I imagine is finding what part of a machine is emitting a noise, much like how you would use a thermal camera for finding what part of it is running hot.
Imagine a wall of microphones next to a truck - you can tell which part of the wheel makes the sound. That sort of stuff.
As for an actual "camera" that turns sound into an image - i've never heard of one.
SOund Navigation And Ranging: https://en.m.wikipedia.org/wiki/Sonar
Think laser rangefinder on a scanner platform vs a camera.
I'd challenge the notion however that this isn't 'just' a passive SONAR device. Keeping the camera analogy, active SONAR use spread spectrum pings, much like the white light of a camera's flash. Your camera lacks a flash function, and its lens is maybe not yet as in focus as it could be.
If you did want to add a flash, you're likely a $0.02 piezo device and maybe $20 in other various parts (and heaps of numerical processing on a PC) away from having the ability to produce a nice 3D reconstruction of the physical scene.
If you get direct data capture going, please do post some raw data online. I'm sure others would have heaps of fun playing around with it.
1: http://graphics.stanford.edu/projects/lightfield/
One file per cell, 16bit integer samples. Stored row by row, 128 slices per frame.
Not sure where to post it, however - here it would be buried.
This project is similar in that it involves visualization from audio, but apart from that, it's quite difference. This is the ability to see a sound. To derive the positions of arbitrary sounds in a viewing plane, and render it like you would a picture or video.
https://en.wikipedia.org/wiki/Sonar#Passive_sonar
Most sonar is active (sends out pings) and has only one input (one microphone)
It builds pictures by directing the ping and measuring time until the response.
This is fundamentally different than an array of microphones passively listening to, and creating a fine grained picture of some sound source.
While I believe the US Navy doesn't prefer the term "Passive Sonar", submarines and some surface ships are equipped with hydrophone arrays (either on the hull, or towed arrays, or both) as well as active sonar; this isn't an uncommon thing. (I was under the impression that the active sonar didn't use a single hydrophone, but instead used the same hydrophone array used for passive detection.)
[0] https://en.wikipedia.org/wiki/Towed_array_sonar
http://blueprintsubsea.com/starfish/images/sonar/452/sf452_s...
That's just one example - look for side scan sonars for the earliest stuff (https://en.wikipedia.org/wiki/Side-scan_sonar) or sonar imaging for the latest stuff (http://www.tritech.co.uk/product/gemini-720i-300m-multibeam-...)
https://en.wikipedia.org/wiki/Fishfinder
For my undergraduate project years ago I built a microphone array that could be used to locate noisy objects. I demonstrated it by tracking aircraft taking off at a local airport.
Whilst I didn't make 'acoustic photos' with it, I did plot the flight track of the aircraft in real time, and could have a decent crack at identifying it based on frequency content.
I have. His name is Daniel Kish [1, 2].
Apparently his visual cortex has learned to process sound rather then light, so he literally sees, based on the sounds he is hearing.
[1] http://www.abc.net.au/catalyst/stories/4441831.htm
[2] https://en.wikipedia.org/wiki/Daniel_Kish
Your point about only him appreciating his own experience is interesting, as the same question can be asked of all of us [1]. In audio, maybe apples and oranges are the same colour and strawberries are different? It would be an interesting application of a sonic camera, to render the image so that equivalent audio colours are consistently mapped to the same light colours.
[1] https://www.quora.com/Visual-Perception-Is-everyones-experie...
Cool stuff!
I'm unaffiliated, but saw one of their demos.
- Kai @ nlacoustics.com
The use case here is being able for a diver or ROV to see in murky water
[0]http://www.soundmetrics.com/image-gallery [1]http://www.apl.washington.edu/programs/z_DIDSON/Pages/LIMIS....
It's used my military & events for drone tracking by sound, and concert venues for seeing where sound leaks out to the surrounding neighborhood. (Among other uses)
Edit: Their example screenshot is even showing an engine block overlaid with acoustic output of the components: http://static1.1.sqspcdn.com/static/f/1431009/18268254/13373...
(Unless i missed some sort of karma/like/wow monetization :) )
[0] https://www.patreon.com/
Wouldn't be fair to people who signed up for something like this camera.
[0] http://orbides.org/random.php
[1] http://orbides.org/randg.php
Could there be a version of Patreon, where not the author themself but instead the fans start the pledges. The author could choose to take the money or not, and could prove their identity by putting their bitcoin address in a <meta> tag on their website.
So we could create a project on the site say "support sound camera posts from ribbonfarm.com" and money would be gathered for it, then when a new qualifying post would be made the site owner would be contacted "hey, you have $500 waiting for you, put this meta tag in your <head> to accept it".
This way modest authors wouldn't need to think "am I good enough for Patreon?" and people could support them. People could provide wallet addresses to which the money would be returned if not accepted.
https://en.wikipedia.org/wiki/Reflection_seismology
Listen to this with headphones: https://www.youtube.com/watch?v=IUDTlvagjJA
It’s amazing how much you can geolocate just in even a recording.
While such a contraption should, in theory, be able to distinguish between dozens of voices in a room, it would take much, much heavier signal processing to achieve.
Consider that most of us are carrying around multiple, networked mics. Some may already do it, but I doubt it will be too long before everyone can virtually tour real-life spaces and visualize the sounds there in real-time.
Interestingly it's also doable with a single mic - previously with algorithms like NMF and ICA, more recently with neural networks. Our brains do it very well so there is probably more to discover.
Google blind source separation or cocktail party problem if you want to find out more.
Interestingly it's also doable with a single mic - previously with algorithms like NMF and ICA, more recently with neural networks. Our brains do it very well so there is probably more to discover.
Nitpick; Duga 3 was never built. The one near Chernobyl is Duga 1.
But I didn't mean to detract, that's really a very cool project that you did!
you mean a $10 EZ-USB FX2LP dongle?
You might be interested in this https://sourceforge.net/p/manyears/wiki/Main_Page/
Im sure you know about SeeSV-S205, its basically the same thing, but polished and packaged neatly for FAT commercial customers ($30K).
posted on slashdot in 2013 about the same thing https://tech.slashdot.org/comments.pl?sid=3742015&cid=437167...
Modern i7 cpu has surprising amount of flops and could handle this no problem (not to mention modern gpus). Consider modifying one of your boards, instead, or in addition to doing fft on fpga try pumping raw data over usb to a PC, 64 3MHz PDM streams should be 25MB/s? comfortably within usb 2.0 limits. FX2LP breaks no sweat while pumping ~40MB/s
top ebay link $6 http://www.ebay.com/itm/CY7C68013A-56-EZ-USB-FX2LP-USB2-0-De... application: http://www.cypress.com/documentation/application-notes/an613...
That's "64 3MHz PDM" per cell, of which there should be 16. And, they have to be synced to a sub-microsecond precision.
If all you want is the current setup then above is valid with 2x $6 modules.
Shouldn't 3 mics be somewhat sufficient for intermediate values as well? More mics => more redundancy => clearer image of the waves, but why do the pixels/blobs "snap" to a microphone?
edit: ah, he does not use timing (at 10fps obviously not) but needs continuous sound to see the wave pattern across the array...
edit 2: so do these gunfire locators¹ only time stamp impulses? Or do they do continuous analysis? With natural, irregular sound, it should be possible to match and measure the delay between mics, no?
edit 3: the gunfire locators would need to analyze for gunfire characteristics anyways, to weed out false positives, but that could be separated from the locating.
edit x: oh cool, if you zoom out of the final picture so it's smaller on screen, you can even see some parallax effect happening: http://206hwf3fj4w52u3br03fi242.wpengine.netdna-cdn.com/wp-c...
¹ http://wordpress.mrreid.org/wp-content/uploads/2013/06/boome...
https://www.newscientist.com/article/mg21929364-400-matchsti...
http://www.npr.org/programs/invisibilia/378577902/how-to-bec...
(downsampled back to 16x16, than upsampled with photoshops more sophisticated resize and doubled speed for more fluent movement)
[0] https://en.wikipedia.org/wiki/Sound_localization
http://www.blueview.com/videos/2d-imaging-sonar/
I have an idea for you. You mention that you want to use a webcam so that you can "overlay" this information on top of an image.
What if you could send it directly to your brain?
There is a technology called Brainport[0], which is effectively a 16x16 grid of electrodes small enough to place on your tongue. By feeding electrical impulses through these electrodes, you stimulate the nerves on your tongue. After two minutes of a "champagn bubbling" sensation, your brain rewires the senses from your tongue to your visual cortex and you can "sense" any input.
This technology has been used to give sight to the blind, balance to those with inner ear problems, and sonar to NAVY seals.
It would be amazing if you were the first person in the world to truly "see" sound.
[0] https://en.wikipedia.org/wiki/Brainport
[1]http://www.thinkgeek.com/product/f358/ (sorry, I couldn't find a better source - but the product description explains it)
[1] https://www.indiegogo.com/projects/feelspace-follow-your-gut...
I've actually been collecting the vibration motors from disposable electric toothbrushes to build my own belt; as they were designed to run form a single AA battery powering them should be easy.
I've tried a prototype version of this belt once (for a couple of hours), it worked surprisingly well. But integrating the vibrations as a "new sense" into your brain (so that your subconscious can use it) takes at least some days.
Nicknamed 'batbelt' in-house.
I dropped it after looking into insurance requirements and ADA compliance.
Before I ditched the idea, I was going to put the vibrator belt as a second belt worn under the clothing, but there was no Bluetooth or WiFi or easily accessible way of going wireless between the belts in a practical fashion.
I had my neighbor, blind man who had a guide dog, trial it. He was a great guy, and helped me quite a bit to 'see' past my vision biases. Unfortunately, he passed away. He used to work for the IRS, and we would joke about me getting my deductions past him! He did like what I had accomplished, but many blind people do not have the money for something that is not subsidized, and then you get caught up in the compliance insurance issues. Not sure if that has changed much nowadays.
It was a great maker project before the 'maker' movement. Too bad there was no startup, or crowd-funding web sites back then!
Nicknamed 'batbelt' in-house.
I dropped it after looking into insurance requirements and ADA compliance.
It's not so far from that
Once you've managed to fully adapt to the new mapping, is it just as jarring to revert and go back to normal, or have you learned how to handle both?
Say you rotated for an entire year - or longer - on a weekly schedule between normal vision, vertically mirrored vision, and horizontally mirrored vision. Do you gain full training in all three to the point that you could transition between normal/vertical/horizontal at any time without any disorientation or readjustment period? Or can the brain really only handle one mapping at a time? This would be the ultimate question to answer.
I suspect the results would be similar to how we handle multiple languages. Your primary vision (ie: the one you learned first, or the one you have used exclusively for a decade or more) would remain the easiest to work with, while acquired alternatives may approach the "fluency" of the primary, while not being perfect.
Later on in the video, he mentions that having gotten used to the backwards bike, a regular bike was nearly impossible to ride until he adjusted to it again.
Fascinating stuff.
The thing basically scanned a line across the image, and produced sound based on the brightness. So, with your eyes closed you could quite easily imagine a basic shape of the image from the sound.
Maybe someone would recognize the description.
I understand that the rest of your brain 'trains' this pathway to occur. So because you know that there is a sound source and you can see it move in space, and you 'feel' this on your tongue as well, you are capable of correlating the two senses together and associating them with each other.
[0] https://en.wikipedia.org/wiki/Sensory_substitution
And it's about time someone did it! Yours is the first hobbyist project I've encountered since my own extremely crude ten-dollar version in 1981. No square array or processing, mine was a mechanical scanner-disk: eight LEDs glued in a spiral to an old vinyl record album, each with an electret microphone and and op amp. Spin the disk with a motor at about 6Hz. Wind noise, so add foam to the mics. It worked. A mechanical acoustic-phosphor raster sensor and display screen, where sound patterns caused light patterns. The best part was to hold some headphones near it while playing 10KHz sine wave. This produced interference stripes on the scanning disk! I hoped to build a huge version someday, but somebody beat me to it: Tom Zimmerman, an MIT student who patented the dataglove, then used the money to work at Exploratorium Museum for a year (as Artist in Residence), building a scanning-microphone LED disk about 6ft wide.
Since then we have companies selling DIDSON full-blown video for underwater (with sonic illumination,) also a euro company with a 3D phase-array microphone imager positioned on a ring or a wire sphere. See:
http://www.soundmetrics.com/Image-Gallery/Fisheries/Habitat-...
Heh, the divers appear to be skeletons wearing scuba tanks.
http://www.acoustic-camera.com/en/products/microphone-arrays...
It works quite well in my experience with it. The spherical geometry enables a 3D capture and visualization.
Don't miss multi-pages video gallery:
http://www.soundmetrics.com/Image-Gallery