I was hoping that someone came out with a camera that not only had not only sensors for visible light, but for infrared and UV. It's just another color to add to the sensors; I think we have enough megapixels, seems like going for other bands is reasonable.
I don't understand how from 3 independent values per pixel (RGB) they claim to derive 200+ independent values per pixel. Unless they are assuming a smooth "image" (all pixels the same RGB), perturbed only by the color card? Not exactly a camera then
This doesn't really seem like "hyperspectral imaging". I think the idea is having a reference colour chart of known emission characteristics and photographing it through a transparent substance gives you an idea of how much that substance attenuates each wavelength.
It's a cool trick if it works, but it seems very finicky and I guess would be limited to transparent/homogeneous liquids?
On top of that it only works in the VIS range, thanks to the filters in front of the camera sensors - and most of the interesting information is in UV or IR. VIS only contains information about a few elements. (see Fraunhofer lines https://en.wikipedia.org/wiki/Fraunhofer_lines)
This is really cool and very clever. But i want to raise one thing.
> designed a special color reference chart that can be printed on a card
My rudimentary understanding of physics makes me suspect this sentence is a simplification.
A normal printer use Cyan Magenta Yellow Black to print. A photo of such a print would already destroy alot of spectral information for the same reason the individual rgb sensors do.
So i suspect those colored dots are a very careful and deliberate concoction of very particular inks with very specific spectral color bands.
I suspect alot of effort went into finding, mixing and algoritmically combining the right inks.
I'm guessing it works similarly to a how a narrow band florescent lamp makes only materials that reflect a very specific frequency be visible, which makea alot of prints and pigments look wierd. (If you do the opposite; use ink with very specific spectral band, you can instead measure the lamp)
I would assume that with time, you could just print it on a generic $30 inkjet, and calibrate the card, sensor, and whitelte balance to a stored reference image.
It won't be as accurate, but it might be enough to offer some insights into whether liquid photographed in the article is in fact whisky, not urine (which to me seems to be a much more noble demonstration subject).
I was initially surprised too, but I think there is a some trick...
I'm not sure what happens on paper, but when you have ink disolved in water the abortion is not linearly proportional to the concentration, it's exponential. For example, consider a red ink and 5 magical selected frequencies and the absortions at 1% of concentration are: 99%, 99%, 50%, 10%, 1%
If you double the ink at 2% you get 99.99%, 99.99%, 75%, 19%, 1.99%
So increasing or decreasing the ink concentration may give you information of different frequencies, even with only one ink and only one sensor. In this case mostly about the 3rd and 4th. With more concentration you may kill all the light in the 3rd and measure the absortions ratio between the 4th and the 5th.
One problem I see here is how to order them, but I guess it's possible with a few sensors and a few inks. Each sensor sees all the frequencies, but with different weight. I'm not sure if it's possible to solve this, but perhaps you need some initial approximated model of the inks(???).
Now, when you put the inks on paper and have a unreliable light source and perhaps other technical problems, ...
In conclusion, I think it's possible to use different saturation and mixes of the inks to get different spectral distribution of the light that bounce on the card. Then use the three sensors to get three averages and try to use big linear algebra book to reconstruct what happens in between. But I should read the paper to be sure.
Patent-pending... again someone trying to rent-seek a high-school physics fair idea. Measuring light absorption with a camera is almost as old as the camera.
Using a known reflectance chart in-scene to recover spectral information is a standard calibration technique.
Someone needs to build a phone that is leaning towards a tricorder; I'd buy that for myself and my kids. My Pixel 10 has a temp sensor on it, which is cool, but I've had minimal use so far.
I've always wanted to build a tricorder with my son, was just thinking about it last week when he was putting together a digital compass (with RasPi Nano, magnetic sensor, GPS, and LED light ring + OLED).
I notice the article doesn't say anything about accuracy. This is not my area, but I think the _other_ hacky way to try to do spectroscopy with a phone is with a diffraction
grating (and maybe a box with a slit in it). Diffraction gratings are cheap, probably not so different from a specially-printed reference card. If you have a choice, which is better?
More specific info on the reference card is available in the paper's supplemental information. https://ieeexplore.ieee.org/ielx8/83/10795784/11125864/supp1.... Basically, they used special paper with a pro-Canon inkjet, along with a special ICC color profile.
Every now and then we discover new traces left by the original light field (by using a fuller picture to model the image formation process, instead of an easier oversimplified one).
It makes one wonder how much of the noise is actually misinterpreted signal.
I do not understand how this could possibly work. From a camera with RGB filters we get essentially three different integrals over the spectrum per pixel, how would we recover the spectrum from that? Even assuming you can account for the spectrum of the light sources and the color filters in the camera - which should be doable with a color chart with known spectra - I do not see how you could go from three data points to a full spectrum without making assumptions about the possible object spectra.
EDIT: Okay, after going to the actual paper I at least get transmission mode - you photograph the color chart through the sample and this will of course imprint the absorption spectrum onto the know spectrum of the color chart and you can then look at the difference to the color chart without the sample in between. But I do not get the logic behind their reflectance mode.
This is a combination of things that almost breaks my brain, but it works, and it's brilliant!
First, you need to understand coded masks[1,2]. This is a way to use an LCD or other array to mask off parts of a scene so that very specific parts of it are sampled, but the rest aren't, to get a single, high resolution analog value. Then you switch to other masks, and get more values. You can then work backwards in the math through the known mask shapes, to get the original image with far fewer samples that would be required one at a time.
Think of the above as a 2d visual version of the Fourier transform[3,4]. This transform is used heavily to compress images throwing away most of the bits in an image without losing it's essence.
The analysis they're talking about uses a very specially printed card. It isn't just something generated with a standard 4 ink printer, each "dot" is a separate unique ink with tightly controlled spectral curves, these form a virtual version of the above masks. When you view these through a sample, it can then give an idea of the spectral response of the camera, and the liquid, by using the many different known response curves of the "dots" to work backwards, and generate the 1 dimensional very tight response curve of a hyperspectral imager, by figuring out where each "dot" is in the scene, then averaging that dot's intensity across the RGB values of the picture taken by the camera. Today's cameras have sufficient resolution and bit depth that you get the original bit depth (usually 8 bits) and an additional bit for each doubling of the number of pixels in a given "dot". This is degraded by Bayer pattern filters[5,6], and the nature of cameras, but it's not unrecoverable.
Like with Coded Masks, and Fourier transforms, you then take your high resolution analog values, and work backwards to get the things you want to measure.
21 comments
[ 9.8 ms ] story [ 96.4 ms ] thread> “Every photo carries hidden spectral information waiting to be uncovered. By extracting it, we can turn everyday photography into science.”
And with our patent, extract rent from anyone who wants to do it!
It's a cool trick if it works, but it seems very finicky and I guess would be limited to transparent/homogeneous liquids?
> designed a special color reference chart that can be printed on a card
My rudimentary understanding of physics makes me suspect this sentence is a simplification.
A normal printer use Cyan Magenta Yellow Black to print. A photo of such a print would already destroy alot of spectral information for the same reason the individual rgb sensors do.
So i suspect those colored dots are a very careful and deliberate concoction of very particular inks with very specific spectral color bands.
I suspect alot of effort went into finding, mixing and algoritmically combining the right inks.
I'm guessing it works similarly to a how a narrow band florescent lamp makes only materials that reflect a very specific frequency be visible, which makea alot of prints and pigments look wierd. (If you do the opposite; use ink with very specific spectral band, you can instead measure the lamp)
Insanely clever. (Whatever they did)
It won't be as accurate, but it might be enough to offer some insights into whether liquid photographed in the article is in fact whisky, not urine (which to me seems to be a much more noble demonstration subject).
I'm not sure what happens on paper, but when you have ink disolved in water the abortion is not linearly proportional to the concentration, it's exponential. For example, consider a red ink and 5 magical selected frequencies and the absortions at 1% of concentration are: 99%, 99%, 50%, 10%, 1%
If you double the ink at 2% you get 99.99%, 99.99%, 75%, 19%, 1.99%
So increasing or decreasing the ink concentration may give you information of different frequencies, even with only one ink and only one sensor. In this case mostly about the 3rd and 4th. With more concentration you may kill all the light in the 3rd and measure the absortions ratio between the 4th and the 5th.
One problem I see here is how to order them, but I guess it's possible with a few sensors and a few inks. Each sensor sees all the frequencies, but with different weight. I'm not sure if it's possible to solve this, but perhaps you need some initial approximated model of the inks(???).
Now, when you put the inks on paper and have a unreliable light source and perhaps other technical problems, ...
In conclusion, I think it's possible to use different saturation and mixes of the inks to get different spectral distribution of the light that bounce on the card. Then use the three sensors to get three averages and try to use big linear algebra book to reconstruct what happens in between. But I should read the paper to be sure.
Using a known reflectance chart in-scene to recover spectral information is a standard calibration technique.
What "investment" is patent law protecting here?
I've always wanted to build a tricorder with my son, was just thinking about it last week when he was putting together a digital compass (with RasPi Nano, magnetic sensor, GPS, and LED light ring + OLED).
Every now and then we discover new traces left by the original light field (by using a fuller picture to model the image formation process, instead of an easier oversimplified one).
It makes one wonder how much of the noise is actually misinterpreted signal.
Custom processors to accelerate generation of AI text: Go ahead
Slightly thicker to fit a bigger battery in: How dare you
EDIT: Okay, after going to the actual paper I at least get transmission mode - you photograph the color chart through the sample and this will of course imprint the absorption spectrum onto the know spectrum of the color chart and you can then look at the difference to the color chart without the sample in between. But I do not get the logic behind their reflectance mode.
First, you need to understand coded masks[1,2]. This is a way to use an LCD or other array to mask off parts of a scene so that very specific parts of it are sampled, but the rest aren't, to get a single, high resolution analog value. Then you switch to other masks, and get more values. You can then work backwards in the math through the known mask shapes, to get the original image with far fewer samples that would be required one at a time.
Think of the above as a 2d visual version of the Fourier transform[3,4]. This transform is used heavily to compress images throwing away most of the bits in an image without losing it's essence.
The analysis they're talking about uses a very specially printed card. It isn't just something generated with a standard 4 ink printer, each "dot" is a separate unique ink with tightly controlled spectral curves, these form a virtual version of the above masks. When you view these through a sample, it can then give an idea of the spectral response of the camera, and the liquid, by using the many different known response curves of the "dots" to work backwards, and generate the 1 dimensional very tight response curve of a hyperspectral imager, by figuring out where each "dot" is in the scene, then averaging that dot's intensity across the RGB values of the picture taken by the camera. Today's cameras have sufficient resolution and bit depth that you get the original bit depth (usually 8 bits) and an additional bit for each doubling of the number of pixels in a given "dot". This is degraded by Bayer pattern filters[5,6], and the nature of cameras, but it's not unrecoverable.
Like with Coded Masks, and Fourier transforms, you then take your high resolution analog values, and work backwards to get the things you want to measure.
[1] https://www.youtube.com/watch?v=_ezhdhHNku0
[2] https://en.wikipedia.org/wiki/Coded_aperture
[3] https://en.wikipedia.org/wiki/Fourier_transform
[4] https://www.youtube.com/watch?v=spUNpyF58BY&pp=ygURZm91cmllc...
[5] https://www.youtube.com/watch?v=LWxu4rkZBLw&pp=ygUMYmF5ZXIgZ...
[6] https://en.wikipedia.org/wiki/Bayer_filter