Unless you're in your own home, I think it's basically a guarantee at this point that you're being recorded. Could be CCTV, trail cameras, some random recording a TikTok, etc...
That will only hold while being watched is rare. See Clarke and Baxter's Light of Other Days for an examination of the consequences of ubiquitous surveillance.
The watchers would be able to blackmail/control anybody who engages in private activities that they don't want to be public. So who watches the watchers? And who watches them? No. Privacy is 100% required in a free society.
The rayban metaglasshole comes to mind. Now its just journalists who fool people in the street with AI face recognition tricks, and its all still fun and games. But this is clearly a horror invention, merrily introduced by jolly zuck, boss of facelook.
> produce full-color images that are equal in quality to those produced by conventional cameras
I was really skeptical of this since the article conveniently doesn't include any photos taken by the nano-camera, but there are examples [1] in the original paper that are pretty impressive.
How does this work? If it's just reconstructing the images with nn, a la Samsung pasting a picture of the moon when it detected a white disc on the image, it's not very impressive.
I had the same thought, but it sounds like this operates at a much lower level than that kind of thing:
> Then, a physics-based neural network was used to process the images captured by the meta-optics camera. Because the neural network was trained on metasurface physics, it can remove aberrations produced by the camera.
I'd like to see some examples showing how it does when taking a picture of completely random fractal noise. That should show it's not just trained to reconstruct known image patterns.
Generally it's probably wise to be skeptical of anything that appears to get around the diffraction limit.
I believe the claim is that the NN is trained to reconstruct pixels, not images. As in so many areas, the diffraction limit is probabalistic so combining information from multiple overlapping samples and NNs trained on known diffracted -> accurate pairs may well recover information.
You’re right that it might fail on noise with resolution fine enough to break assumptions from the NN training set. But that’s not a super common application for cameras, and traditional cameras have their own limitations.
Not saying we shouldn’t be skeptical, just that there is a plausible mechanism here.
we've had very good chromatic aberration correction since I got a degree in imaging technology and that was over 20 years ago so I'd imagine it's not particularly difficult for name your flavour of ML.
My concern would be that if it can't produce accurate results on a random noise test, then how do we trust that it actually produces accurate results (as opposed to merely plausible results) on normal images?
Multilevel fractal noise specifically would give an indication of how fine you can go.
"Accurate results" gets you into the "what even is a photo" territory. Do today's cameras, with their huge technology stack, produce accurate results? With sharpening and color correction and all of that, probably not.
I agree that measuring against such a test would be interesting, but I'm not sure it's possible or desirable for any camera tech to produce an objectively "true" pixel by pixel value. This new approach may fail/cheat in different ways, which is interesting but not disqualifying to me.
> Ultrathin meta-optics utilize subwavelength nano-antennas to modulate incident light with greater design freedom and space-bandwidth product over conventional diffractive optical elements (DOEs).
Is this basically a visible-wavelength beamsteering phased array?
Interesting. This idea appears pretty much exactly at the end of Bob Shaw's 1972 SFnal collection Other Days, Other Eyes. The starting premise is the invention of "slow glass" that looks like an irrelevant gimmick but ends up revolutionizing all sorts of things, and the final bits envisage a disturbing surveillance society with these tiny passive cameras spread everywhere.
It's a good read; I don't think the extrapolation of one technical advance has ever been done better.
Those images are certainly impressive, but I certainly don't agree with the statement "equal in quality to those produced by conventional cameras": they're quite obviously lacking in sharpness and color.
There's one of those Taboola type ads going around with a similar image that suggests it is a close up of belly fat. Given the source and their propensity for using images unrelated to topic, so not sure if that's what it really is.
I wonder how they took pictures with four different cameras from the exact same position at the exact same point in time. Maybe the chameleon was staying very still, and maybe the flowers were indoors and that's why they didn't move in the breeze, and they used a special rock-solid mount that kept all three cameras perfectly aligned with microscopic precision. Or maybe these aren't genuine demonstrations, just mock-ups, and they didn't even really have a chameleon.
what happens when you go too far from trusting what you see/read/hear on the internet? simple logic gets tossed out like a baby in the bathwater.
now, here's the rig I'd love to see with this: take a hundred of them and position them like a bug's eye to see what could be done with that. there'd be so much overlapping coverage that 3D would be possible, yet the parallax would be so small that makes me wonder how much depth would be discernible
Given the size of their camera, you could glue it to the center of another camera’s lens with relatively insignificant effect on the larger camera’s performance.
They didn't really have a chameleon. See "Experimental setup" in the linked paper [emphasis mine]:
> After fabrication of the meta-optic, we account for fabrication error by performing a PSF calibration step. This is accomplished by using an optical relay system to image a pinhole illuminated by fiber-coupled LEDs. We then conduct imaging experiments by replacing the pinhole with an OLED monitor. The OLED monitor is used to display images that will be captured by our nano-optic imager.
But shooting a real chameleon is irrelevant to what they're trying to demonstrate here.
At the scales they're working at here ("nano-optics"), there's no travel distance for chromatic distortion to take place within the lens. Therefore, whether they're shooting a 3D scene (a chameleon) or a 2D scene (an OLED monitor showing a picture of a chameleon), the light that makes it through their tiny lens to hit the sensor is going to be the same.
(That's the intuitive explanation, at least; the technical explanation is a bit stranger, as the lens is sub-wavelength – and shaped into structures that act as antennae for specific light frequencies. You might say that all the lens is doing is chromatic distortion — but in a very controlled manner, "funnelling" each frequency of inbound light to a specific part of the sensor, somewhat like a MIMO antenna "funnels" each frequency-band of signal to a specific ADC+DSP. Which amounts to the same thing: this lens doesn't "see" any difference between 3D scenes and 2D images of those scenes.)
Chalk another one up for Vernor Vinge. This tech seems like it could directly enable the “ubiquitous surveillance” from _A Deepness in the Sky_. Definitely something to watch closely.
The other side to the localizers is the communication / mesh networking, and the extremely effective security partitioning. Even Anne couldn't crack them! It's certainly a lot to package in such a small form
I've been interested in smart dust for a while; recently the news seems to have dried up, and while that may have been other stuff taking up all the attention (and investment money), I suspect that many R&D teams went under government NDAs because they are now good enough to be interesting.
3 or 4 mm in diameter, according to a scene in chapter 6, big enough to have similar resolution to that of a human eye, according to Paul, but able to look in any direction without physically rotating.
In chapter 13 the enemy describes them as using Fourier optics, though that seemed to be their speculation - not sure whether it was right.
The paper says that reconstructing an actual image from the raw data produced by the sensor takes ~58ms of computation, so doing it for 10,000 sensors would naively take around ten minutes, though I'm sure there's room for optimization and parallelization.
The sensors produce 720x720px images, so a 100x100 array of them would produce 72,000x72,000px images, or ~5 gigapixels. That's a lot of pixels for a smartphone to push around and process and store.
Sensor size is super important for resulting quality, that's why pros still lug around huge full frame (even if mirrorless) cameras and not run around with phones. There are other reasons ie speed for sports but lets keep it simple (also speed is affected by data amount processed, which goes back to resolution).
Plus higher resolution sensors have this nasty habit of producing too large files, processing of which slows down given devices compared to smaller, crisper photos and they take much more space, even more so for videos. That's probably why Apple held to 12mpix main camera for so long, there were even 200mpix sensors available around if wanted.
Are they not? Every modern camera does the same thing. Upscaling, denoising, deblurring, adjusting colors, bumping and dropping shadows and highlights, pretty much no aspect of the picture is the way the sensor sees it once the rest of the pipeline is done. Phone cameras do this to a more extreme degree than say pro cameras, but they all do it.
To point out the obvious, film cameras don't, nor do many digital cameras. Unless you mean modern in the sense of "cameras you can buy from best buy right now", of course. But that isn't very interesting: best buy has terrible taste in cameras.
There are a lot of steps like that provided you want an image that you want to show to the user (i.e. Jpeg).
You do have somehow merge the 3 Bayer filter detections on rectangular grid, which involves interpolation. You do have to subtract some sort of bias in a detector, possibly correct for different sensitivity across the detector. You have to map the raw 'electron counts' into Jpeg scale which involves another set of decisions/image processing steps
There is clear processing in terms of interpreting the raw sensor data as you're describing. Then there are blurrier processes still, like "denoising" and "upscaling", which straddle the line between bias-correction and alteration. Then there's modification of actual color and luminance as the parent was describing. Now we're seeing full alterations applied automatically with neural nets, literally altering shapes and shadows and natural lighting phenomena.
I think it's useful to distinguish all of these even if they are desired. I really love my iPhone camera, but there's something deeply unsettling about how it alters the photos. It's fundamentally producing a different image you can get with either film or through your eyes. Naturally this is true for all digital sensors but we once could point out specifically how and why the resulting image differs from what our eyes see. It's no longer easy to even enumerate the possible alterations that go on via software, let alone control many of them, and I think there will be backlash at some point (or stated differently, a market for cameras that allow controlling this).
I've got to imagine it's frustrating for people who rely on their phone cameras for daily work to find out that upgrading a phone necessarily means relearning its foibles and adjusting how you shoot to accommodate it. Granted, I mostly take smartphone photos in situations where i'd rather not be neurotic about the result (candids, memories, reminders, etc) but surely there are professionals out there who can speak to this.
iPhone’s camera supports raw output, so if you use an appropriate app with an appropriate option you can definitely disable most of the funky stuff the default app does.
However, it is likely the more you turn off, the more the physical constraints will show. A regular dumb camera with a big sensor provides much more space for deterministic creative processing.
"The Neural network Image Processing features in this camera are arguably even more important here than they are in the R5 Mark II. A combination of deep learning and algorithmic AI is used to power In-Camera Upscaling, which transforms the pedestrian-resolution 24.2MP images into pixel-packed 96MP photos – immediately outclassing every full-frame camera on the market, and effectively hitting GFX and Hasselblad territory.
"On top of that is High ISO Noise Reduction, which uses AI to denoise images by 2 stops. It works wonders when you're pushing those higher ISOs, which are already way cleaner than you'd expect thanks to the flagship image sensor and modest pixel count."
The cameras themselves might not, but in order to get a decent picture you will need to apply demosaicing and gamma correction in software at the very least, even with high end cameras.
Right, and the point ppl are making upthread is that deterministic signal processing and probabilistic reconstruction approaches are apples and oranges.
Deterministic within a single image yes, but not within arbitrary subsections. Classical filters aren’t trying to reconstruct something resembling “other images it’s seen before”. Makes a difference both in theory and practice.
It's interesting how they mention beneficial impacts on medicine and science in general, but everyone knows that the first applications will likely be military and surveillance.
And since it's AI improved, all of th will hurt people because of hallucinations.
I don't trust human to avoid taking shorcuts once the tech is available, it's too convenient to have "information" for so cheap, and less costly to silence the occasional scandal.
Can we agree that in the field of cameras we surpassed science fiction?
I can remember watching a TV series as a child where a time traveler went back to the 80s and some person told him that everything is about miniaturization. Then he pointed to a little pin on the time traveler's jacket, which was actually a camera, and said: "This little pin for example could one day hold a full video camera", which seemed a bit ridiculous at that time.
Maybe I'm being too skeptical, and certainly I am only a layman in this field, but the amount of ANN-based post-processing it takes to produce the final image seems to cast suspicion on the meaning of the result.
At what point do you reduce the signal to the equivalent of an LLM prompt, with most of the resulting image being explained by the training data?
Yeah, I know that modern phone cameras are also heavily post-processed, but the hardware is at least producing a reasonable optical image to begin with. There's some correspondence between input and output; at least they're comparable.
I've seen someone on this site comment to the effect that if they could use a tool like dall-e to generate a picture of "their dog" that looked better than a photo they could take themselves, they would happily take it over a photo.
The future is going to become difficult for people who find value in creative activities, beyond just a raw audio/visual/textual signal at the output. I think most people who really care about a creative medium would say there's some kind of value in the process and the human intentionality that creates works, both for the creator who engages in it and the audience who is aware of it.
In my opinion most AI creative tools don't actually benefit serious creators, they just provide a competitive edge for companies to sell new products and enable more dilettantes to enter the scene and flood us with mediocrity
It is possible to create realistic images and videos with AI, making anyone do anything. Whether a photo or video is real or not will soon be impossible to distinguish, and it won't matter to those who want to cause harm
> Whether a photo or video is real or not will soon be impossible to distinguish,
You're forgetting something. Chain of custody, trust and reputation. The source of an image or video matters as to whether it is considered a reliable representation of reality or not.
We will develop better methods of verifying sources as well, possibly using cryptography and new social networks where members authenticate each other in-person and trust is built up.
We've always had this sort of stuff. Back in the 70s you had cameras the size of lighters. There's solutions for anyone determined enough. Even with authoritarian states, you'll find counter measures with sufficient demand. It's reed in the wind shit. Hopefully we won't kill ourselves in the process.
Everyone here is thinking about privacy and surveillance and here I am wondering if this is what lets us speed up nano cameras to relativistic speeds with lasers to image other solar systems up close.
We'll need even bigger[1] breakthroughs in propulsion if it's going to be self-propelling itself back to Sol at relativistic speeds.
1. A "simpler" sci-fi solution foe a 1-way trip that's still out of our reach is a large light sail and huge Earth-based laser, but his required "smaller" breakthroughs in material science
I meant to say the "simpler" (but still very complicated) solar sail approach was for a one-way trip. On paper, our civilization can muster the energy required to accelerate tiny masses to relativistic speeds. A return trip at those speeds would require a nee type of science to concentrate that amount of energy in a small mass and use it for controlled propulsion.
Well if you can propel something forward you can propel it backwards as well.
I'm assuming some sort of fixed laser type propulsion mechanism would leverage a type of solar sail technology. Maybe you could send a phased laser signal that "vibrates" a solar sail towards the source of energy instead of away.
> Well if you can propel something forward you can propel it backwards as well
Not necessarily - at least with currently known science. Light sails work ok transferring momentum from photons, allowing positive acceleration from a giant laser Earth. Return trip requires a giant laser on the other side.
Honestly even if they are size of a jellybean, it would be a massive boon for space exploration. Just imagine sending them for reconnaissance work around the solar system to check out potential bodies to explore for bigger probes later down the track. Even to catch interesting objects suddenly appearing with minimal delay, like ʻOumuamua.
This kind of thing -- that humans can do today with current technology -- is why if an ET intelligence that could travel interstellar distances wanted to observe us we would never know unless they wanted us to know.
Their probes could be the size of sand grains, maybe even dust. Maybe not quite sophons, but not much better as far as our odds of finding anything. I suppose there would have to be something larger to receive signals from these things and send them back (because physics), but that could be hanging out somewhere we'd be unlikely to see it.
Yet another Fermi paradox answer: we are looking for big spacecraft when the universe is full of smart dust.
From what I can tell it's using a neural network to derive an image from the interference patterns of light.
I imagine you could do this using a standard computational model, it would just be very intensive. So I guess it would be 'enhanced' in the same way a JPEG stores an image in a lossy format.
The nature article shows some sort of cmos like sensor with a surface made of pegs which seem to be conveniently close in size to the wavelengths of visible light. That passes through some sort of meta optic which presumably measures the diffraction off the sensor surface. Both sensor and "meta optic" data combined and extrapolated to form an image.
It's quite a clever way of designing a "lens" like that. Because you can generate an image from practically a flat surface. Of course the output image is "calculated" instead of just bending light through a series of glass lenses.
Years ago I saw an interview with a futurist that mentioned the following:
"One day, your kids will go to the toy store and get a sheet of stickers. Each sticker is actually a camera with an IPv6 address. That means they can put a sticker somewhere, go and point a browser at that address and see a live camera feed.
I should point out: all of the technology to do this already exists, it just hasn't gotten cheap enough to mass market. When economies of scale do kick in, society is going to have to deal with a dramatic change in what they think 'physical privacy' means."
135 comments
[ 2.8 ms ] story [ 187 ms ] threadMaybe on the other side it's good news as ppl are usually their best selves when they are being watched.
I don't think that view holds up.
A, it very much depends on who is watching, what their incentives are, and what power they hold.
And B, it also depends on who is being watched - not everyone thrives under a microscope. Are they the type to feel stifled? Or rebellious?
Has been published in 2021. Also here https://news.ycombinator.com/item?id=29399828
I was really skeptical of this since the article conveniently doesn't include any photos taken by the nano-camera, but there are examples [1] in the original paper that are pretty impressive.
[1] https://www.nature.com/articles/s41467-021-26443-0/figures/2
> Then, a physics-based neural network was used to process the images captured by the meta-optics camera. Because the neural network was trained on metasurface physics, it can remove aberrations produced by the camera.
Generally it's probably wise to be skeptical of anything that appears to get around the diffraction limit.
You’re right that it might fail on noise with resolution fine enough to break assumptions from the NN training set. But that’s not a super common application for cameras, and traditional cameras have their own limitations.
Not saying we shouldn’t be skeptical, just that there is a plausible mechanism here.
Multilevel fractal noise specifically would give an indication of how fine you can go.
I agree that measuring against such a test would be interesting, but I'm not sure it's possible or desirable for any camera tech to produce an objectively "true" pixel by pixel value. This new approach may fail/cheat in different ways, which is interesting but not disqualifying to me.
Is this basically a visible-wavelength beamsteering phased array?
It's a good read; I don't think the extrapolation of one technical advance has ever been done better.
https://news.ycombinator.com/item?id=42217251
now, here's the rig I'd love to see with this: take a hundred of them and position them like a bug's eye to see what could be done with that. there'd be so much overlapping coverage that 3D would be possible, yet the parallax would be so small that makes me wonder how much depth would be discernible
> After fabrication of the meta-optic, we account for fabrication error by performing a PSF calibration step. This is accomplished by using an optical relay system to image a pinhole illuminated by fiber-coupled LEDs. We then conduct imaging experiments by replacing the pinhole with an OLED monitor. The OLED monitor is used to display images that will be captured by our nano-optic imager.
But shooting a real chameleon is irrelevant to what they're trying to demonstrate here.
At the scales they're working at here ("nano-optics"), there's no travel distance for chromatic distortion to take place within the lens. Therefore, whether they're shooting a 3D scene (a chameleon) or a 2D scene (an OLED monitor showing a picture of a chameleon), the light that makes it through their tiny lens to hit the sensor is going to be the same.
(That's the intuitive explanation, at least; the technical explanation is a bit stranger, as the lens is sub-wavelength – and shaped into structures that act as antennae for specific light frequencies. You might say that all the lens is doing is chromatic distortion — but in a very controlled manner, "funnelling" each frequency of inbound light to a specific part of the sensor, somewhat like a MIMO antenna "funnels" each frequency-band of signal to a specific ADC+DSP. Which amounts to the same thing: this lens doesn't "see" any difference between 3D scenes and 2D images of those scenes.)
PS It's "Vernor"
[0] https://en.wikipedia.org/wiki/The_Peace_War
In chapter 13 the enemy describes them as using Fourier optics, though that seemed to be their speculation - not sure whether it was right.
The paper says that reconstructing an actual image from the raw data produced by the sensor takes ~58ms of computation, so doing it for 10,000 sensors would naively take around ten minutes, though I'm sure there's room for optimization and parallelization.
The sensors produce 720x720px images, so a 100x100 array of them would produce 72,000x72,000px images, or ~5 gigapixels. That's a lot of pixels for a smartphone to push around and process and store.
edit: mixed up bits and bytes
Plus higher resolution sensors have this nasty habit of producing too large files, processing of which slows down given devices compared to smaller, crisper photos and they take much more space, even more so for videos. That's probably why Apple held to 12mpix main camera for so long, there were even 200mpix sensors available around if wanted.
They're not comparable, in the intuitive sense, to conventional cameras.
I think it's useful to distinguish all of these even if they are desired. I really love my iPhone camera, but there's something deeply unsettling about how it alters the photos. It's fundamentally producing a different image you can get with either film or through your eyes. Naturally this is true for all digital sensors but we once could point out specifically how and why the resulting image differs from what our eyes see. It's no longer easy to even enumerate the possible alterations that go on via software, let alone control many of them, and I think there will be backlash at some point (or stated differently, a market for cameras that allow controlling this).
I've got to imagine it's frustrating for people who rely on their phone cameras for daily work to find out that upgrading a phone necessarily means relearning its foibles and adjusting how you shoot to accommodate it. Granted, I mostly take smartphone photos in situations where i'd rather not be neurotic about the result (candids, memories, reminders, etc) but surely there are professionals out there who can speak to this.
However, it is likely the more you turn off, the more the physical constraints will show. A regular dumb camera with a big sensor provides much more space for deterministic creative processing.
"The Neural network Image Processing features in this camera are arguably even more important here than they are in the R5 Mark II. A combination of deep learning and algorithmic AI is used to power In-Camera Upscaling, which transforms the pedestrian-resolution 24.2MP images into pixel-packed 96MP photos – immediately outclassing every full-frame camera on the market, and effectively hitting GFX and Hasselblad territory.
"On top of that is High ISO Noise Reduction, which uses AI to denoise images by 2 stops. It works wonders when you're pushing those higher ISOs, which are already way cleaner than you'd expect thanks to the flagship image sensor and modest pixel count."
Edit: by default.
I don't trust human to avoid taking shorcuts once the tech is available, it's too convenient to have "information" for so cheap, and less costly to silence the occasional scandal.
military, surveillance and porn
I can remember watching a TV series as a child where a time traveler went back to the 80s and some person told him that everything is about miniaturization. Then he pointed to a little pin on the time traveler's jacket, which was actually a camera, and said: "This little pin for example could one day hold a full video camera", which seemed a bit ridiculous at that time.
At what point do you reduce the signal to the equivalent of an LLM prompt, with most of the resulting image being explained by the training data?
Yeah, I know that modern phone cameras are also heavily post-processed, but the hardware is at least producing a reasonable optical image to begin with. There's some correspondence between input and output; at least they're comparable.
The future is going to become difficult for people who find value in creative activities, beyond just a raw audio/visual/textual signal at the output. I think most people who really care about a creative medium would say there's some kind of value in the process and the human intentionality that creates works, both for the creator who engages in it and the audience who is aware of it.
In my opinion most AI creative tools don't actually benefit serious creators, they just provide a competitive edge for companies to sell new products and enable more dilettantes to enter the scene and flood us with mediocrity
You're forgetting something. Chain of custody, trust and reputation. The source of an image or video matters as to whether it is considered a reliable representation of reality or not.
We will develop better methods of verifying sources as well, possibly using cryptography and new social networks where members authenticate each other in-person and trust is built up.
How would someone excrete an array of these cameras if ingested?
1. A "simpler" sci-fi solution foe a 1-way trip that's still out of our reach is a large light sail and huge Earth-based laser, but his required "smaller" breakthroughs in material science
I'm assuming some sort of fixed laser type propulsion mechanism would leverage a type of solar sail technology. Maybe you could send a phased laser signal that "vibrates" a solar sail towards the source of energy instead of away.
Not necessarily - at least with currently known science. Light sails work ok transferring momentum from photons, allowing positive acceleration from a giant laser Earth. Return trip requires a giant laser on the other side.
It's been a while since I've heard anyone talk about the Starshot project[0]. Maybe this would help revitalize it.
Also even without aiming for Proxima Centauri, it would be great to have more cameras in our own planetary system.
--
[0] - https://en.wikipedia.org/wiki/Breakthrough_Starshot
https://www.centauri-dreams.org/2024/01/19/data-return-from-...
Their probes could be the size of sand grains, maybe even dust. Maybe not quite sophons, but not much better as far as our odds of finding anything. I suppose there would have to be something larger to receive signals from these things and send them back (because physics), but that could be hanging out somewhere we'd be unlikely to see it.
Yet another Fermi paradox answer: we are looking for big spacecraft when the universe is full of smart dust.
Then there is the whole "neural" part. Do these get "enhanced" by a generative AI that fills the blur based on the most statistically likely pixels?
The article is pretty bad.
I imagine you could do this using a standard computational model, it would just be very intensive. So I guess it would be 'enhanced' in the same way a JPEG stores an image in a lossy format.
It's quite a clever way of designing a "lens" like that. Because you can generate an image from practically a flat surface. Of course the output image is "calculated" instead of just bending light through a series of glass lenses.
"One day, your kids will go to the toy store and get a sheet of stickers. Each sticker is actually a camera with an IPv6 address. That means they can put a sticker somewhere, go and point a browser at that address and see a live camera feed.
I should point out: all of the technology to do this already exists, it just hasn't gotten cheap enough to mass market. When economies of scale do kick in, society is going to have to deal with a dramatic change in what they think 'physical privacy' means."