raises hand I interpreted it as the news company, looked at Github and came back going "what does CNN have to do with this, it looks like they're uni students." Thank you for the actual acronym expansion, it's obtuse if you don't know anything about this stuff. Terrible title.
The title needs to be changed so brains recognize it as such. It either needs a preceding adjective or letter indicating what type of convolutional network it is.
The other option is spelling it out.
Most people will read CNN as the news channel. Even those familiar with neural networks.
I'm not from Europe, but if I saw "BBC converts night images to perfect daylight in ~1 second", I would assume it meant the British Broadcasting Corporation. CNN is just as big of a name. His point is absolutely relevant.
Said differently: The percentage of people who are not from the US - but are aware of CNN as the Cable News Network, is higher than the percentage of people who are not machine learning experts - but are aware of CNN as a Convolutional Neural Network
I don't think it needs to be changed. I also read it as the news channel, wondered how that would make sense, and guessed it was the other sort of CNN. It's confusing but not misleading.
See also Ted Chiang's What’s Expected of Us, where there's a device with a button and a LED, and the LED always lights up one second before you press the button.
I understand that the title may have brought confusion, and I think your comment calls attention to this whilst also being somewhat funny.
But still, could we make an effort not to devolve into what has happened on Reddit, i.e. comment sections which mainly consist of puns and other low effort jokes?
Sure, and ideally those are downvoted. But low-grade jokes fit in here even less and are usually downvoted. At least that's how it is before HN finishes its transition into Reddit, at which point they won't be downvoted and we'll be back to scrolling through bad jokes like "lol CNN in the title looks like CNN the news network lmao, anyone else notice?" to find serious comments.
It's a fight worth fighting. "C'mon it's just a joke lol" or "it's only a comment about the title" only assist in that transformation.
Meta: this is when I like the Slashdot method, you allow the jokes, tag them as jokes, and let people use their own settings to show/hide the jokes. Basically be permissive on content, but demand proper tagging, then allow people to filter out what they don't want.
I was just wondering a couple days ago why the image from my phone is so grainy, while my eyes+brain can see everything clear in the dark (it wasn't completely dark, of course).
This seems to replicate the post-processing we do in our brain (which is also a giant neural network). I wonder if the process is similar?
To be clear, the parent did not fail to include a citation. The parenthetical note is that rod cells are so sensitive that they react to being struck by a single photon.
Your brain doesn’t make a 2 dimensional image based entirely on photons entering your eye. You generate a complex physical model of your surroundings based only partially on visual input and rely substantially on memory.
Kind of like this well trained CNN is no longer relying entirely on the raw pixel values, but is statistically inferring a brighter image from the baseline.
Not to harp on this, but the point is that, as I understand it, both “systems” are using exogenous information to extrapolate more data than is actually present in the source image.
That’s not to say that the same “thing” is happening at the granular level at all.
But this is distinctly different from standard filtering functions, which can only work with entropy already present in the source image. So there’s a neat distinction.
The output from the CNN is essentially an “artist interpretation” of the source image. As such there could be “clarifying details” in the output that were in fact totally invented and not actually present in the source.
Also other senses, including proprioception. In a completely dark environment, you could swear that you see your hand waving in front of your face. That's because your brain actually does know it's there, and it's trying to create a unified model.
”The human eye can detect a luminance range of 10¹⁴, or one hundred trillion (100,000,000,000,000) (about 46.5 f-stops), from 10−6 cd/m2, or one millionth (0.000001) of a candela per square meter to 10⁸ cd/m2 or one hundred million (100,000,000) candelas per square meter. This range does not include looking at the midday sun (10⁹ cd/m2)[21] or lightning discharge.”
You shouldn't be downvoted - with a big recession/depression looming, link rot and many sorts of repositories shutting down are a big issue that will slow down the pace of research.
It's surprising how little code [1] is needed to do this. On the other hand I feel this is quite dependent on the specific camera models and might not work on the RAW data downloaded from my phone. Happy to be corrected.
It's a huge amount of code, hidden behind the tensorflow import statements. It's common to credit GPUs for the rapid spread of deep learning, but good GPUs were available for quite a few years before deep learning really took off. As someone who wrote * a lot* of OpenCL code, including my own python wrappers, I'm fairly certain this code would be thousands of lines without a computation graph framework library. These frameworks are really amazing pieces of software engineering and deserve some non-trivial fraction of the credit for the rise of deep learning.
If you want to know what the next hot thing in software engineering will be, just pay attention to whatever Jeff Dean is doing.
I don't know that I agree with this first statement, but even if I do, everything is abstracted by import statements even outside ML. You say this is a huge amount of code abstracted, but it wouldn't be difficult to reimplement this in numpy and pandas directly without using tensorflow at all. The code would expand a bit, and you'd have to deal directly with backprop and calculating derivatives but it wouldn't expand things too much. But then you could make the same claim about numpy abstracting the linear algebra, and I could show you that I could extract that and do it without numpy but then it would be the python math library. It's turtles all the way down. My point is, your comment applies to everything.
Yup, I absolutely agree. Almost all big leaps in software engineering and applied computer science come from building a powerful and simple abstraction. Powerful and simple abstractions are surprisingly difficult to get right.
When people say "little code" what they mean an should say is "little customization of the tool".
There's also the issue of how hard it is to select the line of code (API call) that does the job (because the API surface is huge), which is nearly invisible attribute.
Mathematica is famous for being incredibly powerful with low custom coding, but also very hard to find the API call that does the precise thing you need.
We don't have good GPUs that is as fast until 580 (or to some extents, the first Titan). Previous generations only about 2 to 5x faster depending on what types of CPU you compare against.
IMHO, credit should always go to Alex Krizhevsky for the rapid spread of deep learning. He has shown us it was possible. Even without Tensorflow and PyTorch, we will be fine with Caffe, torch, mxnet or Julia.
This goes to show why we can't take HN seriously: hyping up whatever is blessed by Google, regardless of actual value. What novel thing did exactly TensorFlow bring to the table? Also, has Jeff Dean invented CNNs?
I implemented neural networks before the advent of the good python frameworks. It sucked. And CNNs existed for decades before AlexNet. Honestly, the software and hardware engineers are the real heroes of the deep learning revolution.
By way of analogy, David Heinemeier Hansson didn't invent the webapp or even the MVC design pattern. But Ruby on Rails did change the way webapps were built, and enabled a bunch of stuff that wouldn't have been possible otherwise (or, at least, would've taken longer and been more expensive). Lots of websites were built because of Ruby on Rails that probably wouldn't have been built otherwise, even if people would've kept on doing the web thing regardless. We can say the same thing about lisps, about linux, and about a lot of other software infrastructure.
Any high schooler who's capable of learning python and can afford a gaming desktop can build and train a neural network. That's pretty amazing, and definitely wouldn't be the case without computation graph frameworks.
I've never worked for Google or with Jeff, and I'm not a huge fan of the ad tech industry, although I don't understand why either of those things should matter.
This could actually be shortened, maybe simplified, significantly. For example there is a lot of redundancy in the layers and that could be pulled out into a function per block. This is what I often do with deep networks as it helps avoid errors in code and shortens everything significantly, at the potential expense of being able to grok it initially as quickly.
But, many DNN concepts (and ML concepts themselves) can be described with a few lines of pseudocode. CNNs, RNNs, etc. can all be described in a few lines.
It's really quite amazing, most of the work goes into first creating the net work from theory, then training and tuning it until you get good results.
Generally, you just need to subtract the right black level and pack the data in the same way of Sony/Fuji data. If using rawpy, you need to read the black level instead of using 512 in the provided code. The data range may also differ if it is not 14 bits. You need to normalize it to [0,1] for the network input."
The Sony and Fuji training code looks mostly the same - they haven't bothered to pull out common code and re-use.
Which is extremely lossy, because any ISO other than the sensor's native level is the result of in-camera processing. Unlike film, adjusting the "ISO" in a digital camera doesn't increase sensitivity; that's physically impossible. Instead, very strong overgain processing is applied.
So in this instance they're processing lossily on top of an image already processed lossily in-camera.
The A7g image is used for comparison rather than input data, as that camera is widely regard as the state of the art for low-light photography on a non-scientific/military budget.
As a photographer, the comparison to "raw" results without color balance or noise removal seems somewhat deceptive. The effects visible in the video seem easy to quickly replicate with existing techniques, such as the "surface blur" filter that averages out pixel values in areas with similar color.
This happens at the expense of detail in low-contrast areas, producing a plastic-like appearance of human skin and hair, and making low-contrast text unintelligible, which is why it's generally not done by default.
It would indeed be interesting to see a comparison with for instance non-local means on the scaled raw image. The speed is superior in any case, I suspect.
Your example strikes me as the kind of thing neural networks are much better at than a fixed filter. You or I could easily identify regions of an image where it's safe vs unsafe to do the surface averaging, and boundaries where we wouldn't want to mix up the averages. (For example, averaging text should be fine, so long as you don't cross the text boundaries.) A CNN should also be able to learn to do this pretty easily.
What you are describing is a class of filters known as edge preserving filters. You can look at bilateral filters and guided filters for examples that have been around for decades at this point.
So we can do a decent job with hand designed filters... Why aren't they in use in the problem the parent describes? Are they not good enough to deal with small text boundaries?
A lot of hand built filters (I see a lot of these in the audio space) have many hand tuned parameters, which work well in certain circumstances, and less well in other circumstances. One of the big advantages of NN systems is the ability to adapt to context more dynamically. The NN filters can generally emulate the hand designed system, and pick out weightings appropriate to the example.
This is effectively noise reduction, which bilateral and guided filters are actually used for. They take the weights of their kernels based on local pixels and statistics. You can also look up other edge preserving filters like BM3D and non-local means.
I don't know what you mean by hand made filters and I don't know why that's a conclusion you jumped to.
Interestingly, this effect is notably visible in their example image [0]. Notice the distinctly "plasticized" appearance of the book cover, and how the text is not intelligible in the low-contrast areas of the reflection.
Note (a) and (b) are separate photographs (different angles and everything), and that (c) is based on (a), not (b); comparing the glare between (b) and (c) isn't quite an even comparison.
Oh gosh. Taking one second to think about it, _of course_ (a) and (b) are separate photographs -- that is the entire point of that diagram. Somehow my brain farted right over that when making my previous comment.
Thank you, not only for setting me straight, but also for doing so as kindly as you did.
i always have this complaint too. its fundamentally a lossy process, in the hand wavy sense. its more "impressive" looking, but actually conveying less real detail.
To me the results seem vastly superior to those sort of simple DSP algorithms. The video shows a comparison with some denoising: https://youtu.be/qWKUFK7MWvg?t=102
The comparison is fair because it tries to automate expertise.
I'm sure you know exactly how much of which filter to apply for similar results. Laymen like ourselves will need a lot more trial and error. Their contribution here is to provide a push-button, automated mechanism.
I would have probably also tried something simple and given up due to the noise. So this is definitely interesting.
What you are describing is usually called automatic tone mapping. This is basically noise reduction and possibly color normalization from brightening a dark image. Them showing their black image as the starting point is silly, because jpg will make a mess of the remaining information. What they should show is the raw image brightened by a straight multiplier to show the noisy version that you would get from trying to increase brightness in a trivial way.
Image on Github is JPEG made from RAW. Since RAW file has more dynamic range and contains a lot more information than JPEG you can take that photo in an editor and crank up the brightness. You will get a noisy image but it will be a lot brighter and will probably resemble the image with the high ISO in the middle. Then in an editor you can apply some de-noiser to get results similar to the last one.
So presumably this neural net more or less does it for you.
The *PNG is there just to show the results produced by the CNN, if you watch the linked video they do exactly what you are suggesting and then compare both results.
Pretty cool but seems like there’s a big limitation on this for now
“ The pretrained model probably not work for data from another camera sensor. We do not have support for other camera data. It also does not work for images after camera ISP, i.e., the JPG or PNG data.”
Would be cool to see how they come up with better models that would allow them to overcome the above limitations
The problem with techniques like this is that they fundamentally amount to ‘making a plausible guess as to what the image would look like’, since essentially they can’t extract information that is simply not there. There is a Shannon entropy limit here.
Machine learning is really machine-enhanced educated-guesswork, which has its place but also has its limits.
Yes and that's what this "lossy" conversion to daytime does as well, incorporate prior knowledge, but that prior knowledge is about how images of real world things function during night versus day.
First of all, if you were to use the image as a communication channel, how much you could theoretically communicate is exactly the entropy of messages (by definition), and optimal communication means maximum entropy. Information theory already assumes shared existing knowledge in the form of codes; the codes essentially encode all this knowledge, which you could make an analogy in images to digits and shapes, etc. -- what makes them decodable is there is statistical redundancy, shapes do not occur arbitrarily (i.e. not every possible shape occurs, at least not with equal probability) and exhibit dependence between different pixels of a shape and even between shapes elsewhere in the image. Again the dependence is (generally) based on the statistics of the distribution of all possible images -- it essentially encodes all prior knowledge.
This redundancy allows reconstruction of losses in on part of the image from data elsewhere. It's the same principle used in error correcting codes, except codes are designed, while shapes are mostly natural (except things like alphabets, which are designed and indeed follow some principles of codes). But because they're not designed there's not guarantee of having a unique/reliable decoding (i.e. you can get a distribution).
I think that's a significant issue, because if your estimate doesn't match reality it could have important consequences for the use of the image: perhaps text goes from 'X is good' to 'X sucks'.
In this case a few things could be done:
1) Have some kind of watermark indicating the image was enhanced by a neural network, and possibly contains false information;
2) Have some kind of indication of reliability of the image: it should encode the multimodality/confidence of the decoding distribution -- how many different solutions does this have. If it is more or less unique, it would show as high confidence; otherwise it would show a low confidence indicator;
3) Instead of trying to convey uncertainty, the system could simply give up in cases where there is too much uncertainty, i.e. leaving the image dark. This could be done locally or globally, although locally it introduces a lightning consistency problem.
---
There's another important observation w.r.t. Information Theory/Statistics: it essentially assumes unbounded computational power (since this distribution could require analyzing arbitrarily large datasets). Of course this isn't true in reality. For example, the entropy of an encrypted of a redundant text is exactly the entropy of the plaintext string plus the entropy of the key (given an encryption ensamble or encryption prior) -- the process of (e.g. through brute force) finding the key doesn't concern statistics. However, with reasonable computational power, the (properly) encrypted stream is indistinguishable from a random string, hence it would have maximum entropy. So there are further computational limits beside statistical limits. In the case of encryption the function is again designed (to be not tractable), while in natural images the correlations are of simpler and hopefully more tractable nature (although I'm sure not always the case).
The courts & hopefully the jurors should be aware that humans are fallible & capable of lying, but may have hard time believing that cameras can lie as well.
Two 2D Images. The brain doesn't have to guess much when it can use the parallax effect created by both eyes. That's why quite a few animals have two eyes. And I believe that's why we have two ears, too.
I'm also not a fan of how the only part actually readable in the (a) original, which is part of the title in the front book, becomes completely whitewashed in (c). Where the model actually had the most information, it completely removed it in the result...
I guess I might have misinterpreted the goal. If the goal was to make the image look like it was daylight, then maybe whitewashing that light reflection was the correct choice. If the goal was to "see in the dark", then it seems like a very bad choice.
EDIT: Finally got the paper to load via the helpful wayback machine link provided in another thread. It looks like the goal was to simulate a long exposure with a short exposure. So whitewashing of "bright" areas in the original might be expected.
Regardless of the goal, there's no way to get a more readable result if the data just isn't there. Whitewashing might simply be a result of that absence.
My point was that it whitewashed exactly those areas which had the most information. However, this is inline with their stated goal of mimicking a long exposure. It's not inline with "seeing in the dark", but that's not their goal.
If you insist on the same point, you didn't understand my initial reply. Again, I'm assuming the input is (a), not (b). But maybe you mean the lighter areas have the most information. If so, why? Just because there's more light? More light != more information. It could just be a bunch of noise.
> More light != more information. It could just be a bunch of noise.
You do realize that, in a camera sensor, light is the signal, right? So the more light, the higher the signal-to-noise ratio, which means that yes it does have more information available to extract.
And yes, I quite realize that the input is (a). I'm guessing that in your display you are not seeing that there is a brighter spot in the middle of (a) corresponding to the whitewashed area in (c). Try maxing out your brightness if you're on a phone or laptop and you should see it. I can even make out letter shapes in (a) within this bright spot.
But wouldn't that level of glare be what would actually happen if you took the original image, and shot it in the amount of light required to make it look like the output image?
It's not trying to make things readable; it's trying to make things look like there was more light in the room when they were shot. In rooms with high lighting, some objects have glare. That's "correct"—it's what would appear in the training data.
Agreed. It gets to an important point of the purpose of the photo. Photo as a record versus photo as an aesthetic piece. This hurts the photo as a record but improves the photo as an aesthetic piece. This would be a bad addition to a security camera, but perhaps a good addition to an instagram pipeline. There are plenty of other issues there, like is it good/healthy for stuff like instagram to be diverging away from records, creating unrealistic (or perhaps literally unreal) expectations. We're taking what are slowly inching closer to imaginative art pieces and presenting them as records.
Photographs have never been faithful records. The map never the territory.
There are always judgement calls.
The whole concept of JPEG is to throw away information.
Nothing's a perfect record, but a thing's purpose can be to serve as a record. Like if you stick seismometer readings into an academic paper, everyone will know there's noise, but that's okay. You did your best. But there are some changes we can agree are bad there. Now, if you stuck your seismometer into an autotuner and sync'd it with led zeppelin, that's okay. Just not if you're putting that into your paper.
For a long time, photographs were typically used as records. Even when they were art, they were typically records of something. Soon, we'll be typically doing so much more with them and will have to accept that photos can, but frequently won't serve as records.
It's more than 'good enough' for most purposes. Matching other shots for Hollywood quality, probably not. For surveillance or the like it's fine. The things it's guessing poorly about are textures or colors.
Being able to read the title on the books in the example photo is great; you could rely on the title for evidentiary purposes, the smaller text probably not so much. So for a security camera it would do poorly at identifying the color of a car, but might well be sufficient to read the license plate.
Reading a license plate seems like precisely the kind of circumstances where spurious 'plausible interpretation' of limited data can cause trouble.
You show in a courtroom a CNN-enhanced low light image of a car and it's there, literally 'clear as day' - the jury will find it pretty compelling. But maybe the data really wasn't there in the original image, and the CNN just filled in some blanks based on previous images of license plates, letters, and just random noise it had seen in the past.
The worry is when these kinds of algorithms get built in to basic image capture processes, so you never even see the raw data, only data that has already been filtered through the inbuilt prejudices of the CNN enhancement suite.
The camera never lies, but now it doesn't have to, because it can convince itself it saw something that wasn't really there...
Of course, but what are the odds that the algorithm just lucked into the correct book title and other cover text? It doesn't have a dictionary or semantic network.
You are right that the raw sensor data should always be preserved. But sticking with the license plate example, you could challenge a picture of a single car with a visible license plate far away in a wooded area, but it would be hard to refute a picture of the same car in a parking lot surrounded by other (non-suspect) vehicles whose presence there at the same time could be independently verified. In other words, if I can show that it accurately read the license plates of 9 other cars, the chances that it got yours wrong go way down.
That's assuming a single photo taken in the dark by an investigator. With a fixed security camera you would have an even larger basis of comparison, with a population of hundreds or thousands of license plates against which to rate it. I predict that before long we'll see preemptive certification for devices warranting the reliability of their image pipeline out to a certain distance at either the manufacturing or installation stage.
License plates are an ideal breeding ground for false enhancement owing to standardisation of appearance; an ML algo trained on lots of examples might, without due care, learn to replace as a well-known texture.
The pre-emptive certification I mention would be a validation of due care. It doesn't matter how many theoretical arguments you want to throw up against this, once there's sufficient empirical evidence for its reliability (and there will be) it will be accepted as evidence.
Also, y'all need to think more like prosecutors. Say you are dragged you into court on the basis of photos showing your car in the dark, and you object that the photo is from the ML 9000 security camera and it might be just imaging your license plate. The police/prosecutors will just 'borrow' your car and leave it there for a night and leave it up to the jury.
Forensic evidence can be and is regularly abused, but it can also be quite easily validated and it's massively persuasive to juries.
I personally think juries are a great way to convict innocent people, and that adversarial court systems privilege people who can afford to pay for the best storyteller, so arguments from that direction start out hobbled.
Let’s assume I’m innocent but some neural net has placed my car at the location of a crime.
You’re saying that if I challenge the evidence, the prosecutors will counter that by showing that if my car were there, the neural net would have produced a picture of my car? They don’t need to do that and it adds nothing to their argument. I’m not challenging that the neural net is capable of producing an image of my car.
No, the point is I am placed in the position of having to demonstrate that there exists some other car which under those lighting conditions the neural net would mistake for mine. That’s a far harder burden of proof for me to reach.
Honestly this is similar to the way fingerprint, DNA and hair sample matches are presented to courtrooms all the time so it isn’t a new problem. As you say, forensics are persuasive.
And you can see some of its biases in the results, yeah. Look at the thick book's spine on the right, and compare the box around the title - "our result" has pretty significant staircasing instead of being a slightly-off-vertical line.
If you could show a basic level of consistency. Take the correctly rendered title on the book vs the incorrect colors; the odds that it got lucky with the text instead of a different title or a book with random letters on the cover are negligible. But if your evidence revolved around the color of the book the villain stole from the library, not so much.
So if you're planning to do crime, make choices where the evidence relies on spectra rather than geometry. Steal Rothkos rather than Mondrians; baggy coveralls are in, form-fitting ninja wear is out.
Xerox copiers had a bug caused by a failed image (re)construction, which caused it to replace similar (but not identical) parts of an image with other pieces of the image.
An interpolation that looks for movement of a few anchor points? I imagine that would entail much less computation and so deliver apparent real-time night vision. Though sudden big movements in scene would cause blackout regions of about a second?
If you take the dark image (a) from that and balance its color, the information that is present in it simply cannot contain the text from the book covers and so on. In fact, it's full of JPEG artifacts despite the image being a PNG. It would be useful if they presented a histogram equalized image of (a).
It would be good to get a comparison of a brightened version of the sample image compared with the CNN version. Right now the sample image just looks black, but if you scale up the brightness you get an image that looks more like the higher ISO image. That would be a better comparison since it shows what improvements the CNN gives over naive techniques like just bumping up the pixel values.
178 comments
[ 3.3 ms ] story [ 299 ms ] threadhttps://en.wikipedia.org/wiki/Ted_Turner
The other option is spelling it out.
Most people will read CNN as the news channel. Even those familiar with neural networks.
Agree that title should be changed.
Said differently: The percentage of people who are not from the US - but are aware of CNN as the Cable News Network, is higher than the percentage of people who are not machine learning experts - but are aware of CNN as a Convolutional Neural Network
2. CNN exists outside of the US
See also: https://en.m.wikipedia.org/wiki/Thiotimoline
The major peculiarity of the chemical is its "endochronicity": it starts dissolving before it makes contact with water.
The heart of each Predictor is a circuit with a negative time delay — it sends a signal back in time.
But still, could we make an effort not to devolve into what has happened on Reddit, i.e. comment sections which mainly consist of puns and other low effort jokes?
Would make sense to add Tensorflow to make it more specific.
And it is even more "HN" to comment on details of the title or the article because you don't really know what to say about the article.
Look at my comment.
It's a fight worth fighting. "C'mon it's just a joke lol" or "it's only a comment about the title" only assist in that transformation.
https://www.thedrive.com/the-war-zone/25803/this-is-what-col...
This seems to replicate the post-processing we do in our brain (which is also a giant neural network). I wonder if the process is similar?
That’s not to say that the same “thing” is happening at the granular level at all.
But this is distinctly different from standard filtering functions, which can only work with entropy already present in the source image. So there’s a neat distinction.
The output from the CNN is essentially an “artist interpretation” of the source image. As such there could be “clarifying details” in the output that were in fact totally invented and not actually present in the source.
”The human eye can detect a luminance range of 10¹⁴, or one hundred trillion (100,000,000,000,000) (about 46.5 f-stops), from 10−6 cd/m2, or one millionth (0.000001) of a candela per square meter to 10⁸ cd/m2 or one hundred million (100,000,000) candelas per square meter. This range does not include looking at the midday sun (10⁹ cd/m2)[21] or lightning discharge.”
[1] - https://github.com/cchen156/Learning-to-See-in-the-Dark/blob...
If you want to know what the next hot thing in software engineering will be, just pay attention to whatever Jeff Dean is doing.
There's also the issue of how hard it is to select the line of code (API call) that does the job (because the API surface is huge), which is nearly invisible attribute.
Mathematica is famous for being incredibly powerful with low custom coding, but also very hard to find the API call that does the precise thing you need.
IMHO, credit should always go to Alex Krizhevsky for the rapid spread of deep learning. He has shown us it was possible. Even without Tensorflow and PyTorch, we will be fine with Caffe, torch, mxnet or Julia.
You're a stupid lemming.
By way of analogy, David Heinemeier Hansson didn't invent the webapp or even the MVC design pattern. But Ruby on Rails did change the way webapps were built, and enabled a bunch of stuff that wouldn't have been possible otherwise (or, at least, would've taken longer and been more expensive). Lots of websites were built because of Ruby on Rails that probably wouldn't have been built otherwise, even if people would've kept on doing the web thing regardless. We can say the same thing about lisps, about linux, and about a lot of other software infrastructure.
Any high schooler who's capable of learning python and can afford a gaming desktop can build and train a neural network. That's pretty amazing, and definitely wouldn't be the case without computation graph frameworks.
I've never worked for Google or with Jeff, and I'm not a huge fan of the ad tech industry, although I don't understand why either of those things should matter.
But, many DNN concepts (and ML concepts themselves) can be described with a few lines of pseudocode. CNNs, RNNs, etc. can all be described in a few lines.
It's really quite amazing, most of the work goes into first creating the net work from theory, then training and tuning it until you get good results.
Generally, you just need to subtract the right black level and pack the data in the same way of Sony/Fuji data. If using rawpy, you need to read the black level instead of using 512 in the provided code. The data range may also differ if it is not 14 bits. You need to normalize it to [0,1] for the network input."
The Sony and Fuji training code looks mostly the same - they haven't bothered to pull out common code and re-use.
>Still images: ISO 100-102400 (expandable to ISO 50-409600),
[1]:https://www.sony.co.uk/electronics/interchangeable-lens-came...
So in this instance they're processing lossily on top of an image already processed lossily in-camera.
This happens at the expense of detail in low-contrast areas, producing a plastic-like appearance of human skin and hair, and making low-contrast text unintelligible, which is why it's generally not done by default.
A lot of hand built filters (I see a lot of these in the audio space) have many hand tuned parameters, which work well in certain circumstances, and less well in other circumstances. One of the big advantages of NN systems is the ability to adapt to context more dynamically. The NN filters can generally emulate the hand designed system, and pick out weightings appropriate to the example.
I don't know what you mean by hand made filters and I don't know why that's a conclusion you jumped to.
[0]: https://raw.githubusercontent.com/cchen156/Learning-to-See-i...
Thank you, not only for setting me straight, but also for doing so as kindly as you did.
I'm sure you know exactly how much of which filter to apply for similar results. Laymen like ourselves will need a lot more trial and error. Their contribution here is to provide a push-button, automated mechanism.
I would have probably also tried something simple and given up due to the noise. So this is definitely interesting.
What you are describing is usually called automatic tone mapping. This is basically noise reduction and possibly color normalization from brightening a dark image. Them showing their black image as the starting point is silly, because jpg will make a mess of the remaining information. What they should show is the raw image brightened by a straight multiplier to show the noisy version that you would get from trying to increase brightness in a trivial way.
So presumably this neural net more or less does it for you.
Huh? At 1:40 in the video that's exactly what they do.
“ The pretrained model probably not work for data from another camera sensor. We do not have support for other camera data. It also does not work for images after camera ISP, i.e., the JPG or PNG data.”
Would be cool to see how they come up with better models that would allow them to overcome the above limitations
Machine learning is really machine-enhanced educated-guesswork, which has its place but also has its limits.
There is an entropy limit to the message, but the message isn't actually the only data.
One thing humans are great at is integrating existing knowledge into a messy situation and intuiting more than is available just from the raw message.
I.e. The message has an entropy limit, but the message isn't the whole dataset.
First of all, if you were to use the image as a communication channel, how much you could theoretically communicate is exactly the entropy of messages (by definition), and optimal communication means maximum entropy. Information theory already assumes shared existing knowledge in the form of codes; the codes essentially encode all this knowledge, which you could make an analogy in images to digits and shapes, etc. -- what makes them decodable is there is statistical redundancy, shapes do not occur arbitrarily (i.e. not every possible shape occurs, at least not with equal probability) and exhibit dependence between different pixels of a shape and even between shapes elsewhere in the image. Again the dependence is (generally) based on the statistics of the distribution of all possible images -- it essentially encodes all prior knowledge.
This redundancy allows reconstruction of losses in on part of the image from data elsewhere. It's the same principle used in error correcting codes, except codes are designed, while shapes are mostly natural (except things like alphabets, which are designed and indeed follow some principles of codes). But because they're not designed there's not guarantee of having a unique/reliable decoding (i.e. you can get a distribution).
I think that's a significant issue, because if your estimate doesn't match reality it could have important consequences for the use of the image: perhaps text goes from 'X is good' to 'X sucks'.
In this case a few things could be done:
1) Have some kind of watermark indicating the image was enhanced by a neural network, and possibly contains false information;
2) Have some kind of indication of reliability of the image: it should encode the multimodality/confidence of the decoding distribution -- how many different solutions does this have. If it is more or less unique, it would show as high confidence; otherwise it would show a low confidence indicator;
3) Instead of trying to convey uncertainty, the system could simply give up in cases where there is too much uncertainty, i.e. leaving the image dark. This could be done locally or globally, although locally it introduces a lightning consistency problem.
---
There's another important observation w.r.t. Information Theory/Statistics: it essentially assumes unbounded computational power (since this distribution could require analyzing arbitrarily large datasets). Of course this isn't true in reality. For example, the entropy of an encrypted of a redundant text is exactly the entropy of the plaintext string plus the entropy of the key (given an encryption ensamble or encryption prior) -- the process of (e.g. through brute force) finding the key doesn't concern statistics. However, with reasonable computational power, the (properly) encrypted stream is indistinguishable from a random string, hence it would have maximum entropy. So there are further computational limits beside statistical limits. In the case of encryption the function is again designed (to be not tractable), while in natural images the correlations are of simpler and hopefully more tractable nature (although I'm sure not always the case).
Eyewitness testimony is awful but given gold status.
Wait, did it? Isn't the middle photo being shown for comparison only, rather than as an input?
EDIT: Finally got the paper to load via the helpful wayback machine link provided in another thread. It looks like the goal was to simulate a long exposure with a short exposure. So whitewashing of "bright" areas in the original might be expected.
You do realize that, in a camera sensor, light is the signal, right? So the more light, the higher the signal-to-noise ratio, which means that yes it does have more information available to extract.
And yes, I quite realize that the input is (a). I'm guessing that in your display you are not seeing that there is a brighter spot in the middle of (a) corresponding to the whitewashed area in (c). Try maxing out your brightness if you're on a phone or laptop and you should see it. I can even make out letter shapes in (a) within this bright spot.
It's not trying to make things readable; it's trying to make things look like there was more light in the room when they were shot. In rooms with high lighting, some objects have glare. That's "correct"—it's what would appear in the training data.
For a long time, photographs were typically used as records. Even when they were art, they were typically records of something. Soon, we'll be typically doing so much more with them and will have to accept that photos can, but frequently won't serve as records.
Being able to read the title on the books in the example photo is great; you could rely on the title for evidentiary purposes, the smaller text probably not so much. So for a security camera it would do poorly at identifying the color of a car, but might well be sufficient to read the license plate.
You show in a courtroom a CNN-enhanced low light image of a car and it's there, literally 'clear as day' - the jury will find it pretty compelling. But maybe the data really wasn't there in the original image, and the CNN just filled in some blanks based on previous images of license plates, letters, and just random noise it had seen in the past.
The worry is when these kinds of algorithms get built in to basic image capture processes, so you never even see the raw data, only data that has already been filtered through the inbuilt prejudices of the CNN enhancement suite.
The camera never lies, but now it doesn't have to, because it can convince itself it saw something that wasn't really there...
You are right that the raw sensor data should always be preserved. But sticking with the license plate example, you could challenge a picture of a single car with a visible license plate far away in a wooded area, but it would be hard to refute a picture of the same car in a parking lot surrounded by other (non-suspect) vehicles whose presence there at the same time could be independently verified. In other words, if I can show that it accurately read the license plates of 9 other cars, the chances that it got yours wrong go way down.
That's assuming a single photo taken in the dark by an investigator. With a fixed security camera you would have an even larger basis of comparison, with a population of hundreds or thousands of license plates against which to rate it. I predict that before long we'll see preemptive certification for devices warranting the reliability of their image pipeline out to a certain distance at either the manufacturing or installation stage.
https://www.theregister.co.uk/2013/08/06/xerox_copier_flaw_m...
License plates are an ideal breeding ground for false enhancement owing to standardisation of appearance; an ML algo trained on lots of examples might, without due care, learn to replace as a well-known texture.
Also, y'all need to think more like prosecutors. Say you are dragged you into court on the basis of photos showing your car in the dark, and you object that the photo is from the ML 9000 security camera and it might be just imaging your license plate. The police/prosecutors will just 'borrow' your car and leave it there for a night and leave it up to the jury.
Forensic evidence can be and is regularly abused, but it can also be quite easily validated and it's massively persuasive to juries.
Let’s assume I’m innocent but some neural net has placed my car at the location of a crime.
You’re saying that if I challenge the evidence, the prosecutors will counter that by showing that if my car were there, the neural net would have produced a picture of my car? They don’t need to do that and it adds nothing to their argument. I’m not challenging that the neural net is capable of producing an image of my car.
No, the point is I am placed in the position of having to demonstrate that there exists some other car which under those lighting conditions the neural net would mistake for mine. That’s a far harder burden of proof for me to reach.
Honestly this is similar to the way fingerprint, DNA and hair sample matches are presented to courtrooms all the time so it isn’t a new problem. As you say, forensics are persuasive.
people bring this up all the time as hot-takes in these areas. it's conditional inference. it's no more disingenuous than linear regression.
So if you're planning to do crime, make choices where the evidence relies on spectra rather than geometry. Steal Rothkos rather than Mondrians; baggy coveralls are in, form-fitting ninja wear is out.
http://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_...?
- Did they create a special network topology for this problem?
- Does the network need to see the entire image, or only an NxN subblock at a time?
- How did they obtain the training data? Is it possible to take daylight images and automatically turn them into nighttime images somehow?
https://github.com/cchen156/Learning-to-See-in-the-Dark/blob...
If you take the dark image (a) from that and balance its color, the information that is present in it simply cannot contain the text from the book covers and so on. In fact, it's full of JPEG artifacts despite the image being a PNG. It would be useful if they presented a histogram equalized image of (a).