> This grain, formed from tiny particles during the film’s development, is more than just a visual effect. It plays a key role in storytelling by enhancing the film’s depth and contributing to its realism.
I never understood the “grain = realism” thing. my real eyes don’t have grain. I do appreciate the role of grain as an artistic tool though, so this is still cool tech
It's funny, because I've always had a subtle visual snow that looks like a film grain on top of everything I see, and for a long time I thought everybody had it and that that's why artificial grain is added, to make pictures appear more realistic.
I've since learned that not everybody sees the world like I do, but I still do love to see grain and noise in pictures.
Only RGB noise I often find dreadfully ugly when looked at up close, which is a shame, since that is exactly what most color cameras include.
Honestly to me it reads like a solution looking for a problem. I’ve never considered non deterministic imperfections that happen during recording of the movie to be essential to storytelling.
There are definite philosophical questions over the merits of adding noise, but the problem with their example here is their denoising process appears to excessively blur everything, so both it and the synthesized grain image look noticeably less sharp than the source. The grain itself also looks too much like basic noise, and not really grain like.
I don't think it's just that here. Our brains simply prefer high entropy signals over low entropy ones. It's why mobile phones (and digital cameras before them) use sharpen filters to the point where the result looks unnatural.
The "at scale" part is the real story here. Film Grain Synthesis has been available in the usual AV1 encoders for a while, but required some amount of manual tweaking to avoid creating problems, meaning it was only used in production when you had a very limited catalog, or for particularly important titles. They do not provide a lot of details here about how they overcame those problems, but it is nice to see it being deployed more broadly.
There's an influx of high-profile directors/films right now and in pipeline filmed for IMAX (F1: The Movie I think, Mission Impossible, etc) and Christopher Nolan's Odyssey coming next year shot entirely on IMAX film with newly developed smaller/quieter cameras made to accomplish it.
> Picture this: you’re watching a classic film, and the subtle dance of film grain adds a layer of authenticity and nostalgia to every scene
It just adds visual noise that obscures details of the authentic scene, and nothing prevents nostalgia from being tied to many of the more prominent visual cues like old actors or your own old memories from when you watched it first...
> contributing to [film's] realism
But there is no grain in reality, so it does the opposite
Otherwise I'm glad AV1 marches along and instead of wasting bitrate encoding visual garbage has an algorithmic replacement mechanism- which also means you could turn it off easier.
happier and happier about leaving behind digital media to return to physical. to me this is literally slop. i want the uncompressed file stop selling me stepped on product
Everything is fake now. I want a technology which works with a raw film scans, not even compressing them to JPEG, which is a 1st step in loosing the details BTW. Motion detection, key frames, delta frames - fine. But with a lossless video. On a Blu Ray off course, i don't care much about streaming.
It's a bit frustrating that the footage is first shot, then denoised in post, then renoised in post, then denoised in encoding and then renoised at decoding.
Between 24fps and film grain people are way too attached to fundamentally inferior technology. With how strongly people resist frame rates faster than 24 I'm surprised people accepted color and sound, which were much bigger changes.
I like this. Not really because I feel modern media should have added grain, but because for older media this is a method to get closer to the original but at much lower bitrates without excessive smoothing. What’s not to like?
Also, the author had me at God of Gamblers 2. So good. I will take him up on his recommendation to rewatch.
To the comments hating on grain: everything naturally has some amount of noise or grain - even the best digital sensors. Heck, even your eyes do. It's useful beyond just aesthetics. It tends to increase perceived sharpness and hides flaws like color banding and compression artifacts.
That's not to say that all noise and grain is good. It can be unavoidable, due to inferior technology, or a result of poor creative choices. It can even be distracting. But the alternative where everything undergoes denoising (which many of our cameras do by default now) is much worse in my opinion. To my eyes, the smoothing that happens with denoising often looks unrealistic and far more distracting.
Case in point: the HBO into animation. It uses static from the old analog days. It looks like shit even at 4k because random noise is impossible to compress without the exact strategy outlined here (stripping it out and rendering it later).
The original grain that is captured is actually a detail and not total random noise. I believe you can make up the vague sense of original scene if you could somehow extract that grain/noise alone.
It's like reducing an image to tiny dots with dithering (reminds of Atinkson dithering). Those grains are not a noise, they are a detail, actual data. That's why real grain looks good IMO.
When you talk on a cellphone the codec AMR-WB nominally captures 50 Hz - 7000 Hz. However that's only on the optional highest bitrate 23.85 Kbps. The most common bitrate 12.65 Kbps only goes up to 6400 Hz and synthesizes 6400 - 7000 Hz from lower frequencies and noise as it sounds better than not having the noise!
This fails to acknowledge that synthesized noise can lack the detail and information in the original noise.
When you watch a high-quality encode that includes the actual noise, there is a startling increase in resolution from seeing a still to seeing the video. The noise is effectively dancing over a signal, and at 24 fps the signal is still perfectly clear behind it.
Whereas if you lossily encode a still that discards the noise and then adds back artificial noise to match the original "aesthetically", the original detail is non-recoverable if this is done frame-by-frame. Watching at 24 fps produces a fundamentally blurrier viewing experience. And it's not subtle -- on old noisy movies the difference in detail can be 2x.
Now, if h.265 or AV1 is actually building its "noise-removed" frames by always taking into account several preceding and following frames while accounting for movement, it could in theory discover the signal of the full detail across time and encode that, and there wouldn't be any loss in detail. But I don't think it does? I'd love to know if I'm mistaken.
But basically, the point is: comparing noise removal and synthesis can't be done using still images. You have to see an actual video comparison side-by-side to determine if detail is being thrown away or preserved. Noise isn't just noise -- noise is detail too.
To illustrate the temporal aspect: consider a traditional film projector. Between every frame, we actually see complete darkness for a short time. We could call that darkness "noise", and if we were to linger on that moment, we'd see nothing of the original signal. But since our visual systems tend to temporally average things out to a degree, we barely even notice that flicker (https://en.wikipedia.org/wiki/Flicker_fusion_threshold). I suspect noise and grain are perceived in a similar way, where they become less pronounced compared to the stable parts of the signal/image.
Astrophotographers stack noisy images to obtain images with higher SNR. I think our brains do a bit of that too, and it doesn't mean we're hallucinating detail that isn't there; the recorded noise - over time - returns to the mean, and that mean represents a clearer representation of the actual signal (though not entirely, due to systematic/non-random noise, but that's often less significant).
Denoising algorithms that operate on individual frames don't have that context, so they will lose detail (or will try to compensate by guessing). AV1 doesn't specify a specific algorithm to use, so I suppose in theory, a smart algorithm could use the temporal context to preserve some additional detail.
I love this concept/principle, one similar example I often bring up when I talk about machine learning, is comparing how a human would analyse night footage from a camera, and how a ML algorithm can pick up things no human would think about, even artifacts from the sensors which can be used as features. Noise is rarely ever just noise.
Some of the new 4K discs use DRR and the denoising process seems to remove the pores on people's faces occasionally, leaving actors looking like their face is made of wax.
Don't want to sound too snarky, but aren't you just saying that good denoisers must be temporal in addition to spatial? Something like an improved V-BM3D (cf https://arxiv.org/abs/2001.01802) and even traditional V-BM3D works fine even on non-Gaussian noise.
Together with external noise generation (something like https://old.reddit.com/r/AV1/comments/r86nsb/custom_photonno...) that'd support external denoising (pass the reference and denoised video to get the grain spec), the whole FGS thing would be much more interesting.
51 comments
[ 0.16 ms ] story [ 113 ms ] threadI never understood the “grain = realism” thing. my real eyes don’t have grain. I do appreciate the role of grain as an artistic tool though, so this is still cool tech
I've since learned that not everybody sees the world like I do, but I still do love to see grain and noise in pictures. Only RGB noise I often find dreadfully ugly when looked at up close, which is a shame, since that is exactly what most color cameras include.
ANY noticeable percieved "flaw" in any creative media will eventually become an aesthetic choice.
There's an influx of high-profile directors/films right now and in pipeline filmed for IMAX (F1: The Movie I think, Mission Impossible, etc) and Christopher Nolan's Odyssey coming next year shot entirely on IMAX film with newly developed smaller/quieter cameras made to accomplish it.
It just adds visual noise that obscures details of the authentic scene, and nothing prevents nostalgia from being tied to many of the more prominent visual cues like old actors or your own old memories from when you watched it first...
> contributing to [film's] realism
But there is no grain in reality, so it does the opposite
Otherwise I'm glad AV1 marches along and instead of wasting bitrate encoding visual garbage has an algorithmic replacement mechanism- which also means you could turn it off easier.
Also, the author had me at God of Gamblers 2. So good. I will take him up on his recommendation to rewatch.
That's not to say that all noise and grain is good. It can be unavoidable, due to inferior technology, or a result of poor creative choices. It can even be distracting. But the alternative where everything undergoes denoising (which many of our cameras do by default now) is much worse in my opinion. To my eyes, the smoothing that happens with denoising often looks unrealistic and far more distracting.
It's like reducing an image to tiny dots with dithering (reminds of Atinkson dithering). Those grains are not a noise, they are a detail, actual data. That's why real grain looks good IMO.
When you watch a high-quality encode that includes the actual noise, there is a startling increase in resolution from seeing a still to seeing the video. The noise is effectively dancing over a signal, and at 24 fps the signal is still perfectly clear behind it.
Whereas if you lossily encode a still that discards the noise and then adds back artificial noise to match the original "aesthetically", the original detail is non-recoverable if this is done frame-by-frame. Watching at 24 fps produces a fundamentally blurrier viewing experience. And it's not subtle -- on old noisy movies the difference in detail can be 2x.
Now, if h.265 or AV1 is actually building its "noise-removed" frames by always taking into account several preceding and following frames while accounting for movement, it could in theory discover the signal of the full detail across time and encode that, and there wouldn't be any loss in detail. But I don't think it does? I'd love to know if I'm mistaken.
But basically, the point is: comparing noise removal and synthesis can't be done using still images. You have to see an actual video comparison side-by-side to determine if detail is being thrown away or preserved. Noise isn't just noise -- noise is detail too.
To illustrate the temporal aspect: consider a traditional film projector. Between every frame, we actually see complete darkness for a short time. We could call that darkness "noise", and if we were to linger on that moment, we'd see nothing of the original signal. But since our visual systems tend to temporally average things out to a degree, we barely even notice that flicker (https://en.wikipedia.org/wiki/Flicker_fusion_threshold). I suspect noise and grain are perceived in a similar way, where they become less pronounced compared to the stable parts of the signal/image.
Astrophotographers stack noisy images to obtain images with higher SNR. I think our brains do a bit of that too, and it doesn't mean we're hallucinating detail that isn't there; the recorded noise - over time - returns to the mean, and that mean represents a clearer representation of the actual signal (though not entirely, due to systematic/non-random noise, but that's often less significant).
Denoising algorithms that operate on individual frames don't have that context, so they will lose detail (or will try to compensate by guessing). AV1 doesn't specify a specific algorithm to use, so I suppose in theory, a smart algorithm could use the temporal context to preserve some additional detail.
Together with external noise generation (something like https://old.reddit.com/r/AV1/comments/r86nsb/custom_photonno...) that'd support external denoising (pass the reference and denoised video to get the grain spec), the whole FGS thing would be much more interesting.
Eastman Business Park in Rochester has been demolished.
Also, please stop putting dust and scratches on YouTube videos. Thank you.