When you use an VAE, you implicitly assume that your data never ever leaves the area between 0 and 1 and that it'll never ever touch the borders.
But if you have pictures of scanned-in handwritten digits, there's a chance that in reality, the line was blacker than the darkest color that your image file format can store. For example, if you did contrast enhancement after scanning, you likely cut off some extra dark or extra bright pixels.
A regular VAE cannot handle this situation that multiple brightness=0 (black) pixels actually had varying negative brightnesses which were clamped to 0 for technical reasons. A continuous Bernoulli can model 0 really meaning <=0. And 1 meaning >=1.
That's why with a continuous Bernoulli you get more saturated white and blacks than with just a Bernoulli. See the image in 5.1, for example. That's a good thing because it increases contrast, thus making your features better and dependent learning tasks easier.
Having a quick read, they're pointing out that the Bernoulli distribution is only supported for values of 0 and 1 (i.e. it's binary), whereas pixel values for a grayscale image are a decimal value in the interval [0, 1]. When you train a VAE, it's pretty standard to use a BCE loss, but this is wrong because the data isn't binary (i.e. it's not a Bernoulli distribution). They define a continuous analogue of a Bernoulli distribution that is supported in [0, 1], and use this as the loss function for training a VAE, which gives them reconstructions that are closer to the input.
Yeah, agree having skimmed the paper now. It seems kinda meh to me: it's obvious that improving the loss function is a good thing to do, and for any real application (ie, not greyscale mnist) you'd never use this binary value target.
> the continuous Bernoulli is not a beta distribution (the main difference between these two distributions is how they concentrate mass around the extrema, see appendix 1 for details)
This right here means I will never look at this again. Not because of the misspelling, but because of the fact that I do not know what conditions there are on my use.
I made the mistake of buying this book https://www.amazon.com/dp/1119482089 at full price, only to find out that all code examples are some version of GPL. Only, this information is hidden in the small print.
Please tell people upfront what they are allowed to do with the code and documentation you share.
11 comments
[ 2.8 ms ] story [ 36.5 ms ] threadI've dabbled in VAEs, but haven't read this paper. Anyone want to try an ELI5thYearGradStudent?
When you use an VAE, you implicitly assume that your data never ever leaves the area between 0 and 1 and that it'll never ever touch the borders.
But if you have pictures of scanned-in handwritten digits, there's a chance that in reality, the line was blacker than the darkest color that your image file format can store. For example, if you did contrast enhancement after scanning, you likely cut off some extra dark or extra bright pixels.
A regular VAE cannot handle this situation that multiple brightness=0 (black) pixels actually had varying negative brightnesses which were clamped to 0 for technical reasons. A continuous Bernoulli can model 0 really meaning <=0. And 1 meaning >=1.
That's why with a continuous Bernoulli you get more saturated white and blacks than with just a Bernoulli. See the image in 5.1, for example. That's a good thing because it increases contrast, thus making your features better and dependent learning tasks easier.
Edit: I see it’s equivalent to a Beta distribution.
It's not.
The beta distribution is p(x) ∝ x^(α-1) (1-x)^(β-1). x is in the bases.
The continuous Bernoulli distribution is p(x) ∝ λ^x (1-λ)^x. x is in the exponents.
As https://arxiv.org/abs/1907.06845 says:
> the continuous Bernoulli is not a beta distribution (the main difference between these two distributions is how they concentrate mass around the extrema, see appendix 1 for details)
The appendix is available here (under "Supplemental"): https://papers.nips.cc/paper/2019/hash/f82798ec8909d23e55679...
This right here means I will never look at this again. Not because of the misspelling, but because of the fact that I do not know what conditions there are on my use.
I made the mistake of buying this book https://www.amazon.com/dp/1119482089 at full price, only to find out that all code examples are some version of GPL. Only, this information is hidden in the small print.
Please tell people upfront what they are allowed to do with the code and documentation you share.
Also, `s/lamda/lambda/`.