To evaluate this experimental compression codec, I didn’t use any of the standard test images or images found online in order to ensure that I’m not testing it on any data that might have been used in the training set of the Stable Diffusion model (because such images might get an unfair compression advantage, since part of their data might already be encoded in the trained model).
I think it would be very interesting to determine if these images do come back with notably better compression.
There's something to be said about compression algorithms being predictable, deterministic, and only capable of introducing defects that stand out as compression artifacts.
Plus, decoding performance and power consumption matters, especially on mobile devices (which also happens be the setting where bandwidth gains are most meaningful).
This reminds me of a question I have about SD: why can’t it do a simple OCR to know those are characters not random shapes? It’s baffling that neither SD nor DE2 have any understanding of the content they produce.
> why can’t it do a simple OCR to know those are characters not random shapes?
It's pretty easy to add this if you wanted to.
But a better method would be to fine tune on a bunch of machine-generated images of words if you want your model to be good at generating characters. You'll need to consider which of the many Unicode character sets you want your model to specialize in though.
You could certainly apply a “duct tape” solution like that, but the issue is that neural networks were developed to replace what were previously entire solutions built on a “duct tape” collection of rule-based approaches (see the early attempts at image recognition). So it would be nice to solve the problem in a more general way.
With compression you often make a prediction then delta off of it. A structurally garbled one could be discarded or just result in a worse baseline for the delta.
Remember the Xerox scan-to-email scandal in which tiling compression was replacing numbers in structural drawings? We're talking about similar repercussions here.
I was told (on the Unstable Diffusion discord, so this info might not be reliable) that even with using the same seed the results will differ if the model is running on a different GPU. This was also my experience when I couldn't reproduce the results generated by the discord's SD txt2img generating bot.
I'm not sure about the different GPU issue. But if that is an issue, the model can be made deterministic (probably compromising inference speed), by making sure the calculations are computed deterministically.
While that is kind of true it is also sort of the point.
The optimal lossy compression algorithm would be based on humans as a target. it would remove details that we wouldn't notice to reduce the target size. If you show me a photo of a face in front of some grass the optimal solution would likely be to reproduce that face in high detail but replace the grass with "stock imagery".
I guess it comes down to what is important. In the past algorithms were focused on visual perception, but maybe we are getting so good at convincingly removing unnecessary detail that we need to spend more time teaching the compressor what details are important. For example if I know the person in the grass preserving the face is important. If I don't know them then it could be replaced by a stock face as well. Maybe the optimal compression of a crowd of people is the 2 faces of people I know preserved accurately and the rest replaced with "stock" faces.
But I do like the Stephen Wolfram idea of consciousness being the way a computationally bounded observer develops a coherent view of a branching universe.
This is related to compression because it a (lossy!) reduction in information.
I understand that Wolfram is controversial, but the information-transmission-centric view of reality he works with makes a lot of intuitive sense to me.
"Consciousness" is a pretty useless word without being very carefully defined, because people use it to mean a variety of different things. And often in the most ambiguous way possible such as this comment.
But also often some related but very specific and different things such as the reply that assumes it means only "self-awareness".
To me, the main purpose of the word is to prove the insufficiency of language and how imprecise most people's thinking is.
I can imagine some uses for this. Imagine having to archive a massive dataset where it’s unlikely any individual image will be retrieved and where perfect accuracy isn’t required.
I heard Stable Diffusion's model is just 4 GB. It's incredible that billions of images could be squeezed in just 4 GB. Sure it's lossy compression but still.
I don't think that thinking of it as "compression" is useful, and more than an artist recreating the Mona Lisa from memory is "decompressing" it. The process that diffusion models use is fundamentally different to decompression.
For example, if you prompt Stable Diffusion with "Mona Lisa" and look at the iterations, it is clearer what is happening - it's not decompressing so much as drawing something it knows looks like Mona Lisa and then iterating to make it look clearer and clearer.
It clearly "knows" what the Mona Lisa looks like, but what is is doing isn't copying it - it's more like recreating a thing that looks like it.
(And yes I realize lots of artist on Twitter are complaining that it is copying their work. I think "forgery" is a better analogy than "stealing" though - it can create art that looks like a Picasso or whatever, but it isn't copying it in a conventional sense)
I think it's easy to explain. If we split all those images into small 8x8 chunks, and put all the chunks into a fuzzy and a bit lossy hashtable, we'll see that many chunks are very similar and can be merged into one. To address this "space of 8x8 chunks" we'll apply PCA to them, just like in jpeg, and use only the top most significant components of the PCA vectors.
So in essense, this SD model is like an Alexandria library of visual elements, arranged on multidomensional shelves.
Before I clicked through to the article, I thought maybe they were taking an image and spitting out a prompt that would produce an image substantially similar to the original.
Something interesting about the San Francisco test image is that if you start to look into the details, its clear that some real changes have been made to the city. Rather than losing texture or grain or clarity, the information lost in this is information about the particular layout of a neighborhood of streets, which has now been replaced as if some one were drawing the scene from memory. A very different kind of loss that with out the original might be imperceptible because the information that was lost isn't replaced with random or systematic noise, but rather new, structured information..
There was a scandal when it was discovered that Xerox machines were doing this; in that case, the example showed "photocopies" replacing numbers in documents with other numbers.
During my PhD this issue came up amongst those in the group looking into compressed sensing in MRI. Many reconstruction methods (AI being a modern variant) work well because a best guess is visually plausible. These kinds of methods fall apart when visually plausible and "true" are different in a meaningful way. The simplest examples here being the numbers in scanned documents, or in the MRI case, areas of the brain where "normal brain tissue" was on average more plausible than "tumor".
If you are making an image of a cityscape to illustrate an article it probably doesn't matter what the city looks like. But if the article is about the architecture of the specific city, it probably does, so you need to 'be aware' that the image you are showing people isn't correct, and reduce the compression.
> The right amount of compression in a photocopy machine is zero.
This isn't an obvious statement to me. If you've had the misfortune of scanning documents to PDF and getting the 100MB per page files automatically emailed to you then you might see the benefit in all that white space being compressed somehow.
> But what does it mean to “be aware of” compression that may give you a crisp image of some made up document?
This isn't something I said. A good compression system for documents will not change characters in any circumstances.
> not the complete showstoppers some people seem to think that they are.
idk if I had to second guess every single result coming out of a machine it would be a showstopper for me. This isn't pokemon go, tumor detection is serious matter
Why would you want to lossily compress any medical image is beyond me. You get equipment to make precise high-resolution measurements, it goes without saying that you do not want noise added to that.
In medical images, you don't record first and then compress later. Instead, you make sparse measurements and then reconstruct. Why? Because people move, so getting more frames/sec is a thing; you don't want people to stay for too long in the machine; and (ideally) with the same setup, you can focus on a smaller area and get a higher resolution than standard measurements too.
You are talking about compressed sensing which is not lossy compression (compressed sensing can be lossless unless you're dealing with noisy measurements).
But say you're doing noisy measurements, and you are under-measuring like you say, and you have to fabricate non-random non-homogenous reconstruction noise. In that case it would be a very good idea to produce, as they do for lossy compression, both the standard overall bit rate vs. PSNR characterization against alternate direct (non-sparse) measurement ground truths (that have to exist, or else the reconstruction method should be called into question), and the bit rate for each particular sparse measurement. So this way people can see how reliable the reconstruction is. Ideally the image should be labeled at the pixel level with reconstruction probabilities, or presented in other ways to demonstrate the ratio of measured vs. fabricated information, like 95% confidence-interval extremal reconstructions or something.
It's not clear that community is doing this level of due diligence, so then the voices here are right: it's not a good idea to use.
If the compression is lossless that's fine. I have not seen an AI system being used in this manner but I don't doubt it's possible. All lossy compression methods output false information, that's the point of lossy compression and why it works so well. Remove details that the compression algorithm deems unimportant.
One thing that worries me about generative AI is the degradation of “truth” over time. AI will be the cheapest way to generated content, by far. It will sometimes get facts subtly wrong, and eventually that AI generated content will be used to train future models. Rinse and repeat.
The nice thing about math is that often it's much harder to find a proof than to verify that proof. So math AI is allowed to make lots of dumb mistakes, we just want it to make the occasional real finding too.
We are getting closer and closer to a simulacrum and hyperreality.
We used to create things that were trying to simulate (reproduce) reality, but now we are using those "simulations" we'd created as if they were the real thing. With time we will be getting farther away from the "truth" (as you put it), and yes - I share your worry about that.
EDIT: A good example I heard that explains what a simulacrum is was this:
Ask a random person to draw a photo of a princes and see how many will draw a disney princess (which already was based on real princesses) vs how many will draw one looking like Catherine of Aragon or another real princess.
Yes indeed. I've been looking for an auto summarizer that reliably doesn't change the content. So far everything I've tried will make up or edit a key fact once in a while.
Anywhere that truth matters will be unaffected. If such deviations from truth can withhold, then the truth never mattered. False assumptions will never hold where they can't, because reality is quite pervasive. Ask anyone who's had to productionize an ML model in a setting that requires a foot in reality. Even a single-digit drop in accuracy can have resounding effects.
Certainly possible, though we also have many hundreds of millions of people walking the globe taking pictures of things with their phones (not all of which are public to be used for training, but still).
The interesting thing is that is some ways this is a return to pre-modern era of lossy information transmission between the generations. Every story is re-molded by the re-teller. Languages change and thus the contextual interpretations. Even something a seemingly static as a book gets slowly modified as scribes rewrite scrolls over centuries.
True, but I'd like to continue using products that produce close-to-real images. Phones nowadays already process images at lot. The moment they start replacing pixels it'll all be fake.
And… Some manufacturer apparently already did it on their ultra zoom phones when taking photos of the moon.
Meh. Cameras have been "replacing pixels" for as long as I've been alive. Consider that a 4K camera only has 2k*4k pixels whereas a 4K screen has 2k*4k*3 subpixels.
2/3 of the image is just dreamed up by the ISP (image signal processor) when it debayers the raw image.
I'm not aware of any consumer hardware that has open source ISP firmware or claims to optimize for accuracy over beauty.
“When used in lossy mode, JBIG2 compression can potentially alter text in a way that's not discernible as corruption. This is in contrast to some other algorithms, which simply degrade into a blur, making the compression artifacts obvious.[14] Since JBIG2 tries to match up similar-looking symbols, the numbers "6" and "8" may get replaced, for example.
In 2013, various substitutions (including replacing "6" with "8") were reported to happen on many Xerox Workcentre photocopier and printer machines, where numbers printed on scanned (but not OCR-ed) documents could have potentially been altered. This has been demonstrated on construction blueprints and some tables of numbers; the potential impact of such substitution errors in documents such as medical prescriptions was briefly mentioned.”
Invisibly changing the content rather than the image quality seems like a really concerning failure mode for image compression!
I wonder if it'd be possible to use SD as part of a lossless system - use SD as something that tells us the liklihood of various pixel values given the rest of the image and combine that liklihood with a huffman encoding. Either way, fantastic hack, but we really should avoid using anything lossy built on AI for image compression.
Imagine a world where bandwidth constraints meant transmitting a hidden compressed representation that gets expanded locally by smart TVs that have pretrained weights baked into the OS. Everyone sees a slightly different reconstitution of the same input video. Firmware updates that push new weights to your TV result in stochastic changes to a movie you've watched before.
"The weather forecast was correct as broadcast, sir, it's just your smart TV thought it was more likely that the weather in your region would be warm on that day, so it adjusted the symbol and temperature accordingly"
You could still use some kind of adaptive huffman coding. Current compression schemes have some kind of dictionary embedded in the file to map between the common strings and the compressed representation. Google tried proposing SDCH a few years using a common dictionary for wep pages. There isn't any reason why we can't be a bit more deterministic and share a much larger latent representation of "human visual comprehension" or whatever to do the same. It doesn't need to be stochastic once generated.
I mean, that is already happening. Almost all modern TV's do some signal processing before outputting the pixels, and the image looks slightly different on each model.
But it'd definitely be cool to have some latent representation of a video that then gets rendered on tv - you could apply latent style sheets to the content, like what actors you want to play the roles, or turn everything into a steam-punk anime on the fly. The more abstract the representation, the more interesting alterations you could apply
I would've thought anyone relying on lossy-compressed images of any sort already needs to be aware of the potential effects, or otherwise isn't really concerned by the effect on the image (and I'd guess that the vast majority of use cases actually don't care if parts of the image are essentially "imaginary")
The basic premise of these kinds of compression algorithms is actually pretty clever. Here's a very very trivialization of this style of approach:
1. both the compressor and decompressor contain knowledge beyond the algorithm used to compress/decompress some data
2. in this case the knowledge might be "all the images in the world"
3. when presented with an image, the compressor simply looks up some index or identifier of the the image
4. the identifier is passed around as the "compressed image"
5. "decompression" means looking up the identifier and retrieving the image
I've heard this called "compression via database" before and it can give the appearance of defeating Shannon theorem for compression even though it doesn't do that at all.
Of course the author's idea is significantly more sophisticated than the approach above, and trades a lossy approach for some gains in storage and retrieval efficiency (we don't have to have a copy of all of the pictures in the world in both the compressor and the decompressor). The evaluation note of not using any known image for the tests further challenges the approach and helps sus-out where there are specific challenge like poor reconstruction of specific image constructs like faces or text -- I suspect that there are many other issues like these but the author honed in on these because we (as literate humans) are particularly sensitive to them.
In these types of lossy compression approaches (as opposed to the above which is lossless) the basic approach is:
1. Throw away data until you get to the desired file size. You usually want to come up with some clever scheme to decide what data you toss out. Alternative, just hash the input data using some hash function that produces just the right number of bits you want, but use a scheme that results in a hash digest that can act as a (non-unique) index to the original image in a table of every image in the world.
2. For images it's usually easy to eliminate pixels (resolution) and color (bit-depth, channels, etc.). In this specific case, the author uses an variational autoencoder to "choose" what gets tossed. I suspect the autoencoder is very good at preserving information rich, or high-entropy, information dense slices of a latent space or something. At any rate, this produces something that to us sorta kinda looks like a very low resolution, poorly colored postage stamp of the original image, but actually contains more data than that. I think at this point it can just be considered the hash digest.
3. this hash digest, or VAE encoded image or whatever we want to call it, is what's passed around as the "compressed" data.
4. just like above, "decompression" means effectively looking up the value in a "database". If we are working with hash digests, there was probably a collision during the construction of the database of all images, so we lost some information. In this case we're dealing with stable diffusion and instead of a simple index->table entry, our "compressed" VAE image wraps through some hyperspace to find the nearest preserved data. Since the VAE "pixels" probably align close to data dense areas of the space you tend to get back data that closely represents the original image. It's still a database lookup in that sense, but it's looking more for "similar" rather than "exact matches" which when used to rebuild the image give a good approximation of the original.
Because it's an "approximation" it's "lossy". In fact I think it'd be more accurate to say it's "generally lossy" as there is a chance the original image can be reproduced exactly, especially if it's in the original training data. Which is why the author was careful not to use anything from that set.
Because we've stored so much information in the compressor and decompressor, it can also give ...
This is not really "stable-diffusion based image compression", since it only uses the VAE part of "stable diffusion", and not the denoising UNet.
Technically, this is simply "VAE-based image compression" (that uses stable diffusion v1.4's pretrained variational autoencoder) that takes the VAE representations and quantizes them.
(Note: not saying this is not interesting or useful; just that it's not what it says on the label)
Using the "denoising UNet" would make the method more computationally expensive, but probably even better (e.g., you can quantize the internal VAE representations more aggressively, since the denoising step might be able to recover the original data anyway).
It does use the UNet to denoise the VAE compressed image:
"The dithering of the palettized latents has introduced noise, which distorts the decoded result. But since Stable Diffusion is based on de-noising of latents, we can use the U-Net to remove the noise introduced by the dithering."
The included Colab doesn't have line numbers, but you can see the code doing it:
# Use Stable Diffusion U-Net to de-noise the dithered latents
latents = denoise(latents)
denoised_img = to_img(latents)
display(denoised_img)
del latents
print('VAE decoding of de-noised dithered 8-bit latents')
print('size: {}b = {}kB'.format(sd_bytes, sd_bytes/1024.0))
print_metrics(gt_img, denoised_img)
What they do is essentially a fractal compression with an external library of patterns (that was IIRC pattented but the patent should be long expired).
2. It would be great to see the best codecs included in the comparison - AVIF and JPEG XL. Without those it's rather incomplete. No surprise that JPEG and WEBP totally fall apart at that bitrate.
3. A significant limitation of the approach seems to be that it targets extremely low bitrates where other codecs fall apart, but at these bitrates it incurs problems of its own (artifacts take the form of meaningful changes to the source image instead of blur or blocking, very high computational complexity for the decoder).
When only moderate compression is needed, codecs like JPEG XL already achieve very good results. This proof of concept focuses on the extreme case, but I wonder what would happen if you targeted much higher bitrates, say 5x higher than used here. I suspect (but have no evidence) that JPEG XL would improve in fidelity faster as you gave it more bits than this SD-based technique. Transparent compression, where the eye can't tell a visual difference between source and transcode (at least without zooming in) is the optimal case for JPEG XL. I wonder what sort of bitrate you'd need to provide that kind of guarantee with this technique.
The comparison doesn't make much sense because for fair comparisons you have to measure decompressor size plus encoded image size. The decompressor here is super huge because it includes the whole AI model. Also, everyone needs to have the exact same copy of the model in the decompressor for it to work reliably.
Only if decompressor and image are transmitted over the same channel at the same time, and you only have a small number of images. When compressing images for the web I don't care if a webp decompressor is smaller than a jpg or png decompressor, because the recipient already has all of those.
Of course stable diffusion's 4GB is much more extreme than Brotli's 120kb dictionary size, and would bloat a Browser's install size substantially. But for someone like Instagram or a Camera maker it could still make sense. Or imagine phones having the dictionary shipped in the OS to save just a couple kB on bad data connections.
Even if dictionaries were shipped, the biggest difficulty would be performance and resources. Most of these models require beefy compute and a large amount of VRAM that isn't likely to ever exist on end devices.
Unless that can be resolved it just doesn't make sense to use it as a (de)compressor.
The vae used in stable diffusion is not ideal for compression. I think it would be better to use the vector-quantized variant (by the same authors of latent diffusion) instead of the KL variant, then store the indexes for each quantized vector using standard entropy coding algorithms.
From the paper the VQ variant also performs better overall, SD may have chosen the KL variant only to lower vram use.
just checked the paper again and yes you're right, the KL version is better on the openimages dataset. The VQ version is better in the inpainting comparison.
In this case you'd still want to use the VQ version though, it doesn't make sense to do an 8bit quantization on the KL vectors when there's an existing quantization learned through training.
While this is great as an experiment, before you jump into practical applications, it is worth remembering that the decompressor is roughly 5GB in size :-)
This is why for compression tests, they incorporate the size of everything needed to decompress the file. You can compress down to 4.97KB all you want, just include the 4GB trained model.
Do you also include the library to render a jpeg? And maybe the whole OS required to display it on your screen?
There are very many uses where any fixed overhead is meaningless. Imagine archiving billions of images for long term storage. The 4GB model quickly becomes meaningless.
Yes, but each image needs access to this 4GB (actually, I have no idea how much RAM it takes up), plus whatever the working set size is. It is a non-trivial overhead that really limits throughput of your system, so you can process less images in parallel, so compressing billion of images in reasonable time suddenly may cost much more than the amount of storage it would save, compared to other methods.
Is that true? I have never seen this done for any image compression comparisons that I have seen (i.e. only data that is specific to the image that is being compressed is included, not standard tables that are always used by the algorithm like the quantisation tables used in JPG compression)
So a compressor of a few gigabyte would make sense if you would have a set of pictures of more then a few gigabyte. It's a bit similar to preprocessing text compression with a dictionary and adding the dictionary to the extractor to squeeze a bit more bytes.
However, several people here are conflating "best compression as determined for a competition" and "best compression for use in the real world". There is an important relationship between them, absolutely, but in the real world we do not download custom decoders for every bit of compressed content. Just because there is a competition that quite correctly measures the entire size of the decompressor and encoded content does not mean that is now the only valid metric to measure decompression performance. The competitions use that metric for good and valid reasons, but those good and valid reasons are only vaguely correlated to the issues faced in the normal world.
(Among the reasons why competitions must include the size of the decoder is that without that the answer is trivial; I define all your test inputs as a simple enumeration of them and my decoder hard-codes the output as the test values. This is trivially the optimal algorithm, making competition useless. If you could have a real-world encoder that worked this well, and had the storage to implement it, it would be optimal, but you can't possibly store all possible messages. For a humorous demonstration of this encoding method, see the classic joke: https://onemansblog.com/2010/05/18/prison-joke/ )
In theory, it would be possible to benefit from the ability of Stable Diffusion to increase perceived image quality without even using a new compression format. We could just enhance existing JPG images in the browser.
There already are client side algorithms that increase the quality of JPGs a lot. For some reason, they are not used in browsers yet.
A Stable Diffusion based enhancement would probably be much nicer in most cases.
There might be an interesting race to do client side image enhancements coming to the browsers over the next years.
This but for video using the "infilling" version for changing parts between frames.
The structural changes per frame matter much less. Send a 5kB image every keyframe then bytes per subsequent image with a sketch of the changes and where to mask them on the frame.
Modern video codecs are pretty amazing though, so not sure how it would compare in frame size
I've been thinking about more or less the same idea, but the computational edge inference costs probably makes it impractical for most of today's client devices. I see a lot of potential in this direction in the near future though.
I think it's unclear how much computational resources the uncompression steps take.
At the moment it's fairly fast, but RAM hungry. But this article makes it clear that quantizing the representation works well (at least for the VAE). It's possible quantized models could also do decent jobs.
It is really interesting to talk about semantic lossy compression, which is probably what we get.
Where recreating with traditional codices introduce syntactic noise, then this will introduce semantic noise.
Imagine seeing a high res perfect picture, just until you see the source image and discover that it was reinterpreted..
It is also going to be interesting, to see if this method will be chosen for specific pictures, eg. pictures of celebrity objects (or people, when/if issues around that resolve), but for novel things, we need to use "syntactical" compression.
201 comments
[ 2.0 ms ] story [ 234 ms ] threadI think it would be very interesting to determine if these images do come back with notably better compression.
There's something to be said about compression algorithms being predictable, deterministic, and only capable of introducing defects that stand out as compression artifacts.
Plus, decoding performance and power consumption matters, especially on mobile devices (which also happens be the setting where bandwidth gains are most meaningful).
It's pretty easy to add this if you wanted to.
But a better method would be to fine tune on a bunch of machine-generated images of words if you want your model to be good at generating characters. You'll need to consider which of the many Unicode character sets you want your model to specialize in though.
I do tend to use the HuggingFace version though.
The optimal lossy compression algorithm would be based on humans as a target. it would remove details that we wouldn't notice to reduce the target size. If you show me a photo of a face in front of some grass the optimal solution would likely be to reproduce that face in high detail but replace the grass with "stock imagery".
I guess it comes down to what is important. In the past algorithms were focused on visual perception, but maybe we are getting so good at convincingly removing unnecessary detail that we need to spend more time teaching the compressor what details are important. For example if I know the person in the grass preserving the face is important. If I don't know them then it could be replaced by a stock face as well. Maybe the optimal compression of a crowd of people is the 2 faces of people I know preserved accurately and the rest replaced with "stock" faces.
See e.g. the Hutter Prize.
Written language is human thought compressed into words
Digital images are light detection compressed into bits
Text to images AI compress digital images into written language
Then how do the AI weights relate to human thought?
Memory <> Compression <> Language <> Signal Strength <> Harmonics and Ratios
But I do like the Stephen Wolfram idea of consciousness being the way a computationally bounded observer develops a coherent view of a branching universe.
This is related to compression because it a (lossy!) reduction in information.
I understand that Wolfram is controversial, but the information-transmission-centric view of reality he works with makes a lot of intuitive sense to me.
https://writings.stephenwolfram.com/2021/03/what-is-consciou...
It’s not a well defined ontology yet. So whatever it is, at its irreducible size pinpointing it as a thing in which gives rise to such other things.
But also often some related but very specific and different things such as the reply that assumes it means only "self-awareness".
To me, the main purpose of the word is to prove the insufficiency of language and how imprecise most people's thinking is.
Could cut down storage costs a lot.
For example, if you prompt Stable Diffusion with "Mona Lisa" and look at the iterations, it is clearer what is happening - it's not decompressing so much as drawing something it knows looks like Mona Lisa and then iterating to make it look clearer and clearer.
It clearly "knows" what the Mona Lisa looks like, but what is is doing isn't copying it - it's more like recreating a thing that looks like it.
(And yes I realize lots of artist on Twitter are complaining that it is copying their work. I think "forgery" is a better analogy than "stealing" though - it can create art that looks like a Picasso or whatever, but it isn't copying it in a conventional sense)
I think using that language is better than "stealing", because the immoral act is the passing off, not training of the model.
So in essense, this SD model is like an Alexandria library of visual elements, arranged on multidomensional shelves.
But no, it's the real deal. Great job author.
During my PhD this issue came up amongst those in the group looking into compressed sensing in MRI. Many reconstruction methods (AI being a modern variant) work well because a best guess is visually plausible. These kinds of methods fall apart when visually plausible and "true" are different in a meaningful way. The simplest examples here being the numbers in scanned documents, or in the MRI case, areas of the brain where "normal brain tissue" was on average more plausible than "tumor".
[1]: http://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_...
Compression that gives you a blurred image is a trade-off.
But what does it mean to “be aware of” compression that may give you a crisp image of some made up document?
This isn't an obvious statement to me. If you've had the misfortune of scanning documents to PDF and getting the 100MB per page files automatically emailed to you then you might see the benefit in all that white space being compressed somehow.
> But what does it mean to “be aware of” compression that may give you a crisp image of some made up document?
This isn't something I said. A good compression system for documents will not change characters in any circumstances.
idk if I had to second guess every single result coming out of a machine it would be a showstopper for me. This isn't pokemon go, tumor detection is serious matter
But say you're doing noisy measurements, and you are under-measuring like you say, and you have to fabricate non-random non-homogenous reconstruction noise. In that case it would be a very good idea to produce, as they do for lossy compression, both the standard overall bit rate vs. PSNR characterization against alternate direct (non-sparse) measurement ground truths (that have to exist, or else the reconstruction method should be called into question), and the bit rate for each particular sparse measurement. So this way people can see how reliable the reconstruction is. Ideally the image should be labeled at the pixel level with reconstruction probabilities, or presented in other ways to demonstrate the ratio of measured vs. fabricated information, like 95% confidence-interval extremal reconstructions or something.
It's not clear that community is doing this level of due diligence, so then the voices here are right: it's not a good idea to use.
It's perfectly possible to build neural network based compression systems that do not output false information.
We used to create things that were trying to simulate (reproduce) reality, but now we are using those "simulations" we'd created as if they were the real thing. With time we will be getting farther away from the "truth" (as you put it), and yes - I share your worry about that.
https://en.wikipedia.org/wiki/Simulacrum
EDIT: A good example I heard that explains what a simulacrum is was this: Ask a random person to draw a photo of a princes and see how many will draw a disney princess (which already was based on real princesses) vs how many will draw one looking like Catherine of Aragon or another real princess.
And… Some manufacturer apparently already did it on their ultra zoom phones when taking photos of the moon.
2/3 of the image is just dreamed up by the ISP (image signal processor) when it debayers the raw image.
I'm not aware of any consumer hardware that has open source ISP firmware or claims to optimize for accuracy over beauty.
“When used in lossy mode, JBIG2 compression can potentially alter text in a way that's not discernible as corruption. This is in contrast to some other algorithms, which simply degrade into a blur, making the compression artifacts obvious.[14] Since JBIG2 tries to match up similar-looking symbols, the numbers "6" and "8" may get replaced, for example.
In 2013, various substitutions (including replacing "6" with "8") were reported to happen on many Xerox Workcentre photocopier and printer machines, where numbers printed on scanned (but not OCR-ed) documents could have potentially been altered. This has been demonstrated on construction blueprints and some tables of numbers; the potential impact of such substitution errors in documents such as medical prescriptions was briefly mentioned.”
https://en.m.wikipedia.org/wiki/JBIG2
Invisibly changing the content rather than the image quality seems like a really concerning failure mode for image compression!
I wonder if it'd be possible to use SD as part of a lossless system - use SD as something that tells us the liklihood of various pixel values given the rest of the image and combine that liklihood with a huffman encoding. Either way, fantastic hack, but we really should avoid using anything lossy built on AI for image compression.
But it'd definitely be cool to have some latent representation of a video that then gets rendered on tv - you could apply latent style sheets to the content, like what actors you want to play the roles, or turn everything into a steam-punk anime on the fly. The more abstract the representation, the more interesting alterations you could apply
IE, the algorithm ignores and loses the 'irrelevant' information, but holds the important stuff?
1. both the compressor and decompressor contain knowledge beyond the algorithm used to compress/decompress some data
2. in this case the knowledge might be "all the images in the world"
3. when presented with an image, the compressor simply looks up some index or identifier of the the image
4. the identifier is passed around as the "compressed image"
5. "decompression" means looking up the identifier and retrieving the image
I've heard this called "compression via database" before and it can give the appearance of defeating Shannon theorem for compression even though it doesn't do that at all.
Of course the author's idea is significantly more sophisticated than the approach above, and trades a lossy approach for some gains in storage and retrieval efficiency (we don't have to have a copy of all of the pictures in the world in both the compressor and the decompressor). The evaluation note of not using any known image for the tests further challenges the approach and helps sus-out where there are specific challenge like poor reconstruction of specific image constructs like faces or text -- I suspect that there are many other issues like these but the author honed in on these because we (as literate humans) are particularly sensitive to them.
In these types of lossy compression approaches (as opposed to the above which is lossless) the basic approach is:
1. Throw away data until you get to the desired file size. You usually want to come up with some clever scheme to decide what data you toss out. Alternative, just hash the input data using some hash function that produces just the right number of bits you want, but use a scheme that results in a hash digest that can act as a (non-unique) index to the original image in a table of every image in the world.
2. For images it's usually easy to eliminate pixels (resolution) and color (bit-depth, channels, etc.). In this specific case, the author uses an variational autoencoder to "choose" what gets tossed. I suspect the autoencoder is very good at preserving information rich, or high-entropy, information dense slices of a latent space or something. At any rate, this produces something that to us sorta kinda looks like a very low resolution, poorly colored postage stamp of the original image, but actually contains more data than that. I think at this point it can just be considered the hash digest.
3. this hash digest, or VAE encoded image or whatever we want to call it, is what's passed around as the "compressed" data.
4. just like above, "decompression" means effectively looking up the value in a "database". If we are working with hash digests, there was probably a collision during the construction of the database of all images, so we lost some information. In this case we're dealing with stable diffusion and instead of a simple index->table entry, our "compressed" VAE image wraps through some hyperspace to find the nearest preserved data. Since the VAE "pixels" probably align close to data dense areas of the space you tend to get back data that closely represents the original image. It's still a database lookup in that sense, but it's looking more for "similar" rather than "exact matches" which when used to rebuild the image give a good approximation of the original.
Because it's an "approximation" it's "lossy". In fact I think it'd be more accurate to say it's "generally lossy" as there is a chance the original image can be reproduced exactly, especially if it's in the original training data. Which is why the author was careful not to use anything from that set.
Because we've stored so much information in the compressor and decompressor, it can also give ...
Technically, this is simply "VAE-based image compression" (that uses stable diffusion v1.4's pretrained variational autoencoder) that takes the VAE representations and quantizes them.
(Note: not saying this is not interesting or useful; just that it's not what it says on the label)
Using the "denoising UNet" would make the method more computationally expensive, but probably even better (e.g., you can quantize the internal VAE representations more aggressively, since the denoising step might be able to recover the original data anyway).
"The dithering of the palettized latents has introduced noise, which distorts the decoded result. But since Stable Diffusion is based on de-noising of latents, we can use the U-Net to remove the noise introduced by the dithering."
The included Colab doesn't have line numbers, but you can see the code doing it:
[1] https://en.wikipedia.org/wiki/Fractal_compression
1. This is a brilliant hack. Kudos.
2. It would be great to see the best codecs included in the comparison - AVIF and JPEG XL. Without those it's rather incomplete. No surprise that JPEG and WEBP totally fall apart at that bitrate.
3. A significant limitation of the approach seems to be that it targets extremely low bitrates where other codecs fall apart, but at these bitrates it incurs problems of its own (artifacts take the form of meaningful changes to the source image instead of blur or blocking, very high computational complexity for the decoder).
When only moderate compression is needed, codecs like JPEG XL already achieve very good results. This proof of concept focuses on the extreme case, but I wonder what would happen if you targeted much higher bitrates, say 5x higher than used here. I suspect (but have no evidence) that JPEG XL would improve in fidelity faster as you gave it more bits than this SD-based technique. Transparent compression, where the eye can't tell a visual difference between source and transcode (at least without zooming in) is the optimal case for JPEG XL. I wonder what sort of bitrate you'd need to provide that kind of guarantee with this technique.
Of course stable diffusion's 4GB is much more extreme than Brotli's 120kb dictionary size, and would bloat a Browser's install size substantially. But for someone like Instagram or a Camera maker it could still make sense. Or imagine phones having the dictionary shipped in the OS to save just a couple kB on bad data connections.
Unless that can be resolved it just doesn't make sense to use it as a (de)compressor.
From the paper the VQ variant also performs better overall, SD may have chosen the KL variant only to lower vram use.
In this case you'd still want to use the VQ version though, it doesn't make sense to do an 8bit quantization on the KL vectors when there's an existing quantization learned through training.
There are very many uses where any fixed overhead is meaningless. Imagine archiving billions of images for long term storage. The 4GB model quickly becomes meaningless.
No, what does that have to do with reconstructing the original data?
If the fixed overhead works for you, that's fine, but including it is not meaningless.
Long term storage of billions of images is meaningless, if it takes billions of years to archive these images.
Matt doesn't do this on the Silesia corpus compression benchmark, even though it would make sense there as well: http://mattmahoney.net/dc/silesia.html
So a compressor of a few gigabyte would make sense if you would have a set of pictures of more then a few gigabyte. It's a bit similar to preprocessing text compression with a dictionary and adding the dictionary to the extractor to squeeze a bit more bytes.
However, several people here are conflating "best compression as determined for a competition" and "best compression for use in the real world". There is an important relationship between them, absolutely, but in the real world we do not download custom decoders for every bit of compressed content. Just because there is a competition that quite correctly measures the entire size of the decompressor and encoded content does not mean that is now the only valid metric to measure decompression performance. The competitions use that metric for good and valid reasons, but those good and valid reasons are only vaguely correlated to the issues faced in the normal world.
(Among the reasons why competitions must include the size of the decoder is that without that the answer is trivial; I define all your test inputs as a simple enumeration of them and my decoder hard-codes the output as the test values. This is trivially the optimal algorithm, making competition useless. If you could have a real-world encoder that worked this well, and had the storage to implement it, it would be optimal, but you can't possibly store all possible messages. For a humorous demonstration of this encoding method, see the classic joke: https://onemansblog.com/2010/05/18/prison-joke/ )
There already are client side algorithms that increase the quality of JPGs a lot. For some reason, they are not used in browsers yet.
A Stable Diffusion based enhancement would probably be much nicer in most cases.
There might be an interesting race to do client side image enhancements coming to the browsers over the next years.
The structural changes per frame matter much less. Send a 5kB image every keyframe then bytes per subsequent image with a sketch of the changes and where to mask them on the frame.
Modern video codecs are pretty amazing though, so not sure how it would compare in frame size
At the moment it's fairly fast, but RAM hungry. But this article makes it clear that quantizing the representation works well (at least for the VAE). It's possible quantized models could also do decent jobs.
Where recreating with traditional codices introduce syntactic noise, then this will introduce semantic noise.
Imagine seeing a high res perfect picture, just until you see the source image and discover that it was reinterpreted..
It is also going to be interesting, to see if this method will be chosen for specific pictures, eg. pictures of celebrity objects (or people, when/if issues around that resolve), but for novel things, we need to use "syntactical" compression.