HN Explorer

Ask HN: AI algorithms for HTTP Compression

3 points by AlexanderTheGr8 ↗ HN
Hello, I am working on some compression research. And I was wondering if new compression algorithms can improve overall web experience by improving HTTP compression.

Can someone with knowledge of this tell me if HTTP already supports zstd (default is brotli afaik). If it doesn't support zstd, why not? zstd offers pretty fast decompression which is useful (you only compress once, but you decompress millions of times).

I am working on an AI based compression alg which is very slow at compressing but extremely fast at decompressing. AI has huge scope in lossy compression (considering that AIs can guess pretty well what lossy content human beings can perceive). Any advice for me?

4 comments

[ 3.3 ms ] story [ 21.0 ms ] thread
mikewarot
Good compression is always helpful, another working algorithm to fill out the available spectrum of choices is a plus.

Why tie it to HTTP? Most transfers are https, which this wouldn't help.

gus_massa
Also, most transfered bytes are in images and videos, so compressing the text part of html/css/js will change the total number very little.
AlexanderTheGr8
That's true, but extremely good algorithms to compress text/image/video already exist. Devs just don't consider using them when putting them on their web servers. I was curious whether it is possible for the protocol to automatically compress such images/videos using SOTA algorithms such as zstd.
toast0
Yes, zstd is one of the officially registered content encodings https://www.iana.org/assignments/http-parameters/http-parame...

I don't know if any clients or servers support it, but it's registered.

It's not going to help with images or videos though. Well it might help with bitmaps, but not as much as an image compressor would. Anything that's already compressed isn't going to get smaller with a general purpose compression tool, unless the existing compression is really bad at its job. There are some purpose built compression tools that can recompress specific formats for smaller results, the one I can remember is dropbox's lepton https://dropbox.tech/infrastructure/lepton-image-compression...