From time to time I've run an analysis on the use of alternative image formats on the web and it never comes out to be worth it.
New formats are not supported on all browsers so you wind up having to support the mainstream formats as well as one or more new formats. Rather than benefiting from reduced storage costs, your storage costs get multiplied.
Performance is also a problem, particularly when people add "yet another polyfill" to get an image to work without native support.
Once I get into eyeballing images closely I also end up questioning if the compression gains are real. For instance, one I read a comment on the article here
and looked closely I couldn't unsee that the AV1 had very different artifacts than the other compressed images. Not necessarily worse, but definitely different. Unlike the poster I liked the way the suit rendered but definitely there was a lot of blocking around edges that the poster interprets as "pixelation on the flag".
Like many things, the costs of switching image formats are definite, but the benefits are questionable.
> From time to time I've run an analysis on the use of alternative image formats on the web and it never comes out to be worth it.
I just checked the Google homepage, and even the New Year's Eve Google Doodle currently being shown is a good ol' GIF, rather than a modern format such as WebP. And this is with me using a desktop Chrome browser that definitively supports WebP.
> Like many things, the costs of switching image formats are definite, but the benefits are questionable.
One easy way to get some of the benefits without incurring the switching costs is to leverage Cloudflare Polish[1]. I've used it at a few places, and have had good experiences. Although the majority of gains came from optimizing images that were poorly optimized to begin with, rather than the incremental improvements that occurred from enabling WebP support.
I don't like WebP because it's either it or HW accelerated browsing - latter I like a tad more because it saves me a bit of battery juice on my laptop.
There's the same problem with Google insisting on VP8/9 for YouTube in Chrome. If your Mac only supports h.264 in hardware you'll get better battery life in Safari.
Apple has a bigger share of the blame. They purposely ignore any codec that is outside of their favorites, even if the hardware they ship does support it (i.e. Intel) or doesn't need any hardware support (Vorbis, Flac). On the other hand, they are quick to support any codec, that is problematic to implement elsewhere.
> I just checked the Google homepage, and even the New Year's Eve Google Doodle currently being shown is a good ol' GIF, rather than a modern format such as WebP. And this is with me using a desktop Chrome browser that definitively supports WebP.
chroma subsampling, shallow gradients, ringing, the lack of transparency are all quite visible issues when trying to encode cartoons, pixel graphics or screen captures with the standard profiles of lossy video codecs.
In terms of compression, AV1 is a clear win. Note that artifacts that bother you are at about half the file size. You can increase the quality and still have an image that is both nicer and smaller than JPEG and WebP.
However, as TFA mentions, better compression is currently a double-edged sword, because the CPU has become a bottleneck again: even top-of-the-line Android phones have weak Qualcomm CPUs, and sites use so much JavaScript that even high-end machines can struggle.
So we may end up living with JPEG for a few more years. This format has been designed when CPUs had 25Mhz, and we've had over 25 years to optimize the implementations, so we get a lot of bang for the buck from it.
I hate that most websites are so heavy. I enjoyed browsing on my Kobo eReader but loading times of several seconds to a minutes makes it cumbersome. I limit my browsing to "lite" websites and HN suits that well. I also use the Pocket service because it's builtin and reduce the computational need required to read an article.
Most of which has nothing to do with compression format, to be honest. No image compression format is going to save us from websites that use 2000x2000 sized photos for 200x200 author thumbnails.
As far as I can tell, nobody has ever made it easy to fix this problem. I don't believe there is any webpack (etc.) plugin that will just recompress images for you. You have to do it manually. As a result, people don't do it, even those that have a proper "build process" for their website.
This has been a solved problem for years in some frameworks, e.g. all the major image upload handling libraries in Rails have this built in an it's very easy to use. There are also plenty of webpack plugins for static assets:
https://github.com/tcoopman/image-webpack-loader exists. The plugin will recompress png, jpeg, and svg images, as well as optionally converting the first two to webp. That said, it seems to have a persistent bug where jpegs are always converted to webp while retaining their original extention, and the maintainer doesn't seem to have the time to fix it or merge pull requests. If you wanted to help them out with that, they'd likely be amenable.
Something that I've not seen is a plugin/loader that will automatically use the <picture> tag to make available multiple auto-generated source images and a fallback <img>. It'd be great to be able to
Flif gives you a lower resolution image of you download only part of the file. Keep downloading and the image gets better resolution. If browsers supported it, they could have settings so you download partial images when roaming or the full file when on wifi.
> No image compression format is going to save us from websites that use 2000x2000 sized photos for 200x200 author thumbnails.
This sounded like an obvious truth but I think you're wrong!
JPEG has progressive rendering, so a browser could easily decide to stop once it has rendered enough for the image size. Once you zoom or if it's used elsewhere, it can render a little more.
Technically no, but JPEG divides the image into 8x8 segments (does that vary by channel? Can’t remember) so using full resolution images as thumbnails will still suck everyone’s bandwidth like correctly resized .bmp files
That's the point of progressive rendering, it renders less and less blurry the more the image loads. On slow connections with big images, I've seen the effect in the wild.
For a site with limited assets, people care more about bandwidth costs than storage costs and ideally your webserver can automatically transcode and serve the best format based on content negotiation.
Except for JPEG2000, which Safari supports but does not include in its Accepts: header. So Akamai just sniffs the user-agent and serves JPEG2000 to anything that looks like Safari, whether they support it or not.
Uuh, i thought the whole point of a static server was to not need an application that needs to prepare the content?
It's supposed to be a dumb web server that accesses a folder structure or similar storage backed...
Is that the point, though? I think the point is more that you tell the webserver "here is some content, I want you to serve it in the best way possible" and it figures out how to do it without any special configuration. In the same way I'd expect a "dumb" webserver to be able to negotiate gzip compression without having to point it at a directory of gzipped files.
Edit: I agree that something like the pagespeed module rewriting html probably goes too far and html rewriting is fraught with weird edge cases, but content negotiation is something only a webserver (or CDN) can do.
A directory of gzipped files is often incredibly useful, I've managed to reduce both latency and time it takes to load pages by just precompressing resources. It's more than incredibly useful on low-powered hardware like Raspberry Pis, some applications can be made to load in 3 seconds instead of 10.
That’s somewhat surprising that it makes that much of a difference, a Raspberry Pi 2 can gzip compress at around 20MB/s. Maybe there’s some inefficiencies in the webserver implementation, or just really big files?
I work with video codecs a lot; to me the blocking on the AV1 picture looks like chroma subsampling rather than a compression artifact. It's much more noticeable on red than in other colors. Unfortunately, at least as of last time I checked, the AV1 still image converters were mostly only 4:2:0 and it's quite noticeable on some images. But this should go away entirely once they're used without subsampling. In other words: don't use that part of the picture to judge the quality of AV1.
It's definitely chroma upsampling artifacts. This is a problem with the decoder, not the encoder. The chroma channels are upscaled with nearest-neighbor filtering. I resampled them in ImageMagick with a Mitchell-Netravali cubic filter, which is reasonably cheap to calculate, and it looks much better:
This is best done at the edge/CDN layer now. Many of them support real-time optimization (along with resizing, etc) to send the best version to the client that's requesting it. Also recommend Cloudinary and Filestack for a completely managed assets system.
> New formats are not supported on all browsers so you wind up having to support the mainstream formats as well as one or more new formats. Rather than benefiting from reduced storage costs, your storage costs get multiplied.
There's a big thing that bugs me about that when it comes to the web: Software patents and licenses basically ruined wavelet-based image compression schemes on the web.
Wavelets would have been ideal. Basically, the lossy compression would work about the same as the cosine transform in JPEG, but by using the discrete wavelet transform, you basically get thumbnailing built in: Imagine transmitting a tiny little thumbnail of your image. Then, you transmit the information used to derive a version 4x that size (2x width, 2x height) from that. Then, you transmit the data used to derive the next 4x layer. Rinse, repeat -- that's exactly how image data is organized after a wavelet transform.
With some clever adaptions, this could have done away with so much cruft and complexity we have today (like image sets and @media queries for background images): Imagine just putting a 4K version of your background on the server and telling the browser how to render it (an <img> tag with a specified size, or background-size in css). The browser could then start loading the image until it realizes it has all the data for your resolution (say on your phone), and just terminate the transmission before the whole image is downloaded.
JPEG 2000 uses wavelet, and a few other I can't remember their names, none of them has better image / pref than the best ones on the market or soon to be released image format based on X Codec. So in reality much like Wavelet in video, it hasn't worked out well at all in real world implementation.
Jpeg was released the MPEG-1 ( H.261 ) era. Since then we have had H.262 / MPEG2, H.263 / Divx, H.264 /AVC, H.265 / HEVC, and the coming H.266 / VVC. If we consider the amount of improvement we got from Video Encoding, images hasn't changed much at all, we happen to fine tune the heck out of Jpeg. And I hardly call a new image format with 50% reduction bitrate compared to a nearly 30 years old tech a major breakthrough, and that is without mentioning the huge decoding resources requirement.
There are many project that just improves on JPEG, Dropbox had a lossless compressor for JPEG, and Cant remember if Pik was originally based on jpeg as well.
Unless there is a real major breakthrough that gives the current best JPG encoder quality at 25% or less its size, focusing on 200KB, 100KB, 50KB images size, ( i.e the new image will have to be 50KB, 25KB, 12.5KB ) while using within 2x decoding power. I am convinced our roadmap for Network bandwidth improvement in the next 3 - 5 years are going to make JPEG to last forever.
They don't have to be; Microsoft could maintain that as a separate fork/add-in to Blink. In theory it sounds like it should be highly modular, so that shouldn't be a huge ongoing maintenance burden.
I don't know why they wouldn't just switch over to WebP in preference to that, though. It's better to have fewer of these formats around and the simplest thing to do is use the one that already has support.
It’s great to see a summary at the top of the article.
However, the summary (and the article) leaves out a relevant question that immediately comes to mind: Are other image formats decoded off the main thread or is it just that decoding JPEG-XR is just more taxing compared to other formats, thus negating its advantages?
I'm fairly certain the article may be poorly worded. Certainly standard JPGs are decoded on the CPU side as well, are they not?
To the best of my knowledge, I don't think any browsers are doing JPG decoding in hardware; I think they use the GPU for primitives and compositing, etc.
I would look forward to being corrected if I'm wrong!
The CPU portion of JPEG decoding is relatively cheap (just Huffman + RLE) and easy to parallelize (DCT blocks are independent). Conversion and upsampling of YCbCr can be done on the fly on the GPU, and this even saves memory compared to using raw RGB/RGBA bitmaps.
Other formats use more expensive entropy coding, more block sizes, smoothing filters, and inter-block prediction which forces them to be decoded mostly serially.
Isn't the JPEG instruction set built into the CPUs now anyways? I would be shocked to find out they are not seeing as MPEG2 has been part of the CPU since the Pentium days.
There is no "JPEG instruction set" as such --- and what you may be referring to by MPEG2 is MMX, which is just a SIMD extension that certainly helps with JPEG and MPEG decoding, among other things.
Interesting. I just remember the days of the add-on hardware boards for real-time MPEG2 encoding for DVDs that were pretty much obsoleted when some sort of upgrade to the CPU was made. I officially went 100% software/CPU encoding in 2005. However, it looks like something more specific from Intel called Quick Sync came out in 2011.[0] This is a dedicated core on the die. Do browsers get access to that level of hardware?
Hardly surprising. JPEG is a very mature technology with deep support and optimizations going all the way to hardware support built into many (most?) cpus. If the real world savings are ~ 20%, it probably won't be worth it.
The author says that JPEG-XRs are decoded on "the software-side on the CPU" a couple of times that it seems like it might be a term of art but it sounds very awkward to me.
Is the TLDR actually "jpeg xr implementation is poorly optimized and negatively affects browser performance which decreased conversions."
If the compression is really much better than with JPEG, there is probably a point where the connection is so slow, that JPEG-XR would be faster than JPEG, because the file-size would out-weight the decoding performance.
This piece gives a hint where Trivago as a company went with Holacracy. The lack of focus is so obvious and seems to let individual developers create some work which is not just poorly worded but also nonsense:
Why should anyone test a format which is used by just two browsers with a very small market share and which will be never adopted by the company with the biggest market share (because Google has WebP). JPEG-XR is DOA and nobody should care and waste time on this (except the OP gone astray).
I like that there is a tl;dr right at the beginning though.
I think that’s unfair though. It’s not necessarily about percentages; potentially they could easily be seeing a few tens of millions of monthly visits from IE and Edge, and if putting in place an automated system to convert images to a (theoretically) more efficient system had increased conversions they’d be really happy for the payback.
Thinking about it, assuming they’ve got a WebP output for Chrome already this could be as simple as adding one more build step with a different encoder, and one more negotiation rule on the CDN.
I'm not sure I'd be quite so blunt, but I do agree with your argument in the main: this is so low on the list of priorities that I can't help but wonder if someone at Trivago thinks they've run out of any actual revenue-generating development work.
It indicates to me that either the inmates are running the asylum - that devs are being given too much leeway to pursue personal interests - or (and?) that Trivago's day to day work is proving so dull that developers in some teams are inventing R&D work just to stay sane.
Yeah my tone was a bit to harsh and I didn't want to insult anyone but I am still wondering that nobody at Tribago gave him this feedback. However and hopwfully this useless piece of work leads him to some other more relevant stuff.
> -1% negative impact on core business metrics including conversions in our test of JPEG-XRs [...] Let’s not use JPEG-XR on the Web.
Is this an overreaction, or a subversive call to improve the decode perf of JPEG-XR? I'm certainly on the performance-is-a-feature bandwagon myself, but 1% ux impact is pretty small. And shouldn't we assume that decode perf will improve in the near future? Is there any reason to assume that this problem is permanent and unfixable?
The best you can say here is that it’s within the margin of error, in which case the effect is so small it’s literally unmeasurable. In that case the null hypothesis still wins, and you should not switch.
> Is there any reason to assume that this problem is permanent and unfixable?
IE11 as tested in the article is not getting anything other than security patches. The users who use IE11 are probably also the users most affected by poor performance since they most likely have the oldest hardware.
Interesting response, thanks. Your comment gives me even more questions, though!
> The best you can say here is that it’s within the margin of error
What is preventing JPEG-XR from becoming faster and a net ux positive, tomorrow or in the future? Isn't the best I can say that today's results might be a bug or fluke or a quick and dirty implementation, and not a permanent problem?
> IE11 as tested in the article is not getting anything other than security patches.
The title and conclusion both said we should avoid JPEG-XR on the web, not that we should avoid it on IE11.
You're right, the article focused on the problem with IE11. Is it already a net positive on Edge? (The article didn't say.) Should the shim to make it work in the old browser that fewer people use really override the potential benefits on the newer browser that more people are already using?
Does it seem reasonable or unreasonable to make future web decisions depend on an old browser that's on life support today and has small and declining market share?
It's a 1% impact to conversion rate which, if you have over $1B in annual revenue like Trivago, is quite a large amount of money.
And even if you don't, taking a 1% hit to your conversion rate and hurting user experience just because you want to use a fancy new image format is probably not a good idea.
Agreed, with all of that. It doesn’t answer my question, though, right? If the IE11 problem is temporary, and the IE11 perf gets improved next week in response to this article, then Trivago might have to do a 180, no? Even if we wait a while, IE11 usage will dwindle. Couldn’t the mere passage of time easily turn the story of JPEG-XR into a 1% or more net positive for Trivago, especially if the format is already a win on Edge? Every new technology fails to stack up to decades old standards by some metrics, yet we improve on and build new things anyway because we can see the future potential. This article isn’t acknowledging future potential. My question is what justification is there for declaring the format permanently unusable?
Without a breakdown of the stats and methodology, we have no way to know if this result was statistically meaningful, nor whether Trivago ran any null hypothesis tests. We don't even know that there were no confounding factors - just none the author thought of.
Except that the author claims it's a statistically significant result immediately before reporting the 1% number. The full phrase (which was truncated in the other comment) is:
"we were able to statistically verify a -1% negative impact on core business metrics including conversions"
Even if so: the relationship between page conversion and revenue is not direct. You don't know whether the lost leads were high quality, nor whether they were reacquired through other means.
Ultimately though the issue is more fundamental: you can't make assertions about performance without publishing actual figures. All we have to go on is the author's assertion that the test methodology was sound and the results were out of the bounds of chance. That's not really enough.
I have worked with many, many teams who earnestly believed they were running "scientific" MVTs, with solid statistical analysis free of bias. They weren't.
This is a little bit wrong in that the other CPU-bound tasks during loading are generally in the main thread, whereas image decoding is usually on a different thread (so, depending on how many images you have, and how many CPUs, it may not increase overall page load time in reality).
Now, overall time to decode still matters. If it's a gigantic hero image above the fold, then it could take a noticeable amount of time to decode, but it's probably still a win.
With the only good JPEG 2000 encoder, its quality at small file sizes is easily better than webp, but in practice it seems virtually nobody uses JPEG 2000, because the open source encoders are not really that great (probably because the patents are a guessing game, although technically the known holders are willing to grant royalty-free licenses).
Going a little further than just JPEG-XR - definitely do performance tests for new releases. We had a bug that affected one page in our app. It made people cringe using the feature set that invoked it, but essentially something in angular was firing, and the thing it did caused a re-fire. It just kept looping when on these very specific pages in the app. I finally found it and killed it. But definitely don't go to production with a client CPU murdering bug. It can't be measured the same as network latency, etc... so you really need to use your apps.
I was surprised to see a post from Trivago - I might have to book my next hotel through them.
Exotic file formats, like exotic JavaScript language constructs, tend to perform poorly because they are not optimisation targets. And they are not optimisation targets because - obviously - no-one uses them. To my knowledge there is no particular intrinsic reason that JPEG-XR decoding cannot be moved off-thread.
The solution to this problem is to start using these new technologies, so the browser vendors can prioritise work to make them performant. It is not to write articles that entrench that neglect and tie us to flawed, ancient technologies from the 1990s.
It gets worse. If there's one kind of meme that survives a long time in the development world, it's performance memes. Rumours about poor perf invariably end up haunting technologies with FUD long after any issues have ceased to be relevant.
I have several somewhat orthogonal thoughts on this:
1. This is exactly what I've feared and suspected about JPEG-XR, webp, and JPEG-2000 (which Safari ever so quietly supports). I especially wondered about JPEG-XR after reading how Microsoft had done some great things with GPU-based hardware acceleration for JPEG decoding in IE11: https://blogs.msdn.microsoft.com/ie/2013/09/12/using-hardwar...
I was puzzled by the fact that they never mention JPEG-XR and have never published a similar post about hardware acceleration for the more modern format. They've been incredibly lazy and unfocused on XR for years. It's effectively dead now.
2. The author of the don't use it post keeps saying it gets "software decoded". This implies that this is unusual. But it isn't. Does he think JPEGs are not software decoded? They are in Firefox, and probably every other browser but IE11 and Edge. Someone correct me if I'm wrong. Browser makers have been surprisingly inert on using GPUs for image decoding. libjpeg-turbo is used by Chrome and Firefox, and possibly Safari. It's pure CPU, albeit with some SIMD. That's "software decoding". Hardware acceleration normally means GPU, or a fixed function unit/ASIC like you see for some video codecs.
3. One percent? This was all about a -1% drop in some metric? What's the error margin?
4. He implies that there's a lot of JS in this SPA. This seems like a possible confound that won't necessarily apply to rigorous web developers who don't overload on JS (who are admittedly extremely rare in 2019). Sure, they showed that XR was using more CPU, but so do all post-JPEG formats.
5. Which leads to my last point. What about webp? He strangely never mentions it. It's like only JPEG and XR exist. webp performs very poorly in CPU efficiency compared to JPEG. What are trivago and Cloudinary seeing on that front? Is webp less taxing than XR?
A few clarifying points:
* the author specifically said -1% Conversion. If this were AWS, that would mean a daily reduction of $6million in revenue!
* the author specifically says this is for IE users. This implies two things: a) webp is not an option and b) likely we are also talking about lower class of hardware (mom & dad). I would expect WebP had older hardware probably has the same kind of performance tax, but it's harder to identify because at least with IE/Edge there is an implied age-of-hardware
You think you know the value users attribute to improved image quality and shorter battery life (let's not forget that most folks carry chargers or can charge at work and home).
But you don't. Perhaps you shouldn't judge for others but merely notice this trade off.
92 comments
[ 3.2 ms ] story [ 175 ms ] threadNew formats are not supported on all browsers so you wind up having to support the mainstream formats as well as one or more new formats. Rather than benefiting from reduced storage costs, your storage costs get multiplied.
Performance is also a problem, particularly when people add "yet another polyfill" to get an image to work without native support.
Once I get into eyeballing images closely I also end up questioning if the compression gains are real. For instance, one I read a comment on the article here
https://calendar.perfplanet.com/2018/is-avif-the-future-of-i...
and looked closely I couldn't unsee that the AV1 had very different artifacts than the other compressed images. Not necessarily worse, but definitely different. Unlike the poster I liked the way the suit rendered but definitely there was a lot of blocking around edges that the poster interprets as "pixelation on the flag".
Like many things, the costs of switching image formats are definite, but the benefits are questionable.
I just checked the Google homepage, and even the New Year's Eve Google Doodle currently being shown is a good ol' GIF, rather than a modern format such as WebP. And this is with me using a desktop Chrome browser that definitively supports WebP.
> Like many things, the costs of switching image formats are definite, but the benefits are questionable.
One easy way to get some of the benefits without incurring the switching costs is to leverage Cloudflare Polish[1]. I've used it at a few places, and have had good experiences. Although the majority of gains came from optimizing images that were poorly optimized to begin with, rather than the incremental improvements that occurred from enabling WebP support.
[1] https://support.cloudflare.com/hc/en-us/articles/36000060737...
https://chrome.google.com/webstore/detail/h264ify/aleakchihd...
HEIF is made by MPEG. HEIG is extensible (see AVIF fusing the AV1 royalty-free codec). Also an application format (MIAF) will ensure interoperability.
While waiting for native browser support, some js implementation would do the job.
What else?
Formats do not provide HW acceleration, implementations do. And so far there is zero HW accelerated HEIF implementations for the web.
You sure of that? Apple[1] and Qualcomm[2] might argue otherwise.
[1] https://developer.apple.com/videos/play/wwdc2017/513/
[2] https://techreport.com/news/34306/qualcomm-snapdragon-855-sp...
new-years-eve-2018-4995722058399744.2-law.gif
299 972 bytes
new-years-eve-2018-4995722058399744.2-law.webp
293 636 bytes
new-years-eve-2018-4995722058399744.2-law.apng
251 516 bytes
Interesting...
On some old and buggy machines, that might cause the browser to crash too.
Not something Google wants to risk for a doodle - that's why all non-trivial doodles require a click to load.
OTOH it might be cheaper (relatively) on phones as they can offload the work to hardware components and ramp CPU down (or avoid ramping it up).
with the resulting out.webp being 285 932 bytes, so it seems apng does a very good job here, unless I'm using suboptimal options for webp.
However, as TFA mentions, better compression is currently a double-edged sword, because the CPU has become a bottleneck again: even top-of-the-line Android phones have weak Qualcomm CPUs, and sites use so much JavaScript that even high-end machines can struggle.
So we may end up living with JPEG for a few more years. This format has been designed when CPUs had 25Mhz, and we've had over 25 years to optimize the implementations, so we get a lot of bang for the buck from it.
Most of which has nothing to do with compression format, to be honest. No image compression format is going to save us from websites that use 2000x2000 sized photos for 200x200 author thumbnails.
Maybe I should write one, I guess.
Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
https://github.com/metalabdesign/sharp-loader
https://github.com/herrstucki/responsive-loader
https://github.com/tcoopman/image-webpack-loader
(and many more, just grabbing a few examples)
Something that I've not seen is a plugin/loader that will automatically use the <picture> tag to make available multiple auto-generated source images and a fallback <img>. It'd be great to be able to
import myImage from 'res/img/my_img.png'
and have myImage be a string like
<picture><source srcset="[generated-name-of-my-image.webp]"><img src="generated-name-of-my-image.png"]></picture>
This sounded like an obvious truth but I think you're wrong!
JPEG has progressive rendering, so a browser could easily decide to stop once it has rendered enough for the image size. Once you zoom or if it's used elsewhere, it can render a little more.
Except for JPEG2000, which Safari supports but does not include in its Accepts: header. So Akamai just sniffs the user-agent and serves JPEG2000 to anything that looks like Safari, whether they support it or not.
Edit: I agree that something like the pagespeed module rewriting html probably goes too far and html rewriting is fraught with weird edge cases, but content negotiation is something only a webserver (or CDN) can do.
There's a big thing that bugs me about that when it comes to the web: Software patents and licenses basically ruined wavelet-based image compression schemes on the web.
Wavelets would have been ideal. Basically, the lossy compression would work about the same as the cosine transform in JPEG, but by using the discrete wavelet transform, you basically get thumbnailing built in: Imagine transmitting a tiny little thumbnail of your image. Then, you transmit the information used to derive a version 4x that size (2x width, 2x height) from that. Then, you transmit the data used to derive the next 4x layer. Rinse, repeat -- that's exactly how image data is organized after a wavelet transform.
With some clever adaptions, this could have done away with so much cruft and complexity we have today (like image sets and @media queries for background images): Imagine just putting a 4K version of your background on the server and telling the browser how to render it (an <img> tag with a specified size, or background-size in css). The browser could then start loading the image until it realizes it has all the data for your resolution (say on your phone), and just terminate the transmission before the whole image is downloaded.
Jpeg was released the MPEG-1 ( H.261 ) era. Since then we have had H.262 / MPEG2, H.263 / Divx, H.264 /AVC, H.265 / HEVC, and the coming H.266 / VVC. If we consider the amount of improvement we got from Video Encoding, images hasn't changed much at all, we happen to fine tune the heck out of Jpeg. And I hardly call a new image format with 50% reduction bitrate compared to a nearly 30 years old tech a major breakthrough, and that is without mentioning the huge decoding resources requirement.
There are many project that just improves on JPEG, Dropbox had a lossless compressor for JPEG, and Cant remember if Pik was originally based on jpeg as well.
Unless there is a real major breakthrough that gives the current best JPG encoder quality at 25% or less its size, focusing on 200KB, 100KB, 50KB images size, ( i.e the new image will have to be 50KB, 25KB, 12.5KB ) while using within 2x decoding power. I am convinced our roadmap for Network bandwidth improvement in the next 3 - 5 years are going to make JPEG to last forever.
I don't know why they wouldn't just switch over to WebP in preference to that, though. It's better to have fewer of these formats around and the simplest thing to do is use the one that already has support.
Note that parent said "Chrome would gain support for JPEG XR", which I believe is what 05 is replying to.
However, the summary (and the article) leaves out a relevant question that immediately comes to mind: Are other image formats decoded off the main thread or is it just that decoding JPEG-XR is just more taxing compared to other formats, thus negating its advantages?
I'm fairly certain the article may be poorly worded. Certainly standard JPGs are decoded on the CPU side as well, are they not?
To the best of my knowledge, I don't think any browsers are doing JPG decoding in hardware; I think they use the GPU for primitives and compositing, etc.
I would look forward to being corrected if I'm wrong!
Other formats use more expensive entropy coding, more block sizes, smoothing filters, and inter-block prediction which forces them to be decoded mostly serially.
[0]https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video
Is the TLDR actually "jpeg xr implementation is poorly optimized and negatively affects browser performance which decreased conversions."
Why should anyone test a format which is used by just two browsers with a very small market share and which will be never adopted by the company with the biggest market share (because Google has WebP). JPEG-XR is DOA and nobody should care and waste time on this (except the OP gone astray).
I like that there is a tl;dr right at the beginning though.
Thinking about it, assuming they’ve got a WebP output for Chrome already this could be as simple as adding one more build step with a different encoder, and one more negotiation rule on the CDN.
It indicates to me that either the inmates are running the asylum - that devs are being given too much leeway to pursue personal interests - or (and?) that Trivago's day to day work is proving so dull that developers in some teams are inventing R&D work just to stay sane.
Is this an overreaction, or a subversive call to improve the decode perf of JPEG-XR? I'm certainly on the performance-is-a-feature bandwagon myself, but 1% ux impact is pretty small. And shouldn't we assume that decode perf will improve in the near future? Is there any reason to assume that this problem is permanent and unfixable?
The best you can say here is that it’s within the margin of error, in which case the effect is so small it’s literally unmeasurable. In that case the null hypothesis still wins, and you should not switch.
> Is there any reason to assume that this problem is permanent and unfixable?
IE11 as tested in the article is not getting anything other than security patches. The users who use IE11 are probably also the users most affected by poor performance since they most likely have the oldest hardware.
> The best you can say here is that it’s within the margin of error
What is preventing JPEG-XR from becoming faster and a net ux positive, tomorrow or in the future? Isn't the best I can say that today's results might be a bug or fluke or a quick and dirty implementation, and not a permanent problem?
> IE11 as tested in the article is not getting anything other than security patches.
The title and conclusion both said we should avoid JPEG-XR on the web, not that we should avoid it on IE11.
You're right, the article focused on the problem with IE11. Is it already a net positive on Edge? (The article didn't say.) Should the shim to make it work in the old browser that fewer people use really override the potential benefits on the newer browser that more people are already using?
Does it seem reasonable or unreasonable to make future web decisions depend on an old browser that's on life support today and has small and declining market share?
And even if you don't, taking a 1% hit to your conversion rate and hurting user experience just because you want to use a fancy new image format is probably not a good idea.
The may be a spurious figure.
"we were able to statistically verify a -1% negative impact on core business metrics including conversions"
Ultimately though the issue is more fundamental: you can't make assertions about performance without publishing actual figures. All we have to go on is the author's assertion that the test methodology was sound and the results were out of the bounds of chance. That's not really enough.
I have worked with many, many teams who earnestly believed they were running "scientific" MVTs, with solid statistical analysis free of bias. They weren't.
Now, overall time to decode still matters. If it's a gigantic hero image above the fold, then it could take a noticeable amount of time to decode, but it's probably still a win.
With the only good JPEG 2000 encoder, its quality at small file sizes is easily better than webp, but in practice it seems virtually nobody uses JPEG 2000, because the open source encoders are not really that great (probably because the patents are a guessing game, although technically the known holders are willing to grant royalty-free licenses).
I was surprised to see a post from Trivago - I might have to book my next hotel through them.
The solution to this problem is to start using these new technologies, so the browser vendors can prioritise work to make them performant. It is not to write articles that entrench that neglect and tie us to flawed, ancient technologies from the 1990s.
It gets worse. If there's one kind of meme that survives a long time in the development world, it's performance memes. Rumours about poor perf invariably end up haunting technologies with FUD long after any issues have ceased to be relevant.
That's a hilarious thought to me. How can a 1% change in a sales funnel be considered statistically relevant?
1. This is exactly what I've feared and suspected about JPEG-XR, webp, and JPEG-2000 (which Safari ever so quietly supports). I especially wondered about JPEG-XR after reading how Microsoft had done some great things with GPU-based hardware acceleration for JPEG decoding in IE11: https://blogs.msdn.microsoft.com/ie/2013/09/12/using-hardwar...
I was puzzled by the fact that they never mention JPEG-XR and have never published a similar post about hardware acceleration for the more modern format. They've been incredibly lazy and unfocused on XR for years. It's effectively dead now.
2. The author of the don't use it post keeps saying it gets "software decoded". This implies that this is unusual. But it isn't. Does he think JPEGs are not software decoded? They are in Firefox, and probably every other browser but IE11 and Edge. Someone correct me if I'm wrong. Browser makers have been surprisingly inert on using GPUs for image decoding. libjpeg-turbo is used by Chrome and Firefox, and possibly Safari. It's pure CPU, albeit with some SIMD. That's "software decoding". Hardware acceleration normally means GPU, or a fixed function unit/ASIC like you see for some video codecs.
3. One percent? This was all about a -1% drop in some metric? What's the error margin?
4. He implies that there's a lot of JS in this SPA. This seems like a possible confound that won't necessarily apply to rigorous web developers who don't overload on JS (who are admittedly extremely rare in 2019). Sure, they showed that XR was using more CPU, but so do all post-JPEG formats.
5. Which leads to my last point. What about webp? He strangely never mentions it. It's like only JPEG and XR exist. webp performs very poorly in CPU efficiency compared to JPEG. What are trivago and Cloudinary seeing on that front? Is webp less taxing than XR?
You think you know the value users attribute to improved image quality and shorter battery life (let's not forget that most folks carry chargers or can charge at work and home).
But you don't. Perhaps you shouldn't judge for others but merely notice this trade off.