This potential problem is overstated if the quality of the scans is decent. I have scanned literally hundreds of books, and encoded them in both djvu and pdf/jbig2 formats, and have never once found a bad character. (Yes, having heard of this issue, I did initially try to find examples) The free djvu tools have parameters you can adjust to make them more or less aggressive at combining similar characters, and above 200DPI it's hard to even force the issue to happen for something like an 8 versus a 6. That was my experience anyway.
It's also telling that the example trotted out is always a decades-old photocopier implementation. Unless someone does some comparisons of recent scans and finds a bunch of altered characters, I just don't buy this concern. MRC+JBig2 is really great and the size savings have not been trivial for me personally.
(I hesitate to add that the alternative would be, I assume, DCT/jpeg page images, which introduce lots of noisy artifacts of their own... destroying our history one DCT-block at a time!)
There are a number of scans I've come across on archive.org where the person uploading has used jbig2 and there are pages where letters like 'e' get swapped for an 's'.
My general rule when creating pdfs with scans is: If it's mostly text or line drawings and it will be used in a professional setting use png/flate. If it's a photo use jpg (especially if the source is a jpg).
I'd love to see some examples (especially if the scan appeared to be of good quality and not a grainy 100dpi affair). Do you have any idea what kinds of documents you were browsing when you found them?
There's apparently 22 pages of fluff at the beggining, so you should add that the page numbers. Also "the third line" was meant to be the 3rd line from the bottom.
Thanks! When I convert the jp2k file for that page 58 with ABBYY default ("balanced") jbig2 settings I don't get any of the errors we see in the examples. If I convert to DJVU and purposefully degrade the image to 150DPI, I still don't get the errors on display in this example. Besides the errors talked about in your linked discussion (germauic), the pdf clearly has many 'c' -> 'e' errors which aren't in the jp2k files. Again, I can't reproduce this with the software I use. I've also extensively used djvulibre and you just don't see errors like this from clean scans.
So, I have to conclude that the original conversions were done with very aggressive settings (or very bad jbig2 implementations). If that's they way the internet archive did all its encodings, then I admit there must be several problem documents, and they should re-do the conversions from the original JP2k files they seem to have kept. Not sure if google books made similar mistakes.
Unless something has changed, Google books has always used the lossy encoding. Quoting the original paper:
The most obvious way to compress pages is to losslessly encode them as a single 1 bpp image (see Figure 2). However, we get much better compression by using symbol encoding and accepting some loss of image data. Although our compression is lossy it is not clear how much information is actually being lost - the letter forms on a page are obviously supposed to be uniform in shape, the variation comes from printing errors and lack of resolution in image capture.
Probably only a small fraction of errors are JBIG2 substitutions though.
The problem lies therein that JBIG2 has a compression factor, that is controlled by people who have an incentive to increase it, where few people know what it actually does. And how much testing is really done when is changed? Will the test include text in other scripts, like Chinese or Japanse? It can go wrong in so many ways.
Jpeg artifacts on text are ugly but they tend to stand out. You will see artifacts long before the text becomes unreadable or ambiguous, and that is a good thing.
My choice would be the recently standardized JPEG XL designed to replace all of: JPEG, JPEG 2000, PNG, GIF. Among other things, it is supposed to be the first choice for long-term storage. I guess mass adoption will begin after including it in the PDF standard. It is already experimentally in Chrome and Firefox.
I've been reading some books on my ereader and every now and then I've come across a word that has been misspelled with similar looking characters. now I'm wondering if it's this jbig2 issue
If this is such a serious problem, please demonstrate at least one example in Google Books where a scan shows the wrong letter or number.
That really can’t be hard to do, since there are lots of common books in GB, there are even OCRs available so you could even do the error checking automatically for thousands of books.
If you can’t show a single case of JBIG data corruption in GOogle Books, you have absolutely no justification in calling it such a serious problem!
Google Book has atrocious scan quality in general, though (compared to, say, the Microsoft-sponsored archive.org ones). Some pages are completely unreadable and sometimes images are completely deleted as the compressor mistakes them for background.
Sorry, this article is low on content and high on fearmongering. This issue was known years ago and the only example of it seems to be one implementation from Xerox, which was also set to lossy mode --- and I believe that was subsequently fixed too.
The other comments here have linked to the previous articles about this, which do give far more detailed information about the problem. JBIG2 in lossless mode won't do this.
The according talk is linked here in the comments - no, it wasn't just lossy mode. It was much harder, but in the end it was reproduced on other quality levels.
But if it happened once, who's gonna guarantee this won't happen again? When it happens with numbers, worst case it can have fatal consequences.
Saying "jbig2 in lossless mode won't do this" is as relevant as saying "png won't do this"
The problem is the existence and use of a codec for a purpose, that harms that purpose. It doesn't matter what it's name is.
Are scans being produced with this codec? And is it not true that not only is there data loss or corruption like with jpg, but that unlike jpg artifacts you can sometimes not know that the data loss or corruption occurred?
That does make it not merely a problem of "I wish they scanned this in higher quality" but a far worse problem of "this document looks good so I trust it" when in fact it was corrupt.
> JB2 has strong similarities with the forthcoming JBIG2 standard developed by the "ISO/IEC JTC1 SC29 Working Group 1" which is responsible for both the JPEG and JBIG standards. This is hardly surprising since JB2 was our own proposal for the JBIG2 standard and remained the only proposal for years. The full JBIG2 standard however is significantly more complex and slighlty less efficient than JB2 because it addresses a broader range of applications.
Wait till you get ML based compression and you get replacements with entirely different but plausibly looking alternatives that go beyond single character substitutions.
"A new research report demonstrates the connection between the rising amount of mass produced housing featuring a dedicated sex dungeon and biased content in the training set used for the compression algorithm in a popular line of architectural plotters"
Yeah, this kind of lossyness can be very bad for serious archival purposes of critical documents, because this small corruption wouldn't be self-evident nor detectable after the fact in an easy way.
Most probably its raison d'être is for non-critical stuff, such as literature, where character flips may be easily detectable during encoding and after the fact (eg. with a spell-checker) and won't possibly subvert the meaning of a whole literary work anyway.
37 comments
[ 2.6 ms ] story [ 82.4 ms ] threadIt's also telling that the example trotted out is always a decades-old photocopier implementation. Unless someone does some comparisons of recent scans and finds a bunch of altered characters, I just don't buy this concern. MRC+JBig2 is really great and the size savings have not been trivial for me personally.
(I hesitate to add that the alternative would be, I assume, DCT/jpeg page images, which introduce lots of noisy artifacts of their own... destroying our history one DCT-block at a time!)
There's apparently 22 pages of fluff at the beggining, so you should add that the page numbers. Also "the third line" was meant to be the 3rd line from the bottom.
Direct links (to be opened in DjView):
https://ia800204.us.archive.org/28/items/cu31924026442156/cu...
https://ia800204.us.archive.org/28/items/cu31924026442156/cu...
So, I have to conclude that the original conversions were done with very aggressive settings (or very bad jbig2 implementations). If that's they way the internet archive did all its encodings, then I admit there must be several problem documents, and they should re-do the conversions from the original JP2k files they seem to have kept. Not sure if google books made similar mistakes.
Jpeg artifacts on text are ugly but they tend to stand out. You will see artifacts long before the text becomes unreadable or ambiguous, and that is a good thing.
That really can’t be hard to do, since there are lots of common books in GB, there are even OCRs available so you could even do the error checking automatically for thousands of books.
If you can’t show a single case of JBIG data corruption in GOogle Books, you have absolutely no justification in calling it such a serious problem!
Immediately this talk from David Kriesel comes to mind. :)
The other comments here have linked to the previous articles about this, which do give far more detailed information about the problem. JBIG2 in lossless mode won't do this.
But if it happened once, who's gonna guarantee this won't happen again? When it happens with numbers, worst case it can have fatal consequences.
The problem is the existence and use of a codec for a purpose, that harms that purpose. It doesn't matter what it's name is.
Are scans being produced with this codec? And is it not true that not only is there data loss or corruption like with jpg, but that unlike jpg artifacts you can sometimes not know that the data loss or corruption occurred?
That does make it not merely a problem of "I wish they scanned this in higher quality" but a far worse problem of "this document looks good so I trust it" when in fact it was corrupt.
I don't belive that's correct; it's rather the other way round.
https://github.com/barak/djvulibre/blob/release.3.5.28/libdj...
> JB2 has strong similarities with the forthcoming JBIG2 standard developed by the "ISO/IEC JTC1 SC29 Working Group 1" which is responsible for both the JPEG and JBIG standards. This is hardly surprising since JB2 was our own proposal for the JBIG2 standard and remained the only proposal for years. The full JBIG2 standard however is significantly more complex and slighlty less efficient than JB2 because it addresses a broader range of applications.
https://news.ycombinator.com/item?id=6156238
https://news.ycombinator.com/item?id=29223815
"A new research report demonstrates the connection between the rising amount of mass produced housing featuring a dedicated sex dungeon and biased content in the training set used for the compression algorithm in a popular line of architectural plotters"
Most probably its raison d'être is for non-critical stuff, such as literature, where character flips may be easily detectable during encoding and after the fact (eg. with a spell-checker) and won't possibly subvert the meaning of a whole literary work anyway.