This is a great example of why a PDF’s text layer is not necessarily the same as its visible content.
I’d be cautious about making “non-black means invisible” the general rule. A possible middle ground between raw text extraction and full OCR would be to render the page, map each extracted glyph’s bounding box back to the rendered pixels, and discard glyphs that have almost no contrast with their local background.
That would preserve the exact embedded text for visible characters while using rasterization only to determine visibility.
I agree with you, it's not safe to have a rule saying "non-black means invisible".
I should have made it clear in the blog post, but the first thing I do when processing a PDF is I identify the document_type. Then I run special rules for that document_type.
So the "non-black means invisible" rule only applies when processing this document_type.
I was converting some PDF documents recently and I noticed a few pages were missing their text. I wondered if the text was white-on-white. I got Claude to check it, and I was partially right. The original (image) text layers were gone, but the OCR layer remained. Lost all the font data, but most of the text was recoverable.
Is it not dangerous to ignore text based on its color? For this particular case it might work but looks like you have a generic service that is supposed to work with any bank statement.
It's the ground truth, and it's not like it's more complex than parsing a pdf, at this point, the technology for OCR might even be better than PDF parsing, which is full of accidental instead of natural complexity.
Why not skipping all the minus signs unless they do not immediately precede a number. I see all the minus signs are after the number, not before, so just skip if they are not before. Preferably in the same BT/ET block as the number.
I was expecting something about double-entry accounting and how it can (initially) be non-intuitive whether to mark a given entry as a debit (-100) or credit (+100), especially for liabilities (when you enter a credit card payment in your ledger, the signs aren't what you might intuitively expect when you haven't wrapped your head around the system yet).
Instead I get a nice blog post about a knotty little problem caused by someone's abuse of semantic markup for style purposes. Shows that HTML doesn't have a monopoly on that issue. PDFs are already bad enough as a format for extracting data from, we don't need dumb HTML-inspired tricks on top of that.
Nice job finding the issue and explaining it clearly, OP. Thanks for an interesting writeup.
If anyone sent me data in PDF format I would immediately reject it and ask for a more suitable format. This isn't because I know anything about PDF, it's just intuition. Why don't more people come with this intuition? I feel the same way about basic security principles.
12 comments
[ 0.30 ms ] story [ 32.0 ms ] threadI’d be cautious about making “non-black means invisible” the general rule. A possible middle ground between raw text extraction and full OCR would be to render the page, map each extracted glyph’s bounding box back to the rendered pixels, and discard glyphs that have almost no contrast with their local background.
That would preserve the exact embedded text for visible characters while using rasterization only to determine visibility.
Has anyone tried this kind of hybrid approach?
I should have made it clear in the blog post, but the first thing I do when processing a PDF is I identify the document_type. Then I run special rules for that document_type.
So the "non-black means invisible" rule only applies when processing this document_type.
Did they try one method, and what was the feedback?
It's the ground truth, and it's not like it's more complex than parsing a pdf, at this point, the technology for OCR might even be better than PDF parsing, which is full of accidental instead of natural complexity.
Instead I get a nice blog post about a knotty little problem caused by someone's abuse of semantic markup for style purposes. Shows that HTML doesn't have a monopoly on that issue. PDFs are already bad enough as a format for extracting data from, we don't need dumb HTML-inspired tricks on top of that.
Nice job finding the issue and explaining it clearly, OP. Thanks for an interesting writeup.
If you use a real minus sign, and not an ASCII hyphen, it should be the same width as the corresponding plus sign in any reasonable font.
Use a plus sign and make that invisible.
Here is an idea: columns of figures should use a monospaced font.
You don't just want right or left edges to align, but all corresponding digit positions should align:
This consideration overrides any nonsense about whether a space is the same width as a minus.