Show HN: Benchmarking VLMs vs. Traditional OCR (getomni.ai)
We've been continuously evaluating different models since we released the Zerox package last year (https://github.com/getomni-ai/zerox). And we wanted to put some numbers behind it. So we’re open sourcing our internal OCR benchmark + evaluation datasets.
Full writeup + data explorer here: https://getomni.ai/ocr-benchmark
Github: https://github.com/getomni-ai/benchmark
Huggingface: https://huggingface.co/datasets/getomni-ai/ocr-benchmark
Couple notes on the methodology:
1. We are using JSON accuracy as our primary metric. The end goal is to evaluate how well each OCR provider can prepare the data for LLM ingestion.
2. This methodology differs from a lot of OCR benchmarks, because it doesn't rely on text similarity. We believe text similarity measurements are heavily biased towards the exact layout of the ground truth text, and penalize correct OCR that has slight layout differences.
3. Every document goes Image => OCR => Predicted JSON. And we compare the predicted JSON against the annotated ground truth JSON. The VLMs are capable of Image => JSON directly, we are primarily trying to measure OCR accuracy here. Planning to release a separate report on direct JSON accuracy next week.
This is a continuous work in progress! There are at least 10 additional providers we plan to add to the list.
The next big roadmap items are: - Comparing OCR vs. direct extraction. Early results here show a slight accuracy improvement, but it’s highly variable on page length.
- A multilingual comparison. Right now the evaluation data is english only.
- A breakdown of the data by type (best model for handwriting, tables, charts, photos, etc.)
41 comments
[ 2.6 ms ] story [ 102 ms ] threadIt also has a 1M token context window, though from personal experience it seems to work better the smaller the context window is.
Seems like Google models have been slowly improving. It wasn't so long ago I completely dismissed them.
I had gemini 2.0 pro read my entire hand written, stain covered, half English, half french family cookbook perfectly first time
It's _crazy_ good. I had it output the whole thing in latex format to generate a printable document immediately too
VLMs are every bit as susceptible to the (unsolved) hallucination problem as regular LLMs are. I would not use them to do OCR on anything important because the failure modes are totally unbounded (unlike regular OCR).
The article says they evaluated "Traditional OCR providers (Azure, AWS Textract, Google Document AI, etc.)"
Are those not paid OCR engines?
I'm using computers since I can read, and when somebody says "traditional OCR", I think about the older systems like Tessaract or ABBYY's FineReader which can be again automated for batch processing, albeit mostly locally.
Sending huge amount of PDFs to a cloud server to get them processed is still a bit alien to me, since it can be done on-premises (or on a VPS with the said software) very efficiently from my perspective.
Looks like they've got deterministic metrics to me: For each document they've got a ground truth set of JSON extracted data, and they use json-diff to calculate the fields that disagree.
There is GPT-4o in their evaluation pipeline - but only as a means of converting the OCRed document into their target JSON schema.
There is none, because CLOUD Act.
I'm interested in the same thing for audio transcription too, for models like Gemini or GPT-4o audio accepting audio input.
Also, I'd be useful to understand how an OCR context differs from standard injection attacks. One thing I can think of is potential tabular injection attacks. But also image-based, especially for VLMs, are relevant. So a OCR injection attack benchmark might just be a combination of different domain-specific benchmarks formated as images.
So yeah, would be curious how susceptible they are to more refined approaches. Are there some known examples?
However there are two big bugs we've found with VLMs:
1. Correcting the document. If you have an income statement, and all the line items add up to $1,001. But the total says $1000. The model will frequently correct the final output. Which would be terrible if you were trying to build a "identify mistakes in these documents" type tool.
2. Infinite loops. Sometimes the models will get hung up on a particular token and repeat that until it times out. This gets triggered a lot in markdown tables |---|---|----------------->
I wrote in a previous post about how NLP services were dead because of LLMs and obviously people in NLP took great offense to that. But I was able to use the NLP abilities of an LLM without needing to know anything about the intricacies of NLP or any APIs and it worked great. This post on OCR pretty much shows exactly what I meant. Gemini does OCR almost as good as OmniAI (granted I've never heard of it), but at 1/10th the cost. OpenAI will only get better very quickly. Kudos to OmniAI for releasing honest data, though.
Sure you might get an additional 5% accuracy from OmniAI vs Gemini but a generalized LLM can do so much more than just OCR. I've been playing with OpenAI this entire weekend and literally the sky's the limit. Not only can you OCR images, you can ask the LLM to summarize it, transform it into HTML, classify it, give a rating based on whatever parameters you want, get a lexile score, all in a single API call. Plus it will even spit out the code to do all of the above for you to use as well. And if it doesn't do what you need it to do right now, it will pretty soon.
I think the future of AI is going to be pretty bleak for everyone except the extremely big players that can afford to invest hundreds of billions of dollars. I also think there's going to be a real battle of copyright in less than 5 years which will also favor the big rich players as well.
Whilst where OCR's tend to be used it's often a no go.... Just saying this trying to remember all the places where I've implemented it or seen it implemented. A common one was billing stuff.
The price of any of these services pales in comparison to getting a human involved in any fraction of cases.
It is likely reasonable to expect the base LMs to keep getting better and for there to not be a moat on accuracy in the long term, but businesses are not just built on benchmark accuracy and have plenty of other ways to survive, even if the technology under the hood changes.
>>5% accuracy can be worth a lot.
Most surprising to me about these results is the BEST error rate was over 8% errors (91.7% accuracy) and the worse was 40%.
Their method of calculating errors seems quite good:
>> Accuracy is measured by comparing the JSON output from the OCR/Extraction to the ground truth JSON. We calculate the number of JSON differences divided by the total number fields in the ground truth JSON. We believe this calculation method lines up most closely with a real world expectation of accuracy.
>> Ex: if you are tasked with extracting 31 values from a document, and make 4 mistakes, that results in an 87% accuracy.
Especially where dealing with numbers and money, having 10% of them being wrong seems unusable, often worse than doing nothing.
Having humans check the results instead of doing the transcriptions would be better, but humans are notoriously bad at maintaining vigilance doing the same task over many documents.
What would be interesting is finding which two OCR/AI systems make the most different mistakes and running documents against both. Flagging only the disagreements for human verification would reduce the task substantially.
There have been OCR products that do that for decades, and I would hope all the ocr startups are doing the same already. Often times something is objectively difficult to read and the various models will all fail in the same place, reducing the expected utility of this method. It still helps of course. I forget the name of the product, there was one that used about 5 ocr engines and would use consensus to optimize its output. It could never beat ABBYY finereader though, it was a distant second place.
But you get most of the bang for the buck for 1/10th the cost so I think overall it's far, far superior.
> They [VLMs] are generally more capable of "looking past the noise" of scan lines, creases, watermarks. Traditional models tend to outperform on high-density pages (textbooks, research papers) as well as common document formats like tax forms.
Which is a bit confusing? Did they test that or what? It doesn't seem that way from their limited dataset.
I fine-tuned a Llama 3.2 Vision on a small dataset I created for extracting text without heavy cropping. Results are simply amazing in comparison with OCR-based approaches. It can be tried here: https://news.ycombinator.com/item?id=43192417