This is actually the thing I really desperately need. I'm routinely analyzing contracts that were faxed to me, scanned with monstrously poor resolution, wet signed, all kinds of shit. The big LLM providers choke on this raw input and I burn up the entire context window for 30 pages of text. Understandable evals of the quality of these OCR systems (which are moving wicked fast) would be helpful...
And here's the kicker. I can't afford mistakes. Missing a single character or misinterpreting it could be catastrophic. 4 units vacant? 10 days to respond? Signature missing? Incredibly critical things. I can't find an eval that gives me confidence around this.
The OCR leaderboards I’ve seen leave a lot to be desired.
With the rapid release of so many of these models, I wish there were a better way to know which ones are actually the best.
I also feel like most/all of these models don’t handle charts, other than to maybe include a link to a cropped image. It would be nice for the OCR model to also convert charts into markdown tables, but this is obviously challenging.
The best leader board I have used is ocrarena.ai. I agree it is not detailed enough. I wish people could rate what part of the ocr went well or bad (layout, text recognition, etc). However, my more specific results using custom prompts and my own images on their playground page are relatively closely aligned with the rankings as others have voted.
Is it possible for such a small model to outperform gemini 3 or is this a case of benchmarks not showing the reality? I would love to be hopeful, but so far an open source model was never better than a closed one even when benchmarks were showing that.
There was so many OCR models released in the past few months, all VLM models and yet none of them handle Korean well. Every time I try with a random screenshot (not a A4 document) they just fail at a "simple" task. And funnily enough Qwen3 8B VL is the best model that usually get it right (although I couldn't get the bbox quite well). Even more funny, whatever is running on an iphone locally on cpu is insanely good, same with google's OCR api. I don't know why we don't get more of the traditional OCR stuff. Paddlepaddle v5 is the closest I could find. At this point, I feel like I might be doing something wrong with those VLMs.
I tested this pretty extensively and it has a common failure mode that prevents me from using: extracting footnotes and similar from the full text of academic works. For some reason, many of these models are trained in a way that results in these being excluded, despite these document sections often containing import details and context. Both versions of DeepseekOCR have the same problem. Of the others I’ve tested, dot-ocr in layout mode works best (but is slow) and then datalab’s chandra model (which is larger and has bad license constraints).
I can get multiple sets of footnotes (critical + content notes) reliably recognized and categorized using gemini-3-flash-preview. I took 15-20 hours to iterate on my prompt for a specific format. Otherwise it would not produce good enough results. It was a slow process because results from batch did not mirror what I was getting from the chat mode, and you have to wait for batch results while analyzing the last set. There was also a bit of debugging of the batch protocol going on at the same time. Flash is also surprisingly affordable for the results I am getting, 4-5x less than I had anticipated. I gave up on gemini-3-pro pretty quickly because it overthinks and messes things up.
I've been trying different OCR models on what should be very simple - subtitles (these are simple machine-rendered text). While all models do very well (95+% accuracy), I haven't seen a model not occasionally make very obvious mistakes. Maybe it will take a different approach to get the last 1%...
Has anyone experiment with using VLM to detect "marks"? Thinking of pen/pencil based markings like underlines, circles,checkmarks.. Can these models do it?
> Option 1: Zhipu MaaS API (Recommended for Quick Start)
> Use the hosted cloud API – no GPU needed.
...
> Option 2: Self-host with vLLM / SGLang
So, first off, this looks really cool and, given I'm looking for OCR at the moment, I'm pretty interested in this and other OCR models.
With that said, the README implies that option 2 requires a GPU. That's fine but it would be incredibly helpful if the README were explicit about requirements, and especially the amount of memory it needs.
EDIT: Looking at the links under option 3, the docs for macOS setup suggest 8GB of unified memory is enough to run the model, which is pretty modest, so I'd imagine Option 2 is similar. Ollama also offers a CPU only option (no idea how that will perform - not amazingly, I'm guessing), but that would suggest to me that if your volume requirements are low and you can't shell out for or source a beefy enough GPU and don't want to pay the sometimes exhorbitant hire costs, you should be able to punt it on to a machine with enough memory to run the model without too much difficulty.
19 comments
[ 3.1 ms ] story [ 41.7 ms ] threadAnd here's the kicker. I can't afford mistakes. Missing a single character or misinterpreting it could be catastrophic. 4 units vacant? 10 days to respond? Signature missing? Incredibly critical things. I can't find an eval that gives me confidence around this.
I’ve also heard very good things about these two in particular:
- LightOnOCR-2-1B: https://huggingface.co/lightonai/LightOnOCR-2-1B
- PaddleOCR-VL-1.5: https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.5
The OCR leaderboards I’ve seen leave a lot to be desired.
With the rapid release of so many of these models, I wish there were a better way to know which ones are actually the best.
I also feel like most/all of these models don’t handle charts, other than to maybe include a link to a cropped image. It would be nice for the OCR model to also convert charts into markdown tables, but this is obviously challenging.
What more are you looking for?
That doesn't sound great
EDIT: https://github.com/overcuriousity/pdf2epub looks interesting.
...
> Option 2: Self-host with vLLM / SGLang
So, first off, this looks really cool and, given I'm looking for OCR at the moment, I'm pretty interested in this and other OCR models.
With that said, the README implies that option 2 requires a GPU. That's fine but it would be incredibly helpful if the README were explicit about requirements, and especially the amount of memory it needs.
EDIT: Looking at the links under option 3, the docs for macOS setup suggest 8GB of unified memory is enough to run the model, which is pretty modest, so I'd imagine Option 2 is similar. Ollama also offers a CPU only option (no idea how that will perform - not amazingly, I'm guessing), but that would suggest to me that if your volume requirements are low and you can't shell out for or source a beefy enough GPU and don't want to pay the sometimes exhorbitant hire costs, you should be able to punt it on to a machine with enough memory to run the model without too much difficulty.