thanks. yeah there is an annoying limit on vercel's end for body size of 4.5 mb. there are solutions to get around it by uploading directly to a storage solution but i wanted to keep the app simple for folks who want to dive into the code.
I find it challenging to accept something that talks about "OCR" but then I upload a PDF with text in images, and when I query the document after upload, I get a message that says "I can't interpret images"..
Then are you actually doing OCR, or are you just extracting embedded text?
I’d imagine their capabilities mirror that of Mistral OCR [1]. Mistral outputs markdown, the image would have to be convertible to a reasonably useful markdown structure (charts, tables etc).
This highlights the biggest issue I've found with Mistral OCR. Many of the documents I upload are entirely classified as images, which means no OCR is being run.
Pretty much anything with a different colored background gets returned as (image)[image_001].
LLMs tend to be a hammer in search of a nail when it comes to documents that have imagery. We decided on CV models which results in a high 90s midpoint for the docs our customers care about. If you can afford to go with a cv pipeline, it can outperform all of the LLMs by some margin.
Same vein - YouTube most (all?) llm integrations just scrape the transcript. I -think- google's aistudio does more but I'm unsure.
I mean I get it bulk video processing would be crazy expensive, but at least mention you're only analyzing the transcript especially if you're a paid product.
Whisper does do text to speech but yes, nearly all just read off the subtitles. There's a video by f4mi on YouTube where she tricked the summarizing bots with off-screen captions filled with nonsense.
I have a question about Mistral OCR. If I give the model a PDF that is 90% text, is it actually performing OCR on an image representation of the text? Or is it smart enough to extract the text directly and only use OCR on images?
Honestly, the vibes aren't great. Gemini is a lot more flexible for handling PDFs - you can prompt it to do a bunch of other things - and Mistral OCR appears to hallucinate if it can't correctly read handwriting, a common problem with vision LLM based OCR tools.
The way Mistral OCR handles images within the text is disappointing - it doesn't attempt to interpret them, just extracts them out as binary blobs. A vision LLM can usually do a great job of describing an image, but with Mistral OCR you have to manually run that as a separate step.
Knowing that you have to do that as a separate step adds a whole additional level of complexity too.
For example, if some content has the images and some don't, you need to add whole additional steps to your processing and potentially add hallucinations in.
What are you using for document extraction lately, Simon?
45 comments
[ 3.3 ms ] story [ 109 ms ] threadSo cool in fact, I got distracted and ended up building an open source PDF parser and chat app!
Presenting Auntie PDF - your all-knowing guide that unpacks every PDF into clear, actionable insights.
You can upload a pdf or point to a public link, parse it, and then ask questions. All open source and free.
Not working for me on a file like this: https://files.catbox.moe/gii0pu.pdf It says that is larger than 10MB (it's 7MB), or failed on url.
Then are you actually doing OCR, or are you just extracting embedded text?
[1] https://mistral.ai/en/news/mistral-ocr
Pretty much anything with a different colored background gets returned as (image)[image_001].
Example: https://omni-demo-data.s3.us-east-1.amazonaws.com/test/17398...
I mean I get it bulk video processing would be crazy expensive, but at least mention you're only analyzing the transcript especially if you're a paid product.
Honestly, the vibes aren't great. Gemini is a lot more flexible for handling PDFs - you can prompt it to do a bunch of other things - and Mistral OCR appears to hallucinate if it can't correctly read handwriting, a common problem with vision LLM based OCR tools.
The way Mistral OCR handles images within the text is disappointing - it doesn't attempt to interpret them, just extracts them out as binary blobs. A vision LLM can usually do a great job of describing an image, but with Mistral OCR you have to manually run that as a separate step.
For example, if some content has the images and some don't, you need to add whole additional steps to your processing and potentially add hallucinations in.
What are you using for document extraction lately, Simon?
Would be nice with a [Download Combined Rendered] button to download a self-contained .html web page of the rendered combined page.