HN Explorer

Ask HN: What are you using to parse PDFs for RAG?

163 points by carlbren ↗ HN
Hi, I'm looking for a simple way to convert PDFs into markdown with integrated images and tables. Tried Llamaindex, but no integrated images. Tried Langchain, but some PDFs will have the footer being parsed before the top. Tried to use Adobe PDF API, but have to pay $25K upfront!

101 comments

[ 3.9 ms ] story [ 189 ms ] thread
brudgers
have to pay $25K upfront

That's a lot of your money.

It's not a big dose of OPM (Other People's Money).

When building a business, adequate capitalization solve a lot technical problems and it is no different when a business is built. If you aren't building a business, money is different and there's nothing wrong with not building a business. Good luck.

muzani
Idk man, it's a massive chunk of Other People's Money too. Prices like this is why Microsoft Teams has dominance over Slack.

There's really two use cases:

1. If you don't use the budget, you have less budget. I see this happen a lot in construction too, where each project has a set budget and it goes down or gets stolen in time. They'd rather pay $25k for the lifetime of a 2 year project than pay $500/month. (the other fear is that these startups shut down in a year)

2. Tax exemptions, or some sort of money laundering where there's more value to pay a big name lots of money.

sshine
There's a lot of valid business cases that rely on parsing PDFs but does not warrant spending $25k on it.

I've helped prototype something using PyMuPDF. It worked as well as it had to, and it didn't cost $25k.

Kiro
Strange comment and bad advice in general. This is a problem that doesn't need $25k to be solved. You're basically telling them to give up or raise money for no reason at all.
manquer
For me it wouldn't be about the 25k upfront,it is just working with Adobe is incredibly painful, same for Oracle. I really really don't want to work with them in any capacity if i can avoid.

If say MS charged the same or even double that I would still work with them, they at-least they don't treat their customers as criminals.

constantinum
I would recommend give LLMWhisperer a try with the documents pertaining to your use case.

https://unstract.com/llmwhisperer/

Try demo in playground: https://pg.llmwhisperer.unstract.com/

Quick tutorial: https://unstract.com/blog/extract-table-from-pdf/

ipsum2
not open source, and OP seems to be the owner.
lumos_maxima93
it is open-source, the main platform is - Unstract https://github.com/Zipstack/unstract
ipsum2
Nope, LLMWhisperer to parse PDFs is called through an paid API.
constantinum
I'm not sure why the comment is downvoted! Let me see; the OP did not specifically try/ask for open-source solutions; at least, that is what I read.

Let me break it down!

As one of the commenters mentioned, he/she uses four different tools to parse PDFs to handle common parsing cases — tables, tables with images, OCR, layouts, handwriting, etc.

With LLMwhisperer, you don't need that.

Parsing is just a part of the problem. Engineers still need to figure out what LLM models work/are sufficient, reduce costs(tokens) and performance(parsing a million pages), and make the AI stack production-ready.

LLMWhisperer at least handles most use cases and moves out of your way fast.

Also, LLMwhisperer is not open-source; it's API is charged based on pages parsed.

alexliu518
I understand your idea, but if you are sure that a software is excellent, then paying for it is a good habit.
jmnicolas
25k isn't exactly pocket change!
arthurcolle
We have an excellent solution for this at Brainchain AI that we call Carnivore.

We're only a few days away from deploying an SDK for this exactly use case among some others.

If you'd like to speak with our team, please contact us! We would love to help you get through your PDF and other file type parsing issues with our solution. Feel free to ping us at data [at] brainchain.ai

esquivalience
PyMuPDF seems to be intended for this use-case and mentions images:

https://medium.com/@pymupdf/rag-llm-and-pdf-conversion-to-ma...

(Though the article linked above has the feeling, to me, of being at least partly AI-written, which does cause me to pause)

> Update: We have now published a new package, PyMuPDF4LLM, to easily convert the pages of a PDF to text in Markdown format. Install via pip with `pip install pymupdf4llm`. https://pymupdf4llm.readthedocs.io/en/latest/

jwilk
What's RAG?
wkat4242
Retrieval Augmented Generation. It's a way to automatically provide an LLM model with the correct context based on the user's question, providing information directly into the model context, used for information that's not part of the model training.
gforce_de
from https://en.wikipedia.org/wiki/Retrieval-augmented_generation

"Retrieval augmented generation (RAG) is a type of information retrieval process. It modifies interactions with a large language model (LLM) so that it responds to queries with reference to a specified set of documents, using it in preference to information drawn from its own vast, static training data."

noufalibrahim
It's a way of giving LLMs extra information (usually fast changing) that they were no trained with so that you can have them return relevant information. Think of asking an LLM "Who is Paul Graham" (assuming PG is relatively unknown) and it would say it doesn't know. But if you search your own knowledge base and then augment the prompt to something like "Paul Graham is a well known Venture Capitalist. Who is Paul Graham?", it can give you that information back. The idea of adding the extra information is the augmenting and you do that by retrieving relevant information from a knowledge base before you involve the LLM.
exe34
nowadays you might have some luck feeding PNG images to multimodal LLMs.
nicoboo
I've experimented with GCP's Stack using Agent Builder and relying on Gemini Pro 1.5.

I also experimented with pretty large of various files (around 6000 video games full notices) where I used OCR parsing in a similar configuration with mixed results due to the visual complexity of the original content.

jumploops
In my experience Azure’s Form Recognizer (now called “Document Intelligence”) is the best (cheapest/most accurate) PDF parser for tabular data.

If I were working on this problem in 2024, I’d use Azure to pre-process all docs into something machine parsable, and then use an LLM to transform/structure the processed content into my specific use-case.

For RAG, I’d treat the problem like traditional search (multiple indices, preprocess content, scoring, etc.).

Make the easy things easy, and the hard things possible.

mkaszkowiak
Did you encounter hidden costs when using Azure Document Intelligence? I processed some PDFs using the paid tier, but the resulting costs were way higher than expected, despite using a prebuilt layout model for only structured extraction. Have no clue what could cause it, no extra details on the billing page. Not sure if the price is misleading, or if it's a skill issue on my part :)
jumploops
We did not, I remember costs matching our expectations.

With that said, I have only used the previous tool (Form Recognizer) in production. Not sure if the new rebrand/product suite has more opaque costs.

rayxi271828
Have you tried unstructured.io? So far seems promising.
whakim
We have been using different things for text, images, and tables. I think it's worth pointing out that PDFs are extremely messy under-the-hood so expecting perfect output is a fool's errand; transformers are extremely powerful and can often do surprisingly well even when you've accidentally mashed a set of footnotes into the middle of a paragraph or something.

For text, unstructured seems to work quite well and does a good job of quickly processing easy documents while falling back to OCR when required. It is also quite flexible with regards to chunking and categorization, which is important when you start thinking about your embedding step. OTOH it can definitely be computationally expensive to process long documents which require OCR.

For images, we've used PyMuPDF. The main weakness we've found is that it doesn't seem to have a good story for dealing with vector images - it seems to output its own proprietary vector type. If anyone knows how to get it to output SVG that'd obviously be amazing.

For tables, we've used Camelot. Tables are pretty hard though; most libraries are totally fine for simple tables, but there are a ton of wild tables in PDFs out there which are barely human-readable to begin with.

For tables and images specifically, I'd think about what exactly you want to do with the output. Are you trying to summarize these things (using something like GPT-4 Vision?) Are you trying to present them alongside your usual RAG output? This may inform your methodology.

Tade0
> I think it's worth pointing out that PDFs are extremely messy under-the-hood so expecting perfect output is a fool's errand

This.

A while ago someone asked me why their banking solution doesn't allow to paste payment amounts (among other things) and surely there must be a way to do it correctly.

Not with PDF. What a person reads as a single number may be any grouping of entities which may or may not paste correctly.

Some banks simply don't want to deal with this sort of headache.

mkaszkowiak
How do you combine the outputs? Wouldn't there be data duplication between unstructured text and tables?
whakim
We just skip several of unstructured's categories, such as tables and images. We also do some deduplication post-ANN as we want to optimize for novelty as well as relevance. That being said, how are you planning to embed an image or a table to make it searchable? It sounds simple in theory, but how do you generate an actually good image summary (without spending huge amounts of money filling OpenAI's coffers for negligible benefit)? How do you embed a table?
mkaszkowiak
Thanks for answering! In my case, I don't directly use RAG; but rather post-process documents via LLMs to extract a set of specific answers. That's also why I've asked about deduplication - asking LLM to provide an answer from 2 different data sources (invalid unstructured table text & valid structured table contents) quickly ramps up errors.
zbyforgotp
Tables are a hard case for RAG, even if you parse them perfectly into Markdown, the LLMs still tend to struggle with interpreting them.
[deleted]
(comment deleted)
martincollignon
Have you tried https://github.com/VikParuchuri/marker ?
mkaszkowiak
For my use case, overall Marker seems to work pretty well - but it has issues with tables. Merged cells, misplaced headers, and so forth. I'm currently extracting Polish PDFs that are //not// scanned

When compared to Azure Document Intelligence, Marker is really cheap when self-hosted (assuming you fall under the license requirements), but it does not produce high quality data. YMMV.

cpursley
Yeah, the header stuff (and empty cells) for tables needs some work.
vikp
Working on improving tables soon (I'm the author of marker)
mkaszkowiak
Glad to hear that :) Thanks for developing Marker!
chandrai
2nd that. Marker work pretty well as async internal service for us! Thanks!
pookee
We're currently implementing this with https://mathpix.com/, it is not free but really not that expensive. It looks very promising
Teleoflexuous
My use case is research papers. That means very clear text, combined with graphs of varying form and quality and finally occasional formulas.

Two approaches I had most, but not full, success with are: 1) converting to image with pdf2image, then reading with pytesseract 2) throwing whole pdfs into pypdf 3) experimental multimodal models

You can get more if you make content more predictable (if you know this part is going to be pure text just put it in pypdf, if you know this is going to be a math formula explain the field to the model and have it read it back for high accessibility needs audience) the better it will go, but it continues to be a nightmare and a bottleneck.

authorfly
I have great news I wish someone delivered to me when I was in your shoes - try "GROBID". It parses papers into objects with abstract/body/figures! It will help you out a great deal. It is designed for papers and can extract the text almost flawlessly, but also give information on graphs for separate processing. I have several years experience with academic text processing (including presentations) working with an Academic Publisher if I could be helpful to anything?
Teleoflexuous
I have no idea how did I miss them last time I was looking around, unless they grew significantly over last half a year or so. I'll check it out when I get back to this project, thanks.

I wish I was hiring, if that's what you're asking ;) Otherwise, if you have any ideas for processing formulas (even just for reading them out, but any extra steps towards expressing what they mean - ' 'sum divided by count' is 'mean'/'average' value ' being the most simple example I can think of) I'd love to hear them. Novel ideas in technical papers are often expressed with formulas which aren't that complicated conceptually, but are critical to understanding the whole paper and that was another piece I was having very mixed results with.

authorfly
No worries. Sure, as to Formulas... I suspect many of them are LaTeX. If it is possible to parse that, it could help? At sufficient picture quality, vision models can accurately parse images of formulas to photos.

Neither will probably help you with a "readable" formula system because in my experience the readers that do this for LaTeX or normal formula text have flaws any way (it's also slightly cultural and dependent on field of study). Maybe the best bet is a prompt to a vision model with "read this formula out loud in a digestible, understandable concise way".. though this may have issues with the recall accuracy.

freethejazz
Depending on how much structure you want to extract before passing the pdf contents to the next step in your pipeline, this paper[1] might be helpful in surfacing more options. It's a review/benchmark of numerous tools applied to the information extraction of academic documents. I haven't been through to evaluate the solutions they examined, but it's how I discovered GROBID and IMO lays out the strengths of each approach clearly.

[1] https://arxiv.org/pdf/2303.09957

postepowanieadm
mupdf's mutool gives access to most data of all solutions I have checked.