They did. The KV-Markdown is essentially a dict with ``` wrapper, and INI which is similar scored very high as well. The worst performers were index-based rows like CSV or Markdown tables. JSON is in the middle with high context and more syntactic noise and less clear record labels.
The odd ones to me are HTML which uses th and td to make indexed-based rows but did better than JSON somehow, and XML which is like JSON with even more syntactic noise placing better than INI. If I had to guess I'd say because vast amounts of the web were in the training set.
I am not an expert on the subject but i suggest that you can also save context space by using shorter XML element names (like f instead of function, c instead of class, etc.). Just add a legend at the top or bottom to explain what each abbreviation means, LLMs can figure out the mapping without issues. I use this approach when generating project structure maps with Tree-sitter. I did a quick comparison and didn't notice much degradation with claude, so the context space you save may make it worthwhile. I would be interested to see a proper comparison.
Common enough words like `function` and `class` are generally encoded as a single token by the tokenizer and may provide a slightly better context to the LLM. For openai you can test this stuff at https://platform.openai.com/tokenizer
Not to mention that the least poorly performing format is probably the stupidest way to encode tabular data, beating even XML. But I guess that’s the new normal because we’re trying to shoehorn conversational AI models to every use case rather than, say, training finetunes that are better at particular tasks. (Yes, of course you can’t train finetunes when the model is a proprietary black box on someone else’s computer.) Something about hammers and nails…
The test really needed to be run on multiple data sizes (50, 100, 500, 1000, 5000). The more token efficient formats would probably eventually overtake the token heavy ones due to context pollution. All this test really says is what performs best for 1 particular model at one particular context length.
Bizarre conclusions when on average all the formats perform poorly with average accuracy of 50%. Sure 60% is better than 40% but they are both unusable if you actually care about numbers...
This is an interesting theoretical exercise but please for the love of god don't actually use an LLM to search tabular data. This is a solved problem. Free software does this with 100% accuracy and insane efficiency.
Inputs were not long enough to properly see either of the true wins in terms of reduced token counts for terser formats or their benefits in terms of avoiding stuffing the context window thereby potentially reducing accuracy. The test really needs to be conducted across multiple dimensions!
Tbh I am more interested in processing data and formatting it to tabular forms than extracting data from tabular forms. One of the main uses I see in LLMs is structuring unstructured/semistructured data. I may occasionally feed a table to an LLM and ask such kinds of questions when I feel lazy, but I see no serious application of this as compared with using whatever language/library to process the data from the table (whether using an llm or not in the whole process). The point of having structured data is exactly this. But much more often I feed data to an llm and ask it to create a table.
The current OCR approach typically relies on a Vision-Language Model (VLM) to convert a table into a JSON structure. However, a table inherently has a 2D spatial structure, while Large Language Models (LLMs) are optimized for processing 1D sequential text. This creates a fundamental mismatch between the data representation and the model’s input format.
Most existing pipelines address this by preprocessing the table into a linearized 1D string before passing it to the LLM — a question-agnostic step that may lose structural information.
Instead, one could retain the original table form and, when a question is asked, feed both the question and the original table (as an image) directly into the VLM. This approach allows the model to reason over the data in its native 2D domain, providing a more natural and potentially more accurate solution.
Only testing GPT-4.1-nano makes this basically useless. Most people are almost certainly using GPT-5 mini or better. This very poor analysis is like an LLM literacy test for readers.
This is a bit silly way to use LLMs to process tabular data. In reality, you'd ask it to write functions and execute them. First you'd ask it to create a type definition from the table, then ask it to create functions to process the data.
"Write a function to find years of experience by name? Return just the number, e.g. '12'."
It works much better, and it can single-shot many of the processing requirements just from type definitions it can infer from the data.
This way it's easier to stick to tabular formats that have easy reading libraries, like with TypeScript/JavaScript JSON, and with Python, maybe CSV...
52 comments
[ 2.3 ms ] story [ 61.1 ms ] threadThe odd ones to me are HTML which uses th and td to make indexed-based rows but did better than JSON somehow, and XML which is like JSON with even more syntactic noise placing better than INI. If I had to guess I'd say because vast amounts of the web were in the training set.
> accuracy: 60%
Not to mention that the least poorly performing format is probably the stupidest way to encode tabular data, beating even XML. But I guess that’s the new normal because we’re trying to shoehorn conversational AI models to every use case rather than, say, training finetunes that are better at particular tasks. (Yes, of course you can’t train finetunes when the model is a proprietary black box on someone else’s computer.) Something about hammers and nails…
* Multiple tasks vs 1
* O3/o3-mini + 4o/4o-mini instead of nano
* Extra credit: Inside a fixed cost/length reasoning loop
Ex: does the md-kv benefit disappear with smarter models that you'r typically use, and thus just become a 2-3x cost?
> We only tested OpenAI’s GPT-4.1 nano.
https://ochagavia.nl/blog/configuration-files-are-user-inter...
https://news.ycombinator.com/item?id=45291858 (135 comments)
Most existing pipelines address this by preprocessing the table into a linearized 1D string before passing it to the LLM — a question-agnostic step that may lose structural information.
Instead, one could retain the original table form and, when a question is asked, feed both the question and the original table (as an image) directly into the VLM. This approach allows the model to reason over the data in its native 2D domain, providing a more natural and potentially more accurate solution.
They can transform information in tables but information is lost due to that lack of understanding.
"Write a function to find years of experience by name? Return just the number, e.g. '12'."
It works much better, and it can single-shot many of the processing requirements just from type definitions it can infer from the data.
This way it's easier to stick to tabular formats that have easy reading libraries, like with TypeScript/JavaScript JSON, and with Python, maybe CSV...