Really interesting piece, the bit about tight vs loose bounding boxes got me thinking. Small inaccuracies can add up fast, especially in edge cases or when training on limited data.
Has anyone here found good ways to handle bounding box quality in noisy datasets? Do you rely more on human annotation or clever augmentation?
Thanks for this post — I'm doing something similar for a personal/hobby project (just trying to work with very old scanned PDFs in Sanskrit etc), and the bounding box next to "Sub-TOI" in your screenshot (https://www.sergey.fyi/images/bboxes/annotated-filing.webp) is like something I'm encountering too: it clearly “knows” that there is a box of a certain width and height, but somehow the box is offset from its actual location. Do you have any insights into that kind of thing, and did anything you try fix that?
For what it's worth, I worked around this by first calling Google's older OCR api (Cloud Vision), which gives the coordinates and bounding boxes for each word in the document. Then I pass this whole response to Gemini, and it's finally able to give me much more reasonable bounding boxes. Clearly overkill, and would be ridiculously expensive to do at scale, but works for me.
Given that the author is using the specific `box_2d` format, it suggests that he is taking advantage of this feature, so I wanted to highlight it. My intuition is that a base multimodal LLM without this type of post-training would have much worse performance.
Thanks for this post; it's inspiring — for a personal project I'm trying just to get bounding boxes from scanned PDF pages (around paragraphs/verses/headings etc), and so far did not get great results. (It seems to recognize the areas but then the boxes are offset/translated by some amount.) I only just got started and haven't looked closely yet (I'm sure the results can be improved, looking at this post), but I can already see that there are a bunch of things to explore:
- Do you ask the multimodal LLM to return the image with boxes drawn on it (and then somehow extract coordinates), or simply ask it to return the coordinates? (Is the former even possible?)
- Does it better or worse when you ask it for [xmin, xmax, ymin, ymax] or [x, y, width, height] (or various permutations thereof)?
- Do you ask for these coordinates as integer pixels (whose meaning can vary with dimensions of the original image), or normalized between 0.0 and 1.0 (or 0–1000 as in this post)?
- Is it worth doing it in two rounds: send it back its initial response with the boxes drawn on it, to give it another opportunity to "see" its previous answer and adjust its coordinates?
I ought to look at these things, but wondering: as you (or others) work on something like this, how do you keep track of which prompts seem to be working better? Do you log all requests and responses / scores as you go? I didn't do that for my initial attempts, and it feels a bit like shooting in the dark / trying random things until something works.
I might be completely off here but it kinda feels like Multimodal LLMs is our silver bullet to applying different technological solutions? From text analysis, to video generation to bounding boxes, it's kinda incredible!
And hopefully with diffusion based llms, we might even see real-time appliances?
I wonder how the power consumption compares. I’d expect the classic CNN to be cheaper just because it is more specialized.
> The allure of skipping dataset collection, annotation, and training is too enticing not to waste a few evenings testing.
How’s annotation work? Do you actually have to mark every pixel of “the thing,” or does the training process just accept images with “a thing” inside it, and learn to ignore all the “not the thing” stuff that tends to show up. If it is the latter, maybe Gemini with it’s mediocre bounding boxes could be used as an infinitely un-bore-able annotater instead.
Not directly related but still kind of, I've more or less settled on Gemini lately and often use it "for fun", not to do the task but see if it could do it better than me and or in novel or efficient way. NotebookLM and Canvas work nicely and it felt easy to use.
I've been absurdly surprised at how good it is at things, and how bad it is at others, and notably that the thing it seems the worst at are the easy picking parts.
Let me give an exemple; I was checking with it the payslip of my employees for the last few months, various wires related to their salaries and the various taxes, and my social declaration papers for labor taxes (which in France are very numerous and complex to follow).I had found a discrepency in a couple of declaration that ultimately led to a few dozen euros losts over some back and forth. Figuring it out by myself took me a while, and was not fun; I had the right accounting total and almost everything was okay, and ultimately it was a case of a credit being applied while an unrelated malus was also applied, both to some employees but not others, and the collision meant a pain to find.
Providing all the papers to gemini and asking it to check if everything was fine, it found me a bazillion "weird things", all mostly correct but worth checking, but not the real problem.
Giving it the same papers, telling him the problem I had and where to look without being sure, it found it for me with decent details, making me confident that next time I can use it not to solve it, but to be put on the right track much much faster than without gemini.
Giving it the same papers, the problem but also the solution I had but asking it to give me more details, again provided great result and actually helped me clarified which lines collided in which order, again not a replacement but a great add on. Definitely felt like the price I'm paying for it is worth it.
But here is the funny part : in all of those great analysis, it kept trying to tally me totals, and there was always one wrong. We're not talking impressive stuff here, but quite literal case of here is a 2 column 5 rows table of data, and here is the total, and the total is wrong, and I needed to ask it like 3 or 4 times in a row to fix its total until it agreed / found its issue (which was, literally).
Despite being a bit amused (and intrigued) at the "show thinking" detail of that, where I saw it do the same calculation in half a dozen different way to try and find how I came up with my number, it really showed to me how weirdly different from us those thing work (or "think", some would say).
It it's not thinking but just emergent behavior for text assimilation, which it's supposed to be, then it figuring it something like that in such details and clarity was impressive in a way I can't quite grasp. But if it's not that but a genuine thought process of some sort, how could he miss so many time the simplest thing beside being told.
I don't really have a point here, other than I used to know where I sat on "are the models thinking or not" and the waters have really been murkied for me lately.
There have been lots of talk about these things replacing employees or not, and I don't see how they could, but I also don't see how an employee without one could compete with one helped by one as an assistant; "throw ideas at me" or "here is the result I already know but help me figure out why". That's where they shine very brightly for me.
We benchmarked Gemini 2.5 on 100 open source object detection datasets in our paper: https://arxiv.org/abs/2505.20612 (see table 2)
Notably, performance on out of distribution data like those in RF100VL is super degraded
It worked really well zero-shot (comparatively to the foundation model field) achieving 13.3 average mAP, but counterintuitively performance degraded when provided visual examples to ground its detections from, and when provided textual instructions on how to find objects as additional context. So it seems it has some amount of object detection zero-shot training, probably on a few standard datasets, but isn't smart enough to incorporate additional context or its general world knowledge into those detection abilities
I wish temperature was a dimension. I believe the Gemini docs even recommend avoiding t=0 to avoid the kinds of spirals the author was talking about with masks.
Genuine question: How does this work? How does an LLM do object detection? Or more generally, how does an LLM do anything that is not text? I always thought tasks like this are usually just handed to an other (i.e. vision) model, but the post talks about it as if it's the _same_ model doing both text generation and vision. It doesn't make sense to me why would Gemini 2 and 2.5 would have different vision capabilities, shouldn't they both have access to the same, purpose trained state of the art vision model?
i find these discussions comparing the "vision language models" to the old computer vision tech pretty interesting
since there are still strengths the computer vision has, i wonder why someone hasn't made an "über vision language service" that just exposes the old CV APIs as MCP or something, and have both systems work in conjunction to increase accuracy and understanding
One thing that has surprised me (and I should've known that it wasn't great at it), but it is terrible at creating bounding boxes around things it's not trained on (like bounding parts on a PCB schematic.)
Oh yes, its been good for a while. When we created our Android-use[1] (like computer use) tool, it was the cheapest and the best option among Openai, Claude, llama etc.
We have a planner phase followed by a "finder" phase where vision models are used. Following is the summary of our findings for planner and finder. Some of them are "work in progress" as they do not support tool calling (or are extremely bad at tool calling).
+------------------------+------------------+------------------+
| Models | Planner | Finder |
+------------------------+------------------+------------------+
| Gemini 1.5 Pro | recommended | recommended |
| Gemini 1.5 Flash | can use | recommended |
| Openai GPT 4o | recommended | work in progress |
| Openai GPT 4o mini | recommended | work in progress |
| llama 3.2 latest | work in progress | work in progress |
| llama 3.2 vision | work in progress | work in progress |
| Molmo 7B-D-4bit | work in progress | recommended |
+------------------------+------------------+------------------+
This isn't surprising at all - most VLMs today are quite poor on localization even though they've been explicitly post-trained on object detection tasks.
One insight that the author calls out is the inconsistencies in coordinate systems used in post-training these - you can't just swap models and get similar results. Gemini uses (ymin, xmin, ymax, xmax) integers b/w 0-1000. Qwen uses (xmin, ymin, xmax, ymax) floats b/w 0-1. We've been evaluating most of the frontier models for bounding boxes / segmentation masks, and this is quite a footgun to new users.
One of the reasons we chose to delegate object-detection to specialized tools is essentially due to the poor performance (~0.34 mAP w/ Gemini to 0.6 mAP w/ DETR like architectures). Check out this cookbook [1] we recently released, we use any LLM to delegate tasks like object-detection, face-detection and other classical CV tasks to a specialized model while still giving the user the dev-ex of a VLM.
I'm rather puzzled by how bad the COCO ground truth is. This is the benchmark dataset for object detection? Wow. I would say Gemini's output is better than the ground truth in most of the example images.
Cool post. We did a similar evaluation for document segmentation using the DocLayNet benchmark from IBM: https://ds4sd.github.io/icdar23-doclaynet/task/ but on modern document OCR models like Mistral, OpenAI, and Gemini. And what do you know, we found similar performance -- DETR-based segmentation models are about 2x better.
can't there be specialise models that gemini can take help from just like humans do take help from another human ? I have guts feeling that adding everything in single model may degrade its overall competence. For almost all model, as I go deeper, ( and most here must have experienced it. ) I clearly recognize that gemini ( or most other well known ) starts to fail at task, so subtly that you may think it is correct but it is not. This is dangerous, and increases work of verification at our end instead of reducing the work.
24 comments
[ 2.5 ms ] story [ 69.0 ms ] threadHas anyone here found good ways to handle bounding box quality in noisy datasets? Do you rely more on human annotation or clever augmentation?
Gemini is really impressive at these kinds of object detection tasks.
https://www.sergey.fyi/articles/using-gemini-for-precise-cit...
Given that the author is using the specific `box_2d` format, it suggests that he is taking advantage of this feature, so I wanted to highlight it. My intuition is that a base multimodal LLM without this type of post-training would have much worse performance.
- Do you ask the multimodal LLM to return the image with boxes drawn on it (and then somehow extract coordinates), or simply ask it to return the coordinates? (Is the former even possible?)
- Does it better or worse when you ask it for [xmin, xmax, ymin, ymax] or [x, y, width, height] (or various permutations thereof)?
- Do you ask for these coordinates as integer pixels (whose meaning can vary with dimensions of the original image), or normalized between 0.0 and 1.0 (or 0–1000 as in this post)?
- Is it worth doing it in two rounds: send it back its initial response with the boxes drawn on it, to give it another opportunity to "see" its previous answer and adjust its coordinates?
I ought to look at these things, but wondering: as you (or others) work on something like this, how do you keep track of which prompts seem to be working better? Do you log all requests and responses / scores as you go? I didn't do that for my initial attempts, and it feels a bit like shooting in the dark / trying random things until something works.
And hopefully with diffusion based llms, we might even see real-time appliances?
> The allure of skipping dataset collection, annotation, and training is too enticing not to waste a few evenings testing.
How’s annotation work? Do you actually have to mark every pixel of “the thing,” or does the training process just accept images with “a thing” inside it, and learn to ignore all the “not the thing” stuff that tends to show up. If it is the latter, maybe Gemini with it’s mediocre bounding boxes could be used as an infinitely un-bore-able annotater instead.
I've been absurdly surprised at how good it is at things, and how bad it is at others, and notably that the thing it seems the worst at are the easy picking parts.
Let me give an exemple; I was checking with it the payslip of my employees for the last few months, various wires related to their salaries and the various taxes, and my social declaration papers for labor taxes (which in France are very numerous and complex to follow).I had found a discrepency in a couple of declaration that ultimately led to a few dozen euros losts over some back and forth. Figuring it out by myself took me a while, and was not fun; I had the right accounting total and almost everything was okay, and ultimately it was a case of a credit being applied while an unrelated malus was also applied, both to some employees but not others, and the collision meant a pain to find.
Providing all the papers to gemini and asking it to check if everything was fine, it found me a bazillion "weird things", all mostly correct but worth checking, but not the real problem.
Giving it the same papers, telling him the problem I had and where to look without being sure, it found it for me with decent details, making me confident that next time I can use it not to solve it, but to be put on the right track much much faster than without gemini.
Giving it the same papers, the problem but also the solution I had but asking it to give me more details, again provided great result and actually helped me clarified which lines collided in which order, again not a replacement but a great add on. Definitely felt like the price I'm paying for it is worth it.
But here is the funny part : in all of those great analysis, it kept trying to tally me totals, and there was always one wrong. We're not talking impressive stuff here, but quite literal case of here is a 2 column 5 rows table of data, and here is the total, and the total is wrong, and I needed to ask it like 3 or 4 times in a row to fix its total until it agreed / found its issue (which was, literally).
Despite being a bit amused (and intrigued) at the "show thinking" detail of that, where I saw it do the same calculation in half a dozen different way to try and find how I came up with my number, it really showed to me how weirdly different from us those thing work (or "think", some would say).
It it's not thinking but just emergent behavior for text assimilation, which it's supposed to be, then it figuring it something like that in such details and clarity was impressive in a way I can't quite grasp. But if it's not that but a genuine thought process of some sort, how could he miss so many time the simplest thing beside being told.
I don't really have a point here, other than I used to know where I sat on "are the models thinking or not" and the waters have really been murkied for me lately.
There have been lots of talk about these things replacing employees or not, and I don't see how they could, but I also don't see how an employee without one could compete with one helped by one as an assistant; "throw ideas at me" or "here is the result I already know but help me figure out why". That's where they shine very brightly for me.
> Sometimes Gemini is better than the ground truth
That ain’t ground truth, that’s just what MS-COCO has.
See also https://en.wikipedia.org/wiki/Ground_truth.
Notably, performance on out of distribution data like those in RF100VL is super degraded
It worked really well zero-shot (comparatively to the foundation model field) achieving 13.3 average mAP, but counterintuitively performance degraded when provided visual examples to ground its detections from, and when provided textual instructions on how to find objects as additional context. So it seems it has some amount of object detection zero-shot training, probably on a few standard datasets, but isn't smart enough to incorporate additional context or its general world knowledge into those detection abilities
since there are still strengths the computer vision has, i wonder why someone hasn't made an "über vision language service" that just exposes the old CV APIs as MCP or something, and have both systems work in conjunction to increase accuracy and understanding
We have a planner phase followed by a "finder" phase where vision models are used. Following is the summary of our findings for planner and finder. Some of them are "work in progress" as they do not support tool calling (or are extremely bad at tool calling).
1. https://github.com/BandarLabs/clickclickclickOne insight that the author calls out is the inconsistencies in coordinate systems used in post-training these - you can't just swap models and get similar results. Gemini uses (ymin, xmin, ymax, xmax) integers b/w 0-1000. Qwen uses (xmin, ymin, xmax, ymax) floats b/w 0-1. We've been evaluating most of the frontier models for bounding boxes / segmentation masks, and this is quite a footgun to new users.
One of the reasons we chose to delegate object-detection to specialized tools is essentially due to the poor performance (~0.34 mAP w/ Gemini to 0.6 mAP w/ DETR like architectures). Check out this cookbook [1] we recently released, we use any LLM to delegate tasks like object-detection, face-detection and other classical CV tasks to a specialized model while still giving the user the dev-ex of a VLM.
[1] https://colab.research.google.com/github/vlm-run/vlmrun-cook...
Disclosure: I work for https://aryn.ai/