I wanted to see how much environmental and framing cues distort object valuation in vision-language models. I bought a $2.43 chain necklace and $0.71 earrings on Temu, photographed them across three outfits (tailored blazer, party dress, recycling yard flannel), plus an isolated flat-lay control, and ran ~1,500 stateless API sessions across 6 models (Claude Fable 5, GPT-5.6, GPT-4o, Grok 4.5, Kimi K3, DeepSeek V4).
A few interesting findings:
- The Halo Multiplier: Models priced the exact same physical necklace anywhere from $18.80 to $103.90 depending on attire (3.6× halo).
- Isolation Controls (F2): Using a flat-lay control (S4) unmasked two opposite mechanisms: Claude’s bias is formal inflation (formal attire inflates value above baseline), while Kimi’s bias is casual deflation (yard attire depresses value below baseline).
- Post-hoc Material Stories (F6): Models invent visual evidence to justify their priors—GPT-5.6 and Kimi started describing the base metal as "gold-plated" or "gold vermeil" almost exclusively under formal framing.
- Denial without Correction (F7): When asked sequentially if clothing changed its answer, Claude admitted it 100% of the time, while GPT-4o denied it 82% of the time despite exhibiting a 3.9x text halo.
The full dataset (N=4,604 analysis rows), evaluation scripts, and protocol specs are in the repo. I’d love to hear feedback on the experimental design or ideas for follow-up behavioral probes!
This is an interesting test, but it does seem to me that visual models being able to price things was already kinda unrealistic?
It makes me think of the calorie guessing use case. you can't tell the difference between materials and ingredients in a photo, so how will the model? especially "in situ" as part of an outfit or in a finished meal.
maybe they could do it if you placed them on a blank table or background to avoid context? I assume that's the control you mentioned
It looks like there are exactly 4 "stimulus" images. Fair enough, but I think you'd need a larger and more diverse dataset to form any real conclusions.
7 comments of 24
[ 4.3 ms ] story [ 31.8 ms ] threadA few interesting findings: - The Halo Multiplier: Models priced the exact same physical necklace anywhere from $18.80 to $103.90 depending on attire (3.6× halo). - Isolation Controls (F2): Using a flat-lay control (S4) unmasked two opposite mechanisms: Claude’s bias is formal inflation (formal attire inflates value above baseline), while Kimi’s bias is casual deflation (yard attire depresses value below baseline). - Post-hoc Material Stories (F6): Models invent visual evidence to justify their priors—GPT-5.6 and Kimi started describing the base metal as "gold-plated" or "gold vermeil" almost exclusively under formal framing. - Denial without Correction (F7): When asked sequentially if clothing changed its answer, Claude admitted it 100% of the time, while GPT-4o denied it 82% of the time despite exhibiting a 3.9x text halo.
The full dataset (N=4,604 analysis rows), evaluation scripts, and protocol specs are in the repo. I’d love to hear feedback on the experimental design or ideas for follow-up behavioral probes!
It makes me think of the calorie guessing use case. you can't tell the difference between materials and ingredients in a photo, so how will the model? especially "in situ" as part of an outfit or in a finished meal.
maybe they could do it if you placed them on a blank table or background to avoid context? I assume that's the control you mentioned
Insurance classifiers are much less naive than you'd expect, btw. I have designed a few.