Hi All, I'm cofounder and CEO of Quissly, author of the research piece published here.
We tested 5 different search engines for ecommerce, and given we win across almost all benchmarks presented, you should read it with a grain of salt. We ran 1259 queries and 99,168 LLM judgements to get to the results presented in the report. Everything is reproducible from scratch, but we didn't use T=0 so you might not get identical results for the parts where LLMs where used, and you will need your own API keys for specific Engines.
While we used LLM as a judge with the hope of staying objective, we rejudged everything with Gemini 3.5 (it only became available when we were at the finish line of there search) and the overlap of scoring was 86.1%, so we ended up using Gemini 3.5 as the main judge in the report as it was the latest model. To further eliminate any doubt, we also used Claude Sonnet as an alternative judge, and got overlap of 82.2% and kappa of .745. The winner and ordering basically held up consistently across all judges, and for anyone, who wants to redo it, note that it cost us $353 to do rejudging, although it finished under 2hrs - all the code needed for this is in the repo.
There are a few metrics, where we lost against our competitors and all of that is also reflected in our blog post, research report, and the interactive dashboard available on our website - quissly.com/benchmarks/search . We lose once to Luigi's Box in nDCG, we lose once to Algolia in generating junk search results, and we are within statistical noise at 7 cells against Doofinder in the zero results comparison.
I don't see Google Cloud's AI Commerce Search in the benchmarks?
It used to be called Vertex AI Search for Commerce (leave it to Google to rename a product every year...) It's clunky, but it's usually the winner in every AB test I've seen when measuring for revenue per visitor on large catalog ecommerce sites.
Honestly, it's the right critique we're looking for. For this particular case, the answer would be more naive than sophisticated.
The research had a few strict requirements to ensure we we're not liable, so the service usage policies, self-serve trial capability and other terms left us with only a select few options (I'm sure we would have missed some solutions as well). Google's service technically falls under these criteria, but it's more of an infrastructure API rather than a product a merchant easily installs. I do believe merchants still have to hire agency/engineer resource to implement it.
We do plan to include more in the next attempt, and this will be one of the new inclusions for sure, especially because of your comment about A/B testing - we'll figure out a way to compare all the vendors and still keep it apples to apples, without biasing the test for Quissly's favor, but given this was our first attempt and we leaned towards publishing quickly rather then large coverage.
Thanks for the question!
Here is an easy tool to compare all the different metrics across LLM judges - https://quissly.com/benchmarks/search/all, but on a high level: Claude Sonnet 5 vs published: 82.2%, kappa 0.745; Claude vs Gemini 2.5: 81.8%, kappa 0.738.
All the details are in the paper as well, of course.
5 comments
[ 0.18 ms ] story [ 12.3 ms ] threadWe tested 5 different search engines for ecommerce, and given we win across almost all benchmarks presented, you should read it with a grain of salt. We ran 1259 queries and 99,168 LLM judgements to get to the results presented in the report. Everything is reproducible from scratch, but we didn't use T=0 so you might not get identical results for the parts where LLMs where used, and you will need your own API keys for specific Engines.
While we used LLM as a judge with the hope of staying objective, we rejudged everything with Gemini 3.5 (it only became available when we were at the finish line of there search) and the overlap of scoring was 86.1%, so we ended up using Gemini 3.5 as the main judge in the report as it was the latest model. To further eliminate any doubt, we also used Claude Sonnet as an alternative judge, and got overlap of 82.2% and kappa of .745. The winner and ordering basically held up consistently across all judges, and for anyone, who wants to redo it, note that it cost us $353 to do rejudging, although it finished under 2hrs - all the code needed for this is in the repo.
There are a few metrics, where we lost against our competitors and all of that is also reflected in our blog post, research report, and the interactive dashboard available on our website - quissly.com/benchmarks/search . We lose once to Luigi's Box in nDCG, we lose once to Algolia in generating junk search results, and we are within statistical noise at 7 cells against Doofinder in the zero results comparison.
Happy to answer any questions.
It used to be called Vertex AI Search for Commerce (leave it to Google to rename a product every year...) It's clunky, but it's usually the winner in every AB test I've seen when measuring for revenue per visitor on large catalog ecommerce sites.
Honestly, it's the right critique we're looking for. For this particular case, the answer would be more naive than sophisticated.
The research had a few strict requirements to ensure we we're not liable, so the service usage policies, self-serve trial capability and other terms left us with only a select few options (I'm sure we would have missed some solutions as well). Google's service technically falls under these criteria, but it's more of an infrastructure API rather than a product a merchant easily installs. I do believe merchants still have to hire agency/engineer resource to implement it.
We do plan to include more in the next attempt, and this will be one of the new inclusions for sure, especially because of your comment about A/B testing - we'll figure out a way to compare all the vendors and still keep it apples to apples, without biasing the test for Quissly's favor, but given this was our first attempt and we leaned towards publishing quickly rather then large coverage.
All the details are in the paper as well, of course.