While the other response was unflattering but true, a better answer would be that Bard ___ is trained to be a general purpose chat bot, specific for the Bard user experience. While Gemini ___ API is a developer focused API for LLM use. Similar to how ChatGPT performs different to the OpenAI APIs or Copilot in Bing, or whatever. Basically, they’re fine tuned and prompted to respond differently.
Kind of, ChatBot Arena was updated on Dec 7th, so hard to know for sure if Bard is just fine tuned against it.
That being said Chatbot Arena is a pretty wide variety of test scenarios. If fine tuning made the model perfect, all the small models would get similar scores to GPT4, which they don't. Essentially it ranks how people believe a ChatBot should respond, rather than just zero shot, 1 shot and COT type benchmarks.
How would you game it? I think it is clearly the least gameable leaderboard we have. A more valid criticism might be that you don't like the metric it's measuring, but I think it is a useful metric, though certainly not the only useful metric.
I think there is a theoretical threat for models hosted on an external server.
You could ask a question on lmsys, check your server logs for the generated response, go back to lmsys and pick the response that your model generated.
Maybe you could also use a better model for requests from lmsys. E. g. use an unquantized model, disable censorship, etc.
I doubt any of the big players are doing that, but you never know.
I could tell from the second interaction which was Bard Pro because it uses a specific formatting with 2 levels of indentation I never saw in other models.
3000 matches should be more than enough for an Elo-style ranking system to converge. But the notebook[0] linked to from the site includes a section with an equal number of matches being sampled for each pairwise comparison, and the results are pretty much the same.
It's actually better. All the people claiming unannounced updates make OpenAI model significantly worse were misled by their own lying eyes. It's so hard to believe, I know, but it's true.
It isn’t, because performance isn’t scalar. It seems to be a more preferable chatbot in this arena. It is objectively less capable for many other domain specific tasks.
No clue, but it wouldn't surprised me if they identified inputs and training that weren't actually helping. Just as human beings aren't necessarily more helpful by having more varied input, the same seems to apply to LLMs.
The interesting thing is it still has a 128k context length. This is awesome because GPT became way more useful to me once it reached this level of context.
Gemini was a bit disappointing on launch. Good to see them improving.
My own experience doesn't really match the arena's leaderboard. I find myself using Claude 2 much more than GPT 4/Turbo. I prefer its out-of-the-box response style, and its answers for my queries seem just as good, if not better.
Interesting, Kagi (where I use the chatbots) ranks Claude 1 as equal to (and faster than) GPT 4 (non-Turbo), and marks Claude 2's quality as on-par with 4 Turbo (albeit slower). Worth noting it's a simple 1-4-star ranking, unlike the arena's ELO numbers.
Completely concur with your experience. I've always suspected that the questions asked on these arena sites tend towards being a bit different qualitatively than real-world use cases.
I rarely hear people talk about liking Claude. I’m curious, what do you use it for?
I personally use bard for general internet search stuff, because I like how google has set up the linking (and the UI). I use GPT-* for technical stuff (eg GitHub codepilot) because I find it can synthesize code better.
Where GPT really improved for me subjectively is long context - at least since launch. Such ranking can't compare it.
Also I felt I keep getting more personalized results, i.e. models are somehow biased towards user. I heard they plan it, I don't know it's launched, but I feel it.
And there's also fine-tuning in the other direction - my brain got used to ways of interacting with GPT. Same as with Google, I just somehow subconsciously know how to write prompts that get me what I want.
Typically benchmarks have limited aspects they are measuring. I can imagine another suite of benchmarks with longer contexts, but in that case, it might be more difficult to do it in a blind comparison form. At the least, it would be quite costly to run such benchmarks.
Google is one of the most cost conscious companies in tech when it comes to compute costs. Sure they are blind to other kinds of cost like reputation damage due to stupid leadership decisions. But in terms of their tech, they run their servers and their network to very high utilization, definitely exceeding competitors like Amazon.
I don’t really know tbh. I don’t think their current position is because of penny pinching. They invented the transformer and slept on it while OAI ate their lunch.
It absolutely can. Try asking ChatGPT4 something like "Please draw a chart of the population of Switzerland over the past 20 years".
It will write Python code to use matplotlib and the resulting chart will appear in the chat. I'd show an example but their sharing feature doesn't work with images for some reason.
My favourite by miles was asking bard for some more mathematical examples after explaining quantum field theory quite well in words, to which it said "Ok here are some examples of mathematics: 2+2=4"
Wow. I've suspected for a while that Bard's performance has been limited mostly by cost. Google isn't charging for Bard and they didn't want to run a gigantic model for everyone for free forever. Maybe they made a breakthrough in inference cost for their better models? Or maybe they got tired of everyone clowning on them for being behind and decided to eat the cost for a while.
I still think they ought to launch a subscription so we can see their absolute best model running in public.
The trick is to access the "bard-jan-24-gemini-pro" model, available in direct chat mode here: https://chat.lmsys.org/. Significantly better than the prior model.
New information from a Google employee: this new leaderboard entry (Bard - Gemini Pro) is a different fine-tune than the previous one (Gemini Pro - Dev API), but more importantly it "has access to the Internet" which I assume means it uses Google Search when generating answers. I bet this is responsible for the boost!
Does anyone know if the GPT-4 Turbo version used on the leaderboard has access to web search? I always assumed it did not, but now it doesn't seem like an apples-to-apples comparison.
Edit: I used the "Direct Chat" feature on lmsys to ask Bard and GPT-4 Turbo "What is the current price of Bitcoin?". Sure enough GPT-4 Turbo said it can't browse the Internet and Bard gave a real time answer from Google Search. This means GPT-4 outperforms Bard overall even without the ability to browse the web at all. Pretty impressive.
These seem like different categories; one is a model and one is a system with a model plus tools. I think it is useful to compare them, since there is a real difference in user experience. However, they ought to be prominently marked as different categories. And the lmsys guys ought to put a ChatGPT model on the leaderboard with its own search integration enabled, for a fairer comparison. And it would be cool to have other LLM+tools entries like Perplexity, Phind, etc.
It’s pretty good got newer info than gpt4, but refuses on stuff that doesn’t make sense when I ask it to write code using a certain public library, it refuses due to privacy reasons. Where as gpt4 will do it.
Getting close, but this will force OpenAI to come out with gpt5 for another big round of catch up
Moderation means no sex, hate, illegal things and religion. I tried to talk about Allah being merciful and got my ass moderated away. I am Buddhist and when I talked about reincarnation being misunderstood that was fine. So the limits are not clear.
Does not completely answer the question, all aspects of a question, or has a response otherwise cut off. This was common for gpt4 and coding questions it would simply stop responding in the middl
The code snippet is literal - the model will actually write out a comment with something like "// TODO: Implement the rest" instead of actually doing it.
I had a colleague complain to me the other day that he'd asked GPT to help him with a trivial reformatting task and it had said something like "I can't do that but I can guide you on how to do it".
It's not a valid comparison. Bard uses Google (ie the Internet) to answer things, while GPT4 doesn't. So bard can answer things like, "what's the weather in SF today" or "who won the basketball game last night".
> Bard, powered by the Gemini Pro-scale model, debuts at the #2 position on the independent lmsys leaderboard.
According to Jeff Dean's tweet, it looks like they have a new "Gemini Pro-scale model" being rolled out, not sure what it means by "Pro-scale" though. Also not sure if everyone already got it...
Yeah, I was evaluating Bard Pro vs Starling and up to the end I wasn't sure what to pick. Starling 7B was almost as good. People don't realize there are ways to
"mommy, can we get GPT?" "we have GPT at home darling"
Some how my experience doesn't reflect this. I keep going back to GPT if i need an answer. I even say most of the time I prefer 3.5 better than Google Bard. GPT 4 is definitely way better than Bard for sure.
95 comments
[ 4.6 ms ] story [ 155 ms ] threadThat being said Chatbot Arena is a pretty wide variety of test scenarios. If fine tuning made the model perfect, all the small models would get similar scores to GPT4, which they don't. Essentially it ranks how people believe a ChatBot should respond, rather than just zero shot, 1 shot and COT type benchmarks.
You could ask a question on lmsys, check your server logs for the generated response, go back to lmsys and pick the response that your model generated.
Maybe you could also use a better model for requests from lmsys. E. g. use an unquantized model, disable censorship, etc.
I doubt any of the big players are doing that, but you never know.
[0] https://colab.research.google.com/drive/1KdwokPjirkTmpO_P1WB...
But yeah we tried some things.
The interesting thing is it still has a 128k context length. This is awesome because GPT became way more useful to me once it reached this level of context.
My own experience doesn't really match the arena's leaderboard. I find myself using Claude 2 much more than GPT 4/Turbo. I prefer its out-of-the-box response style, and its answers for my queries seem just as good, if not better.
Interesting, Kagi (where I use the chatbots) ranks Claude 1 as equal to (and faster than) GPT 4 (non-Turbo), and marks Claude 2's quality as on-par with 4 Turbo (albeit slower). Worth noting it's a simple 1-4-star ranking, unlike the arena's ELO numbers.
I personally use bard for general internet search stuff, because I like how google has set up the linking (and the UI). I use GPT-* for technical stuff (eg GitHub codepilot) because I find it can synthesize code better.
I gave Bard a go, after seeing Jeff Dean's tweet.
It's just as frustrating as it was, compared to GPT-4. It's simply off the question and unable to realize it's off.
I asked it to generate a chart and 3 times it came back with "here's a chart" with no chart, finally saying it doesn't have that ability.
Also I felt I keep getting more personalized results, i.e. models are somehow biased towards user. I heard they plan it, I don't know it's launched, but I feel it.
And there's also fine-tuning in the other direction - my brain got used to ways of interacting with GPT. Same as with Google, I just somehow subconsciously know how to write prompts that get me what I want.
If this aint anecdotal evidence, I dont know what is. You need to assume a whole lot to draw these conclusions from prompt output comparison.
Heck, just compare a couple outcomes from ChatGPT and you should conclude the same thing, assuming rationality of course.
Also what a wild comparison, because afaik chat gpt can’t make charts either.
Wait so if Google suddenly started charging for Bard, it would be instantly better?
They can spend billions giving away compute (cost) and be extremely efficient (high utilization) in doing so.
It's either to expensive or they just don't think it's that important to have a competitive, accessible model out there (which is a valid stance imo).
It will write Python code to use matplotlib and the resulting chart will appear in the chat. I'd show an example but their sharing feature doesn't work with images for some reason.
I tried this on the free version you can get to without paying anything, I didn't realize the advanced version supported this.
https://i.imgur.com/b0fIGyS.png
Here you go https://imgur.com/a/MT04viM
Disclaimers: I don’t work for Google
I still think they ought to launch a subscription so we can see their absolute best model running in public.
it's better to let more people interact with it because this will help training the model (get more data) so it must be free to use.
I don't know about that second part - but it would make sense that google (and others) may want to use lmsys's arena to benchmark their models.
After all, Human A/B tests are far better then the current automated benchmarks.
I would like more info from lmsys as to how they're accessing these though.
From the vertex ai:
and from the makersuite:https://github.com/dssjon/gemini/blob/main/app.py
Everyone else needs to pay nvidia margins.
Training is murkier as it’s more about the total performance and scalability of the system.
Does anyone know if the GPT-4 Turbo version used on the leaderboard has access to web search? I always assumed it did not, but now it doesn't seem like an apples-to-apples comparison.
https://x.com/asadovsky/status/1750983142041911412?s=20
Edit: I used the "Direct Chat" feature on lmsys to ask Bard and GPT-4 Turbo "What is the current price of Bitcoin?". Sure enough GPT-4 Turbo said it can't browse the Internet and Bard gave a real time answer from Google Search. This means GPT-4 outperforms Bard overall even without the ability to browse the web at all. Pretty impressive.
These seem like different categories; one is a model and one is a system with a model plus tools. I think it is useful to compare them, since there is a real difference in user experience. However, they ought to be prominently marked as different categories. And the lmsys guys ought to put a ChatGPT model on the leaderboard with its own search integration enabled, for a fairer comparison. And it would be cool to have other LLM+tools entries like Perplexity, Phind, etc.
Getting close, but this will force OpenAI to come out with gpt5 for another big round of catch up
Moderation means no sex, hate, illegal things and religion. I tried to talk about Allah being merciful and got my ass moderated away. I am Buddhist and when I talked about reincarnation being misunderstood that was fine. So the limits are not clear.
For example it might tell me to read some documentation and find the answer myself. So I'm thinking "Yes well, what do I need you for then?"
A: Certainly! Here is a function that makes a couple of text edits and a button.
I had a colleague complain to me the other day that he'd asked GPT to help him with a trivial reformatting task and it had said something like "I can't do that but I can guide you on how to do it".
agreed. that's pretty "lazy"
> Bard, powered by the Gemini Pro-scale model, debuts at the #2 position on the independent lmsys leaderboard.
According to Jeff Dean's tweet, it looks like they have a new "Gemini Pro-scale model" being rolled out, not sure what it means by "Pro-scale" though. Also not sure if everyone already got it...
"mommy, can we get GPT?" "we have GPT at home darling"