On certain abstract tests it might be on par, but in terms of instructability and general coherence I've found it very lacking. I tried using it in an application I'm developing and was quite disappointed: https://youtu.be/mzAij61O2L0
In identifying categories and repopulating them Gemini Pro seems to be slightly more diverse than GPT-3.5-Turbo, while the latter tries harder to stay within the constraints given.
It's honestly sad to watch. Google has the talent and research to be smoking everyone in this space, but it seems like they have zero product or marketing leadership/vision. For years they've been shipping half baked products built on killer tech that are destined to fail.
By the way, I linked to this report when it came out, but was labeled a "dupe" because someone else had already posted a link to the flashy website without any technical details.
Google's misleading demonstration video shows they can't be trusted to honestly evaluate their own model. Until we can actually get our hands on Ultra and try for ourselves, I wouldn't give too much credit to their numbers.
"Important benchmarks" aren't the same as being useful and reliable. I tried using Gemini Pro for tasks that GPT-3.5 does very well on and it fell apart completely. From reading the reviews of people using it, I get the impression Gemini Pro is good at writing marketing copy and other stuff that GPT-3 was good at, and I suspect a lot of benchmarks are over focused on that same text completion modality.
Honestly Gemini Pro is so unimpressive I expect even GPT-3.5 will beat Gemini Ultra in terms of utility, at least for the next 6 months.
I have been playing around with various GPTs lately by getting them to help me with my management consulting work. Usually analyzing docs and merging them or two me what the overlap is, etc.
My experience with Gemini pro is that it often completely and spectacularly misunderstands the ask. No other GPT comes close to getting it that wrong. As a result, I don't have a high level of confidence in it.
I find Mixtral to be quite impressive. It is by far the fastest and gives me good output. At least for big documents kind of work.
> Our experiments across four LLMs and two MLLMs demonstrate Gemini's competitive commonsense reasoning capabilities.
I couldn't glean from the post; does anyone know what the test suite is?
I notice failure in common sense reasoning quite often in Bard and ChatGPT and while I haven't done a systematic analysis of the categories of mistakes, some patterns I've seen include: lack of self-reflection, not noticing internal inconsistency in output, lack of awareness of what would be "common" common sense, etc.
The original ‘think step by step’ also made this observation.
Other models provide better answers if you tell them it’s important for your career or that you’ll tip them $200 if they’re right, or if you tell them that it’s allowed to not know… the preamble to get max performance gets long.
It's the other way around, that they work on humans is suspect to it working on LLMs. These LLMs are trained on predicting what the human training data would say next. Since these kind of things often result in the following responses in the data being different it results in the model learning that difference in what comes next as well.
I was surprised to see in the paper that they appear to ask for an answer to a multiple-choice question, and only after getting the answer to the multiple-choice question do they ask the model for an explanation. The model will do much better if you ask it to explain its reasoning before it produces its answer to the multiple-choice question.
The reason this works is it puts the reasoning into the context window. Transformers pay attention to everything in the context window and having a rationale sitting there - even if the model generates the rationale - helps the model do a better job of issuing the correct multiple choice answer.
Or like giving the machine time and tape to perform computation. Never tried that but an interesting experiment would be making it output irrelevant things - it could also help to get the right answer just due to the time and space.
Funnilly enough this also works with human children. When you ask them something their first attempt us a guess. Only when you ask them to answer and explain the answer they start thinking.
Good initiative but it's not a fair comparison IMO. The Gemini Ultra is the one that should've been compared to GPT-4. And, yes, I know it's not available behind an API yet. But does it justify comparing a < 20 billion params model to a model possibly 50x-100x larger without even mentioning this evident gap in their analysis? In the light of scaling law, I believe no.
Do we actually know that Gemini Pro is < 20 billion params? Their paper revealed Nano-1 is 1.8B and Nano-2 is 3.25B, but they seemed to omit any details about Gemini Pro or Ultra, and I couldn't find any further information online beyond just rumors and wildly ranging speculation
Not the person you replied to, but… Microsoft apparently revealed that GPT-3.5 Turbo is 20 billion parameters. Gemini Pro seems to perform only slightly better than GPT-3.5 Turbo according to some benchmarks, and worse in others. If Gemini Pro is significantly larger than 20 billion, that would be embarrassing for Google. If it is significantly smaller, that would be good for Google.
It seems reasonable to me to assume it’s somewhere in the neighborhood of 20 billion, but I agree it is worthwhile to recognize that we don’t actually know.
I don't think it would necessarily be embarrassing for Google because Gemini Pro is multimodal, while GPT-3.5 Turbo is text-only. Given this difference it wouldn't seem too unrealistic to me if Gemini Pro was bigger, but it seems like we just don't know.
Even so, Google treats the Gemini Pro Vision model as a separate model from Gemini Pro, so it could have separate parameters that are dedicated to vision (like CogVLM does), and that wouldn’t impact the size of the model as far as text-tasks are concerned.
38 comments
[ 2.9 ms ] story [ 49.3 ms ] threadThen, when they do get over the 3.5 line, they will just graveyard it anyway.
They've fully earned a reputation as a marketing parasite.
This is the impression I have of google as well. Especially in the AI space.
Remember that “AI” scheduling a haircut or something that google presented in 2017?
That doesn’t mean you’re any good at building cars.
See https://storage.googleapis.com/deepmind-media/gemini/gemini_...
By the way, I linked to this report when it came out, but was labeled a "dupe" because someone else had already posted a link to the flashy website without any technical details.
I cannot believe someone in Google approved that video but then again I cannot believe that a company would be launching 10 different chat apps.
Honestly Gemini Pro is so unimpressive I expect even GPT-3.5 will beat Gemini Ultra in terms of utility, at least for the next 6 months.
My experience with Gemini pro is that it often completely and spectacularly misunderstands the ask. No other GPT comes close to getting it that wrong. As a result, I don't have a high level of confidence in it.
I find Mixtral to be quite impressive. It is by far the fastest and gives me good output. At least for big documents kind of work.
It just added support for configuring an arbitrary number of custom endpoints for your locally hosted models, and is improving a ton.
https://docs.librechat.ai/index.html
I couldn't glean from the post; does anyone know what the test suite is?
I notice failure in common sense reasoning quite often in Bard and ChatGPT and while I haven't done a systematic analysis of the categories of mistakes, some patterns I've seen include: lack of self-reflection, not noticing internal inconsistency in output, lack of awareness of what would be "common" common sense, etc.
To get to this, I asked him to predict the return value of some simple python function (for """def func(a,b): return a*3 + 2*b)""" what's func(2)).
without requiring the explanation, the answers were not only wrong but also changed between attempts for the same input.
Asking the model to provide an explanation, however, resulted in it giving the correct answer each time.
This method still allows for improvement in the quality of answers in more complex problems, although at a bit lower correctness rate.
Other models provide better answers if you tell them it’s important for your career or that you’ll tip them $200 if they’re right, or if you tell them that it’s allowed to not know… the preamble to get max performance gets long.
The reason this works is it puts the reasoning into the context window. Transformers pay attention to everything in the context window and having a rationale sitting there - even if the model generates the rationale - helps the model do a better job of issuing the correct multiple choice answer.
It seems reasonable to me to assume it’s somewhere in the neighborhood of 20 billion, but I agree it is worthwhile to recognize that we don’t actually know.
Even so, Google treats the Gemini Pro Vision model as a separate model from Gemini Pro, so it could have separate parameters that are dedicated to vision (like CogVLM does), and that wouldn’t impact the size of the model as far as text-tasks are concerned.