How is that hard to understand? Yes its gemini 1.5 pro, they haven't released ultra or nano, like this isn't rocket science they didnt introduce Gemini 1.5 ProLight or something, lol its the Pro size model's 1.5 version.
They can't decide on a single name for a chat application so I think expecting them to come up with a sensible naming suggestion is optimistic at best.
Their inability to name things sensibly has been called out for years and it doesn't look like they care?
I'm not sure what the deal is, it has to be a marketing hinderance as every major tech company is trying to claw their way up the AI service mountain. Seems like the first step would be cogent naming.
Google Bard to Google Gemini is what they call Gemini 1.0.
Gemini consists of Gemini Nano, Gemini Pro, & Gemini Ultra.
Gemini Nano is for embedded and portable devices I guess?
The free version of Gemini (gemini.google.com) is Gemini Pro.
The paid version, called Gemini Advanced is using Gemini Ultra.
What we're reading now is about Gemini Pro version 1.0 switching to version 1.5 as of today.
That just made my head spin even more. (Like, I get it, but it's just a very tortuous naming system.) The free version is called Pro, Gemini Advanced is actually Gemini Ultra, the less powerful version upgraded to the more powerful model but the more powerful version is on the less powerful model.
People make fun of OpenAI for not using product names and just calling it "GPT" but at least it's straightforward: 2, 3, 3.5, 4. (On the API side it's a little more complicated since there's "turbo" and "instruct" but that isn't exposed to users, and turbo is basically the default.)
But you don't pay for GPT-4, you pay for a product called ChatGPT Plus, which allows you to write 40 messages to GPT-4 within a three-hour time window, after which you need to switch to 3.5 in the menu.
but if Vertex AI is using Gemini Ultra, then why makersuite (aisuite now? hmmm) showing only "Gemini 1.0 Pro 001" (001: a version inside a version)
and why have makersuite/aisuite in the first place, if Vertex AI is the center for all things AI? and why aitestkitchen?
I'm seeing only Gemini 1.0 Pro on Vertex AI.
So even if I enabled Google Gemini Advanced (Ultra?), enabled Vertex AI API access, I have to first be blessed by Google to access advanced APIs.
It seems paying for their service doesn't mean anything to Google at this point. As a developer, you have to jump through hoops first.
It was probably not a wise choise to give the model itself and the product the same name: "Gemini Advanced is using Gemini Ultra".
Also: "The free version ... is Gemini Pro" - is not what you usually see out there.
Their LLM brand is now Gemini. Gemini comes in three different sizes, Nano/Pro/Ultra.
They recently released 1.0 versions of each, most recently (a few months after Nano and Pro) Ultra.
Today they are introducing version 1.5, starting with the Pro size. They say 1.5 Pro offers comparable performance to 1.0 Ultra, along with new abilities (token window size).
It’s really not. Substitute Gemini for iPhone. Apple releases an iPhone model in mini, standard, and pro lines. They announce iPhone model+1 but are releasing the pro version first. Still difficult?
> Apple releases an iPhone model in mini, standard, and pro lines.
not an iphone user but just looked at iphone 15. Don't see any mini version. I am guess 'standard' is called just 'iphone' ? Is pro same thing as plus ?
Now you’re being intentionally difficult. Do you want it to be cars? Last year $Automaker released $Sedan 2023 in basic, standard, and luxury trims. This year $Automaker announced $Sedan 2024 but so far have only announced the standard trim. If I had meant the iPhone 15 specifically I would’ve said iPhone 15. I think the 12 was the last mini? The point is product families are often released in generations (versions in the case of Gemini) and with different available specs (ultra/pro/nano etc) that may not all be released at the same time.
I think it's the "iPhone +1 Mini is as fast as the old Standard" that confuses people here. This is obvious and expected but not how it's usually marketed I guess ...
So Google will be upgrading the version number of each model at the same time? Based on other comments here, that's not the case - some are 1.5 and some are 1?
Apple doesn't announce the iPhone 12 Mini and compare it to the iPhone 11 Pro.
Gemini Advanced seems to be the brand name for the higher price tier for the end-user frontend that gets you Ultra access, similar how ChatGPT Plus gets you ChatGPT 4.
I get it, but it does beg the question whether you will need Advanced now to get 1.5 Pro. Or does everyone get Pro, making it useless to pay for 1.0 Ultra?
I still don't think it's confusing, but that part is definitely messy.
It's like if Apple announced macOS Blabahee, starting with Mini, not long after releasing Pro and Air touting benefits of Sonoma.
Also, just.. this is how TFA begins:
> Last week, we rolled out our most capable model, Gemini 1.0 Ultra, [...] Our teams continue pushing the frontiers of our latest models with safety at the core. They are making rapid progress. [...] 1.5 Pro achieves comparable quality to 1.0 Ultra
Last week! And now we have next generation. And the wow is that it's comparable to the best of the previous generation. Ok fine at a smaller size, but also that's all we get anyway. Oh and the most capable remains the last generation one. As long as it's the biggest one.
It's almost exactly like Apple, actually, with their M1 and M2 chips available in different sizes, launching at different times in different products.
It's really not that confusing. There are different sizes and different generations, coming out at different times. This pattern is practically as old as computing itself.
I can't even imagine what alternative naming scheme would be an improvement.
Don't go thinking I'm an Apple 'fanboy', I don't have any Apple devices at the moment, but I really can't imagine them launching a next gen product that isn't better than the best of the last gen.
I doubt they launched M2 MBAs while the MBP was running M1, for example. Or more directly, a low-mid spec M3 MBP while the top-spec M2 MBP (I assume that would out-benchmark it?) still for sale and no comparable M3 chip yet.
It's not having the matrix of size/power & generation that's confusing, it's the 'next generation' one initially launched not being the best. I think that's mainly it for me anyway.
> but I really can't imagine them launching a next gen product that isn't better than the best of the last gen.
But they have. The baseline M2 is significantly less powerful than the M1 Max.
What Google's doing is basically exactly like that. It happens all the time that the mid tier of the next generation isn't as good as the top tier of the previous generation. It might even be the norm.
I don't understand what that has to do with anything.
There isn't a set order to things. Sometimes companies release a higher powered version first and then the budget version later, sometimes an entry-level version first and a pro version after. Sometimes both simultaneously. All of these are normal, and can even follow different orders generation to generation.
Google got caught completely flat footed by OpenAI. I'm going to cut them some slack that they want to show the world a bit of flex with their AI chops as soon as they have results.
Yes, my understanding is also there will be a 1.5 Ultra.
It's however nowhere explicitly said that I could find. The Technical Report PDF also avoids even hinting at it.
Advanced is a price/service tier for the end-user frontend. At the moment it gets you 1.0 Ultra access vs. 1.0 Pro for the free version. Similar to how ChatGPT Plus gives you 4 instead of 3.5.
I agree this part is messy. Does everyone who had Pro already get 1.5 Pro? If 1.5 Pro is better than 1.0 Ultra, why pay for Advanced? Is 1.5 Pro behind the Advanced paywall? etc.
Thank you, is more clear to me now. But I also read in some Google announcement about "Gemini Advanced", do you know what is that and the relation with the Nano/Pro/Ultra levels?
Gemini is also the brand name for the end-user web and phone chatbot apps, think ChatGPT (app) vs. GPT-# (model).
Gemini Advanced is the paid subscription service tier that at the moment gets you access to the Ultra model, similar to how a ChatGPT Plus subscription gets you access to GPT-4.
Honestly, they should have called this part Gemini Chat and Gemini Chat Plus, but of course ego won't let them follow the competitor's naming scheme.
Oh, I understand, thank you. To me with the "Gemini Advanced" they screwed the naming scheme.
With an already complex naming for regular consumers (Nano/Pro/Ultra each one with a 1.x), adding this Advanced thing it becomes and spaghetti.
I understand that for most people may be just a chat input and don't care, but if people will consider to pay, they will research a bit and is confusing.
I’m sure they had a discussion that size-based identifiers might imply models are primarily differentiated on the amount of knowledge they have. From that standpoint I don’t agree S/M/L would have been better.
Dear OpenAI please fix your names and versioning. Why do you have GPT-3 and GPT-3.5? What happened between 3 and 3.5? And why isn't GPT-3 a single model? Why are there variations like GPT-3-6.7B and GPT-3-175b? And why is there now a turbo version? How does turbo compared to 4? And what's the relationship between the end-user product ChatGPT and a specific GPT model?
“One of the key differentiators of this model is its incredibly long context capabilities, supporting millions of tokens of multimodal input. The multimodal capabilities of the model means you can interact in sophisticated ways with entire books, very long document collections, codebases of hundreds of thousands of lines across hundreds of files, full movies, entire podcast series, and more.”
This is nice, but it’s hard to judge how nice without knowing more about how much compute and memory is involved in that level of processing. Obviously Google isn’t going to tell us, but without having some idea it’s impossible to judge whether this is an economically sustainable technology on which to start building dependencies in my own business.
Sustainable? The countdown to cancellation on this project is already underway.
"Does it make sense today?" is really the only question you can ask and then build dependencies with the understanding that the entire thing will go away in 3-7 years.
Apple is good about only announcing real products you can buy. They don't do tech demos. It's always, "here's a problem. the new apple watch solves it. here're five other things the watch does. $399."
I agree that Apple does a better job, but wasn't Apple Vision Pro announced 240 days before you could get it? I think it's a pretty safe bet that Gemini 1.5 (or something better) will be available anyone who wants to use it in the next 240 days.
Apple aggressively keeps products under wraps before launch fires employees and vendors for leaking any sort of news to the press .
Also an hardware product that is miles ahead of competition in terms of components and also needs complex setup workflow (for head and eyes) something apple has not done before being 7-8 months after announcing is not really comparable with a SaaS API in terms of delays
They can't do that because only they are the incorruptible stewards empowered with the ability to develop these models, making them accessible to the unwashed masses would be irresponsible!
If that was true, they wouldn't have named it Gemini 1.5 to follow the half-point increment of ChatGPT, they desperately want "people" to care about their product to gain back their mindshare.
Anthropic's Claude targets mostly business use cases and you don't see them write self-congratulating articles about Claude v2.1, they just pushed the product.
I work at a very large company and everyone knows about ChatGPT and Gemini (in part because we for our sins have a good chunk of GCP stuff), but I doubt anyone here not doing some LLM-flavored development has ever even heard of Anthropic, let alone Claude.
And look at how well it's going for Claude. Their primary claim to fame is being called "an annoying coworker" and that's it.
Why would anyone look to form a contract with Anthropic right now? I'd say they're in danger here, because their models and offerings don't have clear value propositions to customers.
I'm generally an excited early adopter, but this kills my excitement immediately. I don't know if Gemini is out (or which Gemini is out) because I've associated Google with "you can't try their stuff", so I've learned to just ignore everything about Gemini.
Google is really good at diluting any possible anticipation hardcore users might have for new stuff they do. 10 years ago I loved when there was a big update to one of their Android apps and I could sideload the apk from the internet to try it out early. Then they made all those changes A/B tests controlled by server side flags that would randomly turn themselves on and off, and there was no way to opt in or out. That was one of the (many) moves that contributed to my becoming disenchanted with Android.
There is a Gemini service that you can use with your Google account, but it's kind of meh as it repeats your input, makes all sorts of assumptions. I am confused as well about the version. There's a link to another premium version (1.5?) on its page, to which I don't have access to without completing a quest which likely ends with a credit card input. That kills it for me.
Or can't use ... I have a newish work account and downloaded Gemini on a Pixel 8 Pro and get "Gemini isn't available" and "Try again later" with no explanation of why not and when.
Look, it now has totally useless suggestions like it was trained on burned out woke IT workers. I asked it about the weather, sea temperature and wave height and period in Malaga, which is much less boring than the choices it came up with. First it tried to talk me out of it waving away responsibility, then it provided useful climate data, which I would have wasted too much time doing a Google search on. I guess it's good for checking on the weather if you can put up with the waivers. Also it knows fishing for garfish in Denmark in May is not a total waste of your time, a great way to experience local culture and a sustainable activity.
I also asked it about the version: "I am currently running on the Gemini Pro 1.01.5 model".
I think the way to understand this is to realize that this isn’t targeted at a Hacker News audience and they don’t care what we think. The world doesn’t revolve around us.
What’s the goal? Maybe, being able to work with partners without it being a secret project that will inevitably leak, resulting in inaccurate stories in the press. What are non-goals? Driving sales or creating anticipation with a mass audience, like a movie trailer or an Apple product launch.
So they have to announce something, but most people don’t read Hacker News and won’t even hear about it until later, and that’s fine with them.
What's worse is that I can't seem to find a way to let Google know where I actually live (as opposed to where I am temporarily traveling, what country my currently inserted SIM card is from etc). And apparently there is no way to do this at all without owning an Android device!
Apple at least lets me change this by moving my iTunes/App Store account, which is its own ordeal and far from ideal, but at least there's a defined process: Tell us where you think you live, provide a form of payment from that place, maybe we'll believe you.
Yeah Google aggressively uses geolocation throughout their services, regardless of your language settings. The flipside of that is that it's really easy to access the latest Gemini or whatever by just using a VPN.
Wait, does that mean if I subscribe to Gemini Pro in country A where it's available (e.g. the US) but travel to Europe, I can't use it?
I'm really frustrated by Google's attitude of "we know better where you are than you do". People travel sometimes and that's not the same thing as moving!
I really, really hate all of these geo heuristics. Sure, don't advertise services to people outside of your market, I get that. Do ask for a payment method from that country too to provide your market-specific pricing if you must.
But once I'm a paying customer, I want to use the thing I'm paying for from where I am without jumping through ridiculous hoops!
The worst variant of this I've seen is when you can neither use nor cancel the subscription from outside a supported market.
To be clear, I didn't pay for any of them. I just signed up for early access to every product that uses some form of ML that can remotely be called "AI"...
Once I got accepted, some of them work outside of the US and some don't
Eh, I think it's about as bad as the OpenAI method of officially announcing something and then "continuously rolling it out to all subscribers" which may be anything between a few days and months.
Yeah. I ended up with an eight letter @gmail.com because I dithered, but if I'd signed up by any means necessary when I'd first heard of it, I would've gotten a four letter one.
I bartered on gmailswap.com, sending someone a bicentennial 50¢ US coin in exchange for an invite.
The envelope made it to the recipient, but the coin fell out in transit because I was young and had no idea how to mail coinage. They graciously gave me the invite anyway.
This is bad practice across the board IMO. There seems to be an idea that this builds anticipation for new products. Sounds good in a PowerPoint presentation by an MBA but doesn't work in practice. Six months (or more!) after joining a waitlist, I'm not seeing it for the first time, so I don't really care when yet another email selling me something hits my inbox. I may not even open the email. This could be mitigated somewhat by at least offering a demo, but that's rare.
As someone who worked in Google Cloud's partnerships team, the way the Early Access Program, not to mention the Alpha --> Beta --> GA launch process for AI products, works, is really dysfunctional. Inevitably what happens is that a few strategic customers or partners get exceptionally early (Alpha) access and work directly with the product team to refine things, fix bugs and iron out kinks. This is great and the way market driven product development should work.
The issues arise with the subsequent stagegate graduation processes, requirements and launches to less restricted markets. It's inconsistent, the QoS pre-GA customers receive is often spotty and the products come with no SLAs, and -- just like Gmail on the consumer side -- things frequently stay in EAP/Beta phase for years with no reliable timeline for launch. ... and then often they're killed before they get to GA, even though they may have been being used by EAP customers for upwards of 1-2 years.
I drafted a new EAP model a few years ago when Google's Cloud AI & Industry Solutions org was in the process of productizing things like the retail recommendation engine and Manufacturing Data Engine, and had all the buy-ins from stakeholders on the GTM side ... but the CAIIS GM never signed off. Subsequently, both the GM & VP Product of that org have been forced out.
In my opinion, this is something Microsoft does very well and Google desperately needs to learn. If they pick up anything from their hyperscaler competitors it should be 1) how to successfully become a market driven engineering company from MSFT and 2) how to never kill products (and not punish employees for only doing KTLO work) from AMZN.
Agreed. OpenAI also doesn't need to grock with Shareholders fearing a GDPR like-fine. Sadly the larger you are the bigger the pain is from small mistakes.
Its because they don't want you to actually use it and see how far behind they are compared to other companies. These announcement are meant to placate investors. "See, we are doing a lot of SotA AI too".
You might be right, but other things from Google tell the same story. For example, I recently tried to get ahold of Pixel 8 Pro. Had to import one from UK, and when I did, turns out that new feature of using thermometer on humans isn't available outside of USA. It doesn't even seem that process to certificate it outside of USA is in play. Google and sales/support just aren't a thing like with Apple, as a contrast. Which is a total shame. I know Google is strong, if not strongest in the game of tech, they just need to get their act together and I believe in them succeeding in that, but sales and support was never in their DNA. Not sure if that can be changed.
I'm more than happy to transfer my monthly $20 to google from OpenAI, on top of my youtube and google one subscription. It's up to Google to take it.
It lets the company control the narrative, without the distraction of fifty tech bloggers test-driving it and posting divergent opinions or findings. Instead, the conversation is anchored to what the company claims about the product.
It's interesting that it's the opposite of the gaming industry. There, because the reviewers dictate the narrative, the industry is better at ferreting out bogus claims. On the flip side, loud voices sometimes steamroll over decent products because of some ideological vendetta.
Totally agree with this. I can see the desire to show off, but I don't understand how anyone can believe this is good marketing strategy. Any initial excitement I get from reading such announcements will be immediately extinguished when I discover I can't use the product yet. The primary impression I receive of the product is "vaporware." By the time it does get released I'll already have forgotten the details of the announcement, lost enthusiasm, and invested my time in a different product. When I'm choosing between AI services, I'll be thinking "no, I can't choose Gemini Pro 1.5 because it's not available yet, and who knows when it will be available or how good it'll be." Then when they make their next announcement, I'll be even less likely to give it any attention.
After the complete farce that was the last 90% faked video of their tech, maybe just give us a text box we can talk to the thing and see it working ourselves next time.
Like it's shocking to me, are management really so clueless they don't realize how far behind they are? This isn't 2010 Google, your not the company that made your success anymore and in a decade the only two sure fire things that will still exist are android and chrome. Search, Maps, Youtube are all in precarious positions that the right team could dethrone.
I believe this is a standard practice in Google whenever they need to launch a change expected to consume huge resources and they cannot reasonably predict the demand. Though I agree that this is a bad PR practice; waitlist should be considered as a compromise, not a PR technique.
These announcements are mainly for investors and other people interested in planning purposes. It's important to know the roadmap. More information is better.
I get that it's frustrating not to be able to play with it immediately, but that's just life. Announcing things in advance is still a valuable service for a lot of people.
Plus tons of people have been claiming that Google has somehow fallen behind in the AI race, so it's important for them to counteract that narrative. Making their roadmap more visible is a legitimate strategy for that.
I wrote off the PS5 because of waitlists. I was surprised to learn just yesterday that they are now actually, honestly purchasable (what I would consider "released").
I guess I let my original impression anchor my long-term feelings about the product. Oh well.
It's probably going to be dead/deprecated in a year, so maybe there's a silver lining to how hard it is to get to use the service. I, for one, wouldn't "build with Gemini".
I don't think I've ever engaged with a product after "joining their waitlist". By the time they end up utilizing that funnel, competitors have already released feature upgrades or new products cannibalizing their offering.
"Studying the limits of Gemini 1.5 Pro's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens"
Well that's really the right question .. what can, and will, Google do with this that can move their corporate earnings needle in a meaningful way? Obviously they can sell API access and integrate it into their Google docs suite, as well as their new Project IDX IDE, but do any of these have potential to make a meaningful impact ?
It's also not obvious how these huge models will fare against increasingly capable open source ones like Mixtral, perhaps especially since Google are confirming here that MoE is the path forward, which perhaps helps limit how big these models need to be.
In the long run it could move the needle in enterprise market share of Workspace and GCP. They have a lot of room to grow and IMO have a far superior product to O365/Azure which could be exacerbated by strong AI products. Only problem is this sales cycle can take a decade or more, and Google hasn’t historically been patient or strategic about things like this.
Outperform is dependent on the RAG approach (and this would be a RAG approach anyways, you can already do this with smaller context sizes). A simplistic one, probably, but dumping in data that you don't need dilutes the useful information, so I would imagine there would be at least _some_ degradation.
But there is also the downside of "tuning" the RAG to return less tokens you will miss extra context that could be useful to the model.
A perfect RAG system would probably outperform everything in a larger context due to prompt dilution, but in the real world putting everything in context will win a lot of the time. The large context system will also almost certainly be more usable due to elimination of retrieval latency. The large context system might lose on price/performance though.
are you going to upload 10M tokens to Gemini on every request? That's a lot of data moving around when the user is expecting a near realtime response. Seems like it would still be better to only set the context with information relevant to the user's prompt which is what plain rag does.
10M tokens is absolutely jaw dropping. For reference, this is approximately thirty books of 500 pages each.
Having 99% retrieval is nuts too. Models tend to unwind pretty badly as the context (tokens) grows.
Put these together and you are getting into the territory of dumping all your company documents, or all your departments documents into a single GPT (or whatever google will call it) and everyone working with that. Wild.
If any of this is remotely true, not only did it catch up, it’s wiping the floor with how useful it can be compared to GPT4. Not going to make a judgement until I can actually try it out though.
In the demo videos gemini needs about a minute to answer long context questions. Which is better than reading thousands of pages yourself. But if it has to compete with classical search and skimming it might need some optimization.
Replacing grep or `ctrl+F` with Gemini would be the user's fault, not Gemini's. If classical search for a job already a performant solution, use classical search. Save your tokens for jobs worthy of solving with a general intelligence!
I think some of the most useful apps will involve combining this level of AI with traditional algorithms. I've written lots of code using the OpenAI APIs and I look forward to seeing what can be done here. If you type, "How has management's approach to comp changed over the past five years?" it would be neat to see an app generate the greps needed to find the appropriate documents and then feed them back into the LLM to answer the question.
The input you give it can be very long. This can qualitatively change the experience. Imagine, for example, copy pasting the entire lord of the rings plus another 100 books you like and asking it to write a similar book...
I doubt it’s smart enough to write another (coherent, good) book based on 103 books. But you could ask it questions about the books and it would search and synthesize good answers.
I just googled it, and the LOTR trilogy apparently has a total of 480,000 words, which brings home how huge 1M is! It'd be fascinating to see how well Gemini could summarize the plot or reason about it.
One point I'm unclear on is how these huge context sizes are implemented by the various models. Are any of them the actual raw "width of the model" that is propagated through it, or are these all hierarchical summarization and chunk embedding index lookup type tricks?
Reading Lord of the Rings, and writing a quality book in the same style, are almost wholly unrelated tasks. Over 150 million copies of Lord of the Rings have been sold, but few readers are capable of "writing a similar book" in terms of quality. There's no reason to think this would work well.
It's how much text it can consider at a time when generating a response. Basically the size of the prompt. A token is not quite a word but you can think of it as roughly that. Previously, the best most LLMs could do is around 32K. This new model does 1M, and in testing they could put it up to 10M with near perfect retrieval.
As the other comment mentions, you can paste the content of entire books or documents and ask very pointed question about it. Last year, Anthropic was showing off their 100K context window, and that's exactly what they did, they gave it the content of The Great Gatsby and asked it questions about specific lines of the book.
Similarly, imagine giving it hundreds of documents and asking it to spot some specific detail in there.
Great explanation. I was amazed when I started using Claude because I could find a recently-transcribed novella, upload it, and ask specific questions. I'm downright giddy to try a 1M+ model.
>Finally, we highlight surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person learning from the same content.
I think this somewhat is mostly due to the ability to handle high context lengths better. Note how Claude 2.1 already highly outperforms GPT-4 on this task.
> This new generation also delivers a breakthrough in long-context understanding. We’ve been able to significantly increase the amount of information our models can process — running up to 1 million tokens consistently, achieving the longest context window of any large-scale foundation model yet.
When OpenAI raced through 3/3.5/4 it was "this team ships" and excitement.
This cargo-cult hate train is getting tiresome. Half the comments on anything Google-related are like this now, and it doesn't add anything to the conversation.
The difference, though, as someone who really doesn't have a particular dog in this fight, is that I can go use GPT-4 right now, and see for myself whether it's as exciting as the marketing materials say.
When OpenAI launched GPT-4, API access was initially behind a waitlist. And they released multiple demo stills of LMM capacilities on launch day that for months were in a limited partner program before they became generally available only 7 months later.
I also want the shiny immediately when I read about it, but I also know when I am acting entitled and don't go spam comment threads about it.
But really, mostly I mean this: It's fine to criticize things, but when half a dozen people have already raised a point in a thread, we don't need more dupes. It really changes signal-to-noise.
Gemini Ultra was announced two months ago. It just launched in the last week. It literally is still the featured post on the AI section of their blog, above this announcement. https://blog.google/technology/ai/
There’s “this team ships” and there’s “ok maybe wait until at least a few people have used your product before you change it all”.
Did OpenAI then announce GPT-5 two weeks after launching GPT-4?
No, of course they didn’t. And you’re comparing one specific feature (image input) and equating it to a whole model’s release date.
Maybe compare apples to apples next time.
People pointing out release/announcement burnout is a reasonable thing; people in general can only deal with the “next new thing” with some breaks to process everything.
yep they hide things throughout the prompt and then ask it about that specific thing, imagine hiding passwords in a giant block of text and then being like, what was bobs password 10 million tokens later.
According to this it's remembering with 99% accuracy, which if you think about it is NUTS, can you imagine reading a 22x 1000 page books, and remembering every single word that was said with 100% accuracy lol
Interestingly, there's a decent chance I'd remember if there was an out of context passage saying "the password is FooBar". I wonder if it would be better to test with minor edits? E.g., "what color shirt was X wearing when..."
i feel you would recognise that more as a quirk of how humans think, remember that LLMs think fundamentally differently to you and i. i would be curious about someone making a benchmark like that and using it to compare as an experiment however
I'm not trying to anthropomorphize the model, but it's not hard to imagine that a model would attribute significance to something completely out of context, and hence "focus" on it when computing attention.
Another possible synthetic benchmark would be to present a list of key value pairs and then ask it for the value corresponding to different keys. Or present a long list of distinct facts and then ask it about them. This latter one could probably be sourced from something like a trivia question and answers data set. I bet there's something like that from Jeopardy.
I think instead you could just do a full doc of relationships. "Tina and Chris have five children named ..."
Then you can ask it who is Tina's (great)^57 grandmother's twice removed cousin on her father's side?
It would have to be able to remember the context of the relationships up and down the document and there'd be nothing to key into as you could ask about any relationship.
Google is somehow truly awful at this. I thought it was funny when branding messes happened in 2017. I cried when they announced "Google Meet (original)." Now I don't even know what to do.
I'm stunned that Google hasn't appointed some "name veto person" that can just say "no, you aren't allowed to have three different things called 'Gemini Advanced', 'Gemini Pro', and 'Gemini Ultra.'" Like surely it just takes Sundar saying "this is the stupidest fucking thing I've ever seen" to some SVP to fix this.
The only thing that's different is the standard people apply to different companies due to their biases. There are more Apple fanboys on HN than Google fans (Of course, since Google's reputation has been going down for quite a while). Therefore Apple gets a pass. Classic double standard.
It’s different because Apple didn’t release the M1 Ultra at the same time as the M2 Pro. That would be confusing to buyers because it wouldn’t be immediately obvious which one is the better purchase, both being new offerings presented to customers at the same time.
It’s understandable that later generations are better and higher tiers are also better, but usually there is some period of time in between generations to help differentiate them. Here we have Google advancing capability on two axes at the same time.
I give them a pass as this field is advancing rapidly. So good for them. But I think it’s a legitimate call that it adds complexity to their branding. It is different.
This is something close to CPU versioning. You have two axis; performance branding and its generation. Nano, Pro and Ultra is something similar to i3, i5 and i7. The numbered versions 1.0, 1.5, ... can be mapped to 13th gen, 14th gen, ... so on. And people usually don't need to understand the generation part this unless they're enthusiasts.
Isn't that usually the case with many products? Like the M3 Pro CPU in the new Macs is more powerful than the M1 Max in the old Macs.
The Nano < Pro < Ultra is an in-revision thing. For their LLMs it's a size thing. Then there's newer releases of Nano, Pro, and Ultra. Some Pro might be better than some older Ultra.
A lot of people seem confused about this but it feels so easy to understand that it's confusing to me that anyone could have trouble.
Adam Osborne’s wife was one of my dad’s patients so I’m not unacquainted with the risk of early announcements. But surely they do not prevent comprehension.
I like that they are rushing with this and don't care enough to make it Gemini 2 or even really release it, to me it looks like they are concerned to share progress.
Hope they do a good job and once OpenAI releases GPT 5 they are competitive with it with their offerings, it will be better for everyone.
Obsolete if you don't take cost in consideration. Having 10 millions of token going through each layer of the LLM is going to cost a lot of money each time. At gpt4 rate that could mean 200 dollars for each inference
Is there any meaningful valuation on OpenAI? It’s not for sale, there is no market.
Google … has no ability to commercialize anything. Their only commercial successes are ads and YouTube. Doing deceptive launches and flailing around with Gemini isn’t helping their product prospects. I wouldn’t take a bet between open ai and anyone, but I also wouldn’t take a bet on Google succeeding commercially on anything other than pervasive surveillance and adware.
> Is there any meaningful valuation on OpenAI? It’s not for sale, there is no market.
Its shares are already for sale on private markets for accredited investors and for a valuation of over $100BN lead by Thrive Capital.
> Google … has no ability to commercialize anything.
Absolute nonsense.
So Google Cloud, Android (Play Store) are not already commercialized? You well know that they are.
> Doing deceptive launches and flailing around with Gemini isn’t helping their product prospects.
Gemini already caught up to (and surpassed) GPT-4V. What is your point?
> I wouldn’t take a bet between open ai and anyone, but I also wouldn’t take a bet on Google succeeding commercially on anything other than pervasive surveillance and adware.
OpenAI's greatest competitor is Google DeepMind which has the advantage of Google's infrastructure to scale up their models quickly and they have direct access to Google's billions. OpenAI cannot afford to make mistakes or delay anything and a single mistake can cost them hundreds of millions of dollars. The majority of the investment from Microsoft is in Azure credits and not in dollars. [0]
> Whether Google has the stomach to put these models out publicly without neutering their creativity or their existing business model is a different discussion.
Google has a serious GPU (well, TPU) build out, and the fact that they're able to train moe models on it means there aren't any technical barriers preventing them from competing at the highest levels
0 trust to what they put out until I see it live. After the last "launch" video which was fundamentally a marketing edit not showing the real product, I don't trust anything coming out of Google that isn't an instantly testable input form.
The videos shown in these demos have clearly learnt from that as they're using a real live product, filmed on their computers with timers in the bottom showing how long the computations take.
I completely share the same views as you after their last video - and it appears that they've learnt their lesson this time.
If you watch the videos in the blog post, you can see it's a screen recording on a computer without any editing/stitching of different scenes together.
It's good to be sceptical but as engineers we should all remain open.
Essentially, the focus seems to be on leveraging the media buzz around Gemini 1.0 by highlighting the development of version 1.5. While GPT-4's position relative to Gemini 1.5 remains unclear, and the specifics of ChatGPT 4.5 are yet to be disclosed, it's worth noting that no official release has taken place until the functionality is directly accessible in user chats.
Google appears to be making strides in catching up.
When it comes to my personal workflow and accomplishing tasks, I still find ChatGPT to be the most effective tool. My familiarity with its features has made it indispensable. The integration of mentions and tailored GPTs seamlessly enhances my workflow.
While Gemini may match the foundational capabilities of LLMs, it falls short in delivering a product that efficiently aids in task completion.
I don't mean this in a bad way, but when I read a comment like yours which includes phrases like "seamlessly enhances my workflow" and "efficiently aids in task completion", I can't help but feel like it's ChatGPT-generated, and if so I think it's a shame, just write like yourself.
But maybe you do, and I am seeing patterns in sand.
> Google appears to be making strides in catching up.
I say it's even more than that. OpenAI had a bigger lead when it released GPT-2 than it does now. They're burning through cash to try to hold on to a lead of a few months over the competition.
If I understand correctly, they're releasing this for Pro but not Ultra, which I think is akin to GPT 3.5 vs 4? Sigh, the naming is confusing...
But my main takeaway is the huge context window! Up to a million, with more than 100k tokens right now? Even just GPT 3.5 level prediction with such a huge context window opens up a lot of interesting capabilities. RAG can be super powerful with that much to work with.
I am reaching a bit, however, I think its a bit of a marketing technique. The Pro 1.5 being compared to the Ultra 1.0 model seems to imply that they will be releasing a Ultra 1.5 model which will presumably have similar characteristics to the new Pro 1.5 model (MOE architecture w/ a huge context window).
Apparently the technical report implies that Ultra 1.5 is a step-up again, I'm not sure it's just context length, that seems to be orthogonal in everything I've read so far.
So Pro and Ultra are from my understanding link to the number of parameters. More parameters means more reasonning capabilities, but more compute needed.
So Pro is like the light and fast version and Ultra the advanced and expensive one.
1-2 years probably. There will still be a question around who determines what "adequately" is for a while though. Presumably even if an LLM can do something in theory you wouldn't actually want it doing anything without human oversight.
And we should keep in mind that to understand a code change in depth is often just as much work as making the change. When review PRs I don't really know exactly what every change is doing. I certain haven't tested it to be 100% certain I understand fully. I'm just checking the logic looks mostly right and that I don't see anything clearly wrong, and even then I'll often need to ask for clarifications why something was done.
I can't imagine LLMs being used in most large code bases for a while yet. They'd probably need to be 99.9% reliable before we can start trusting them to make changes without verifying every line.
Gemini (or whatever google ai) will be all about ads. I’m not adopting this shit. Their whole business model is ads. Why would I adopt a product from a company that only cares about selling more ads?
Not true - Gemini looks to be marketed towards companies, where it's far more profitable to just charge thousands of dollars. Ads wouldn't fund AI usage anyway. GPU's are extremely expensive (even Google's fancy TPU's).
Agreed, people continually forget that Google has fundamentally failed at everything besides selling ads despite decades of moonshots and other attempts to shift the business. Very skeptical that any company getting 80% revenue from ads will be able to resist the pressure to advertise
Probably because EU/national governments have regulations with respect to the safety and privacy of the users, and the purveyors must evaluate the performance of their products against the regulatory standards.
i saw this announcement on twitter and i was excited to check it out, only to see that "we’re offering a limited preview of 1.5 Pro to developers and enterprise customers via AI Studio and Vertex AI".
please google, only announce things when people can actually use it.
616 comments
[ 8.0 ms ] story [ 519 ms ] threadThe 1 million token context window + Gemini 1.0 Ultra level performance seems like it’ll unlock a wide range of incredible use cases!
HN, what are you going to use/build with this?
Gemini Pro, Gemini Ultra... but was 1.0?
now upgraded but again Gemini Pro? jumping from 1.0 to 1.5?
wait but not Gemini Pro 1.5... Gemini "1.5" Pro
What actually happened between 1.0 and 1.5?
I'm not sure what the deal is, it has to be a marketing hinderance as every major tech company is trying to claw their way up the AI service mountain. Seems like the first step would be cogent naming.
Ultra vs pro vs nano with Ultra unlocked by buying Gemini Advanced is confusing.
I'm also not sure why they make base Gemini available after you have Advanced, because presumably there's no reason to use a worse model.
Google Bard to Google Gemini is what they call Gemini 1.0.
Gemini consists of Gemini Nano, Gemini Pro, & Gemini Ultra.
Gemini Nano is for embedded and portable devices I guess? The free version of Gemini (gemini.google.com) is Gemini Pro. The paid version, called Gemini Advanced is using Gemini Ultra.
What we're reading now is about Gemini Pro version 1.0 switching to version 1.5 as of today.
People make fun of OpenAI for not using product names and just calling it "GPT" but at least it's straightforward: 2, 3, 3.5, 4. (On the API side it's a little more complicated since there's "turbo" and "instruct" but that isn't exposed to users, and turbo is basically the default.)
and why have makersuite/aisuite in the first place, if Vertex AI is the center for all things AI? and why aitestkitchen?
I'm seeing only Gemini 1.0 Pro on Vertex AI. So even if I enabled Google Gemini Advanced (Ultra?), enabled Vertex AI API access, I have to first be blessed by Google to access advanced APIs.
It seems paying for their service doesn't mean anything to Google at this point. As a developer, you have to jump through hoops first.
"Gemini 1.0 Ultra, our most sophisticated and capable model for complex tasks, is now generally available on Vertex AI for customers via allowlist."
https://cloud.google.com/blog/products/ai-machine-learning/g...
Their LLM brand is now Gemini. Gemini comes in three different sizes, Nano/Pro/Ultra.
They recently released 1.0 versions of each, most recently (a few months after Nano and Pro) Ultra.
Today they are introducing version 1.5, starting with the Pro size. They say 1.5 Pro offers comparable performance to 1.0 Ultra, along with new abilities (token window size).
(I agree Small/Medium/Large would be better.)
not an iphone user but just looked at iphone 15. Don't see any mini version. I am guess 'standard' is called just 'iphone' ? Is pro same thing as plus ?
https://www.apple.com/shop/buy-iphone/iphone-15
> Still difficult?
yes your example made it even more confusing.
Apple doesn't announce the iPhone 12 Mini and compare it to the iPhone 11 Pro.
Did you watch the announcements for the M2 and M3 pros? They compared it to the previous generations all the time.
Gemini Advanced seems to be the brand name for the higher price tier for the end-user frontend that gets you Ultra access, similar how ChatGPT Plus gets you ChatGPT 4.
I get it, but it does beg the question whether you will need Advanced now to get 1.5 Pro. Or does everyone get Pro, making it useless to pay for 1.0 Ultra?
I still don't think it's confusing, but that part is definitely messy.
This is where it gets confusing IMO.
It's like if Apple announced macOS Blabahee, starting with Mini, not long after releasing Pro and Air touting benefits of Sonoma.
Also, just.. this is how TFA begins:
> Last week, we rolled out our most capable model, Gemini 1.0 Ultra, [...] Our teams continue pushing the frontiers of our latest models with safety at the core. They are making rapid progress. [...] 1.5 Pro achieves comparable quality to 1.0 Ultra
Last week! And now we have next generation. And the wow is that it's comparable to the best of the previous generation. Ok fine at a smaller size, but also that's all we get anyway. Oh and the most capable remains the last generation one. As long as it's the biggest one.
It's really not that confusing. There are different sizes and different generations, coming out at different times. This pattern is practically as old as computing itself.
I can't even imagine what alternative naming scheme would be an improvement.
I doubt they launched M2 MBAs while the MBP was running M1, for example. Or more directly, a low-mid spec M3 MBP while the top-spec M2 MBP (I assume that would out-benchmark it?) still for sale and no comparable M3 chip yet.
It's not having the matrix of size/power & generation that's confusing, it's the 'next generation' one initially launched not being the best. I think that's mainly it for me anyway.
But they have. The baseline M2 is significantly less powerful than the M1 Max.
What Google's doing is basically exactly like that. It happens all the time that the mid tier of the next generation isn't as good as the top tier of the previous generation. It might even be the norm.
There isn't a set order to things. Sometimes companies release a higher powered version first and then the budget version later, sometimes an entry-level version first and a pro version after. Sometimes both simultaneously. All of these are normal, and can even follow different orders generation to generation.
Google got caught completely flat footed by OpenAI. I'm going to cut them some slack that they want to show the world a bit of flex with their AI chops as soon as they have results.
It's however nowhere explicitly said that I could find. The Technical Report PDF also avoids even hinting at it.
Advanced is a price/service tier for the end-user frontend. At the moment it gets you 1.0 Ultra access vs. 1.0 Pro for the free version. Similar to how ChatGPT Plus gives you 4 instead of 3.5.
I agree this part is messy. Does everyone who had Pro already get 1.5 Pro? If 1.5 Pro is better than 1.0 Ultra, why pay for Advanced? Is 1.5 Pro behind the Advanced paywall? etc.
There are three models: nano/pro/ultra and all are at v1.0
There are two tiers of chat service: basic and pro
There is AIStudio from google through which you can interact with / use directly gemini llms.
Chat service Gemini basic (free) uses Gemini Pro 1.0 llm.
Chat service Gemini advanced uses Gemini Ultra 1.0 llm.
What was shown is ~~Ultra~~ Pro 1.5 LLM which is / will be available to select few for preview to be used via AIStudio.
That leaves a question, what's nano for, and is it only used via AIStudio/API?
Jesus, Google..
How this relates to the end-user chat service/price tiers is still unknown.
The best scenario would be that they just move Gemini free and Advanced tiers to Pro 1.5 and Ultra 1.5, I guess.
That's very difficult.
Vertex AI is their developer API platform.
I agree OpenAI is a bit better at launching for customers on ChatGPT alongside API.
Gemini Advanced is the paid subscription service tier that at the moment gets you access to the Ultra model, similar to how a ChatGPT Plus subscription gets you access to GPT-4.
Honestly, they should have called this part Gemini Chat and Gemini Chat Plus, but of course ego won't let them follow the competitor's naming scheme.
With an already complex naming for regular consumers (Nano/Pro/Ultra each one with a 1.x), adding this Advanced thing it becomes and spaghetti.
I understand that for most people may be just a chat input and don't care, but if people will consider to pay, they will research a bit and is confusing.
You see this problem isn't unique to Google.
and if Pro 1.5 is this good holy shit what will Ultra be...
Nano/Pro/Ultra are the model sizes, 1.0 or 1.5 is the version
"Does it make sense today?" is really the only question you can ask and then build dependencies with the understanding that the entire thing will go away in 3-7 years.
The first of it's type AR/VR hardware has, understandably, a longer release cycle. Also, Apple announced early to drive up developer interest.
Apple aggressively keeps products under wraps before launch fires employees and vendors for leaking any sort of news to the press .
Also an hardware product that is miles ahead of competition in terms of components and also needs complex setup workflow (for head and eyes) something apple has not done before being 7-8 months after announcing is not really comparable with a SaaS API in terms of delays
They’re an enterprise software company doing an enterprise sales motion.
Anthropic's Claude targets mostly business use cases and you don't see them write self-congratulating articles about Claude v2.1, they just pushed the product.
I work at a very large company and everyone knows about ChatGPT and Gemini (in part because we for our sins have a good chunk of GCP stuff), but I doubt anyone here not doing some LLM-flavored development has ever even heard of Anthropic, let alone Claude.
Why would anyone look to form a contract with Anthropic right now? I'd say they're in danger here, because their models and offerings don't have clear value propositions to customers.
Seems reasonably similar in tone to the Google post.
Really? Someone ought to tell them.
https://gemini.google.com/
Look, it now has totally useless suggestions like it was trained on burned out woke IT workers. I asked it about the weather, sea temperature and wave height and period in Malaga, which is much less boring than the choices it came up with. First it tried to talk me out of it waving away responsibility, then it provided useful climate data, which I would have wasted too much time doing a Google search on. I guess it's good for checking on the weather if you can put up with the waivers. Also it knows fishing for garfish in Denmark in May is not a total waste of your time, a great way to experience local culture and a sustainable activity.
I also asked it about the version: "I am currently running on the Gemini Pro 1.01.5 model".
What’s the goal? Maybe, being able to work with partners without it being a secret project that will inevitably leak, resulting in inaccurate stories in the press. What are non-goals? Driving sales or creating anticipation with a mass audience, like a movie trailer or an Apple product launch.
So they have to announce something, but most people don’t read Hacker News and won’t even hear about it until later, and that’s fine with them.
Apple at least lets me change this by moving my iTunes/App Store account, which is its own ordeal and far from ideal, but at least there's a defined process: Tell us where you think you live, provide a form of payment from that place, maybe we'll believe you.
I'm really frustrated by Google's attitude of "we know better where you are than you do". People travel sometimes and that's not the same thing as moving!
But once I'm a paying customer, I want to use the thing I'm paying for from where I am without jumping through ridiculous hoops!
The worst variant of this I've seen is when you can neither use nor cancel the subscription from outside a supported market.
Once I got accepted, some of them work outside of the US and some don't
The envelope made it to the recipient, but the coin fell out in transit because I was young and had no idea how to mail coinage. They graciously gave me the invite anyway.
But still, compared to Hotmail etc the free storage space (something like 1GB vs 10MB) was well worth $6
The issues arise with the subsequent stagegate graduation processes, requirements and launches to less restricted markets. It's inconsistent, the QoS pre-GA customers receive is often spotty and the products come with no SLAs, and -- just like Gmail on the consumer side -- things frequently stay in EAP/Beta phase for years with no reliable timeline for launch. ... and then often they're killed before they get to GA, even though they may have been being used by EAP customers for upwards of 1-2 years.
I drafted a new EAP model a few years ago when Google's Cloud AI & Industry Solutions org was in the process of productizing things like the retail recommendation engine and Manufacturing Data Engine, and had all the buy-ins from stakeholders on the GTM side ... but the CAIIS GM never signed off. Subsequently, both the GM & VP Product of that org have been forced out.
In my opinion, this is something Microsoft does very well and Google desperately needs to learn. If they pick up anything from their hyperscaler competitors it should be 1) how to successfully become a market driven engineering company from MSFT and 2) how to never kill products (and not punish employees for only doing KTLO work) from AMZN.
I'm more than happy to transfer my monthly $20 to google from OpenAI, on top of my youtube and google one subscription. It's up to Google to take it.
It's interesting that it's the opposite of the gaming industry. There, because the reviewers dictate the narrative, the industry is better at ferreting out bogus claims. On the flip side, loud voices sometimes steamroll over decent products because of some ideological vendetta.
Like it's shocking to me, are management really so clueless they don't realize how far behind they are? This isn't 2010 Google, your not the company that made your success anymore and in a decade the only two sure fire things that will still exist are android and chrome. Search, Maps, Youtube are all in precarious positions that the right team could dethrone.
I get that it's frustrating not to be able to play with it immediately, but that's just life. Announcing things in advance is still a valuable service for a lot of people.
Plus tons of people have been claiming that Google has somehow fallen behind in the AI race, so it's important for them to counteract that narrative. Making their roadmap more visible is a legitimate strategy for that.
I guess I let my original impression anchor my long-term feelings about the product. Oh well.
"Studying the limits of Gemini 1.5 Pro's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens"
https://storage.googleapis.com/deepmind-media/gemini/gemini_...
It's also not obvious how these huge models will fare against increasingly capable open source ones like Mixtral, perhaps especially since Google are confirming here that MoE is the path forward, which perhaps helps limit how big these models need to be.
But there is also the downside of "tuning" the RAG to return less tokens you will miss extra context that could be useful to the model.
Having 99% retrieval is nuts too. Models tend to unwind pretty badly as the context (tokens) grows.
Put these together and you are getting into the territory of dumping all your company documents, or all your departments documents into a single GPT (or whatever google will call it) and everyone working with that. Wild.
Input is parsed one token at a time right? Can you cache the state after the initial prompt has been provided?
One point I'm unclear on is how these huge context sizes are implemented by the various models. Are any of them the actual raw "width of the model" that is propagated through it, or are these all hierarchical summarization and chunk embedding index lookup type tricks?
The Encyclopedia Britannica is ~44M words.
As the other comment mentions, you can paste the content of entire books or documents and ask very pointed question about it. Last year, Anthropic was showing off their 100K context window, and that's exactly what they did, they gave it the content of The Great Gatsby and asked it questions about specific lines of the book.
Similarly, imagine giving it hundreds of documents and asking it to spot some specific detail in there.
>Finally, we highlight surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person learning from the same content.
Results - https://imgur.com/a/qXcVNOM
Sweet, this opens up so many possibilities.
This cargo-cult hate train is getting tiresome. Half the comments on anything Google-related are like this now, and it doesn't add anything to the conversation.
I also want the shiny immediately when I read about it, but I also know when I am acting entitled and don't go spam comment threads about it.
But really, mostly I mean this: It's fine to criticize things, but when half a dozen people have already raised a point in a thread, we don't need more dupes. It really changes signal-to-noise.
There’s “this team ships” and there’s “ok maybe wait until at least a few people have used your product before you change it all”.
Google announced a fancy model two months early and released it in the promised timeframe.
Seems par for the course.
No, of course they didn’t. And you’re comparing one specific feature (image input) and equating it to a whole model’s release date.
Maybe compare apples to apples next time.
People pointing out release/announcement burnout is a reasonable thing; people in general can only deal with the “next new thing” with some breaks to process everything.
Because they actually shipped ... (!)
https://openai.com/sora
"Remember val="XXXX" .........10M tokens later....... Print val"
It’s difficult to make that array longer because training time explodes.
According to this it's remembering with 99% accuracy, which if you think about it is NUTS, can you imagine reading a 22x 1000 page books, and remembering every single word that was said with 100% accuracy lol
Another possible synthetic benchmark would be to present a list of key value pairs and then ask it for the value corresponding to different keys. Or present a long list of distinct facts and then ask it about them. This latter one could probably be sourced from something like a trivia question and answers data set. I bet there's something like that from Jeopardy.
Then you can ask it who is Tina's (great)^57 grandmother's twice removed cousin on her father's side?
It would have to be able to remember the context of the relationships up and down the document and there'd be nothing to key into as you could ask about any relationship.
I asked the free model (whatever that is) and it wasn't very helpful, alterating betweens a sales bot for Ultra and being somewhat confused itself.
Edit: apparently it goes 1.0 Pro, 1.0 Ultra, 1.5 Pro, 1.5 Ultra and so on.
Gemini ultra 1.0 is still on version 1.0
If you look at the Gemini report it refers to "Gemini 1.5", then refers to "Gemini 1.5 Pro" and "Gemini 1.0 Pro" and "Gemini 1.5 Pro".
- Gemini 1.5 is the new version of the model Gemini.
- They are at the moment testing it on Gemini Pro and calling it Gemini Pro 1.5
- The testing has shown that Gemini Pro 1.5 is delivering the same quality as Gemini Ultra 1.0 while using less computing power
- Gemini Ultra is still using Gemini 1.0 at the moment
For history of pre-Gemini models at Google, see: https://news.ycombinator.com/item?id=39304441
I'm stunned that Google hasn't appointed some "name veto person" that can just say "no, you aren't allowed to have three different things called 'Gemini Advanced', 'Gemini Pro', and 'Gemini Ultra.'" Like surely it just takes Sundar saying "this is the stupidest fucking thing I've ever seen" to some SVP to fix this.
It’s understandable that later generations are better and higher tiers are also better, but usually there is some period of time in between generations to help differentiate them. Here we have Google advancing capability on two axes at the same time.
I give them a pass as this field is advancing rapidly. So good for them. But I think it’s a legitimate call that it adds complexity to their branding. It is different.
So 1M tokens is going to be astronomical.
So Pro is better than Ultra, but only if the version numbers are higher?
The Nano < Pro < Ultra is an in-revision thing. For their LLMs it's a size thing. Then there's newer releases of Nano, Pro, and Ultra. Some Pro might be better than some older Ultra.
A lot of people seem confused about this but it feels so easy to understand that it's confusing to me that anyone could have trouble.
Hope they do a good job and once OpenAI releases GPT 5 they are competitive with it with their offerings, it will be better for everyone.
OpenAI is extremely overvalued and Google is closing their lead rapidly.
Google … has no ability to commercialize anything. Their only commercial successes are ads and YouTube. Doing deceptive launches and flailing around with Gemini isn’t helping their product prospects. I wouldn’t take a bet between open ai and anyone, but I also wouldn’t take a bet on Google succeeding commercially on anything other than pervasive surveillance and adware.
Its shares are already for sale on private markets for accredited investors and for a valuation of over $100BN lead by Thrive Capital.
> Google … has no ability to commercialize anything.
Absolute nonsense.
So Google Cloud, Android (Play Store) are not already commercialized? You well know that they are.
> Doing deceptive launches and flailing around with Gemini isn’t helping their product prospects.
Gemini already caught up to (and surpassed) GPT-4V. What is your point?
> I wouldn’t take a bet between open ai and anyone, but I also wouldn’t take a bet on Google succeeding commercially on anything other than pervasive surveillance and adware.
OpenAI's greatest competitor is Google DeepMind which has the advantage of Google's infrastructure to scale up their models quickly and they have direct access to Google's billions. OpenAI cannot afford to make mistakes or delay anything and a single mistake can cost them hundreds of millions of dollars. The majority of the investment from Microsoft is in Azure credits and not in dollars. [0]
[0] https://www.semafor.com/article/11/18/2023/openai-has-receiv...
While I'm linking semianalysis, though, it's probably worth talking about how everyone except Google is GPU poor: https://www.semianalysis.com/p/google-gemini-eats-the-world-... (paid)
> Whether Google has the stomach to put these models out publicly without neutering their creativity or their existing business model is a different discussion.
Google has a serious GPU (well, TPU) build out, and the fact that they're able to train moe models on it means there aren't any technical barriers preventing them from competing at the highest levels
If you watch the videos in the blog post, you can see it's a screen recording on a computer without any editing/stitching of different scenes together.
It's good to be sceptical but as engineers we should all remain open.
Google appears to be making strides in catching up.
When it comes to my personal workflow and accomplishing tasks, I still find ChatGPT to be the most effective tool. My familiarity with its features has made it indispensable. The integration of mentions and tailored GPTs seamlessly enhances my workflow.
While Gemini may match the foundational capabilities of LLMs, it falls short in delivering a product that efficiently aids in task completion.
But maybe you do, and I am seeing patterns in sand.
I say it's even more than that. OpenAI had a bigger lead when it released GPT-2 than it does now. They're burning through cash to try to hold on to a lead of a few months over the competition.
But my main takeaway is the huge context window! Up to a million, with more than 100k tokens right now? Even just GPT 3.5 level prediction with such a huge context window opens up a lot of interesting capabilities. RAG can be super powerful with that much to work with.
So Pro is like the light and fast version and Ultra the advanced and expensive one.
Nano/Pro/Ultra are model SIZES. 1.0/1.5 is generations of the architecture.
How long until it shows similar results on middle-sized and large codebases? And do the job adequately?
And we should keep in mind that to understand a code change in depth is often just as much work as making the change. When review PRs I don't really know exactly what every change is doing. I certain haven't tested it to be 100% certain I understand fully. I'm just checking the logic looks mostly right and that I don't see anything clearly wrong, and even then I'll often need to ask for clarifications why something was done.
I can't imagine LLMs being used in most large code bases for a while yet. They'd probably need to be 99.9% reliable before we can start trusting them to make changes without verifying every line.
I mention Google One because you can access Gemini Ultra through it.
Those Gemini queries will be no exception.
But we'll see, maybe Gemini will become profitable eventually.
please google, only announce things when people can actually use it.