What I am not really liking about Gemini is how they haven't bundled this into an existing tier of Google One. I am paying for 2TB and I have to cough up extra to get access to the best model.
You may mean Gemini 1.0 Pro, which is the paid version of what used to be bard. Unfortunately Gemini Ultra is not available right now, and it seems Gemini 1.5 Pro (potentially similar to Ultra 1.0) may be available by waitlist.
The Google One AI plan does provide access to Gemini Advanced (backed by Gemini Ultra 1.0) at a significant premium to the current highest Google One plan: https://one.google.com/explore-plan/gemini-advanced
The free tier of Gemini (formerly Bard) has been on Gemini Pro 1.0 for a bit now.
Gemini Pro is free at gemini.google.com, Gemini Ultra is the paid one. The remaining tier is Gemini Nano for on-device use.
This has some serious implications for OpenAI as Gemini Pro 1.5 that is tested here indeed seems to beat their premium tier, assuming Google will keep using these tiers for what's paid and not.
I'm excited about what Google comes up with but only as an intellectual curiosity. In practice, I'll never use it in my apps or workflow because I simply don't trust Google and they'll pull the plug any day with short notice. If Google's brand image wasn't tarnished due to their abandoning products that users loved, I'd probably have a different opinion.
Well, maybe at an API and prompt level. But if Google pull ahead in this space then you may become dependent on what it alone can do functionally. Even if you can trivially switch LLM and prompt, if the others aren't able to do something equivalent (or at the same level of quality) then you're still locked in. Until now we've basically had this situation with OpenAI.
My interest in LLMs lies in the static program analysis space and these results are genuinely exciting. LLM-guided analysis (e.g. invariant modelling or detecting bugs by identifying unusual implementation patterns) is going to be a thing.
OpenAI is gradually turning into the Fisher-Price of AI. I cancelled my ChatGPT Pro subscription after 12mo due to increasingly low quality answers since ~Nov 2023. I occasionally try the API but the quality there is only slightly better than ChatGPT.
I keep trying alternatives to gpt 4 based on people shitting on it and you know what, every single time I try something else, I hit a bruh moment on the first try.
I asked Gemini v1 2 days ago about programming and the first response was some nonsense about elections being a complex topic.
if gpt 4 is Fisher Price then it's competitors are ill fitting kinder egg toys with missing pieces.
Gemini in their interface has overly strong political correctness and safety which creates such strange responses, but if you avoid triggering that then it is pretty good.
Edit: And in your screenshot you did get real responses, you see gemini generates several look at response 2, looks like what you wanted.
It's so sad that we got AIs only after everybody got so sensitive about some utterances. I think the language itself was never supposed to be that serious as we are currently making it.
I did see the other examples. But the fact is, a question about double metaphone that is 99% code should not trigger some tangent about politics.
Not once in my thousands of questions to gpt did it go so far off base. Maybe in the gpt3 days sometimes it would get caught in a repeating character loop.
Yet my fourth or so time back to Bard and it shits the bed on the first try makes me think, yeah... Still kinder surprise
Also, the suggested changes from the remaining responses are incredibly naive. It suggests pulling numbers out and appending them back in, which breaks if the numbers aren't simply at the end of the strings, as well as not going with the pronunciation spirit of metaphone.
I built a smart contract decompiler (eveem.org), and got deep into formal verification / code analysis path during that time.
There was a ton more I could do back then if I had access to gpt-4. Some analysis methods were extremely slow using solvers, and it was a pita to write code for dedicated code pattern recognition. Gpt4 improves that, and it's easy to verify if it's guesses are correct using formal methods.
Nice, I've used your decompiler! The smart contract decompilation/binary lifting space is pretty active and I'm not sure how much low hanging fruit is left there. But there are now active bug bounty/audit contests in this space, so my focus is on projects where I already have the source. I'm exploring some unconventional domain-specific ideas that leverage ASTs and (eventually) LLMs.
That being said, I recently read a paper whose authors made use of traditional statistical analysis to find unusual implementation patterns in an IR decompilation of smart contracts. I wonder if custom LLMs can be used to perform that analysis.
I'm very much not on the LLM hype-train, but I've been saying from day one that I'd really like something like this. Code has strong local similarities, so it's not far fetched to imagine that future LLMs -- if not current ones -- could be used as a stochastic linter of sorts.
Programming is hard, it's not insane to make use of all the help we can get.
I feel the same, ChatGPT barely beats Google searches for me nowadays.
For example, on a side project I used to explain it shortly and the issue I am facing, mostly looking to discover packages or libraries that can help me build it.
Over 3 months ago GPT4 used to jump straight to the point and recommend alternative scenarios and packages. Now it just gives me a giant text with considerations for the whole project, but rarely actually tackles the issue I am having. Even if I ask it to actually go in-depth into whatever I am interested in it usually doesn't and repeats the same surface level information.
> I feel the same, ChatGPT barely beats Google searches for me nowadays.
I started feeling similar recently, but then I added the "Custom Instructions" [0] feature in ChatGPT, and the results are much better. The instruction that I've put there is that I'm a highly information-aware person and that I require detailed answers. The reason that I feel this works is that GPT is essentially trained (caveat: the RLHF part could partially falsify my "theory" here) on internet textual data whereby the usual method of answering on internet forums (and other sites) has a relatively low-to-moderate detailed way of answering things; but given that we know that GPT has in its latent space pretty much has all the useful knowledge on the internet, that if one asks for very detailed (information diverse/inclusive) answers by default that it can perform really well.
Impressive! It's too bad it's not possible to opt-out of model training on Googles Gemini (free/paid*) plans.
Which includes:
Conversations, Location, Feedback, Usage information
And specifically this clause:
"Please don't enter confidential information in your conversations or any data you wouldn't want a reviewer to see or Google to use to improve our products, services, and machine-learning technologies."
There is a setting on Gemini paid, however it isn't about disabling model training, only activity storage length.
Also (On manual deletion):
"Even when Gemini Apps Activity is off, your conversations will be saved with your account for up to 72 hours to allow Google to provide the service and process any feedback."
And in any case:
"Conversations that have been reviewed or annotated by human reviewers (and related data like your language, device type, location info, or feedback) are not deleted when you delete your Gemini Apps activity because they are kept separately and are not connected to your Google Account. Instead, they are retained for up to three years."
I’m guessing this is more for abuse prevention and regulatory compliance. Google being in the news yet again for some misuse of their AI and journalists spinning that off as Google’s embodiment of values is the last thing that they want or need at this point.
If you turn off Gemini Apps Activity, future conversations won’t be sent for human review or used to improve our generative machine-learning models."
The way I interpret the policy is that if Gemini Apps Activity is off:
1. Conversations after activity is turned off that are linked to your account are retained for up to 72 hours for "safety and security", but are not sent to human reviewers or used for model training.
2. In the event that you submit feedback (eg good/bad response), then the conversation (disassociated from your account) can be used for training. This is also flagged when attempting to submit feedback: "Even when Gemini Apps activity is off, feedback submitted will also include up to the last 24 hours of your conversations to help improve Gemini."
I suspect a good way to omit yourself from data collection would be to include something that looks like PII at the beginning and end of every conversation. I suspect a fake social security number and a first, middle initial, and last name, all next to each other, would be enough. Maybe throw in a DOB in there too.
Confused. Many articles about how good Gemini 1.5 is. Yet I am in Generative AI Studio right now and I can't get access to it. If its so good, why make it so hard to access?
1. If you are an enterprise customer - the waitlist page is a blank black page as of 2/18/24
2. If you are non-enterprise customer - you can signup for waitlist.
Seems like it wouldn't. In the OP post, Gemini was able to reason about that codebase through the _text_. For example, comments and function names. It doesn't really understand the code. So binary code would be gibberish to it.
Since OpenAI has started early to give access to developers so that they can create new software with their models, it says alot about the company, and they are truly commited to create better solutions. I think all other LLMs are way back in this space and will have a hard time as soon as OpenAI continues to the next model.
Say I design a language, and instead to writing an implementation, I train an AI with tons of input/output examples, the kinda of errors to emit for violations of the grammar. Will be cool to try this out.
I imagine it might be more feasible to transpilation rsther than real compilation.
If you have the compute time to pull it off. I'll wager these kinds of models are only really possible by half a dozen companies so "other companies" might more accurately be written "Google and a few other companies" or something like that.
65 comments
[ 4.6 ms ] story [ 133 ms ] threadThe free tier of Gemini (formerly Bard) has been on Gemini Pro 1.0 for a bit now.
This has some serious implications for OpenAI as Gemini Pro 1.5 that is tested here indeed seems to beat their premium tier, assuming Google will keep using these tiers for what's paid and not.
Et viola, your application is now free from LLM lock-in.
OpenAI is gradually turning into the Fisher-Price of AI. I cancelled my ChatGPT Pro subscription after 12mo due to increasingly low quality answers since ~Nov 2023. I occasionally try the API but the quality there is only slightly better than ChatGPT.
I asked Gemini v1 2 days ago about programming and the first response was some nonsense about elections being a complex topic.
if gpt 4 is Fisher Price then it's competitors are ill fitting kinder egg toys with missing pieces.
https://i.imgur.com/gxh0mUZ.jpeg
Edit: And in your screenshot you did get real responses, you see gemini generates several look at response 2, looks like what you wanted.
Not once in my thousands of questions to gpt did it go so far off base. Maybe in the gpt3 days sometimes it would get caught in a repeating character loop.
Yet my fourth or so time back to Bard and it shits the bed on the first try makes me think, yeah... Still kinder surprise
Also, the suggested changes from the remaining responses are incredibly naive. It suggests pulling numbers out and appending them back in, which breaks if the numbers aren't simply at the end of the strings, as well as not going with the pronunciation spirit of metaphone.
"Google search for election info" https://g.co/gemini/share/7afd63de0079
Custom GPTs exist for a reason. Grimoire (custom GPT) outperforms raw ChatGPT by miles.
My workflow is usually Grimoire to get a broad overview of things, then a model that can use my repo via tools like Cursor.
I built a smart contract decompiler (eveem.org), and got deep into formal verification / code analysis path during that time.
There was a ton more I could do back then if I had access to gpt-4. Some analysis methods were extremely slow using solvers, and it was a pita to write code for dedicated code pattern recognition. Gpt4 improves that, and it's easy to verify if it's guesses are correct using formal methods.
That being said, I recently read a paper whose authors made use of traditional statistical analysis to find unusual implementation patterns in an IR decompilation of smart contracts. I wonder if custom LLMs can be used to perform that analysis.
I'm very much not on the LLM hype-train, but I've been saying from day one that I'd really like something like this. Code has strong local similarities, so it's not far fetched to imagine that future LLMs -- if not current ones -- could be used as a stochastic linter of sorts.
Programming is hard, it's not insane to make use of all the help we can get.
For example, on a side project I used to explain it shortly and the issue I am facing, mostly looking to discover packages or libraries that can help me build it.
Over 3 months ago GPT4 used to jump straight to the point and recommend alternative scenarios and packages. Now it just gives me a giant text with considerations for the whole project, but rarely actually tackles the issue I am having. Even if I ask it to actually go in-depth into whatever I am interested in it usually doesn't and repeats the same surface level information.
I started feeling similar recently, but then I added the "Custom Instructions" [0] feature in ChatGPT, and the results are much better. The instruction that I've put there is that I'm a highly information-aware person and that I require detailed answers. The reason that I feel this works is that GPT is essentially trained (caveat: the RLHF part could partially falsify my "theory" here) on internet textual data whereby the usual method of answering on internet forums (and other sites) has a relatively low-to-moderate detailed way of answering things; but given that we know that GPT has in its latent space pretty much has all the useful knowledge on the internet, that if one asks for very detailed (information diverse/inclusive) answers by default that it can perform really well.
[0] https://openai.com/blog/custom-instructions-for-chatgpt
Which includes:
Conversations, Location, Feedback, Usage information
And specifically this clause:
"Please don't enter confidential information in your conversations or any data you wouldn't want a reviewer to see or Google to use to improve our products, services, and machine-learning technologies."
Update: *There is a Gemini setting here https://myactivity.google.com/u/1/product/bard but it does not explicitly state that it disables training, just activity history.
Presumably Gemini 1.5 will be available there soon.
Allows training on input:
1. Gemini (formerly Bard): https://gemini.google.com/. Policy: https://support.google.com/gemini/answer/13594961. Can opt out of training by disabling history.
2. AI Studio free plan: https://ai.google.dev/.
Does not allow training on input:
1. AI Studio paid plan: https://ai.google.dev/.
2. Google Cloud Vertex AI: https://cloud.google.com/vertex-ai. Policy: https://cloud.google.com/vertex-ai/docs/generative-ai/data-g...
3. Gemini for Workspace (formerly Duet AI for Workspace).
Also (On manual deletion):
"Even when Gemini Apps Activity is off, your conversations will be saved with your account for up to 72 hours to allow Google to provide the service and process any feedback."
And in any case:
"Conversations that have been reviewed or annotated by human reviewers (and related data like your language, device type, location info, or feedback) are not deleted when you delete your Gemini Apps activity because they are kept separately and are not connected to your Google Account. Instead, they are retained for up to three years."
The setting is explicitly about the activity history (which is 3-mo auto-delete, 72h minimum).
"Who has access to my Gemini Apps conversations?
How you can control what’s shared with reviewers
If you turn off Gemini Apps Activity, future conversations won’t be sent for human review or used to improve our generative machine-learning models."
The way I interpret the policy is that if Gemini Apps Activity is off:
1. Conversations after activity is turned off that are linked to your account are retained for up to 72 hours for "safety and security", but are not sent to human reviewers or used for model training.
2. In the event that you submit feedback (eg good/bad response), then the conversation (disassociated from your account) can be used for training. This is also flagged when attempting to submit feedback: "Even when Gemini Apps activity is off, feedback submitted will also include up to the last 24 hours of your conversations to help improve Gemini."
1. If you are an enterprise customer - the waitlist page is a blank black page as of 2/18/24
2. If you are non-enterprise customer - you can signup for waitlist.
1.5 is not available in the Gemini ChatGPT like interface. It is only available through AI Studio. But you have to signup for a waitlist.
It seems like a very interesting project with quite a steep learning curve, given interaction nets look like a whole new way of thinking.
At some point they run out of training data and others can catch up.
ChatGPT 4 doesn't work as well as ChatGPT 3.5. The little brother is smarter han the big brother.
ChatGPT 3.5 works better than Eliza.
No-one has heard of any other AIs.
That's pretty much it - the entire state of AI early 2024.
Say I design a language, and instead to writing an implementation, I train an AI with tons of input/output examples, the kinda of errors to emit for violations of the grammar. Will be cool to try this out.
I imagine it might be more feasible to transpilation rsther than real compilation.
Honestly suspicious if it even shares much with 4 as it disappoints more times than it impresses.
Yup. If a french company, Mistral, can do it you know the cat is out of the bag (ok, ok, started by an ex- OpenAI person I think).
I tried mistral-next and it is impressive.
The "moat" at this point seems to be: "Invest and train models using $$$ of hardware and electricity".
It's both exciting (nobody has an edge) and depressing (the edge consist in buying billions worth of NVidia AI chips).
The issue is that it's only democratized if you have the money.
What's with the France slander? France is not some technological backwater - it's given us BeOS[1], VLC and Fabrice Bellard.
1. I know Be Inc was an American company, but check the nationalities of its alumni.