Gemini is so bad it's literally unusable. I can't think of any situation in which I could accept the quality of the output of Gemini, no matter how low the cost.
Because we've been testing and for our tagging duse case it's been quite quite good. 4o still out performs by 4% (86% vs 90%) but it's acceptable for us given the 35x decrease in cost. We've been able to get 88% on gemini-pro so we're still debating which one we will finalize on given speed, cost, accuracy.
Mistral's partnership with Cerebras for inference hardware has received less commentary than I expected. They're basically blowing the competition out of the water, with Le Chat getting 1,100+ tokens per second of per-user throughput.
I'm curious when someone will do the right experiment in a way that some LLM on Cerebras will do the reasoning so well so big so fast, that it does something very novel
I have found testing coding prompts in mistral and Claude lets me pick, they differ in some details of how to implement my goals (python3, numpy, matplotlib, json, requests sourced data, CSV handling, linear regression)
They are similar speed. I am probably travelling the well worn road so in some equivalent of the LRU cache
The Le Chat Web UI, after having some code and text generated, slowed down to unusable levels for me(the UI itself, probably has some JS code that goes through all the DOM every time). That's why I downloaded the app.
Generally, I feel like all the AI models are about the same at this point. Grok in Twitter has the ability to access real time events information but the rest seems to be interchangeable at this point.
I pay for ChatGPT for higher usage limits, then use all the rest for different things in order to keep history for different things separated(not because one is better than the other in the smartness department).
It should be noted that as a customer of the French ISP Free you get a one year free subscription of Le Chat Pro (Free CEO Xavier Niel is an investor).
Mistral is great, I love their image generation and speed at which it replies. They really don't benefit as much hype from the others contenders but it feels like they are the silent undertaker.
I stopped doing business with Mistral when I got an API subscription and then watched one of their devs break and try to fix their oauth live over several hours over what clearly was something they didn't bother trying in a non-prod environment.
24 comments
[ 2.5 ms ] story [ 78.4 ms ] threadI think I wasted my time reading it this time. Just my opinion.
Quality gap for sure but a 35x gap? Definitely worth testing for your use case.
Because we've been testing and for our tagging duse case it's been quite quite good. 4o still out performs by 4% (86% vs 90%) but it's acceptable for us given the 35x decrease in cost. We've been able to get 88% on gemini-pro so we're still debating which one we will finalize on given speed, cost, accuracy.
A bit more about the collaboration can be found here:
https://cerebras.ai/blog/mistral-le-chat
https://chat.mistral.ai/chat
I'm curious when someone will do the right experiment in a way that some LLM on Cerebras will do the reasoning so well so big so fast, that it does something very novel
They are similar speed. I am probably travelling the well worn road so in some equivalent of the LRU cache
Generally, I feel like all the AI models are about the same at this point. Grok in Twitter has the ability to access real time events information but the rest seems to be interchangeable at this point.
I pay for ChatGPT for higher usage limits, then use all the rest for different things in order to keep history for different things separated(not because one is better than the other in the smartness department).
For your amusement too: https://imgur.com/EgmQ0Ph
https://lmarena.ai/?leaderboard
Do they not take part, or is the list not complete?
That probably helped downloads.
See https://chat.mistral.ai/chat/01a9ee32-a8fe-4305-8f74-a5af959... as an example.
Try the same on other chats with websearch.
Here is me trying (and finally succeeding) to persuade Le Chat to generate image using filename as a prompt...
https://chat.mistral.ai/chat/9940f6bf-b2e5-4db2-bb64-adcbd9f...
I mean... "pretty please" as a debugging technique. I kind of do not look forward to my future conversations with tea kettle and door knob.
Cute.