Anthropic has been killing it. I subscribe to both chatgpt pro and claude, but I spend probably 90% of my time using Claude. I usually only go back to open ai when I want another model to evaluate or modify the results.
I was worried how they'd do as it felt like Opus was very expensive compared to GPT-4o but with worse performance. They're now claiming to beat GPT-4o AND do it cheaper, that's impressive.
I can't speak to any new features announced today but the API version of Claude has been superior in every way when paired with a more feature rich front end.
Same here. I said this somewhere else already, but honestly GPT4o feels worse than 4 to me. So that's what drove me over to using Claude more which lead to me discovering it is generally superior for most of my use cases.
Not listed there and not well known, but I really like that the bigger models know how to produce ArgDown output. You can do things like "give me a detailed/nuanced tree in argdown format for topic: static typing improves development speed". It's quite useful for exploring topics.
This is a very strong offering. I've been really impressed with 3.0 Haiku for smaller tasks, but I'm going to have to test 3.5 Sonnet as our primary pipeline model.
This is amazing - I far prefer the personality of Claude to GPT-4 series models. Also, with coding tasks, Claude-3-Opus and been far better for me vs gpt-4-turbo and gpt-4o both. Looking forward to giving it a spin.
Seems like it's doing better than GPT-4o in most benchmarks though I'd like to see if its speed is comparable or not. Also, eagerly awaiting the LMSYS blind comparison results!
I agree. There are some corner cases that GPT-4o reliably fails that Claude does well in, and vice versa. GPT-4 and GPT-4o consistently generates very poor cv2 Python code for human face/boundary box work - it's a strange reproducible failure in my experience.
For coding Claude Opus-3 provides far more mature code and good at finding bugs (when present with the error code) compared to GPT-4-Turbo and GPT-4o. Last few days I've been using both for some python+pyspark project. Not sure how come in their comparison GPT-4o is showing that good!
GPT4(o) is quite good at advanced math, it's been helpful when I was learning differential geometry. Not sure how Claude compares though, this 3.5 release has tempted me to try it out. Also, it's finally available in Canada!
One huge leg up here is ChatGPT defaults to outputting (and actually displaying, if you're using the default client) LaTeX. Between that and this being one of the few places high verbosity is actually helpful I preferred GPT4/4o for helping learn calc 2. It's well possible Claude 3.5 Sonnet gets the final answer right on the first try more often though.
After a few months of writing all my homework in LaTeX I'm finding my thinking slid towards the raw latex rather than the rendered form. I'll have to wait till fall semester to give 3.5 a good whirl.
I'm surprised there isn't a single mention of Gemini 1.5 Pro. I've been using it for about a month because it came for free with my Google setup and I've been pretty happy. Not for coding but mostly for business tasks like writing minutes from transcripts, summarizing long legal documents,... and the long context length has been awesome. It also conveniently integrates with the rest of my google setup like Drive.
IIRC it also ranked only behind gpt4o on benchmarks.
Gemini in general is terrible. Way too many mistakes. If you use it via the API it repeats itself constantly. At least it's the model that is the easiest to jailbreak and will happiliy give you a tutorial on how to make a bomb if you ask politely ;) Very ironic considering how Google emphasizes "safety".
I've also had good results with Gemini 1.5 Pro for some tasks. Just yesterday, it produced very good analysis and comments based on a 200-page document. ChatGPT 4o was much weaker, and the document was too large for Claude 3 Opus. (This was a few hours before 3.5 was released.)
This is awesome! Until GPT-4o dropped, Claude 3 Opus was hands down my go-to for code generation.
Between these model performance improvements and their new "artifacts" handling, I get the impression this update may sway me strongly back towards Anthropic (at least for this use case).
Someone who has built infrastructure or system prompts that use Opus will probably continue with Opus until they verify that everything works on Sonnet 3.5
Benchmarks don't cover all possible use cases, for one. There's always the possibility that a model does better on every benchmark thrown at it, but for your specific use case it does worse in practice.
Woah, this is a marked improvement. I just threw a relative complex coding problem at it and 3.5 sonnet did a really good job across several language. I asked it to rewrite a Qt6 QSyntaxHighlighter subclass to use TreeSitter to support arbitrary languages and not only did it work (with a hardcoded language) but it even got the cxx-qt Rust bindings almost right, including the extra header.
Curious to see how well it handles QML because previous models have been absolutely garbage at it.
An AI chat interface ironically. All of the chat apps are slow as heck electron so I figured there's a market for an actually usable desktop app, especially one that can inline code files and create a RAG index of local documents.
Plume looks great! I'm curious how you implemented the markdown formatted text editing component - I need to implement something similar for the chat input.
So funny! I thought about building the same thing for the exact same reason! And the block editor I created for Plume is a great candidate for that.
I implemented the editor from scratch. The model is a C++ QAbstractListModel and the view is in QML. I'm writing a blog post about the implementation, it should pop up soon on my personal website (https://rubymamistvalove.com). But I can (and wish) to send you a draft soon, if you would like.
You're welcome to use other models; as for me, I started using Claude 3 shortly after it came out and I've never felt like switching to the "competition". Their stance on safety aligns with my take on it, and they don't use user data for training purposes. Matter of fact, I just got my first customer for using AI services last night when I live-coded a demo in front of them, and that was with Opus. I'm thrilled to see what I can do with the new tech! I've been trying it out a little bit this morning but haven't seen much improvement (yet).
Awesome, can’t wait to try this. I wish the big AI labs would make more frequent model improvements, like on a monthly cadence, as they continue to train and improve stuff. Also seems like a good way to do A/B testing to see which models people prefer in practice.
True, but they could have multiple parallel running training processes going on at the same time. And they could release models that result from partial training checkpoints if they can quantify that they are better than the last released model and also "safe."
Anthropic is the new king. This isn't even Claude 3.5 Opus and it's already super impressive. The speed is insane.
I asked it "Write an in depth tutorial on async programming in Go" and it filled out 8 sections of a tutorial with multiple examples per section before GPT4o got to the second section and GPT4o couldn't even finish the tutorial before quitting.
I been a fan of Anthropic models since Claude 3. Despite the benchmarks people always post with GPT4 being the leader, I always found way better results with Claude 3 than GPT4 especially with responses and larger context. GPT responses always feel computer generated, while Claude 3 felt more humanlike.
Agree. They're like the quiet achievers. The new experimental sidebar 'artifacts' feature is super cool (it keeps a convenient version history also). I just fed it a json object and asked for a collapsible table app using next and shadcn. First code worked perfectly and code doesn't get lost in the chat history like chatgpt. Response was super fast.
And latest training data date for 3.5 is April, 2024.
I don't think that is the case. AWS is a very significant investor and if you meet with their business development team they will recommend deploying on bedrock (which is Nvidia). There are also press releases like this[1] stating they use Nvidia.
Like its predecessors, Claude 3 models employ various training methods, such as unsupervised learning and Constitutional AI [6]. These models were trained using hardware from Amazon Web Services (AWS) and Google Cloud Platform (GCP), with core frameworks including PyTorch [7], JAX [8], and Triton [9].
JAX's GPU support is practically non-existent, it is only used on TPUs.
I agree. I've been really impressed with Anthropic. The issue for me comes when I want to take arbitrary user input and ask Claude questions about the user provided input. Claude is very, very, very ethical. Which is great, but it won't provide a response if the user tends to use a lot of curse words.
Do some masking of curse words with sht, ?!, verybad, or similar? Something that Claude will accept. It might work, if users are just generally badmouthed, not actively trying to trigger the model/system.
Our internal blinded human evals for summarization/creative work have always preferred Claude 3.0 Opus by a huge margin, so we've been using it for months - GPT-4o didn't unseat it either.
GPT-4o IMO was better for coding (still using GPT-4 original w/ Cursor, but long-form stuff GPT-4o seemed better) but with this new launch, will definitely have to retest.
One thing Anthropic did that I loved and think was very smart was building a prompt generator[1] into the developer console. The generator is tuned to generate prompts the way Claude prompts are supposed to be, which improves responses. And you can use it to improve your user prompt as well, not just your system prompt, which make responses even better.
You can see examples of the prompts it generates here[2]. It significantly improved my experience with LLMs; I haven't touched GPT4 in quite a while, and GPT4o didn't change that.
This is impressive. I've just ran a couple of gpt4o workloads for getdot.ai on Sonnet and the quality is great.
Plus it's super fast right now ~110 token/s (but gpt4o was also super fast when they launched). But what will stay is the 40% price drop for input tokens. I love it. :)
Just tried it. This is the first model that immediately gives me the correct answer to my test prompt: "Hi <model>, can you give me an exact solution to pi in python?". All other models I've tried first give an approximation, taking several prompts to come to the correct conclusion: it's impossible.
In numerical computation, "exact" is a term of art that means providing accurate bounds, that are not affected by random numerical errors. So an "exact" computation of Pi is one that, e.g. might just ask for a minimum precision as input, then provides an interval around Pi that's at least that small and accurately bounds Pi. If the AI answered "it can't be done" that's not really the best outcome, though it's understandable because that use of "exact" is quite niche.
This is impressive. I just tested a debug problem I encountered in real life, which previous models including got-4o can’t do. Claude 3 sonnet nailed it in first try.
If anyone would like to try it for coding in VSCode, I just added it to http://double.bot on v93 (AI coding assistant). Feels quite strong so far and got a few prompts that I know failed with gpt4o.
fyi for anyone testing this in their product, their docs are wrong, it's claude-3-5-sonnet-20240620, not claude-3.5-sonnet-20240620.
Before I read your comment I was looking for a solution to use Claude as co-pilot in Neovim. I've seen in Double's website FAQ that it's not supported yet. Do you have an idea if this feature is expected to land anytime soon?
Which is the goto leaderboard for determining which AI model is best for for answering devops / computer science questions / generating code? Wondering where Claude falls on this.
Recently canceled openai subscription because too much lag and crashes. Switched to Gemini because their webinterface is faster and rock solid. Makes me think the openai backend and frontend engineers don't know what they are doing compared to the google engineers.
A while ago I tested the image recognition skills of GPT-4o, Claude 3, and Gemini using a random street plan diagram I happened to have in my downloads folder (https://i.imgur.com/9WZpK0L.png). It's a top-down CAD rendering showing the planned renovation of a street in my neighborhood in Tampere, Finland. I uploaded the image and simply asked each model "What can you tell me about the attached picture?"
GPT-4o's answer was excellent and highly detailed, recognizing essentially all the relevant aspects of the image [GPT4o]. Claude 3 Sonnet was correct on a general level, but its answer was much less detailed and showed more uncertainty in the form of "A or B" sentences [CL3]. Gemini's answer was, well, hilariously wrong [GEM].
I just tried this with Claude 3.5 Sonnet and it did very well. Its answer was still not as detailed as GPT-4o's, but it did ask me if I want it to elaborate on any aspect of the image [CL35].
I think this was an interesting experiment because street plan CAD diagrams probably aren't very common in the training data of these models.
I'm very impressed! Using Gpt-4o and Gemini, I've rarely had success when asking the AI models to create a PlantUML flowchart or state machine representation of any moderate complexity. I think this is due to some confusing API docs for PlantUML. Claude 3.5 Sonnet totally knocked it out of the park when I asked for 4-5 different diagrams and did all of them flawlessly. I haven't gone through the output in great detail to see if its correct, but at first glance they are pretty close. The fact that all the diagrams were able to be rendered is an achievement.
I wish they'd implement branching conversations like in ChatGPT. And convenient message editing, that doesn't paste large chunks of text as an non-editable attachment or break formatting.
Seems like such a simple thing to do, relative to developing an AI, yet the minor differences in the UI/UX are what prevents me from using claude a lot more.
Good history search (including non "main" conversation branches) and convenient conversation management (bookmarking, folders, maybe something smarter) would be great.
Also, maybe some convenient way to create message templates? I don't know how I'd implement this, I just know that I often write one long prompt that I reuse multiple times, with multiple minor tweaks/edits, and it'd be amazing to have a convenient tool to manage that.
Also, good mobile/tablet support, convenient to use and without bugs (as I happen to spend most of my time writing prompts on my ipad, but that's just me).
If you already have a demo - please share a link, I'd be happy to beta test it and maybe become one of the early customers.
wow, reading your comment is a great mood boost for me because these are literally the exact features I want from my llm chat experience. It's great to see someone with the exact same problem set.
I just followed you on Twitter (I'm @NamanyayG there as well), I'll definitely ping you when I have something to test.
Thanks for the resource! I've seen some of them and the main issue I had is I don't want to self host anything, and I want to use latest third party models as soon as they are released.
Maybe something like what I'm talking about exists already, but I think I'll still try and make my own open source version to fulfill my personal requirements.
We (disclosure: founder) do something similar at Trelent[1] but with an emphasis on security. Paid accounts can use OpenAI & Anthropic models, free ones just OpenAI. We have 3.5 sonnet live already. If you want to try it out lmk! Also totally respect building your own open-source :)
So typically these providers only offer ZDR to "managed" customers, after a lengthy application process. For example, on Azure, "managed" means companies with >$1m, possibly more now, in annual spend. They don't want to waste their time going through this long application process with smaller companies, so we take some of that weight off their shoulders. They get the same revenue at the end of the day, so in many ways it groups smaller companies' LLM spend and sends it straight to their bottom line, and they still get to claim their rolling out AI "responsibly".
Once one provider is cracked, the others fall as well, as these AI companies are all competing viciously for customers. Et voila, ZDR across multiple providers for the small(er) companies out there :)
I just mean that when you click the button to generate a new version of the response (or edit your own message), ChatGPT shows you the arrow buttons enabling you to go to the previous version of it, and that works for all the messages, so you can go back up a few messages and try a different version of the conversation, without losing what you've had before.
Pretty much all of the features you mention are already in LibreChat (MIT License). If you don't mind self-hosting, then it has branching, convo search, change models mid-chat, "presets" (save system prompts), and a whole lot more. I've deployed it in my gov agency for months now, and I've had amazing feedback. https://github.com/danny-avila/LibreChat
Hi! I’m a product engineer on the Claude.ai team. Claude.ai does support branching conversations. If you hover on a message, there should be an edit button, and once you edit the message, you can again hover on it, which will show you left/right arrows that will switch between the branches. Please let me know if you have any troubles with this!
Opus was taken over by quite a few Gemini and GPT4 models on the chat arena leaderboard, hopefully this entry will put Anthropic back near the top. Nice work!
Might look small, but the needle in a haystack numbers they report in the model card addenda at 200k are also a massive improvement towards “Proving a negative”… I.e. your answer does not exist in your text. %99.7 vs 98.3 for Opus
https://cdn.sanity.io/files/4zrzovbb/website/fed9cc193a14b84...
Could you explain how these two are related? That benchmark seems to be asking for very specific information inside a large body of text. For LLMs, that seems quite a different task compared to proving a negative. Any improvements on proving a negative would mean less hallucinations and would be a huge deal.
286 comments
[ 5.6 ms ] story [ 288 ms ] threadwhat frontend are you talking about?
Sillytavern also supports prefills which is an API feature not allowed on the web version.
Editing the system prompt is also not permitted in the web version but should be doable in any third party front-end.
I also use Poe sometimes which doesn't have all those features but at least allows custom system prompts when using Claude.
this is new and I just tried a simple dice roll into a React component, and it works perfectly.
Seems like it's doing better than GPT-4o in most benchmarks though I'd like to see if its speed is comparable or not. Also, eagerly awaiting the LMSYS blind comparison results!
What kind of coding tasks is Claude 3 opus doing for people ?
This new Sonnet seems way less human-like than even old Sonnet, let alone Opus. It's practically devoid of character. It's smart, though.
IIRC it also ranked only behind gpt4o on benchmarks.
Between these model performance improvements and their new "artifacts" handling, I get the impression this update may sway me strongly back towards Anthropic (at least for this use case).
Ah it is "The model costs $3 per million input tokens and $15 per million output tokens"
Curious to see how well it handles QML because previous models have been absolutely garbage at it.
Plume looks great! I'm curious how you implemented the markdown formatted text editing component - I need to implement something similar for the chat input.
I implemented the editor from scratch. The model is a C++ QAbstractListModel and the view is in QML. I'm writing a blog post about the implementation, it should pop up soon on my personal website (https://rubymamistvalove.com). But I can (and wish) to send you a draft soon, if you would like.
With so much competition, I wonder why everyone else makes it hard to try out something.
I asked it "Write an in depth tutorial on async programming in Go" and it filled out 8 sections of a tutorial with multiple examples per section before GPT4o got to the second section and GPT4o couldn't even finish the tutorial before quitting.
I been a fan of Anthropic models since Claude 3. Despite the benchmarks people always post with GPT4 being the leader, I always found way better results with Claude 3 than GPT4 especially with responses and larger context. GPT responses always feel computer generated, while Claude 3 felt more humanlike.
And latest training data date for 3.5 is April, 2024.
[1] https://nvidianews.nvidia.com/news/aws-and-nvidia-collaborat... and https://press.aboutamazon.com/2023/3/aws-and-nvidia-collabor... search for "anthropic"
https://www.datacenterdynamics.com/en/news/anthropic-to-use-...
https://www.anthropic.com/news/anthropic-partners-with-googl...
From Claude 3's technical report:
Like its predecessors, Claude 3 models employ various training methods, such as unsupervised learning and Constitutional AI [6]. These models were trained using hardware from Amazon Web Services (AWS) and Google Cloud Platform (GCP), with core frameworks including PyTorch [7], JAX [8], and Triton [9].
JAX's GPU support is practically non-existent, it is only used on TPUs.
GPT-4o IMO was better for coding (still using GPT-4 original w/ Cursor, but long-form stuff GPT-4o seemed better) but with this new launch, will definitely have to retest.
Pretty big news.
You can see examples of the prompts it generates here[2]. It significantly improved my experience with LLMs; I haven't touched GPT4 in quite a while, and GPT4o didn't change that.
[1]: https://docs.anthropic.com/en/docs/build-with-claude/prompt-...
[2]: https://sr.ht/~jamesponddotco/llm-prompts/
Seems like the API docs haven't been updated yet: https://docs.anthropic.com/en/docs/about-claude/models
And it's free!
Plus it's super fast right now ~110 token/s (but gpt4o was also super fast when they launched). But what will stay is the 40% price drop for input tokens. I love it. :)
fyi for anyone testing this in their product, their docs are wrong, it's claude-3-5-sonnet-20240620, not claude-3.5-sonnet-20240620.
https://github.com/frankroeder/parrot.nvim/
Recently canceled openai subscription because too much lag and crashes. Switched to Gemini because their webinterface is faster and rock solid. Makes me think the openai backend and frontend engineers don't know what they are doing compared to the google engineers.
Or the scale.ai private benchmarks
It places Claude 3.5 Sonnet in third position.
GPT-4o's answer was excellent and highly detailed, recognizing essentially all the relevant aspects of the image [GPT4o]. Claude 3 Sonnet was correct on a general level, but its answer was much less detailed and showed more uncertainty in the form of "A or B" sentences [CL3]. Gemini's answer was, well, hilariously wrong [GEM].
I just tried this with Claude 3.5 Sonnet and it did very well. Its answer was still not as detailed as GPT-4o's, but it did ask me if I want it to elaborate on any aspect of the image [CL35].
I think this was an interesting experiment because street plan CAD diagrams probably aren't very common in the training data of these models.
--
[GPT4o] https://gist.github.com/jdahlstrom/844bda8ac76a5c3248c863d20...
[CL3] https://gist.github.com/jdahlstrom/ecccf31c8305f82519f27af53...
[GEM] https://gist.github.com/jdahlstrom/2e12a966c0d603a7b1446ba08...
[CL35] https://gist.github.com/jdahlstrom/60ca9352630934bec6e2f4e37...
GPT-4o 30.7
Claude 3.5 Sonnet 27.9
Claude 3 Opus 27.3
Llama 3 Instruct 70B 24.0
Gemini Pro 1.5 0514 22.3
Mistral Large 17.7
Qwen 2 Instruct 72B 15.6
Seems like such a simple thing to do, relative to developing an AI, yet the minor differences in the UI/UX are what prevents me from using claude a lot more.
For a long while I've wanted a good "pro" UI that can connect to multiple different llm APIs
Convenient editing and branching is one of the items in my roadmap already, what else do you think I could include?
Also, maybe some convenient way to create message templates? I don't know how I'd implement this, I just know that I often write one long prompt that I reuse multiple times, with multiple minor tweaks/edits, and it'd be amazing to have a convenient tool to manage that.
Also, good mobile/tablet support, convenient to use and without bugs (as I happen to spend most of my time writing prompts on my ipad, but that's just me).
If you already have a demo - please share a link, I'd be happy to beta test it and maybe become one of the early customers.
I just followed you on Twitter (I'm @NamanyayG there as well), I'll definitely ping you when I have something to test.
https://old.reddit.com/r/LocalLLaMA/comments/1847qt6/llm_web...
Maybe something like what I'm talking about exists already, but I think I'll still try and make my own open source version to fulfill my personal requirements.
[1]: https://trelent.com
Once one provider is cracked, the others fall as well, as these AI companies are all competing viciously for customers. Et voila, ZDR across multiple providers for the small(er) companies out there :)
Can you say more about this?
I Google'd and I'm not finding much. I asked ChatGPT and its response was not the assumption I held about what "branching" meant [0].
[0] https://chatgpt.com/c/6b2e0f7c-c4e6-44df-9116-ac7f618200f2