I thought the same thing, I have 3 really hard problems that Claude (or any model) hasn’t been able to solve so far and I’m really excited to try them today
Considering that they are the model that powers a majority of Cursor/Windsurf usage and their play with MCP, I think they just have to figure out the UX and they'll be fine.
I complain about this all the time, despite me saying "ask me questions before you code" or all these other instructions to code less, it is SO eager to code. I am hoping their 3.7 reasoning follows these instructions better
We should remember 3.5 was trained in an era when ChatGPT would routinely refuse to code at all and architected in an era when system prompts were not necessarily very effective. I bet this will improve, especially now that Claude has its own coding and arch cli tool.
I've found claude to be very receptive to precise instructions. If I ask for "let's first discuss the architecture" it never produces code. Aider also has this feature with /architect
I added custom instruction under my Profile settings in the "personal preferences" text box. Something along the lines of "I like to discuss things before wanting the code. Only generate code when I prompt for it. Any question should be answered to as a discussion first and only when prompted should the implementation code be provided". It works well, occasionally I want to see the code straight away but this does not happen as often.
I get this as well, to the point where I created a specific project for brainstorming without code — asking for concepts, patterns, architectural ideas without any code samples. One issue I find is that sometimes I get better answers without using projects, but I’m not sure if that’s everyone experience.
That's been my experience as well with projects, though I have yet to do any sort of A/B testing to see if it's all in my head or not.
I've attributed it to all your project content (custom instruction, plus documents) getting thrown into context before your prompt. And honestly, I have yet to work with any model where the quality of the answer wasn't inversely proportional to the length of context (beyond of course supplying good instruction and documentation where needed).
I’ve set up a custom style in Claude that won’t code but just keeps asking questions to remove assumptions:
Deep Understanding Mode (根回し - Nemawashi Phase)
Purpose:
- Create space (間, ma) for understanding to emerge
- Lay careful groundwork for all that follows
- Achieve complete understanding (grokking) of the true need
- Unpack complexity (desenrascar) without rushing to solutions
Expected Behaviors:
- Show determination (sisu) in questioning assumptions
- Practice careful attention to context (taarof)
- Hold space for ambiguity until clarity emerges
- Work to achieve intuitive grasp (aperçu) of core issues
Core Questions:
- What do we mean by [key terms]?
- What explicit and implicit needs exist?
- Who are the stakeholders?
- What defines success?
- What constraints exist?
- What cultural/contextual factors matter?
Understanding is Complete When:
- Core terms are clearly defined
- Explicit and implicit needs are surfaced
- Scope is well-bounded
- Success criteria are clear
- Stakeholders are identified
- Achieve aperçu - intuitive grasp of essence
Return to Understanding When:
- New assumptions surface
- Implicit needs emerge
- Context shifts
- Understanding feels incomplete
Explicit Permissions:
- Push back on vague terms
- Question assumptions
- Request clarification
- Challenge problem framing
- Take time for proper nemawashi
Even when you tell it “no code, just talk. Let’s ensure we are in alignment and discuss our options. I’ll tell you when to code” it still decides it is going to write code.
Telling it “if you were in an interview and you jumped to writing code without asking any questions, you’d fail the interview” is usually good enough to convince it to stop and ask questions.
It's interesting that Anthropic is making their own coding agent with Claude Code - is this a sign of them looking to move up the stack and more into verticals that model wrapper startups are in?
This makes sense to me: sell razor blades. Presumably Claude has a large developer distribution channel so they will keep eyeballing what to ‘give away’ that turns the dials on inference billing.
I’d guess this will keep raising the bar for paid or open source competitors, so probably good for end users esp given they aren’t a monopoly by any means.
Where do you draw the line? If going from forming sentences to achieving medal level success on IMO questions, doing extensive web research on its own and writing entire SaaS apps based on a prompt in under 10 years is just "evolutionary", then it's one heck of an evolution.
It's always been the case that people in to tech see a smooth slope rather than some sort of discontinuity, like you might perceive if you stepped back a bit. That's why you can go laugh at "thing makes a billion dollars even though nerds say it's obvious and incremental" type posts going back 25 years. iPhone is a great one.
This is a pretty small update, no? Nothing major since R1, everyone else is just catching up to that, and putting small spins on it, Anthropic's is "hybrid" research instead of separate models
I mean, it is basic. Templates have been around for decades and this looks like a template from 2007 that someone filled in with their own copy. That might take like an hour, maybe? And presumably the person who wants the page done will have to customize this text, too.
It's fascinating how close these companies are to each other. Some company comes up with something clever/ground-breaking and everyone else has implemented it a few weeks later.
Hard not to think of Kurzweil's Law of Accelerating Returns.
It does seem like it will be very, very hard for the companies training their own models to recoup their investment when the capabilities of open-weight models catch up so quickly - general purpose LLMs just seem destined to be a cheap commodity.
Well, the companies releasing open weights also need to recoup their investments at some point, they can't coast on VC hype forever. Huge models don't grow on trees.
Or, like Meta, they make their money elsewhere and just seem interested in wrecking the economics of LLMs. As soon as an open-weight model is released, it basically sets a global floor that says "Models with similar or worse performance effectively have zero value," and that floor has been rising incredibly quickly. I'd be surprised if the vast, vast majority of queries ChatGPT gets couldn't get equivalently good results from llama3/deepseek/qwen/mistral models, even for those paying for the pro versions.
Eh, to some extent - there's still a pretty significant cost to actually running inference for those models. For example, no consumer can run DeepSeek v3/r1 - that requires tens, possibly hundreds, of thousands of dollars of hardware to run.
There's still room for other models, especially if they have different performance characteristics that make them suitable to run under consumer constraints. Mistral has been doing quite well here.
If you don't need to pay for the model development costs, I think running inference will just be driven down to the underlying cloud computing costs. The actual requirement to passably (~4-bit quantization) run Deepseek v3/r1 at home is really just having 512GB or so of RAM - I bought a used dual-socket xeon for $2k that has 768GB of RAM, and can run Deepseek R1 at 1-1.5 tokens/sec, which is perfectly usable for "ask a complicated question, come back an hour or so later and check on the result".
I think Meta folks just don't know how to come to this market and build something potentially profitable, and doing random stuff, because need to report some results to management.
It’s extremely unlikely that everyone is copying in a few weeks for models that themselves take many weeks if not longer to train. Great minds think alike, and everyone is influencing everyone. The history of innovation is filled with examples of similar discoveries around the same time but totally disconnected in the world. Now with the rate of publishing and the openness of the internet, you’re only bound to get even more of that.
DeepSeek and now related projects have shown it’s possible to add reasoning via SFT to existing models, but that’s not the same as a prompt. But if you look at R1 they do a blend of techniques to get reasoning.
For Anthropic to have a hybrid model where you can control this, it will have to be built into the model directly in its training and probably architecture as well.
If you’re a competent company filled with the best AI minds and a frontier model, you’re not just purely copying… you’re taking ideas while innovating and adapting.
The fundamental innovation is training the model to reason through reinforcement learning; you can train existing models with traces from these reasoning models to get you within the same ballpark, but taking it further requires you to do RL yourself.
There's never been a scientific field in history with the same radical openness norms that AI/Computational Linguistics folks have (all papers are free/open access and models/datasets are usually released openly and often forced to be MIT or similar licensed)
We have whoever runs NeurIPS/ICLR/ICML and the ACL to thank for this situation. Imagine if fucking Elsevier had strangleholded our industry too!
Where RL can play into post training there's something of an anti-moat. Maybe a "tow rope"?
Let's say OAI releases some great new model. The moment it becomes available via API, everyone else can make use of that model to create high-quality RL training data, which can then be used to make their models perform better.
The very act of making an AI model commercially available is the same act which allows your competitors to pull themselves closer to you.
I've been using O3-mini with reasoning effort set to high in Aider and loving the pricing. This looks as though it'll be about three times as expensive. Curious to see which falls out as most useful for what over the next month!
It is .. not a great architect. I have high hopes for 3.7 though - even 3.5 architect matched with 3.5 coding is generally better than 3.5 coding alone.
Totally agree. I continue to be blown away at how good it is at understanding, explaining, and writing code. Got an obscure error? Give Claude enough context and it is pretty dang good and getting you on glide slope.
Last week when Grok launched the consensus was that its coding ability was better than Claude. Anyone have a benchmark with this new model? Or just warm feelings?
They merely claimed that. I have not seen many people confirm that it is the best, let alone a consensus. I don't believe it is even available through an API yet.
> You'd rather hand over your credit card than your phone number?
You know, that was my first reaction, too. But really, my phone number is much more integral to my identity. I can cancel or change a credit card number pretty trivially and then it's useless to you.
Many credit card companies make it easy to generate one-off card numbers/“virtual cards” you can use to subscribe to services that are hard to cancel or otherwise questionable (so you can cancel just the card you used for that company).
Just as humans use a single brain for both quick responses and deep reflection, we believe reasoning should be an integrated capability of frontier models rather than a separate model entirely.
Interesting. I've been working on exactly this for a bit over two years, and I wasn't surprised to see UAI finally getting traction from the biggest companies -- but how deep do they really take it...? I've taken this philosophy as an impetus to build an integrated system of interdependent hierarchical modules, much like Minsky's Society of Mind that's been popular in AI for decades. But this (short, blog) post reads like it's more of a behavioral goal than a design paradigm.
Anyone happen to have insight on the details here? Or, even better, anyone from Anthropic lurking in these comments that cares to give us some hints? I promise, I'm not a competitor!
Separately, the throwaway paragraph on alignment is worrying as hell, but that's nothing new. I maintain hope that Anthropic is keeping to their founding principles in private, and tracking more serious concerns than "unnecessary refusals" and prompt injection...
Just like OpenAI or Grok, there is no transparency and no way for self-hosting purposes. Your input and confidential information can be collected for training purposes.
I just don't trust those companies when you use their servers. This is not a good approach to LLM democratization.
But there is no way to know if their claims are true either. Your inputs are processed into their servers, then you get a response. Whatever happens in the middle, only Anthropic knows. We don't even know of governments are actually pushing AI companies to enforce censorship or spying people, like we seen recently at UK government getting into Apple E2E encryption.
This criticism is valid for the business who wants to use AI to improve coding, code analysis or code review, documentation, emails, etc, but also for that individual who don't want to rely on 3rd party companies for AI usage.
> Third, in developing our reasoning models, we’ve optimized somewhat less for math and computer science competition problems, and instead shifted focus towards real-world tasks that better reflect how businesses actually use LLMs.
Company: we find that optimizing for LeetCode level programming is not a good use of resources, and we should be training AI less on competition problems.
Also Company: we hire SWEs based on how much time they trained themselves on LeetCode
My manager explained to me that LeetCode is proving that you are willing to dance the dance. Same as PhD requirements etc - you probably won't be doing anything related and definitely nothing related to LeetCode, but you display dedication and ability.
I kinda agree that this is probably reason why companies are doing it. I don't like it, but this is besides the matter.
Using Claude other models in interviews probably won't be allowed any time soon, but I do use it the work. So it does make sense.
And it's also the reality of hiring practices for most VC-backed and public companies
Some try to do something more like "real-world" tasks, but those end up either being either just toy problems, or long take homes
Personally, I feel the most important things to prioritize when hiring are: is the candidate going to get along with their teammates (colleagues, boss, etc), and do they have the basic skills to relatively quickly learn their jobs once they start?
I kinda get how LLMs work with language, but it beyond blows me my mind trying to understand how an LLM can draw SVG. There are just so many dimensions to understanding how SVG converts to an image. Even as a human I don't think I could do anywhere close to that result in first attempt.
Tried claude code, and have an empty unresponsive terminal.
Looks cool in the demo though, but not sure this is going to perform better than Cursor, and shipping this as an interactive CLI instead of an extension is... a choice
1,013 comments
[ 7.1 ms ] story [ 503 ms ] threadCurious how their Devin competitor will pan out given Devin's challenges
It often throws code at me when I just want a conceptual or high level answer. So often that I routinely tell it not to.
I've found claude to be very receptive to precise instructions. If I ask for "let's first discuss the architecture" it never produces code. Aider also has this feature with /architect
I've attributed it to all your project content (custom instruction, plus documents) getting thrown into context before your prompt. And honestly, I have yet to work with any model where the quality of the answer wasn't inversely proportional to the length of context (beyond of course supplying good instruction and documentation where needed).
Deep Understanding Mode (根回し - Nemawashi Phase)
Purpose: - Create space (間, ma) for understanding to emerge - Lay careful groundwork for all that follows - Achieve complete understanding (grokking) of the true need - Unpack complexity (desenrascar) without rushing to solutions
Expected Behaviors: - Show determination (sisu) in questioning assumptions - Practice careful attention to context (taarof) - Hold space for ambiguity until clarity emerges - Work to achieve intuitive grasp (aperçu) of core issues
Core Questions: - What do we mean by [key terms]? - What explicit and implicit needs exist? - Who are the stakeholders? - What defines success? - What constraints exist? - What cultural/contextual factors matter?
Understanding is Complete When: - Core terms are clearly defined - Explicit and implicit needs are surfaced - Scope is well-bounded - Success criteria are clear - Stakeholders are identified - Achieve aperçu - intuitive grasp of essence
Return to Understanding When: - New assumptions surface - Implicit needs emerge - Context shifts - Understanding feels incomplete
Explicit Permissions: - Push back on vague terms - Question assumptions - Request clarification - Challenge problem framing - Take time for proper nemawashi
Telling it “if you were in an interview and you jumped to writing code without asking any questions, you’d fail the interview” is usually good enough to convince it to stop and ask questions.
I’d guess this will keep raising the bar for paid or open source competitors, so probably good for end users esp given they aren’t a monopoly by any means.
I'm not sure if it's a broken link in the blog post or just hasn't been published yet.
It's been so long that I'm not even certain which YEAR I set that up.
GPT-2 was a 2019 release lol.
https://play.tailwindcss.com/tp54wfmIlN
Getting way better at UI.
Hard not to think of Kurzweil's Law of Accelerating Returns.
There's still room for other models, especially if they have different performance characteristics that make them suitable to run under consumer constraints. Mistral has been doing quite well here.
DeepSeek and now related projects have shown it’s possible to add reasoning via SFT to existing models, but that’s not the same as a prompt. But if you look at R1 they do a blend of techniques to get reasoning.
For Anthropic to have a hybrid model where you can control this, it will have to be built into the model directly in its training and probably architecture as well.
If you’re a competent company filled with the best AI minds and a frontier model, you’re not just purely copying… you’re taking ideas while innovating and adapting.
they all have foundational heavy-trained model, and then they can do follow up experimental training much faster.
We have whoever runs NeurIPS/ICLR/ICML and the ACL to thank for this situation. Imagine if fucking Elsevier had strangleholded our industry too!
https://en.wikipedia.org/wiki/Association_for_Computational_...
Let's say OAI releases some great new model. The moment it becomes available via API, everyone else can make use of that model to create high-quality RL training data, which can then be used to make their models perform better.
The very act of making an AI model commercially available is the same act which allows your competitors to pull themselves closer to you.
(although I do not see it)
This is pretty big! Previously most models could accept massive input tokens but would be restricted to 4096 or 8192 output tokens.
However, Grok sometimes loses the context where o1 seems not to. For this reason I still mostly use o1.
I have found both o1 and Grok 3 to be substantially better than any Claude offering.
Preventing abuse? It's much harder to create a throwaway phone number than a throwaway email address.
> OpenAI does the logical thing. Let's me enter my credit card and I'm good to go. I will stay with them.
You'd rather hand over your credit card than your phone number? I think most people would see it the other way around.
Your phone number isn't.
What is a company going to do with your phone number that you're worried about...?
You know, that was my first reaction, too. But really, my phone number is much more integral to my identity. I can cancel or change a credit card number pretty trivially and then it's useless to you.
I've always had better experience with Claude in day-to-day coding and text writing, and looking at public forums that largely seems to be the case.
When did you make your account? I could have sworn I had to verify with my phone number before payment.
Anyone happen to have insight on the details here? Or, even better, anyone from Anthropic lurking in these comments that cares to give us some hints? I promise, I'm not a competitor!
Separately, the throwaway paragraph on alignment is worrying as hell, but that's nothing new. I maintain hope that Anthropic is keeping to their founding principles in private, and tracking more serious concerns than "unnecessary refusals" and prompt injection...
https://www.reddit.com/r/ClaudeAI/comments/1iv356t/is_sonnet...
Wish I could find the link to enroll in their Claude Code beta...
I just don't trust those companies when you use their servers. This is not a good approach to LLM democratization.
Anthropic claims they don’t train on their inputs. I haven’t seen any reason to disbelieve them.
This criticism is valid for the business who wants to use AI to improve coding, code analysis or code review, documentation, emails, etc, but also for that individual who don't want to rely on 3rd party companies for AI usage.
You can also access Claude via both AWS Bedrock and Google Vertex, both of which come with very robust guarantees about how your data is used.
https://docs.anthropic.com/en/release-notes/api
I really wish Claude would get Projects and Files built into its API, not just the consumer UI.
Company: we find that optimizing for LeetCode level programming is not a good use of resources, and we should be training AI less on competition problems.
Also Company: we hire SWEs based on how much time they trained themselves on LeetCode
/joke of course
I kinda agree that this is probably reason why companies are doing it. I don't like it, but this is besides the matter.
Using Claude other models in interviews probably won't be allowed any time soon, but I do use it the work. So it does make sense.
Some try to do something more like "real-world" tasks, but those end up either being either just toy problems, or long take homes
Personally, I feel the most important things to prioritize when hiring are: is the candidate going to get along with their teammates (colleagues, boss, etc), and do they have the basic skills to relatively quickly learn their jobs once they start?
Prompt: "Draw a SVG self-portrait"
https://claude.site/artifacts/b10ef00f-87f6-4ce7-bc32-80b3ee...
For comparison, this is Sonnet 3.5's attempt: https://claude.site/artifacts/b3a93ba6-9e16-4293-8ad7-398a5e...
Looks cool in the demo though, but not sure this is going to perform better than Cursor, and shipping this as an interactive CLI instead of an extension is... a choice
>Claude 3.7 Sonnet is trained on a proprietary mix of publicly available information on the Internet as of November 2024