> I noticed this too as did many others. Had to bump my default reasoning but it’s not the same. Seems to often happen before new model drops
I heard that Qwen4 was dropping some time soon, I wonder if Anthropic is trying to announce a new model immediately after Qwen4 - both for their IPO, and to kill the competition.
This has happened to consistently across OpenAI and Anthropic models that I can't believe people are still denying it. Obviously they're silently quantising their models or making other efficiency changes. If they can save 40% RAM and only hurt intelligence by a few percent in their benchmarks, they're going to do that. Unfortunately their benchmarks are obviously not representative of the real world, and these downgrades are very noticeable for people who rely on these models for work.
Of all the reasons to look at local inference, this is the one which pushes me in that direction the most. I want to rely on a model for consistency in my workflows. If I wake up on Monday and suddenly it's refactoring things it built on Friday, that's a huge problem. That makes it unreliable.
Probably because those "nerfs" do not show on sites that track model performance over time. Sometimes Anthropic/OpenAI mess stuff up in their harness, but usually not in the models themselves.
People are denying it because it's simply not true and for sure has not happened consistently.
In my own work I have not once noticed a degradation where I could have said with any certainty that it's not within usual deviation or me having a bad day.
Conspicuously the model seems to be especially dumb and annoying when I did not get a lot of sleep the previous night.
When OpenAI and Anthropic engineers post here on HN that they do not quantize models and silently replace checkpoints in the API, I believe them.
I saw nothing that convinced me the people who are constantly making these complaints don't just mistake any suboptimal result for changes in the model.
It’s also true that people that perform their own evals see wide variance uncorrelated with “nerfing”. Human cognitive bias makes them only see that variance at specific times and then they start projecting.
Everyone really should keep a few tasks around that are indicative of the work they do and run them a few times with the same model and a few times across model families and when new models drop.
It’s extremely informative to see with your own eyes the variance that always exists. Even if you can’t afford statistical relevance it’s valuable.
I've "called" the last 3 out of 4 new model releases. Maybe my account it perma-tagged as one that gets downgraded to re-allocate resources before a release but about 5-7 days before a new model Claude suddenly gets dumber and dumber on executing canned skills that I've used for 6+ months with no issues. The cynic in me thinks they are dumbing down models right before a release so the new model seems much smarter.
Yes I have similar anecdotal experiences - it gets dumb before a new model. What I imagine in my head is Anthropic using all their GPUs to polish off the next model and they need to borrow GPU cycles so they cut down the thinking.
I am using Opus 5.5 high almost exclusively. I definitely felt this and looked for nerfing watch sites. It felt like downgrading to the old version of deepseek. Things have improved a bit now.
You can still control it if you can trivially spin up a node with eight B200s and 2TB of memory to run your Deepseek v4 instance. Nobody on a subsidized plans is doing that. Local models that fit 128GB memory don't come close even to a "nerfed" frontier model.
Actually Qwen3.8-flash-next comes very close to a "nerfed frontier model" the way I host it locally at 4.75bpw with int8 kv. It is pretty much equivalent to opus 4.6 up to 4.8 on coding. What it lacks is some factual knowledge - this is to be expected, it's a far smaller model. But if you run the kind of coding tasks where you have to spec them in high detail anyway (large very custom environment) this model is very usable.
I run it on a 6*rtx3090 consumer PC (no threadripper etc sadly) with very constrained pcie topology (4 cards sit on 2 lanes basically) and i still get 7k prefill and 50t/s single user or 200t/s 7 users. All at full 262k contexts of course and with 96GB of ram kv cache that means I can easily have 10+ Claude code sessions open with 1-7 generating at any time.
I've been the person that said, before opus 4.6 came out that this AI stuff is absolute crap for programming. Then it suddenly wasn't and I was using it many hours each day.
Local AI was the same. It was all crap. Models like GLM5.3 Flash that are many times larger than that qwen, lots of people are oooing and ahhing over I found way too inferior. Previous DeepSeek V4.1 Flash 0731 was atrocious. They benchmaxed it to hell, but at least it was small.
The current model of same name but release 1017 and double the size is currently the best open source model IMO. It even beats my self hosted Qwen3.8-flash-next.
I've tried hosting that DeepSeek model, but even adding another rtx3090 for a total of 7 cards and my 192gb of ram bought when it cost under $1000 I get only about 25t/s decode and few hundred prefill. That is far too slow for me to use.
But, I believe once this pyramid scheme of circular financing finally bursts there will be a glut of hardware on the market where people like me will be able to buy equipment to run 750B models like we now run 120B.
I've been very impressed by it and recommended it to everyone. 32gb ram, 12gb 3080ti, and I run 3.8 flash next coder (half the experts ripped out, leaving in coding) int8 kv 128k context at 35-55tk/s and I agree with the opus comparison; I personally say 4.7 is what it seems like the most.
I read stuff like this and just think that if these open models work well enough financially you'd still be better off using them via OpenRouter or some other hoster.
Definitely a significant change. I noticed it in just daily work. Quality of work down, mistakes up, repetitive errors in scripts - rare earlier, now a few every session.
I noticed Fable decline in quality a couple weeks ago. And I've noticed Opus drop over the last week, even after bumping to a default of Extra High reasoning for almost everything.
Tonight has been maddening with the kinds of absurd mistakes it's repeatedly making. It's so frustrating.
I'm the kind of person that wants something to be consistent and reliable. And Anthropic has been great, except for in this way. Which is enough of a recurring frustration (about every couple months this year) that I genuinely can't wait for other options to catch up. Then I'll be leaving as soon as I can.
I've been using Fable for a couple months as a relatively independent, long-horizon task executor and it worked well. About two weeks ago it completely stopped working. Like ~85% success rate down to ~10% in my workflows.
Totally possible that I'm just hallucinating/projecting/etc or my workflows somehow changes in terms of difficulty, but it's been kind of shocking.
I have the same exact experience. Working on one project, brainstorming. It was brilliant in July and August, like a seriously smarter, more experienced version of myself, reading in my mind, one step ahead of me, when pointed out a mistake it would remedy on the spot with the exact brilliant solution.
Now it's like talking to my hungover self, after 3 days bender, who just wanted to be left alone.
Same for me, especially the last couple of days. It was really frustrating and it felt like Claude was sick of working with me and just didn’t give a f*ck anymore
I guess there are experimenting with they can enshittify. But even if they undo this current change, all commercials model will eventually be enshittified.
I read somewhere that the test might switch models mid conversation which stops the other from reading the previous model's reasoning. I don't know if that's true, but it sure feels like that quite often in the last days.
I'm a pretty voracious 20x user, and OAI Biz Pro user; however I'm not an engineer by training, a previous tech adjacent dev team leader who always relied on pseudo code and broadening my horizons when dealing with my team. ( AdTech, DTC marketing )
So I'm AI pilled and in another career, and started to think, what if the problem is the inputs? They release Opus X.X and at first it's a rush of their best users, best use cases, upgraded lanes. As weeks go on, adoption increases, cruft uses start to poison the model? Is this how the ongoing evolution works? I'd love to get Sonnet 5.5 and Opus for review in a place where I could reliably daily drive for X months / Y years etc.
I just cancelled my x20 anthropic sub after their last usage reduction shenanigans, and moved over to DeepSeek and opencode go. Yeah opus is a bit smarter, but not even close to justify the 20x price relative to DeepSeek.
There is random input to make the models evaluate differently every time, but without that (or if you fix it) then the model will always give the same output with the same input.
There's no somehow to it, it's actually surprisingly hard to make a computer do something in a non-deterministic fashion.
Nope, the issue was not the 5 h limit, but that the 5 hour limit was gone in some minutes and that this burned trough 50% of my weeks fable limit. Before I could do roughly one full 5h session (fable till max) per day.
They're actually extremely predictable. What you are experiencing is the provider you are entrusting with your time, effort, and money fucking you because they need the capacity to train the next one. So without any real notice or indication other than things vaguely seeming broken in ways everyone seems to be experiencing differently, they're serving up some shitty quant of what you already paid to use.
Extremely predictable is an exaggeration. I was teaching machine learning and we needed half a dozen lines to make even simple networks deterministic. LLMs are not really predictable, even if we want to, they will sometimes behave almost chaotic depending on the input.
My understanding of LLMs are that they are giant functions and generate output deterministically. However, the way these providers setup and route traffic means that your traffic lands on different boxes where each box has different inputs that run alongside your prompt -- essentially creating "non-deterministic" behavior. Basically it's BS to make LLMs look sentient. Each model could be burned onto an ASIC if you really wanted to be locked into that model and get incredible token speeds. If I'm wrong I would like to be corrected, thanks.
Eagerly awaiting Artificial Analysis to redo their Anthropic benchmarks and kick it down 15-20 index points, in the interests of accuracy. Those guys love accuracy. They moved Heaven and Earth to kick Kimi off the #2 spot, out of their deep, abiding passion for having the most accurate benchmarks in the industry.
I have no doubt they will make sure their users will get an accurate picture of Opus 5.5 within hours.
If I were a cynic I’d also say that it wouldn’t be particularly difficult for them to detect when a benchmark/eval context is active and to route those requests to larger models or higher quants.
IMHO the only defensible way to benchmark this is to use secret-shoppers via subscriptions. As is standard practice in other product-review fields, where the stakes are a lot lower.
The purpose of AI benchmarking is to inform people making large spending decisions what alternatives are available to them.
I *do* wish we wouldn't let software companies call this "A/B testing". That's not what that term means to a typical reader. In any other market than software, you'd call it something like "misleading consumers in respect to product composition" and "an unfair commercial practice". That's verbiage from the EU "dual quality" food scandal, where transnational food brands (Coca-Cola was one) adulterated their recipes to use cheaper ingredients in select markets, though the external packaging was kept the same. (A very apt analogy, isn't it?)
The way Anthropic continues to get away with testing these nerfing "experiments" on paying customers without their consent is wrong.
The Windows Insider program has shown plenty of people will willingly sign up to be testers for a taste of the rawhide, Anthropic could do this in a transparent way but they continue to choose not too.
They can get away with it because they are so far ahead of OpenAI models for coding, it's not like you can switch to anything else if you don't like it. Astra is even worse than nerfed 5.5.
Pretty much the only alternative is loading up some credits and using Fable.
My experience is that 6.1 Sol is a little behind Opus 5.5 but it’s in spitting range. Their cost per task is generally higher, but if you’re on a quota plan the difference in cost doesn’t really matter until you’re paying API rates.
Both get the job done these days. I still hate Opus 5.5’s output, 6.1 Sol does an acceptable job of doing a second cleanup pass to it.
Astra when it hit the ground felt like “A Fable monent”: it was so good and concise and just was done with whatever task in a short time, where opus and fable sometimes can take forever. I think it lasted a week or two until it started to deteriorate to where we are now. Coincided nicely with the stopping of codex20 subscriptions etc.
Sol/Astra are sweet but they also have pretty much no discipline or executive function, which is where Opus 5.5 comes in, and I've not seen that capability in it nerfed too bad in my experience.
Yes this is the experiment I have been running for a while, that shows the constant comments on Claude models here have to be push from Claude team friends...The experience of output of OpenAI models is consistently better.
1)- Get a coding task and get the output of Opus and Astra.
2)- Open a new session and ask each other to review in detail the solution from the other model without mentioning where is it coming from.
3)- After the review, ask the original model to comment on it and state which solution is better and if the criticisms are valid...all in a new session.
4)- At the end Opus and others from Anthropic always end up agreeing the OpenAI model solution is better. My human review agrees.
After doing this for a few weeks...and consistently having their models rating higher the OpenAI solutions...I cancelled the Anthropic subscription...
Isn't this inherently task/domain/programming language/even prompting style specific? Why can't other people have genuinely different experience than you?
Yep, just one hour ago I again reached my weekly limit on claude code (3 days till reset btw), using Fable as Head-Orchestrator + Codex sub-agents via custom mcp.
I tried $20 of usage credits as a test once the limit was reached.
I shit you not, it lasted for less than 5 minutes.
Yeah it's a compete joke/scam. I bought $200 of credits once... gone in an hour. Never again. Part of me is thinking they decreased usage partly to force users to buy credits before the IPO... but at least in this case they ended up losing a user.
I get 5x more done (of equivalent quality) on my opencode go + DeepSeek flash sub for $40, than $200 anthropic sub.
Yeah it works great, maybe not quite as good as opus 5, but it's literally 20x-50x cheaper, and several times faster, an extra review/refinement loop per task brings the work up to opus level while still being much faster and cheaper overall. I have my own setup/tool (which I hope to open source soon) for planning/tasking/tracking which I found helpful to keep things on track an not forget/redo things - regardless of model - essentially tasks, dependencies/hierarchy, priorities, requirements, and task completion evidence tracked in the repo in structured plaintext/markdown files. DeepSeek was able to very easily understand and follow the workflow right off the bat.
Ok, thanks, I'm going to try to setup the pi harness and add on some needed extensions for my workflow. I'd appreaciate any advice in regards to any addons you would recommend if you have them.
Eh sometimes it cuts the other way, like when Opus 5.5 was being shadow-routed on Opus 5. I am on a team plan and seem to not get any of these A/B tests.
Realistically it can't match Fable, let alone Opus today. It can however beat basically all of the models from 1 year ago. If last years state of the art is "good enough" for you today then local is great.
Reasonable throughput depends a lot on your hardware. I'm running Qwen3.8-Flash-Next on a 128GB Strix Halo box at 50t/s gen (using the gufo inference engine) and it feels like Opus4.6/4.8 level performance. It's an amazingly good model. I've got it coding a C->microcode compiler in OCaml and it's doing great (there's also verilog code generation involved as well so this is a pretty good stress-test, I think).
Flash Next is really great, but you still need pretty decent hardware. A DGX Spark, Strix Halo, or Mac with a lot of memory. You can use things like Strata to squeeze it into a system with only 32GB of VRAM by offloading a lot of it to system memory and still get something that "works", but the quality falls off a cliff too. At that point you get better results out of Qwen 3.8 27b in int4.
Never go below a four bit quant if you want good results. A worse quant of a bigger model is often easily beaten by the small model at decent quality. A lot of the people who think local AI suck think that because they don't understand that low quants == dumbass model. You aren't just compressing data, you're reducing the amount of compute it uses which effectively lowers it's IQ.
If you want to run something actually useful for cheap Qwen 3.8 27b on a 32GB GPU in int4 should be passable. It's more like Sonnet for a year ago. To match last years Opus you really need at least 128Gb.
Personally I have two Sparks connected with ConnectX 7. Now I can run GLM 5.3 locally at 30-40 tps and it's honestly been great. It really cut my dependence on cloud providers and I can run an abliterated version that never refuses my Cyber related tasks.
Some of the weird new quants really do push the limits, but at the end of the day that compute you're eliminating is the actual 'work' the model is doing.
Folks like to think of 'weights' as just storage, but they aren't (except with stuff like engrams). Those digits of precision encode actual computation, not just storage. There is always a cost, it's lossy compression after all. It's like MP3 vs AAC. If you can't hear the difference it just means you haven't encountered it YET.
That's not to say that you ever will. The cost could be in weird far out edge cases, but whatever it once encoded... It's gone now and it ain't coming back.
Maybe at Opus 4.6 level for the "easier" tasks. But honestly if you were given a choice between a computer with Qwen 3.8 or one with even older Sonnet 5, both local, same tok/s, window size, etc... what are you picking?
100% agree. A local model might not have the same stretch goal capabilities, but at least I don't have to constantly second guess myself in terms of the inconsistencies I think I'm noticing.
No longer am I constantly made to feel like Ingrid Bergman in that movie with all the lights.
They have lowered the price several times; and often released more for the same price. But it is normal to consider the best ever price to performance the benchmark and anything below that disappointing.
IPO is coming so they will have to maximize profits going forward which will mean constraining the compute they give everyone to ensure the subscriptions make money.
By selling subscription tokens for less than they cost to generate, of course.
What you said doesn't yield any information whatsoever about whether or not subscriptions are profitable, all that says is that subscriptions are less profitable than API.
Things are able to be less profitable than other things while still being profitable.
If something like the API is very very profitable, it's even possible for profitable things to be much less profitable!
i've seen people on social media claiming all sorts of things about Fable 5.5, Opus 5.6 and Sol 6.2 and what-not for the past few weeks, but it just seems like engagement farming with no proof
They claim they've used the same cli version, but don't publish their methodology (maybe I've missed it), eg how do I know they didn't accidentally turn on claude code auto update after all? At least https://github.com/ninjahawk/livenerf seems unchanged so far and is much more thorough...
I guess it's just capacity problems. They probably have a fixed amount of compute and they need to make space for new models that will be released. So they quantize existing models e.g. in half so that can use 50% of current capacity for new models. They have such massive scale that you just cannot dynamically get that amount of new compute online so they need to work with what they have. No provider has thousands of GPUs idle.
There are a couple documented things where it has degraded: in late 2025 or so when they transitioned to TPUs, they had a bug in the compiler affecting sampling logits in a cheaper approximate way than GPUs and that caused a big degradation. They never told users "sometimes we'll sample logits in a cheaper approximate way," but the main issue was the bug and not really an intentional thing at least from how the characterized it.
TPUs only made up a small percentage of the fleet, but, exacerbating that bug, if you got routed to a TPU instance at some point in your chat then another bug kept you routed there for the remainder of the chat.
The second one I'm aware of is earlier in 2026 they decided to save usage by stripping out all thinking tokens if you left your chat idle for one hour (cache timeout) and then came back to it, causing massive model performance degradations. This one also made me very skeptical of Chinese distillation, until the reports of how you could use Haiku to extract the raw reasoning of the larger models.
They fixed that, but recently they switched to doing auto-compactions for long running chats in anticipation of them going stale in cache. I think that one is usually what you want anyway and there is at least a way to recover to the uncompacted chat if you really need it, but it might also be a source of some degradation people are seeing now.
I think the bug one wasn't bad intent, but it was a fact that even without the bug you got different behavior with the approximate logit sampling on TPUs. I say that was jut a minor platform optimization though and wouldn't be too upset about them not disclosing it.
But the harness one was not just a random harness regression; they were getting a huge influx of users and pretty clearly did it to save capacity. Not a great thing when you might be relying on attention to those thinking traces to reduce the risk of prompt injection etc. and it could have gone beyond major task degradation into a safety/security issue.
Yeah, perhaps but they are also trying to make the harnesses more efficient so they are competitive with their competitors on efficiency. Reducing token use can also reduce context size and improve aligned accuracy in general..
Through the lens of a conspiracy theory this could be viewed narrowly as "cheating" their users to reduce load.
But there are other motivations for the constant work going on in the harnesses that paint a fuller picture.
This is what I've been wondering. I mean we all know they are compute constrained so I get that intuition to a point.
However, how would less compute result in degraded performance as opposed to more latency? Like is amount of compute something you can actually tune to control for latency? Wouldn't that imply they are actually serving requests in other models? If I run a very large model locally on a shit computer vs. something maxed out, it's the latency that differs.
The major providers like Azure along with OpenAI and Anthropic have the concept of "priority processing" which comes with service level guarantees on latency and token throughput. It's more expensive.
They basically dynamically allocate compute into this pool from the general pool to meet demand and SLAs. If demand for priority processing spikes latency in the general pool spikes.
Almost all subscriptions use general pool. That's why it gets really slow sometimes. Unless you use "fast" which acts like priority processing and is super expensive.
So that's part of how they balance compute based on demand and fund the "excess" capacity. This is a bit reductive and basic, but it's all the usual Cloud on demand, reserved, and spot capacity techniques at play.
Thank you for the information but I'm not sure you understand my question. I'm not surprise to hear these details on how they control for latency.
Im asking on how lower compute could translate to worse model performance. That's the implicit claim being made when people are saying Anthropic nerfs a model (result is bad, not necessarily slow). The only way I can think that could happen is if they actually swapped Opus 5.5 out for a worse model that takes less resources to do inference.
Yes, that's what they are claiming; that the full(er) precision models get heavily quantized and/or distilled to lesser, "nerfed" models and requests get routed to them based on load, time since release, time to new model release, and reading chicken bones.
There is no proof, the frontier labs deny it, and certainly it's not happening on hosts like Azure, Amazon, and etc.. The claim is that it's "trivial" to detect observation, so they serve the normal model if you try to measure it..
Basically the conspricy theory is:
* OpenAI/Anthropic nerf their models regularly to reduce compute demand
LLMs are such a great business, they are non-deterministic, hence you can do whatever you want, because customers can't do anything about it:
* nerf them based on supply and demand, when you have a high demand, nerf it, but keep charging same price.
* do you want to show revenue growth to investors? increase thinking tokens, customers will keep paying, because again, this is non-deterministic system, sometimes it thinks more
* do you want to increase profits? start serving 4-bit, 5-bit quantizations and keep charging same price
* want even more? publish a paper "Advanced Auto Super Smart Prompt Router - AASSPiRation", and start routing requests to Qwen3.8-27B or GemmaE2B from time to time on the edge
* Not enough? introduce "Complex Chained Workflows - CCW" in your coding harness, which spawns multiple deterministic workflows (which you can't stop or control) and they will use Haiku to write the code, then use Opus 5.5 high to verify the code (which is obviously wrong) and Opus 5.5 produces recommendations on how to fix the code, delegates to Haiku, which again fails writing proper code, and keep doing it 5-6 times, you made lots and lots of money (don't forget, all should run in parallel, to pick one best result among 10 outputs).
You can say the same thing about nearly any business. Lyft and Uber, for instance. You can lower quality, raise price, whatever. This is why open markets matter so much - when you have competition, businesses can't materially worsen their customers' outcomes for long or those customers switch.
How do you even lower the quality - serve half broken car for a luxury order? No way person who orders luxury car accepts it.
Or do you increase wait times? - sure, order gets cancelled, you won't get a payment for that.
There are of course ways, but you still can do something, in case of LLMs, you blame yourself for poor prompting and retry, which again makes money to LLM providers
- You can lower the quality standard of which car counts as which class. There are many classes of vehicle on both platforms.
- You can delete or not consider low ratings from certain users. Both companies do this already.
- You can increase wait times, of course. It doesn't at all follow that the order gets cancelled. It depends on the market and what each company believes the other is doing.
Uber/Lyft are infamous for using their control over both ends of the transaction to milk both ends for money. They have heavy use of manipulative dark patterns. They know the entire market on both ends so they can manipulate the spread to optimize profit for themselves... and they can hide much of it simply by not telling users.
I noticed the same and posted about it three days ago!
Opus is suddenly fumbling basic tool calls and failing to finish tasks even at high, where I had been permanently using it on low effort with success since it came out. It is very frustrating.
My guess is they quantized it and/or reallocated DC capacity to train a new model, this has happened before.
Not just frustrating, it can also be dangerous. What if you trust Claude with certain security-sensitive things (because experience), and suddenly you cannot trust it anymore like that?
Right lol, i didn't know we'd built a machine that was capable of getting critical level security deterministically correct. That's not me, at least...
It really though?? Even if you set temperature to 0 I saw research showing that the scheduling of the order of operations in the GPU firmware itself can change trailing decimals places and affect the output token
I'm aware operations can technically finish at different rates throughout all of a GPU's cores, but I am not exactly sure that GPU firmware would change the outcome of a calculation (unless there's something specifically in the firmware for it)
If we're being pedantic, it may be deterministic from the point of view of Anthropic. They can guarantee that my prompt X results in actual prompt X+Y (Y being all the settings they control), which would hopefully result in deterministic output Z. From my point of view, Y may as well be random, resulting in non-deterministic output Z for my input X.
I noticed its been trying to drag out the conversation on every topic. Maxxing token consumption for the dollars instead of staying on point. Makes sense for them but its frustrating for me as a user. I suspect this is going to be the AI version of enshittification. Now its only missing some badly needed in-stream ads.
Multiple instances of failing a Bash tool, calling bash (lowercase) and getting rejected and even reprimanded by the API "No such tool, call Bash instead".
Killing it's own shell, multiple, multiple times in the same session. I'm sure I saw a phrase "I did it again".
It's so shit that it must be damage control. The alternative is to limit usage and that'd affect the IPO even more.
Anecdotal, but I tend to get by with medium thinking on Opus and I noticed sometime last week that it seemed to have become noticeably slower to think about things.
I wouldn't say I've especially noticed its effectiveness having been reduced, just slower.
438 comments
[ 3.1 ms ] story [ 67.9 ms ] thread> I noticed this too as did many others. Had to bump my default reasoning but it’s not the same. Seems to often happen before new model drops
I heard that Qwen4 was dropping some time soon, I wonder if Anthropic is trying to announce a new model immediately after Qwen4 - both for their IPO, and to kill the competition.
Of all the reasons to look at local inference, this is the one which pushes me in that direction the most. I want to rely on a model for consistency in my workflows. If I wake up on Monday and suddenly it's refactoring things it built on Friday, that's a huge problem. That makes it unreliable.
Probably because those "nerfs" do not show on sites that track model performance over time. Sometimes Anthropic/OpenAI mess stuff up in their harness, but usually not in the models themselves.
In my own work I have not once noticed a degradation where I could have said with any certainty that it's not within usual deviation or me having a bad day.
Conspicuously the model seems to be especially dumb and annoying when I did not get a lot of sleep the previous night.
When OpenAI and Anthropic engineers post here on HN that they do not quantize models and silently replace checkpoints in the API, I believe them.
I saw nothing that convinced me the people who are constantly making these complaints don't just mistake any suboptimal result for changes in the model.
Everyone really should keep a few tasks around that are indicative of the work they do and run them a few times with the same model and a few times across model families and when new models drop.
It’s extremely informative to see with your own eyes the variance that always exists. Even if you can’t afford statistical relevance it’s valuable.
Specifically
> YOU get to decide when the model changes
I run it on a 6*rtx3090 consumer PC (no threadripper etc sadly) with very constrained pcie topology (4 cards sit on 2 lanes basically) and i still get 7k prefill and 50t/s single user or 200t/s 7 users. All at full 262k contexts of course and with 96GB of ram kv cache that means I can easily have 10+ Claude code sessions open with 1-7 generating at any time.
I've been the person that said, before opus 4.6 came out that this AI stuff is absolute crap for programming. Then it suddenly wasn't and I was using it many hours each day.
Local AI was the same. It was all crap. Models like GLM5.3 Flash that are many times larger than that qwen, lots of people are oooing and ahhing over I found way too inferior. Previous DeepSeek V4.1 Flash 0731 was atrocious. They benchmaxed it to hell, but at least it was small.
The current model of same name but release 1017 and double the size is currently the best open source model IMO. It even beats my self hosted Qwen3.8-flash-next.
I've tried hosting that DeepSeek model, but even adding another rtx3090 for a total of 7 cards and my 192gb of ram bought when it cost under $1000 I get only about 25t/s decode and few hundred prefill. That is far too slow for me to use.
But, I believe once this pyramid scheme of circular financing finally bursts there will be a glut of hardware on the market where people like me will be able to buy equipment to run 750B models like we now run 120B.
Also, how much power does that consume at full tilt ?
Truly insane times
That's around 1% of the price of 8x B200.
Tonight has been maddening with the kinds of absurd mistakes it's repeatedly making. It's so frustrating.
I'm the kind of person that wants something to be consistent and reliable. And Anthropic has been great, except for in this way. Which is enough of a recurring frustration (about every couple months this year) that I genuinely can't wait for other options to catch up. Then I'll be leaving as soon as I can.
I just don't know how long that will be.
Totally possible that I'm just hallucinating/projecting/etc or my workflows somehow changes in terms of difficulty, but it's been kind of shocking.
Now it's like talking to my hungover self, after 3 days bender, who just wanted to be left alone.
That's exactly why they did it. Nerf Opus 5.5, then drop Fable 5.5 just before IPO so it looks better next to its weaker predecessor.
And people wonder why open models are a big deal, and why some people love them so much, despite being 30% worse than the frontier.
Having control over something that writes most of your code is important.
Only local AI capable hardware and frontier open weight models can provide that.
So I'm AI pilled and in another career, and started to think, what if the problem is the inputs? They release Opus X.X and at first it's a rush of their best users, best use cases, upgraded lanes. As weeks go on, adoption increases, cruft uses start to poison the model? Is this how the ongoing evolution works? I'd love to get Sonnet 5.5 and Opus for review in a place where I could reliably daily drive for X months / Y years etc.
Surely everyone is that kind of person.
Who wants to pay for a service that doesn't work properly half the time?
"More reasoning" != "better"
This is the #1 thing i HATE with LLMs. They are not predictable like a compiler is.
There is random input to make the models evaluate differently every time, but without that (or if you fix it) then the model will always give the same output with the same input.
There's no somehow to it, it's actually surprisingly hard to make a computer do something in a non-deterministic fashion.
6 lines per model? Can't you put that 6 lines in a function/macro/decorator/library so you only have to call it?
I have no doubt they will make sure their users will get an accurate picture of Opus 5.5 within hours.
The purpose of AI benchmarking is to inform people making large spending decisions what alternatives are available to them.
I *do* wish we wouldn't let software companies call this "A/B testing". That's not what that term means to a typical reader. In any other market than software, you'd call it something like "misleading consumers in respect to product composition" and "an unfair commercial practice". That's verbiage from the EU "dual quality" food scandal, where transnational food brands (Coca-Cola was one) adulterated their recipes to use cheaper ingredients in select markets, though the external packaging was kept the same. (A very apt analogy, isn't it?)
https://ec.europa.eu/commission/presscorner/detail/en/qanda_...
(I suppose Coca-Cola's CEO would prefer we talk about what they did as "A/B testing").
https://github.com/ninjahawk/livenerf
https://www.reddit.com/r/ClaudeCode/comments/1xr9x5y/the_ner...
The Windows Insider program has shown plenty of people will willingly sign up to be testers for a taste of the rawhide, Anthropic could do this in a transparent way but they continue to choose not too.
I don't think that's an excuse, but it is likely the reasoning.
https://news.ycombinator.com/newsguidelines.html
The model itself should not be substituted(for example, quantized) without an explicit SKU change.
Pretty much the only alternative is loading up some credits and using Fable.
Both get the job done these days. I still hate Opus 5.5’s output, 6.1 Sol does an acceptable job of doing a second cleanup pass to it.
1)- Get a coding task and get the output of Opus and Astra.
2)- Open a new session and ask each other to review in detail the solution from the other model without mentioning where is it coming from.
3)- After the review, ask the original model to comment on it and state which solution is better and if the criticisms are valid...all in a new session.
4)- At the end Opus and others from Anthropic always end up agreeing the OpenAI model solution is better. My human review agrees.
After doing this for a few weeks...and consistently having their models rating higher the OpenAI solutions...I cancelled the Anthropic subscription...
DeepSeek, Qwen.
I tried $20 of usage credits as a test once the limit was reached.
I shit you not, it lasted for less than 5 minutes.
I get 5x more done (of equivalent quality) on my opencode go + DeepSeek flash sub for $40, than $200 anthropic sub.
I have been using DeepSeek flash for a few projects and it is actually crazy capable, And I don't think I have spent $20 via api in a month yet.
A bit like "You can order the steak for $75, or you can have whatever the chef is making for $30"
And with Strata running at 50tok/s on my desktop is all I need. Would I like faster? Sure! But it's super useful in all sorts of tasks now.
Flash Next is really great, but you still need pretty decent hardware. A DGX Spark, Strix Halo, or Mac with a lot of memory. You can use things like Strata to squeeze it into a system with only 32GB of VRAM by offloading a lot of it to system memory and still get something that "works", but the quality falls off a cliff too. At that point you get better results out of Qwen 3.8 27b in int4.
Never go below a four bit quant if you want good results. A worse quant of a bigger model is often easily beaten by the small model at decent quality. A lot of the people who think local AI suck think that because they don't understand that low quants == dumbass model. You aren't just compressing data, you're reducing the amount of compute it uses which effectively lowers it's IQ.
If you want to run something actually useful for cheap Qwen 3.8 27b on a 32GB GPU in int4 should be passable. It's more like Sonnet for a year ago. To match last years Opus you really need at least 128Gb.
Personally I have two Sparks connected with ConnectX 7. Now I can run GLM 5.3 locally at 30-40 tps and it's honestly been great. It really cut my dependence on cloud providers and I can run an abliterated version that never refuses my Cyber related tasks.
Folks like to think of 'weights' as just storage, but they aren't (except with stuff like engrams). Those digits of precision encode actual computation, not just storage. There is always a cost, it's lossy compression after all. It's like MP3 vs AAC. If you can't hear the difference it just means you haven't encountered it YET.
That's not to say that you ever will. The cost could be in weird far out edge cases, but whatever it once encoded... It's gone now and it ain't coming back.
No longer am I constantly made to feel like Ingrid Bergman in that movie with all the lights.
How are they making money in your view?
What you said doesn't yield any information whatsoever about whether or not subscriptions are profitable, all that says is that subscriptions are less profitable than API.
Things are able to be less profitable than other things while still being profitable.
If something like the API is very very profitable, it's even possible for profitable things to be much less profitable!
https://x.com/tokengremlin/status/2119783808773841101
i've seen people on social media claiming all sorts of things about Fable 5.5, Opus 5.6 and Sol 6.2 and what-not for the past few weeks, but it just seems like engagement farming with no proof
That would be the less bad reason.
Under load requests queue and take longer.
TPUs only made up a small percentage of the fleet, but, exacerbating that bug, if you got routed to a TPU instance at some point in your chat then another bug kept you routed there for the remainder of the chat.
The second one I'm aware of is earlier in 2026 they decided to save usage by stripping out all thinking tokens if you left your chat idle for one hour (cache timeout) and then came back to it, causing massive model performance degradations. This one also made me very skeptical of Chinese distillation, until the reports of how you could use Haiku to extract the raw reasoning of the larger models.
They fixed that, but recently they switched to doing auto-compactions for long running chats in anticipation of them going stale in cache. I think that one is usually what you want anyway and there is at least a way to recover to the uncompacted chat if you really need it, but it might also be a source of some degradation people are seeing now.
But the model NERF conspiracies are bunk.
But the harness one was not just a random harness regression; they were getting a huge influx of users and pretty clearly did it to save capacity. Not a great thing when you might be relying on attention to those thinking traces to reduce the risk of prompt injection etc. and it could have gone beyond major task degradation into a safety/security issue.
Through the lens of a conspiracy theory this could be viewed narrowly as "cheating" their users to reduce load.
But there are other motivations for the constant work going on in the harnesses that paint a fuller picture.
However, how would less compute result in degraded performance as opposed to more latency? Like is amount of compute something you can actually tune to control for latency? Wouldn't that imply they are actually serving requests in other models? If I run a very large model locally on a shit computer vs. something maxed out, it's the latency that differs.
They basically dynamically allocate compute into this pool from the general pool to meet demand and SLAs. If demand for priority processing spikes latency in the general pool spikes.
Almost all subscriptions use general pool. That's why it gets really slow sometimes. Unless you use "fast" which acts like priority processing and is super expensive.
So that's part of how they balance compute based on demand and fund the "excess" capacity. This is a bit reductive and basic, but it's all the usual Cloud on demand, reserved, and spot capacity techniques at play.
Im asking on how lower compute could translate to worse model performance. That's the implicit claim being made when people are saying Anthropic nerfs a model (result is bad, not necessarily slow). The only way I can think that could happen is if they actually swapped Opus 5.5 out for a worse model that takes less resources to do inference.
There is no proof, the frontier labs deny it, and certainly it's not happening on hosts like Azure, Amazon, and etc.. The claim is that it's "trivial" to detect observation, so they serve the normal model if you try to measure it..
Basically the conspricy theory is:
* OpenAI/Anthropic nerf their models regularly to reduce compute demand
* It's impossible to measure and thus prove
* nerf them based on supply and demand, when you have a high demand, nerf it, but keep charging same price.
* do you want to show revenue growth to investors? increase thinking tokens, customers will keep paying, because again, this is non-deterministic system, sometimes it thinks more
* do you want to increase profits? start serving 4-bit, 5-bit quantizations and keep charging same price
* want even more? publish a paper "Advanced Auto Super Smart Prompt Router - AASSPiRation", and start routing requests to Qwen3.8-27B or GemmaE2B from time to time on the edge
* Not enough? introduce "Complex Chained Workflows - CCW" in your coding harness, which spawns multiple deterministic workflows (which you can't stop or control) and they will use Haiku to write the code, then use Opus 5.5 high to verify the code (which is obviously wrong) and Opus 5.5 produces recommendations on how to fix the code, delegates to Haiku, which again fails writing proper code, and keep doing it 5-6 times, you made lots and lots of money (don't forget, all should run in parallel, to pick one best result among 10 outputs).
How do you even lower the quality - serve half broken car for a luxury order? No way person who orders luxury car accepts it.
Or do you increase wait times? - sure, order gets cancelled, you won't get a payment for that.
There are of course ways, but you still can do something, in case of LLMs, you blame yourself for poor prompting and retry, which again makes money to LLM providers
- You can lower the quality standard of which car counts as which class. There are many classes of vehicle on both platforms.
- You can delete or not consider low ratings from certain users. Both companies do this already.
- You can increase wait times, of course. It doesn't at all follow that the order gets cancelled. It depends on the market and what each company believes the other is doing.
That's thirty seconds of brainstorming.
Opus is suddenly fumbling basic tool calls and failing to finish tasks even at high, where I had been permanently using it on low effort with success since it came out. It is very frustrating.
My guess is they quantized it and/or reallocated DC capacity to train a new model, this has happened before.
Not just frustrating, it can also be dangerous. What if you trust Claude with certain security-sensitive things (because experience), and suddenly you cannot trust it anymore like that?
then $corp should remove your access to these security-sensitive things ;)
It is deterministic, if we're being pedantic
Killing it's own shell, multiple, multiple times in the same session. I'm sure I saw a phrase "I did it again".
It's so shit that it must be damage control. The alternative is to limit usage and that'd affect the IPO even more.
I wouldn't say I've especially noticed its effectiveness having been reduced, just slower.