> a unique message ID in the system prompt broke the entire KV cache. Since the cache requires byte-exact matches, that changing ID forced a full re-compute on every turn, turning warm contexts into cold fills.
This is (part of) the same problem that initially lead Anthropic to ban non-Claude Code clients from using the subsidized subscription: A full to-the-second datetime stamp in the system prompts of OpenCode, and I believe Pi as well, invalidated the caches, making this a very expensive use of their compute very quickly.
They even had Anthropic employees submit PRs (or maybe just open issues, I’d have to check) to these other clients/harnesses because the cache misses were hitting them so hard.
I second this, I read too much AI slop already so when something triggers that part of my brain, at this stage I immediatley lose the capacity to engage outside of work, largly because it feels like work. Scrolling through, this article looks like it holds useful info. Info i'd likely love to engage with, but realistically I cannot force myself to spend my weekend reading more ai outout, even if human seeded.
Fair enough, nothing more infuriating than the sycophantic way LLMs write.
But I’d definitely be dishonest if I said I’d stop using LLMs to tidy my writing (out of principle or otherwise), my hold on the English language has seriously degenerated over the last eight years since I pivoted into IT from customer facing roles.
Nonstop. There is too much wholesale reliance on LLMs to generate content. When I use LLMs for scientific writing, I approach it differently: I write the paragraph dirty, then ask an LLM to perform a minor rewrite for "clarity", using Claude's now retired Concise Mode. This has been a great approach for scientific writing. It tends to prevent these overly used turns of phrase, it makes sure that the writing is making the points I want to make, and shortens writing time by cleaning up the dirty edges of my grammar (especially since my writing can tend towards convoluted constructions). More artistic/creative writing, I'd probably not use it at all, because then, it's usually (for me) about rthym and emotional flow.
BTW It IS an effective rhetorical phrase, but given it's ubiquity in Claude's output, I have to avoid it.
Ahhh yeah, I’m not going to sit here and say I don’t use Claude to clean up my writing (ie make it actually coherently laid out). In all honestly I tend to write in a rambling stream of consciousness style across random scrap markdown files (ha), but point is taken.
I was thinking to myself as I was reading the post that here we go again - it's like how much can a man take. It's not that it's unclear... because it is easy to read. I also long for regular old human writing, warts and all.
I don't mind reading stream of consciousness, thats how I think anyway. Not everything you write has to be tight and polished, it just has to have value to the reader one way or the other.
Try using claude to suggest structural improvements, but take the time to implement those yourself. Don't use claude to make the changes directly. You'll find that what you want to say changes when you have to write it out.
I'd rather have a ai-written article then no article at all. I think this is a real struggle. Writing an article takes time that could be spent on improving the product. Would I prefer a human article, yes. But if the alternative is "no article", then that's how it is.
I pointed my Claude/Retort evaluator at this blog post and it ran an experiment to test what it found. We got better cache hit refill but it didn’t improve the overall coding results on my M5Pro/64GB system. Results in experiment-24 at GitHub.com/adrianco/retort
Is this issue specific to 122B or will it lurk in other 3.5 r 3.6 models? I run a 9B model locally so while this fix doesn't help me I'm wondering if it's general?
35 comments
[ 0.23 ms ] story [ 59.2 ms ] threadThis is (part of) the same problem that initially lead Anthropic to ban non-Claude Code clients from using the subsidized subscription: A full to-the-second datetime stamp in the system prompts of OpenCode, and I believe Pi as well, invalidated the caches, making this a very expensive use of their compute very quickly.
They even had Anthropic employees submit PRs (or maybe just open issues, I’d have to check) to these other clients/harnesses because the cache misses were hitting them so hard.
How is that counter-intuitive? You changed the key, so it didn't return a value. Thats a KV cache working as intended.
That said, I have to read LLM output all day all the time, and I would implore you to take the time to explore your own voice a bit more.
> Two separate things then happened, and it is worth keeping them apart.
Is one of those phrases claude spits out nonstop.
But I’d definitely be dishonest if I said I’d stop using LLMs to tidy my writing (out of principle or otherwise), my hold on the English language has seriously degenerated over the last eight years since I pivoted into IT from customer facing roles.
BTW It IS an effective rhetorical phrase, but given it's ubiquity in Claude's output, I have to avoid it.
At this point that'd be a welcome respite from every single blog post being written in the same exact AI tone.
Try using claude to suggest structural improvements, but take the time to implement those yourself. Don't use claude to make the changes directly. You'll find that what you want to say changes when you have to write it out.
Thanks for your work, thanks for sharing, thanks for the fork.