LLMs reward expertise (seangoedecke.com)

LLMs reward architecture knowledge of how to structure things and how to not just say "Claude, make me Microsoft Flight Simulator, make no mistakes".

Agree with this. LLMs multiply the human user's ability. More ability, more impact!

Not sure I agree with this. The math guy at anthropic's prompts are essentially:

  "suppose you’ve gotta resolve the $CONJECTURE, like absolutely have to, everything depends on it. think really hard, and try to come up with a bunch of ideas to try. but remember to trust yourself and not necessarily in conventional wisdom!!"

  https://claude.ai/share/25740bd5-aa97-4bd7-bf58-c4df3793fda7
  https://xcancel.com/__alpoge__/status/2083855298239078748

Tao's chat was for him to gain intuition, not to solve the problem from the outset.

What's funny is that every other person gets a different conclusion about who these models reward/empower. I've seen people say that the generalist stands to gain the most and others say that it's the experts. Like all of life, maybe the "winner" is the person who just does stuff.

Prompt an image or video generator without knowledge in photography or art skills and your results will look sloppy.

I am feeling this a lot lately. Getting the most out of agents seems to require being able to ask the right question. And how can you ask the right questions without deep domain expertise?

You're basically playing the role of team lead to the LLM's junior dev.

This matches my experience. Just Talk To It is the best method for working with LLMs if you're an expert.

I've seen this at work (as eng manager/lead/principal/whoevenknowsanymore) – all the big APIs give you stats. We see how much people burn in tokens and we know how much output they produce. There is a pretty strong inverse correlation between token burn and output.

The more tokens people burn, the less likely they are to produce a good outcome.

I do find that "signalling expertise" is important. "I have a significant background in biblical scholarship. You can assume I've read the most important works in NT studies in particular. Do not translate Greek, Latin, Hebrew, or Syriac. Now, I would like to know . . ." That changes things significantly. So does telling it you have 20+ years of experience with C programming, that you have a robust understanding of machine organization, memory layouts, embedded systems, etc.

I have lengthy conversations with my LLM, almost like an interview. I agree on the expertise part, because I wouldn't be able to go in depth on a subject with it if I lacked the expertise. Some work is a result of design and negotiations in those designs.

I don't think Tao's style works with everyone/thing, especially if we don't know what style he's tuned his LLM on.

This is true but also false.

In my experience (scientific programming) AI is a giant multiplier for people with specialized knowledge.

But it’s also a giant devaluer for that same knowledge as people with no idea what they’re doing can clog the field with plausible bullshit.

It’s now the case that if someone tells me they’ve done something, and I look into it and find out it’s completely AI slop, then I will have spent more time on the project than the person who “made” it. The situation is completely untenable and only serves to drain time and resources from people with better things to do.

The counterexample of the Dinitz-Garg-Goemans conjecture was basically just "keep going" and finally "enough of partial results. now finish with a complete unconditional counterexample" lol

https://x.com/DmitryRybin1/status/2079904005652893709

https://chatgpt.com/share/6a60b2eb-0b64-83ee-9c76-7931ca1de0...

I often do my best to represent a genuine interest in the subject at hand and learning in general to models. Imagine the model's response prose and mannerisms being on the other polar end of answering questions simply to get the correct answers as they're often scoped for on quantitative benchmarks. Not sure I explained this well, sorry. An LLM could help

Yes. I agree with most, if not all of this. For instance, I am seeing folks either relying in the LLM as an _assumed_ expert or, assuming someone - who knows the structure of skill definitions - also has some expertise (in the area of the skill). It's a difficult situation; there is not much point in explaining _why_ the LLM output or skill in use (on a domain problem) isn't what the person actually _needs_ to address the domain problem, because the person isn't a domain expert or indeed, adjacent to domain expertise. But, it is an interesting experiemnt to arm folk with little domain expertise with the _skill_ necessary to be able to extract the right solution from the model.

This is also why people's experience with LLMs/AI varies so much, because some people can see a use for AI for their needs and go about using the tool, while others do not as it relates to whatever they're working on and so they may say "LLMs/AI are useless" (it doesn't mean they're not experts though, although some people who have totally no expertise might also see no use for AI for themselves).

Overall, I agree, when I ask things I'm an expert in and do professionally every day. I get very good useful answers. When, for example, our marketing people, ask about the science, they often get confusing and wrong answers.

Skilled use may or may not matter, depending on the task. Do you need to do what Terence Tao is doing?

[dead]

> In the 2010s, if you had technical gaps (say, you couldn’t write CSS), you had to either rely on a skilled colleague or just hope that the answer to your exact problem was out there on the internet.

You could read some general reference/guide/tutorial documentation on CSS, and then probably solve your problem (without searching for "how to center a div", or whatever your exact problem was, and copy&pasting the answer and moving on), also becoming more knowledgeable in the process.

The rest of the short blog post has some good points, but the first sentence sounds like it's targeted at the percentage of developers who did StackOverflow copy&paste to close Jira tickets, never becoming experts.

Delegating to LLM-ish AI is just a natural evolution of that. The question is whether they can still add value if kept in the loop.

The article author suggests that the answer is to be expert, and is addressing people who... "either rely on a skilled colleague or just hope that the answer to your exact problem was out there on the internet."

I cant have an overall opinion but in my personal experience i have analysed that LLMs do reward concreteness

Yes. If you use the right technical terms together it’s lights up more specific feature spaces to your task.

Specificity matters to LLMs a lot.

There's a growing and fascinating divide between people who see LLMs as more of a "bicycle for the mind" in the vein of Jobs vs those who see them as whollly supplanting the role of human intelligence. I can't help but wonder if these aren't primarily two human archetypes more than anything - the LLMs can be both and they erect a mirror of the human using them. Some humans really don't want deep individual expertise and intelligence to be the deciding factor because they don't identify with that. Others are completely the opposite. We really can't tell which will be more effective yet, because LLMs are very good in both modes. But most of the predictions currently are people executing on wishful thinking about what they hope will be the outcome.

The short version I give to non-technical people who ask me about whether "AI will replace coding" is this: it accelerates you. You can get much further much more quickly.

If you don't know where you're going or how to get there, or even if you're just not paying enough attention, it will get you very far in the wrong direction before you've realised.

Yes. This is called the Matthew Principle:

> For to every one who has will more be given, and he will have abundance; but from him who has not, even what he has will be taken away.

This feels like a moment in time, not the end state of AI.

Like I read there was a time when teams of people + AI could beat pure AI at chess. But that these days, pure AI wins.

For all the things people say about "how AI works" you have to add the missing piece "how current AI works".