Ask HN: How do you prompt the "advanced" models
With the apparently more advanced reasoning models I thought that that would change. In Windsurf I have DeepSeek R1 as well as o3-mini available. I had thought that they would improve my outcomes to the prompts that I'm giving. They did not, far from it. Even though in benchmarks they consistently pull ahead of Claude 3.5 Sonnet, in reality, with the way I am prompting, Claude almost always comes up with the better solution. So much so, that I can't remember a time where Claude couldn't figure it out and then switching to another model fixed it for me.
Because of the discrepancy between benchmarks and my own experience I am wondering if my prompting is off. It may be that I am prompting Claude specific having used it for a while now. Is there a trick to know to prompt the reasoning models "properly"?
13 comments
[ 2.4 ms ] story [ 41.0 ms ] threadOne thing I always make sure is to never get it to just spit out code. I always go back and forth a few times to ensure alignment before I say “Bombs Away” and let it write code.
These LLM’s seem to assume their job is to just write code instantly. No! Tell them to not do that! “No code yet, let’s make sure we are on the same page first. This is requirements and discussion, I’ll tell you when it’s time to write code” and go back and forth on requirements and stuff. If fact bake that into your “system prompt” or whatever your tool provides to set up the pre-prompt—I will straight up tell it what I just used as an example about failing an engineer who jumped right into writing code. That LLM had plenty of training data about why that is a bad idea! You just need to “remind it” by telling it.
These tool makers should honestly build that in by default but I bet a lot of people expect the magic genie to start writing magic code by just reading your mind or something. If they had it ask all those pesky requirements first, I bet they’d start failing some idiotic benchmarks or something.
In addition consider asking it if it has all the context it needs to help you. Sometimes it will say “no, can you let me see such and such class/file”.
Also I make sure to include the full path and file name at the top of all my files to help the LLm since I doubt that kind of metadata gets passed through otherwise. I’ll sometimes give it a “tree” of my own entire project so it knows where all the files are at.
Providing proper context is absolutely critical. This tool cannot read your mind! Tell it exactly what its role is and give it the context to do the job properly.
These things aren’t magic. Learn how they work a bit and then you’ll realize they are a really fancy command line interface or something… dunno how to describe it, but it’s just another way to interact with a computer. That’s all it is.
[1]: https://github.com/OpenInterpreter/open-interpreter
I suspect we'll be getting to a point where the "code" is just instructions, codified in a special markup file, and llms just write the worst possible, kiss code you can think of - but is extremely secure because it's just like direct database access with all the security constraints you define, but always applied correctly. In other words think of the actual code as a non-committed artifact, and it's just emitted if the descriptors change.
The long term of llms writing code isn't to give us human quality code, it's to give us what we'd think of assembly but rigorously output to all auth requirements.
We'll start seeing things like Java, Python and Ruby code as we see Assembly now, and the structures you're describing as the program.