I’m an independent consultant building agent systems. I kept running into the same problem where my clients and I tried to use LLMs when regular code was the better tool. The last straw for me was a prospecting agent that ruined 10-20% of the records it inserted into my CRM, and I had to review all 800+ entries by hand. I thought I’d write up what I’ve learned from this experience - I landed on a six-question framework that has helped me greatly when deciding if a solution could benefit from an LLM. I’d be interested to hear what boundary questions others use.

This really resonates with me.

So many tasks given to LLMs can be performed more reliably and efficiently with deterministic code.

In those cases the LLM can help you write the code. But doesn’t need to be invoked every time the task executes.

I think in many (most?) cases the best approach is to build a tool that defines exactly what the LLM model can/can’t do based on the requirements and then have it use that. Mostly about minimising the choices the model can make. Of course things can always go off the rails if you’re not careful or watching/validating!