HN Explorer

How much does CoT Reasoning typically cost for frameworks like LlamaIndex?

2 points by manili ↗ HN
I'm curious to know, based on your experience in production, how much does Chain-of-Though Reasoning typically cost in terms of tokens for frameworks like LlamaIndex, LangChain, CrewAI, etc.?

I understand it depends on many different factors including the complexity of the product and the architecture of the agents involved, but I'd love to hear about your experiences in creating real-world-class application.

4 comments

[ 3.0 ms ] story [ 19.7 ms ] thread
tikkun
tough to answer. depends on the llm, on the api provider or hosting method, on the prompt

but you could get a simple estimate by running a CoT prompt, copying the output tokens, and then using an online token counter to count the length & tokens of the CoT part of the completion

manili
The issue is that I can give it a try on a set of toy problems, but finding a real-world-class problem is totally a different story IMHO. That is exactly the reason I asked about people's "experiences".

Anyway thanks for the response.

kingkongjaffa
> real-world-class problem

The issue is that this notion is nonsense. There are just problems nothing about them makes them “world class” whatever that means.

manili
Fair enough, it's "nonsense". Curious to know about YOUR experience. Sounds "nonsense" yet?