Love your idea. We have timeout mechanisms and we originally would be pretty aggressive with timeouts based on both time and response length to balance accuracy and speed. There’s research that longer responses tend to…
As of right now, we do not. I'm working on these other benchmarks, but unfortunately they cost quite a bit of money to run, which I'm hoping will come from many people using Sup :)
I mentioned in another comment that I make sure the cost/time is within 1.25x of the next best single-model run. So it's not perfect, but I think that aspect will only get better with time. Of course I'm biased, but…
Love the idea of learning the skill of vibe coding. The subtitle of the book, "Idea → Prompt → AI → Edit → Ship" doesn't quite go with the premise, "There is no magic prompt that turns your idea into a finished product…
Well-written article. Does a great job walking through why any robust system will need what DSPy provides. Though there are many libraries and frameworks that will provide the basics, RAG, exponential back-off, etc.…
[dead]
Is their scaffold available? Does it do anything special beyond feeding the warmup, single challenge, and full problem to an LLM? Because it's interesting that GPT-5.2 Pro, arguably the best model until a few months…
Love your idea. We have timeout mechanisms and we originally would be pretty aggressive with timeouts based on both time and response length to balance accuracy and speed. There’s research that longer responses tend to…
As of right now, we do not. I'm working on these other benchmarks, but unfortunately they cost quite a bit of money to run, which I'm hoping will come from many people using Sup :)
I mentioned in another comment that I make sure the cost/time is within 1.25x of the next best single-model run. So it's not perfect, but I think that aspect will only get better with time. Of course I'm biased, but…
Love the idea of learning the skill of vibe coding. The subtitle of the book, "Idea → Prompt → AI → Edit → Ship" doesn't quite go with the premise, "There is no magic prompt that turns your idea into a finished product…
Well-written article. Does a great job walking through why any robust system will need what DSPy provides. Though there are many libraries and frameworks that will provide the basics, RAG, exponential back-off, etc.…
[dead]
Is their scaffold available? Does it do anything special beyond feeding the warmup, single challenge, and full problem to an LLM? Because it's interesting that GPT-5.2 Pro, arguably the best model until a few months…