Did you try it with high reasoning effort?
Yeah, +1. Looking back to the WebVoyager [1] and GPT4V generalist agent [2] papers from last January, it feels like we haven't come that far. But there are now several major technical unlocks - fine tuning for cursor…
From my experience there are three key issues with agents today: 1. They usually don't end up completing the right set of steps required to complete tasks when using our human-defined frameworks (react, rewoo,…
I think the defining story of 2025 will be AI agents getting very good with computer use, largely enabled by RL fine tuning.
Many are incorrectly citing 85% as human-level performance. 85% is just the (semi-arbitrary) threshold for the winning the prize. o3 actually beats the human average by a wide margin: 64.2% for humans vs. 82.8%+ for o3.…
It actually beats the human average by a wide margin: - 64.2% for humans vs. 82.8%+ for o3. ... Private Eval: - 85%: threshold for winning the prize [1] Semi-Private Eval: - 87.5%: o3 (unlimited compute) [2] - 75.7%: o3…
I did, and then promptly used it for 2 hours straight. It's excellent. Going to save me so much time.
My $0.02: it's too hard to build and iterate on complex workflows. Every agent uses a meta-workflow (eg. ReAct is plan->act->observe, with some added steps to check for completion etc.). The teams that have been…
Thank you for building this! It looks excellent and geared at exactly the same problems I've been facing. In fact, I've been working on a very similar package and this may have just saved me a ton of time. Excited to…
Yes, they are overblown, with some caveats. In terms of API usage, OpenAI has never used the prompts for training but this is very poorly understood among enterprise CEOs and CIOs. Executives heard about the Samsung…
Open auctions will help. In the last year, we've ended up #2 in 6 bidding wars (as disclosed by the listing agents) in one particular area of the GTA. In each case we reached our absolute max and wouldn't have paid any…
Did you try it with high reasoning effort?
Yeah, +1. Looking back to the WebVoyager [1] and GPT4V generalist agent [2] papers from last January, it feels like we haven't come that far. But there are now several major technical unlocks - fine tuning for cursor…
From my experience there are three key issues with agents today: 1. They usually don't end up completing the right set of steps required to complete tasks when using our human-defined frameworks (react, rewoo,…
I think the defining story of 2025 will be AI agents getting very good with computer use, largely enabled by RL fine tuning.
Many are incorrectly citing 85% as human-level performance. 85% is just the (semi-arbitrary) threshold for the winning the prize. o3 actually beats the human average by a wide margin: 64.2% for humans vs. 82.8%+ for o3.…
It actually beats the human average by a wide margin: - 64.2% for humans vs. 82.8%+ for o3. ... Private Eval: - 85%: threshold for winning the prize [1] Semi-Private Eval: - 87.5%: o3 (unlimited compute) [2] - 75.7%: o3…
I did, and then promptly used it for 2 hours straight. It's excellent. Going to save me so much time.
My $0.02: it's too hard to build and iterate on complex workflows. Every agent uses a meta-workflow (eg. ReAct is plan->act->observe, with some added steps to check for completion etc.). The teams that have been…
Thank you for building this! It looks excellent and geared at exactly the same problems I've been facing. In fact, I've been working on a very similar package and this may have just saved me a ton of time. Excited to…
Yes, they are overblown, with some caveats. In terms of API usage, OpenAI has never used the prompts for training but this is very poorly understood among enterprise CEOs and CIOs. Executives heard about the Samsung…
Open auctions will help. In the last year, we've ended up #2 in 6 bidding wars (as disclosed by the listing agents) in one particular area of the GTA. In each case we reached our absolute max and wouldn't have paid any…