You'll get some hostility around here for all the slop-text, but the board of personas with anti-cheat public attestation seems like the beginning of a useful forecasting tool.
I tried building something similar to make 72-hour predictions about the US war on Iran, but found that the subagents for each persona were far too naive. They believed official statements and media reports at face value, they failed to read between the lines or apply principles of bounded distrust, they trusted the spin and not the underlying incentives, and they didn't learn from their mistakes from earlier rounds.
It must be possible to fix all these issues with memory, system prompts, progressively changing subagent weights based on historical performance, etc.
Presumably the hosted version will have a leaderboard of some sort. Each board might not be able to cheat, but if users can cheaply create many sockpuppet boards, you'll see the Baltimore Stockbroker Scam emerge on the leaderboard. If the public attestation is to be meaningful, it must be difficult and expensive to create new boards.
You are right and sorry about that, I am new here, already learned my lesson )
I build this as an advisory board for myself and very quickly understood I can use it for predsictions.
Politics and war (sport and weather as well) have too much noise and more unknows that knowns to the public and very hard to predict, on the other hand fed numbers (and stocks) can be compared and verified.
I was already thinking abot post-mortems after each round (self-improve loop), thanks for sharpening it out!
Cheating is impossible, because numbers are written into git before the real-world answer come out, so they can't be faked.
Regarding "Baltimore Stockbroker Scam", you are right and thanks for pointing out!
The current release is local OSS library with your own API keys, no hosted leaderboard exist yet.
In the futere tho, I can prevent it with one real account per person and rate-limitting the board creation. In addition min track record and transparency (can't delete or hide recordd) will do the job.
@panarky thanks for insights! any more grill is wellcomed )
7 comments
[ 3.0 ms ] story [ 23.4 ms ] threadI tried building something similar to make 72-hour predictions about the US war on Iran, but found that the subagents for each persona were far too naive. They believed official statements and media reports at face value, they failed to read between the lines or apply principles of bounded distrust, they trusted the spin and not the underlying incentives, and they didn't learn from their mistakes from earlier rounds.
It must be possible to fix all these issues with memory, system prompts, progressively changing subagent weights based on historical performance, etc.
Presumably the hosted version will have a leaderboard of some sort. Each board might not be able to cheat, but if users can cheaply create many sockpuppet boards, you'll see the Baltimore Stockbroker Scam emerge on the leaderboard. If the public attestation is to be meaningful, it must be difficult and expensive to create new boards.
I build this as an advisory board for myself and very quickly understood I can use it for predsictions.
Politics and war (sport and weather as well) have too much noise and more unknows that knowns to the public and very hard to predict, on the other hand fed numbers (and stocks) can be compared and verified.
I was already thinking abot post-mortems after each round (self-improve loop), thanks for sharpening it out!
Cheating is impossible, because numbers are written into git before the real-world answer come out, so they can't be faked.
Regarding "Baltimore Stockbroker Scam", you are right and thanks for pointing out! The current release is local OSS library with your own API keys, no hosted leaderboard exist yet. In the futere tho, I can prevent it with one real account per person and rate-limitting the board creation. In addition min track record and transparency (can't delete or hide recordd) will do the job.
@panarky thanks for insights! any more grill is wellcomed )