Interesting direction but the 98.8% FPR in Table 1 seems like a dealbreaker. Anyone understand what's going on with the contradictory results between the text and tables?
Based on Table 1: This method is actually worse than generating a random number (0-100% independent of the program) and testing if it is less than 98.8%. That would achieve a better detection rate without increasing the false positive rate.
It doesn't seem worth it to try to follow the math to see if there is something interesting.
4 comments
[ 4.3 ms ] story [ 17.8 ms ] threadFrom Sec 3, end of second to last paragraph
by program hash, and *bounds false positives via the chosen percentile and gap parameters.*I believe this is a choice, though I think it is suspect that the FPR is pushed this high to get the TP results.
Disclaimer: I only gave this a very cursory skim so don't rely on me too much
It doesn't seem worth it to try to follow the math to see if there is something interesting.
https://www.youtube.com/watch?v=Xx4Tpsk_fnM
"The Hard Problem of Controlling Powerful AI Systems" (Computerphile)
https://www.youtube.com/watch?v=JAcwtV_bFp4
Attempting to guide statistical salience of LLM reasoning model procedures, usually just created an evasive interface facade in the output. =3