If the developer’s Christmas bonus depends on scoring high on a particular benchmark, it is not inconceivable that benchmark somehow would make its way into the training set - directly or indirectly.
I don’t think the management would have to directly encourage it as stated in the Chinese text.
“ We've also heard claims that we trained on test sets -- that's simply not true and we would never do that. Our best understanding is that the variable quality people are seeing is due to needing to stabilize implementations.”
4 comments
[ 0.21 ms ] story [ 21.2 ms ] threadI don’t think the management would have to directly encourage it as stated in the Chinese text.
“ We've also heard claims that we trained on test sets -- that's simply not true and we would never do that. Our best understanding is that the variable quality people are seeing is due to needing to stabilize implementations.”