Nice concept. Similar techniques can be found in neuroevolution, where genetic algorithms optimize neural networks. One major benefit is the ability to compare the performance of both subpar and exceptional models on a relatively fair basis. Considering this evidence, I anticipate that this approach will gain greater significance as the quality of language models surpasses our existing benchmarks. However, it's worth noting that a potential pitfall arises when certain models outshine the rest of the group.
It's blind test, and the code to compute scores is open. So unless they are deliberately modifying the data it should be a fair ranking. Just I think openassistant llama(30B) version is better than vicuna 13b, but it is not included in the test.
10 comments
[ 3.0 ms ] story [ 37.6 ms ] threadWould mosaic mpt be useful to benchmark as well?