2 comments

[ 3.0 ms ] story [ 16.2 ms ] thread
Love this kind of experiment. Would the model perform better with word tokens?
A friend of mine forked the repo and tried it with BPE (Byte-Pair Encodings), and it did noticeably improve performance.