Interesting read, since the author brought up future work on large corpora, allow me to share our experience. Our work "A partition cover approach to tokenization" https://openreview.net/forum?id=prGyR9id7X, explored…
Interesting read, since the author brought up future work on large corpora, allow me to share our experience. Our work "A partition cover approach to tokenization" https://openreview.net/forum?id=prGyR9id7X, explored…