4 comments

[ 3.7 ms ] story [ 17.1 ms ] thread
Super fast BERT tokenizer. It will save you 100x time in preprocessing text.
Mainly it will help improve latency at inference time. For preprocessing training one can always add more threads, but not at inference time.