springkim

↗ HN profile [ 2073 ms ] full profile

A developer specializing in optimization and automation.

Show HN: FlashTokenizer – 10x faster C++ tokenizer for Python (github.com)

I built a tokenizer in C++ with a Python binding that outperforms HuggingFace tokenizers by 10x on large inputs. It's optimized for minimal memory usage and latency. Benchmarks and comparison included in README. Would…

5 points 1y ago 0