1 comment

[ 4.2 ms ] story [ 16.4 ms ] thread
Simhash is an extremely fast and simple algorithm for detecting near duplicate text at scale which makes it particularly useful for deduplicating AI training datasets.