Dedup-ing LAION (60M duplicates) and ImageNet (1.2M duplicates) with fastdup (youtube.com) 3 points by dnth 3y ago ↗ HN
[–] dnth 3y ago ↗ The authors at fastdup ran an analysis on LAION 400M and Imagenet21K. Here's what they found.LAION 400M> 60M duplicates. > 962K broken images. > Various label discrepancies.ImageNet21K> 1.2M duplicate images. > 104K train/val leak.fastdup GitHub repo - https://github.com/visual-layer/fastdup
1 comment
[ 1.7 ms ] story [ 13.9 ms ] threadLAION 400M
> 60M duplicates. > 962K broken images. > Various label discrepancies.
ImageNet21K
> 1.2M duplicate images. > 104K train/val leak.
fastdup GitHub repo - https://github.com/visual-layer/fastdup