HN Explorer

Ask HN: Do image hosting sites remove redundant images?

7 points by mgallivan ↗ HN
I've started reading: http://www.aosabook.org/en/distsys.html

And his first example is that of an image hosting website. My mind went on a bit of a tangent and I was curious...

Do image hosting sites keep a hash of their images and only store 1 copy of each image? I'm not sure the effort it would take to ensure there was a single copy of each image but... it seems like it would save some space.

Does anyone know?

5 comments

[ 3.7 ms ] story [ 24.3 ms ] thread
UnoriginalGuy
Maybe they do and don't even know it. Some database engines will compare blobs, and then delete duplicates.
true_religion
No serious image host is going to stick images in the database.
macowar
Well, technically a file system is a specialized typed of database. NTFS has some support for removing duplicate files and replacing them with links. Windows server 2008 calls this feature "Single Instance Storage".
relix
I don't think so. The average filesize of a JPEG image on the web is what, 50KB? At those sizes, it's just not worth it to put a system in place that could introduce more bugs through extra complexity. Especially since duplicate images would be relatively rare, as in 0.00001% rare. Completely not worth it. You'd still need extra database entries too, so you're not even saving on the overhead, database size or queries.

Look at it this way: during the time spent coding this feature, I'm pretty sure disk drive space would grow more than the extra space you'd need for the duplicates (relatively speaking over the long-term of course).