Ask HN: The best way store thousands of Images on a server
Hello. I've got a server of my own handling the multimedia files for a website. It has around 200Gb of images (all JPG's) stored in nested folders.
I have Apache 2 on Linux and nothing else. I don't know if it is the best solution, but I think is not.
What system (http server, cache, or anything) would you recommend me?
12 comments
[ 0.30 ms ] story [ 27.4 ms ] threadThanks, but I was looking for technology to install on my own server. Not moving to Amazon S3
My own 200mbps connection costs me the equivalent of 300$ a month.
http://www.danga.com/mogilefs/
with an nginx or lighttpd frontend.
The documentation is a little sparse though the mailing list and irc are useful for help.
Also it would somewhat depend on your traffic profile. If you've got thousands of random reads (e.g. thumbnails with dozens per page, ssds might be better; specifically the Intel X-25).
Otherwise MogileFS with an array of fast drives would probably do the job. Mogile can also do load balancing based on drive I/O because each image is replicated across different files/machines.
EDIT: Make sure you've got browser caching set up correctly (though admittedly that's a tiny bit harder with Mogile)
If you can use a expiry time that's far in the future. Also don't send a Last-Modified nor an ETag Header. Only send Expires.
Facebook - Needle in a Haystack: Efficient Storage of Billions of Photos
http://www.flowgram.com/p/2qi3k8eicrfgkv/
You might also want to try thttpd: http://www.acme.com/software/thttpd/ I heard all the porn sites use it to host their images.
With only the http server for the images running it should work quite well to rely on the filesystem cache.
Super duper secure and very, very quick, especially for smaller files.
I'd try and run some tests against it, thttpd and your existing apache setup.