I have written a crawler for buzzfeed and 9gag but I don't have them running because there is a lot of overlap between 9gag and reddit and they require more work.
To be honest this was a weekend project I did awhile ago. If there is viable interest I may pursue expanding it. But for now I'm going to keep it as is.
Ruby on Rails for the backend, Angular on the front. It's hosted on Heroku, it uses cloudfront, S3 for storage, Postgres for DB along with httpClient and mechanize for crawling and Paperclip for image manipulation.
Not particularly. The one thing that helps but most people don't know about reddit is the fact that adding a .json to the end of each url displays the content of that page in json format.
for example:
reddit.com/r/funny.json
This make crawling/fetching content from reddit much more trivial than old school web crawling.
I aggregate from Buzzfeed and 9gag as well, although I don't have them actively running.
I try to give the images a clutter-free, mobile friendly, reliable, fast and ad-free home. (I compress and host all the images myself)
I try to provide an easy mechanism for navigating to previous snapshots of what was aggregated for anyone who is passively browsing and has viewed all the recent images.
Because many people hotlink directly from the site. And If I'm going to host the images myself and eat the cost of bandwidth I might as well derive some value from it.
I provide a link to where the original images was taken from, so if anyone is interested in the image without the watermark they can get it from there.
11 comments
[ 3.6 ms ] story [ 35.7 ms ] threadTo be honest this was a weekend project I did awhile ago. If there is viable interest I may pursue expanding it. But for now I'm going to keep it as is.
for example: reddit.com/r/funny.json
This make crawling/fetching content from reddit much more trivial than old school web crawling.
I try to give the images a clutter-free, mobile friendly, reliable, fast and ad-free home. (I compress and host all the images myself)
I try to provide an easy mechanism for navigating to previous snapshots of what was aggregated for anyone who is passively browsing and has viewed all the recent images.
I provide a link to where the original images was taken from, so if anyone is interested in the image without the watermark they can get it from there.