This approach (scraped data on FS + metadata in DB) works well for storing scraped data. It was the first thing I prototyped when we started the project to move away from MongoDB. We've worked on similar designs in the…
Take the example of crawl logs. Each log entry has a log level, timestamp and message. Typical use would be to view all ERROR (or higher) log levels, show all entries with a specific text in the message, or download the…
The problems with pagination are explained better on this SO post: http://stackoverflow.com/questions/7228169/slow-pagination-o... Mongo docs used the "walk" terminology
I would look at "delta-scraping" as one of the things that is easier with scrapy. There is an existing extension for it: https://github.com/scrapinghub/scrapylib/blob/master/scrapyl... and lots of support and help from…
This approach (scraped data on FS + metadata in DB) works well for storing scraped data. It was the first thing I prototyped when we started the project to move away from MongoDB. We've worked on similar designs in the…
Take the example of crawl logs. Each log entry has a log level, timestamp and message. Typical use would be to view all ERROR (or higher) log levels, show all entries with a specific text in the message, or download the…
The problems with pagination are explained better on this SO post: http://stackoverflow.com/questions/7228169/slow-pagination-o... Mongo docs used the "walk" terminology
I would look at "delta-scraping" as one of the things that is easier with scrapy. There is an existing extension for it: https://github.com/scrapinghub/scrapylib/blob/master/scrapyl... and lots of support and help from…