HN Explorer

Ask HN: Best way to keep the raw HTML of scraped pages?

32 points by vitorbaptistaa ↗ HN
I'm scraping information regarding civil servants' calendars. This is all public, text-only information. I'd like to keep a copy of the raw HTML files I'm scraping for historical purposes, and also in case there's a bug and I need to re-run the scrapers.

This sounds like a great usage for a forward proxy like Squid or Apache Traffic Server. However, I couldn't find in their docs a way to both:

* Keep a permanent history of the cached pages

* Access old versions of the cached pages (think Wayback Machine)

Does anyone know if this is possible? I could potentially mirror the pages using wget or httrack, but a forward cache is a better solution as the caching process is driven by the scraper itself.

Thanks!

17 comments

[ 48.5 ms ] story [ 341 ms ] thread
sbricks
i'd just apply intelligent file naming strategy, based on timestamps and urls. keep in mind, that a folder should not contain more than 1000 files or other folders, otherwise it's slow to list.
nf-x
Did you try using some of the cheap cloud storage, like AWS S3?
compressedgas
WARC.
PaulHoule
Content addressable storage. Generate names with SHA-3, split off bits of the names into directories like

   name[0:2]/name[0:4]/name[0:6]/name
to keep any of the directories from getting too big (even the filesystem can handle huge directories, various tools you use with it might not) Keep a list of where the files came from and other metadata so you can find things in a database.
marginalia_nu
You're going to incur insane overhead doing this, since each file's actual size is a multiple of the filesystem's block size, which is similar order of magnitude as a compressed HTML file.
PaulHoule
Granted. It's fair to stuff them into SQLite or some other kind of composite file but you trade one problem for another. The same content-addressible indexing still makes sense.
toast0
It would be nice if compressed HTML was on a similar order to filesystem block sizes, but most pages are much larger than 4k bytes even compressed. May depend on the specific site that's being scraped.
marginalia_nu
This isn't a hypothetical. I've used this exact workflow for this exact use case. I'm telling you it's no good.

I run a search engine crawler and my average across 100M docs is about 7 Kb when compressed with zstd (fs block size is typically 4 Kb). Some much larger than that of course, but many smaller still. HTML in general compresses absurdly well.

mdaniel
If you weren't already aware, Scrapy has strong support for this via their HTTPCache middleware; you can choose whether to have it actually behave like a cache, choosing to returned already scraped content if matched or merely to act as a pass-through cache: https://docs.scrapy.org/en/2.7/topics/downloader-middleware....

Their OOtB storage does what the sibling comment says about sha1-ing the request and then sharding the output filename by the first two characters: https://github.com/scrapy/scrapy/blob/2.7.1/scrapy/extension...

placidpanda
When doing this in the past, I settled on an sqlite database with one table that stores the compressed html (gzip or lzma) along with other columns (id/date/url/domain/status/etc.)

Also made it easy to alert on when something broke (query the table for count(*) where status=error) and rerun the parser for failures.

crazygringo
Yup. A database gives you all the performance AND flexibility you need. MySQL or PostgreSQL will work well too.

Storing pages as files is a no-go because it wastes way too much disk space due to block sizes. While more customized cache tools will never be as flexible or have as much tooling as a widely supported relational database.

For even better compression use a preset dictionary as well tuned to a wide sample of HTML, but it doesn't sound like you need to go that far.