It is not uncommon for enterprises to intercept HTTPS for inspection and logging. They may or may not also do caching of responses at the point where HTTPS is intercepted.
I previously experimented a bit with Squid Cache on my home network for web archival purposes, and set it up to intercept HTTPS. I then added the TLS certificate to the trust store on my client, and was able to intercept and cache HTTPS responses.
In the end, Squid Cache was a little bit inflexible in terms of making sure that the browsed data would be stored forever as was my goal.
This Christmas I have been playing with using mitmproxy instead. I previously used mitmproxy for some debugging, and found out now that I might be able to use it for archival by adding a custom extension written in Python.
It’s working well so far. I browse HTTPS pages in Firefox and I persist URLs and timestamps in SQLite and write out request and response headers plus response body to disk.
My main focus at the moment is archiving some video courses that I paid for in the past, so that even the site I bought the courses from ceased operation I will still have those video courses. After I finish archiving the video courses, I will proceed to archiving other digital things I’ve bought like VST plugins, sample packs, 3d assets etc.
And after that I will give another shot at archiving all the random pages on the open web that I’ve bookmarked etc.
For me, archiving things by using an intercepting proxy is the best way. I have various manually organised copies of files from all over the place, both paid stuff and openly accessible things. But having a sort of Internet Archive of my own with all of the associated pages where I bought things and all the JS and CSS and images surrounding things is the dream. And at the moment it seems to be working pretty well with this mitmproxy + custom Python extension setup.
I am also aware of various existing web scrapers and internet archival systems for self hosting and have tried a few of them. But for me the system I am doing is the ideal.
CDNs manage user TLS certificates and that is one of the advantages of using them.
A node server could negociate https close to the user, do caching stuff and create an other https connection to your local server (or reuse an existing one).
As is traditional with most explanations of HTTP caching, it doesn't mention Vary header. Although apparently some CDNs (e.g. Cloudflare) straight up ignore it for some reason [0].
As many have pointed out here, the nature of caching has changed in the current climate of ubiquitous HTTPS, and I want to add a paragraph or two about it. Is there a good summary somewhere that I could reference? What are the the usual, most prevalent uses of HTTP intermediaries involving caches, besides CDNs and origin-controlled caches (eg Varnish)?
Wanted to highlight MDN's HTTP caching guide[0] that OP links in the conclusion. It's written at a higher level than the underlying reference material and has been a great resource I've turned to several times in the last few years.
I think setting FileETag None solved it. With that setup, the browser won't use stale JS/CSS/whatever bundles, instead always validating them against the server, but when the browser already has the correct asset downloaded earlier, it will get a 304 and avoid downloading a lot of stuff. Pretty simple and works well for low traffic setups.
It was surprisingly easy to mess up, or having your translation bundles have cached out of date versions in the browser.
(nothing against other web servers, Apache2 was just a good fit for other reasons)
12 comments
[ 4.5 ms ] story [ 22.2 ms ] threadI previously experimented a bit with Squid Cache on my home network for web archival purposes, and set it up to intercept HTTPS. I then added the TLS certificate to the trust store on my client, and was able to intercept and cache HTTPS responses.
In the end, Squid Cache was a little bit inflexible in terms of making sure that the browsed data would be stored forever as was my goal.
This Christmas I have been playing with using mitmproxy instead. I previously used mitmproxy for some debugging, and found out now that I might be able to use it for archival by adding a custom extension written in Python.
It’s working well so far. I browse HTTPS pages in Firefox and I persist URLs and timestamps in SQLite and write out request and response headers plus response body to disk.
My main focus at the moment is archiving some video courses that I paid for in the past, so that even the site I bought the courses from ceased operation I will still have those video courses. After I finish archiving the video courses, I will proceed to archiving other digital things I’ve bought like VST plugins, sample packs, 3d assets etc.
And after that I will give another shot at archiving all the random pages on the open web that I’ve bookmarked etc.
For me, archiving things by using an intercepting proxy is the best way. I have various manually organised copies of files from all over the place, both paid stuff and openly accessible things. But having a sort of Internet Archive of my own with all of the associated pages where I bought things and all the JS and CSS and images surrounding things is the dream. And at the moment it seems to be working pretty well with this mitmproxy + custom Python extension setup.
I am also aware of various existing web scrapers and internet archival systems for self hosting and have tried a few of them. But for me the system I am doing is the ideal.
This website is chock full of site operators raging mad at web crawlers created by people that didn't bother to implement proper caching mechanisms.
A node server could negociate https close to the user, do caching stuff and create an other https connection to your local server (or reuse an existing one).
Https everywhere with your CDN in middle.
[0] https://news.ycombinator.com/item?id=38346382
Wanted to highlight MDN's HTTP caching guide[0] that OP links in the conclusion. It's written at a higher level than the underlying reference material and has been a great resource I've turned to several times in the last few years.
[0]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Cac...
I think setting FileETag None solved it. With that setup, the browser won't use stale JS/CSS/whatever bundles, instead always validating them against the server, but when the browser already has the correct asset downloaded earlier, it will get a 304 and avoid downloading a lot of stuff. Pretty simple and works well for low traffic setups.
It was surprisingly easy to mess up, or having your translation bundles have cached out of date versions in the browser.
(nothing against other web servers, Apache2 was just a good fit for other reasons)