The author dismisses WARC, but I don't see why. To me, Gwtar seems more complicated than a WARC, while being less flexible and while also being yet another new format thrown onto the pile.
I gave up a long time ago and started using the "Save as..." on browsers again. At the end of the day, I am interested in the actual content and not the look/feel of the page.
I find it easier to just mass delete assets I don't want from the "pageTitle_files/" directory (js, images, google-analytics.js, etc).
Hmm, I’m interested in this, especially since it applies no compression delta encoding might be feasible for daily scans of the data but for whatever reason my Brave mobile on iOS displays a blank page for the example page. Hmm, perhaps it’s a mobile rendering issue because Chrome and Safari on iOS can’t do it either https://gwern.net/doc/philosophy/religion/2010-02-brianmoria...
Gwtar seems like a good solution to a problem nobody seemed to want to fix.
However, this website is... something else. It's full of inflated self impprtantance, overly bountiful prose, and feels like someone never learned to put in the time to write a shorter essay. Even the about page contains a description of the about page.
I don't know if anyone else gets "unemployed megalomaniacal lunatic" vibes, but I sure do.
I was on board until I saw that those can't easily be opened from a local file. Seems like local access is one of the main use case for archival formats.
It sounds like it would be pretty easy to write a super simple app with a browser in it that you could associate with the file type to spin these up. IMO.
It's fairly common for archivers (including archive.org) to inject some extra scripts/headers into archived pages or otherwise modify the content slightly (e.g. fixing up relative links). If this happens, will it mess up the offsets used for range requests?
Very cool idea. I think single-file HTML web apps are the most durable form of computer software. A few examples of Single-File Web Apps that I wrote are: https://fuzzygraph.com and https://hypervault.github.io/.
I would like to know why ZIP/HTML polyglot format produced by SingleFile [1] and mentioned in the article "achieve static, single, but not efficiency". What's not efficient compared to the gwtar format?
Hmm, so this is essentially the appimage concept applied to web pages, namely:
- an executable header
- which then fuse mounts an embedded read-only heavily compressed filesystem
- whose contents are delivered when requested (the entire dwarf/squashfs isn't uncompressed at once)
- allowing you to pack as many of the dependencies as you wish to carry in your archive (so, just like an appimage, any dependency which isn't packed can be found "live"
- and doesn't require any additional, custom infrastructure to run/serve
I’ve thought about doing something similar, but at the Service Worker layer so the page stays the same and all HTTP requests are intercepted.
Similar to the window.stop() approach, requests would truncate the main HTML file while the rest of that request would be the assets blob that the service worker would then serve up.
The service worker file could be a dataURI to keep this in one file.
Interesting, but I'm kind of confused why you'd need lazy loads for a local file? Like, how big are these files expected to be? (Or is the lazy loading just to support lazy loading its already doing?)
In case the author is reading: Please consider to add official fields for an optional screenshot of the page in BASE64 encoding and permit to add an (optional) description. Would also help to have official fields to specify the ISO time stamp when the archival took place.
As final wish list, would be great to have multiple versions/crawls of the same URL with deduplication of static assets (images, fonts) but this is likely stretching too much for this format.
Allowing more metadata might be useful. You can add anything to the manifest at build time as assets are not required to be loaded or ever used (because this is impossible to statically check). I suppose we'd have to define an official prefix like 'gwtar-metadata-*' with like a 'gwtar-metadata-screenshot' and 'gwtar-metadata-desciption'... Not obvious what the best way forward is there, you don't want to add a whole bunch of ad hoc metadata fields, everyone will have a different one they want. Exif...?
Multiple versions or multiple pages (maybe they can be the same thing?) would be nice but also unclear how to make that. An iframe wrapper?
I considered and rejected deduplication and compression. Those can be done by the filesystem/server transparent to the format. (If there's an image file duplicated across multiple pages, then it should be trivial for any filesystem or server to detect or compress those away.)
> The main header JS starts using range requests to first load the real HTML, and then it watches requests for resources; the resources have been rewritten to be deliberately broken 404 errors (requesting from localhost, to avoid polluting any server logs)
what if a web server on localhost happens to handle the request? why not request from a guaranteed unaccessable place like http://0.0.0.0/ or http://localhost:0/ (port zero)
either it optimizes and dont try to connect or does not recognize as never accessible and does try to connect – both are better than accitentally fetching something from a web service running on localhost.
30 comments
[ 4.6 ms ] story [ 52.2 ms ] threadApparently every important browser has supported it for well over a decade: https://caniuse.com/mdn-api_window_stop
Here's a screenshot illustrating how window.stop() is used - https://gist.github.com/simonw/7bf5912f3520a1a9ad294cd747b85... - everything after <!-- GWTAR END is tar compressed data.
Posted some more notes on my blog: https://simonwillison.net/2026/Feb/15/gwtar/
I find it easier to just mass delete assets I don't want from the "pageTitle_files/" directory (js, images, google-analytics.js, etc).
I don't know if anyone else gets "unemployed megalomaniacal lunatic" vibes, but I sure do.
Works locally, but it does need to decompress everything first thing.
Would W3C Web Bundles and HTTP SXG Signed Exchanges solve for this use case?
WICG/webpackage: https://github.com/WICG/webpackage#packaging-tools
"Use Cases and Requirements for Web Packages" https://datatracker.ietf.org/doc/html/draft-yasskin-wpack-us...
https://gwern.net/doc/philosophy/religion/2010-02-brianmoria...
I will try on Chrome tomorrow.
[1] https://github.com/gildas-lormeau/Polyglot-HTML-ZIP-PNG
- an executable header
- which then fuse mounts an embedded read-only heavily compressed filesystem
- whose contents are delivered when requested (the entire dwarf/squashfs isn't uncompressed at once)
- allowing you to pack as many of the dependencies as you wish to carry in your archive (so, just like an appimage, any dependency which isn't packed can be found "live"
- and doesn't require any additional, custom infrastructure to run/serve
Neat!
Similar to the window.stop() approach, requests would truncate the main HTML file while the rest of that request would be the assets blob that the service worker would then serve up.
The service worker file could be a dataURI to keep this in one file.
As final wish list, would be great to have multiple versions/crawls of the same URL with deduplication of static assets (images, fonts) but this is likely stretching too much for this format.
Multiple versions or multiple pages (maybe they can be the same thing?) would be nice but also unclear how to make that. An iframe wrapper?
I considered and rejected deduplication and compression. Those can be done by the filesystem/server transparent to the format. (If there's an image file duplicated across multiple pages, then it should be trivial for any filesystem or server to detect or compress those away.)
great job
what if a web server on localhost happens to handle the request? why not request from a guaranteed unaccessable place like http://0.0.0.0/ or http://localhost:0/ (port zero)