248 comments

[ 2.9 ms ] story [ 257 ms ] thread
Love this. Use it all the time. Handy for saving huge pages with all the styling intact for reading offline (like on a plane). You could save a webpage as a PDF, but I prefer this over a PDF.
I also want to give praise about the demo. It's one of the best demos I've ever seen with such a project. Nice job!
A 16MB gif with no playback controls, so you had to go through the tedium.
I would be surprised that the author wasn't using WebM to get a smaller filesize (not to mention higher quality) but the project itself leads me to believe that the author has a lot of free disk space to use
There’s no need to make further assumptions about the author (who btw took the time to build a very useful tool and share to in the Internet for free). Just point out the issue of the GIF and move along.
I never made an assumption about the author and certainly never said that the tool wasn't useful. You can feel free to move along yourself, though.
I think you missed something here; parent wasn't insulting the author. They were inferring that someone who made an archiving tool probably has resources that help with archiving (bandwidth means access to documents and storage space means a place to save them-- you get the idea).
FYI - there’s an official standard (MHTML) for doing this that has existed for 20+ years and exists natively in browsers.

https://en.m.wikipedia.org/wiki/MHTML

And it generally does not do a good job
What are the issues?
From my experience, wrong layout,missing pictures.
The big one in my experience is it doesn't play well at all with JavaScript. Single file to my knowledge (I experimented with it briefly) allows all js to load on page and can then embed loaded media as base64. I think it also has heuristics to embed relevant js as well. It still only gets you 90% of the way there, and I came to the conclusion that unless you are doing web archive type work or need audio / video a composite image works well
> MHTML, (...) is a web page archive format used to combine, in a single computer file, the HTML code and its companion resources (such as images, Flash animations, Java applets, (...)

Well that goes to show its longevity I guess.

The browser compatibility section suggests MHTML is unsupported in current versions of Firefox and Safari.
Safari supports webarchive, which does basically the same thing
The problem is that it is a proprietary format. The advantage of the format produced by SingleFile (HTML) is that as long as your browser is capable of interpreting HTML, you will be able to read your archives without worries.
I don't think it was ever native in Firefox, there is/was the excellent unMHT extension that was broken by Quantum/WebExtensions and The Great XUL Silliness. Shame.

I have Waterfox-Classic and unMHT (fished out of the Classic Addons Archive, just remember to turn off Waterfox's multiprocess feature) since I occasionally need to archive web pages - and more importantly, reopen them later.

mhtml is just MIME, literally every discrete URL as a MIME part with its origin in a Content-Location header, all wrapped in a multipart container. I don't understand why it's not a default format.

I can see WebExtensions breaking it (as it's a completely new set of APIs for extensions, and the losses do definitely still hurt)... but quantum/xul? How is that related, aside from "it happened around the same time"?
IANA firefox dev: XUL/XPCOM = old APIs, WebExtensions = new (multi-browser) API

Quantum was the the project name to re-engineer Firefox internals, with lots of design changes, not just plugins. XUL/XPCOM APIs were dropped, as an occasional programmer I understand why, "Quantum broke my plugins" is a reasonable first approximation for most users.

Over a decade ago I had a laptop but no internet at home. This was one of the ways I taught myself programming (and also downloading dozens of manga) by using internet explorer at a cafe which had an option to save to mhtml which was one file and had everything self contained. Legit owe a portion of my success to this. I still have some of these files, old crusty hello world c++ tutorials etc.
For people who cannot afford internet access now, and for perhaps more in the future if times get more difficult, I believe this is a very important use-case.
I have fantastic internet, and I still do something similar. Local docs just load so much faster, and if something happens (which it still does, even on Fiber in the US), I have docs and can program.

Lemme see if I can pull up the command I use to mirror doc sites.

    wget \
      --recursive \
      --level=5 \
      --convert-links \
      --page-requisites \
      --wait=1 \
      --random-wait \
      --timestamping \
      --no-parent \
      $1
Unfortunately mhtml is not widely supported.
IIRC, back in the day mhtml won’t save java applets.
Are any sites still using applets these days?
80% of server IPMI Web control panels. But who whould want to save those anyway? :)
A lot of those are getting HTML5/Canvas based implementations and most of the old AST BMCs can get it through upgraded firmware.
None of my machines had any such upgrades and never will :(
I’ve extensively looked into this as I can’t find a good light and easy backup options that isn’t extreme overkill.

I thought MHTML was NOT standardized which is why it wasn’t across all browsers yet. From what I remember, every company was doing their own implementation of it. Maybe it’s gotten more standardized the last few years though.

I've always thought the "M" stood for "Microsoft" -- wasn't even aware any browsers other than IE supported it.
There is also CHM which is actually a Microsoft only file format for "Compiled HTML Help" files.
I love this format. Very fast and compact. Entire Visual Studio help was in it once. Worked VERY well. And there's a KDE/Qt reader.
> FYI

The alternative format (used by the Internet Archive and Wayback Machine) is WARC. It's also a single file, but it's preserving the HTTP headers as well; so its applications is specifically for archival purposes. [1] The "wget" tool which is co-maintained by the Web Archive people also has support for it via CLI flags.

Though when it comes to mobile browser support I'd recommend to use MHTML, because webkit and chromium both have support for it upstream.

[1] http://iipc.github.io/warc-specifications/

[2] https://www.gnu.org/software/wget/wget.html

Is there any objection to adding WARC support to webkit/chromium? Seems like a not-so-complex project...
I know that WebKit relies on either libsoup [1] (on Linux/Unices) or curl [2] (legacy Windows and maybe WPE(?)) as a network adapter, so the header handling and parsing mechanisms would have to be implemented in there.

Though, on MacOS, WebKit tries to migrate most APIs to the Core Foundation Framework, which makes it kind of impossible to implement as a non-Apple-employee because it's basically a dump-it-and-never-care Open Source approach. [3]

Don't know about chromium (my knowledge is ~2012ish about their architecture, and pre-Blink).

[1] https://github.com/WebKit/WebKit/tree/main/Source/WebKit/Net...

[2] https://github.com/WebKit/WebKit/tree/main/Source/WebKit/Net...

[3] https://github.com/opensource-apple/CF

GTK/WPE use libsoup. Playstation/Windows uses curl. And yes Apples networking is proprietary.
I wasn't sure about WPE in regards to libsoup due to the glib dependencies and all the InjectedBundle hacks that I thought they wanted to avoid.

I mean, in principal curl would run on the other platforms, too...but as far as I can tell there's an initiative to move as much as possible to the CF framework (strings, memory allocation, https and tls, sockets etc) and away from the cross-platform implementations.

(comment deleted)
I remember saving webpages in MHTML when I was using dial-up so that I could read them offline later.

I would also download entire websites using a software which name I forgot, to read them offline. Back when websites held in a single floppy disk.

Good times!

I remember using HTTrack for this a while back. Still have a few of those sites lying around, I think.
Does anyone else get two security warnings whenever you try to save an MHTML page using a Chrome extension? I have to click on one warning's button to confirm that I indeed want to save the "dangerous" file and another to confirm I'm really sure. It's gotten very annoying. I've looked all over for an option to disable this behavior but haven't been able to.
I was gonna say Opera (the old, good one) had this. When saving a page there were some options and one was a single file IIRC.
This would be very useful in many situations, and a great demo!
I love, love this extension. I am working on an app to turn this into a single click bookmark system on Linux. Run an inotify service to watch your downloads and then process any Single file downloads to a database and update a browsable index.
I'm using Recoll for this exact purpose. Just without inotify.
I archived (privately) some documentation pages from some of our vendors that were behind a login page using that just in case it became inaccessible at a critical time for us.
TELL ME MORE.

I think I basically get the idea, what kind of database are you using? Recoll sounds like a good idea, but I'm also thinking about how I might also make this public-ish.

(i.e. I teach in college and would love to have a centralized way to store and search all my assigned readings, which are most often webpages)

I am not a trained software engineer but...

Each html page is processed by (1) getting url, title, time saved (this is under-rated as approximate time of saving is useful if you want to rediscover) and then (2) taking a screenshot and finally (3) extracting text with readability.js and hopefully doing some keyword analysis.

Right now it is stored in a local SQLite Database, although the article content is stored in text files. For search, I can use ripgrep to look through the associated text files.

The eventual goal is to create a flask app which will allow for interactive management of the bookmarks (tagging, searching). I've already got static generation of bookmarks.

Here's a screenshot: https://imgur.com/5YP4sP5

Awesome! Let's bring it up a notch - save every page automatically.

Then add a search engine over that, for 'what was that article about long term effects of DDT on ecosystems I was reading a long time ago?' queries.

And you get a memex - a way to outsource part of the brain to a computer :).

This is great for a page. I'd love to see it expanded to include an entire site.
I have been using WebScrapBook (an add-on for Firefox) for some time. I really like it. Has anyone else some experience with this add-on? Good or bad.
I've been using it for a couple years (2 maybe) and I like it quite a bit as a quick and easy way to save pages. ArchiveBox looks fantastic, but I just don't have the motivation to set up the service and maintain it since I don't save enough links to make it worthwhile. SingleFile might be worth a shot, but it looks like WebScrapBook has been handling your needs just fine (they seem to have 90% of the same functionality).
As a webscrapbook user, do you know if there is a migration path from pocket or another hosted service?
Don't know about a migration option, but I do remember there's a lot of custom configuration possible.
Thanks!

ArchiveBox does indeed look fantastic. Their homepage alone is beautiful.

I bookmarked both ArchiveBox and now also SingleFile, but WebScrapBook gets the job done (in almost all cases).

Unfortunately that won't allow you to click links in your offline version. you can do this properly with wget: (sorry I don't know how to do code formatting in hackernews)

wget --mirror \ --convert-links \ --html-extension \ --wait=2 \ -o log \ https://example.com

Code formatting is just blockquotes.

  So one empty space followed by indented text (2 or more spaces)
Looks like SingleFile helps with sites where you have to be logged in, something that is not that easy with wget.
What are you talking about? I have hundreds of pages saved with SingleFile and I can click links in all of them.
Oh maybe it does work then. I assumed it didn’t follow links because they didn’t show it in the video.
Do you mean converting links so that they point to your local copy of other pages from the same site? Yeah it doesn't do that; it isn't a tool meant for archiving whole websites. It's just for archiving a single page, with all the content baked into it. Like a bookmark but doesn't linkrot.
You don't need to newline every flag of a trivial command.
I'm guessing the user's intent was to have the command formatted across multiple lines.
(comment deleted)
Are you suggesting to mirror e.g. the entire Wikipedia through wget?

That is not only suboptimal, it is stressing on the server. At least you added a --wait=2, but on any large site/hoster/CDN, this might still get your IP banned or throttled. And on e.g. the English wikipedia this will then take 149 days. Which means that by the time you hit the last page, the first ones (and their links) are out of date.

If you add '--no-parent' (doesn't request anything that's not a page dependency above the requested URI) and a '--level=5' (only follows link 5 deep), you won't get all of a site. It makes it more realistic for getting wikipedia articles.
This is great. I've always wondered why this isn't the default behaviour for page saving in browsers. To an ordinary user saving a page implies saving a single file, not a file plus a directory of stuff. HAR can be useful but seems only for niche or specialised reasons.
The most impressive part of the demo is seeing how tidy his Downloads folder is!
Can we please stop with the 17MB GIF images used as demos? They use up lots of data immediately as you open the page, and are impractical, you don't know how long the animation is, can't forward/rewind, and you can't press fullscreen on a mobile.

And GitHub supports embedded videos in README.md files, videos are generally smaller than GIF files and their disabled autoplay is a feature = you save your data until you press play.

GitHub only recently expanded video support from gif to decent video formats, and many github enterprise installs don't have those new features yet. So, keep spreading the word.
Giving a massive upvote for this, disappointed and confused to see you've been downvoted here. There's literally no reason to use GIFs like this, and - as you stated, it's massively disrespectful to those not fortunate enough to have broadband connections, but would like access to the information.

Using data so wastefully like this always reeks of privilege to me - especially on something like GitHub. Wikipedia, for instance, never allows things like this.

> disappointed and confused to see you've been downvoted here

Because it's a relatively new feature, and probably, a lot of devs don't know about it (I didn't).

I did this [animated gif] once actually, before the feature was introduced, and I definitely hated it, but I had no choice.

Thanks for bringing this to the general attention, though :)

Sharing a project with the world and taking time to document it reeks of privilege? I really can't understand your reasoning.
I think there is some nuance here.

If the demo sequence is <5 seconds, I have never found myself becoming impatient. Gif is perfect for very brief demos. Anything longer than that and I'd like to have some idea where I am at in the video stream (and other controls as indicated)

Just the "intro" splash takes more than 5 seconds, and it's totally unnecessary

Yes, the gif bothered me too :D

It would seem that this subject is very subjective. However, I admit that I let myself go a bit on the intro and I can understand your point of view.
> GitHub supports embedded videos in README.md files

True since May 2021 so I think a lot of people are still finding this out...

In my experience GIF is still the most set-it-and-forget-it way to know a video will play, to get cross-platform support out of mp4 you may have to provide two different codecs. Anyway, not disagreeing with you and most gifs could drop 90% of their size with better choice of resolution and framerate. This readme is particularly egregious doing a screen capture with scrolling.

As for saving bandwidth until you want to play, I haven't tried this yet but it seems adequately clever to wrap a loading=lazy gif inside a details/summary tag: https://css-tricks.com/pause-gif-details-summary/

Not to mention that H264 can take quite a bit of horsepower to decode and play as well (assuming your machine doesnt have a hardware chip specifically for doing just that)
Which machine doesn't? Anything in the last 10 or so years will decode H264 with much less power than GIF because of it. Even a Pi supports it.
My 2014 Thinkpad X1 Carbon (gen 3) doesn't have hardware transcoding as far as I can tell made Zoom and Discord impossible to use for class, especially because there was no way (that I knew of) to disable all video except the presenter. Even playing a YouTube video on it makes it ramp up.
I'm not sure which CPU you have specifically but the lowest-end model of the X1 Carbon Gen3 has an i5-5200U [1] that lists Intel Quick Sync Video support.

From the wiki page for Quick Sync [2]:

> Intel Quick Sync Video is Intel's brand for its dedicated video encoding and decoding hardware core. Quick Sync was introduced with the Sandy Bridge CPU microarchitecture on 9 January 2011 and has been found on the die of Intel CPUs ever since.

I can't confirm but I'd guess your performance issues lie elsewhere than in the h264 decoding specifically.

[1] - https://ark.intel.com/content/www/us/en/ark/products/85212/i...

[2] - https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video

If you check out the generation-codec table in that wikipedia article [1], under Broadwell (I believe that's the 5200U's generation name), it says there is support for AVC (which I believe is H264, I'm not a codec wiz), so that's a really good point. I'm not sure why I've consistently had issues with this on my machine then. I wonder if this is something with a configuration on Linux then?

Thanks for pointing that out. I've looked at this table before and payed attention to HEVC, not AVC, so I believe that's where my mistake came from.

[1] https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video#Hardwar...

AVC is H.264, yes.

Accelerated video decode is often disabled by default on Linux versions of browsers and can be quite dependent on versions of drivers/mesa/X-vs-Wayland/etc.

YouTube by default prefers newer, bitrate saving codecs over old ones if it thinks your CPU can handle software decoding them. On my 2017 Dell XPS 1080p and lower resolutions on YouTube play in software decoded AV1, only 1440p and higher play in hardware decoded VP9, so playing 4K video on YouTube is less taxing for my CPU than playing a 1080p video....
You can use h264ify extension to fix it.
The problem is Zoom and Discord are doing multiple streams. But it really shouldn't be a problem.

H.264, even on the high profile is not CPU intensive on a 2014 machine. Unless you are watching 1080P with 5-10Mbps, which is not the norm for internet video.

Is this really still an issue in 2022? How many people are browsing the internet on a device that can't do hardware H264 decoding?
Some browsers have poor hw decoding support on Linux (their problem, not drivers) but its gotten a lot better recently.
> to get cross-platform support out of mp4 you may have to provide two different codecs

Video codecs are not my area of expertise. Which codecs are these and what tool(s) would you typically use to ensure you provide them?

Author here, sorry for the GIF file. I created it because people were not happy with the video hosted on Youtube. AFAIK, video files did not work when I did this demo. I'll try to improve this in the future.
I wanted to comment on how useful that demo was to me. It did a great job at demonstrating why this is useful and how well it works compared to the native browser implementation. Thank you both for the demo and for the project!
> And GitHub supports embedded videos in README.md files

Any documentation on this? Because I have tried to embed video in issues and PRs before, and did not manage. I'm hoping such documentation will explain how this extends to issues and PRs.

In issues its just drag-n-drop.
I wish browsers came standard, preconfigured with warning dialogs that triggered if assets attempting to load were beyond some threshold. That threshold could be decided by the browser vendors group based on some collection of network statistics and be adjusted on an annual basis or so.
FYI, I updated the README page. Now, it includes a mp4 video weighting ~9MB.
If it’s a single file, then how do the images get stored?
I use SavePageWE, it can save the page (into single file) as it was modified by JS after load, which is often useful.

The only thing I miss I wish it was easier to script.

Related: I used to keep a collection of locally mirrored web pages a long time ago, with a legendary Firefox extension called ScrapBook [0] (now long retired). The surprise for me is that after all these years I still remembered the name...

While writing this comment I found that it lived on as a (now "legacy") new extension named ScrapBook X [1], and then yet another one named WebScrapBook [2], which seems to still be alive!

[0]: http://www.xuldev.org/scrapbook/

[1]: https://github.com/danny0838/firefox-scrapbook

[2]: https://addons.mozilla.org/en-US/firefox/addon/webscrapbook/

Author here, it makes me really happy to see SIngleFile on the front page of HN. Thank you! I take the opportunity to make you aware of the upcoming impacts of the Manifest V3 [1], and for those who prefer zip files, I recommend you to have a look here [2].

[1] https://github.com/gildas-lormeau/SingleFile-Lite

[2] https://github.com/gildas-lormeau/SingleFileZ

If I start using SingleFile today, will I still be open saved pages after the update to Manifest V3?

I mean, if I want to save pages over the next 11 months, should I install SinglePage or SinglePage-lite?

In fact, you simply do not need an extension to open pages saved with SingleFile (or SingleFile Lite) because they are standard HTML pages. So you don't have to worry about that.
This alone is fantastic. I've been looking for an mhtml replacement that worked well across all browsers.
Thanks for this project. I found SingleFile a year or two ago, and used it to take "HTML Screenshots" of third party sites I could embed in guided walkthroughs with modified/example data changed, instead of just PNGs.

SingleFile was ultra-valuable for this.

If anyone has a similar use-case, I wrote some pretty rough (and slow) code to post-process SingleFile's output to remove any HTML that wasn't contributing to the presentational render by launching puppeteer and comparing pixels. It's available here: https://github.com/mieko/trailcap

It's interesting! I had started something similar as part of testing but hadn't really finished my work. I will have a look at your project.
Oh, that’s such a good idea for documentation - thanks for sharing!
One very useful thing you could add to this (if you feel like it) would be to make it work with snapshotted directories, rather than a single HTML file with inlined data. You can get the former with SingleFileZ and then extracting the resulting zip file.

I like these because it makes it easier for me to make manual edits when necessary and it's a better solution for long term archiving (IMO). But I would love to add your project to my workflow.

Single File is one of my favorite addons since it allows me to keep offline copies of articles, tutorials, etc i see online without losing images, etc (there have been a ton of articles lost over the years and while some are preserved in archive.org, they often lack things like images, etc, so i prefer to save anything i come across). So thank you for making it :-).

Now, having said that, the text in SingleFile-Lite's "Notable features of SingleFile Lite" sound like a list of issues :-P. It looks like these are issues with Chrome, but do you know if/how these "improvements" will affect Firefox?

Thanks for the work you have done, its a lazy man heaven especially for bulk downloads and helped me a lot. About a month ago I have decided to backup my bookmarks via archivebox, it was more than 1k bookmarks, most reliable methods were singlefile and wget.
Is it possible to use this within the context of the current web page, without the extension portion?

Taking a snapshot of my user's screen and then display it to them later (maybe in an iFrame)?

It's possible but it's a bit limited. It won't be able for example to save images coming from a different origin.
FYI: Figure tags don't convert their hrefs to base64.

For example, try saving my home page: https://andrewrondeau.herokuapp.com/

The img tags are converted correctly, but there's still <figure class=image><a href="https://andrewrondeau.herokuapp.com/... in the single HTML file.

I cannot reproduce your issue, I just did a test on this page and I see the expected `<img src="data:image/jpeg;base64,...` in the saved page.
Look for figure tags, not image tags

Even better, just search for ="http

All I can find is a <a> tag.

Edit: I think I now understand the issue. I confirm SingleFile saves only the current page and not linked images, for example.

Twelve year project with nearly 7000 commits shows a lot of dedication. Good work.
I've been using SingleFile for the last year or so, it's amazing!

I'm going to hijack your post for a question! I love the way you can use the editor and select "format for better readability," then save just the stripped down version of the page. I use this to send it to my e-ink device.

The question I have is whether it's possible to toggle the default save to use the formatted version automatically? I dug into the options and didn't turn anything up!

You can enable these options for this:

- Annotation editor > default mode > edit the page

- Annotation editor > annotate the page before saving

Sorry, I was wrong, you have to select "format the page" instead of "edit the page" (first item).
Is there a configuration for the zip version where I can avoid duplicating the static assets? Thanks
I guess you're referring to SingleFileZ. This option is not needed because zip files (i.e. what SingleFileZ produces) already provide this feature.
Thank you for the Manifest V3 critique, the examples you give make it really clear how many things are regressing with this upcoming change :/
Thank you, very useful and works like a charm: a must have.
Well done, gildas! You have really built a quality product and marketed your work well.
I love it and use it. But product? Nothing is being sold and product sounds almost condescending. :)
> But product? Nothing is being sold and product sounds almost condescending. :)

Really!? You don't think it's trying to be condescending to try to find criticism on someone's praise, haha? Heh, aanyway :) You feel product is bad!? So weird!! I guess you find there what you bring to it. Wonder what you're protecting there, if you share more of your thinking, we can do know you more. Even so, I think we can just celebrate gildas' achievement! :)

Thank you for your work! I've been using start.me for my new tab page (it's the page you see every single time you open a tab, I can't believe most people don't make it useful), but it's way too slow so I SingleFile it and have a local Firefox extension to set it as my new tab page.

It's too complicated but at least it works!

That's a nice and simple tool, good work. I'm personally using Zotero to save copies of web pages: https://www.zotero.org/. With the browser extension you can save a snapshot in a few seconds.
Zotero is actually using SingleFile under the hood to save web pages ;)
Similar project -> https://github.com/Y2Z/monolith

(I used both and ended up favoring monolith, but can’t remember why. I think they’re pretty comparable/am grateful for both of them)

I love monolith too, especially because it is so easy to modify. Only 2300 sloc!

Sure, it uses libraries to do the heavy lifting, but these are all popular, well-tested libraries with well-scoped feature sets (html5ever for parsing HTML, url for parsing urls, etc).

If you're looking for a tool like this but think you might need to tweak it, you should give monolith a try.

Iran has a habit of using tools like this to trick defense contractors into using their page.
(comment deleted)
Nice project! This project, and a similar project called Monolith[0], was a bit of an inspiration for making my own single HTML file tool called Humble[1] to solve a few edges cases I was having with bundling pages (and since I wanted a TypeScript API for making page bundles).

[0] https://github.com/Y2Z/monolith

[1] https://github.com/assemblylanguage/humble

Maybe a little OT, but founders should take a careful look at this landing page. That's how you sell something. The demo is clear about the problem they're trying to solve and it convinced me that their product actually solves it. It's not just all the information they've included, but also the lack of irrelevant clutter.
you actually sat there and watched a 17MB gif video at 1x speed with no controls? lol. worst landing page i've ever seen
I think this is the worst comment I've ever seen as well! ;)

BTW, it's not a landing page. This is the README on GitHub.

I scrolled past the gif because I didn't realize that it was an informative gif. First few seconds looked like just an animated logo and I never stayed to watch it. Or it could've just started with an action instead of animating the logo.