This is why I've been saving PDFs/HTMLs or even just taking screenshots of webpages I find especially meaningful or important to me... Archiving things as files this way can get kind of tedious and definitely feels primitive at times (we made the LHC but I'm here not expecting the same pages of cool interviews, designs, etc. to be up next month), but what can you do?
But it's nice to know you're not alone in wanting nice things to last :)
Yes, I wish I had started doing that years ago. I went back through all my old bookmarks that I had saved from the past 2 decades and a significant portion are dead now.
pURL are hard. I have one which is an X.509 enforced pointer to a PDF which is 'terms and conditions' and the pain to ensure the link never dies (its embedded in binary objects which are long lived, and out in the world beyond our control) is non-zero. Change publishing model, change CMS, the pURL is at risk.
Publication models where you pay the cost for somebody to be the archival reference make some sense. URI pointing into a store. Buckets in Google? But if you decided to move on, could you get a retained 302 redirect to point to the new home?
Imagine sticking a proxy between our browser and the internet that automatically archives all webpages we attempt to browse to, and then only lets you view the archive. How much of the internet would you be able to see?
HTML may contain style and script tags. Sometimes minification may break CSS or JS - however rare that is, if you want your archive to be reliable you don't want functionality that may break something. If you intend to reduce storage space usage, HTML may be gzipped - this is "losless" so it does not carry a risk of breaking anything
I would imagine it’s because one is markup of text, the other serialized image data. One potentially has meaning when viewed raw, one probably doesn’t.
Indeed, it should be use "plain svg" image instead. Some drawing program like Inkscape use SVG as their save format and add a lot of internal tags/properties that are just junk for other programs. Generally they offer the option to save as "plain svg" without all of this tags.
There's no reason why a web browser bookmark action doesn't automatically create a WARC (web archive) format.
Heck, with the cost of storage so low, recording every webpage you ever visit in searchable format is also very realistic. Imagine having the last 30 years of web browsing history saved on your local machine. This would especially be useful when in research mode and deep diving a topic.
EDIT: I forgot to mention https://github.com/webrecorder/webrecorder (the best general purpose web recorder application I have used during my previous research into archiving personal web usage)
In practice how is this different from MHTML? I think most browsers have built-in support for MHTML so it should be possible to build that part easily.
The state of mhtml support is fairly pathetic at the moment. Firefox broke mhtml compatibility with the quantum overhaul. Chrome's mht support had been a hit and miss over the years, sometimes removing the GUI option entirely and requiring one to manually launch the browser with a special tag to enable it. The only browser with a history of consistent mhtml support happens to be....Internet Explorer, followed by a bunch of even more obscure vendors that nobody really uses.
I am currently dealing with the problem of parsing large mht files (several megabytes and up). A regular web browser would hang and crash upon opening these files and most ready made tools I could find struggle with the number of embedded images. It's very much a neglected format with very little support in 2019.
It's a pain, like most of the WARC ecosystem. It's been several months since I dug in, so maybe it's had some spitshine the last little bit, but I usually end up using combinations of wpull, grab-site, and a smattering of other utilities to reliably capture a page/set of pages, and have had to make some quick hacks as well as manually merging in some PRs to get things to work with Python3. Once I have the WARC, I typically end up using warcat to extract the contents into a local directory and explore that way.
WARC as a format seems promising, but at least last I checked, open-source tooling to make it a pleasant and/or transparent experience is not really there, and worse, at least as of several months ago, doesn't really seem actively worked on. Definitely an area you'd expect to be further along.
>There's no reason why a web browser bookmark action doesn't automatically create a WARC (web archive) format.
Indeed.
And I still remember the modem days where I would download entire websites because the ISP charged by the hour, and I'd read them offline to save money.
Yeah. In the dialup days, layer 1 and 2 of a home internet connection was a long-running phone call between your own modem and a modem of an ISP. You payed via your phone bill, for the duration of the call.
You can ask Safari to do that by enabling 'Reading List: Save articles for offline reading automatically'. It's not WARC but it is an offline archive. The shortcut is cmd-shift-D which is almost the same as the bookmark one. It's also the only way I know of to get Safari to show you bookmarks in reverse chronological order. And it syncs to iOS devices.
This could be done in better and more specialized ways, one problem is browser extension APIs don't provide very good access to the browser's webpage-saving features.
> SingleFileZ is a fork of SingleFile that allows you to save a webpage as a self-extracting HTML file. This HTML file is also a valid ZIP file which contains the resources (images, fonts, stylesheets and frames) of the saved page.
Off topic, but could I ask how you knew your software was being talked about? Did you just happen by or have you some monitoring agent looking for mentions? Just curious
Sorry, I didn't see your question. I check the posts on HN very regularly. The title of the post made me think that people might have been talking about SingleFile.
Sometimes, friends of mine tell me someone on the Internets is talking about SingleFile :). I also sometimes use the integrated search engine.
Whoa. I just installed SingleFileZ for FF and it is working great. Before I was using wget and that was clunky. This is working great since I can just toss a single file up on my server and we are good to go. Thanks for this!
Safari used to (and still) do this automatically but in a limited way. In the browsing history view (Command Y), you can search visited pages by its content, and this is extremely useful. But there's no way(†) to tell Safari to display that saved content. If you revisit a URL in the history, Safari fetches it again, losing the original saved content.
Firefox actually purges history automatically. For instance, the oldest history I have on this browser right now is from January 2018. I found about this the hard way.
It looks like Firefox has been doing it since 2010[1]. I wonder how long Chrome has been doing it, since launch, 2008? Here's a Chrome bug discussing it[2].
I noticed this behavior in Firefox too. So I started writing personal Python scripts to scrape FF's SQLite database where it stores all the browsing history information.
I have this problem. Some bits of history are gone except from old backups of profile directories and profiles where I've already set places.history.expiration.max_pages to some absurdly high number.
I need to do a handful of experiments to see exactly how this interacts with Sync, even though I've (foolishly) already synced the important profiles. I'd hope that the cloud copies of the places database just keeps growing, but in any case, I'd rather combine them all offline anyway.
Even if you set the setting, how can you be sure that it won't be reset on an upgrade or that you'll remember to set it if you need a new profile (perhaps your old one becomes buggy, crufty, corrupt, or all three)? I thought I had all my history retained until one day I couldn't find a website I knew I had visited years ago, and took a closer look at my history and was very unpleasantly surprised... What happened? I'll never know, but my suspicion is that Firefox reset the history retention setting at some point along the way. If you do any web dev, you know Firefox occasionally backstabs you and changes on updates. The only way to be sure over a decade-plus is to regularly export to a safe text file where the Mozilla devs can't mess with it. I can't undo my history loss, but I do know I have lost little history since.
I can't be sure. When I say 'combine them all offline', I mean using something like [1] which refuses to do anything for me because the Waterfox database version is a rather old Firefox version, and that seems to expect all the db's versions to be up-to-date and equal, which seems pointless. #include <sqlite3.h> was my next step-- only I don't walk very well, so that didn't happen "yet". Or I'm lazy, or distracted, or depressed, or something. When I recently got tired of realizing a thing was on the other machine, I bit the bullet and synced them, if only to see how well that worked.
This was what made me convert from bookmarking to clipping pages into Evernote around 6-7 years ago. I realized I had this huge archive of reference bookmarks that were almost useless because 1) I could rarely find what I was looking for, if I even remembered I'd bookmarked something in the first place, and 2) if I did, it was likely gone anyway. With Evernote I can full text search anything I've clipped in the past (and also add additional notes or keywords to ease in finding or add reference info).
Since starting with replacing bookmarks, I've moved other forms of reference info in there, and now have a whole GTD setup there as well, which is extremely handy since I can search in one place for reference info and personal tasks (past and future). Only downside is I'm dependent on Evernote, but hopefully it manages to stick around in some form for a good while, and if it ever doesn't, I expect I'll be able to migrate to something similar.
I was an Evernote user when I was on macOS. When I switched to Linux, a proper web clipper was something I really missed. I'm now on Joplin and it does everything I used to use Evernote for and then some.
It even has vim bindings now!
As far as longevity goes, I think they got their archive / backup format right - it's just a tarball with markdown in it.
No need of proprietary code and apps why not build it into browsers. I have seen Firefox and Chrome can download web pages. So it will be nicer if they can download the bookmarked pages and store in a local html, css, image folder. I think it's pretty easy to achieve.
Also people need to move away from those esoteric reactjs, angular, vuejs and plethora of CMS as API or static site generators relying on some js framework which won't last even 2-3 years. Use a static site generator which can generate a plain html, like static site generators built on pandoc, python docutils or similar.
Personally I like restructuredText as the preferred format for content as its a complete specification and plain text. So the only thing in this article I will change is that content can also be in rst format and then generate html from it. Markdown is not a specification as each site implements their own markdown directives unlike restructuredtext specification and most of the parsers and tooling are little different from each other.
It is stil not a specification like restructuredText[1]. Also wikiMarkup (which really started this markdown) is different from GitHub markdown, which is different from other markdown editors. Also many sites use their own markdown versions.
If you are in restructuredText world there is one specification and all implementation adhere to it, be it pandoc, sphinx, pelican, nikola. The beauty of it is that it has extension mechanisms which provides enough room for each tool to develop it. But markup can be parsed by any tool.
Because its inherently appealing, close to what you wanted intuitively, and if you're only dealing with a single implementation of it, works fairly well.
You don't really get bit by its lack of a standard and extensibility until after you've bought in.
It's essentially designed by the opposite of a committee -- rather than including everything but the kitchen-sink, it contains support for almost no usecases except the one. Which is very appealing, when you only have the one usecase.
Well rst is better than markdown from day one. The only reason it became famous is thanks to wikimarkup.
So markdown needs to thank the popularity of Wikipedia for its success, as rst did not have any application like Wikipedia. But still rst is used widely enough with its killer Sphinx, readthedocs and now its kind of de-facto documentation writing markup in Python and many open source software world.
Because you can teach someone markdown in five minutes. And even if they don't know all the ins and outs, the basics are pretty foolproof (paragraphs, headings, bold and italic).
> Personally I like restructuredText as the preferred format for content as its a complete specification and plain text.
I have used rst intensively on a project. A few years later, I would be hard pressed to write anything in ti and would need to start with a Quick Start tutorial. With all its faults, Markdown is simple enough that it can be (and is) used anywhere, so there is no danger of me forgetting its syntax (even if it wasn't much simpler to star with).
This... I just found a plugin for the static site generator Pelican that is 7 years old that still works.
After running Pelican you get plain HTML that can be hosted anywhere.
I like Netlify, but other options like GitHub pages are also great.
The author recommends not putting on GitHub Pages because they haven't found a working business model and might not be here in the future.
But... GitHub has been taken over by Microsoft which is most likely not going bankrupt soon and Microsoft loves their backward compatibility so I am confident they won't screw GitHub up too much.
You can say the same about geocities when it was acquired by yahoo. But it didn't last and then now it's happening with yahoo-groups. So I am not hopeful if GitHub becomes a liability microsoft will keep it.
Joplin is open source, which is a big part of the sell to me. It definitely isn't the best of all possible note taking systems that could ever exist, but it's the best open source one I've found so far, and I don't have time to write a better one at the moment.
> why not build it into browsers. I have seen Firefox and Chrome can download web pages. So it will be nicer if they can download the bookmarked pages and store in a local html, css, image folder. I think it's pretty easy to achieve
This is solving a different problem though. WARC/MHT and other solutions can do this. Joplin is more of a note taking system that allows ingesting content from the web into one's own local notebook, which is relevant to what the GP post was talking about - Evernote.
However, it would seem that "the modern web" is the now popular standard. 10 years ago it might have been Flash or Java web applets or whatever. Now it's JS. I'm not convinced that JS is any better than what it has replaced. However, people keep paying developers to write them, so presumably someone likes them.
> Also people need to move away from those esoteric reactjs, angular, vuejs and plethora of CMS as API or static site generators relying on some js framework which won't last even 2-3 years. Use a static site generator which can generate a plain html, like static site generators built on pandoc, python docutils or similar.
Agreed, but that's also not a problem that Joplin, Evernote, or any other such tool is going to be able to solve. Unless you are complaining that Joplin is an Electron app? That's my biggest issue with it personally. It runs well enough, but is definitely the heaviest application I use regularly, which is a little sad for a note taking program. On the other hand, I haven't found a better open source replacement for _Evernote_. There are lots of other open source note-taking programs though.
> Personally I like restructuredText as the preferred format for content as its a complete specification and plain text. So the only thing in this article I will change is that content can also be in rst format and then generate html from it. Markdown is not a specification as each site implements their own markdown directives unlike restructuredtext specification and most of the parsers and tooling are little different from each other.
reST is indeed very nice. At one point, I kept my personal notes as a Sphinx wiki with everything stored in reST. I found this to be less ergonomic than Evernote/Joplin, although in principle it could do all the same things that Joplin can do, and then some.
Thanks a lot for the recommendation! I have been a little annoyed with Evernote not having an app for Ubuntu, which I recently started using quite heavily. So this looks very interesting!
>if I even remembered I'd bookmarked something in the first place
I had recently participated in a discussion on the problem of forgetting bookmarks[1].
Copying my workflow from there,
1. If the entire content should be easily viewed, then store via pocket extension.
2. If a partial content should be easily viewed i.e. some snippet with link to entire source, then store in notes (Apple).
3. If the content seem useful in the future, but it is okay to forget it; then I store it in the browser bookmarks.
But, my workflow doesn't address the problem raised by Mr. Jeff Huang; if Pocket app or notes disappear so goes my archives. I think self hosted archive as mentioned by the parent is the way to go, but I don't think it's a seamless solution to a common web browser user.
My solution for a small subset of the forgetting problem:
I frequently see something and want to try it out the next time I want to do something else. So I emulate User Agent strings and append lots of "like [common thing I search for a lot]" to the bookmark. When I start typing into the search bar for those other things I'll be reminded of the bookmark.
For example, since file.io is semi-deprecated I decided to try out 0x0.st . But I kept forgetting when I actually needed to transfer a file, so I made a bookmark titled "0x0.st Like file io".
As a side note, I have a similar bash function called mean2use that I use to define aliases that wrap a command and ask me if I'd like to do it another way instead or if I'm sure I want to use the command. I've found this is a nice way to retrain my habits.
Exported sure, it's all there. But importing that into your new favorite notes application is not going to be trivial, especially not for regular users.
That's why I've decided to stick mostly to regular files in a filesystem.
Presumably "regular users" will not be individually writing XML parsing code to convert the notes. The developers of their "new favorite notes application" will do it (and if they can't be bothered, maybe it shouldn't be your "new favorite notes application").
Joplin, for example, can import notes exported from Evernote. It's just a menu option that even regular users should have no trouble employing.
> Only downside is I'm dependent on Evernote, but hopefully it manages to stick around in some form for a good while, and if it ever doesn't, I expect I'll be able to migrate to something similar.
I have used Evernote and OneNote, but have finally, after a long interim period, resorted to using only markdown.
I have a "Notes" root folder and organize section groups and sections in subfolders. VSCode (or Emacs), with some tweaks, shortcuts, and extensions, provides a good-enough markdown editing experience. Like an extension that allows you to paste in images, storing it in a resources folder in the note's current location (yes, I see small problems with this down the road when re-organizing, but nothing that can't be handled).
For Firefox, I use the markdown-clipper extension the few times I would like to copy a whole article, it works well enough. Or I copy/paste what I need; mostly, I take my own summarized notes.
For syncing, I use Nextcloud, which also makes the notes available both for reading and editing on Android and iOS (I use both).
Up until very recently, I used Joplin, which also uses markdown, but there were two things I could not live with: it does not store the markdown files with a readable filename, e.g., its title, and being tied to a specific editor.
If you are mostly clipping and not writing your own notes, I can imagine my setup won't work well, or be very efficient.
I want to use a format that has longevity, and storing in a format that I cannot grep is out of the question.
It would be nice if we had browsers that were actually user-agents that allow full pluggable customizability for all cookie, header, UI, request, and history behavior. Then, this would just be a plugin that anyone could install.
And then people install garbage extensions that break the browser and people think "Wow, this firefox browser is so buggy and slow" and switch to chrome. And then your extensions break with every single browser update because they are tampering with internal code.
Everyone is free to fork a browser and apply any changes they want. Allowing extensions to change anything at all essentially is the same as forking and merging your changes with upstream every update.
This is what started me clipping everything to OneNote instead of bookmarking. Unfortunately, it becomes difficult to maintain, the formatting is off, things subtly break, pages clipped on mobile use different fonts for god knows what reason, some content is discarded silently because the clipper deems it's not part of the main article, I could go on.
It's better than nothing but it's also increasingly frustrating to deal with.
WorldBrain's Memex (https://addons.mozilla.org/en-US/firefox/addon/worldbrain/) has an option to perform a full-text index (not archive) of bookmarks, or pages you visited for 5 seconds (default) down to 1 second (no option to index all pages). It stores this stuff into a giant Local Storage (etc) database, which Firefox implements as a sqlite file.
Having to pay $15/month ($180/yr!) to be able to search stuff on my own computer for years seems awfully expensive. I'd rather depend on some simple open-source piece of software that I can understand and maintain if necessary.
Decades ago there was an amazing piece of software from lotus when I worked there called magellan. I remember the first time I saw someone search, and find results in text documents, spreadsheets and many other of the common formats of the day.
That was in 1989 and today I mostly search my computer using find and grep commands, since that's what just keeps working.
Yeah, the sheer idea of paying a subscription for software that is running on my computer to index local resources is crazy. This kind of software should be should sold as one-time buy license.
The main disadvantage was disk space. This is particularly true when some pages are 10 MB or larger. I would periodically prune the archive directory for this reason.
I stopped using Shelve when I started running out of disk space, and now I can't use Shelve because the addon is no longer supported. The author of Shelve has some suggestions for software with similar functionality:
It installs and loads in Waterfox, but of course it still hasn't been touched in 3.5 years.
I used Scrapbook in the past (which also still works) but I usually just save random things in ~/webpages/ since (apparently) 2011. The earliest is a copy of the landing page at bigthink.com. Of course now almost every link is broken, excluding social media buttons, About Us, Contact Us, RSS, Privacy, Terms of Use, Login, and the header logo pointing at the same page.
Pinboard happens to be a web service run along the same principles as the article we're discussing.
The bus factor is high, but I suspect that Maciej has a plan that'll let us download our archive even if he does get grabbed by the mainland Chinese government let alone a forecasted going out of business action.
Pinboard is a profitable online equivalent to a mom and pop shop. It’s sustainable and its founder isn’t chasing growth at all costs. It also has a cult following, so OOB is highly unlikely.
Better than a bookmark action would be a commandline option, similar to Firefox's -screenshot, which will work without starting X11. Something like -archive:warc
Personal wayback machins should be standard computing kit. I have had one since around 2013. Very bare bones demo: https://bpaste.net/show/3FBH6 it does much more than that. file:// is supported for example, so you can recursively import a folder tree, and re-export it later if you wanted to.
Or in some random script: "from iridb import iri_index" "data = iri_index.get('https://some/url')" I'm skipping lots, you can ref by hash, url, url+timestamp. It hands back a fh, you dont know if the data you are reffing even fits in memory. Extensive caching, all the iri/url quirks, punycode, PSL etc.
Some random pdf in ~/Downloads, "import doc.pdf" and dmenu pops up, you type a tag, hit enter and the pdf disappears into the hash tree, tagged, and you never need to remember where you put it. Later on you only need to remember part of the tag, and a tag is just a sequence of unicode words.
Chunks are on my github (jakeogh/uhashfs, it's heka-outdated dont use it yet), I'll be posting the full thing sometime soonish.
>Heck, with the cost of storage so low, recording every webpage you ever visit in searchable format is also very realistic.
I tend to do that, I also save a lot of scientific papers, ebooks and personal notes. I've found that doing so does not help me at all. The main problem I have is that when I need to look something up (an article, a book, a bit of info) I reach for google first, usually end up finding the answer and go to save it, only to find that I had already found the answer beforehand (and perhaps already made clarifying notes to go along with it) and then forgot about it.
This, and not dead links, is the fundamental problem with bookmarks for me. Not only bookmarks, it extends to my physical notes and pretty much everything I do. If I haven't actively worked on something for a couple of months, I forget all about it and when I come back to it I usually have to start from scratch until I (hopefully) refresh my memory. Some of it is also usually outdated information.
I think this is a big, unsolved problem and I'm not even sure how to go about starting to solve it. I can envision some form of AI-powered research assistant, but only in abstract terms. I can't envision how it would actually work to make my life better or easier. It would need to be something that would help blur the line between things I know and things that are on my computer somehow. If I think of my brain like it has RAM and cache, things I'm working on right now are in the cache and things I've worked on recently or work on a lot are in RAM, but what's for me lacking is a way to easily move knowledge from my brain-RAM to long term storage and then move that knowledge back into working memory faster than I can do so now. I'm not even talking about brain uploading or mind-machine interfaces, but just something that can remind me of things I already know but forgot about faster than I can do so by myself.
I am convinced that figuring out how to do this will lead to the next leap in technological development speed and efficiency. Not quite the singularity that transhumanists like to talk about, but a substantial advancement.
What I've found is that I need to spend more time deciding what is important, and less time consuming frivolous information. That's hardly a technology problem.
For things I really don't want to forget, I'm using Anki [0], a Spaced Repetition System (SRS). Anki is supremely good at knowing when you're about to forget an item and prompting you to review it.
Spaced practice and retrieval practice, both of which are used in SRS, are two learning techniques for which there is ample evidence that they actually work [1].
You still need to decide what is worth remembering, but that's something technology can't help with, I think.
I like the idea, but wanted to know how realistic it would be so I made a quick and dirty Python script to download all my bookmarks. If you want to make the same experiment, you can get it from here: https://gist.github.com/ksamuel/fb3af1345626cb66c474f38d8f03...
It requires Python 3.8 (just the stdlib) and wget.
I have 3633 bookmarks, for a total of 1.5 Go unziped, 1.0 Go zipped (and we know we can get more from better algo and using links to files with the same checksum like for JS and css deps).
This seems acceptable IMO, espacially since I used to consider myself a heavy bookmarker and I was stunned by how few I actually had and how little disk they occupied. Here are the types of the files:
It should probably be opt-in though, like a double click on the "save as bookmark icon" to download the entire file, and the star becomes a different color. Mobile phones, chrome books and raspy may not want to use the spaces, not to mention there are some bookmark content that you don't want your OS to index, and show you preview of in every search.
But it would be fantastic: by doing this experiment I noticed that many bookmarks were 404 now, and I will never get their content back. Beside, searching bookmark, and referencing them is a huge pain.
So definitely something I wish mozilla would consider.
> definitely something I wish mozilla would consider
There used to be this neat little extension called Read It Later that let you do just that. Bookmark and save it so you could read it when you were offline or the page disappeared. Later they changed their name and much later Mozilla bought it and added it to Firefox without a way to opt out. It was renamed to Pocket.
Pocket is not integrated with your bookmarks. For offline consultation, you need a separate app. Of course this app is not available on Linux, where you have to get some community provided tools.
Bookmark integration would mean one software, with the same UI, on every platform, and only one listing for your whole archive system.
I've actually been saving every page I visit for a good two years now and it has barely caused in dent in my NAS storage space. As usual though, I wrote a crappy extension and Python script to do that because I never bothered to look online. Thanks for introducing me to WarcProxy - I'll probably be making the switch very soon.
- Any comprehensive archive of your activity is itself going to be a tremendously "interesting" resource for others -- advertisers, law enforcement, business adversaries, and the like. Baking in strong crypto and privacy protections from the start would be exceedingly strongly advised.
- That's also an excellent reason to have this outside the browser, by default, or otherwise sandboxed.
- Back when I was foolish enough to think that making suggestions to Browser Monopoly #1 was remotely useful, I pointed out that the ability to search within the set of pages I currently have open or have visited would be immensely useful. It's (generally) a smaller set than the entire Web, and comprises a set of at least putatively known, familiar, and/or vetted references. I may as well have been writing in Linear A.
- Context of references matters a lot to me. A reason I have a huge number of tabs open, in Firefox, using Tree-Style Tabs, is that the arrangement and relationships between tabs (and windows) is itself significant information. This is of course entirely lost in traditional bookmarks.
- A classification language for categorising documents would be useful. I've been looking at various of these, including the Library of Congress Subject Headings. A way of automatically mapping 1-6 high-probability subjects to a given reference would be good, as well as, of course, tools for mapping between these.
- I've an increasing difference of opinion with the Internet Archive over both the utility and ultimately advisability of saving Web content in precisely the format originally published. Often this is fragile and idiosyncratic. Upconverting to a standardised representation -- say, a strictly semantic, minimal-complexity HTML5, Markdown, or LaTeX, is often superior. Both have their place.
On that last, I've been continuing to play with the suggestion a few days ago for a simplified Washington Post article scrubber, and now have a suite of simple scripts which read both WashPo articles and the homepage, fetching links from the homepage for local viewing. These tend to reduce the total page size to about 3-5% of the original, are easier to read than the source, and are much more robust.
I'm reading HN at the moment from w3m (which means I've got vim as my comment editor, yay!), and have found that passing the source to pandoc and regenerating HTML from that (scrubbing some elements) is actually much preferable, for the homepage. Discussion pages are ... more difficult to process, and the default view in w3m is unpleasant, though vaguely usable.
Upshot: saving a WARC strictly for archival purposes is probably useful, but generating useful formats as noted above would be generally preferable in addition.
With the increasing untenability of mainstream Web design and practices, a Rococco catastrophe of mainstream browsers, the emergence of lightweight and alternative browsers and user-agents (though many based on mainstream rendering engines), the tyranny of the minimum viable user attacking any level of online informational access beyond simple push-stream based consumption, and more, it seems that at the very least there's a strongly favourable environment to rethinking what the Web is and what access methods it should support. Peaks in technological complexity tend to lead to a recapitulation phase in which former, simpler, ideas are resurrected, returned to, and become the basis of further development.
Does this also strip the megabytes of superfluous tracking JS? It's probably what'll be the bulk of the size on the modern web, and I don't feel any particular need to store it.
(I believe that for historical purposes, enough complaining about ads and tracking will survive that future historians can easily deduce the existence of this practice)
The point is to make a webpage that lasts. So people can link to it and get the page. That means making a maintainable webpage and a url that does not change.
It is great that you can archive every page you visit for yourself, but that is not the same as making a lasting web.
FileCoin is a blockchain and is the incentivization layer on top of IPFS that allows users to rent hard drive space. It is intrinsically connected with IPFS.
The parent comment is still accurate. IPFS isn't a blockchain and shouldn't be dismissed for this use case, which is more or less what it was designed for.
Filecoin is just for incentives, but IPFS has been running for years without it.
I had no intention of bring up "blockchain", only that cryptographic hashes must play an important part of any archival scheme intended to last generations.
Otherwise you must trust that an archiver hasn't changed history. Why trust when you can verify?
Disappearing content is a blessing, not a curse. Let it all be replaced by new people doing the same things slightly differently, instead of constantly having to confront prior art.
This - so, so, much. I don't know where the assumption that everything is worth preserving comes from. To me, this is even part of the beauty of the internet: Things appear, then vanish again.
Often, the antique content is irreplaceable. If it disappears, then valuable and important reference material is lost forever. Consider, for example, if you are working with hardware or software created decades ago -- if information about such systems is lost, you're pretty much hosed.
As another example, I keep every line of code I write (that wasn't written for an employer) forever. I often pull up code that I've written decades ago to use in new projects.
The beauty of the internet is that the old and the new aren't mutually exclusive -- there's plenty of room for it all.
I fundamentally agree with the principle -- that pages should be designed to survive a long time -- however the steps the author lays out I completely disagree with.
"The more libraries incorporated into the website, the more fragile it becomes" is just fundamentally untrue in a world where you're self-hosting all of your scripts.
"Prefer one page over several" is diametrically opposed to the hypertext model. Please don't do this.
"Stick with the 13 web safe fonts" assumes that operating systems won't change. There used to be 3 web safe fonts. Use whatever typography you want, so long as you self host the woff files.
"Eliminate the broken URL risk" by... signing up for two monitoring services? Why?
I think this list of suggestions does a great disservice to people who just want to be able to post their thoughts somewhere. There's an assumption here that you'll need to be technically capable in order to create a page "designed to last" and frankly that is not what the internet is about. Yes, Geocities went away. Yes, Twitter and Facebook and even HN will go away. But the answer sure as hell isn't "I teach my students to push websites to Heroku, and publish portfolios on Wix" because that is setting up technical gatekeeping that is completely unnecessary.
The phrases "simple" and "news websites" don't combine well these days. Even the NPR website downloads 13.2 MB of content over 91 individual requests, and takes just over 3.6 seconds to load (6.5 to finish).
- CSS Stylesheets: 3
- Animated gifs: 1
- Individual JS files: 11 (around 2MB of JS decompressed (but not un-minimized))
- Asynchronous Requests: 14 (and counting)
And that's with uBlock Origin blocking 12 different ad requests.
That's not simple in any form. So, the possibility of something on this page breaking? High. There's a lot of surface area for things to break over time. And that's not counting what happens when the NPR's internal APIs change for those asynchronous requests.
If the site is being served over HTTP/2 then the 11 separate JS files is a good thing compared to a single 2MB JS file.
-
In my case it also has the added benefit of being able to cache JS for a long(er) period of time, with users only having to download maybe 0-30kb of JS when only 1 component is updated instead of invalidating the entire JS served (Way under 1MB however)
> "The more libraries incorporated into the website, the more fragile it becomes" is just fundamentally untrue in a world where you're self-hosting all of your scripts.
There are more problems though. older library versions might be vulnerable to XSS attacks, or use features removed by browsers in the future for security reasons (eval?). Or you might want to change something involving how you use the API but the docs are long gone. Generally, libraries imply complexity and when it comes to reliability, complexity will always be your enemy.
Also unless you're very diligent about semantic markup and separation of content, presentation, interaction logic, the more complicated s site is the more difficult it is to port.
I have run into this problem trying to migrate very old web pages or blog posts off of SaaS sites that are shutting down or just decaying. It's not just that complicated sites make it difficult to extract the content in the first place; it's difficult to publish that content on another site in a high-fidelity, and sometimes even readable, way.
The hard part isn't keeping the old site (page) running (although that's not always easy either). The hard part is when you want to do something _else_ with that content -- more complicated means less (easily) flexible.
>Use whatever typography you want, so long as you self host the woff files.
or use Google Web fonts, and set let last option in your font-family to be "serif" or "sans-serif" to let an appropriate typeface be used if your third-party font is unreachable. That's the beauty of text, the content should still be readable even if your desired font is unavailable.
Google Web Fonts are not an "or", here. Fonts have disappeared from it, and there is no reason to not expect Google to, at some point in the future, go: "you know what, this costs too much without any substantial return." And now it's just another killedbygoogle.com product. Just like images, self-hosting woff/woff2 should be step 1.
Fonts disappearing is not a big issue that will ultimately render your page useless. If the font is gone, the look of the page is slightly affected, but the content of the page remains. It's honestly not a big deal at all.
Here as we enter the 2020s, there are no longer any web safe fonts. Those 1990s Core Fonts for the Web (Verdana, Georgia, Trebuchet, etc.) are no longer universal across all widely used platforms.
Yeah okay, but the initial suggestion (just specify "sans serif") still holds. Or really, if we're talking about a webpage to last, why do we even care about what font is being used? If you care enough about a font that the glyphs used are important for layout, then obviously you're going to need to include the font. If the specific look of the page is essential to the content conveyed, it seems likely to me you won't be using a standard font anyway.
For typical "the words matter more than how the words look" content...can someone explain to me why we care about including the font?
There’s also layout issues caused when replacing a font with another font, unless the metrics are precisely duplicated. There’s a reason RedHat paid a lot of money to have Liberation Sans with the exact same metrics as Arial, Liberation Serif have the same metrics as Times New Roman, and Liberation Mono have the same metrics as Courier New.
His "or" was to suggest that instead of only self-hosting the font file, you simply use a google one with a "fall-back" that happens to be a super-standard font that won't reasonably disappear from most OSes in the near future. That way, you get a reasonable "best of both".
Google web fonts were a great way to make my site slower. I don’t know if it’s the latency here in Australia or what, but (especially for developing locally) google web fonts were a big headache for having snappy webpages. I took the time one day to produce my own webfont files and self-host those, and the difference in site load speed is like night and day.
And that's where they are not banned. Many pages simply won't load at all in the PRC because someone thought a Google analytics tracker or a hosted library should load before the content (which then never does).
Google Fonts also isn’t blocked but I recall it being hit-and-miss in terms of responsiveness when I was working on a website that targeted Chinese audience a few years ago. However, I just tried resolving fonts.googleapis.com and fonts.gstatic.com on a Chinese server of mine, and they both resolve to a couple of Beijing IP addresses belonging to AS24424 Beijing Gu Xiang Information Technology Co. Ltd., so it’s probably very much usable now.
Not sure "working fine in PRC" is really something you can say about anything web related.
I do occasional web dev from within China and had to eliminate external references to get manageable page load times. At least from where I work pulling in practically anything from outside the Great Firewall will have a high probability of killing page load time. Anything hosted by Google in particular will often have you staring at your screen for 30 seconds.
Yes, any additional domain you request to has a non-negligible chance of killing the entire connection. To GP, noticing that one request works once from a server (not a home or mobile connection) really means nothing. Every ISP has different and constantly changing failure modes.
The trouble is that the defaults tend not to be the best fonts that are available, and very few users change them. I have changed them myself, but I don’t know of anyone else that has.
For myself, I wish that people would leave Arial, Verdana, Helvetica Neue, Helvetica, &c. out of their sans-serif stack, having only their one preferred font and sans-serif, or better still sans-serif alone; but as a developer I understand exactly why they do it all.
Unfortunately, I'm one of those developers :( My font stack is:
font-family: system-ui, Helvetica, sans-serif;
for prose and
font-family: ui-monospaced, Menlo, monospace;
for monospaced text. The first being the user's preferred font, the second as a good (IMO?) default that I impose on them, and the third as a full fallback. I'm conflicted on whether this is the right balance between user choice and handling browsers that support nothing.
It would be great if web browsers had a way to actually indicate the user's preference of typeface, but what we've actually got is the browser's preference, and the browsers almost all have chosen really terrible default typefaces. It's fine to say "just use the default" for Mac users who get a decent default, but then the poor windows users have to suffer through some terrible serif.
The users who actually know how to change the default font also know how to use stylish.
When you go to a restaurant you let the chef prepare food for you.
Telling him to back off and let you cook because he can't know better than you (his user) would be absurd.
Same thing with design and typography. It requires skill and taste, and hopefully people will be delighted or simply consume the content for what it is, because the design/cooking just reveals that content in a convenient/useful shape.
Can you elaborate on why/how they suck? Do you have example links, to set a common ground for the conversation?
I think most fonts that get your attention suck, the best ones are invisible and get you directly to the meaning of text, without getting in the way. So maybe there's a kind of bias (selection or sampling bias?) operating here?
I can’t speak for the parent poster, but, yes, back in the Myspace days, end users would do really tasteless CSS like Comic Sans or an italic font everywhere. Back then, I told my browser “I don’t care what font they tell you to use, just render it with Verdana”.
These days, people either use their social network’s unchangeable CSS, or they use a Wordpress theme with an attractive and perfectly readable font. Even Merriweather, which I personally don’t care for, is easy enough to read.
The only time I have seen a page use obnoxious fonts in the 2010s is when the LibreSSL webpage used Comic Sans as a joke to highlight that the project could use more money:
Edit It may be a case that the parent poster likes using a delta hinted font, either Verdana or Georgia, on a low resolution monitor, and doesn’t like the blurry look of an anti-aliased font on a 75dpi screen.
> back in the Myspace days, end users would do really tasteless CSS like Comic Sans or an italic font everywhere.
Indeed, typography is a skill. Most designers should have it though, which is why I asked more information to OP.
> The only time I have seen a page use obnoxious fonts in the 2010s is when the LibreSSL webpage used Comic Sans as a joke to highlight that the project could use more money
Ah, the infamous Comic Sans. It's a shame because as a typeface on its own, in its category, it is pretty good. Sadly, it's misused all the time in contexts where it's not appropriate at all.
> It may be a case that the parent poster likes using a delta hinted font, either Verdana or Georgia, on a low resolution monitor, and doesn’t like the blurry look of an anti-aliased font on a 75dpi screen.
Without more details we cannot guess. You're right: a lot of things can go wrong and ruin a typeface, regardless of how the characters are designed. Anti-protip: a reliable way to make any font look like shit is to keep the character drawings as they are and mess up the tracking (letter-spacing) and kerning.
I think one of the reasons Comic Sans got such a bad rep is because it was one of the relatively few available fonts back in the pre-woff “web safe fonts” era of a decade ago. Microsoft should had given us a more general purpose font, such as a nice looking slab serif to fill the gap between the somewhat old-fashioned looking Georgia and the very stylized Trebuchet MS font.
Because they they are not the single system default sans-serif and single system default sans-serif-monospace fonts that all websites MUST use, period, no discussion. As you put it:
> fonts that get your attention suck
If I can tell the difference between your font and the system default font, your font sucks; if I can't tell the difference, what's the damned point?
Well, if it makes you feel any better, my website renders just fine on Lynx (no Javascript nor webfonts needed to render the page), complete with me putting section headings in '==Section heading name==', which is only visible in browsers without CSS. Browsers with modern CSS support see the section headings as a larger semibold sans-serif, to contrast with the serif font for body text. [1]
[1] There are some rendering issues with Dillo, with made the mistake of trying to support CSS without going all the way, making sure that http://acid2.acidtests.org renders a smiley face, but even here I made sure the site still can be read.
[2] Also, no cookies used on my website. No ads, no third party fonts, no third party javascript, no tracking cookies, nothing. The economic model is that my website helps me get consulting gigs.
[3] I do agree with the general gist of what you’re trying to say: HTML, Javascript, and CSS have become too complicated for anything but the most highly funded of web browsers to render correctly. Both Opera and Microsoft have given up with trying to make a modern standards compliant browser, because the standards are constantly updating.
> Well, if it makes you feel any better, my website renders just fine on Lynx
It doesn't; I only use lynx when someone tricks apt-get into updating part of my graphics stack (xorg, video dirvers, window manager, etc) and researh is needed to figure out how to forcibly downgrade it, and then only because I can't use a proper browser without a working graphics stack.
> the general gist of what you're trying to say: HTML, Javascript, and CSS have become too complicated for anything but the most highly funded of web browsers to render correctly.
This is subtly but critically wrong; I am saying that it is necessary than web browsers do not render websites 'correctly'. The correct behaviour is to actively refuse to let websites specify hideous fonts, snoop on user viewing activity, or execute arbitrary malware on the local machine.
> Browsers with modern CSS support see [...] the serif font for body text.
Fair nitpick - "haven't noticed the difference yet" would be more accurate - but I don't see how that changes the argument; if I haven't noticed a difference, what's the point?
Most people have the means to cook for themselves without going anywhere and do so at least the vast majority of the time. Even if you do go to a restaurant, they almost always have menus rather than just making one dish for everyone since some people have styles of cooking that they prefer or don't like. People rarely design their own font but rather pick from professionally designed fonts. Additionally, at restaurants people pay for food so incentives are aligned while on the web people generally don't pay for content and any design professional involved is likely an advertiser. I rarely read stuff on the web for a design experience but for the content. I suspect most people would be unhappy with a newspaper that changed fonts for every story or a book that changed fonts every chapter.
Personally, I've been setting my browser to use only DejaVu fonts with a 16pt minimum for years (maybe a decade now) and every time I briefly use a default browser profile I notice the fonts and think not just "this is bad" but "how can people live like this?". Even with the usually minor issues that often appear, setting my own fonts is a way better experience than not doing so. My default experience is much closer to Firefox reader mode than it is to what the page specifies in most cases.
IMO, font speicification should be limited to serif, sans-serif, or monospace and let the user or browser set the actual font. Desingers should not rely on exact sizes of fonts or use custom icon fonts.
: "in a world where you're self-hosting all of your scripts."
Anything self-hosted is already fragile: it will go away when you don't continue to actively maintain it (paying for a domain, keeping a computer connected to the Internet etc.) or when you die.
I guess you can last 10 years, which is apparently what "This Page is Designed to Last" aspires to, but what if we have greater ambitions? Like 100 years?
You can design and mark-up content that will still be useful and readable in 100 years. You might be able to preserve the presentation logic (CSS-style) for 100 years.
You probably won't be able to preserve the interaction design for 100 years (without a dedicated effort -- that's why they bury computers along with the software in time capsules).
But I think it is optimistic to think that _most_ SasS hosts are going to archive content for 100 years. Preserving digital content is an _active_ process. It takes resources and requires deliberate effort.
Postscript: I'm trying to think of modern companies that would preserve content for 100 years, assuming they make it that for.
Facebook is the only significant current platform that I can even imagine preserving content for 100 years, but even that seems like s stretch. Historians might step in to archive it, but is there real value to Facebook to maintain and publish 50 year old comments on 2.5 billion unremarkable walls?
Twitter won't. Certainly Insta, SnapChat, WhatsApp etc. won't. Flickr probably could do it relatively easily but won't. YouTube maybe, but there's more to store. Something like GitHub maybe?
Right, look at SourceForge. There's a lot of broken links and/or references to no-longer accessible content in some of the older Apache.org projects too.
Also maybe cvs/svn/git repo generally don't contain content worth preserving for 100 years. There are some historically significant or interesting repos, but for the most part you'll have a bunch of unremarkable (and duplicated) code that may not have run then and certainly won't run now.
> a bunch of unremarkable (and duplicated) code that may not have run then and certainly won't run now
100 years is a long time, but I do run 20+ years old Common Lisp libraries and expect them to work without modification; I'd be really pissed if they disappeared from the Internet because someone thought that 5 years of inactivity means something doesn't work anymore.
If you have built something worth lasting 100 years, other people will help you ensure that it does. That reduces the concerns in this article considerably.
I didn't perceive the author to be doing any technical gatekeeping, quite the opposite. I feel like their article was targeted at people like me or others who use stuff like Hugo/Jekyll, or those who use free website builders or use large frameworks for simple websites.
I agree a couple of the points seem out of place (the monitoring service one made me laugh. visiting my website is the first thing I do after uploading a new page), but the intent of this article I wholeheartedly agree with:
Reduce dependencies, use 'dumb' solutions, and do a little ritualistic upkeep of your website to keep it around for a decade or more. The things you propose are the norm and the reason nothing sticks around, IMO.
> (the monitoring service one made me laugh. visiting my website is the first thing I do after uploading a new page)
I think what you want is not just monitoring your internal links, but also external ones - if a page you linked to in your article starts 404-ing or otherwise changes significantly, it's something you'd likely want to know about. That said, just like preferring GoAccess over Google Analytics, it's something I'd like to have running locally somewhere (on my server, or even on my desktop), instead of having to sign up to some third-party service.
> "Stick with the 13 web safe fonts" assumes that operating systems won't change. There used to be 3 web safe fonts. Use whatever typography you want, so long as you self host the woff files.
Indeed. 10 years ago, “font-family: Georgia, Serif” was guaranteed to work and look the same on pretty much all computers out there. Windows had all of the “web core” fonts (Georgia, Verdana, Trebuchet, Arial, even Comic Sans). Macintosh computers had all of the “web core” fonts. Even most Linux computers had them because it was legal to mirror, download, and install the files Microsoft distributed to make the fonts widely available.
In the last decade, Android has become a big player, and the above font stack with Georgia will look more like Bitstream Vera than it looks like Georgia on Android.
The only way to have a website have the same typography across computers and phones here in the soon-to-be 2020s is to supply the .woff files. Locally (because Google Webfonts might be offline some day). Either via base 64 in CSS or via multiple files; I prefer base 64 in CSS because sites are more responsive loading a single big text file than 4 or 5 webfont files. Not .woff2: Internet Explorer never got .woff2 support, and we can’t do try-woff2-then-woff CSS if using inline base64.
Even with very aggressive subsetting, and using the Zopfli TTF-to-WOFF converter to make the woff files as small as possible, this requires a single 116 kilobyte file to be loaded with my pages. But, it allows my entire website to look the same everywhere, and it allows my content to be viewed using 100% open source fonts.
Then again, for CJK (Asian scripts), webfonts become a good deal bigger; it takes about 10 megabytes for a good Chinese font. In that case, I don’t think it’s practical to include a .woff file; better to accept some variance in how the font will look from system to system.
Edit In terms of having a 10-year website, my website has been online for over 22 years. The trick is to treat webpages as <header with pointers to CSS><main content with reasonably simple HTML><footer closing all of the tags opened in the header> and to use scripts which convert text in to the fairly simple HTML my website uses for body content (the scripts can change, as long as the resulting HTML is reasonably constant). CSS makes it easy for me to tweak the look and fonts without having to change the HTML of every single page on my site, but as the site gets older, I am slowing decreasing how much I change how it looks.
I disagree with keeping fonts inline in the page. It means an additional 100kb per page at the very least. Which adds up very quickly. Remember that most of the world still doesn't have broadband (including yourself if you're using roaming services abroad). It also means extremely redundant information is transmitted when people watch more than one page on your site.
Exactly. It’s a CSS (not HTML) file with all the inline fonts in that file, with a long cache time, so all of the website’s fonts are loaded once for site visitors.
... wherein the main article content is, as the article suggests, a single, self-contained page.
And yet ... that is just the "tip" - underneath is:
" ... at least one, but possibly many, supporting documents, resources and services. The minimum requirement is simply an expanded form of the tip (the "bummock"), complete with references, notes and links. Other resources and services that might lie under the surface are a wiki, a changelog, a software repository, additional supporting articles and reference pages and even a discussion forum."
[1] Neither wiki nor forum exist yet, but the bummock does...
This is by and large, impossible. The hoops you have to jump through and downsides you have to endure are just a death by 10,000 cuts. Try writing a tabulated container in raw HTML and CSS that flex sizes and behaves nicely with the browser back button. Partial page reloads, containers with native (no reload!) sorting, there's so many features in modern web design that are just downright impossible without JS.
I am hesitant to put words in the author's mouth, but I suspect he would agree with my stance here...
The solution is to not require things like partial page reloads, etc. This shouldn't be a huge hardship -- a properly designed modern site can degrade gracefully anyway, so users that don't have or allow things like JS can still use it, even if in a "degraded" form.
I've put a bit of thought of what'll happen to my website if I were to die. It's hosted on GitHub Pages right now, so at some point GitHub is going to disappear or stop offering the service. Even before that, I honestly don't know what happens to renewing payments–I guess my Google Domains payment will stop and somebody will squat on my domain soon after? archive.org might be the only thing keeping the information around…I hope I've done a good job of making it archivable; as a matter of policy, there's no JavaScript. There's a snapshot from earlier this year and it looks fine, so maybe it'll outlast me?
It's specifically for open source software, but I wonder if it can be spun out into personal archives or websites.
---
Some years ago, I had an idea for a thought experiment called the Eternal Bit - various angles on the practicality of preserving a bit state "forever".
HTML is worth minifying just to get rid of comments. I don't want my "TODO(mullens): " comments to be visible to end users if possible. And gzip doesn't get rid of comments at all.
I do believe that, usually, you should make it possible to be archived. (In one of a few cases where it isn't, consider if HTTP(S) is even the correct protocol for what you are doing; sometimes it isn't.)
I agree that you can (almost always) write HTML without JavaScript codes (CSS is not always needed either, but nevertheless it can be helpful). This improves portability too. If you do want to use JavaScript to generate static documents, consider using Node.js and have it output a plain HTML document, which will then be the hosted document, rather than hosting the JavaScript version. (This way, the code only has to run once.)
About fonts, I think usually the fonts are not an essential part of the document. (Still, you don't usually need to use so many, but often you do not need to specify the font at all, except you may still need to specify if it is monospace, bold, etc.) Also, you do not usually need so many pictures on your web page (except picture galleries).
I do think if you need a copy of a web page then you can copy it.
You can also use plain text documents (without HTML); it is what I often do.
Regarding fonts, I think it's good practice to archive fonts together with your site's source, if the site relies on third party resources, like Google Fonts. (Switching the link target should be easy. Embedding a commented link to a stylesheet linking to these local resources along with the third party resource link may be of help.)
"monospace" is a property of a given font but is not (usually) a variant of a single font family as is "bold" (font weight).
While bold text is often used for emphasis and setting it to "regular" doesn't change the meaning much, using a monospace font signals specific things (often, it's used to represent code snippets, but in some contexts it can mean "work in progress"). Changing that font to a proportional one strips a lot of meaning and readability from the text.
In that regard, fonts families are an essential part of the document, and not just their stylistic/emphasis variants (e.g. size and weight, which can still convey important meaning on their own. Think titles and headings.)
Honestly, in almost all places that monospaced fonts are used, they’re purely a stylistic choice. (The remainder is mostly ASCII art, or plain-text representations of tables—in which case alignment is important, not the font—or something where tabular figures are desirable for number alignment.) I don’t say that that makes it useless in any way, but honestly monospaced text in code editing is overrated. You can live without it easily, and will probably get used to it very quickly.
I agree about monospaced text in code editing. I was thinking about uses such as code excerpts in literature (class names, inline bash commands, etc). Monospaced fonts[1] plays a major role in helping reading and understanding the content. This is more than a stylistic choice.
[1] actually the font doesn't need to be monospaced, but it must be very different than the font used for main copy.
While in some cases the font doesn't necessary need to be monospaced, some documents might require that a monospaced font is used, so it should be monospaced anyways. Although, as you say, different from the main copy; so if all of the text is monospaced, there should be some way to distinguish it, such as by selecting a different monospaced font, or displaying it with a different colour.
(One case where a monospace font is required is if you are automatically including text from another document which uses plain text format; there is no way to determine what format is needed, but monospace will always work.)
Even Glk, which has styles meant for meaning and use rather than formatting, nevertheless has a monospace style, since some things need to be displayed using a monospace font to be displayed correctly. However, you could just as well display all text using a monospace font, if you are using only one font, so you might not be able to "signal specific things" by the use of a monospace font.
For the case of headings, this can be done without specifying the font yourself; if you specify it is a heading, then the rendering software can decide how to display it. (HTML has <H1> and <H2> and so on for headings, and Glk has style_Header and style_Subheader.)
> Well, people may prefer to link to them since they have a promise of working in the future.
In a world where 95% of the google search results for <common problem> are forum threads, with everyone 'answering' the question by saying 'just google it, dumbass'[1], I don't think people - as in - the common homo sapiens - cares about the longevity of the content they link to as much as the author thinks.
Quality of the content (Does it have the information you want) is king. Longevity is a 'future me' problem, and 'future me' is incredibly shortsighted.
[1] Thanks, guys, how did you think I found this forum thread?
For this exact reason I've been pushing to use static HTML on my personal blogs and I spend a lot of time optimizing images for both retina and 1x resolutions. Basic HTML & CSS are good now, and there is no reason fairly basic content can't be stored as static pages.
You can convert PHP or other dynamic pages to static fairly easily, it takes about half a day of scripting and a little time with configuring your web server. The pages load quicker and the load on your server is lower.
I've long considered the idea of a permanent hosting solution, where people pay to get their content up and I guarantee it's longevity, but the business model on that is tough.
FYI. The content looks real nice in "reader" mode (at least in Firefox).
I kind of think this is the best approach. Write your content with as minimal markup as possible. Let the user-agent render the page in my own styling.
I wish the web was a bit more like the gopher protocol, where the page markup was very minimal. Just give me the plain text (or close to it) and let my client render it the way I want to read it.
Markdown (or similar) would be an ideal representational format. This would push all the rendering decisions to the client.
Even better (IMO, because that’s what I do) is to provide your own markup that is reasonable, but let the underlying semantic HTML be there too so people can discard it and use their browser’s reader mode to view it too.
> On my large screen, the lines are so long that it's difficult to read.
The text on the page adapts to the width of the browser window in which it is viewed. The result is that everyone can obtain the line width they prefer by adjusting the width of their local browser that is rendering the page.
If the lines are too long, just pop the page out into its own window, narrow the width of just that window until the lines are at a comfortable length for you, and you'll have lines that are just right for you. This page is an example of where you are in control of how the site renders and can easily adjust it to your own tastes.
With the exceptions of my root index.html generated by the tree command and the subdirectory listings handled by apache, my entire site is plain .txt files. Should be good for a while...
Kinda sounds like you don't link to any other content, though (I guess you can include links in plain text, but that's a bit of a pain for your readers) which means your site doesn't really fit the use case.
Yes, it's a dead end, a cul de sac that links to nowhere. And that's exactly what I intended. But if you see something that is of further interest, highlight it and search. That will almost always produce topical results...
I was thinking about this exact subject matter the other day. We invest so much time in our digital content but what happens with death? How long can we really expect our content to last?
Ultimately we are all dependent on the functionality of the internet and making a physical copy is the best way to make something last. However if we intend to not only have it last but be accessible, some of the suggestions are helpful but the most significant is: where can I host something indefinitely?
Will Netlify, Github Pages, AWS, etc all be around in 50 years? 100 years? 500 years? Heck, the internet hasn't even been around for all that long.
As I write this, my only thought is that you need a system of fallbacks. Frankly, this seems like a business opportunity in which the cost of the infrastructure is purchased based on some formula. DNS is configured programmatically. File locations are distributed and redundant. I'm not sure what the best approach is but one thing is certain: the page accessibility is the least of this author's issues. He just needs the site to be hosted...
I was expecting the solution to be mirror your generated pages on IPFS (https://ipfs.io), so they just don't go away at all (as long as someone has them pinned).
The proposed solution set seems extremely convoluted and don't actually solve the issue.
That’s quite the caveat, and speaking as somebody who has attempted it, you’ve introduced quite a bit of complexity. The whole point is that complexity militates against keeping it online. Keeping it simple, the author’s theory seems to go, is the single most effective way to make something likely to be able to be available long term.
None of what he wrote mitigates the case where he no longer maintains his site and stops paying for the server, which is the very reason all those links he mentioned were dead (defunct websites/hosts).
The article was about not having to maintain a site for non techies like us. WE can happily maintain a range of things, but a chemistry doctoral student will hit a hurdle and may give up.
Paying for a server - mentioned in the article about the provider changing access.
You might keep the content format mighty simple, but keep in mind that the hosting and delivery mechanism consisting of http/html are extremely demanding. In particular, your website's availability is fundamentally limited by the uptime of your host machine (which you are only leasing) and its HTTP server and network. If you forget to renew the lease on your host machine, or there is some mishap by the service provider, your website will vanish without a trace -- even if it happened to have a high rate of visitors.
I find it an interesting social phenomenon that the finite nature of content on the Web is lamented, when content has been temporal for all of human history.
Carved stone is a bit more durable that that nice Myspace page was, however. Yes it was not widespread and was difficult to setup but millennia of duration is nice.
I think that the medium, be it a hosting site or you own domain, will fail. What we need is maybe : a naming scheme that can transfer to different people (replicated files on different servers indexed by content hash? Aka anonymous ftp mirrors ... ) and common media ie readable by anyone, gracefully degrading.
HMTL was a bad idea that just kept sprouting cancerous growths on top of itself. Just like the author recommends not using non-web fonts, we honestly don't need much markup beyond "monospace", "em", "strong", "title 1-5", and "p". Everything else HTML does is a huge distraction.
In the past thirty years, there have only been a few actual kinds of content:
- Article
- Comment
- Reference manual
Nothing really deviates from this. If it does, then it's probably in the "application" domain and deserves all that javascript stuff.
I imagine a new semantic web inspired content type that bakes in permanent content URLs, author signatures, and supports sharing P2P and rich archival:
<article>
<id>
(Computed immutable content hash and signature. Subsequent revisions
will invalidate this and require new content hashes. Links can be made
to the old version.)
</id>
<authors>
(metadata, homepage, public key)
</authors>
<contents>
<title> Semantic Web was actually brilliant </title>
<body>
<h1>The Web was Embraced, Extended, and Extinguished</h1>
<p>Lorem ipsum dolor sit amet...</p>
</body>
</contents>
</article>
It can be a really simple model. No divs or presentational CSS or anything. (Well, maybe some support for LaTeX and I18N.) The client decides how it looks, which is how it always should have been.
If every entity in the universe has an ID, they can link to each other in a distributed fashion. It won't matter if the original source URL goes down as long as someone somewhere has a cache.
It's also probably better to binary encode these (protobuf or something). Images and media could be inlined instead of href'd.
Instead of building on the layer cake, we should take a moment to reflect on what we're trying to do. We have a lot of history and bad evolution that we can garbage collect and streamline.
If our objective is to publish and share content, a lightweight version of the semantic web is the way to go.
Oh, and check out my follow-up to the author:
<article>
<id> (my own post's content hash) </id>
<parent> (the author I'm responding to) </parent>
<author> me </author>
<contents>
Hey you! I disagree!
</contents>
</article>
Again, these can be distributed on a web. It can be p2p, federated, or live atop the classical WWW. It doesn't matter. The clients can richly use the semantic data model.
The anger for disappearing content, walled gardens, and AMP will eventually reach a boil. As technologies go, the pendulum is always swinging. This is the direction we'll eventually head back to.
I agree, it is correct, we do not need much markup other than the five you specified, plus hyperlinks; those are good enough. Yes, the client should decides how it looks.
I'd include lists and tables for data presentation. Also, we may want some (additional) tags to manage footnotes and references (these were in SGMLguid, but didn't make it into HTML.)
You could just link to them I think, possibly with a hint to specify inlining; the client decides whether to obey that hint or to ignore that hint. (This would be done the same whether the picture is part of the document or is a separate file, I should think. It makes many considerations easier to work with.)
I guess, the Netscape Navigator 1.0 specs (tables and forms, but before frames and JS) are quite what we may want (with the possible exception of the font tag and, certainly, without the blink tag), augmented by some footnote/reference tags and a reduced set of CSS.
It's one thing to dream up a minimalist markup language that ought to be enough for everybody, but it's a whole different problem to preserve the large body of existing content out there. Actually, SGML (from 1986, and on which HTML is based) has the capability to render/transform your mini markup language to HTML, and also render markdown and other custom Wiki syntax.
457 comments
[ 3.2 ms ] story [ 308 ms ] threadBut it's nice to know you're not alone in wanting nice things to last :)
Publication models where you pay the cost for somebody to be the archival reference make some sense. URI pointing into a store. Buckets in Google? But if you decided to move on, could you get a retained 302 redirect to point to the new home?
Imagine sticking a proxy between our browser and the internet that automatically archives all webpages we attempt to browse to, and then only lets you view the archive. How much of the internet would you be able to see?
Heck, with the cost of storage so low, recording every webpage you ever visit in searchable format is also very realistic. Imagine having the last 30 years of web browsing history saved on your local machine. This would especially be useful when in research mode and deep diving a topic.
[1] https://github.com/machawk1/warcreate
[2] https://github.com/machawk1/wail
[3] https://github.com/internetarchive/warcprox
EDIT: I forgot to mention https://github.com/webrecorder/webrecorder (the best general purpose web recorder application I have used during my previous research into archiving personal web usage)
Edit: it can also auto-save pages, like SingleFile.
I still keep an old ESR with this plugin for archiving, and accessing MHTs.
I am currently dealing with the problem of parsing large mht files (several megabytes and up). A regular web browser would hang and crash upon opening these files and most ready made tools I could find struggle with the number of embedded images. It's very much a neglected format with very little support in 2019.
WARC as a format seems promising, but at least last I checked, open-source tooling to make it a pleasant and/or transparent experience is not really there, and worse, at least as of several months ago, doesn't really seem actively worked on. Definitely an area you'd expect to be further along.
Indeed.
And I still remember the modem days where I would download entire websites because the ISP charged by the hour, and I'd read them offline to save money.
I think this says something kind of profound about information and capitalism and whatnot.
This could be done in better and more specialized ways, one problem is browser extension APIs don't provide very good access to the browser's webpage-saving features.
https://github.com/gildas-lormeau/SingleFile
https://github.com/gildas-lormeau/SingleFile/issues/320
> SingleFileZ is a fork of SingleFile that allows you to save a webpage as a self-extracting HTML file. This HTML file is also a valid ZIP file which contains the resources (images, fonts, stylesheets and frames) of the saved page.
https://github.com/gildas-lormeau/SingleFileZ
(†): short of direct plist manipulation
[1] https://web.archive.org/web/20151229082536/http://blog.bonar...
[2] https://bugs.chromium.org/p/chromium/issues/detail?id=500239
I need to do a handful of experiments to see exactly how this interacts with Sync, even though I've (foolishly) already synced the important profiles. I'd hope that the cloud copies of the places database just keeps growing, but in any case, I'd rather combine them all offline anyway.
Anyway, thanks for the guide.
[1] https://github.com/crazy-max/firefox-history-merger
Since starting with replacing bookmarks, I've moved other forms of reference info in there, and now have a whole GTD setup there as well, which is extremely handy since I can search in one place for reference info and personal tasks (past and future). Only downside is I'm dependent on Evernote, but hopefully it manages to stick around in some form for a good while, and if it ever doesn't, I expect I'll be able to migrate to something similar.
I was an Evernote user when I was on macOS. When I switched to Linux, a proper web clipper was something I really missed. I'm now on Joplin and it does everything I used to use Evernote for and then some.
It even has vim bindings now!
As far as longevity goes, I think they got their archive / backup format right - it's just a tarball with markdown in it.
Also people need to move away from those esoteric reactjs, angular, vuejs and plethora of CMS as API or static site generators relying on some js framework which won't last even 2-3 years. Use a static site generator which can generate a plain html, like static site generators built on pandoc, python docutils or similar.
Personally I like restructuredText as the preferred format for content as its a complete specification and plain text. So the only thing in this article I will change is that content can also be in rst format and then generate html from it. Markdown is not a specification as each site implements their own markdown directives unlike restructuredtext specification and most of the parsers and tooling are little different from each other.
Not by that name... https://commonmark.org/
If you are in restructuredText world there is one specification and all implementation adhere to it, be it pandoc, sphinx, pelican, nikola. The beauty of it is that it has extension mechanisms which provides enough room for each tool to develop it. But markup can be parsed by any tool.
[1] https://docutils.sourceforge.io/docs/ref/rst/restructuredtex...
It's better than "designed by a committee" standards, but it lacks elegance or maybe craftsmanship.
You don't really get bit by its lack of a standard and extensibility until after you've bought in.
It's essentially designed by the opposite of a committee -- rather than including everything but the kitchen-sink, it contains support for almost no usecases except the one. Which is very appealing, when you only have the one usecase.
So markdown needs to thank the popularity of Wikipedia for its success, as rst did not have any application like Wikipedia. But still rst is used widely enough with its killer Sphinx, readthedocs and now its kind of de-facto documentation writing markup in Python and many open source software world.
Joplin is free and open source.
I have used rst intensively on a project. A few years later, I would be hard pressed to write anything in ti and would need to start with a Quick Start tutorial. With all its faults, Markdown is simple enough that it can be (and is) used anywhere, so there is no danger of me forgetting its syntax (even if it wasn't much simpler to star with).
So personally I would prefer md over rst anytime.
Joplin is open source, which is a big part of the sell to me. It definitely isn't the best of all possible note taking systems that could ever exist, but it's the best open source one I've found so far, and I don't have time to write a better one at the moment.
> why not build it into browsers. I have seen Firefox and Chrome can download web pages. So it will be nicer if they can download the bookmarked pages and store in a local html, css, image folder. I think it's pretty easy to achieve
This is solving a different problem though. WARC/MHT and other solutions can do this. Joplin is more of a note taking system that allows ingesting content from the web into one's own local notebook, which is relevant to what the GP post was talking about - Evernote.
However, it would seem that "the modern web" is the now popular standard. 10 years ago it might have been Flash or Java web applets or whatever. Now it's JS. I'm not convinced that JS is any better than what it has replaced. However, people keep paying developers to write them, so presumably someone likes them.
> Also people need to move away from those esoteric reactjs, angular, vuejs and plethora of CMS as API or static site generators relying on some js framework which won't last even 2-3 years. Use a static site generator which can generate a plain html, like static site generators built on pandoc, python docutils or similar.
Agreed, but that's also not a problem that Joplin, Evernote, or any other such tool is going to be able to solve. Unless you are complaining that Joplin is an Electron app? That's my biggest issue with it personally. It runs well enough, but is definitely the heaviest application I use regularly, which is a little sad for a note taking program. On the other hand, I haven't found a better open source replacement for _Evernote_. There are lots of other open source note-taking programs though.
> Personally I like restructuredText as the preferred format for content as its a complete specification and plain text. So the only thing in this article I will change is that content can also be in rst format and then generate html from it. Markdown is not a specification as each site implements their own markdown directives unlike restructuredtext specification and most of the parsers and tooling are little different from each other.
reST is indeed very nice. At one point, I kept my personal notes as a Sphinx wiki with everything stored in reST. I found this to be less ergonomic than Evernote/Joplin, although in principle it could do all the same things that Joplin can do, and then some.
Safari does this. pages added to the reading list archive the content for offline reading.
> Only downside is I'm dependent on Evernote
No special software nor database required.
2. it works with any browser
3. I can move it to any machine and use it
4. It is not transmitted to the browser vendor
5. Being a text file, it is under my control
6. I can back it up
7. I don't need some database manager to access it
8. I can add notes and anything else to the file
9. It's stupid simple
I had recently participated in a discussion on the problem of forgetting bookmarks[1].
Copying my workflow from there,
1. If the entire content should be easily viewed, then store via pocket extension.
2. If a partial content should be easily viewed i.e. some snippet with link to entire source, then store in notes (Apple).
3. If the content seem useful in the future, but it is okay to forget it; then I store it in the browser bookmarks.
But, my workflow doesn't address the problem raised by Mr. Jeff Huang; if Pocket app or notes disappear so goes my archives. I think self hosted archive as mentioned by the parent is the way to go, but I don't think it's a seamless solution to a common web browser user.
[1]https://needgap.com/problems/57-i-forget-my-web-bookmarks-qu...
I frequently see something and want to try it out the next time I want to do something else. So I emulate User Agent strings and append lots of "like [common thing I search for a lot]" to the bookmark. When I start typing into the search bar for those other things I'll be reminded of the bookmark.
For example, since file.io is semi-deprecated I decided to try out 0x0.st . But I kept forgetting when I actually needed to transfer a file, so I made a bookmark titled "0x0.st Like file io".
As a side note, I have a similar bash function called mean2use that I use to define aliases that wrap a command and ask me if I'd like to do it another way instead or if I'm sure I want to use the command. I've found this is a nice way to retrain my habits.
Disclaimer: needgap is a problem validation platform I built.
It is true that it is propietary software but it is worth mentioning that all the content can be exported as an .enex file, which is xml.
So, the data can be easily exported.
Have you actually looked at such an xml: https://gist.github.com/evernotegists/6116886
Exported sure, it's all there. But importing that into your new favorite notes application is not going to be trivial, especially not for regular users.
That's why I've decided to stick mostly to regular files in a filesystem.
Joplin, for example, can import notes exported from Evernote. It's just a menu option that even regular users should have no trouble employing.
You bookmark in Pinboard or Delicious and ArchiveBox saves the page. Handy.
I have used Evernote and OneNote, but have finally, after a long interim period, resorted to using only markdown.
I have a "Notes" root folder and organize section groups and sections in subfolders. VSCode (or Emacs), with some tweaks, shortcuts, and extensions, provides a good-enough markdown editing experience. Like an extension that allows you to paste in images, storing it in a resources folder in the note's current location (yes, I see small problems with this down the road when re-organizing, but nothing that can't be handled).
For Firefox, I use the markdown-clipper extension the few times I would like to copy a whole article, it works well enough. Or I copy/paste what I need; mostly, I take my own summarized notes.
For syncing, I use Nextcloud, which also makes the notes available both for reading and editing on Android and iOS (I use both).
Up until very recently, I used Joplin, which also uses markdown, but there were two things I could not live with: it does not store the markdown files with a readable filename, e.g., its title, and being tied to a specific editor.
If you are mostly clipping and not writing your own notes, I can imagine my setup won't work well, or be very efficient.
I want to use a format that has longevity, and storing in a format that I cannot grep is out of the question.
https://addons.mozilla.org/en-US/firefox/addon/markdown-clip...
I know about DevonThink, i read good recommendations. But it's IOS/Mac only.
Any Evernote alternative for Win/Android with great search ?
So simple! Thank you!
Edit: doesn't appear to be down, they're just using a self-signed cert.
Everyone is free to fork a browser and apply any changes they want. Allowing extensions to change anything at all essentially is the same as forking and merging your changes with upstream every update.
I guess “with a reasonably stable hook api” was supposed to be implicit in my statement.
It's better than nothing but it's also increasingly frustrating to deal with.
Not useful to me personally, but useful to someone!
[0] https://apse.io
That was in 1989 and today I mostly search my computer using find and grep commands, since that's what just keeps working.
https://en.m.wikipedia.org/wiki/Spotlight_(software)
I believe the name for that experience was/is "Microsoft Windows".
http://shelve.sourceforge.net/
The main disadvantage was disk space. This is particularly true when some pages are 10 MB or larger. I would periodically prune the archive directory for this reason.
I stopped using Shelve when I started running out of disk space, and now I can't use Shelve because the addon is no longer supported. The author of Shelve has some suggestions for software with similar functionality:
https://groups.google.com/forum/#!topic/shelve-firefox-addon...
I used Scrapbook in the past (which also still works) but I usually just save random things in ~/webpages/ since (apparently) 2011. The earliest is a copy of the landing page at bigthink.com. Of course now almost every link is broken, excluding social media buttons, About Us, Contact Us, RSS, Privacy, Terms of Use, Login, and the header logo pointing at the same page.
https://pinboard.in/
Just pay the yearly subscription so pinboard can cache your bookmarks.
The bus factor is high, but I suspect that Maciej has a plan that'll let us download our archive even if he does get grabbed by the mainland Chinese government let alone a forecasted going out of business action.
https://www.zotero.org/
Most websites I used to visit during demoscene high days, are now gone.
Or in some random script: "from iridb import iri_index" "data = iri_index.get('https://some/url')" I'm skipping lots, you can ref by hash, url, url+timestamp. It hands back a fh, you dont know if the data you are reffing even fits in memory. Extensive caching, all the iri/url quirks, punycode, PSL etc.
Some random pdf in ~/Downloads, "import doc.pdf" and dmenu pops up, you type a tag, hit enter and the pdf disappears into the hash tree, tagged, and you never need to remember where you put it. Later on you only need to remember part of the tag, and a tag is just a sequence of unicode words.
Chunks are on my github (jakeogh/uhashfs, it's heka-outdated dont use it yet), I'll be posting the full thing sometime soonish.
I tend to do that, I also save a lot of scientific papers, ebooks and personal notes. I've found that doing so does not help me at all. The main problem I have is that when I need to look something up (an article, a book, a bit of info) I reach for google first, usually end up finding the answer and go to save it, only to find that I had already found the answer beforehand (and perhaps already made clarifying notes to go along with it) and then forgot about it.
This, and not dead links, is the fundamental problem with bookmarks for me. Not only bookmarks, it extends to my physical notes and pretty much everything I do. If I haven't actively worked on something for a couple of months, I forget all about it and when I come back to it I usually have to start from scratch until I (hopefully) refresh my memory. Some of it is also usually outdated information.
I think this is a big, unsolved problem and I'm not even sure how to go about starting to solve it. I can envision some form of AI-powered research assistant, but only in abstract terms. I can't envision how it would actually work to make my life better or easier. It would need to be something that would help blur the line between things I know and things that are on my computer somehow. If I think of my brain like it has RAM and cache, things I'm working on right now are in the cache and things I've worked on recently or work on a lot are in RAM, but what's for me lacking is a way to easily move knowledge from my brain-RAM to long term storage and then move that knowledge back into working memory faster than I can do so now. I'm not even talking about brain uploading or mind-machine interfaces, but just something that can remind me of things I already know but forgot about faster than I can do so by myself.
I am convinced that figuring out how to do this will lead to the next leap in technological development speed and efficiency. Not quite the singularity that transhumanists like to talk about, but a substantial advancement.
What I've found is that I need to spend more time deciding what is important, and less time consuming frivolous information. That's hardly a technology problem.
For things I really don't want to forget, I'm using Anki [0], a Spaced Repetition System (SRS). Anki is supremely good at knowing when you're about to forget an item and prompting you to review it.
Spaced practice and retrieval practice, both of which are used in SRS, are two learning techniques for which there is ample evidence that they actually work [1].
You still need to decide what is worth remembering, but that's something technology can't help with, I think.
[0] https://apps.ankiweb.net/
[1] https://www.learningscientists.org/
It requires Python 3.8 (just the stdlib) and wget.
I have 3633 bookmarks, for a total of 1.5 Go unziped, 1.0 Go zipped (and we know we can get more from better algo and using links to files with the same checksum like for JS and css deps).
This seems acceptable IMO, espacially since I used to consider myself a heavy bookmarker and I was stunned by how few I actually had and how little disk they occupied. Here are the types of the files:
It should probably be opt-in though, like a double click on the "save as bookmark icon" to download the entire file, and the star becomes a different color. Mobile phones, chrome books and raspy may not want to use the spaces, not to mention there are some bookmark content that you don't want your OS to index, and show you preview of in every search.But it would be fantastic: by doing this experiment I noticed that many bookmarks were 404 now, and I will never get their content back. Beside, searching bookmark, and referencing them is a huge pain.
So definitely something I wish mozilla would consider.
There used to be this neat little extension called Read It Later that let you do just that. Bookmark and save it so you could read it when you were offline or the page disappeared. Later they changed their name and much later Mozilla bought it and added it to Firefox without a way to opt out. It was renamed to Pocket.
Bookmark integration would mean one software, with the same UI, on every platform, and only one listing for your whole archive system.
There are a few issues to consider:
- Any comprehensive archive of your activity is itself going to be a tremendously "interesting" resource for others -- advertisers, law enforcement, business adversaries, and the like. Baking in strong crypto and privacy protections from the start would be exceedingly strongly advised.
- That's also an excellent reason to have this outside the browser, by default, or otherwise sandboxed.
- Back when I was foolish enough to think that making suggestions to Browser Monopoly #1 was remotely useful, I pointed out that the ability to search within the set of pages I currently have open or have visited would be immensely useful. It's (generally) a smaller set than the entire Web, and comprises a set of at least putatively known, familiar, and/or vetted references. I may as well have been writing in Linear A.
- Context of references matters a lot to me. A reason I have a huge number of tabs open, in Firefox, using Tree-Style Tabs, is that the arrangement and relationships between tabs (and windows) is itself significant information. This is of course entirely lost in traditional bookmarks.
- A classification language for categorising documents would be useful. I've been looking at various of these, including the Library of Congress Subject Headings. A way of automatically mapping 1-6 high-probability subjects to a given reference would be good, as well as, of course, tools for mapping between these.
- I've an increasing difference of opinion with the Internet Archive over both the utility and ultimately advisability of saving Web content in precisely the format originally published. Often this is fragile and idiosyncratic. Upconverting to a standardised representation -- say, a strictly semantic, minimal-complexity HTML5, Markdown, or LaTeX, is often superior. Both have their place.
On that last, I've been continuing to play with the suggestion a few days ago for a simplified Washington Post article scrubber, and now have a suite of simple scripts which read both WashPo articles and the homepage, fetching links from the homepage for local viewing. These tend to reduce the total page size to about 3-5% of the original, are easier to read than the source, and are much more robust.
I'm reading HN at the moment from w3m (which means I've got vim as my comment editor, yay!), and have found that passing the source to pandoc and regenerating HTML from that (scrubbing some elements) is actually much preferable, for the homepage. Discussion pages are ... more difficult to process, and the default view in w3m is unpleasant, though vaguely usable.
Upshot: saving a WARC strictly for archival purposes is probably useful, but generating useful formats as noted above would be generally preferable in addition.
With the increasing untenability of mainstream Web design and practices, a Rococco catastrophe of mainstream browsers, the emergence of lightweight and alternative browsers and user-agents (though many based on mainstream rendering engines), the tyranny of the minimum viable user attacking any level of online informational access beyond simple push-stream based consumption, and more, it seems that at the very least there's a strongly favourable environment to rethinking what the Web is and what access methods it should support. Peaks in technological complexity tend to lead to a recapitulation phase in which former, simpler, ideas are resurrected, returned to, and become the basis of further development.
(I believe that for historical purposes, enough complaining about ads and tracking will survive that future historians can easily deduce the existence of this practice)
The point is to make a webpage that lasts. So people can link to it and get the page. That means making a maintainable webpage and a url that does not change.
It is great that you can archive every page you visit for yourself, but that is not the same as making a lasting web.
Lets make something that others can use too.
A hash can represent any resource.
IPFS, or something like it, is the missing component for long term retrieval.
EDIT: or something like https://github.com/google/trillian
Filecoin is just for incentives, but IPFS has been running for years without it.
Otherwise you must trust that an archiver hasn't changed history. Why trust when you can verify?
As another example, I keep every line of code I write (that wasn't written for an employer) forever. I often pull up code that I've written decades ago to use in new projects.
The beauty of the internet is that the old and the new aren't mutually exclusive -- there's plenty of room for it all.
"The more libraries incorporated into the website, the more fragile it becomes" is just fundamentally untrue in a world where you're self-hosting all of your scripts.
"Prefer one page over several" is diametrically opposed to the hypertext model. Please don't do this.
"Stick with the 13 web safe fonts" assumes that operating systems won't change. There used to be 3 web safe fonts. Use whatever typography you want, so long as you self host the woff files.
"Eliminate the broken URL risk" by... signing up for two monitoring services? Why?
I think this list of suggestions does a great disservice to people who just want to be able to post their thoughts somewhere. There's an assumption here that you'll need to be technically capable in order to create a page "designed to last" and frankly that is not what the internet is about. Yes, Geocities went away. Yes, Twitter and Facebook and even HN will go away. But the answer sure as hell isn't "I teach my students to push websites to Heroku, and publish portfolios on Wix" because that is setting up technical gatekeeping that is completely unnecessary.
If you're worried about fonts changing out from under a site you should surely also be worried about bitrot in, say, jQuery.
- CSS Stylesheets: 3
- Animated gifs: 1
- Individual JS files: 11 (around 2MB of JS decompressed (but not un-minimized))
- Asynchronous Requests: 14 (and counting)
And that's with uBlock Origin blocking 12 different ad requests.
That's not simple in any form. So, the possibility of something on this page breaking? High. There's a lot of surface area for things to break over time. And that's not counting what happens when the NPR's internal APIs change for those asynchronous requests.
-
In my case it also has the added benefit of being able to cache JS for a long(er) period of time, with users only having to download maybe 0-30kb of JS when only 1 component is updated instead of invalidating the entire JS served (Way under 1MB however)
There are more problems though. older library versions might be vulnerable to XSS attacks, or use features removed by browsers in the future for security reasons (eval?). Or you might want to change something involving how you use the API but the docs are long gone. Generally, libraries imply complexity and when it comes to reliability, complexity will always be your enemy.
I have run into this problem trying to migrate very old web pages or blog posts off of SaaS sites that are shutting down or just decaying. It's not just that complicated sites make it difficult to extract the content in the first place; it's difficult to publish that content on another site in a high-fidelity, and sometimes even readable, way.
The hard part isn't keeping the old site (page) running (although that's not always easy either). The hard part is when you want to do something _else_ with that content -- more complicated means less (easily) flexible.
or use Google Web fonts, and set let last option in your font-family to be "serif" or "sans-serif" to let an appropriate typeface be used if your third-party font is unreachable. That's the beauty of text, the content should still be readable even if your desired font is unavailable.
For typical "the words matter more than how the words look" content...can someone explain to me why we care about including the font?
See here: https://news.ycombinator.com/item?id=21841011
There’s also layout issues caused when replacing a font with another font, unless the metrics are precisely duplicated. There’s a reason RedHat paid a lot of money to have Liberation Sans with the exact same metrics as Arial, Liberation Serif have the same metrics as Times New Roman, and Liberation Mono have the same metrics as Courier New.
Google Fonts also isn’t blocked but I recall it being hit-and-miss in terms of responsiveness when I was working on a website that targeted Chinese audience a few years ago. However, I just tried resolving fonts.googleapis.com and fonts.gstatic.com on a Chinese server of mine, and they both resolve to a couple of Beijing IP addresses belonging to AS24424 Beijing Gu Xiang Information Technology Co. Ltd., so it’s probably very much usable now.
I do occasional web dev from within China and had to eliminate external references to get manageable page load times. At least from where I work pulling in practically anything from outside the Great Firewall will have a high probability of killing page load time. Anything hosted by Google in particular will often have you staring at your screen for 30 seconds.
For myself, I wish that people would leave Arial, Verdana, Helvetica Neue, Helvetica, &c. out of their sans-serif stack, having only their one preferred font and sans-serif, or better still sans-serif alone; but as a developer I understand exactly why they do it all.
The users who actually know how to change the default font also know how to use stylish.
Telling him to back off and let you cook because he can't know better than you (his user) would be absurd.
Same thing with design and typography. It requires skill and taste, and hopefully people will be delighted or simply consume the content for what it is, because the design/cooking just reveals that content in a convenient/useful shape.
I think most fonts that get your attention suck, the best ones are invisible and get you directly to the meaning of text, without getting in the way. So maybe there's a kind of bias (selection or sampling bias?) operating here?
These days, people either use their social network’s unchangeable CSS, or they use a Wordpress theme with an attractive and perfectly readable font. Even Merriweather, which I personally don’t care for, is easy enough to read.
The only time I have seen a page use obnoxious fonts in the 2010s is when the LibreSSL webpage used Comic Sans as a joke to highlight that the project could use more money:
https://web.archive.org/web/20140422115234/http://www.libres...
Edit It may be a case that the parent poster likes using a delta hinted font, either Verdana or Georgia, on a low resolution monitor, and doesn’t like the blurry look of an anti-aliased font on a 75dpi screen.
Indeed, typography is a skill. Most designers should have it though, which is why I asked more information to OP.
> The only time I have seen a page use obnoxious fonts in the 2010s is when the LibreSSL webpage used Comic Sans as a joke to highlight that the project could use more money
Ah, the infamous Comic Sans. It's a shame because as a typeface on its own, in its category, it is pretty good. Sadly, it's misused all the time in contexts where it's not appropriate at all.
> It may be a case that the parent poster likes using a delta hinted font, either Verdana or Georgia, on a low resolution monitor, and doesn’t like the blurry look of an anti-aliased font on a 75dpi screen.
Without more details we cannot guess. You're right: a lot of things can go wrong and ruin a typeface, regardless of how the characters are designed. Anti-protip: a reliable way to make any font look like shit is to keep the character drawings as they are and mess up the tracking (letter-spacing) and kerning.
> fonts that get your attention suck
If I can tell the difference between your font and the system default font, your font sucks; if I can't tell the difference, what's the damned point?
The web standards allow a website to use any WOFF (or WOFF2) font they wish to use. Please see https://www.w3.org/TR/css-fonts-3/
[1] There are some rendering issues with Dillo, with made the mistake of trying to support CSS without going all the way, making sure that http://acid2.acidtests.org renders a smiley face, but even here I made sure the site still can be read.
[2] Also, no cookies used on my website. No ads, no third party fonts, no third party javascript, no tracking cookies, nothing. The economic model is that my website helps me get consulting gigs.
[3] I do agree with the general gist of what you’re trying to say: HTML, Javascript, and CSS have become too complicated for anything but the most highly funded of web browsers to render correctly. Both Opera and Microsoft have given up with trying to make a modern standards compliant browser, because the standards are constantly updating.
It doesn't; I only use lynx when someone tricks apt-get into updating part of my graphics stack (xorg, video dirvers, window manager, etc) and researh is needed to figure out how to forcibly downgrade it, and then only because I can't use a proper browser without a working graphics stack.
> the general gist of what you're trying to say: HTML, Javascript, and CSS have become too complicated for anything but the most highly funded of web browsers to render correctly.
This is subtly but critically wrong; I am saying that it is necessary than web browsers do not render websites 'correctly'. The correct behaviour is to actively refuse to let websites specify hideous fonts, snoop on user viewing activity, or execute arbitrary malware on the local machine.
> Browsers with modern CSS support see [...] the serif font for body text.
My point exactly.
Personally, I've been setting my browser to use only DejaVu fonts with a 16pt minimum for years (maybe a decade now) and every time I briefly use a default browser profile I notice the fonts and think not just "this is bad" but "how can people live like this?". Even with the usually minor issues that often appear, setting my own fonts is a way better experience than not doing so. My default experience is much closer to Firefox reader mode than it is to what the page specifies in most cases.
IMO, font speicification should be limited to serif, sans-serif, or monospace and let the user or browser set the actual font. Desingers should not rely on exact sizes of fonts or use custom icon fonts.
Anything self-hosted is already fragile: it will go away when you don't continue to actively maintain it (paying for a domain, keeping a computer connected to the Internet etc.) or when you die.
You can design and mark-up content that will still be useful and readable in 100 years. You might be able to preserve the presentation logic (CSS-style) for 100 years.
You probably won't be able to preserve the interaction design for 100 years (without a dedicated effort -- that's why they bury computers along with the software in time capsules).
But I think it is optimistic to think that _most_ SasS hosts are going to archive content for 100 years. Preserving digital content is an _active_ process. It takes resources and requires deliberate effort.
Postscript: I'm trying to think of modern companies that would preserve content for 100 years, assuming they make it that for.
Facebook is the only significant current platform that I can even imagine preserving content for 100 years, but even that seems like s stretch. Historians might step in to archive it, but is there real value to Facebook to maintain and publish 50 year old comments on 2.5 billion unremarkable walls?
Twitter won't. Certainly Insta, SnapChat, WhatsApp etc. won't. Flickr probably could do it relatively easily but won't. YouTube maybe, but there's more to store. Something like GitHub maybe?
I can see a deletion heuristic that considers both account activity and repo activity being deployed within the next 10 years.
However I expect another evolution in SCM in the same timeframe.
Also maybe cvs/svn/git repo generally don't contain content worth preserving for 100 years. There are some historically significant or interesting repos, but for the most part you'll have a bunch of unremarkable (and duplicated) code that may not have run then and certainly won't run now.
100 years is a long time, but I do run 20+ years old Common Lisp libraries and expect them to work without modification; I'd be really pissed if they disappeared from the Internet because someone thought that 5 years of inactivity means something doesn't work anymore.
I agree a couple of the points seem out of place (the monitoring service one made me laugh. visiting my website is the first thing I do after uploading a new page), but the intent of this article I wholeheartedly agree with:
Reduce dependencies, use 'dumb' solutions, and do a little ritualistic upkeep of your website to keep it around for a decade or more. The things you propose are the norm and the reason nothing sticks around, IMO.
I think what you want is not just monitoring your internal links, but also external ones - if a page you linked to in your article starts 404-ing or otherwise changes significantly, it's something you'd likely want to know about. That said, just like preferring GoAccess over Google Analytics, it's something I'd like to have running locally somewhere (on my server, or even on my desktop), instead of having to sign up to some third-party service.
Indeed. 10 years ago, “font-family: Georgia, Serif” was guaranteed to work and look the same on pretty much all computers out there. Windows had all of the “web core” fonts (Georgia, Verdana, Trebuchet, Arial, even Comic Sans). Macintosh computers had all of the “web core” fonts. Even most Linux computers had them because it was legal to mirror, download, and install the files Microsoft distributed to make the fonts widely available.
In the last decade, Android has become a big player, and the above font stack with Georgia will look more like Bitstream Vera than it looks like Georgia on Android.
The only way to have a website have the same typography across computers and phones here in the soon-to-be 2020s is to supply the .woff files. Locally (because Google Webfonts might be offline some day). Either via base 64 in CSS or via multiple files; I prefer base 64 in CSS because sites are more responsive loading a single big text file than 4 or 5 webfont files. Not .woff2: Internet Explorer never got .woff2 support, and we can’t do try-woff2-then-woff CSS if using inline base64.
Even with very aggressive subsetting, and using the Zopfli TTF-to-WOFF converter to make the woff files as small as possible, this requires a single 116 kilobyte file to be loaded with my pages. But, it allows my entire website to look the same everywhere, and it allows my content to be viewed using 100% open source fonts.
Then again, for CJK (Asian scripts), webfonts become a good deal bigger; it takes about 10 megabytes for a good Chinese font. In that case, I don’t think it’s practical to include a .woff file; better to accept some variance in how the font will look from system to system.
Edit In terms of having a 10-year website, my website has been online for over 22 years. The trick is to treat webpages as <header with pointers to CSS><main content with reasonably simple HTML><footer closing all of the tags opened in the header> and to use scripts which convert text in to the fairly simple HTML my website uses for body content (the scripts can change, as long as the resulting HTML is reasonably constant). CSS makes it easy for me to tweak the look and fonts without having to change the HTML of every single page on my site, but as the site gets older, I am slowing decreasing how much I change how it looks.
I think I agree with you here, in that much of the power of hypertext lies in the hierarchical "tree" model.
And yet, I think it has not been used properly up to this point ...
I hesitate to post this as this is not quite finished[1], but here goes - this is something called an "Iceberg Article":
http://john.kozubik.com/pub/IcebergArticle/tip.html
... wherein the main article content is, as the article suggests, a single, self-contained page.
And yet ... that is just the "tip" - underneath is:
" ... at least one, but possibly many, supporting documents, resources and services. The minimum requirement is simply an expanded form of the tip (the "bummock"), complete with references, notes and links. Other resources and services that might lie under the surface are a wiki, a changelog, a software repository, additional supporting articles and reference pages and even a discussion forum."
[1] Neither wiki nor forum exist yet, but the bummock does...
(1) a single, self-contained page, (2) that is just the "tip", and (3) linked within it is all the stuff mentioned
That site has various opinions about the "tip" being uncluttered of links, etc, but that's just an opinion (and one I disagree with).
Wikipedia articles can be quite long (and justifiably so) - perhaps scrolling many pages.
The "tip" of an "Iceberg Article" is ".. a single page of writing ..."
Perhaps confusing because I don't mean a "single (web)page" I mean, an actual single page.
This is by and large, impossible. The hoops you have to jump through and downsides you have to endure are just a death by 10,000 cuts. Try writing a tabulated container in raw HTML and CSS that flex sizes and behaves nicely with the browser back button. Partial page reloads, containers with native (no reload!) sorting, there's so many features in modern web design that are just downright impossible without JS.
The solution is to not require things like partial page reloads, etc. This shouldn't be a huge hardship -- a properly designed modern site can degrade gracefully anyway, so users that don't have or allow things like JS can still use it, even if in a "degraded" form.
https://archiveprogram.github.com/
It's specifically for open source software, but I wonder if it can be spun out into personal archives or websites.
---
Some years ago, I had an idea for a thought experiment called the Eternal Bit - various angles on the practicality of preserving a bit state "forever".
jeffhuang.com. 60 IN A 162.243.124.123
$ whois 162.243.124.123 | grep Organization
Organization: DigitalOcean, LLC (DO-13)
Designed to last... Hmm... Seems like a tall order for someone forced to maintain their own host.
I agree that you can (almost always) write HTML without JavaScript codes (CSS is not always needed either, but nevertheless it can be helpful). This improves portability too. If you do want to use JavaScript to generate static documents, consider using Node.js and have it output a plain HTML document, which will then be the hosted document, rather than hosting the JavaScript version. (This way, the code only has to run once.)
About fonts, I think usually the fonts are not an essential part of the document. (Still, you don't usually need to use so many, but often you do not need to specify the font at all, except you may still need to specify if it is monospace, bold, etc.) Also, you do not usually need so many pictures on your web page (except picture galleries).
I do think if you need a copy of a web page then you can copy it.
You can also use plain text documents (without HTML); it is what I often do.
While bold text is often used for emphasis and setting it to "regular" doesn't change the meaning much, using a monospace font signals specific things (often, it's used to represent code snippets, but in some contexts it can mean "work in progress"). Changing that font to a proportional one strips a lot of meaning and readability from the text.
In that regard, fonts families are an essential part of the document, and not just their stylistic/emphasis variants (e.g. size and weight, which can still convey important meaning on their own. Think titles and headings.)
[1] actually the font doesn't need to be monospaced, but it must be very different than the font used for main copy.
(One case where a monospace font is required is if you are automatically including text from another document which uses plain text format; there is no way to determine what format is needed, but monospace will always work.)
For the case of headings, this can be done without specifying the font yourself; if you specify it is a heading, then the rendering software can decide how to display it. (HTML has <H1> and <H2> and so on for headings, and Glk has style_Header and style_Subheader.)
In a world where 95% of the google search results for <common problem> are forum threads, with everyone 'answering' the question by saying 'just google it, dumbass'[1], I don't think people - as in - the common homo sapiens - cares about the longevity of the content they link to as much as the author thinks.
Quality of the content (Does it have the information you want) is king. Longevity is a 'future me' problem, and 'future me' is incredibly shortsighted.
[1] Thanks, guys, how did you think I found this forum thread?
You can convert PHP or other dynamic pages to static fairly easily, it takes about half a day of scripting and a little time with configuring your web server. The pages load quicker and the load on your server is lower.
I've long considered the idea of a permanent hosting solution, where people pay to get their content up and I guarantee it's longevity, but the business model on that is tough.
I kind of think this is the best approach. Write your content with as minimal markup as possible. Let the user-agent render the page in my own styling.
I wish the web was a bit more like the gopher protocol, where the page markup was very minimal. Just give me the plain text (or close to it) and let my client render it the way I want to read it.
Markdown (or similar) would be an ideal representational format. This would push all the rendering decisions to the client.
The text on the page adapts to the width of the browser window in which it is viewed. The result is that everyone can obtain the line width they prefer by adjusting the width of their local browser that is rendering the page.
If the lines are too long, just pop the page out into its own window, narrow the width of just that window until the lines are at a comfortable length for you, and you'll have lines that are just right for you. This page is an example of where you are in control of how the site renders and can easily adjust it to your own tastes.
Ultimately we are all dependent on the functionality of the internet and making a physical copy is the best way to make something last. However if we intend to not only have it last but be accessible, some of the suggestions are helpful but the most significant is: where can I host something indefinitely?
Will Netlify, Github Pages, AWS, etc all be around in 50 years? 100 years? 500 years? Heck, the internet hasn't even been around for all that long.
As I write this, my only thought is that you need a system of fallbacks. Frankly, this seems like a business opportunity in which the cost of the infrastructure is purchased based on some formula. DNS is configured programmatically. File locations are distributed and redundant. I'm not sure what the best approach is but one thing is certain: the page accessibility is the least of this author's issues. He just needs the site to be hosted...
The proposed solution set seems extremely convoluted and don't actually solve the issue.
I think he’s probably right.
The moment you have to struggle with NPM dependencies to switch hosts, it’s all over. If all you have is a bit of html, no problem.
Paying for a server - mentioned in the article about the provider changing access.
its all about hurdles.
I think that the medium, be it a hosting site or you own domain, will fail. What we need is maybe : a naming scheme that can transfer to different people (replicated files on different servers indexed by content hash? Aka anonymous ftp mirrors ... ) and common media ie readable by anyone, gracefully degrading.
In the past thirty years, there have only been a few actual kinds of content:
- Article
- Comment
- Reference manual
Nothing really deviates from this. If it does, then it's probably in the "application" domain and deserves all that javascript stuff.
I imagine a new semantic web inspired content type that bakes in permanent content URLs, author signatures, and supports sharing P2P and rich archival:
It can be a really simple model. No divs or presentational CSS or anything. (Well, maybe some support for LaTeX and I18N.) The client decides how it looks, which is how it always should have been.If every entity in the universe has an ID, they can link to each other in a distributed fashion. It won't matter if the original source URL goes down as long as someone somewhere has a cache.
It's also probably better to binary encode these (protobuf or something). Images and media could be inlined instead of href'd.
Instead of building on the layer cake, we should take a moment to reflect on what we're trying to do. We have a lot of history and bad evolution that we can garbage collect and streamline.
If our objective is to publish and share content, a lightweight version of the semantic web is the way to go.
Oh, and check out my follow-up to the author:
Again, these can be distributed on a web. It can be p2p, federated, or live atop the classical WWW. It doesn't matter. The clients can richly use the semantic data model.The anger for disappearing content, walled gardens, and AMP will eventually reach a boil. As technologies go, the pendulum is always swinging. This is the direction we'll eventually head back to.
;)