It always strikes me as hilarious when printed books reference web URLs. Granted, URLs are meant to be permanent (the cool ones don't change, as they say) and any webmaster worth their salt will strive really hard for this.
But c'mon. You think a website will outlive the useful lifespan of a printed book? Hah!
You should see the age of some of the books I've found useful in the past few years.
WebCite [1] was built for this in the past, but using archive.org is a better idea nowadays. Save a copy of the webpage to the archive if it doesn't already exist and link to it.
Has the added advantage that anyone who goes to the link sees exactly what you saw when you linked to it, with no modifications.
I think a service like webcitation is better because
1. Archive.org follows robots.txt rules - also used to remove content on request from owners (not sure if still do but I suppose they do) making it less sure than webcitation.
2. webcitation could also provide other services like - export citations for book or collections of books into one book of citations.
3. Similar citations tracking stuff that you can get on google scholare, how many other publications were citing this text or part of this website.
on edit: obviously this is if webcitation or a similar service was still a going concern, pragmatically yes archive.org is better because functioning and taking new content to archive.
The only reason I recommneded archive.org is that webcite is no longer taking submissions, and their HTTPS cert has expired. Other than that, I agree with you.
The Internet Archive at least since it'll keep a URL alive and consistent a lot longer.
At least plain URLs can be manually passed to the internet archive for a lookup, but I'd love if publishers/authors made a habit of maintaining their source links (via shortlinks, published errata, etc).
That said, even with how easy it is to update, Wikipedia suffers from this immensely. I can't count how many times I've tried to track back a source from a wiki and end up with a 404.
That rests on the assumption that IA will outlast everything else, which is unresasonable, if nothing else because, while it may be supported by the community, when Kahle becomes unavailable to the project for one reason or the other, there's no saying what will really happen to it.
The IA has well over 100 employees, possibly over 200 at this point. With a very strong mission statement, I have to imagine they have a very good culture dedicated to Brewster’s goals. I’m sure he’ll choose a very good successor. I’m sure he’s already taken considerable steps to codify his mission into a charter. He’s a very smart guy and he’s worked too hard to allow his vision to falter when he retires.
Furthermore, archivist is a full blown profession with its own professional society. I’m sure IA will always be able to hire people who are serious about preserving and cataloguing information.
I'm curious who it's going to be and what field his successor will come from. As a coder/developer with an MLIS + a decade of library experience, in my experience there aren't that many people who grok both the tech and the politics/people aspect around preservation, which you would need to head IA.
I wouldn't be shocked if, if IA is around in a few decades, they start expanding into educational partnerships to ensure the vision continues. Or institute fellowships. Or they may just inspire more people; IA's work was one of the things that convinced me to pursue an MLIS (as a girl, I was discouraged at the time from CS). Or we'll start seeing more degree specializations that mesh the two, like HCI concentrations.
> I can't count how many times I've tried to track back a source from a wiki and end up with a 404.
Yes, that's really frustrating. I often have the situation where I click on a reference for a claimed fact and get redirected to the front page of a news website.
Personally I would save a copy in a standard format and make it available on a domain I control. I've done that in the past for scientific publications.
I chuckle at something similar where books on programming languages are titled "The Definitive Guide to <Insert Programming Language>". Our industry changes, and fast. Everything is chaos.
- Those explaining a programming language, framework or application with a specific version. These books clearly have a lifespan related to the thing they are explaining and will often end up in second hand or wholesale shops.
- Books explaining principles. It doesn't matter if they are 2 or 20 years old since the things they are explaining don't really change.
Sure, but the greatest preponderance of programming books explain the first, also some books that seem to be explaining the second are really more tightly tied to a particular programming language or platform than they like to let on - like Clean Code with Java.
Which is almost entirely unique to programming books. Take a book on electronics, and a book discussing principles and a 20 year old book will be nearly as relevant as one published today. Take a book on physics and a 100 year old book will be be nearly as relevant as one published today. That doesn't mean the fields are unchanging. Modern electronics is definitely more sophisticated and much of classical physics has been superseded by modern physics. One definitely needs more modern treatments in order to work at the frontiers of these fields. Yet the principles are less ephemeral. On the other hand, software development is much more fad driven. The most basic principles may still apply, yet the application of those principles change (sometimes subtly and sometimes dramatically) due to a tendency to throw the baby out with the bathwater.
I'm used to that from math, but I wonder if math books have many urls as citations. I guess I haven't read any modern ones but I wouldn't expect it, so do modern electronics books have urls as citations?
From what I have seen, which is admittedly limited, textbooks tend to push additional services from the publisher but very little is lost if you don't use those services. Then again, the books I have seen are older (even though I may be reading a recent edition).
Anything directed towards more casual learners, I would have a hard time commenting on since it tends to be more the history of science rather than being technical in nature (where references tend towards primary sources anyway), or embedded development (which seems to suffer from a similar mentality as software and web development since it has a software bias).
The person you're replying to didn't mention anything about the useful lifetime of technical books specifically. URLs appear in e.g. footnotes and bibliographies of plenty of non-technical works.
This is a use case of IPFS I think. Any IPFS "website" will remain available at its original IPFS content ID as long as at least one person is seeding it, so if a book referenced one the author could seed it to guarantee that the URL will continue to work as long as they're willing to keep it up (and any interested 3rd party could do the same)
What seems a lifetime ago I wrote the backend for the website of Sprite's Sublymonal ad campaign. It mashed up a bunch of wikipedia data and fed it to a slick flash frontend. The whole website was expected to last two weeks.
Funny story: I was expected to use Coca-Cola's Websphere infrastructure, maintained by IBM at some IBM datacenter. It was going to take them months to set up, and I had a week before the campaign launched. I ended up running the whole thing on a couple (rackspace, IIRC) VPSes and never told the client.
I still maintain my personal website which was once pretty popular. It remained neglected for a very long time, and now I just keep it for the archives and for my personal rambling.
Some of the popular links are the likes of "/archives/000129.php", "/archives/2004/10/the_elements_of/", etc. I have tried my best to take care of the top 20-odd links and redirect them to the new/better URL, which is likely to survive my naming method of "/YYYY/title-of-the-link/".
Many of these has survived DCMAs, still enjoys inbounds from the Russians, and even the Chinese.
One of the many reasons for me to get rid of the Analytics is that I feel bad and always wanted to solve the errors. Now, I just don't have a clue besides the basic analytics that comes with Cloudflare, which I see once in few months.
Psssstttt, DDG or Google or Search my name "Brajeshwar". ;-)
Fun fact, some people in the political higher ups in India wants to use that domains and thinks they have rights to it. I have 20+ year of data/info to prove them wrong.
I am so jealous that you were able to get a .com name. I'm Indian as well, but I'm younger and my name is common so by the time I was looking into a .com, it was being sold for thousands of dollars ://
Another family fun-fact -- I also own the .com of our family name "oinam.com". Oinam[1] is a community and the name of a place in our hometown. Well, there are occasional events where people will ask me to buy it, email for questions about stuffs of that place, etc.
Nice; so you stumbled on something. IIRC, I kept Keepass around for a while, even after I started beta-testing 1Password during its early program. I believe by 2008 when iPhone was launched in India (I was there for the Bombay chapter in a mall in Andheri-West), I moved to 1Password along with the iPhone.
We're talking about information technology here right? It's been constantly in flux for 30 years when it comes to the web and keep going as far as PCs and data storage systems before that.
Not saying I wouldn't like to see more long-lasting designs and content archiving is a huge deal/challenge/lost histories real problem, but hard to expect 10-20 year design choices and infrastructure to hold up or be planned for.
Most websites have a few kilobytes of zipped useful text, and a few useful jpegs. By all means, most social media notifications could fit in a single ICMP packet each. Hacker news could be checked over ARPANET.
Wrt. classical probability, useful data divided by total data/metadata, the ratio has steadily been decreasing. Our infrastructure is changing for the worse and 'probably' about to fail.
The reason websites aren't designed to last is not due to ever-changing design trends; it's because the web is inherently ephemeral. For a website to continue to be available requires continuous effort and expense to run a web server, update SSL certs, etc. This automatically shortens the time horizon when you're developing for it.
I think the way to get more "timeless" software is to build a platform where the code and the data can be brought offline. PWAs do the first but not the second. The data needs to be in an enduring format like SQLite that can be queried offline. With that kind of system, an app or game (and the accompanying data) could remain usable 20 years from now, long after its author stopped distributing it.
> The reason websites aren't designed to last is not due to ever-changing design trends; it's because the web is inherently ephemeral. For a website to continue to be available requires continuous effort and expense to run a web server. This automatically shortens the time horizon when you're developing for it.
Interesting thought, but I don't think that's quite it. Everything is ephemeral to some degree. Everything will require a combination of resources to keep something available continuously.
For example keeping books around is not exactly free. Yeah, we usually "just have it in a bookshelf at home", but if you think about what that actually entails you'll probably start wondering if that book actually appreciates all you've done for it.
When running a home server becomes as simple as building an IKEA shelf, and when making a website is a native as writing on pen and paper, we will have a lot less bookshelves and more websites just being around and with that the expectation of them being around and zero understanding of how or why. No consideration being paid to routing, backup, space, bandwidth, because there is nothing to consider. Something that is magically and reliably available for money, like electricity. If you move you just bring the box and it works again as soon as you plug it in, like any desk lamp would.
Technology is still way too hard and inaccessible for that to be a thing.
You could potentially keep a website "on a bookshelf at home" as well, provided the database is small enough. It doesn't even need to be SQLite. No matter the data format, you could get a fully running website on a raspberry pi, maybe with a subset of the data.
I'm picturing a bookshelf of raspberry pis in custom "boxes" like a collection of 8-tracks. Pull one out, plug it it, turn it on, and connect to it.
Considering the majority of my 20 year web-building career is long gone, this idea kind of intrigues me.
I believe that content addressed storage had to be built open-source because it was ultimately not in anyones corporate interest to further it.
It's interesting to reflect on the kinds of technologies which are clearly possible, but never seem to get built because they don't centralise control. Truly offline apps, and actually working device-to-device networking fall in this bucket I think. Yes, there are power and privacy considerations but it's pretty clear none of the big players want those technologies to take off.
As long as you bother to pin the content of course, otherwise you have to hope that someone else does. But if you have to store the data yourself anyway, why not go even simpler and store the content in a folder on your hard drive?
To normies this is incredibly far from a bookshelf. What is a raspberry pi? What is the sqlite in doesn't-even-need-to-be-sqlite? Why do I have to care?
The highest level of questions you would need to be aiming for on the hardware side are: Where do I put this thing? Where did I leave the power chord? How do I turn it on?
If it doesn't turn on, you have someone repair it as you would any other electronic appliance. Can't be repaired? Get a new one. As soon as you plug it in it automatically fetches the automatically created backup. How? From where? What about encryption? The user is not qualified to answer any of that and should not be expected to care or expected to be involved into the decision. If I want to get involved with how electricity gets to my power socket I can look that stuff up on the internet. In the meantime just make it work without any intervention and with 99.9% reliability.
Us techies (maybe subconsciously) still lean way too much towards trying to involve users with the stuff we care about, when most of them just want our work to be invisible. Hide the complexity. Make it seamless. Make all tech a power socket.
All that doesn't matter. What normies did was purchase a CD of a native application, install it and have it work basically forever; with zero need for servers or hosting costs.
In my experience, home servers are delicate. Power outages and brownouts not only temporarily take the server offline, they can corrupt it and take it offline permanently. And you still have to pay for the hardware and the power.
It's much cheaper, easier, and more reliable to go with a cheap webhost like nearlyfreespeech.net to keep it online at almost no cost or maintenance.
> I'm picturing a bookshelf of raspberry pis in custom "boxes" like a collection of 8-tracks. Pull one out, plug it it, turn it on, and connect to it.
Why do your books need onboard compute? Why a Raspberry Pi, and not, like, an MMC/SD or Minidisc?
I think when we recognize what led you to first reach for the Raspberry Pi, we'll be able to better nail down why things are more difficult today than they need to be, and what we should be doing instead to eliminate that difficulty.
I was thinking of this generally. I've built hundreds of websites and while many had plenty of similarities, there would still be a plethora of databases and languages to support. And that's just my own history. There are quite a few developers, languages, data formats, services, etc.
And so the system that can run any "boxed" website - as in a working archive - would need to support all of these things, as they are today, in some form or would need to emulate / translate these things.
But instead of normalizing and archiving, my off-the-cuff suggestion is basically to freeze it in time. OS, languages, databases, services, data, code, everything is exactly as it was last built. Ideally, provided the storage medium held all its bits and the power source were compatible, it should boot up in 200 years without issue.
In a sense this is what we've accomplished with VMs and containers. But those still require an underlying system. If the system is part of the medium, you only need power and a console.
> there would still be a plethora of databases and languages to support
I don't think you absorbed the point I was going for there.
It sounds like you're prioritizing the website as an entity unto itself
above the content. You're not thinking of the content as individual entities,
and the website merely being a place where these entities are collected—like
books on a bookshelf. That first false step in the conceptualization of this
stuff is I believe to be _the_ reason that websites are "hard" to make
manageable for a non-technical audience. All the implementations of
"Information Management: A Proposal" have been executed poorly because the
thing that TBL was outlining was obviously going to require computer systems,
and when computer systems are involved you get the kinds of people who work on
computer systems and their (largely self-serving) tendencies—which a lot of
the time involves making everything in that system another system.
Do you need to support a plethora of databases and languages to maintain the
other books on your bookshelf—that is, the dead-tree ones? Even for digital
artifacts... the music you listen to, do you encode those as programs with
unbounded power that you execute every time you want to hear a song? Or do
you restrict yourself to relying on "dead" formats for what is sufficiently
"dead" content, and then offload the need for computation to a separate
machine that operates across a well-defined interface and that exists outside
the content itself—a music player app? When it comes to backups, do you
exclusively use self-extracting archives instead of something reliable like a
tarball or a ZIP? Why not?
At the end of the day, websites are really just about pieces of content and
the names we assign to them. It's the entanglement of all the unnecessary
violations of the Principle of Least Power in the form of live systems and
running programs that has made this stuff hard to manage (often not just for
non-technical people, but programmers and sysadmins, too).
Thanks for your thorough response. You've made your point well.
As I gather it, and I realize I'm simplifying, you're suggesting that websites need to be compiled for consumption just as a studio recording would be. I'm suggesting otherwise: That it's too varied of an experience to be compiled sufficiently across all the different ways in which the web is composed and consumed.
The content is not alone in what needs to be archived in order for a website (or any software) to live beyond just today. We don't _only_ read websites. We experience them in different ways depending on the purpose.
I have 30 tabs open right now: HN, google music, wordle, twitter, github, docker, nytimes, aws console, a couple blog posts, some software documentation, a couple pricing pages, and google maps open. Most of these things are not even remotely related in functionality, and we're supposed to be able to archive them into some similar format? For what purpose?
Much like a game, the experience of the content can be inherent in the content itself. But games are written to the same system. NES emulators and PS emulators and so on are required to enjoy game archives for systems that are now gone. My suggestion is to skip the emulator and include the environment with the content - which allows every piece of software to exist as it were originally conceived.
I could export a list of places I've been from google, but that's not useful to me in the same way without an interactive map to see the places therein. Same goes for foursquare and yelp and anything else that I allow to track my location for some purpose.
It would be incredible to be able to click around a static localized mostly-functional read only version of slashdot or fark or digg or prodigy or compuserve or geocities, or more recently HN or reddit. The UI for these applications is a _part_ of the content. Because the features are included in the content. Upvotes and Downvotes on HN are not the same as they are on slashdot, because slashdot's came with reasons. How does one experience that without the actual software? And how do we compile those software onto a single system that we'll be able to just load from nothing?
Do we _need_ to support a plethora of databases for books? No. Because they're books. Static, unsearchable, non-functional. Amazing in their simplicity. Perfect for what they are.
Software isn't that, nor should it be.
This isn't even a new suggestion. We used to have arcades - which were filled with separate systems built for single games, and each one was the size of a closet, including a screen and a UI. You can buy a 50 year old arcade game and, provided it was taken care of, it'll still boot up and work as it did in 1972.
Now we can get an entire system, with any software, and any database, onto a computer small enough that you could put hundreds of them on a single bookshelf. We could have terminals at libraries that you could plug these cartridges into and consume the information therein, and interact with them, in the original format.
I'm definitely not saying its the answer and should be done - only that it would be reasonable to accomplish as the world is today.
So much of the web is commercial now, including social media which, whilst full of personal writing, would disappear the moment it no longer promises returns.
To that end, so much of the web has a limited product-lifecycle, and the purpose of the websites are not at all the same as the purpose of a book. I think the way people build websites reflects this commercial reality more than anything else. Could you self host an HTML file on your PC? Totally, and you'll likely always have a PC around so long as you care about websites, so the analogy of the bookshelf I feel is similar to this. But most websites are not books, they're more like storefronts of knowledge, and when the stores shut down we lose access to the knowledge they held.
I can say this with some anecdotal confidence, as my personal website is some HTML that I haven't maintained in years but still works perfectly, yet almost every piece of software I've written in my professional life is gone, in many cases so too has the company that asked me to make it.
I think the main reason is low initial cost of investment. It's just easy and cheap (in terms of money/effort) to create a website, so people make lots of them for whatever far-fetched ideas they have. They don't intent to build a website for decades to come, because they don't know if they'll even continue to maintain it in six months' time - so they naturally gravitate towards the cheapest/easiest/most ephemeral solutions that get the job done for now. At scale, it results in the web resembling a shantytown.
Not to mention how the modern web is also inherently much harder to archive than the old one. Permanently archiving an old site was an easy as dropping the URL into HTTRACK and bam, you have a full working copy on disk. Modern ones have so much dynamic and cross-linked content you might be able to get the text and images in the right layout.
Native software has been doing this for ages, its nothing new or difficult. It is a matter of choosing technologies that we can be sure have a good chance of existing and working the same years into the future. Well written C code is one such example. None of the experimental flavor of the month stuff is going to endure, what endures and keeps enduring is stuff like ANSI C and attempting to use only standard features of POSIX/windows api etc.
In my experience "durability" is not even a metric to those typically deploying most small websites (e.g. think about the website for a small company) consider.
They just put some wordpress on some webspace, that is maintained by some hoster and call it a day. If something doesn't work the company will call them.
If people like these have problems with technology, they call people like me — I ask them why they don't just take a static site instead and they look at me sideways and ask me how they would then do a animated sliding gallery without a wordpress plugin.
They use the tech they use, because that's what worked for them so far. They don't use it because it is rationally the best choice.
The thesis is fine, whatever, but do they actually prescribe a solution here? What can someone do to "design for a long lifetime"? What does a website that is "designed for a long lifetime" look like? This all very hypothetical.
My personal site is dead simple, about 20kb uncompressed (50-odd pages), and uses semantic HTML, minimal CSS (~2-3kb) and no JS. But that's for simplicity and to keep my costs down. There's no saying whether even this basic structure will be supported 20 years down the line because it works on browsers now.
This analogy works for commercial websites. But that's hardly comprehensive. There are two mutually exclusive views of the web. As a set of protocols set to allow individual humans to share information about things they love and the web as a set of protocols to make a living.
Profit motivated web presences want views, they want attention, they need nine 9s uptime, need to be able to do monetary transactions absolutely securely, and they want to be an application not a document. They live and die on the eternal wave of walled garden's recommendation engines because that's the network effect and that's where money flows. They're ephemeral and only last for a couple years. It doesn't matter if this means extremely high barriers to entry because money solves everything.
Individuals' websites are freeform presentations about the things that person is interested in. They are the backyard gardens of the mind and the most important thing is lowering the friction from thought to posting. There's no need to get tons of traffic instantly (or ever), they're mostly time insensitive and often last decades.
Heh, the website I work on went online in 1996, there was a design update in 2006 (when I joined, still as a student), and we are planning to make it mobile-friendly this year. How’s that for longevity? ;)
I don't think people building websites explicitly think about the expected lifetime and plan for whatever specific lifetime they expect.
On the contrary, people do that a lot. Every abstraction, every 'separation of concerns', every micro service or micro-frontend, is a response to the belief that at any moment a developer might need to replace some part of the whole. Pushing content into markdown files that can be loaded with any build process that understands markdown is an attempt to be build something where the website, eg the content, can outlast the current code. Every template-driven design populated by values from an API that's loading a dictionary from a database is decoupling the pages from the code, which greatly extends the life of the website.
The website is not the code. That's just the part of it that you see. The website is the ideas, creativity, content, effort and understanding that went into defining what the code does.
Some parts of the code might change relatively quickly, but that's the nature of code. We find better ways of doing things. Browsers change. We choose to optimize for different things. That's a good thing, even if it means throwing some old code away.
It's a bit like the Ship of Thebes. Everything might have been replaced a few times, but the website is still the same website.
I design websites with the mindset of changing requirements over time. So the website must be created decoupled and scaleable. Start with the URLs.
URL-rules
URL-Rule 1: unique (1 URL == 1 resource, 1 resource == 1 URL)
URL-Rule 2: permanent (they do not change, no dependencies to anything)
URL-Rule 3: manageable (equals measurable, 1 logic per site section, no complicated exceptions, no exceptions)
URL-Rule 4: easily scalable logic
URL-Rule 5: short
URL-Rule 6: with a variation (partial) of the targeted phrase
URL-Rule 1 is more important than 1 to 6 combined, URL-Rule 2 is more important than 2 to 6 combines, … URL-Rule 5 and 6 are a trade-of. 6 is the least important.
The most important part in this context is rule nr 2. permanente, no dependencies with anything. no dependencies with the navigation, not with categorisation, not with the content the URL hosts, if not 100% within the organisations control.
The URLs must be also decoupled from the CMS or any technology choice underneath.
The next things is "don't do redirects". Redirects are only done when there is systematic change of the website (relaunch), but not a daily business tool.
Interlinking only to canonical URLs. No parameters.
Each (Web)Application (that mostly interacts with user input) gets its own subdomain. i.e.: checkout.example.com, map.example.com
Throw away things that you do not need any longer. Cleaning up is an ongoing process of a website, similar to creating things.
Responsibility, 1 person is responsible for the whole website. 1 person is responsible for each section of the website. If a section of the website has nobody who is responsible any more, somebody must be found or the section will be deleted.
Just a few of the rules I put in place. It's a mixture of decoupling and responsibilities. Change is inevitable - and manageable.
As a creative developer, my job is basically to make short life websites.
Her is the way most of the websites I do are considered: when you make a communication campaign, the website is one of the medias used, like tv ad, print ad, ... So the websites only last for the duration of the campaign. Consider them as ads like any other print/tv/radio/.. ads
Let me tell you that all the considerations on accessibility have never been even discussed or even a few times laughed at.
A website is just a medium to publish and present information. It generally has as long a lifetime as the underlying project. The short-livedness of websites is a symptom of short-lived projects and ventures in general.
To me, this phenomenon is a parable for perseverance rather than a technological cautionary tale.
My Wordpress hobby blog is 12 years old and is still using the same theme as day one. A few years back my favicon suddenly stopped working. I called Wordpress and they said my theme was deprecated and to get the favicon back, I’d have to choose a new theme… but once I did, I could never revert back to the old one.
My blog has thousands of short form posts and I very much like how they are presented. I just left it the way it was. I sincerely hope they never make me try to switch, but realistically I have to assume they eventually will.
I have 6000 little posts, each with a 300x300 image... and it's set up the way I like it. I just know that if I tried to import it into a new theme, something would be off that would require fixing manually, post by post, which I definitely wouldn't do.
Theoretically, the way to go about it would be to save several generated static HTML pages from your blog and work backwards from there, adding the necessary Wordpress PHP. Granted it would be intense work, but then you would no longer have the impending cancellation of your theme hanging over your head.
So your Wordpress host doesn’t allow custom themes? Because theoretically, you should be able to make a ‘modern’ theme that creates the same HTML + CSS as your beloved vintage theme.
I sympathise with wanting to keep the presentation. It was made in and for this context. It’s like reading an old magazine, you get so many extra clues about the context from the design. If a magazine does a redesign, it only affects the issues going forward. In my ideal world, blog redesigns work the same way.
I’ve done my share of maintaining and archiving older websites, where I tried to keep the design in place while updating the underlying infrastructure (or converting it to static HTML). The one thing I would sometimes change is to ‘zoom in’, multiplying all pixels in the CSS by a small amount, because the web moved to larger text sizes and screens, so older websites tend to look tiny.
I work on a website that has been around in some form or other for nearly 22 years. In that time it has undergone about 7 major revisions. The last revision was about 5 years ago. But we are still finding bugs and the site requires two of us to work on maintaining it. The site is interesting in that it is far from being our core business site, yet is useful enough that the company will probably keep it around until after I’m retired. There is always work that needs to be done and we have a large backlog list of things that we need to do to keep it running. Our biggest issue right now is the site is running on a cluster of 2008 Windows servers and we want to move it onto AWS EC2 Linux servers. We’ve been trying to get to this work for years, but there is always something else that needs doing that has higher priority.
There are other factors to consider, once a site turns into a desktop app replacement with dynamic content it is expected that it will change and grow, features dropped and added, styles change, the tech stack is replaced. Building a single use product to last doesn't make a lot of sense, the only part that does is the idea of maintaining "permalinks" where the content is stable and "append only", but the style may change. The world has not placed enough requirements to preserve a historical friendly web, which is why archive sites, print to pdf, and other options exist.
It is expensive to plan, and maintain sites with longevity. You don't know what the future holds and longevity might be at odds with relevance.
Perhaps neglected web content should die. Improvement via a kind of Darwinian mechanism would depend on that.
Web content that people that people care about could last forever if we adopted the Federated Wiki patterns, i.e. don't just link, but copy content from other sites and put it on your own (everything there is, in principle, licensed CC share-alike).
It is unfortunate that mirroring is not first class concept on web. There are some concessions like relative urls, but largely mirroring a modern web site is a non-trivial operation. If mirroring was easier and more prevalent, then the longevity of individual websites would be less critical.
You can generalize this conclusion into almost anything being designed with a short lifetime.
Our society is fast and getting ever faster. It's unstable and exponentially changing. In these condition, nobody can afford to do anything foundational or timeless. We're in constant survival mode, just trying to make it to the next financial quarter.
Why worry about 20 years from now when we're not even sure our entity survives the coming 3? Why invest in something "proper" when that proper thing may be entirely obsolete or lose relevance?
Why develop a product that lasts if you can win more when you make one that doesn't?
82 comments
[ 6.3 ms ] story [ 99.7 ms ] threadBut c'mon. You think a website will outlive the useful lifespan of a printed book? Hah!
You should see the age of some of the books I've found useful in the past few years.
Has the added advantage that anyone who goes to the link sees exactly what you saw when you linked to it, with no modifications.
[1]: https://www.webcitation.org/
1. Archive.org follows robots.txt rules - also used to remove content on request from owners (not sure if still do but I suppose they do) making it less sure than webcitation.
2. webcitation could also provide other services like - export citations for book or collections of books into one book of citations.
3. Similar citations tracking stuff that you can get on google scholare, how many other publications were citing this text or part of this website.
on edit: obviously this is if webcitation or a similar service was still a going concern, pragmatically yes archive.org is better because functioning and taking new content to archive.
At least plain URLs can be manually passed to the internet archive for a lookup, but I'd love if publishers/authors made a habit of maintaining their source links (via shortlinks, published errata, etc).
That said, even with how easy it is to update, Wikipedia suffers from this immensely. I can't count how many times I've tried to track back a source from a wiki and end up with a 404.
Furthermore, archivist is a full blown profession with its own professional society. I’m sure IA will always be able to hire people who are serious about preserving and cataloguing information.
I wouldn't be shocked if, if IA is around in a few decades, they start expanding into educational partnerships to ensure the vision continues. Or institute fellowships. Or they may just inspire more people; IA's work was one of the things that convinced me to pursue an MLIS (as a girl, I was discouraged at the time from CS). Or we'll start seeing more degree specializations that mesh the two, like HCI concentrations.
Yes, that's really frustrating. I often have the situation where I click on a reference for a claimed fact and get redirected to the front page of a news website.
The most egregious cases is when the author references their own web content!
- Those explaining a programming language, framework or application with a specific version. These books clearly have a lifespan related to the thing they are explaining and will often end up in second hand or wholesale shops.
- Books explaining principles. It doesn't matter if they are 2 or 20 years old since the things they are explaining don't really change.
Anything directed towards more casual learners, I would have a hard time commenting on since it tends to be more the history of science rather than being technical in nature (where references tend towards primary sources anyway), or embedded development (which seems to suffer from a similar mentality as software and web development since it has a software bias).
As technologies age, so do anything that talks about them.
Funny story: I was expected to use Coca-Cola's Websphere infrastructure, maintained by IBM at some IBM datacenter. It was going to take them months to set up, and I had a week before the campaign launched. I ended up running the whole thing on a couple (rackspace, IIRC) VPSes and never told the client.
Some of the popular links are the likes of "/archives/000129.php", "/archives/2004/10/the_elements_of/", etc. I have tried my best to take care of the top 20-odd links and redirect them to the new/better URL, which is likely to survive my naming method of "/YYYY/title-of-the-link/".
Many of these has survived DCMAs, still enjoys inbounds from the Russians, and even the Chinese.
One of the many reasons for me to get rid of the Analytics is that I feel bad and always wanted to solve the errors. Now, I just don't have a clue besides the basic analytics that comes with Cloudflare, which I see once in few months.
Fun fact, some people in the political higher ups in India wants to use that domains and thinks they have rights to it. I have 20+ year of data/info to prove them wrong.
1. https://en.wikipedia.org/wiki/Oinam
I double-checked and I already had 1Password on the Homescreen of my phone - https://oinam.fyi/brajeshwar/phone-homescreen/
We're talking about information technology here right? It's been constantly in flux for 30 years when it comes to the web and keep going as far as PCs and data storage systems before that.
Not saying I wouldn't like to see more long-lasting designs and content archiving is a huge deal/challenge/lost histories real problem, but hard to expect 10-20 year design choices and infrastructure to hold up or be planned for.
Wrt. classical probability, useful data divided by total data/metadata, the ratio has steadily been decreasing. Our infrastructure is changing for the worse and 'probably' about to fail.
I think the way to get more "timeless" software is to build a platform where the code and the data can be brought offline. PWAs do the first but not the second. The data needs to be in an enduring format like SQLite that can be queried offline. With that kind of system, an app or game (and the accompanying data) could remain usable 20 years from now, long after its author stopped distributing it.
Interesting thought, but I don't think that's quite it. Everything is ephemeral to some degree. Everything will require a combination of resources to keep something available continuously.
For example keeping books around is not exactly free. Yeah, we usually "just have it in a bookshelf at home", but if you think about what that actually entails you'll probably start wondering if that book actually appreciates all you've done for it.
When running a home server becomes as simple as building an IKEA shelf, and when making a website is a native as writing on pen and paper, we will have a lot less bookshelves and more websites just being around and with that the expectation of them being around and zero understanding of how or why. No consideration being paid to routing, backup, space, bandwidth, because there is nothing to consider. Something that is magically and reliably available for money, like electricity. If you move you just bring the box and it works again as soon as you plug it in, like any desk lamp would.
Technology is still way too hard and inaccessible for that to be a thing.
I'm picturing a bookshelf of raspberry pis in custom "boxes" like a collection of 8-tracks. Pull one out, plug it it, turn it on, and connect to it.
Considering the majority of my 20 year web-building career is long gone, this idea kind of intrigues me.
It's interesting to reflect on the kinds of technologies which are clearly possible, but never seem to get built because they don't centralise control. Truly offline apps, and actually working device-to-device networking fall in this bucket I think. Yes, there are power and privacy considerations but it's pretty clear none of the big players want those technologies to take off.
Storing websites on your hard drive tends to break URLs; content addressing solves that.
The highest level of questions you would need to be aiming for on the hardware side are: Where do I put this thing? Where did I leave the power chord? How do I turn it on?
If it doesn't turn on, you have someone repair it as you would any other electronic appliance. Can't be repaired? Get a new one. As soon as you plug it in it automatically fetches the automatically created backup. How? From where? What about encryption? The user is not qualified to answer any of that and should not be expected to care or expected to be involved into the decision. If I want to get involved with how electricity gets to my power socket I can look that stuff up on the internet. In the meantime just make it work without any intervention and with 99.9% reliability.
Us techies (maybe subconsciously) still lean way too much towards trying to involve users with the stuff we care about, when most of them just want our work to be invisible. Hide the complexity. Make it seamless. Make all tech a power socket.
It's much cheaper, easier, and more reliable to go with a cheap webhost like nearlyfreespeech.net to keep it online at almost no cost or maintenance.
Why do your books need onboard compute? Why a Raspberry Pi, and not, like, an MMC/SD or Minidisc?
I think when we recognize what led you to first reach for the Raspberry Pi, we'll be able to better nail down why things are more difficult today than they need to be, and what we should be doing instead to eliminate that difficulty.
And so the system that can run any "boxed" website - as in a working archive - would need to support all of these things, as they are today, in some form or would need to emulate / translate these things.
But instead of normalizing and archiving, my off-the-cuff suggestion is basically to freeze it in time. OS, languages, databases, services, data, code, everything is exactly as it was last built. Ideally, provided the storage medium held all its bits and the power source were compatible, it should boot up in 200 years without issue.
In a sense this is what we've accomplished with VMs and containers. But those still require an underlying system. If the system is part of the medium, you only need power and a console.
I don't think you absorbed the point I was going for there.
It sounds like you're prioritizing the website as an entity unto itself above the content. You're not thinking of the content as individual entities, and the website merely being a place where these entities are collected—like books on a bookshelf. That first false step in the conceptualization of this stuff is I believe to be _the_ reason that websites are "hard" to make manageable for a non-technical audience. All the implementations of "Information Management: A Proposal" have been executed poorly because the thing that TBL was outlining was obviously going to require computer systems, and when computer systems are involved you get the kinds of people who work on computer systems and their (largely self-serving) tendencies—which a lot of the time involves making everything in that system another system.
Do you need to support a plethora of databases and languages to maintain the other books on your bookshelf—that is, the dead-tree ones? Even for digital artifacts... the music you listen to, do you encode those as programs with unbounded power that you execute every time you want to hear a song? Or do you restrict yourself to relying on "dead" formats for what is sufficiently "dead" content, and then offload the need for computation to a separate machine that operates across a well-defined interface and that exists outside the content itself—a music player app? When it comes to backups, do you exclusively use self-extracting archives instead of something reliable like a tarball or a ZIP? Why not?
At the end of the day, websites are really just about pieces of content and the names we assign to them. It's the entanglement of all the unnecessary violations of the Principle of Least Power in the form of live systems and running programs that has made this stuff hard to manage (often not just for non-technical people, but programmers and sysadmins, too).
As I gather it, and I realize I'm simplifying, you're suggesting that websites need to be compiled for consumption just as a studio recording would be. I'm suggesting otherwise: That it's too varied of an experience to be compiled sufficiently across all the different ways in which the web is composed and consumed.
The content is not alone in what needs to be archived in order for a website (or any software) to live beyond just today. We don't _only_ read websites. We experience them in different ways depending on the purpose.
I have 30 tabs open right now: HN, google music, wordle, twitter, github, docker, nytimes, aws console, a couple blog posts, some software documentation, a couple pricing pages, and google maps open. Most of these things are not even remotely related in functionality, and we're supposed to be able to archive them into some similar format? For what purpose?
Much like a game, the experience of the content can be inherent in the content itself. But games are written to the same system. NES emulators and PS emulators and so on are required to enjoy game archives for systems that are now gone. My suggestion is to skip the emulator and include the environment with the content - which allows every piece of software to exist as it were originally conceived.
I could export a list of places I've been from google, but that's not useful to me in the same way without an interactive map to see the places therein. Same goes for foursquare and yelp and anything else that I allow to track my location for some purpose.
It would be incredible to be able to click around a static localized mostly-functional read only version of slashdot or fark or digg or prodigy or compuserve or geocities, or more recently HN or reddit. The UI for these applications is a _part_ of the content. Because the features are included in the content. Upvotes and Downvotes on HN are not the same as they are on slashdot, because slashdot's came with reasons. How does one experience that without the actual software? And how do we compile those software onto a single system that we'll be able to just load from nothing?
Do we _need_ to support a plethora of databases for books? No. Because they're books. Static, unsearchable, non-functional. Amazing in their simplicity. Perfect for what they are.
Software isn't that, nor should it be.
This isn't even a new suggestion. We used to have arcades - which were filled with separate systems built for single games, and each one was the size of a closet, including a screen and a UI. You can buy a 50 year old arcade game and, provided it was taken care of, it'll still boot up and work as it did in 1972.
Now we can get an entire system, with any software, and any database, onto a computer small enough that you could put hundreds of them on a single bookshelf. We could have terminals at libraries that you could plug these cartridges into and consume the information therein, and interact with them, in the original format.
I'm definitely not saying its the answer and should be done - only that it would be reasonable to accomplish as the world is today.
To that end, so much of the web has a limited product-lifecycle, and the purpose of the websites are not at all the same as the purpose of a book. I think the way people build websites reflects this commercial reality more than anything else. Could you self host an HTML file on your PC? Totally, and you'll likely always have a PC around so long as you care about websites, so the analogy of the bookshelf I feel is similar to this. But most websites are not books, they're more like storefronts of knowledge, and when the stores shut down we lose access to the knowledge they held.
I can say this with some anecdotal confidence, as my personal website is some HTML that I haven't maintained in years but still works perfectly, yet almost every piece of software I've written in my professional life is gone, in many cases so too has the company that asked me to make it.
The sites which (actually) require server side code tend to be large and commercial.
They just put some wordpress on some webspace, that is maintained by some hoster and call it a day. If something doesn't work the company will call them.
If people like these have problems with technology, they call people like me — I ask them why they don't just take a static site instead and they look at me sideways and ask me how they would then do a animated sliding gallery without a wordpress plugin.
They use the tech they use, because that's what worked for them so far. They don't use it because it is rationally the best choice.
My personal site is dead simple, about 20kb uncompressed (50-odd pages), and uses semantic HTML, minimal CSS (~2-3kb) and no JS. But that's for simplicity and to keep my costs down. There's no saying whether even this basic structure will be supported 20 years down the line because it works on browsers now.
But somehow, people have the expectation that the entire web is more like a permanent archive of written content. That is simply not true.
Profit motivated web presences want views, they want attention, they need nine 9s uptime, need to be able to do monetary transactions absolutely securely, and they want to be an application not a document. They live and die on the eternal wave of walled garden's recommendation engines because that's the network effect and that's where money flows. They're ephemeral and only last for a couple years. It doesn't matter if this means extremely high barriers to entry because money solves everything.
Individuals' websites are freeform presentations about the things that person is interested in. They are the backyard gardens of the mind and the most important thing is lowering the friction from thought to posting. There's no need to get tons of traffic instantly (or ever), they're mostly time insensitive and often last decades.
On the contrary, people do that a lot. Every abstraction, every 'separation of concerns', every micro service or micro-frontend, is a response to the belief that at any moment a developer might need to replace some part of the whole. Pushing content into markdown files that can be loaded with any build process that understands markdown is an attempt to be build something where the website, eg the content, can outlast the current code. Every template-driven design populated by values from an API that's loading a dictionary from a database is decoupling the pages from the code, which greatly extends the life of the website.
The website is not the code. That's just the part of it that you see. The website is the ideas, creativity, content, effort and understanding that went into defining what the code does.
Some parts of the code might change relatively quickly, but that's the nature of code. We find better ways of doing things. Browsers change. We choose to optimize for different things. That's a good thing, even if it means throwing some old code away.
It's a bit like the Ship of Thebes. Everything might have been replaced a few times, but the website is still the same website.
The most important part in this context is rule nr 2. permanente, no dependencies with anything. no dependencies with the navigation, not with categorisation, not with the content the URL hosts, if not 100% within the organisations control.
So basically you end up with something like
https://www.example.com/%short-one-letter-namespace%/%unique... - or - if you want to keep future market and language expansion in mind
https://www.example.com/%market-language-namespace%/%short-n...
The URLs must be also decoupled from the CMS or any technology choice underneath.
The next things is "don't do redirects". Redirects are only done when there is systematic change of the website (relaunch), but not a daily business tool.
Interlinking only to canonical URLs. No parameters.
Each (Web)Application (that mostly interacts with user input) gets its own subdomain. i.e.: checkout.example.com, map.example.com
Throw away things that you do not need any longer. Cleaning up is an ongoing process of a website, similar to creating things.
Responsibility, 1 person is responsible for the whole website. 1 person is responsible for each section of the website. If a section of the website has nobody who is responsible any more, somebody must be found or the section will be deleted.
Just a few of the rules I put in place. It's a mixture of decoupling and responsibilities. Change is inevitable - and manageable.
Let me tell you that all the considerations on accessibility have never been even discussed or even a few times laughed at.
To me, this phenomenon is a parable for perseverance rather than a technological cautionary tale.
My blog has thousands of short form posts and I very much like how they are presented. I just left it the way it was. I sincerely hope they never make me try to switch, but realistically I have to assume they eventually will.
Granted, I don't know how WordPress works, or any if its technical circumstances.
I sympathise with wanting to keep the presentation. It was made in and for this context. It’s like reading an old magazine, you get so many extra clues about the context from the design. If a magazine does a redesign, it only affects the issues going forward. In my ideal world, blog redesigns work the same way.
I’ve done my share of maintaining and archiving older websites, where I tried to keep the design in place while updating the underlying infrastructure (or converting it to static HTML). The one thing I would sometimes change is to ‘zoom in’, multiplying all pixels in the CSS by a small amount, because the web moved to larger text sizes and screens, so older websites tend to look tiny.
There are other factors to consider, once a site turns into a desktop app replacement with dynamic content it is expected that it will change and grow, features dropped and added, styles change, the tech stack is replaced. Building a single use product to last doesn't make a lot of sense, the only part that does is the idea of maintaining "permalinks" where the content is stable and "append only", but the style may change. The world has not placed enough requirements to preserve a historical friendly web, which is why archive sites, print to pdf, and other options exist.
It is expensive to plan, and maintain sites with longevity. You don't know what the future holds and longevity might be at odds with relevance.
Web content that people that people care about could last forever if we adopted the Federated Wiki patterns, i.e. don't just link, but copy content from other sites and put it on your own (everything there is, in principle, licensed CC share-alike).
In some cases they should be destroyed during the life of the author (e.g. a book filled with CSAM photos).
Our society is fast and getting ever faster. It's unstable and exponentially changing. In these condition, nobody can afford to do anything foundational or timeless. We're in constant survival mode, just trying to make it to the next financial quarter.
Why worry about 20 years from now when we're not even sure our entity survives the coming 3? Why invest in something "proper" when that proper thing may be entirely obsolete or lose relevance?
Why develop a product that lasts if you can win more when you make one that doesn't?