24 comments

[ 2.3 ms ] story [ 51.6 ms ] thread
I like this... right now I'm using a ras pi 3 or 4 as a file server and it seems to mostly work?
I have used perkeep. I still do at least in theory. I love the concept of it but it’s become… not quite abandonware, but it never gained enough traction to really take on a full life of its own before the primary author moved on. A bit of a tragedy because the basic idea is pretty compelling.
They released a new version today, the first release in 5 years. It looks like it was more or less dead until September.
I don't really understand the goal here. It feels like "wouldn't it be nice if instead of organizing a library, we just kept all of the information in a giant unsorted pile of looseleaf paper?"

How is this better then a filesystem with automated replication?

Yeah, I think you perfectly nailed why this is kind of pointless. Better to abstract this thing out into two functions -- file organization and backup, because that second thing is solved easily.
And here I'm still looking for a way, with one click, to create an offline backup of the webpages each of my bookmarks points to. Such that the offline version looks and works exactly like the online version in (say) Google Chrome (e.g. the CTRL+F feature works fine). And such that I can use some key-combo and click a bookmark in my bookmarks manager (in Chrome) to open a webpage from the backup (or the backup can have its own copy of the bookmarks manager... it needs a catalog of some sort or it won't be useful).
FWIW I've had success with self-hosted [LinkDing](https://github.com/sissbruecker/linkding) and the firefox SingleFile plugin (so it archives what I'm seeing / gets around logins etc). LinkDing also links directly to Internet Archive for any URL.
WebRecorder [0] is the best implemention of this that I've tested. It runs as an extension in your browser, intercepting HTTP streams, so as long as you open a page in your browser the data is captured to reproduce it exactly. It outputs WARC files that are (in theory) compatible with the rest of the web archiving ecosystem, and has a WARC explorer interface to browse captured archives.

For pages with dynamic content that can't be trivially reproduced by their HTTP streams— E.G., opening the archive triggers GETs with a mismatched timestamp, even if the file it's looking for is in the WARC under a different URI— There's always SingleFile [1], and Chromium's built-in MHTML Ctrl+S export, which "bake" the content into a static page.

0: https://chromewebstore.google.com/detail/webrecorder-archive...

1: https://github.com/gildas-lormeau/SingleFile

At first glance, this looks like way too much to trust in the long run. I use git-annex since roughly 10 years to archive files I don't want to loose again. Does everything I want, and is pretty simple for what it gives me. A checksum for every file, replication on a file-basis, does not dictate the underlying filesystem I use. Full syncs are rather slow, but in reality, it doesn't really matter if I have to wait 3 hours or 2 days, just let it run in the background and do its thing.
I wish bradfitz had more time to work on it.
I feel like there have been a number of attempts in this content addressed space and that nobody has gotten it quite right, not that the underlying idea is unsound.
Can it be used with AI to create your personal context?
Interesting idea. Pretty timely as I recently started working (again) on a concept cross-platform "superapp" and have been trying to think of a decent state/storage sync solution.
I think many of us builds the same idea nowadays with many different tools and services. It became the "project car" of tech enthusiasts. But it's complicated and subjective enough that I guess it can not be abstracted down this way. We'd need some common platform, something like Synology was vaguely going for.
I've worked on and off on my own personal system which leaves the filesystem stuff to filesystems, and focuses on verifying backups/mirrors and recursing into archive formats. Also interested in warning of near-obsolete formats, like my collection of RealAudio files that are hard to decode these days.
"Blob servers" are essentially leverage cloud provider like AWS/Azure/GCP, not sure how this will help making "your data is entirely under your control".
By saying "under your control," I believe it means you are not locked in by cloud providers. Since it's easy to switch and back up between different blob storages, if one is down, your data remains accessible and manageable.
a plain old filesystem, esp on a small removeable device, is hard to beat