Wow, great! Self-hosted, open-source, solid UI, tie-ins to the broader ecosystem... seems to check all the right boxes. Looking fwd to trying it and if all goes well, maybe see about integrating it into AthensResearch. Thanks for sharing!
I literally just spun up a copy but it looks like it has Sonic full-text integration, however I'm not 100% sure if it's working via the UI as there isn't much feed back letting you know why a site has displayed in the results.
I can't wait for the API to be completed. I want to build something to archive HN (article + comments) and turn it into epub go read offline. Hard to do currently.
Actual ESXI on a decently powerful (33 cores, 512 gigs) machine. VMWare’s been really good to me, minus some points for occasional stupidity on upgrades.
Yes, I can write a long article about why it's the default someday. I've agonized over this decision for many many months, and it's flipped flopped a few times as well.
The short version is that defaults in software are really important (90% of users wont change them), and I don't trust myself to code ArchiveBox 100% correctly so as to never lose data, or the majority of people to store their archives correctly so as to never lose data on their own. Archive.org is the redundant failsafe. Another good reason is that Archive.org is not the only way that your archive content can be leaked, the security model means that archived pages can read each other's content, so I want to make it abundantly clear to users that by default it's designed to only archive content thats already public (in which case it's already fair game for Archive.org).
I've settled on leaving it on as the default, but I do mention 3 times in the README how to disable it, most notably in the CAVEATS section which explains both the security model drawbacks and how to prevent your content from being leaked to Archive.org or other 3rd party APIs.
Although I tend privacy-by-default for most deployed technologies, the context of archiving does change the criteria quite a lot; you've selected a sensible and reasonable default, I reckon. Hopefully integrity is a consideration too? Glad to read that article, one day :)
Integrity is absolutely paramount too of course, which is why I chose Django (because of the mature DB migrations system that makes upgrades deterministic, reversible, and relatively painless). Hand coding a schema migration system would be a recipe for disaster and an easy opportunity for users to lose data.
Nevertheless, no system is perfect, and even with Django helping guard database integrity and multiple redundant index files, it's possible I'll make a mistake someday that leads to data loss on upgrade. I don't want that situation to be the next (mini) library of Alexandria, and saving copies to Archive.org helps serve as a last-resort backup.
Appreciate the project, web archives are becoming an important part of the internet ecosystem.
I personally have no issue with the defaults, but if you've agonized over the defaults, perhaps you should consider clearly documenting it in the main project README instead of leaving it for people to find in the config documentation.
If you're interested in this sort of thing, you might also be interested in Archivy [1], which is somewhat similar but it (thankfully) doesn't upload your stuff to archive.org
Aside from the can of worms that is copyright infringement? There was a recent HN discussion about how much of a pain it is to get something removed from archive.org.
Not uploading people's stuff to permanent, public archives seems like a good rule of thumb.
Something can be public today and not tomorrow. Something can be made public by accident. Something can be publically reachable (i.e. a private URL but one without a login) without the intention of being searchable.
I think it's primarily personal preference of features and how things are stored and presented. While Wallabag is more a Pocket/Read It Later alternative, LinkAce does not save the website itself, but a reference to it including the taxonomy you assign to it. It is intended to be a long-term bookmark archive, but without handling all the website archiving on its own.
I'm not sure how the iCloud Sync works in this case but if you use Desktop Safari where it has your iOS history too you might be able to get it out of the Safari sqlite DB on your computer:
Last time I tried to do this same thing, I didn't know about these, and ended up spending a couple days on wget and httrack. Do all these alternatives work from the command line, or are they their own little proprietary ecosystem?
How do tools like this cope with pages that are rendered by Javascript. What do the tools actually save? For instance if I save a Quora page using Firefox I can open it but if Quora is not accessible it doesn't work.
ArchiveBox is a wrapper around ~12 different extractor modules, each of which saves the page or its assets in a different way. The most relevant to JS is Singlefile, which renders the page in headless chrome and then snapshots the DOM with all assets inlined after a few seconds of JS execution. It's not perfect, but it works well even for the majority of JS-heavy sites.
60 comments
[ 3.4 ms ] story [ 137 ms ] threadCould be very useful now.
Any time I need to do anything, I will full clone the base; with a decent SSD it takes maybe 10 seconds for the full clone and I have a full OS.
https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#...
Not since 2017. https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
They now have a clunky manual process to exclude your site. https://help.archive.org/hc/en-us/articles/360004651732-Usin... ("How can I exclude...")
They don't spoof user agents, but blocking them actively doesn't remove their history.
The short version is that defaults in software are really important (90% of users wont change them), and I don't trust myself to code ArchiveBox 100% correctly so as to never lose data, or the majority of people to store their archives correctly so as to never lose data on their own. Archive.org is the redundant failsafe. Another good reason is that Archive.org is not the only way that your archive content can be leaked, the security model means that archived pages can read each other's content, so I want to make it abundantly clear to users that by default it's designed to only archive content thats already public (in which case it's already fair game for Archive.org).
I've settled on leaving it on as the default, but I do mention 3 times in the README how to disable it, most notably in the CAVEATS section which explains both the security model drawbacks and how to prevent your content from being leaked to Archive.org or other 3rd party APIs.
Nevertheless, no system is perfect, and even with Django helping guard database integrity and multiple redundant index files, it's possible I'll make a mistake someday that leads to data loss on upgrade. I don't want that situation to be the next (mini) library of Alexandria, and saving copies to Archive.org helps serve as a last-resort backup.
I personally have no issue with the defaults, but if you've agonized over the defaults, perhaps you should consider clearly documenting it in the main project README instead of leaving it for people to find in the config documentation.
[1] https://archivy.github.io/
FIY You just have to set the environment variable SUBMIT_ARCHIVE_DOT_ORG=False
Not uploading people's stuff to permanent, public archives seems like a good rule of thumb.
If it isn't already public (i.e., reachable by archive.org), it won't be afterwards?
I really wanna read this but can't find the thread, do you happen to have a link?
Just a heads-up. Found that a while ago and much prefer it over wallabag.
https://stackoverflow.com/questions/28628385/sqlite-safari-h...
https://support.apple.com/guide/safari/keep-a-reading-list-s...
For the very complex sites that really rely on a ton of interactive JS or dynamic requests to APIs to render their content, check out https://ArchiveWeb.page + https://ReplayWeb.page by https://webrecorder.io.