What I would love is a full personal CMS type system, where I can have a subject like "cybersecurity" or "cooking" and I can add bookmarks, notes, pdfs, and they're all treated equally and as first class objects in the system.
I would love a system that treats all media the same as just a source of knowledge instead of being hung up on the source type i.e. is it a video, or a book mark or a pdf....
If you didn't try Notion before, you should probably not judge it solely by my post. It is very versatile and you could probably use it in a way that comes close to what you describe.
There is not much support for PDFs or videos though.
Not exactly CMS, but I'm working on a browser extension along a similar line of thought. It unifies all bookmarks/notes/chat messages etc you have for a certail URL treating them as first-class: https://github.com/karlicoss/promnesia#demo
P.S. It's also relevant to bookmark management as well: you get to see not only notes on the bookmark, but also on the 'child' URLs, e.g. if you open someone's twitter profile you'd be able to see their tweets you've favorited. Or if you open some blog, you'd see posts from that blog you saved on Reddit.
I have been using a text file (actually an org mode file) since 2009. Before that I used a plain text file.
It doesn't handle video/bookmark/pdf. If I have to save those, I put them in a separate directory and make a note of it in the text file.
It's pretty well organized, thanks to org mode. It has a section that is by date, like journal entries, and a section that's by topic. Since it's just a single file, it's very easily searchable. It will never "go down". I don't need to run a database (I tried using wiki software for the same purpose). I can even "link" different sections, by having plain text labels. For example, I can refer to "ZFS Setup 2009-01-01" which is another place in the document I can search for.
I can "jump" to sections by using the orgmode annotation. Searching for "* Computers" will go to that top level section.
I liked this system so much that I also use it at work. People are very impressed that I can find things in my "notebook" so quickly. It's just text search.
Websites disappear. Web apps disappear. Apps disappear. My text file, does not.
As someone with a similar setup, I have a Google Doc named "Weekly Scratchpad" with a shortcut on my Android home screen. At the end of the week, I copy anything worth saving into my plain text file(s) and delete the rest.
> It doesn't handle video/bookmark/pdf. If I have to save those, I put them in a separate directory and make a note of it in the text file...
I've used a similar system in the past (though not based on org mode), but was too brittle if/when "links" broke or were altered. For an over simplistic example, if the references that you made to a video within your master text file(s) breaks because the video's containing folder name has changed even slightly, well that becomes annoying. For a small number of references, types of destination media types, etc., maybe not an issue, but after some point, it can become too much maintenance/annoyance. Is this an issue that you encounter? If so, do you have mitigations for avoiding/resolving the broken links? Just curious.
I've got quite similar system in spirit and aims, just instead of lower Notion layer, using org-mode.
It ends up searchable, synchronised with all devices, available offline and with great tooling for organizing and processing information.
> While documenting my configuration, especially my command-line workflows, I identified some shortcomings
Can't agree more! I've considerably simplified so many things while writing up on my setups and publishing code -- I guess it's easier to sneak in unnecessary complexity when it's all in your head.
> They are, however, not well suited for keeping extensive bookmarks libraries.
Agree state of browser bookmarking is a bit of a shame, interfaces are restricted and bloated with non-functional features. At first I switched to Pinboard [0], but after a while I realized even that was not enough for me and wrote Grasp [1], browser extension to capture stuff directly in org-mode.
> Examples of things that should not be Chrome bookmarks
Yep, eventually reached exactly the same conclusion! I often want to add private notes, more context etc and it's just not compatible with standard bookmark solutions.
> Taking Notes
Also big fan of notetaking, I basically write down any remotely meaningful thought if I don't have time to exercise it at the moment (via org-capture). Also using same file for everything and just processing it now and then. Same for ideas for new projects, blog updates, etc -- these are just entries in org-mode.
> ..it requires some discipline. I tag and annotate new entries, that I added via the web clipper, about twice a month
Yep, doing same, going through clipped links in org-mode, tagging and putting priorities. Then I can sort by priority and start reading/refiling etc.
Eventually non-important stuff just sinks down as I don't have time to process everything, but at least it's searchable.
> For example, it is quite easy to search through questions you have answered on Stack Overflow. It is also not a problem to go back to your Hacker News posts or search through projects you starred on GitHub.
I actually find that even though in theory it's easy, in practice sometimes you don't remember where exactly you need to search for something, so you end up going through 'check reddit saves', 'check HN saves', 'check twitter faves' cycles, etc. So I'm automatically converting these into org-mode and they are searchable as any other org-mode file. I describe it in more detail here [2].
P.S. Great and clean design, especially the sticky navigation on the right! I might borrow the idea :) By the way, I tried in responsive mode and it seems to disappear alltogether, perhaps you could display it on top if the screen is too small?
Thank you for the nice feedback, much appreciated!
> Ad using native bookmark sources, such as HN/Twitter/...
You are right, this is not ideal and it also happens to me that I am no longer sure where to look first. On the plus side, I do find it eventually, even if it's only the second native source I check. Atm, it seems to me that this is still better than duplicating everything in my other bookmarking layers. I will read your posts to find out more about your system, honestly don't know much about org-mode yet.
PS: Very glad to hear that! Especially, since I don't do much frontend work usually. I will think of a way to include a ToC also on small screens. Please, feel free to steal any idea you like. Then I don't have to feel as bad for looking into some of yours. I really like the pilcrows next to the headings and the dotted lines for highlighting sections on your site.
I have been perpetually sitting on the fence about Devon.
My use case: before Google+ folded I saved all my history there, because 99% of this was links to pages on stuff I found interesting/fun/relevant.
So now I have a large amount of urls (some will probably be defunct by now but it is not relevant). What I would like to do is to feed these urls to something (DevonThink?) that could access the article, and index it so that if - as an example - I want to prepare a RPG campaign on modern day pirates I can just write in the search box ["pirates" "shipping" "modern"] and hopefully get a list of web articles that are relevant.
Now, I know that Devon has some sort of automatic indexing/clustering facility for documents you have on your HD, but it is not really clear to me if it works also with stuff you only have an URL to.
(If anyone has some alternatives to suggest I will be very interested - I toyed with the idea of putting together an ElasticSearch VM for this but it remained on the backburner for years).
For managing a personal knowledge base, I highly recommend Tiddly Wiki[1]
It's a self-contained Wiki/Notebook/Journal with tags, it works instantly, and all of it is in a single HTML file with magic JS in it. Or you can use it with a server.
I strongly support this recommendation. Tiddly Wiki has enough features and extensibility that I wasn’t tempted to write my own bibliography management system after discovering it. Yes, I know there is dedicated software for that but they didn’t fit how I want to structure my notes.
How do you recommend running it? The install guide lists a half-dozen different possibilities including PHP, Node, Ruby which all look to be vastly different solutions.
You just download the HTML file and open it to 'run' it. Then, once you've made changes, use your browsers "save as" feature to save a copy. Everything is in browser.
Other options do things like public hosting.
Personally, I just use the HTML file with the Timimi Firefox plugin installed for autosaving the current file.
I use TiddlyDesktop and open with it the HTML file that I downloaded from the site. The desktop app takes care of saving the file when you finished editing a post and save it.
That's the save method I'm using since one year now. It perfectly fits my needs as a single user of this wiki.
No infrastructure to set up, it works out of the box.
As a former ikiwiki user I won't go back.
I use a tool called zim wiki. Its less of a wiki and more of a notes app with a tree sidebar for navigation and linking between pages. Everything is stored as txt files which works awesome.
Joe Armstrong of Erlang fame was a fan of Tiddly Wiki, and there's a video of the Tiddly Wiki creator and him enthusing about it here https://www.youtube.com/watch?v=Uv1UfLPK7_Q
Jeremy Ruston the creator, says a 100MB TiddlyWiki works fine on a 2013 era laptop, and that browsers are some of the best optimized VMs available to work on (in the video I linked in another comment).
I would love an alternative that is free/and or possible to self-host.
However, I do not see how this compares to Notion. How do I edit this HTML file on my phone? How do I add bookmarks to it via a hotkey? What is the equivalent of Notion's databases? Does it have Slash commands? This list of questions could go on for much longer. I really don't want to sound like a Notion shill, especially since I also use other tools (e.g. GitLab) for my knowledge base, but many of the suggested alternatives lack basically all the features that I describe in my post.
I am not sure you understood how Tiddly Wiki works.
>How do I edit this HTML file on my phone?
Open https://tiddlywiki.com/, click the pencil icon on any of the entries. Edit away. The edits can persist with a self-hosted solution. Or you can just download the edited HTML file.
>Does it have Slash commands?
You probably mean "keyboard shortcuts". Yes, a plenty. Get started here:
This doesn't cover all of it; there are shortcuts in the editor (Ctrl+B, for example to make text bold), and they are fully customizeable. Everything can be customized.
>How do I add bookmarks to it via a hotkey?
TiddlyWiki is tag-based. You can add tags with a keyboard. Tags define all the structure (table of contents and search).
>What is the equivalent of Notion's databases?
I am not familiar enough with Notion to answer this. If you ask "can I do ____ with TW", I can tell you.
>but many of the suggested alternatives lack basically all the features that I describe in my post
Sure, because they are different products. Feature-by-feature comparison doesn't make sense; if you want Notion, you need Notion.
If you want to organize your knowledge, Tiddly Wiki is one very fine tool to do that.
TW is not a CRM, it is not a Calendar or Google Sheets alternative (as Notion claims to be). It is not an all-in-one tool.
What it is, it's a personal knowledge base tool - and that's what the title of your post says. At that, it can do better than Notion. Or worse. User's call.
>This list of questions could go on for much longer.
The same can be true going from the other side. Here is one:
How do you access your knowledge in Notion 10 years after Notion-the-company goes the way of the dodo?
With slash commands, I do not mean keyboard shortcuts. The lack of customized keyboard shortcuts is one of my issues with Notion, but their slash commands are great.
It basically means that you can access all kinds of features by just typing ahead after a '/' without knowing a precise shortcut. E.g. if I want my text to appear blue I would type /blue. I would also get there with /color or even by just starting to type either of those words and Notion will make suggestions. Slack, Confluence, and others do it in a similar way.
I think I do understand Tiddly Wiki works, as I have quite a bit of experience with Wiki systems, didn't use TW in particular though.
Some of the features you put aside as not needed for personal knowledge management are actually very nice for personal knowledge management. You might want to look a bit into my post to see what those databases can do ;)
> How do you access your knowledge in Notion 10 years after Notion-the-company goes the way of the dodo?
This is a very valid concern, that was also raised by some other commenters. In my opinion, Notion's exporting capabilities are quite good and I use them regularly. However, I do hope that we will get more automated and sophisticated third-party backup solutions when they have an official API.
It’s all I need, I have tens of thousands of lines of notes at this point, can search them (text, regexes, tags, whatever) essentially instantly, can export to html or slides or pdf or $format, will never go down or close down or disappear behind a paywall, it’s perfect
I also love that since it's plain text, I can still access it anywhere I want to without having to use Emacs (despite the limited functionality without it).
Accessible mostly everywhere, but really exploring, manipulating content across mobile devices and operating systems gets pretty frustrating. Love it in theory, but DynaList is just so simple to use anywhere without a second thought.
Dynalist hits the right spot for me to manage my current work (tasks, quick notes…). Simple but flexible, extra features not overwhelming you if you don't need them.
But their mobile app isn't great.
For Links/Bookmarks I use Google bookmarks - https://www.google.com/bookmarks/
which surprisingly has not been killed and looks like might be left alone by Google.
Other bookmarking services delicious.com & posterous etc all died in due time.
By using browser extensions and a 3rd party Mobile App. I am able sync/store all the links. Though I would love to be able to sync bookmarks to my org-mode notes. Looking at org-capture now to see if that could possibly work (https://github.com/alphapapa/org-protocol-capture-html)
I also love org-mode, but I have to admit that asciidoc and asciidoctor has been growing on me, especially for anything I want others to contribute to.
Bummer. I signed up a few weeks ago (free), and have been enjoying it tremendously. To the point where I'm glad they'll take my money bc it's more than worth it, and I want to feel confident I can rely on it as a customer.
I love writing an blogging. Started reading into org-mode recently. Wish I had a cleaner setup with my own website. I am pretty satisfied with fox's bookmarks export though. The json it generates is easy to work with and carries lots of useful meta data. I built a little processor to make my bookmarks easily readable and organized here: https://l-o-o-s-e-d.net/bookmarks
I've been using Zim [1] as a knowledge database for around 2 years. I don't like depending on online solutions which may suddenly disappear.
Together with some plugins for managing tasks, git and some script for synchronizing repositories, it has been working great.
There are some limitations on this approach (ie, two persons editing the same page concurrently is a no-no), but for my use-case it works perfectly.
I find it much more intuitive than org-mode, and the fact that it auto-generates a "global" task list based on items spread across the whole notebook makes it much easier to prioritize things.
I like Zim. For my Batcave I switched from Tiddlywiki to Tomboy to Zim to Notecase Pro and finally switched from that over to files and folders using the Geany editor and a sync service. So far this is the combination that seems to survive software, OS, and environment changes best. Even on Android it's not perfect, but it makes me happier than e.g. trying to use Zim on Android :-)
Off topic sorry but why bother with the cookie agreement on the overlay footer like this? You are not giving users the ability to opt out so this just annoys people without making the website GDPR compliant.
I will soon disable this for non-EU visitors. A more sophisticated cookie agreement banner is not so trivial to do with static sites. As long as everybody is doing it this way, even big corporations, I will keep this approach for EU visitors.
My blog itself is not open source. I thought about this for a while, but I think the pressure to keep everything cleaned up, documented, and organized is not worth the benefits in this case. If you have some specific requests, feel free to get in touch and I might be able to help you.
The problem with bookmarks is you're never going to have time to reread the whole thing. I make a summary and put the source as a hyperlink so that if I summarized wrong I can go read it again. Summaries are better because you can organize summaries and have everything you know about enums together in a few pages, rather than 20 bookmarks.
You don't need to re-read all of your bookmarks. They're basically a hash table keyed by some concept or some problem that you either list elsewhere or remember in your head, that you can quickly use to pull up the full detail content for the given key. Much like the actual physical bookmarks that are just quick jump references to content you already know the summary of.
Don't use bookmarks as a 'I might come back and read this later'. Use them as 'I will need this later'.
I fully agree with this sentiment. It is very important to not have the feeling that you need to revisit all your bookmarks. In that case, you will eventually feel overwhelmed and a bookmarks library will do more harm than good.
I’d like to look into the features of notion more, but I can’t help but wonder why it isn’t more common to see people using a local SQLite database, or any other self hosted database, to serve as a personal knowledge base management system. A lot of the paid solutions I’ve seen seem to bend over backwards to offer a limited subset of the features that are trivially available in an actual database.
Has anyone tried out using a personal database like this?
I have a postgres instance running somewhere that has a history of stuff that I've recorded.
The reason why one might choose not to do this is that a) it takes a bit of work to get syncing across devices, and b) it takes a bit of work to get markdown, images, search, videos etc working.
It might be an unpopular opinion, but I use Notion and think that it's user interface is quite bad. I doubt they will go in the right direction for what I need. I moved to Notion from Evernote (spammy, terrible, no innovation for years) hoping for a better experience, but I am still unsatisfied. I will soon be looking for another solution.
Twinkle Notes (currently in beta) is close to what you just described. It has a local encrypted sqlite3 database. You need a host (paid) to sync across devices, otherwise it's free to use on any device: https://twinkle.app
Please reach to us at support@twinkle.app if you need a free token to try syncing.
During my first weeks of Notion usage, I was repeatedly baffled that I never thought more about personal usage of relational databases, as they make so much sense for organizing notes, and many other things.
Having Notion, and also other alternatives, this does not seem viable anymore to me. For example, when I want to edit the "Post" column of my bookmarks database, where I relate blog posts to bookmarks, I just start to type and Notion will suggest matching posts. I can not think of any SQLite frontend that offers such usability features, which is not surprising, as they are clearly not built for using them as personal knowledge base GUIs.
Yes, but there are no good tool to do it.
In my case i need to store a a lot of structured data and files. So i need a Document manager / Database. It's hard...
I started with excel a lot of year ago, migrated to access and now i'm trying nixoxdb for android (the only other acceptable alternative was mobidb).
The main problem is the ease to use, you need to create a table for everything you want to store, and nothing support an hybrid free-form AND structured data(while also been able to store files), so the solutions with good text editing are terrible at structured data and viceversa.
Other problem are the ability to sync or even open the database in more than a couple of platform and the ability to access the file embedded in the database from other application (there are others but this are the one that bother me the most).
I'm also evaluating notion, but i really don't want a cloud service since i store basically everything inside my main database and the ability to access years from now is a MUST.
Also i still not have a good solution for my email that are stored mostly in thunderbird and only some message are "exported" to my main database.
I'm thinking of building something myself based on sqlite, .net, dokan/fuse and something similar to syncthing. But it's a big project and i don't have a lot of time
Devonthink[1] seems like a good personal database, but it's Apple only. I'm in a situation where I'm trying to coordinate a few academic projects with people from multiple institutions and I have yet to find a tool that supports our need. If Basecamp had better integration with Google Docs and Google Sheets, it might work. However, the need to have multiple people review and comment on MS/Google Docs leaves the team with project artifacts scattered all over everywhere. I'll have to take a look at Notion.
I did. I created a simple html form to input/edit/enter queries, and I had an html index holding common database queries. It worked extremely well as far as that goes, and it only took 90 minutes or so to get it working, so really a trivial setup cost.
There were two things I didn't like about it. I realized that browsability is really important - an SQL database works well if you know what to query, but it's just easier to have an old Yahoo-style index. I went with a wiki approach instead. The other thing is that it's easy to use version control with a pile of text files, but not so much with a database.
These days I do have some things in a "database" but that is a big text file that I query using Linux tools like grep and sed. Does everything I need and plays well with version control.
This reminds me of SQLite's own Fossil-SCM product [0]. It's a source control manager with wiki and built-in web interface all stored within one SQLite file. They use it to manage the source of SQLite itself.
I played with it briefly a few years ago, and just tried it again now. It's definitely not geared towards note taking the way other apps in this thread are, but it could have value as a self-contained and portable wiki/kb. I'd be interested to hear of people's experience with this, either for its main purpose or even just as a note taking system.
I would love to get a Surface and use OneNote -- especially because it has excellent support for hand-written notes (and I much prefer hand-writing to typing).
However, I'm hit in the face with two problems:
1. Uncertainty about data export. I don't want to be stuck depending on a third-party proprietary solution for something so important to me. (That said, Microsoft is typically quite good about supporting products for a long time)
2. I actually find Windows unpleasant to use, and much prefer linux. (Arch linux + i3/KDE happens to be my preferred setup).
Re 1. I would really not recommend using OneNote. I recently closed an Office 365 Business account and tried to move my notes on similar topics, e.g. pros/cons and how-tos of using different tools, from a group OneNote page to my own and it was very difficult to do so.
The documentation isn't very good (it's unclear which version) and I ended up having to install the local version on my Mac and manually copy/past the pages I wanted because the copy notebook dialog didn't work with notebooks that have many pages. The worst part is not being able to access the raw data and simply copy that from one place to another.
I've switched to markdown files on Sublime with the Markdown Editing package for now, because they are easier to read and edit, portable and I can convert my notes to wiki pages, blog posts or documentation very easily. I'm still not 100% happy because of the different flavours and how pandoc converts files differently sometimes (especially lists).
I'm not sure if learning a new system is worth it, given the simplicity of markdown files - do you have any good resources?
My goal for now is to have a simple way to record (the files) and manage (the folders) notes, collaborate (I currently use gitlab repositories) and potentially use them later, e.g. by copying relevant parts to make posts, documentation etc, or by writing a script that finds relationships based on the content (something I've been thinking about for links I save in Safari or Firefox too - a script that scrapes the pages and builds a tree to organize links automatically).
Maybe a tool that could navigate the files, extract headlines and content to figure out relationships out of the files would be ideal to help me compose new documents and posts.
I am sure that there are things OneNote is even better at than Notion.
However, some of the most important concepts covered in my post are databases, relations between those databases, and editing workflows (markdown, slash commands). As far as I can tell, OneNote does not even attempt to do these things.
I disagree completely. I wouldn't trust anything important to OneNote, and it's hard to get things out of it again, too. Your notes are basically locked in. You can sort of print to PDF, but it uses different pagination to the original PDFs you annotate! PDFs are inserted as bitmap images instead of vector format, and text is only sometimes searchable.
OneNote has a really scary number of obvious bugs, with pen input especially. Writing disappears immediately or jumps left and right, the document randomly stops updating the screen, the program gets stuck in a high-CPU loop making huge input lag and needs restarting, selecting writing doesn't select what you draw around but a smaller area inside that, etc. It's incredibly frustrating and difficult compared to the Android pen-based notes app I use, Squid (which has a much narrower focus, but is much better at it).
Export is really the weakest area of OneNote. As far as I can tell, there is no way of batch exporting that also exports embedded attachments (for example, if you embed a PDF or Word document into a page rather than "printing" to the page).
My biggest issue with onenote is that you can no longer use local notebooks with the new versions. I'm stuck on onenote 2016 or all my local notebooks become read only.
One drive is blocked at work so onenote is effectively useless to me now.
For now, I've started moving my notes to a static site using hugo. It's a bit less convenient, but until I settle on a better permanent solution at least I have text files I can move around and manipulated in bulk if needed.
Yes, lock-in is a big problem we have with Onenote in my company, since Microsoft decided not to offer offline notes anymore. Lots of people have their notes locked-in that format and we cannot store that data on a third party server.
I've mentioned this before, but I'd love to have a note taking app as convenient as EverNote, but that lets me insert code cells a la Jupyter notebook. I can barely think without Jupyter any more. Maybe that's not a good thing, but it's sure handy.
Not OP but I also use jupyter for almost everything.
Jupyter notebooks are great for capturing thought process and showing intermediate state of whatever it is you're doing, which means coming back to them even years later it's a lot easier to recall your own thoughts and not have to reverse engineer a monolithic block of code.
I'm a physicist by day, and work for a company that makes measurement equipment. A lot of my work is quantitative, so I'm frequently "thinking" with numbers, equations, graphs, etc. While I'm not employed as a programmer per se, I use scientific programming quite heavily.
Jupyter has helped me take notes when I need to "think" quantitatively, and be able to look at answers quickly, plus maintain a record that I can read later on.
Try Joplin, its a direct replacement for EverNote, can import its database, and has active development to add more similar function. The only one that I'm missing right now is OCR, not sure if its coming, though. I'm using dropbox to sync my phone with laptop clients.
+1 for Joplin. I've been using it for over a year now and I love how I maintain control over my notes because in the end, they're just Markdown stored on Dropbox. And you can change backend at any time.
Hmm, what is it you don't like about code blocks in Notion? Don't think it can get much more convenient. The only things I miss is auto-format and maybe automated language detection. I don't know of any web-based tool that can do those things though, except some that do only those things.
I have every interesting article or web reference I've ever read since 1997, sitting in a PDF file alongside tens of thousands of others.
It is so convenient to be able to "ls -l | grep subject" and find all the pages related to that subject, and then to mine the data out of the PDF's for further reference.
Or, I just open the PDF and gain the knowledge again.
This works very well and doesn't require an active Internet connection. Every year I spend a few weeks up in a mountain retreat, going through the collection and getting an idea for the variety of topics I've read about for the year.
Its pretty neat to see, also, the changes in my interests over the years - and as well, its pretty wild to go back to the sites after time and realise I've got the only copy of the site - because its gone now.
I'd prefer to see more tools for manipulating PDF's become mainstream in the future. I can't recommend highly enough the productivity I've gained by being able to organise things this way. Its like having my own private Internet of 80,000+ pages, tailored to all the things I'm interested in ..
I've always thought of PDF as an opaque format. How do you search and/or browse your collection? Does the subject show up in grep [e: without being diligent with filenames]?
Many PDFs have compressed content streams, plain text utilities only see metadata in that case. Cached, compressed text-only output is usually tiny, and can be zgrep-ed.
pdfinfo shows document metadata (title, subject, keywords and more), but it's quite uncommon for these to be useful (Adobe and LᴬTᴇX-sourced PDFs tend to have this data).
I always wonder why nobody else does it, but its a very pleasing experience to have an archive like this - and it has proven very valuable in production crunches and peaceful moments of downtime, alike.
Another thing it allows me to do is generalise my reading in real time - I know I can always come back to the subject of the PDF in a few days time, or whenever really, without needing to know where to find the details: I just grep my file tree, and mine the data that way.
Key thing is, though, that when saving the PDF, I always check that it is named a decent subject, derived from the <title> tags .. if there isn't one, I add it myself. This is the only place I pay attention to 'tagging an article' - in naming the file of a poorly-named page, but if the page title fits the subject, I don't change a thing. And it means I have a huge set of data also in the filenames, not just the contents themselves .. Periodically, I use 'detox' on my PDFArchive/ directory, also .. this helps with consistent-naming for regexes, and so on.
I keep getting the itch to put all this into a DB with proper indexing or whatever, but .. actually using plain ol' shell tools is turning out to be just as productive. Its a big set of data, but plain ol' files and pipes is proving to be all I really need to get the data mined ..
I use markdown-clipper for Firefox, it does an OK job but sometimes requires a bit of fixing up. It does not save images, so I have to do this manually; I use VSCode for Markdown, which has an extension for copy/pasting images to Markdown, it inserts a link and saves the image in a subfolder.
A little friction, but if that's too much, then the article is not worth keeping. But yes, someone with the time on their hands could contribute to markdown-clipper, saving images locally.
I'm also interested in how you go about this. Evernote and Notion have good clippers, but I'd rather not rely on them. I tried to use markdownifier[0] for a while, but images are hotlinked and will break if the site goes down. I should really just write my own clipper that does this, but I haven't found the time yet.
I've always disliked pdfs - they just seem too, i don't know, obscure or scrambly/messy, maybe? However, i have recently seriously considered beginning to archive instead of bookmarking. I would prefer archiving as plain text (for easier searching and compressing, etc.)...but enough content that i consume (and wish to archive) contains media (mostly imagery, not really much video), so i guess printing to pdf is more convenient for the "capture" phase. The one concern i have is storage...I mean how much storage would bloated pdfs have to use up i wonder (as opposed to, say, leaner plain text)?? Legitimate concern, not trying to be facetious. I've also heard there's some sort of web archive file format - .warc or something? If so, i hope that becomes an established thing - maybe bst of all worlds...at least for this archiving use-case.
Why would you store all of this valuable information in a proprietary tool? What happens when notion shuts down, jacks up prices, makes feature changes you don't like, etc etc? Make some attempt to migrate all this data? It's gonna be hard to do this in an automated way, if they even have an (export) api for this.
I'm using (vim)wiki/gollum and I feel much safer about my data.
Fully agree. I was turned off notion for the same reason and reverted back to self-hosting dokuwiki. A much more limited feature set, to be sure, but at least I'll still have my notes 10 years from now.
Obviously, you have a point. I am indeed a little bit worried.
Some things that ease my mind:
Notion's export and backup features are quite good. You can export everything in various formats, markdown being the most relevant for me.
This only becomes messy with complex databases that use formulas and are related to each other. You would not have these with local solutions in the first place.
Notion is already profitable and growing rapidly, I do not see them shutting down in the foreseeable future.
However, the other concerns are relevant.
The API is supposed to be released soon.
I intend to either build a backup workflow myself or use other tools that will get developed then.
Have you actually done an export of your Notion workspace?
I looked at their options before using Notion and was happy they had a full export available - but then I actually went ahead and used it because they are putting zero focus into improving the performance of the apps - and the export format is a horrific mess!
They split out code blocks from your notes in separate notes, the filenames are a mess and images aren't saved inline (as they could be with something like base64).
For workspaces of any reasonable size, you're going to be doing a lot of work to make the export actually useful outside the context of Notion - which should be obvious given how they treat things as blocks (and all the abstractions that come with that make it into their underlying data structures)
They've also been saying the "API is coming soon" for about a year now.
I have not yet done a complete export of my workspace. I do however often export specific databases, notes, etc.
My post describes how I use it to draft my blog posts, where I also use the Markdown export feature. The code blocks are fine and within the same file, so I am not sure what you mean there.
Images are not ideal, but I don't know how it could be done much better. Everything is exported into a .zip and the images are also there.
Admittedly, my long term hope for a reliable and automated way of backing up everything relies on the API. Their promise that it will be released soon is a little awkward by now, but I am sure that it will come eventually. I don't think its true that it is already a year since they first said it would be here soon.
Never used notion, but personally I'd prefer a machine readable and raw export, at least you can assemble your stuff back in any way you prefer. More annoying when you have to scrape data you're interested in from htmls.
But yeah, lack of API is a major problem, and main reason I haven't even considered trying it yet.
I think Markdown does qualify as machine-readable. There are very good libraries to do all kinds of things with it.
Edit: I guess you meant human-readable as opposed to things like base64, and not as a critique against Markdown.
By now, there are several unofficial APIs at least, for read-access they are quite good apparently, did not yet use one myself though. E.g. https://github.com/kjk/notionapi
I also found https://dabble.me/ the other day which seemed like a nice slot in for quick thoughts with the ability to email to your journal. I've heard that journaling manually works better but I really dislike writing unless necessary.
One of my biggest gripes is with how many places I have my data in currently, and as a dev, I should have done something already but that's another story.
I'll definitely have a look at Notion and org-mode having read the feedback here along with a few other mentions elsewhere.
I was a paying EverNote user for years, but they stopped evolving while other NoteTaking apps didn't. I forget which feature they removed that was the last straw, but the recipe front-end was hugely important to me.
I have a single God ReadMe file in which I try to record all the facts that I should know and 'recipes' that I might need to follow again. Such as that shutting down windows does not dispose of kernel state whereas restart does. Or how to do a certain task on the income tax site or how to set up an odbc datasource in Windows. I've been keeping this file for years and I periodically read through it as much to remind myself of what's in it as anything else.
It's saved me a lot of frustration when finding myself having to do things again that I know it took me ages to figure out the previous time but I have since forgotten.
A tool is great for collaboration, but how much do you really need for a personal KB? And how much effort do you want to spend migrating?
Then there's vendor lock-in, learning curves, etc.
A well-organized directory structure with plain text files gives you a data store that won't become obsolete and can be used on any OS. I use CLI tools to search (e.g., grep) and that's all the features I need right there.
Does anyone else use plain text notes for everything?
I've tried to optimize things so that while working, I can get things jotted down with close to zero resistance. If the barrier of entry is too high then I don't bother doing it.
That means often being able to spawn a new terminal, write a command and be done with it. Or to be able to pipe something from my clipboard into an auto-dated file.
This is how I use Sublime Text cause I dont have to actually save new files it saves them for me. It loads instantly so its perfect. I paste in code snippets and what have you.
I dont use ST3 for proper dev as much cause I use JetBrains IDEs more.
I still would prefer something better for my more permanent notes.
I use emacs org mode which stores every thing in a text file.The text file is fully human readable even without emacs. Its has few rules that makes the collapsing the parts of the text file. One of the main rule is any line that starts with asterik followed by space ("* ") is interpreted as heading and the whole heading can be collapsed.
Yep, zero resistance: plaintext, with context, metadata, keywords, minimal formatting, grep and a few scripts. The most important thing to remember is that I don't remember everything, so write it down, and use sensible filenames/hierarchy/tags/metadata so it can be found easily.
The less temptation to play with CSS, formatting and markup the better. One day it'll all go in a self-contained minimal wiki. Maybe.
On my android phone I use an ancient app called "unote" (installed from an .apk which I keep, it's long since disappeared from Play). Simple, hierarchical plaintext notes, kept in a local sqlite file. Sadly no search, but that just means I have to be disciplined about what, where and pruning.
I'm experimenting with a single 500K+ line-long flat text file for EVERYTHING, from dream journalling to work email drafts to blog posts etc. -- I've even copied math from my home whiteboard in latex -- screw rendering, I'll transcribe them back when I need them.
At one point I was trying to do a Luhmann's Zettelkasten with one note per txt file, internal linking, etc. I concatenated every piece of txt content I had. It's much simpler.
I have no system anymore. I try to create internal "hashtags" that are ctrl-F searchable. The goal is having something that's always open and at hand. I got plugins for autosaving when the window loses focus, autoloading when changed in disk and it syncs to Dropbox.
I cobbled together in Notepad++ (which has a GUI for this) a small syntax coloring definition that sort of matches my spontaneous habits from old salvaged text. I intend to let this evolve too. E.g. I'm using {braces} to have a little collapse button that hides pieces of text.
I came to the same conclusion: a massive flat text file in emacs. I was using orgmode for a bit, but it was too convoluted. I have a very simple markup to identify TODO items with [ ], and I nest contiguous items with { }, making it easy to navigate chunks with emacs. I then have 3 keyboard macros to mark/unmark and move the TODO items into a DONE section. It's simple enough I can recreate the system from scratch in a few minutes when I'm at a new terminal.
Tangentially related, but after years of trying out (and struggling with) hundreds of calendar and schedule apps, in the end what was best for me turned out to be a simple text file that I keep updating and look at each day. A line on it represents a day and begins with (dd/mm), and events for the day are plaintext separated by commas. Each day gone by gets erased. Absolutely zero friction and I can see an entire month of stuff at a glance so I'm always aware of what to do. Can't believe it took me so long to go back to such bare basics, and now I strive to reach this level of brutal basic frictionless simplicity in my own toy projects.
Brutal basic frictionless simplicity is a good way to put it haha.
That's basically what this script does except it uses separate files. You run `notes something really cool and important` and it creates a YYYY-MM-DD.txt file for you in a configured directory and appends the arguments to the file on a new line. So every day you get a new file.
Or, if you just run `notes` it opens your configured $EDITOR for that specific day so you can go free form.
Yes; I've been using Notational Velocity for years. It gives just enough automatic structure (it's almost like a box filled with index cards, though the size of the note is unlimited) without having to actually fiddle with files. It's effortless to create new entries and incredibly easy to find things again later.
186 comments
[ 2.8 ms ] story [ 178 ms ] threadI'm hoping (and optimistic) that the distributed web brings with it some epiphanies about how to do better local knowledge management!
More specifically in the section Changing results on the fly
I would love a system that treats all media the same as just a source of knowledge instead of being hung up on the source type i.e. is it a video, or a book mark or a pdf....
There is not much support for PDFs or videos though.
P.S. It's also relevant to bookmark management as well: you get to see not only notes on the bookmark, but also on the 'child' URLs, e.g. if you open someone's twitter profile you'd be able to see their tweets you've favorited. Or if you open some blog, you'd see posts from that blog you saved on Reddit.
It doesn't handle video/bookmark/pdf. If I have to save those, I put them in a separate directory and make a note of it in the text file.
It's pretty well organized, thanks to org mode. It has a section that is by date, like journal entries, and a section that's by topic. Since it's just a single file, it's very easily searchable. It will never "go down". I don't need to run a database (I tried using wiki software for the same purpose). I can even "link" different sections, by having plain text labels. For example, I can refer to "ZFS Setup 2009-01-01" which is another place in the document I can search for.
I can "jump" to sections by using the orgmode annotation. Searching for "* Computers" will go to that top level section.
I liked this system so much that I also use it at work. People are very impressed that I can find things in my "notebook" so quickly. It's just text search.
Websites disappear. Web apps disappear. Apps disappear. My text file, does not.
I've used a similar system in the past (though not based on org mode), but was too brittle if/when "links" broke or were altered. For an over simplistic example, if the references that you made to a video within your master text file(s) breaks because the video's containing folder name has changed even slightly, well that becomes annoying. For a small number of references, types of destination media types, etc., maybe not an issue, but after some point, it can become too much maintenance/annoyance. Is this an issue that you encounter? If so, do you have mitigations for avoiding/resolving the broken links? Just curious.
I've got quite similar system in spirit and aims, just instead of lower Notion layer, using org-mode. It ends up searchable, synchronised with all devices, available offline and with great tooling for organizing and processing information.
> While documenting my configuration, especially my command-line workflows, I identified some shortcomings
Can't agree more! I've considerably simplified so many things while writing up on my setups and publishing code -- I guess it's easier to sneak in unnecessary complexity when it's all in your head.
> They are, however, not well suited for keeping extensive bookmarks libraries.
Agree state of browser bookmarking is a bit of a shame, interfaces are restricted and bloated with non-functional features. At first I switched to Pinboard [0], but after a while I realized even that was not enough for me and wrote Grasp [1], browser extension to capture stuff directly in org-mode.
> Examples of things that should not be Chrome bookmarks
Yep, eventually reached exactly the same conclusion! I often want to add private notes, more context etc and it's just not compatible with standard bookmark solutions.
> Taking Notes
Also big fan of notetaking, I basically write down any remotely meaningful thought if I don't have time to exercise it at the moment (via org-capture). Also using same file for everything and just processing it now and then. Same for ideas for new projects, blog updates, etc -- these are just entries in org-mode.
> ..it requires some discipline. I tag and annotate new entries, that I added via the web clipper, about twice a month
Yep, doing same, going through clipped links in org-mode, tagging and putting priorities. Then I can sort by priority and start reading/refiling etc. Eventually non-important stuff just sinks down as I don't have time to process everything, but at least it's searchable.
> For example, it is quite easy to search through questions you have answered on Stack Overflow. It is also not a problem to go back to your Hacker News posts or search through projects you starred on GitHub.
I actually find that even though in theory it's easy, in practice sometimes you don't remember where exactly you need to search for something, so you end up going through 'check reddit saves', 'check HN saves', 'check twitter faves' cycles, etc. So I'm automatically converting these into org-mode and they are searchable as any other org-mode file. I describe it in more detail here [2].
P.S. Great and clean design, especially the sticky navigation on the right! I might borrow the idea :) By the way, I tried in responsive mode and it seems to disappear alltogether, perhaps you could display it on top if the screen is too small?
0. https://pinboard.in
1. https://beepb00p.xyz/grasp.html
2. https://beepb00p.xyz/pkm-search.html#other
> Ad using native bookmark sources, such as HN/Twitter/...
You are right, this is not ideal and it also happens to me that I am no longer sure where to look first. On the plus side, I do find it eventually, even if it's only the second native source I check. Atm, it seems to me that this is still better than duplicating everything in my other bookmarking layers. I will read your posts to find out more about your system, honestly don't know much about org-mode yet.
PS: Very glad to hear that! Especially, since I don't do much frontend work usually. I will think of a way to include a ToC also on small screens. Please, feel free to steal any idea you like. Then I don't have to feel as bad for looking into some of yours. I really like the pilcrows next to the headings and the dotted lines for highlighting sections on your site.
(as an alternative to grasp, because I'm on MacOS/Safari).
My use case: before Google+ folded I saved all my history there, because 99% of this was links to pages on stuff I found interesting/fun/relevant.
So now I have a large amount of urls (some will probably be defunct by now but it is not relevant). What I would like to do is to feed these urls to something (DevonThink?) that could access the article, and index it so that if - as an example - I want to prepare a RPG campaign on modern day pirates I can just write in the search box ["pirates" "shipping" "modern"] and hopefully get a list of web articles that are relevant.
Now, I know that Devon has some sort of automatic indexing/clustering facility for documents you have on your HD, but it is not really clear to me if it works also with stuff you only have an URL to.
(If anyone has some alternatives to suggest I will be very interested - I toyed with the idea of putting together an ElasticSearch VM for this but it remained on the backburner for years).
Here is my original request here, btw: https://news.ycombinator.com/item?id=18882167
It's a self-contained Wiki/Notebook/Journal with tags, it works instantly, and all of it is in a single HTML file with magic JS in it. Or you can use it with a server.
[1]https://tiddlywiki.com/
Big thumbs up from me so far.
The hardest part was setting up a CouchDB instance for the backend, but I found some instructions and fiddled with it until it worked.
Link: https://noteself.github.io/
Other options do things like public hosting.
Personally, I just use the HTML file with the Timimi Firefox plugin installed for autosaving the current file.
> Don't attempt to use the browser File/Save menu option to save changes (it doesn't work)
It looks like TiddlyWiki doesn't recommend that exact method anymore.
You "Save As.." that.
To save changes, click the red checkmark, and "save as..." the updated HTML over the old copy.
Switch to other possibilities if you use it enough to justify that.
https://github.com/djmaze/tiddlywiki-docker
I, a lifetime worth of knowledge in reasonable HTML of Tiddly Wiki would take less toll on your PC then the current CNN front page.
https://github.com/djmaze/tiddlywiki-docker
if you want to host on arm device, https://github.com/djmaze/tiddlywiki-docker/issues/15
However, I do not see how this compares to Notion. How do I edit this HTML file on my phone? How do I add bookmarks to it via a hotkey? What is the equivalent of Notion's databases? Does it have Slash commands? This list of questions could go on for much longer. I really don't want to sound like a Notion shill, especially since I also use other tools (e.g. GitLab) for my knowledge base, but many of the suggested alternatives lack basically all the features that I describe in my post.
>How do I edit this HTML file on my phone?
Open https://tiddlywiki.com/, click the pencil icon on any of the entries. Edit away. The edits can persist with a self-hosted solution. Or you can just download the edited HTML file.
>Does it have Slash commands?
You probably mean "keyboard shortcuts". Yes, a plenty. Get started here:
https://tiddlywiki.com/static/KeyboardShortcuts.html
This doesn't cover all of it; there are shortcuts in the editor (Ctrl+B, for example to make text bold), and they are fully customizeable. Everything can be customized.
>How do I add bookmarks to it via a hotkey?
TiddlyWiki is tag-based. You can add tags with a keyboard. Tags define all the structure (table of contents and search).
>What is the equivalent of Notion's databases?
I am not familiar enough with Notion to answer this. If you ask "can I do ____ with TW", I can tell you.
>but many of the suggested alternatives lack basically all the features that I describe in my post
Sure, because they are different products. Feature-by-feature comparison doesn't make sense; if you want Notion, you need Notion.
If you want to organize your knowledge, Tiddly Wiki is one very fine tool to do that.
TW is not a CRM, it is not a Calendar or Google Sheets alternative (as Notion claims to be). It is not an all-in-one tool.
What it is, it's a personal knowledge base tool - and that's what the title of your post says. At that, it can do better than Notion. Or worse. User's call.
>This list of questions could go on for much longer.
The same can be true going from the other side. Here is one:
How do you access your knowledge in Notion 10 years after Notion-the-company goes the way of the dodo?
It basically means that you can access all kinds of features by just typing ahead after a '/' without knowing a precise shortcut. E.g. if I want my text to appear blue I would type /blue. I would also get there with /color or even by just starting to type either of those words and Notion will make suggestions. Slack, Confluence, and others do it in a similar way.
I think I do understand Tiddly Wiki works, as I have quite a bit of experience with Wiki systems, didn't use TW in particular though.
Some of the features you put aside as not needed for personal knowledge management are actually very nice for personal knowledge management. You might want to look a bit into my post to see what those databases can do ;)
> How do you access your knowledge in Notion 10 years after Notion-the-company goes the way of the dodo?
This is a very valid concern, that was also raised by some other commenters. In my opinion, Notion's exporting capabilities are quite good and I use them regularly. However, I do hope that we will get more automated and sophisticated third-party backup solutions when they have an official API.
I have a single file called "engineering-notebook" and I store pretty much everything in it:
- bookmarks to resources
- personal notes
- personal tutorials/step-by-steps
- cheatsheets
It works fantastically for me
I use a separate file for non-technical stuff.
For Links/Bookmarks I use Google bookmarks - https://www.google.com/bookmarks/ which surprisingly has not been killed and looks like might be left alone by Google. Other bookmarking services delicious.com & posterous etc all died in due time.
By using browser extensions and a 3rd party Mobile App. I am able sync/store all the links. Though I would love to be able to sync bookmarks to my org-mode notes. Looking at org-capture now to see if that could possibly work (https://github.com/alphapapa/org-protocol-capture-html)
Chrome Extension - https://play.google.com/store/apps/details?id=com.amlegate.g...
Firefox extension - https://addons.mozilla.org/ru/firefox/addon/fess-google-book...
Android App for Google Bookmarks (3rd Party) https://play.google.com/store/apps/details?id=com.amlegate.g...
TODO alone is a game changer imho.
https://twitter.com/conaw/status/1214855473876201472?s=21
“Still working out specifics
Right now looking like it'll be $30/month - cheaper for annual - cheaper for students/non-profits/unemployed
I personally think unemployed autodidacts should always get student benefits - scam that they don't”
Not looking good.
Together with some plugins for managing tasks, git and some script for synchronizing repositories, it has been working great.
There are some limitations on this approach (ie, two persons editing the same page concurrently is a no-no), but for my use-case it works perfectly.
I find it much more intuitive than org-mode, and the fact that it auto-generates a "global" task list based on items spread across the whole notebook makes it much easier to prioritize things.
[1] https://zim-wiki.org/
https://github.com/luizdepra/hugo-coder/
My blog itself is not open source. I thought about this for a while, but I think the pressure to keep everything cleaned up, documented, and organized is not worth the benefits in this case. If you have some specific requests, feel free to get in touch and I might be able to help you.
Its your life, and your code. You can post it, and keep it messy if you want to.
Don't use bookmarks as a 'I might come back and read this later'. Use them as 'I will need this later'.
Has anyone tried out using a personal database like this?
The reason why one might choose not to do this is that a) it takes a bit of work to get syncing across devices, and b) it takes a bit of work to get markdown, images, search, videos etc working.
Having Notion, and also other alternatives, this does not seem viable anymore to me. For example, when I want to edit the "Post" column of my bookmarks database, where I relate blog posts to bookmarks, I just start to type and Notion will suggest matching posts. I can not think of any SQLite frontend that offers such usability features, which is not surprising, as they are clearly not built for using them as personal knowledge base GUIs.
I started with excel a lot of year ago, migrated to access and now i'm trying nixoxdb for android (the only other acceptable alternative was mobidb).
The main problem is the ease to use, you need to create a table for everything you want to store, and nothing support an hybrid free-form AND structured data(while also been able to store files), so the solutions with good text editing are terrible at structured data and viceversa.
Other problem are the ability to sync or even open the database in more than a couple of platform and the ability to access the file embedded in the database from other application (there are others but this are the one that bother me the most).
I'm also evaluating notion, but i really don't want a cloud service since i store basically everything inside my main database and the ability to access years from now is a MUST.
Also i still not have a good solution for my email that are stored mostly in thunderbird and only some message are "exported" to my main database.
I'm thinking of building something myself based on sqlite, .net, dokan/fuse and something similar to syncthing. But it's a big project and i don't have a lot of time
edit: couple of fixes
[1]https://www.devontechnologies.com/apps/devonthink
There were two things I didn't like about it. I realized that browsability is really important - an SQL database works well if you know what to query, but it's just easier to have an old Yahoo-style index. I went with a wiki approach instead. The other thing is that it's easy to use version control with a pile of text files, but not so much with a database.
These days I do have some things in a "database" but that is a big text file that I query using Linux tools like grep and sed. Does everything I need and plays well with version control.
I played with it briefly a few years ago, and just tried it again now. It's definitely not geared towards note taking the way other apps in this thread are, but it could have value as a self-contained and portable wiki/kb. I'd be interested to hear of people's experience with this, either for its main purpose or even just as a note taking system.
0: https://www.fossil-scm.org/home/doc/trunk/www/index.wiki
This video covers some of the things you can do with it in an academic setting but relatable. https://m.youtube.com/watch?v=JQD5c8A_D2g
0:33 - Math equations
1:45 - Replay text
2:45 - Ink to text
3:20 - Research tools
4:39 - Immersive reader
6:01 - Web clipper browser extension
7:12 - Save emails to OneNote
NB: https://www.microsoft.com/en-us/microsoft-365/blog/2015/11/1...
1. Uncertainty about data export. I don't want to be stuck depending on a third-party proprietary solution for something so important to me. (That said, Microsoft is typically quite good about supporting products for a long time)
2. I actually find Windows unpleasant to use, and much prefer linux. (Arch linux + i3/KDE happens to be my preferred setup).
The documentation isn't very good (it's unclear which version) and I ended up having to install the local version on my Mac and manually copy/past the pages I wanted because the copy notebook dialog didn't work with notebooks that have many pages. The worst part is not being able to access the raw data and simply copy that from one place to another.
I've switched to markdown files on Sublime with the Markdown Editing package for now, because they are easier to read and edit, portable and I can convert my notes to wiki pages, blog posts or documentation very easily. I'm still not 100% happy because of the different flavours and how pandoc converts files differently sometimes (especially lists).
My goal for now is to have a simple way to record (the files) and manage (the folders) notes, collaborate (I currently use gitlab repositories) and potentially use them later, e.g. by copying relevant parts to make posts, documentation etc, or by writing a script that finds relationships based on the content (something I've been thinking about for links I save in Safari or Firefox too - a script that scrapes the pages and builds a tree to organize links automatically).
Maybe a tool that could navigate the files, extract headlines and content to figure out relationships out of the files would be ideal to help me compose new documents and posts.
https://www.youtube.com/watch?v=fgizHHd7nOo
However, some of the most important concepts covered in my post are databases, relations between those databases, and editing workflows (markdown, slash commands). As far as I can tell, OneNote does not even attempt to do these things.
OneNote has a really scary number of obvious bugs, with pen input especially. Writing disappears immediately or jumps left and right, the document randomly stops updating the screen, the program gets stuck in a high-CPU loop making huge input lag and needs restarting, selecting writing doesn't select what you draw around but a smaller area inside that, etc. It's incredibly frustrating and difficult compared to the Android pen-based notes app I use, Squid (which has a much narrower focus, but is much better at it).
Export is really the weakest area of OneNote. As far as I can tell, there is no way of batch exporting that also exports embedded attachments (for example, if you embed a PDF or Word document into a page rather than "printing" to the page).
I'll give you two guesses as to why that is.
One drive is blocked at work so onenote is effectively useless to me now.
For now, I've started moving my notes to a static site using hugo. It's a bit less convenient, but until I settle on a better permanent solution at least I have text files I can move around and manipulated in bulk if needed.
it blows my mind that there is no good migration strategy for pc base OneNote databases to cloud based ones.
Jupyter notebooks are great for capturing thought process and showing intermediate state of whatever it is you're doing, which means coming back to them even years later it's a lot easier to recall your own thoughts and not have to reverse engineer a monolithic block of code.
Jupyter has helped me take notes when I need to "think" quantitatively, and be able to look at answers quickly, plus maintain a record that I can read later on.
https://joplinapp.org/
Yes depend up server and paid subscription. But use tag and sync across multiple devices
I have every interesting article or web reference I've ever read since 1997, sitting in a PDF file alongside tens of thousands of others.
It is so convenient to be able to "ls -l | grep subject" and find all the pages related to that subject, and then to mine the data out of the PDF's for further reference.
Or, I just open the PDF and gain the knowledge again.
This works very well and doesn't require an active Internet connection. Every year I spend a few weeks up in a mountain retreat, going through the collection and getting an idea for the variety of topics I've read about for the year.
Its pretty neat to see, also, the changes in my interests over the years - and as well, its pretty wild to go back to the sites after time and realise I've got the only copy of the site - because its gone now.
I'd prefer to see more tools for manipulating PDF's become mainstream in the future. I can't recommend highly enough the productivity I've gained by being able to organise things this way. Its like having my own private Internet of 80,000+ pages, tailored to all the things I'm interested in ..
pdfinfo shows document metadata (title, subject, keywords and more), but it's quite uncommon for these to be useful (Adobe and LᴬTᴇX-sourced PDFs tend to have this data).
Both come with xpdf.
It kind of amazes me that you've been doing this for two decades already.
Another thing it allows me to do is generalise my reading in real time - I know I can always come back to the subject of the PDF in a few days time, or whenever really, without needing to know where to find the details: I just grep my file tree, and mine the data that way.
Key thing is, though, that when saving the PDF, I always check that it is named a decent subject, derived from the <title> tags .. if there isn't one, I add it myself. This is the only place I pay attention to 'tagging an article' - in naming the file of a poorly-named page, but if the page title fits the subject, I don't change a thing. And it means I have a huge set of data also in the filenames, not just the contents themselves .. Periodically, I use 'detox' on my PDFArchive/ directory, also .. this helps with consistent-naming for regexes, and so on.
I keep getting the itch to put all this into a DB with proper indexing or whatever, but .. actually using plain ol' shell tools is turning out to be just as productive. Its a big set of data, but plain ol' files and pipes is proving to be all I really need to get the data mined ..
I "print" to markdown instead, even though pdfgrep sort of works I much prefer text files.
A little friction, but if that's too much, then the article is not worth keeping. But yes, someone with the time on their hands could contribute to markdown-clipper, saving images locally.
[0]: http://heckyesmarkdown.com/
Some things that ease my mind:
Notion's export and backup features are quite good. You can export everything in various formats, markdown being the most relevant for me. This only becomes messy with complex databases that use formulas and are related to each other. You would not have these with local solutions in the first place.
Notion is already profitable and growing rapidly, I do not see them shutting down in the foreseeable future. However, the other concerns are relevant.
The API is supposed to be released soon. I intend to either build a backup workflow myself or use other tools that will get developed then.
I looked at their options before using Notion and was happy they had a full export available - but then I actually went ahead and used it because they are putting zero focus into improving the performance of the apps - and the export format is a horrific mess!
They split out code blocks from your notes in separate notes, the filenames are a mess and images aren't saved inline (as they could be with something like base64).
For workspaces of any reasonable size, you're going to be doing a lot of work to make the export actually useful outside the context of Notion - which should be obvious given how they treat things as blocks (and all the abstractions that come with that make it into their underlying data structures)
They've also been saying the "API is coming soon" for about a year now.
My post describes how I use it to draft my blog posts, where I also use the Markdown export feature. The code blocks are fine and within the same file, so I am not sure what you mean there.
Images are not ideal, but I don't know how it could be done much better. Everything is exported into a .zip and the images are also there.
Admittedly, my long term hope for a reliable and automated way of backing up everything relies on the API. Their promise that it will be released soon is a little awkward by now, but I am sure that it will come eventually. I don't think its true that it is already a year since they first said it would be here soon.
But yeah, lack of API is a major problem, and main reason I haven't even considered trying it yet.
Edit: I guess you meant human-readable as opposed to things like base64, and not as a critique against Markdown.
By now, there are several unofficial APIs at least, for read-access they are quite good apparently, did not yet use one myself though. E.g. https://github.com/kjk/notionapi
I also found https://dabble.me/ the other day which seemed like a nice slot in for quick thoughts with the ability to email to your journal. I've heard that journaling manually works better but I really dislike writing unless necessary.
One of my biggest gripes is with how many places I have my data in currently, and as a dev, I should have done something already but that's another story.
I'll definitely have a look at Notion and org-mode having read the feedback here along with a few other mentions elsewhere.
A tool is great for collaboration, but how much do you really need for a personal KB? And how much effort do you want to spend migrating?
Then there's vendor lock-in, learning curves, etc.
A well-organized directory structure with plain text files gives you a data store that won't become obsolete and can be used on any OS. I use CLI tools to search (e.g., grep) and that's all the features I need right there.
I've tried to optimize things so that while working, I can get things jotted down with close to zero resistance. If the barrier of entry is too high then I don't bother doing it.
That means often being able to spawn a new terminal, write a command and be done with it. Or to be able to pipe something from my clipboard into an auto-dated file.
I ended up putting together this ~15 line Bash script which seems to do the trick: https://github.com/nickjj/notes
I've been using plain text notes since 2001, although I only recently created this script. So far so good.
I dont use ST3 for proper dev as much cause I use JetBrains IDEs more.
I still would prefer something better for my more permanent notes.
The less temptation to play with CSS, formatting and markup the better. One day it'll all go in a self-contained minimal wiki. Maybe.
On my android phone I use an ancient app called "unote" (installed from an .apk which I keep, it's long since disappeared from Play). Simple, hierarchical plaintext notes, kept in a local sqlite file. Sadly no search, but that just means I have to be disciplined about what, where and pruning.
At one point I was trying to do a Luhmann's Zettelkasten with one note per txt file, internal linking, etc. I concatenated every piece of txt content I had. It's much simpler.
I have no system anymore. I try to create internal "hashtags" that are ctrl-F searchable. The goal is having something that's always open and at hand. I got plugins for autosaving when the window loses focus, autoloading when changed in disk and it syncs to Dropbox.
I cobbled together in Notepad++ (which has a GUI for this) a small syntax coloring definition that sort of matches my spontaneous habits from old salvaged text. I intend to let this evolve too. E.g. I'm using {braces} to have a little collapse button that hides pieces of text.
Among all the organization/notetaking methods I've tried, this is really the best. No fancy app in the browser, but really effective and fast.
I've written a small article about it here:
https://illdoitlater.xyz/t/plaintext
How do you access your notes on your phone? How do you edit?
That's basically what this script does except it uses separate files. You run `notes something really cool and important` and it creates a YYYY-MM-DD.txt file for you in a configured directory and appends the arguments to the file on a new line. So every day you get a new file.
Or, if you just run `notes` it opens your configured $EDITOR for that specific day so you can go free form.
https://github.com/panos-zamos/zlog-cli#lt