108 comments

[ 2.4 ms ] story [ 180 ms ] thread
Amen. I currently use txt files & Tree 2 (mac app).

I’ve tried Quiver, Quip, & Workflowy and cannot find the perfect app to suit this space for my usage patterns. I’m almost 100% back to text files & sublime text.

The problem is really that every time something new and better comes around, it becomes a mess to port everything over, especially if it is conceptually different (e.g. tag-based or something else). Dying products also seldom give you a convenient means to export your content. So ultimately the only thing that can last for decades, be cross-platform, and not be tied to one company's product, is text files and directories. For now, throw it all on Dropbox, Github, or whatever you like. When Dropbox and Github die, the file-directory structure will be easy to port to whatever is next.
This has been my current approach for a while, after exporting everything out of Tiddlywiki when it looked like that was dying for a while (Browser support drying up, etc).

Now that it's been revived with Tiddlywiki5 and development's picked back up, I'm somewhat tempted to get back into it. Main advantage: Better crosslinking, rich content, etc. Main disadvantage: Less simple to edit, some tedium to export.

I'd love to have a Tiddlywiki-like frontend to a set of Markdown text documents backing it. (This is something that's been gnawing at the back of my mind long enough that I might have to buckle down and make it myself) Maybe Markdown + JSON for rich content data...

Where I need markup, I've been sticking to HTML or LaTeX syntax for the most part.

Partly because I hate Markdown due to its inconsistent syntax (that's my personal opinion), but also because it feels like a fad of recent years, much like a successor of the MediaWiki fad and the BBcode fad. I feel like in another 2 or 3 years someone will invent Markleft or Markright or Visual Markdown#.NET 2.0 or Dustindown that's 10 times better than Markdown, and then I'll have to switch everything over yet again.

HTML and LaTeX on the other hand, aren't dying any time soon due to their extensive deployment, and I'm confident that my syntax will be renderable for at least another 10 years or more.

Emacs and orgmode are probably your next step :)

orgmode is really a superset of markdown with PIM markup added.

It exports to markdown and a ton of other formats.

Cheers.

I like Evernote for a lot of personal stuff, though that's one of the "worser" systems OP lists. Atlassian's Confluence does very well at being a wiki and though you're hostage perhaps to your original layout / organization, at $10/year for self-hosted it's a very good deal. It can store and index PDF, Word, Excel, etc., documents very well. I don't think it does handwriting recognition.

If I were to evolve the product category I'd use Confluence as a basis.

In fact I'd want to toy with using Confluence (+ Postgres backend) as the basis for any I.T.-related product -- write plugins for user input, generate all reports to Confluence. I'm not sure what licensing restrictions might be but Confluence provides a lot of useful UI for $10/year (structured + unstructured) for any customer, and building on top of that seems like a no-brainer. Perhaps this is what others do with Drupal and other CMS products, though ... I'm not familiar with that model.

I'm surprised to read this. Confluence is the most unpleasant "wiki" product (its lack of support for redlinks prevents me from really considering it a wiki) I have ever used. Its built-in hierarchical structure for pages is excessively intrusive and undercooked at the same time, and the search is so bad that I have had it fail to pull up pages when I typed in the literal, complete title.

Like most Atlassian products, its only apparent notable quality to me is that it integrates with other Atlassian products, as well as being a semi-functional platform on which to pile plugins and customization. It's possible that there's a good Confluence install out there, probably administered by the same people who run that really nice JIRA instance that people claim to have used once.

Confluence is... okay. I think Atlassian have successfully managed to sand down a lot of the sharp corners on older wiki-type packages that tend to act as a barrier to the non-technical types - from my perspective, that seems to mean that my non-technical colleagues can be persuaded to contribute to documents held in Confluence markedly more easily than to contribute to a MediaWiki instance that we previously had.

So, while Confluence itself as a piece of software makes me grumble more often than not, I'm more than happy to trade that off against the easier sell to others.

My saving grace at work has been the tidily wiki (http://www.tiddlywiki.com). It's the easiest place to put one-off shell command line snippets, or instructions or whatever. It's search is fast, and the formatting is nice. The only downside is I have to remember to save and copy the downloaded file to google drive every time I edit anything.
Yeah tiddlywiki is by far my preferred way of storing this type of information. Coupled with a FreeNAS box I can't really think of a better way to store this type information.
I never managed to get tiddlywiki working right - pity, because I really wanted it to work for me, but couldn't seem to get to a stage where I'd trust it to have saved something when it said it had.

This was five years ago at least, mind - I've no idea what the problem might've been, and even if it wasn't user error, I suppose it might've evaporated by now.

(note to self: try tiddlywiki again soon)

Have you tried the firefox save plugin? You could use it to save to a synched google drive so you don't have to think about that.
I love love love tiddlywiki. I've been using it for probably 6 or 7 years now. It's simply amazing.
I commented on Discus the site, as well. I've been a long time user of DEVONthink. It can be used for many purposes, including document organizer, email, GTD, tagging, etc. My favorite feature has always been the Magic Hat, which will suggest under what folders a document should be filed. I clip most interesting web pages, and it is surprisingly accurate once you have established a few documents.

http://www.devontechnologies.com

I agree, it's great. I just wish DEVONthink had a better iOS client.
In my opinion, this is not a "problem" that can be solved. What makes something good varies a lot from one person to the next. In my case, I'd need a way to hold equations, computer code, PDF versions of papers, handwritten notes that someone else has given me, and a lot of other things. I want it stored in a Git repo. I want to view it in a browser. I want it to be convenient and I want it automated. The "answer" is a feature-filled scripting language that allows you to build a customized solution.
What about it would you want automated?

What kind of scripts would you add to your personal knowledgebase?

It's not so much what I would want automated as what I have already automated. As an example, consider how I handle notes that I leave for myself as I'm working. These are things like paragraphs to be added to a paper, ideas for a research topic, a list of things I have to do around the yard, or anything not directly related to the task at hand.

I have written a script in D (my programming language of choice). I have a web interface to all of the notes I've jotted down in the past. I can click a topic and see all notes related to that topic. I click the new note button, and it opens Emacs with the file that will hold the note. I type in a note in markdown, and as soon as I close Emacs, the note is converted to html. If I want to change it, I hit the edit button, Emacs opens up, and I make my edits. Upon closing Emacs, the html is rebuilt. Anytime I create a new note or edit an existing note, there is a commit to the Git repo.

That's what I mean by automated. I write scripts that handle everything except the content of the notes. I have written or am writing scripts for just about everything I do.

Confluence for the simple win
I've found that using google docs works well if you frame the document title in the form of a question or statement: - What is a JavaScript constructor? - making rvm permanent in rvspec - postgresql listing, creating and dropping a db

Some of the "answers" are just a single line. Then all I have to remember is either something in the title or the gist of the thing I need to know and google docs search provides me with a list of related stuff.

Evernote. Free and accessible across many devices. For me it has been best of the worst. It has been my one place to store all information. Evernote serves well as a general knowledge base, but fails for storing programming snippets and ideas.
How about Camlistore by Brad Fitzpatrick [1]:

"Camlistore is a set of open source formats, protocols, and software for modeling, storing, searching, sharing and synchronizing data in the post-PC era. Data may be files or objects, tweets or 5TB videos, and you can access it via a phone, browser or FUSE filesystem."

[1]: http://camlistore.org/

Camlistore is a nice append-orientated sync system, but I wasn't aware it included any data modelling or search?
Camlistore is where I'm expecting to go. They've got some basic things modeled in their world, and the rest I'll fill in using inspiration from schema.org.

My plan is to land PDFs of research papers, web bookmarks (and archives of those pages), ebooks, book scans, textfiles/markdown documents, imports of social networks (favorites from Twitter, mostly) and scans of written notes I make to myself in Camlistore. From there, sync to devices and publish to the web as appropriate.

I don't expect that I'll ever find the time to build this all myself, so I'm hopeful that either my data will be in a format that other Camlistore apps can read, or if I get the format wrong I'll be able to write a translator to whatever the Camlistore community decides to model for that specific data type.

Org-mode. A benevolent beast. Once one gets to learn it, it can be made to do anything. Also having a programming language shared with the runtime and other programmes for that runtime (Elisp and Emacs) is just priceless, it just lends itself to be customized in whichever way one would like. A major con for me, though, is the lack of decent iOS-Android integration.
I've been wanting to learn org-mode for years but I never get around to it. I use something similar, but I only worry that if I were to move to something else I would want it to be in the cloud so I could access it from devices and other laptops. Do you have a solution for this?
It's just plain text files, so you can sync it using whatever you want. Many people use Dropbox.
I have a bunch of org-mode files that I save into a Dropbox folder.

That gives me backups, and access from other computers / laptops.

Search is via ack or grep.

It theoretically also gives me access from my phone using MobileOrg[1], but I have never actually set that up.

[1]: http://orgmode.org/manual/MobileOrg.html

> ack or grep

C-c C-j. You're missing out :)

Cool, didn't know about that. Does it search multiple files in a directory? I use it for work logs and keep one file per week.
@spatten and @kristiandupont - orgzly is a nice android client. You can sync the org files with dropbox or with git. You could also run a linux locally and install emacs. A keyboard is essential though when you're using emacs, orgzly works ok with touch but is less full-featured. Just as with vim i only use a small subset of what's possible with org-mode: clock in/out, tables, nested outlines, tags and priorities, links to other files and images.
My main problem with org-mode is the lack of good display for math notations as well as multimedia (picture/video etc). Did I miss something that I can use with it?
You can display images in-line if you're using the graphical emacs.
AFAIK it's possible to use $$ -- $$ for math notation, it exports to LaTeX/MathJax etc, and there's a setting for making it to preview within the buffer (the function is "org-latex-preview-fragment"). And there is very good latex integration, support for latex blocks etc. I do not know the details tho, I'm not a maths guy.

If you set

  #+OPTIONS: inlineimages
you'll see links to local images inline in the buffer. IDK, if video is possible, but I think no.
(comment deleted)
This. Best organizer I've ever used!

I'm a Vim user, but thanks to SpaceMacs/evil-mode I feel right at home. Actually, it works so well that I've come to prefer evil-mode over the real Vim!

http://spacemacs.org/

Exactly, did you know you could install a special spacemacs org-mode layer?
I suspected as much, but I've never tried it.

Thanks for the reminder!

Hey, I found spacemacs a bit intimidating. How do you get started with it? I have plenty of text to write in the coming weeks, so I already have the substrate to practice on. I just need a good introduction to evil-mode+org-mode.
Spacemacs has pretty good docs: http://spacemacs.org/#learning-spacemacs

org-mode and evil-mode alone, without Spacemacs, work pretty well already. It's just additional glue to make it more seamless.

Thanks! For some reason I never saw this page before. I wasn't satisfied with the README on Github. Off to read!
I haven't gotten around Emacs yet. I'm on a MBP and the Emacs controls are bothersome (to say the least) compared to Vim. I tried using Spacemacs, but it kind of assumes one is already familiar with Emacs and seems a bit too bloated for a newcomer.

So, how do you suggest I acquaint myself with Org-mode? Vanilla Emacs + evil-mode?

These are my suggestions:

Take Emacs as a lisp machine, a working environment, a framework. Then start building it up to your liking. A bit of Emacs lisp knowledge is essential, and it is an easy language, though with rough edges. With org, be experimental and incremental, don't try to create a perfect system from day one, start small and tame it to your habits and tendencies. But get used to the capture+refile flow (see info).

I can't comment about evil, but I do not really feed the need for vi-style editing. Typing speed is not the bottleneck for me. When I type prose, I use Emacs standard bindings + visual line mode. When I program, and nowadays I program exclusively Emacs lisp, and exclusively for my needs, I use paredit. I feed good with these, and I do not feel a need for more speed. But everybody is a bit unique in their needs, so if you'll feel better with evil, go for it.

My last suggestion will be that of avoiding things like Prelude and SpaceMacs. Steal from them, but build your own config yourself. Apart from this, I always use a stable release of Emacs and always check in to git the packages I use, so that I will not have to depend on 3rd parties for my setup. My .emacs.d is simply vital to me.

Edit: about controls, mapping caps lock to ctrl is a life saver, though I do not know well the mac keyboard.

In my opinion, FreeMind is the best solution currently out there because of forces mindmapping conventions and places a strong emphasis on folding. A lot of other solutions I've seen are more like concept maps, which are great for project management, but not great for knowledge bases.

http://freemind.sourceforge.net/wiki/index.php/Main_Page

(The one thing of note is that there is a bug in the default version of OS X Java, so it frequently crashes unless you update to Java 8 and edit the config to use that version of Java.)

Freemind is great for mindmapping in the context of a specific topic or project, but isn't a general purpose knowledgebase.

A mindmap starts at a central node and moves outward. This makes it limited in the practical number of nodes you can have--anything over a couple hundred is hard to manage. It also makes it difficult for one node to appear in multiple places within the mindmap.

>A lot of other solutions I've seen are more like concept maps, which are great for project management, but not great for knowledge bases.

I've never understood this preference; would you be willing to explain it? It seems almost exactly backward to me: I model tasks as trees of sub-tasks, and knowledge as a general graph of connected concepts.

So all knowledge can be seen as connected, but all knowledge can also be put into hierarchies. Concepts and how those concepts are related is something that people come up with, rather than being an inherent property of the universe.

The idea of forcing yourself to put knowledge into hierarchies is that it makes your thinking more clear, makes it easier to remember where things are, and perhaps paradoxically makes it easier to see how things are related. (Or at least highlights certain types of relationships that are only visible in the context of hierarchies.)

I'll admit it's pretty weird at first to only get one root node, I think it took me a couple years to get used to that. But once I embraced it the benefits quickly became clear.

I've been using Zim (http://zim-wiki.org) for several years, backed up to Dropbox. It's a significant improvement over plain text files.
OneNote?
I've become a big fan. With the OneNote clipper extension for Chrome, it's how I bookmark everything I might want to read later.
I've been using nvAlt for a while. Very fast search and ease of use. Not sure how well it scales in the long run but thus far I've had no problems.

Edit to add: Some other interesting features. You can sync to Simplenote automatically, and notes can be saved as text files for easy import/export. There is some markdown support (though I see a number of bugs in the preview). The main downsides appear to be: limited color schemes, no syntax highlighting for markdown, development appears to have stalled.

I have a git repo with notes on all the things I'm interested in, a few larger files (some books I always reference, some images etc.).

Over time I've built up some scripts to help with general tasks, eg list TODOs etc.. It's perfectly moulded to my needs. For example, when I find a new utility I run mk_notes.sh [thing], which sets up three files:

  - links (for relevant web links)
  - thing.txt (which has basic info about it)
  - cheat_sheet.txt (which has quick useful things I want to keep)
These files suffice for the majority of cases. I used to have more, but found that was over-engineered.

I'm not sure I could buy such a thing, and if I did, I'd get irritated with some aspect of it sooner or later.

Search is easy (grep et al), git means it's available everywhere (for me).

It's literally changed my life, since I now don't 'lose' any research, and can pick up threads later and build.

That sounds a lot like the org-mode concept.
Interesting; a quick glance at the home page, and it looks like it's for planning. I'm happy using JIRA for that, where I have customized workflows. But again, this is just because I'm familiar with JIRA, not because it's necessarily the best tool.

Also, I'm a vi user :)

I use Microsoft's OneNote for this. The search works amazingly well and is fast.
I've been tempted by this, but I have some reservations:

1. Does it work under less-used OSs? Linux/BSD..?

2. Is it a proprietary format? Will you be able to open the file in a decade? My text files will be able to be opened, I'm not sure anyone can still open their MSWorks files.

Maybe I'm crazy to worry about opening these files after a decade since using them probably isn't worth while unless you keep up with it. However, I've got files from the 90's that I've been transitioning from machine-to-machine since storage costs keeps getting cheaper.

No, and yes. It's proprietary and Windows/Android/iOS-only.
That's false, you can download documentation for the OneNote file format here

https://msdn.microsoft.com/en-us/library/dd924743(v=office.1...

it is only about 5% of the size of the Adobe PDF specification and I am sure there are corner cases, but it is not too bad.

I admittedly did not know that they published a specification . It's still a proprietary file format though. Proprietary does not mean "binary and undocumented", it means that it's not an open standard.
Word can still open Works files to this day.
Any reason a bunch of text files in directories wont work?

Any "advanced" text editor that can search directories should work: Notepad++,Vim,grep,... anything really.

I've been using a bunch of synced markdown files. Works for me, I just have a Markdown editor and a "grep-like" app for android. I'm not sure what you can do for iPhone though. When I've talked to people they couldn't seem to "browse for files/folders" which I thought was strange.

The one thing I'm missing is I'd like to "tag" my notes. I haven't decided how/if I'm going to do that yet.

One of the main reasons for using a personal knowledgebase is surfacing patterns in what you've saved. If there is no way of visualizing the relationship between notes/nodes outside of a flat list, pattern finding is limited to what you can remember to search for.

Basically the core problem isn't saving and searching for data, it's having great ways to structure and visualize it.

Good point, unless I know a search term it doesn't work out great.

I've been thinking of switching to some kind of wiki format, like maybe creole. Then finding an app that can understand links between files or something.

I used to use Delicious but that didn't solve the "search the content of my bookmarks" problem for me.

I've been using Pocket for now and it seems to work fine.

imho the best one ever made is called getguru.com. It's brilliant actually but the team there has priced themselves out of the market charging $7/head as a b2b app because they're from and enterprise sales background and therefor don't realize guru's monster potential in consumer-web. They're happy taking the low hanging fruit $$ from call-centers.

sigh

I've gotta say that org-mode and a git repo do pretty well by me for this kind of thing.
I'm inclined not to see this as surprising. There are as many ways in which people conceptualise their 'offline' knowledge as there are people, and there is never going to be one size to fit all.

Org-mode is probably the closest contender, if only for its sheer flexibility - but the fact that it's wed to something (i.e. emacs) that the gen-pop would immediately recoil from is only ever going to limit its adoption to a tiny subset.

I think vendor lock-in is at least as big a problem as searching and scaling. Arguably bigger for this audience: if I have easy unfettered access to the data itself I can always try something else for search/scale.

I've tried a few and at least twice tried to write my own (and gave up due to scope creep). Right now I just have a bunch of Markdown files and a giant messy DropBox folder and an almost-as-messy set of private GitHub repos. And photos. And videos. And e-mail. And backups. (What does one do with a Zip disk these days?)

I bet a lot of people my age (mid-40's) or older have the same problem: choosing or building a system we think would work long-term, and then gathering and organizing all the "legacy" stuff (which you definitely want!) is already an enormous undertaking, and so instead we live with a bunch of ad-hoc systems that aren't in any way cross-referenced.

Nobody wants to be the sysadmin of their own junk drawer, but I for one don't want years' worth of my stuff locked into somebody's Cloud du Jour.

(Then again, I have this same problem with actual paper documents, so maybe it's just me.)

This. As a near 40 person, I now need to search in 5-10 places for that thing I know I wrote down in Markdown, Vim Notes, Workflowy, Tiddlywiki, 1Password, Confluence, or email at some point in the last 20 years. But maybe we just need to give up on finding one tool to store them all. Instead we need to build more federated search across all these platforms. That way I can have one searcher that returns results all my various old "brains."
I have had very similar thoughts, but I have to wonder if it turned out to be wrong. Jerry's Brain:

http://jerrysbrain.com/

Apparently started in TheBrain in 1997 and now has more than 250K "thoughts". That's breathtaking for both the longevity and the scale. I'm guessing that most mind mapping programs wouldn't handle that well.

Gathering the legacy stuff is not as important as getting started with something, I think. The legacy will make it into a new system as it's needed.

I've been using Evernote for a number of years. I don't particularly like it (cumbersome UI without great navigation), but the search has worked well enough for me to find things and its ubiquity has been great.

Following up to my own point: products come and go all the time and it seems like luck that Jerry happened to pick a product that stuck around. It does seem unfortunate to have to risk your outboard brain on luck!
Totally agree about Evernote. Man did they drop the ball on staying ahead with their UI. Certain things that have been solved in text editors for years are still major issues in Evernote.

That said, I do use it still, and will continue doing so until there is a solid alternative that I don't think will go poof in 3 years.

Does anyone have a system for managing their relationships? Kind of like CRM but on a personal level. Keeping track of people's details - like their family, their passions, their history etc.
(comment deleted)