Ask HN: How do you manage/organize information and knowledge in your life?
We "collect" data all the time, and sometimes it helps us learn something. On one hand we have things that were created by others: funny images, movies, articles on the web, books. On the other we have things creted by us: personal insights, thoughts, snippets of code. How do you manage/organize all of this? Do you have apps/tools for organizing specific type of information and if so, why this particular one? Do you have ideas how to organize/manage this efficiently?
But we're not only collecting information [0], we also learn from it. Most people store all this knowledge in their heads for their whole life, but I think it's not efficient (memory loss, can't search, not always reliable etc.), some people create some kind of personal knowledge bases (e.g. personal dokuwiki/mediawiki on localhost). How do you manage/organize things you learn?
It may seem that I should create 2 different topics, but for me both concepts are connected[1]. What I'm trying to find is an efficient solution to managing (almost) all the information that enters (and already is in) my life. I don't think that there exist a good app for that, I know that there are some that solve parts of this problem (evernote, wikis etc.), but I'm more interested in your ideas on the whole topic, how to approach this problem, where to look, how to think about this etc.
I'm curious about your solutions, ideas and "setups" for this problem(s). If you have any resources (books, research papers etc.) about the topic, I'd love to learn from them. Thank you for your time.
[0] as in bits on the disk, learning can be viewed as collecting new information, I guess
[1] I liked quote from a book, some code from LLVM gave me an insight into a compiler design etc.
85 comments
[ 2.5 ms ] story [ 153 ms ] thread- pinboard for managing bookmarks (database of things that may be useful sometime; probably never) & reading list for articles
- I'm testing tagspaces (http://www.tagspaces.org/) for local files organization (mostly tagging research papers and books; didn't like Mendeley)
- cardav & caldav from owncloud for contacts and events
- anki as a memorization tool (spaced repetition) - from languages to my own mistakes (i.e. "lessons learned", so I won't repeat them)
- for insights, notes, ideas, things I've learned & everything else I use personal wiki (media wiki) on localhost. This is the biggest part of my "system", I have there entries like things to buy someday, current project's notes/resources, useful scripts, configuration snippets, notes from books, journeys, analysis of my own behaviour, personal journal, ideas for startups etc. But it's hard to organize, it becomes a mess very easly after some time. Also, I can't use it on mobile (I don't want to put all this on the web, there is a lot of personal info), it takes time to add new thing/entry (I need to think to which page new piece of information belongs etc.).
- simplified version of gtd as a meta-system managing this system and for projects/things to do
So I switched back to plain text, using the Markdown syntax format. Sync works fine with IMAP based email account (note apps on iOS & Android support that too). For visual complex documents I use a WYSIWYG HTML editor and the HTML format. SVN/Git repo to preserve the edit history. And I coded an desktop/enterprise search engine myself to search through PDFs, HTML and various Office formats - similar to the discontinued Google Desktop search and some enterprise search software.
https://www.youtube.com/watch?v=oJTwQvgfgMM
http://lifehacker.com/im-ryan-holiday-and-this-is-how-i-work...
The take-away there is index cards. I like the idea of information being broken down small enough to fit on an index card. Reading a book, it could be one point, one quote or something of similar size.
In the past I've tried to use POIC system[0][1] but it wasn't really that efficient and the biggest 2 problems were search (mostly full text, I could find entry from a particular date quite easly) and portability - over time it becomes a big physical thing you need to store (I move quite often).
I agree with you about dividing (almost) everything into single pieces of information and adding to the system, but I'm still trying to figure out how to do this well.
[0] http://pileofindexcards.org/wiki/index.php?title=Main_Page
[1] http://lifehacker.com/the-pile-of-index-cards-system-efficie...
I've recently started to use fedwiki instead of OneNote, and things are alrightish. Fedwiki has lots of room for improvement as a wiki, and then as a knowledge store.
I think the ideal personal information/knowledge store would incorporate a tagging filesystem combined with something like OneNote/fedwiki. The tagging filesystem would allow PDFs, movies, etc to show up in searches with fedwiki/OneNote handling the plain text & images. Ideally the client that the user uses is something like fedwiki, where you can have multiple different pages open at once, but also allow you to pull in the PDF/video resources.
Do you have categories in which single entries are grouped or you just add new thing and search for it later?
With fedwiki, I have a bunch of 'Table of Contents' pages grouped around themes, Project 1, Project 2, etc, and some pages then link to other pages, like a normal wiki.
Currently fedwiki just does searching based on page titles, I'm (slowly) working on an elastic search backend for it to get full-text search.
For example, a while ago I read an article that talked about peanut allergies being less prevalent/severe in Europe than the US, and Europe having more cases of other allergies (like apple allergies). It then went on to talk about different pollens that may react with our bodies to trigger certain allergies. I can't recall how long ago I came across this, or where (could have been HN, Slashdot, Digg, etc). I've also not been able to put together a search to turn up this article in Google. Had I been better about bookmarking, maybe I would be able to find this article even if it took a brute-force search through all of my bookmarks.
Another thing is archiving all those bookmarks, since after few years a number of them is 404.
Do you think that solution to the whole personal knowledgebase problem should be solved from a single app perspective or is it better to think about this as a sort of ecosystem - one "module" for adding new things to the system, other for statistics[0], sync etc.? Also, did you consider semantic part of such system? So we can write programs that can make sense of all the things we have in the system?
[0] I was actually thinking about using machine learning to somehow finding the most important things in the wiki and suggesting tags/place in hierarchy of new entries
The primary app is a wiki with unstructured text, outlined/indented text and hyperlinks. But, as mentioned, information comes other forms too: structured data like bookmarks, spreadsheets, images, etc. The web lets you express all those things (spreadsheets being the weak link, since it involves executing code). Reinventing it all within a single app would be a lifetime of work.
The web already has common patterns and facilities for log analysis and so forth. If you just make it a web site, you can track all your actions with web log analysis. Whereas if you make it some "app", then you have to reinvent the analytics.
I definitely considered doing more textual analysis, stuff like PageRank, and even semantic analysis. I think it's interesting to explore but I don't the cost/benefit works out right now.
I have over 2000 pages, but probably 500 or less are "active". That can mostly be managed with full text search, and nothing smarter. You just have to make the search fast, which isn't hard with that amount of data.
But, if you want to write more code and programs against the data, a web site is a good format for doing so. I basically use the Unix philosophy of independent programs and data, rather than the "app" philosophy of data locked up behind code / user interface.
My philosophy is just to optimize for speed and ease of both writing and reading. And then you use it more. And the more you use it, the more valuable becomes. Just like with learning itself, you start to make more connections once you have some existing knowledge/content to graft on to it.
I think the most important thing is the content, and not the tool, and as long as the tool is lightweight and fast, you will keep up the habit of entering content and (just as importantly) retrieving it.
I was surprised the biggest thing for you was speed and full-text search, and also by this comment: "I think people forgot how to make a simple web app with a form and plain buttons."
The biggest thing for me is learning. Not so much categorizing everything correctly.
How do you use Tiddlywiki - do you often search or do you know where did you put some note? What in your opinion are pros and cons of this wiki? What do you think is the biggest problem?
Pros: It's easy to set up, and has been robust. I like it that it stores its data in the filesystem, in a location of my choosing.
Cons: It seems to use a somewhat unconventional formatting syntax. I find a lot of the terminology pretty opaque. Having had a few scary configuration-related incidents, I don't try to push it to do exactly what I want anymore.
I'm glad to have found Tiddlywiki and set it up, because at the time, I was storing notes in a thousands-of-lines-long text file, which just wasn't working. But I should probably resurvey the scene and see if there's anything better.
With Zim everything is in plain text files, and it integrates with version control. This makes things more transparent (to me at least), and you can still use some useful things like filtering with tags, searching, todo-list overviews, etc.
The problem comes from keeping everything organized (impossible with Evernote and Simplenote with their lack of structure, impossible with Quip since it's docs/spreadsheets-only) and in a way that works with my mind and workflow.
I'm working on sketching out what a unified personal knowledge management product might look like, which combines a kind of "inbox" of sorts of resources and notes coming in, and also a "personal wiki" with structured docs, and a number of people I know are also thinking about what a 'perfect' knowledge management system would look like. Let me know if you want to bounce around some ideas around.
1. A little bound paper notebook such as a Moleskine.
2. A mind mapping program -- I use FreePlane -- to store links, including links to files on my hard disk. An advantage is that I don't have to get the organization right on the first, second, or even third try.
3. A lot of the information in my life is not digitized, such as most of the sheet music in the world. So I now rely on my cell phone camera to record a lot of that stuff.
Amusingly, when I was in grad school, it was still considered to be an open question whether a person should get their own computer. The university computer store had a little guide, and the most memorable advice -- which certainly rings true in my life -- was: "Don't expect a computer to make you organized. If you have a messy desk, you will have a messy computer."
I can sympathize with bookmarks problems, not only they rot, but after few years there is a lot of 404 (pinboard provides archiving bookmarks but that is a paid service).
Do you have a lot of Anki decks of just a single "random things from the bookmarks"? Do you use Anki every day or rather infrequently?
The main problems with all these systems are just that we use them to attempt to augment our own intelligence/utility based off of some concept of who we are aiming to be in the future, and then as things change for us personally, our aborted plans lead to a lot of rot that doesn't clear itself away automatically. So only some small percentage of what we collect ends up relevant.
I have a lot of individual Anki decks, which isn't recommended. But I create a filter deck that rebuilds from the other decks every day and properly randomizes the cards, and only study from that. I do it every day except when I forget and then I catch up later. My cards are mostly along the lines of music, logic, math, stats, programming, and rational thinking.
- For meetings, events, social obligations, I carry a small paper-based diary.
- For ideas and thoughts, I write them down. The idea is that the act of writing something down aids recall. For example, story and blog ideas go in a A5 notebook that I carry around with me, and I commit them to a digitally backed up document as soon as possible. Note-to-self lifestyle advice goes in my bedside drawer, to check on if I ever feel like I've forgotten anything.
- For things to learn and interesting articles, I don't do anything and keep my fingers crossed that the salient bits will have rubbed off on me, lurking in my subconscious and subtly improving my life forever after.
I'm not 100% sure that last one works quite that well.
2. Memorize as much as I can. It's an exercise; I memorize phone numbers, schedules, people's names, trivia, all kinds of stuff. I've never found anything that matches the flexibility and utility of my own brain. I should use the best tool I have, and that's in my head. Technology is unreliable and constantly changing and difficult to organize and search. I've been practicing this for long enough that now I'm pretty good at it.
3. For everything else, I use a few simple systems: a few sheets of paper to the right side of my desk for scribbling and note-taking (meant to be discarded after a day or two), a pile of to-do to my left, a tab open in my text editor labeled "notepad" for longer-term stuff, and a well-organized directory of documents on my laptop with subfolders like "projects", "writing", "sysadmin", etc. -- I try to keep this directory as small as possible by dedicating time here and there to either finishing or pruning projects.
I disagree that keeping knowledge in your head isn't efficient. I think a lot of people just don't practice it enough. Smartphones and computers and everything else make it really easy to not bother. But, my brain is always with me, doesn't require batteries (well...), can store any type of information I want, and can instantly recall it without having to craft some kind of search query or organize the information in a rigorous way. It is exactly the kind of database storage we all wish we had. It never changes data formats, it never tries to get acquired by a bigger company and then shut down, and it gets reception everywhere I go. If my brain were an electronic tool, I would want to use it all the time. And, the more I use it, the better it works.
(edit: oh yeah, and pinboard. Looove pinboard.)
Make life as simple as possible ("simplicity is the ultimate sophistication"). Use paper when necessary.
Also, try hard not to multi-task. It's been proven the human brain isn't built to handle it.
I just practice, a lot. I exercise my brain like a power lifter exercises their body.
For instance, I know the names and some basic information of everyone at the bank branch I frequent. I go up, say, "Hi ____, how're the kids, did you have a good time last weekend?" It's nice for them and a little mental workout for me.
When I'm out and about, I briefly memorize random things: movie showtimes, things I heard on the radio and want to look up, prices of things in the store (so I can price-compare later; e.g. I know Safeway has my cereal for $3.99/box with the club card discount, which is slightly better than regular price but not as good as the occasional 2-for-$4.99 sale at Raley's). I try to really look at the people around me and memorize different facial features and notice different things about them, and if I'm not distracted, then I try to flip through them all mentally when I get in the car. (But this also makes big crowds more overwhelming.)
I play Go. I have a small local club with some friends, and we play religiously once a week, and I play or study more often in my spare time. It's a game where you can improve quite a lot by visualizing the game board several moves in advance, and the game board can get pretty complicated. It's a really good workout for my working memory.
When I'm reading, I try to memorize most of the previous paragraph immediately, memorize the bullet points of the previous page when I get to the next page, and the next day I try to remember the gist of what I read. If I can't, I go back and re-read it until it gels.
Stuff like that. It all sounds silly written out, and like a lot of work, but it's second-nature now. I just do it automatically.
Exercising memory is kind of a two-part process. You have to be able to put things in your brain (typically this just means practicing paying attention), and you have to be able to recall them later. I think the recall part might be the bit that a lot of people forget to do. I quiz myself all the time on what I just memorized, so my recall is pretty good.
Oddly, the harder thing I've been struggling with more recently is forgetting on purpose: there's a bunch of stuff in my brain that I don't need, and learning how to forget that stuff is a trick I haven't figured out yet.
I agree that you should rely on your memory. However, when you die, all that information you have is gone to the world. That's why I started writing things down, for when I'm gone.
Take a look at Roy Underhill, if he hadn't started writing books and opened a school, all of the woodworking knowledge he has acquired would be gone forever (or until someone else dedicated a lifetime to finding it all over again).
I also try to use my memory as often as possible and I think it's pretty good (I'm learning mnemonic techniques which are really fun to implement in everyday life). Yes, mind itself is reaaaallly good system, but (at least for me) not all my insights, ideas, notes etc. about particular topic are available on demand, but only some part of them. And if I don't think about one thing very often it can be lost forever. Nevertheless, thank you for taking time to write about your system and giving different perspective.
But also, keeping that stuff in a system you can trust frees your brain to do other things. Instead of falling asleep reminding yourself to buy milk tomorrow, you can leave it to the system to remember. The system being anything - a notebook, an app, or what have you. There is some good evidence that this kind of delegation allows for more high-level thinking to come about, since your brain is more free to do other things.
Now, as you've touched on - sometimes technology screws you one way or another, and that trust is broken. So if you have no redundancy or backup or plan for that dependency, maybe you'd be better off keeping things in your head.
But you are human and your head is human by extension. it is faulty, imperfect, and not nearly as good at storing things as pretty much anything else. Even if its good now, it won't necessarily always be, and isn't in such good shape for a lot of folk. So I would not paint it in such a rosy light.
I'm curious, Have you done any experimentation with Method of loci?
You're right though, brains are pretty fallible. If something's really really really important, I do have backups. Usually email, or text, or paper.
...but, honestly? I'm scared to death of old age. It scares the piss out of me. One of the things about it that really gets to me is the idea that I might be 90 one day and not recognize people I care about, or remember anything about myself, or have any idea what's going on around me. The smartest elderly people I know are all very mentally active, and always have been, so even though there isn't good scientific evidence for the prevention of alzheimer's through puzzles and brain teasers and the like, I do it anyway.
You won't reach your maximum potential if you only try keeping knowledge in your head.
2. Poor solution, but I email myself notes a lot. Usually from my phone, then I drop them into a folder in Outlook when I get back to my desk.
3. OneNote, for more long term collections. Microsoft did an amazing job with this product.
It's a bit time-intensive to find things, but it's not impossible: let's just say the system is optimized for write efficiency and not read efficiency ;)
[1] https://gist.github.com/ivanistheone/9daa23ae2a7abb472cb2
I think ideas are folder-like structures so I usually create a folder to store code/links/pdfs/clippings specific to that idea, then file away that Idea folder as one piece.
My general advice is to spend less time planning and more time doing, else you might get stuck on what Ze Frank calls "Brain Crack" http://youtu.be/0sHCQWjTrJ8?t=8s
One unique thing I started doing, I use this program called ClipX which lets you have a popup of your recent clipboard entries. Its will let you save to a text file. So, everything I clip, code snips, passwords, etc.. goes into a single text file - so I can just Ctrl-C something and forget about it. Six months later I just do a text search in this file. Its gets big (10MB or so) and I back that file up and start with a new file every year.
Something that would combine find,grep+pipes into something more "userfriendly".
For GUI: Windows search can do this with the datemodified:* and size:* operators, or just click the Search tab in the Explorer ribbon to see buttons
So I guess my answer is "poorly," but somehow it seems to work.
Simple text file that's actually more functional then almost any other solution, designed around allowing you to create and adapt new workflows and WILL be available and useful to you for decades.
https://en.wikipedia.org/wiki/Org-mode
The most important purpose, I decided, was personal goals. Knowledge/information which is relevant to helping me achieve my own goals is the most important thing I should be focusing on, and should be extremely well organised and easily accessible. Any other interesting info that falls outside that is a bit of a shame to lose, but ultimately just a distraction and clutter. For this purpose, I developed this tool: https://nachapp.com
I believe the next level of information down would be general learning/knowledge. Stuff that doesn't fall under any specific goals, but is still useful information to know and understand (and may in disparate ways tie into core goals). For this, I'm currently using https://pinboard.in, although it's not ideal as it's again limited to a single medium. I have a solution in mind, but haven't started developing yet. If you're interested, feel free to get in touch and I can keep you updated (contact info in profile).
- Folders like \doc \tutorial \ideas \recipics - Store everything in .txt and .odt - Replicate to favorit cloud storage
- For comunication: Fastmail, Skype
- Contacts & calender: Fastmail
- Keep a diary in a mail folder, I send myself a mail every now and then with a subject to an alias, that gets sorted into a "diary" folder
After trying several solutions, I ended up building a SaaS to manage everything. I found most things out there are fairly boring, and are not at all as powerful as what I wanted. It's basically a brain for my brain; so i can remain a scatterbrain and it can tell me when its time to water the plants, or if food in my fridge is about to expire. But it also handles all my notes, important files, time-series data, and historic dates.
But really, you have to figure out whats important to you and what system best aligns with that; and in the end you'll likely need to make a few tradeoffs to get something working.
But I think there are definitely some principles you can apply to any system you use; I can't recommend Getting Things Done by David Allen enough. His methodology is great, but even if you don't like it or can't use it for whatever reason, there are oodles of great tips; and it'll make you into a natural project manager / information guru.
I have recently started using jrnl: http://jrnl.sh
I really like it because you can export it any many formats which I think would come in handy later to import the data, say a database. I always like the idea of a flat file for right now so you can ack/grep on the command line.
The cons are not great for assets such as images.
Looking forward to reading some other solutions.
Most of it I'll never need again (and probably look online for it first anyway, if ever), so it's really just to satisfy a compulsive need to have the impression I'm not "losing" anything. Rarely did I ever dearly miss anyhing on accidental deletion of my "stores".