Ask HN: How do you deal with Rabbit Hole Syndrome?
This happens when working on something new and unfamiliar, often side projects. In the quest for a "perfect" solution, entire branches of mathematics or programming start to beckon me, and away I go... down a rabbit hole of Wikipedia articles, Stackoverflow questions, and Github projects. Even if I do find an adequate solution, it's not good enough until I can understand the class of problem and derive it for myself.
This also happens based on principle. When I encounter a problem that is actually solvable -- though it may not matter much, then I must conquer it entirely. As an example, for me it is unacceptable that something rated 5.0 with 1 review is ranked higher than a 4.9 with 10 reviews -- and I need to find the solution to this problem, even though it might only be a minor feature and won't matter if there are no reviews to begin with.
I call this "conditional" Rabbit Hole Syndrome. It also sounds an awful lot like the classic interview answer: "My biggest flaw is that I work too hard." Well it's true, damn it. These types of trivial things can easily become my "top idea" for days at time.
I am now able to admit it: I spend time working on things that probably won't matter that much, and I enjoy it. But how do I fix this? How do I learn to leave well enough alone? How do I learn the distinction between frivolous research and actually getting shit done?
A couple of edits: After more thought, this is really only an issue with work on my side projects. I also think it's due to lack of accountability -- I report to myself. As such, I tend to work on "most interesting first" and avoid schlep. So I'm wondering: how do you motivate yourself to schlep?
87 comments
[ 2.3 ms ] story [ 155 ms ] threadBy now it's 9:30pm. It's dark and cold. You realize what your original purpose was: Dinner with friends. That was two hours ago. You missed dinner, but hey, you got some satisfaction.
The above was more of an analogy about Yak Shaving than Rabbit Hole Syndrome, but there are parallels. Like you, I used to be obsessive about details and solving subproblems. I used to come home from work and work on my own side projects for similar reasons as your own.
But then an advisor said something like, "You need to focus. If you want this thing of yours to succeed, you have to focus on making it succeed. Nothing else should matter." So I stopped my side projects and I became so effective at building our product that my employees wondered if I ever slept.
Stay on target. Make it to dinner.
Say you want to try alternative payment processing methods on your site. Your site it tightly integrated with PayPal, and you want to add Stripe. You could (A) spend 1 day to bolt on Stripe with a hack in the HTML, (B) spend 4 days and add enough views to the frontend to make Strip work in a non-disgusting manner, or (C) spend two weeks and refactor the entire backend with a generic interface that will handle an infinite number of payment solutions.
All three lead to success, but you can't optimize for both time and perfection. Set a time budget. Leave breadcrumbs for the next programmer. Instill a culture that says, "Our code isn't perfect, we know this, but we do what we can with what we have." Done is better than perfect.
With forethought, you should have built your system originally with method C in mind. If he had done this at the beginning instead of just hacking it together then all this work after wouldn't have been needed!
1) Apply conscious planning and forethought. Some things are clearly going to be used a lot, some things might not. Use bolt-on hacks where appropriate, invest time for things you expect to be using forever.
2) When the longevity of the tool is not easy to pin down, test it out with a hack. Spend 1 day adding Stripe the hacky way, and keep a slushbucket of time budgeted for revisiting experimental hacks that look like they will last a while.
That's my general approach, anyway. Most of the time, a hack is good enough and anything more is a waste of time. The story is admittedly a little different for me though; for me, software is a tool to do my job- software is not my job.
We have only had to do 2 refactors in that time on 2 controls. That was more about understanding our customers needs more as we progressed rather than not enough planning though.
Previous to that I built a custom ecommerce site for a wholesale distributor. That whole process was more suited to a quick hack over a well thought out decision. That's why I say I agree in principle, so long as the project allows it.
That's how I develop now to be fair and not how I would build a product if I were to build it for myself.
At any rate, I'll tweak the "done-perfect" lever more towards done.
Facebook's motto reminds me of something from a Holman slidedeck: "Nothing great was ever not shipped."
Don't sabotage the larger vision for short-term gratification.
peace
That being said, I guess it depends on what results these random maunderings produce, and it's probably a crapshoot as to whether they are more often than not to result in favorable outcomes. Also, I like your example (obviously :)
Chances are, you'd approach Town Hall with the solution to all their pothole problems and they would ignore it. To make it worthwhile, you'd either have to have earlier signed a contract with Town Hall to figure out their potholes (ie. have a customer) or develop a business based on your solution (ie. develop a market).
It sounds like at least part of your problem may be perfectionism. I did a bit of research on this a while ago, and it turns out there's a lot of literature on perfectionism and how to manage it. A quick look on Amazon under "Perfectionism" should bring up a few interesting books.
My side projects, in particular, are where I allow myself to roam freely, and I often end up prioritizing unimportant (but interesting) work that is only tangential to the core idea. When I am only accountable to myself, then it becomes too easy to get unfocused.
Perhaps the original question should be rephrased as "how do I keep myself focused when working on side projects, since I'm only accountable to myself."
If it is the former then perhaps you shouldn't worry and just let the rabbit hole lead you where it may.
If it is the latter then perhaps it would help to produce some sort of roadmap. Ideally share this and get other people involved (either people who will work with you on it or who want to use the finished product) and they will perhaps keep you somewhat accountable.
If your side project involves multiple difficult problems then perhaps you need to accept that it is not going to be a 1 person job to get it done too the quality standard that you aspire too.
It's actually very easy to generalize.
The answer is endorphins. The harder the problem that you solve, the bigger the fix you get. Hacky solution is not the solution, you know it, so you don't get much. If you realize and accept this as the cause, the solution to your problem (which I think is very common between the programmers) is obvious.
If positive user feedback 'gets you off,' as it were, try releasing something that isn't as perfect as you'd like it to be and see if there's any measurable difference in sentiment. I bet there won't be. Do it a couple more times and hopefully it'll break the pattern.
For the other case, try time boxing yourself so you can get back to the more interesting challenge. Make that other thing 'perfect' if you won't.
I found that to be quite helpful in maximizing your happiness by getting done what you most care about in the long term.
Most of these tickets get resolved as "doesn't matter / won't fix" two or three weeks later - but myself.
First, it forces me to write down clearly the question that's obsessing me, occasionally leading me to realize that doesn't actually make sense or isn't that interesting -- all the more so because it is published so my standards are higher that if I just wrote it in a text file of mine. Second, it gives me the impression that people will work on it now so I can stop thinking about it for a while (even if no one actually cares about the question). Third, in any case the question is documented so it won't be lost.
In some cases I get insightful answers (sometimes, brilliant answers) that save me the time to hunt down a solution in an unfamiliar field or to work it out by myself, other times I end up replying to my own question because I stumbled on the solution later. In any case, it totally defuses the urge to think compulsively about a problem.
1) Try the pomodoro technique, or some other form of time tracking. When you're about to take a side path which may or may not be a distraction, you can decide how long it'd be worth investing in it. Once the timer dings, you can stop and evaluate if it was worth it.
2) A few years ago I started repeating in my head the phrase "real artists ship". Embrace imperfect or partially finished solutions that are viable.
3) Keep a list of things that you'd like to investigate more. I've found that the act of writing down the idea lets me stop obsessing over it. Later when you revisit the list you'll be able to cross off the things that you thought were important, but turned out not to be. By delaying work on these items, you're able to better explore the most important parts of the problem you're solving, and so your future self will be in a better position to evaluate which areas need deep research.
Whilst the desire to strive for perfection is in any craftsmen, ultimately what matters is getting the product to the customers and solving their needs.
Only after solving their needs should you start to indulge in the craft for what the customer cannot see, the internal quality.
I also sometimes describe this as "be lazy"... don't do anything that doesn't need to be done (for the benefit of the business and your customers).
As a recent example, I just spent a long time on a tiny plugin for CarrierWave (the Ruby/Rails file uploading gem). This took me through most of their source code (always good to read code, and I could probably PR on their project now), new Rails internals and techniques, and mocking and stubbing with RSpec (in the process, I turned up a bug with RSpec, which they promptly fixed; and I have a better understanding of mocking best practices and clean RSpec).
I do empathize with feeling I'm "not getting enough done/shipped" (if you feel that way?). To alleviate it, I try to cut corners and just get something out.
Time-boxing helps me do this -- "accomplish X in 1 hour". This doesn't happen too often, however. I know, like you, I prefer the learning itself; and, I view all my spent time as building experience and a critical mass to accomplish work faster in the future.
Also, I find pair programming helps immensely. I am sensitive to the other person's time, and thus naturally refrain from digression when pairing.
As for motivating myself to "schlep". I'm not sure what you mean -- menial tasks? I do those when I'm tired or having down time.
Finally, as in your case, I do this on personal projects and not on "work". That said, it can make it hard to get your startup and product launched - your own "work". Again, disciplined time-boxing helps. I have not mastered this, and find myself regularly looking for good time-boxing tools.
At some point you realize your best days are the days when you delete more lines of code than you write.
But that being said, chasing rabbits is what will eventually make you more skilled than your peers. Which is awesome, except now you'll have put yourself in a perpetual category of being paid less than your worth because you don't fit the same performance evaluation criteria as everyone else.
All anyone cares about is a) are you easy to work with, and do you b) get things done to contribute to profitability.
Continutally stressing yourself out by spending more time on problems than they deserve and eating away at your work/life balance does not contribute to a) or b).
This gets to the crux of the matter. I think we all here have an intuitive sense that being a bit of a perfectionist and regularly chasing a few rabbits down holes can lead to better long-term results. But it's very hard to quantify, certainly it's impossible to quantify objectively; all you can do is be a good salesman about it.
There is another path though—put your money where your mouth is. Start your own business based on your chosen principles and standards of software engineering. It will certainly bring you joy and if your instincts are right then it could even become a competitive advantage as your competitors drown under mountains of technical debt while you nip it in the bud at the first sign of trouble.
If you never find anything and still can't help yourself it's some lethargy/depression ADD brain dysfunction. Make sure to exercise enough to be in shape, eat, etc.
I think it's actually "have kids" that solves this procrastination problem. With having kids you get so many constraints on your time that procrastination is no longer an option.
What I mean is, if you see a problem to solve, and you are able to keep working on your current task and to solve the problem later, then do so... just note the problem. This will immediately make you much more focused.
Then at the end of the day, review your list of rabbit holes and try and determine which ones are necessary for the current project, which ones would be educational / you want to do, and which ones can be discarded.
Basically rabbit holes are a problem because they are long and narrow and do not offer an overview of the entire grounds, so before jumping down a rabbit hole force yourself to survey the big picture and to see if you can step over it instead.
So I suggest you to try to build the minimum viable working program ASAP. Along the way, write all the things you would like to improve in something like an Evernote note, just a few lines for every thing you want to address and make better.
Then if you have something working ship it ASAP, don't care about what other people will say about the sub-optimal parts of your work: many programmers trying to achieve perfection actually end with a mess of complexity that does not serve very well the purpose of the software, so there is little to be embarrassed for a programmer for shipping simple software.
In the second pass, refine every part with the same approach: find a solution that within the timeframe you have is better compared to the previous one, but will make you able to ship a new version.
Also when you face a problem, other than reading the existing literature, papers, and the proper way to do it, check if there is an intuitive solution that is comparable as a result (even if maybe not provable or not perfectly optimal) but much simpler to implement.
But IMHO the golden rule is: don't freaking care, ever, about what other people think about your work. Often perfectionism is just a form of insecurity.
I get it now: If either one is taken to the finish line, they are the same (all nodes visited). But when left mid-way (time constraint), breadth first still is something workable while depth-first has a few components perfectly done while the others not at all.
>> Often perfectionism is just a form of insecurity.
Possibly brilliant too. How do I tell whether it is coming from insecurity or from my drive to not avoid shlep?
For instance I prefer developing software over marketing my product ( http://gridspy.co.nz ). This means that perfecting our product's software is the comfortable and easy option when what I need to be doing is picking up the phone and calling people.
This actually has a label: Maximizers vs Satisficers.
http://en.wikipedia.org/wiki/The_Paradox_of_Choice:_Why_More...
And a little something I wrote about it regarding programming languages:
http://journal.dedasys.com/2006/02/18/maximizers-satisficers...
I think the key is to "choose your battles", and be a maximizer where it really counts, and try and be more of a satisficer for the things that aren't so important.
I'm in the middle of The Paradox of Choice right now, and while the maximizing/satisficing thing applies to many aspects of life—the book was certainly not written about programming—programming is yet another aspect of life where we're confronted with a huge array of choices.
This is especially true in side projects. In the office, there may be a standard set of technologies; you're likely working on an existing product; etc. Additionally, there are probably a number of constraints, including time and product requirements. In side projects, there are hardly any constraints, and every single aspect of your work involves choice. Just deciding what you want to create is one massive choice. Then there are the big questions of "what programming language/framework should I use?" and "what database(-ish) platform do I want to use?" There's a good chance since this is a side project, you'll want to try out something new and cool to build your knowledge and gain experience. Those alone can be huge decisions. Then there are questions of libraries (which library? or build it from scratch?), etc. Heaven forbid you want to start thinking about your choice of IDE (time to learn Vim?) and tabs vs. spaces and every other little decision you can get caught up on.
I'm at least as guilty as anyone else of these things. I spend too much time focusing on the best possible outcome and the best possible technologies among lists of hundreds and thousands—because I want the best and because I don't want to regret any failure—rather than simply settling for something that's good enough and moving on to actually get something done. The result is generally painful and unproductive.
You should definitely set standards for your work—don't produce crap—but try bringing the bar down from "ideal and perfect" to something more attainable like "excellent" or "very good" or even just "adequate." This will help you focus on your real goals and appreciate your own success.
Another useful term/link: http://en.wikipedia.org/wiki/Analysis_paralysis
Reading about interesting stuff is deeply satisfying, and so it's something you want to justify so you get to do more of it. Actually getting stuff done is hard.
It's a bit like why so many techies - myself included! - vastly prefer to code new app features rather than working on marketing or sales copy. One is has a definite end point, and we feel comfortable getting there.
The other one can seem like work!
1. Set goals and fixate on them. Imagine what it will be like to have achieved the goal. Get excited about it and keep reminding yourself. When you're not making progress toward the goal, make yourself imagine the situation where you spend 6 months making unimportant things perfect and never achieve the goal. Imagine all the other goals you won't even be able to set because you're wasting your time.
2. Make other commitments. Make plans to meet a friend for drinks at 9pm. You only have 2 hours after work to get anything done – don't waste those 2 hours! If you can stay consistently busy, you'll notice quite quickly that not using your time effectively will lead you nowhere.
I'm like you. I write fairly vanilla production code, then I go home and work on alien technology. And I used to have the same problem as you: I'd get sidetracked and sidetracked, and somehow I went from writing a fart app to reading about type theory.
So I started skipping the fart app part, and started learning some more abstract theory. The nice thing abstract theory is that it's abstract---it's not incidentally connected to anything, so there are fewer places for you to get sidetracked.
So go sign up for a math course on Coursera, or learn the lambda calculus. They are so alien from your everyday programming experience that you won't have anything to connect them to---until you do. But then you'll be coming at the new topic in the direction of abstract to concrete ("a trivial application of x") rather than concrete to abstract ("there's a greater truth here and I MUST understand it!").
BTW, going down the rabbit hole occasionally isn't a bad thing IMO. But if you go down every rabbit hole, it can slow you down.
I think you're probably not doing projects to help people, but instead projects that you think would be fun to do. If you start with a burning itch, or a person that needs help, you will probably find it much easier to stay on task. This is not to say that there's anything wrong with exploring interesting technologies, or thinking about "the right way" to solve certain problems you find interesting... It's just to say that when there's a burning need the interesting diversions tend to fall away.
Hope that helps :)
It might help to limit going down the rabbit hole too much, by researching what could be improved without actually doing it right then. Save it in a list for later, do the schlep, then when things get bigger chances are you'll actually need to take on some of the challenges on that list. And the more useful they become, the more satisfying it'll be.