Ask HN: Is TDD/BDD hampering your startup's agility?
In practice though, I find myself spending a disproportianate time writing and getting test cases to pass compared to just writing good code and testing in a browser.
- For functionality that I know works, I sometimes have a difficult time writing and passing tests especially when the functionality is unique (e.g. Facebook login?).
- When we pivot or iterate, it seems we always spend a lot of time finding and deleting test cases related to old functionality (disincentive on iterating/pivoting, software is less flexible).
- Test cases (namely Rspec) are just plain slow to run (seconds staring at a screen before getting positive or negative feedback).
- There always seems to be about 3-5x as much code for a feature's respective set of tests as the actual feature (just takes a lot of damn time to write that much code).
- Most of the code in a lean startup are hypotheses that will be tested by the market, and possibly thrown out (even harder to rewrite test cases with slightly different requirements over writing from the beginning).
- Refactoring breaks a lot of test cases (mostly with renaming variables).
I do think TDD is great for client work, but for lean startups, I'm not so sure.
For a startup that is iterating very frequently and trying to reach a product-market fit, I find TDD to be harmful and actually impede agility. Speed trumps reliability here.
Like security (budget vs security) are speed and reliability 2 points of a continuum? Where is your slider as a lean startup?
129 comments
[ 0.20 ms ] story [ 174 ms ] threadWorks for me perfectly well, and I don't give a damn what the TDD True Believers think.
Robert Martin usually says "You are not allowed to write any production code unless it is to make a failing unit test pass."
When I'm writing code to solve a problem I've never solved before, I don't write tests. But then I scrap it and write tests first, implementing code to pass the tests. It may sound wasteful, but I have enough experience behind me to know that those tests come in mighty handy about six months later when I want to add in some crazy new feature I didn't think of before.
Heck, I even do that with new features - I'll branch, write it, see if it works, then branch again off the master and implement it again with tests. It's not much extra work really, and I often do catch little mistakes I made in the "prototype" version.
But it's taken me a long time to get used to TDD, and I feel like I'm still learning. I occasionally find myself over-testing. Like anything else, it's a discipline, but I find it so worth it.
* To me, you should balance the amount of testing on several things. Not all code is created equal - some are plain old quick tests which are meant to be thrown away. Other code is something you expect to be running for a long time.
* The most important balance is this: If you leap quickly over testing of a piece of code, it may or may not cost you more time in the longer run. In other words, not testing increases the risk variance of the code having a bug further down the road. You have to evaluate if that is going to be a problem or not. The problem may also occur because your code is too slow. With a good test-harness it will be easier to optimize and sometimes the tests can be used as a start for benchmarking.
* On the contrary, if you feel the grass is greener on the other side of the road, you may test too much and thus never move fast enough to getting something done. It will cost further down the road, but it hinges on the premise that you will not discard both the code and the idea and rewrite (so tests needs to rewritten anyway).
* Personally, I rarely use a TDD approach. I rather like property-based testing: I "fuzz" out errors. I've just written a protcol encoder and decoder and there is an obvious test: (eq orig (decode (encode orig))). So I automatically generate 1000 "origs" and test that the above property holds. To me, this is much more valuable than TDD/BDD - but I've never been a fan as I said.
* Sometimes the idea of BDD is to shape your process and thinking pattern. In that case, it hardly looks as if it a waste of time: Had you not BDD'ed, well then you may have been in the unlucky case where you implement a lot of code only to realize that you implement the wrong idea because the API has to be different and serve you differently.
Time, Quality, Cost, Scope
If what your coding will have limited impact on use or functionality, scalability or performance later down the road... Fine. However, it is my experience that standard approaches are standardized for the greater good and health of a "more mature company". As a manager, director and head honcho I sure don't want a developer making that kind of evaluation. It will work for you now, but for your own sake and others later down the road; artfully comment your code!
There's not always a "do TDD, develop slowly" vs "do no TDD, develop quickly" dichotomy.
If you're doing it right, TDD should speed you up. It adds some guaranteed extra time to your plans up front but significantly reduces all of the hidden costs of wasting hours debugging bugs that unit tests would have picked up.
So I might know developing a particular app might take 10 hours without tests, 15 hours with tests, but if I waste more than 5 hours fixing a myriad of bugs, the TDD approach still wins. (The constant reward cycle is an important psychological factor too, though, even if the times are equivalent.)
Some differences we're doing from your situation:
We use Cucumber to cover the whole web app(but not flash or video processing), and only have some small rspec model specs on important methods involving billing.
Cucumber coverage is also very powerful per line of test code. We have 1000 lines of cucumber covering 5000 lines of code.
We aren't covering everything by tests. For example, I would have given up on the Facebook login test coverage, and just written some tests that mock a facebook-logged in user but not covered the actual login funtionality itself.
If we were not doing BDD, my time estimates for each ticket would have to double because the hair-pulling debugging time would skyrocket and kill productivity.
I would also hate working on a team that didn't have test-coverage because developer B might build something I don't understand or know about but I inadvertently break it and we find out 4 days later.
Another benefit: we can ruthlessly refactor and tear out code because the tests immediately identify if something broke.
There's also more payoff for your tests over time. The longer your project lasts, the more those tests pay dividends. Even they seem painful now, they are an investment towards maintainable code in the future.
My advice: keep going with TDD/BDD and consider Cucumber for everything but your most important business-logic methods.
I worked with a developer who was pulling his hair out because he couldn't figure out how to test the SMTP emailing with gmail. We all knew it worked, but he spent a few days figuring out the test cases.
A couple points though:
- Ruthlessly refactoring is huge with TDD. However, I don't think there is much refactoring with bootstrapped, pre-revenue startups. Changes are generally functionality changes.
- Payoff over time. With very early stage, I can imagine that shipping a day early is worth a high interest rate of payoff later.
- Solo hackers don't have to worry about other developers, again at the very early stage.
Overall, maybe TDD/BDD doesn't make sense for very early, pre-revenue startups, but instead should start after funding or product-market fit where you need all these things?
At our startup, we learned a lot during the pre-revenue stage, and twice realized that our basic architecture was stupid, wrong, and overly complex. (That was my fault, by the way.)
Both times, I took 3 or 4 days and savagely refactored our product, deleting massive amounts of code. When I was done, it was much easier to make changes.
Mind you, in the end, this wasn't enough to save us: We ran up against the reality of a short runway and a long enterprise buying cycle. But our product was very sweet. :-)
I think this is the most important thing. Tests are an investment, but once you've done them, every time you run them from that point on is free. The amortized cost keeps getting better and better.
Once you pull in a Continuous Integration engine, or even turn it up to 11 and implement continuous deployment, those tests really do pay for themselves. It's just intimidating in the short-term.
Whenever I've gone rogue and thought "Sod it, not today", it's invariably bitten me on the ass twice over the pain it would have been to write them.
True, but they have their own maintenance cost. When features evolve in the future you have to pay to update the tests as well. Worthwhile, but something to bear in mind.
Very true, but when features evolve in the future, you may have to pay the debugging costs if they evolve in ways you did not intend with regard to other features.
A number of these points aren't familiar to me (trouble finding testing for deleted code? harder to modify tests for changing requirements than start from scratch? Renaming variables is hard?). These comments make me wonder if you've been treating your tests the way you treat your code.
When TDD has worked best for me it's because I've spent a lot of time thoughtfully putting some organization into my tests, making sure they're ridiculously fast to write, and ridiculously fast to run. Your source code becomes slave to your tests, that's the whole point. The fact that your tests are in your way, suggests that you're doing it wrong. If you were doing it correctly, and TDD was failing you.. I think the symptom would be your operational code getting in the way instead.
If there's no corresponding production code using the same interface, you have a problem: you're probably testing at the wrong level.
TDD/BDD doesn't fit the mold of startups. Here's why:
TDD/BDD assumes you know the problem and are coding to create a solution. In startups, however, you do not know the problem. Sure, you can imagine some kind of customer that might want some kind of code to do something or another. But that's all pipe dreams. You have no idea if method X is useful or not. Sure, you might know if method X meets the requirements of your imagination, or your architecture dreamed up in your imagination to deal with your imaginary customers with their imaginary problems, but you really don't know.
So it's a waste of time. Startups are all about providing value -- not flexibility, not bug-free code, not code you can hang on a wall and take pictures of. Value. If you want to be the guy who can make business solutions happen, the guy that customers can come to with any problem and you can make it all happen, you need to bone up on this stuff. But in the business world, you've already got the money, your job is to make the solution happen. In the startup world, you don't have the money yet. Big difference. Big, big difference.
Look at it this way: 19 out of 20 startups fail. That means that odds are that you will never see this code again. You'd be a fool to spend any more time on it than absolutely necessary. But the math works out completely differently in the commercial world, where most things you write stay around forever.
What I found over and over again with Agile is teams and individuals buying into the marketing/herd mentality of agile and forgetting about the adaptive/iterative nature. Everybody wants to either use a recipe book or just repeat their last project that they thought was really cool. "True" agile means ditching whatever isn't working. Pronto. There are no sacred cows. Everything is on the table.
The fascinating thing is that once I teach that it's all flexible, guess what? People come in and make it all rigid and orthodox anyway. It's not Agile that's the problem, it's the people that, over and over again, just beat the living hell out of agile trying to make it into some rigid 1-2-3 formula for success. No matter how I beg and plead with them not to do this, still they persist. Strange.
I'm left with the conclusion that the basic problem in software development is people's fear of having a truly unpredictable, flexible, and open way of working -- even the agile adherents (especially the agile adherents, it seems). That's simply too much for some folks. They'd rather have a fixed list of things to do each day, even if they know the list is full of crap, they are taking ten times longer than they should, and they are doing busywork things that provide no value. As an example, I've had teams be very successful keeping their list of things to do on post-its on a wall. Then I leave for a year and come back and they've got some kind of spreadsheet monstrosity that has hundreds of items nobody can remember. Nobody likes the new system (except for perhaps the person who runs it, who will also admit that they don't like it, but only in private), it doesn't fit their needs, it's a huge time sink, yet they don't know how to make it stop. They are prisoners of their own need for fixed systems and over-engineering everything. This is because somehow they feel by over-engineering things that they make them "safer". Even though at some level they realize that this is not the case at all. Very strange.
So don't blame 'Agile' for being inflexible, blame the people who are using it for not actually being it.
It was great, and I always want to share it when the concept of 'Agile' comes up, but can never find a way to sum it adequately.
It's not a blue pill.
I think this is where these posts on HN always end up talking at cross-purposes. "Startup" can mean many different stages. I definitely agree with this. It's one of those things where when I think "startup" I think "they've got a problem, they're solving it" whereas some people read it at that very early stage instead. TDD doesn't suit that stage, because, like you said, you don't even have an idea of what success looks like.
This seems like two ideas conflating into one. The business 'problems' (1) and the problem you're solving in a particular piece of code (2). These aren't usually directly related. For example, if I wanted to build a YouTube killer, the TDD process isn't relevant to the high level "problems" I want to solve (which, as you say, one might not know of yet).
TDD comes to play (from a developer POV) when I actually start to write some code that, say, transcodes a video or provides an authentication system. In those areas, the problem is obvious and contained and TDD can work well.
That means that odds are that you will never see this code again. You'd be a fool to spend any more time on it than absolutely necessary.
Which does not necessarily mean TDD is "a waste of time". If practicing TDD in a particular situation will result in fewer hours spent developing a feature, the upfront "cost" of those hours is not a waste of time if you spend more hours debugging your way through building a non-tested equivalent.
A developer should have a feel for which way works for them. In my case, I know that the time I spend is ultimately lowered through using some TDD principles (though not all) vs tiresome debugging of untested code. The reward cycle of TDD is commonly overlooked. "Code->Yay!->Code->Yay!" beats "Code->Code->Code->Code->2 hours of debugging->FFFFUU!!" any day.
"True" agile means ditching whatever isn't working. Pronto. There are no sacred cows. Everything is on the table.
Agreed, but that runs counter to making absolute statements like "TDD doesn't fit the mold of startups" and that "it's a waste of time." It can fit and it can save time (just not always, sure, or if you're 'doing it wrong' for your situation).
At high level also, Start-ups do need a TDD philosophy. For Start-up business ideas, the market is the test suite (to take the software analogy further) that is already there and changing the execution plan to make sure you pass those tests is vital.
But yeah, there's other things you can use, like - A/B testing (for web apps). You can automate "code smell" detectors, and get a continuous graph on how good your code seems to be.
Of course, things like converters and math functions (anything non-GUI) can benefit from unit tests.
The reward cycle of TDD and the focus that TDD brings to writing code are the reasons I use TDD. The added confidence in correct code is also extremely helpful.
As I see it, if one were "in the zone" from 8-5 for a week, and had a clearly defined and understood spec, most software wouldn't take more than a few weeks to build.
Since being "in the zone" and understanding a well-defined spec aren't really that easy. Most time spent (in my experience) is understanding the spec and wrapping your head around the problem. This is a reason why pair programming can be so helpful.
Regression testing aside, writing tests is valuable to me mostly because it helps to define the problem for me enabling me to more quickly write code.
In most businesses the opposite is true. Test Driven Development isn't really about testing low-level functions, it's about making sure the code and the business logic behave in the same way. TDD really comes into its own when you start using the tests as documentation on how the system is supposed to perform. A company using TDD can very quickly adapt to change because they can be confident in their code.
The reasons TDD isn't suited to startups is more that it's a heavy process which doesn't fit well with a small, tech-heavy group of people who aren't selling code.
One of the things I really don't think you get -- something I had an extremely difficult time getting -- is just how worthless your code is. Startups are not rewarded for writing good, maintainable code. Hell, they're not even rewarded for writing code at all. There's a lot of folks that say to advertise, create a vapor-ware product, and only once you know for sure people are willing to pay money for it should you start programming. Whether you like that idea or not, it shows just how far down the line programming is in the scheme of things for a startup. "Maintainable" or "Great" code is even further down.
Startups are about making things that people want that can scale out to huge numbers. Coding gives you that ability to scale out -- in certain instances and under certain situations. But you could just as easily write 10 lines of javascript and make a million dollars as you could write 100K lines of C. In fact, there's another huge school of thought that says the more you code the farther you've probably drifted in your mind from where the actual market might be. Each little test you create is further reinforcing in your mind -- that little code->yay cycle -- that what you are doing is cool. And it's probably not. "Yay" is people paying you money. Not having a test run.
And that gets to agile principles, which I think are very applicable. Do a little, test the market. Do a little more, test the market. I have been doing that for a few months, and I've found that most people who want stuff could care less whether it involves programming or not, or how well it's programmed. They just want stuff. If a guy is drowning, he's perfectly happy with you throwing him a life saver that falls apart and is crappy -- as long as it saves him from drowning. You might not even need a lifesaver -- you could do something else. In fact, the more you speculate on what the drowning guy might need, the more off-course you are. That means that any structure you put in your code early on is smoke and based on pipe dreams. The iterate-test cycle for me is what has verified beyond a doubt how low on the totem pole programming is. Find something people want. Period. More customers, less anything else.
Coming from many years of loving to learn how to be a better craftsman with my code, kinda sucks, huh? All those guys telling you what a great programmer is and how to be one, and then Average Joe Sixpack doesn't recognize how cool your TDD and programming language are. Dang customers. :)
The one great thing I learned from RUP, a cocktail napkin makes a great realization document. Especially when originally served under a cool drink.
Communication and being clear on what the tree-swing looks like and needs to do is all that matters. Your companies will develop culture as they grow based on their pain points and their experiences.
I take the approach of "automate, automate, automate". I also enforce regular activities in intervals of by day of the week. I insist on profiling. I insist on peer code review; which can be done in the IDE or with a cocktail napkin, but I like my developers to be able to communicate what they've just written in case they get hit by a bus.
As a leader, not just the coder, you DO have to be concerned that you're writing something that can hold a the weight of a business on top of it. Would I worry about TDD or BDD before my Series A? No. But I would go back and reiterate and reinforce everything written prior, shortly after.
If you don't know the problem then you are just winging it. Writing software is always with a goal (can it be without?) and thus you should be able to write a test (first) that will tell you if it does what you expect. If you expect it to record click stream, then it records click stream or whatever.
When I was new to TDD, I too found it to be very cumbersome. It took me some time to realize that I don't have to figure out the entire design before I write any test code. I can start with a test, then write the functionality that it is testing and when I realize that this is not what I want but something else then I change my test to test that new thing. Iterate. That is what agile is about.
Discovery that you want something else, happens whether or not you are doing TDD.
My feeling on automated testing is that it's far more valuable and saves more time than people who are inexperienced with it can understand, and yet it has shortcomings and limitations that go far beyond what any zealot is willing to admit.
I do encourage every developer to spend 3 months doing pure TDD, just to learn how to test. It's not that testing first inherently makes code better (it doesn't; test code is meta), but it's the fact that to realize the value of testing you have to be good at it, and to be good at it you have to practice. Once you are good at testing you realize that in many circumstances writing an automated test takes exactly the same amount of time as manual testing, plus then you have a regression test.
As far as test coverage is concerned, if you are coding a medium to large application in Rails, you should have base coverage on every controller, period. This is a good basic sanity check that is necessary for a language as dynamic as Ruby (if you're using Java it's an order of magnitude less important, if you're using Haskell, two orders of magnitude). The only excuse is if you're really sure that this code is going to either live on in exactly the same state or be erased. The fact that you're a startup doesn't mean that tests have less value. To the contrary, if you are iterating quickly, you need a safety net when you are refactoring. Once the application reaches a certain size, you will never have time to add test coverage, and there will be certain classes of changes that become impossible because they are just too risky.
I actually find this to be a huge time sink for most projects. Usually in a rails app for a client's application there's only a few fancy controller methods that do anything outside of typical REST scaffolding (there's no way I'm testing all of the typical admin CRUD). Add in something like inherited_resources using thin controllers and you're wasting even more time. IMO only write tests for functionality that's non-standard, which is probably going to be a very small percentage of your controller code.
For client work I find that most of the complicated or non-standard functionality happens on the client side, and rarely on the rails side. Which means a huge portion of my tests would just be fluff (like commenting method names and params manually) if I were to test all of my controllers.
Also, I have to question how much time you save by not writing tests for boilerplate. How much boilerplate is there really? Most business code isn't exactly novel or exciting, but neither is it scaffolding. Even if it is, you should just crank it out and be done with it right? It shouldn't be taking a large percentage of your time, and neither should the tests.
Actually, no.. TDD is almost never used when you fully know the problem!
Look at this way.. would you trust that free source clone of facebook with your data knowing the site security holes i the first release simply because they did not test using something like TDD/BDD? I would not..
That being said TDD/BDD is not agile! Neither is waterfall!
WHAT?!
Agile is critical analysis of processes and tools used in the dev process to reach a set of dev and management objectives and goals by modifying those processes and tools.
You got some of it right and some of it wrong
The kinder gentlemen above have explained why the parent is fundamentally incorrect.
Your whole premise is based on the the idea that your startup is going to fail. In that case why write any code at all?
If your startup doesn't fail then you've really made things extra hard on yourself. One of my favorite things about working on my startup is getting to learn and apply good practices.
It's just as easy to write good code as it is to write bad code. I find that TDD speeds me up not slows me down. It's not as much overhead as people seem to suggest.
You can write new code and then keep running the program to see if it works. Or you could write the code and keep running a test until it works. The test is typically faster for this because you don't have to deal with the rest of the program which is unrelated to what you're working on. Once you have working unit tests the implementation is typically trivial.
In practice, I don't write tests for everything but I code in a way that means I could test everything if I wanted to.
"19 out of 20 startups fail. That means that odds are that you will never see this code again. You'd be a fool to spend any more time on it than absolutely necessary."
The solution is to use Spork to have a Rails test environment running permanently (it also does caretaking between sessions) and then RSpec will access it over DRb. Couple this with a good .watchr file to run only the relevant specs when you update them and you get very quick RSpec tests. It's a bit of a pain to set everything up but it's worked great for me.
http://www.rubyinside.com/how-to-rails-3-and-rspec-2-4336.ht...
this seems to be the easiest thing a refactoring tool would handle. Why wouldn't you re-factor your tests at the same time as your code?
Find it here (scroll down to the link for "Show #74"). http://startuppodcast.wordpress.com/2010/07/10/show-74-kent-...
Or, subscribe to the show here: http://itunes.apple.com/podcast/the-startup-success-podcast/... Look for episode 74.
The problem I notice, though, is many people will either TDD the "right/official" way or not at all and that's a false dichotomy. If a particular type of testing is slowing you down or causing you to be less productive, don't do it! But stick with the tests and processes that do allow you to be quick but without abandoning testing in favor of the old "code and pray" approach.
For example, on a recent project I built it almost entirely in the models only (using TDD) before even hitting controllers or views. It only took a few days to tack those on at the end and I didn't bother doing any testing of them beyond some cursory "make sure the site generally works" integration tests. (I see the value in controller and view tests but.. well.. they'd have slowed me down and the models were far more important.) In a contrast to that, I have a 5+ year old project I retroactively added lots of integration tests to. The models are untested but at least I know if a change screws the app up in a big way because so many different use cases are tried in the integration tests.
TLDR: With TDD, stick with the stuff that works and tone down the stuff that doesn't. Don't feel you have to do things the official/"cool" way - come up with your own processes.
Disclaimer: I'm an agile evangelist, spent years at startups, and I love quality code.
Be pragmatic in your adoption of any software engineering process, agile methods included. Tossing TDD/BDD out the window because you believe it's slowing you down is a mistake. Adapt it to your environment; use the aspects that work.
Remember that the goal of TDD/BDD and testing in general is to deliver useful software that works. Even your startup's prototypes, quickly hacked together, should at some level meet that criteria.
If you're writing code, you have an intention for it, a purpose, and that can be tested on some level or another. It's hard to make an argument that manually clicking around in your browser after writing your implementation is superior to having some form of automated tests in place beforehand.
If you're in the exploratory stage of a startup, you'll pivot often, and you'll be building a lot of throwaway prototypes. You should limit the depth of your test coverage, but you can still do TDD. Just don't be strict about it. Not every method/class/behaviour needs to be tested. You may not know the broad direction your software is going in. When you sit down to code, at least the behaviour of that little piece is known, and it's at that point you can write a test or two to express and validate your intentions.
You can speed up RSpec so much by offloading it to Spork, a test server of sorts that loads your environment.
Developers should not be paid to write tests, only code. If the tests are worthwhile, then they'll get written anyway.
I've seen some developers who write lots and lots of pointless tests... hmm does Model.find(:all) return all the items in the test db? Ok one passing test, does :first return one? Ok, another passing test. I'm not exaggerating.
If your test codebase is full of stupid tests that are actually testing your framework, and if your test suite takes 5 minutes to run, maybe that's why your team has so much time to read HN.
Good, useful tests will test the most critical 10% of the codebase at most. The "money" paths that are critical to your core business. Things like credit card processing, account signups, password resets.
Many of the critical 10% of tests may very well be integration tests, not unit tests. There is no reason to write unit tests if the big problems would be caught by an integration test before a deploy.
If your testing ideology makes all this sound like hogwash, then you probably work in a cubicle where it does make sense to do test your codebase more broadly.
- I wrote the specs before I wrote the code. Essentially, I used Cucumber to define how users interacted with the site, and I used RSpec+Shoulda to define how low-level APIs worked. This rarely took longer than testing by hand: I just wrote something like "When I click on 'Sign in', Then I should see 'You are signed in.'", and that was it.
- I kept a watchful eye on the size of the tests. If the test-to-code ratio ever drifted far from 1:1, I figured out why and fixed it. A 3:1 or 5:1 ratio is a sign that your BDD/TDD process has gone way off the rails, at least in my experience. Common causes are (a) not using Shoulda to test models, and (b) relying on controller specs when you should be using Cucumber (or Steak).
- I used Cucumber for specifying user interactions, and RSpec for testing models. I only wrote controller specs for weird edge-case behavior that was a pain to test with Cucumber. Edit: And I virtually never wrote view specs.
- Refactoring was easy, because I could tear into the code and trust the specs to report any breakage.
I agree, however, about the speed of Ruby test suites. I hate hate hate waiting for specs to run. I get some mileage out of autotest and Spork, but not enough for my tastes.
I do you Shoulda and I don't do controller/view specs (so not mocking/stubbing). I do integration (i.e. request) specs.
I've always had a problem with testing view output like you mentioned e.g. I should see "You are signed in." I'm always making last-minute copy changes, and sometimes would make a change like this to "You are logged in". It's a very simple change, but potentially could break some specs. I'd have to run the suite, see what failed, look at the line numbers, figure out why, then realize it wasn't that my app wasn't functioning properly, but the assertion was tied very closely to a transient message. Again, maybe this is just isn't a good way to test.
I'll agree on refactoring. Nothing is scarier than going in and changing internals, hoping nothing breaks.
http://mrjaba.posterous.com/acceptance-testing-and-cucumber-...
It sounds in general like you might not be doing TDD correctly. Your test cases shouldn't be slow to run, either. Are you actually hooking up with the database in your tests, or are you isolating them properly?
Don't do it, use a real IDE.
An IDE is, well, an integrated development environment - I find the idea of mashing my editor, VCS, build system, debugger, etc. into one tool to be incredibly distasteful.
What you want is a competent editor. Unfortunately, most editors that provide support for things like automated refactoring are bundled into IDEs, hence your confusion.
Uh?
The strength of an IDE is that it understands your code. It knows what a method, a class, a variable, a package is. Because of that, it can give you more assistance in writing your code than any editor that bases its highlighting on regular expressions ever will.
Also, in general, it doesn't matter what you program in: be it an IDE, vim, cat, or a magnetized needle. A good hacker finds a way to write good code.
I can't believe you actually think that typing all these characters (imports, renaming by hand, creating methods by hand, etc..), navigating between classes, finding symbols is making you faster.
vim is great for general text editing, but for Java, nothing can beat an IDE. Try it, you'll be surprised.
Technically, they are hooking up with the database since the attributes are automagically generated from the DB schema (using ActiveRecord on Rails). Maybe there's a better way? I'd love to know.
The refactor you mention is single right click in C# (with ReSharper, and without the frankly terrifying idea of using Regexes to do it). Compounding matters is the fact that a huge portion of your unit tests simply evaporate once you realize that you simply can't have typos or type mismatches in an old dinosaur language like C#. So it's entirely possible that that big suite of testcases it took so long to refactor would not be necessary at all (replaced instead by the occasional int x;).
Once somebody gets the tools together so I can be as productive in Rails as I am in ASP.NET, I'm jumping ship. Until then, I'm hanging tight.
In the mean time, test the parts of the application that do not change. How much can the payment flow, sign in or sign up change? Make sure you have test for those in order to catch regressions.
Know where you are. Are you building to last? Are you building to test an idea? balance.
Build better abstractions.
Practice. Practice. Practice.
And I think there's some truth to that. On a macro-level, how would you even begin to write tests for a search engine or some stock market bot or other notoriously hard problem?
search_on("avatar").should_return("http://www.imdb.com)
best_stock_for(:percent_return, 200).should_return("cisco")
?
These problems an inherently non-deterministic. How do you even begin to write a test for that?
On a micro-level, sure, maybe you're working on a single component. And TDD would help you come up with the interface. But if you don't even know if the answer is a genetic algo, or simulated annealing, or using mechanical turk, or whatever, there's really no point in even trying to freeze the interface. Which is what TDD really does as much or even more than it verifies the resulting code. It defines the interface ahead of time. It's a way to trick developers into writing specifications without using that nasty imprecise context-sensitive language know as english.
But then again, right now we're rewriting a pretty critical piece of code. We've thought a lot about how it works. We had a few meetings about the new approach. Wrote up a quick email with a basic API. And doing pairing and TDD from there, well that's actually working out pretty well. And I'm confident we're getting better code quicker because of the approach.
Ultimately it gets back to the statement that real developers ship. In some cases, BDD and pairing will help you ship higher quality software quicker. In other cases it won't, and it'll end up wasting money and time. And real developers will then use their tools accordingly, and not dogmatically.
Of course, the whole search engine isn't specified by tests. However, if IMDB isn't in the top 10 results for [avatar] or barackobama.com isn't in the top 10 for [obama], something is seriously wrong and a human should look into it.
The rest of your post is pretty good. Not sure why you've been downvoted.
First, they are tied to a current time's context. Maybe next year, IMDB is no longer a good source for movie information, and it's so bad that it's on page 2. It could happen. In theory, your test cases should be consistent.
Secondly, the TDD process is always to write code to pass tests. Well it's pretty easy to write code to return IMDB, but it's really hard to write test suite, that when coded against, would produce Google. That test would look like:
search_on("avatar").should_return("www.imdb.com") if is_really_good_result(imdb)
Once you are Google, then you should have these tests to make sure things keep working. However, I think it is really hard (in a bad way) to do TDD on hard problems.
If you have a known snapshot of data, you can set expectations for how you want your system to behave under those circumstances.
If you create a world where IMDB would rank highest for your indexing algorithm for the term "avatar", then you can expect that when you run a search it will be returned as the first result.
These problems are hard because you evaluate your results with some kind of "quality score" instead of a simple "pass/fail" metric. You need to adapt your testing strategy accordingly.
One useful strategy is to define a decent metric, and try to maximize it. Let's say that you have two sources of data: (1) A list of pages that should rank highly for specific queries, and (2) a list of links that users clicked on and stayed at without coming back to click on other links < 30 seconds later.
Your goal is to write a search engine which ranks these links, on average, as highly as possible for the relevant queries. You do this with the usual techniques of experimental science: Split your data in half, develop against part of it, and hold a part in reserve for your final tests.
If you make a clever change, and your average "good link" position drops from 1.7 to 3.9, you back your change out and try again.
One of my clients actually did something similar with mapping software. Before deploying new code (or a new version of the map data), they ran extensive automated tests, and flagged anything weird for human review. I wrote them several test harnesses, one of which discovered a situation where the driving directions took a right hand turn off an overpass and tried to merge into traffic below... :-)
Granted, these types of "tests" are essentially very high-level integration tests. But they serve the major purpose of tests: Automatically ferreting out disastrous mistakes before you ship to customers.
Yes splitting your data into a training set and a test set is a really good idea for problems like these.
But I'd still argue it's near impossible to generate decent grading first, TDD-style, with only a superficial knowledge of the problem you're trying to solve in that case.
Say you've got one obscure Star Wars character in your test set. Top results are probably imdb and wikipidia and that looks good. Then you've got a slightly less obscure character in the test set. And you stick with imdb and wikipedia as the top results for testing purposes. But he's got a huge page on wookiepedia. He's got a huge fansite. He's got a huge personal page. A twitter account more popular than Austin Kutcher. All popular enough to bump IMDB and Wikipedia out of the top five. Now the test you wrote is broken, and you're coding to broken results.
In that case I don't think we're capable of generating good tests to define the result set FIRST, TDD-style. I don't think we're capable of specifying the application behavior FIRST, BDD-style. I think its better to play around with the data and algorithms first, until you start to get a gut feel for the data. And then you can write some decent tests for the test data.
Same with mapping software. It's probably a good idea for the shipping product to have a test suite that does something like generate 100 random or not-so-random trips, then make sure that: you can get there from here, You can do it in +/- 5% of the time/miles that we've already esablished, etc. But I question how much value those tests have for day 1 or week one or even month one when you're writing the software. There's a lot of legwork before those will even come close to passing. And because you now don't have an un-broken build, people will potentially start ignoring problems with tests that should be passing week one.
And I know you're not saying this, but every time people argue about TDD, the TDD proponents seem to think that no-one else tests. Which isn't true at all. You do need tests. The question is when and how. And the answer isn't always before anything else.
You are completely right. In this case you would start prototyping various solutions to the problem. Once you have explored the domain, you either go for implementing a certain approach using tdd, or you do more exploring.
--fg