Poll: Do you test your code?
Do you have tests that run every time you push and ensure that the functionality on your site works?
There's always a lot of debate around testing and I'm interested to see how much people do and how satisfied they are with it
IF YOU'D LIKE TO ENCOURAGE OTHERS TO ANSWER, PLEASE UPVOTE - TY
350 comments
[ 2.1 ms ] story [ 268 ms ] threadHow?
People who don't really write tests will tell me that the advantage of unit testing is being able to see when code changes have broken stuff (which is fair enough and true).
Those who regularly write unit tests will probably bring this up- but often their first point will be 'It helps to structure code properly, make me think about dependencies, modularise code appropriately'
There are people who could write quality software with good test coverage without following TDD style.
what are you even saying
i dont even
(
More concretely, if you use testing to drive your refactors and architecture--as opposed to, say, finding pain points in normal code or actual design time in preproduction--I would be concerned that you are "guardrail programming", as a gentleman put in a talk I saw recently.
When we drive, we don't have guardrails to bounce us back on the road every time we veer off--they're there to protect us against accidents or when something goes seriously wrong with our vehicle. If you told somebody that you drove from city A to city B by hugging the guardrail, they'd say you were nuts.
Similarly, depending on unit tests to do design is strange--they're there to be sure that your code functions according to contract.
)
Writing tests makes you think about how pieces of your code interact with each other, dependencies etc.
As an example, if you're trying to test Function A and are finding you need tens of lines of setup code to be able to do so, then that would be a warning sign that you may want to think about refactoring out some of those dependencies
Unit tests are THE most overrated buzzword of the last 10 years.
If you are using your post as an excuse for not using automated testing at all, I completely disagree. That's the bad kind of developer laziness.
On the other hand, I do have to concede that when competing against people who don't use unit testing on the open market, I come off looking like a wizard in terms of what I can accomplish in a reasonable period of time and the sorts of things I can do (successful major changes to large existing code bases you wouldn't even dream of starting), so maybe I shouldn't try so hard to encourage others to use them sensibly. So, I mean, yeah, totally overrated. Have I also mentioned how overrated syntax highlighting is? You should totally just shut it off. Also, fixing compiler warnings are for wusses, and what moron keeps putting -Werr in compilers?
Every line in a codebase has a cost, including tests. I'd rather deal with a code base that's as trim as possible.
I've done unit tests before, but I don't find that they help that much, because they don't solve the most common source of actual production issues: things you didn't think of.
Problem with that is that, to get that trust, I need to know that unit tests exist, and, preferably have spent time writing or reading them. Question then is whether that time would not be spent better on reading the existing code. I think that, often, the answer to that is "no", but I cannot really argue that.
Perhaps, it is because writing unit tests puts you explicitly in "break this code (that may not have been written) mode". Writing a unit test that calls a function with some invalid arguments and verifies that it throws is often simpler than reading the code to verify that. Also, unit tests may help in the presence of bug foxes and/or changing requirements. Bug report/Requirements change => code change => unit tests break => free reminder "oops, if we change that, feature X will break".
And then these people will be very surprised when I pull off a fairly large-scale invasive refactoring successfully, and deliver product no engineer thought possible.
I'm not hypothesizing; this has been my career path over the past five years, and I have names and faces of the cynical people I'm referring too. You can not do the things I do without testing support. I know you can't, because multiple people who have more raw intelligence than I try and fail.
It is equally true you can't be blind about dogma, 100% coverage being a particularly common bugaboo, but I completely reject the idea that the correct amount of automated testing is zero for any non-trivial project.
As mentioned in my other post, first I had to bring it under test as is, then de-globalize a lot of things, then run the various bits across the network. Also testing the network application. Also, by the way, releases were still being made and many (though not all) of the intermediate stages needed to still be functional as single devices, and also we desire the system to be as reverse-compatible as possible across versions now spanning over a year of releases. (You do not want to be manually testing that your network server is still compatible with ~15 previous versions of the client.) And there's still many other cases I'm not even going into here where testing was critical.
The task I'm currently working on is taking a configuration API that has ~20,000 existing references to it that is currently effectively in "immediate mode" (changes occur instantly) and turning into something that can be managed transactionally (along with a set of other features) without having to individually audit each of those 20,000 references. Again, I had to start by putting the original code under a microscope, testing it (including bug-for-bug compatibility), then incrementally working out the features I needed and testing them as I go. The new code needs to be as behavior-similar as possible, because history has shown small deviations cause a fine spray of subtle bugs that are really difficult to catch in QA.
I could not do this without automated testing. (Perhaps somebody else could who is way smarter, but I have my doubts.) The tests have already caught so many things. Also, my first approach turned out wrong so I had to take another, but was fortunately able to carry the tests over, because the tests were testing behavior and not implementation. (Also it was the act of writing those tests that revealed the performance issues before the code shipped.)
This isn't a matter of large-scale requirement changes on a given project. This is a matter of wanting to take an existing code base and add new features that nobody thought of when the foundation of the code was being laid down 5-7 years ago. (In fact, had they tried to put this stuff in at the time it would have all turned out to be a YAGNI violation and would have been wrong anyhow.) Also, per your comment in another close-by thread, the foundation was all laid down prior to my employment... not that that would have changed anything.
The assumption that large-scale changes could only come from changing requirements is sort of what I was getting at when I was talking about how the limitations of not-using-testing can end up internalized as the limitations of programming itself.
Might I also just say one more time that testing can indeed be used very stupidly, and tests with net negative value can be very easily written. I understand where some opposition can come from, and I mean that perfectly straight. It is a skill that must be learned, and I am still learning. (For example: Code duplication in tests is just as evil as it is in real code. One recurring pattern I have for testing is a huge pile of data at the top, and a smaller loop at the bottom that drives the test. For instance, testing your user permissions system this way is great; you lay out what your users are, what the queries are, and what the result should be in a big data structure, then just loop through and assert they are equal. Do not type the entire thing out manually.) But it is so worth it.
From scratch work is the easier part of programming.
The point I was making was that he used unit tests to confirm his design (as a safety net) and not as a primary design tool.
Few questions:
1) How many lines of code is in that man-century project? Is the number of lines of code ~proportional to the number of man hours, or lines(man-hours) function is ~ logarithmic?
2) How does your typical project look like (or how does that project look like) in terms of testing vs coding? Do you spend few months of covering old code by tests and only then start testing? Or you do "add tests - add features - add tests - add features - ..." cycle?
What's the proportion between time spent on writing tests and writing code?
3) What's the proportion of time you spend directly working (analyzing requirements/testing/writing code) and generally learning (books, HN, etc.)?
4) Do you do most of the work yourself or you mostly leading your team?
5) How do you pick your projects, and when you pick them - what are your relationships with the clients: Fixed contract? Hourly contract? Employment?
Thanks!
But I've never, in 15 years of development, had to rewrite half of an application I've already written.
Spending a large amount of extra time and energy, things I don't have an excess of to begin with, for a "might" or a "maybe" seems like a rather poor choice to me.
Most of the time, testing takes more time than writing the code, so throwing optimality under the bus can be the best choice. If it's Good Enough nobody's going to rewrite, but I wouldn't see it as something inherently negative or shameful, it's just a question of priorities.
I certainly support unit testing, as its essential--and anyone telling you otherwise is bonkers--to ensuring that code follows contract.
That said, if unit testing was great for design but didn't spot errors, it'd be useless. Whereas, if it was useless for design and good for errors, that's okay, because I can do the design work myself.
There are many ways we try to improve code quality and make sure we get it right. Automated test suites are only one of them. Software design needs to take multiple factors into account, and letting one of them arbitrarily dominate all others is a dangerous path to take.
Beyond a certain point, I think automated tests that give simple yes/no answers are no longer a particularly effective way to test certain types of complex algorithm. Sometimes there are just too many possible inputs and interactions between different effects to get a sensible level of coverage and draw any useful conclusions from that kind of testing alone. You might still have some automated tests, but they are more like integration tests than unit tests at that point.
Internally, you could try writing almost-unit-tests for the implementation details, but then you get into the usual concerns about backdoor access and tying tests too closely to implementation details that might change frequently. Alternatively, some form of careful algorithm design with systematic formal proof might be called for. Maybe instrumenting the code and checking the actual values at key points will highlight errors that aren't yet manifesting as faults, things that a boolean automated test would miss because they haven't violated some arbitrary threshold but which form an unexpected pattern to a knowledgable human observer. However, in these cases, you really want the code to be as simple as possible, and hooks to permit internal access to run some automated test cases as well could cause an awful lot of clutter.
My estimate is that 98% of all programming everywhere is as algorithmically trivial as calculating a weighted modulo 11 checksum - probably more so - and it acquires its bugginess from accidental complexity due to poor factoring, and from conflicts at interfaces. Test-driven development is pretty good, in my experience, at helping ameliorate both these problems.
Of course, that doesn't mean I actually do it 100% or even 80% of the time. I'm happy to agree that it's no panacea: testing threads and UIs are particular pain points for me, and usually I substitute with either Thinking Really Hard or just Not Changing Stuff As Much
Formal proof for me is stuff I learnt at college, forgot subsequently, and keep meaning to reread up on. Thank you for prompting it back up my TODO list
I think it depends a lot on your field.
If you're working in a field that is mostly databases and UI code, with a typical schema and most user interaction done via forms and maybe the occasional dashboard-type graphic, then 98% might even be conservative.
On the other hand, if you're doing some serious data munging within your code, 98% could be off by an order of magnitude. That work might be number crunching in the core of a mathematical modelling application, more advanced UI such as parsing a written language or rendering a complex visualisation, other I/O with non-trivial data processing like encryption, compression or multimedia encoding, and no doubt many other fields too.
Generalising from one person's individual experience is always dangerous in programming. I've noticed that developers who come from the DB/business apps world often underestimate how many other programming fields there are. Meanwhile, programmers who delight in mathematical intricacies and low-level hackery often forget that most widely-used practical applications, at least outside of embedded code, are basically a database with some sort of UI on top. And no, the irony that I have just generalised from my own personal experience is not lost on me. :-)
This can lead to awkward situations where practical problems that are faced all the time by one group are casually dismissed by another group as a situation you should never be in that is obviously due to some sort of bad design or newbie programmer error. I'm pretty sure a lot of the more-heat-than-light discussions that surround controversial processes like TDD ultimately come down to people with very different backgrounds making very different assumptions.
Tests are only marginally important at the time you're writing the code they test. The real value comes months later when something else causes the test to fail, and now you a: know the code is broken, and b: have a clear specification what what that code was supposed to do.
Firstly, even if automated testing isn't appropriate for a particular part of the code, there should still be other forms of quality checking going on that would pick up a broken feature before the code is accepted, and certainly before the product ships. If this doesn't happen, you're relying on a limited set of automated tests as a substitute for things like proper code reviews and pre-release QA, in which case IMNSHO you're already doomed to ship junk on bad days.
Secondly, if you can break one piece of code by changing a completely unrelated bit of functionality elsewhere, you have other fundamental problems: your code isn't clearly organised with an effective modular design, and your developers demonstrably don't understand how the code works or the implications of the changes they are going to make before they dive in and start editing (or even afterwards). Again, you're already doomed: no amount of unit testing is going to save you from bugs creeping in under such circumstances.
Finally, unit tests are not a clear specification of anything, ever, other than the behaviour of a specific test.
Basically, if you consider automated unit testing a substitute for any of
(a) maintaining a clean design
(b) doing an impact analysis before making changes to existing code
(c) writing and updating proper documentation, including clear specifications, or
(d) proper peer review and QA processes
then I think you're suffering from precisely the false sense of security I mentioned earlier. In many contexts, unit tests can be great for sounding alarm bells early and giving some basic confidence, but even in the most ideal circumstances they can never replace those other parts of the development process.
The only QA I've ever worked with that was worthwhile spent their time writing automated tests - they were programmers concentrated in test. Otherwise, you're literally saying 'It would be cheaper to pay this room full of people to do what a machine can do instead of paying 1/10th their number to write the same thing as a test', which is essentially never true.
Each test has the opportunity cost of writing some part of a new feature for your customers. But so does every minute spent of fixing bugs that would have been caught with more testing, at a fraction of the cost.
Article about the group that writes the space shuttle software, sort of relevant?: http://www.fastcompany.com/magazine/06/writestuff.html
When I saw my codebase has 'all' functionality tested, I mean we don't commit code without tests included too. I think that's a pretty reasonable definition.
However, even if the sin function is already tested elsewhere, you will still need further testing to ensure that you are calling it correctly (e.g. not confusing degrees and radians).
EDIT: Yes, I read it wrong - clearly need coffee...
That said... that kind of coverage isn't quite as useless as it might seem. If your tests do execute every line, even in a completely contrived way, you will catch a lot if you change your code. You just tend to catch more of the "wrong number of arguments passed to a method" kind of error than "you are allowing the autopilot to try to land the plane 100 feet below the runway" kind of error ;)
A full run of our test suite literally takes months on a cluster of hundreds of CPUs (obviously, there are also faster versions of the tests which are run frequently). While I have a long list of additional test coverage that I would like to add, what we test is much closer to "all functionality" than it is to "a few critical things".
I'd say that a lot of good responses would have been in between those two.
I think it's safe to assume that anyone who selected "All functionality" actually means "Most functionality". Also I think we can assume that a good proportion of people who selected "A few critical" would belong in the "Most" bucket.
However, if you consider functions on the code level (e.g. Java methods) then organisations with 100% coverage will be thin on the ground. If you go further and consider line coverage, almost nobody will have 100% coverage.
A common problem with organisations claiming to test all functions is that they will only test the happy path - there will be few tests for things like unexpected or illegal input etc.
I am seriously considering putting together a "Software Engineering for Small Teams" course or set of articles. With a little bit of expertise, you can inject testing in to most projects, use the minimum of Agile that'll help, and generally massively raise your game - and by that I mean code faster, better, and more reliably, with considerably less stress.
(edited: turns out I forgot which year we're in :-P)
*2012
I used to always write proper full-fledged tests. Then I started my startup, building a product in the few hours left after a demanding high-stress job and a tumultuous private life.
Within a few weeks, I stopped writing tests. Within a few more weeks, I turned off the test suite.
I wrote the product, got it working, received market feedback, realized my model was all wrong, rewrote the entire domain model and UI multiple times all to finally realize that my component boundaries were all wrong and intuitively understanding where they should've been.
Now I feel confident about an architecture that will stay stable for 12+ months and each new component I write is properly tested.
In the meanwhile my lack of tests is starting to bite me very slowly, but I find that I'm just slowly replacing all 'bad parts' with properly tested components with clearly defined boundaries, rather than changing existing code.
And in the end I'm really happy that I decided not to test as much. It has it's place but when your time is really precious and you're trying to mold your software to fit the market needs, it just isn't worth it.
I don't know how many others are in a similar situation but, for me, sometimes it just ain't f*ing worth it.
http://www.writemoretests.com/2011/09/test-driven-developmen...
I'm always amazed by how well the whole 'technical debt' analogy holds up. Yes, leveraged development at the beginning is fast, and sometimes a good idea for getting to MVP. But the cost is still there, and will become apparent, and needs dealing with.
One of the keys to making that work for us is a short feedback loop. We automatically release on every commit, which means every couple of hours. Speculative features get minimally implemented; if they look good then we beef them up more. Our goal is to avoid not just the unneeded tests, but the unneeded feature code too.
I'm personally pretty happy with the testing in that we don't have to spend much time on debugging or manual testing. It's very nice to make a major change, poke at it a little bit, and then ship it with a fair bit of confidence that it will work.
Experimental features may be very short-lived, or require extensive tweaks, and the technical debt that accumulates from not testing may never arise over their lifetime. Once you're sure it's going to stick around forever, do it right and cover it with tests.
When you're in this position, your going to get more value out of creating a smaller set of functional integration tests that cover the critical functions of the project. Sure, adding new tests as you add functionality is a good idea, but it's not going to result in total coverage for a very long time.
Of course, tests don't actually tell you these things. But they can tell you that your assumptions were wrong, or that your trivial change broke feature xxx, and that's crucial information to have.
I appreciate the idea here, and I've done the same in certain circumstances, but typically that means writing tests for bits of functionality that I need to touch.
Without good automated, easy-to-run tests, you're going to blow more time fixing bugs and ad hoc testing in the long run.
Thinking you save time by not testing is a lie on all but the most trivial of projects.
Does an ER surgeon always have the time/bandwidth to scrub hands before surgery?
Does it potentially take the surgeon several days to scrub before an emergency surgery?
Edited to add: I appreciate the analogy, but it's flawed. If someone comes to me and says "here, developer A is on holiday, and we have this bug that it causing massive disruption in the field," is it appropriate for me to say "well, I can do that, but it will likely take me five days so I can understand the codebase and write the appropriate unit test suite."
This is circumstance I thinking about, not necessarily inheriting a codebase and having to add features to it. In that case, certainly, I'm going to take my time, read the code, and write tests.
In fairness, I recognize that this isn't always the case in the real world. Sometimes you really do need to just blindly attempt to fix something, and there's nothing to be done about it. But it should never become a regular occurrence, and you should never get comfortable doing it. First thing I would do is tell my manager exactly why I'm uncomfortable, and what a conservative assessment of the risk is. If we decide to go ahead with the change anyway, I would create two new entries in the bug tracking system, which should be developer A's top priorities as soon as she returns: thoroughly vet my changes, and DEVELOP A SET OF TESTS.
It just troubles me that people are so often willing (and eager!) to waste a lot of time doing half-assed manual testing when they claim not to have any time to write tests. Especially when the state of the art in test automation is better than it has ever been.
This has me thinking that the importance of test automation is related to the proposed frequency of changes. If someone wants a one-off change for something this very second I'll just change it. If someone wants me to inhabit a codebase for any length of time, I'll always set up tests for it. The problem is where you can't tell the difference between those two scenarios until it's too late.
How are you verifying you fixed the bug otherwise? By changing some code, building the app, and running it to verify the bad behavior doesn't happen anymore? I don't really see how not writing a unit test (assuming the code is unit-testable in the first place) saves you any time. You are doing testing anyhow.
And if it was a critical bug, personally I'd want to feel as confident as possible that I fixed all permutations of it.
I feel that it is unreasonable to expect that you will be able to pick up any code base and immediately write sufficient tests to get coverage on a majority of the code base. Speaking from my experience picking up old code bases, just being able to write isolated unit tests would require refactoring most of the code base, which is typically not something you will have time to do before you're expected to do other work.
I can't think of a single manager that I've worked for who would accept me saying, "it's going to take me 3-6 months of refactoring & building tests before I can start fixing bugs and providing enhancements."
I can't think of a single developer I've worked with who would try that approach.
When a bug is identified in a project with few-or-no tests, the approach that I usually see taken is to write some sort of large, slow integration test that exercises bug, then fix that. That allows you to prove that the bug exists and prove that the fix fixes it, at least for the documented case(s).
There's no reason to cover an entire legacy code base with tests if you're only changing a small portion of it.
My area of expertise is adding tests to legacy codebases. Obviously you're not going to hit the whole thing overnight. But that's no excuse for not having /any/
You may say "I'll thank myself later", but in this sort of business there won't be a later if we're not fast enough. It's a lesser of evils thing.
It would be nice if testing were a faster thing to do. The faster you can do it, the lower the threshold would be for this sort of a judgement call.
http://www.bnj.com/cowboy-coding-pink-sombrero/
(article's not mine, but might as well be)
Stay thirsty my friends.
Just as an example, how do you test a parser that processes large amounts of sometimes sloppy semi structured text? Whether a particular defect should be classified as a bug in my parser or as a rare glitch in the source data is undecidable until I know how often the defect occurs.
What I need is a kind of heuristic test framework that makes sure the parser doesn't miss any large chunks that I only find out about weeks later if at all. I cannot supply individual test cases for everything that could possibly be found in the source data.
Perhaps not, but you can supply test cases for known problems you might encounter, as well as ones you've solved after they've been encountered.
I believe this transformation is entirely to do with the fact that I paired with a brilliant developer every day for 6 months who really helped to answer all my questions and show me how to test a variety of different things. I truly believe that unit testing (and testing in general) is a hard thing to grasp without being able to learn from someone over a long(ish) period of time.
I realized that I hated testing because I didn't know how to do it and wasn't good at it. I also didn't understand what the essence of a unit-test was; my tests would often cross multiple integration boundaries (ie: hit the db and the server) and were really more like bloated integration tests. Once I had sorted that out and was able to see a variety of techniques for testing specific scenarios I realized that I actually enjoyed testing and the satisfaction of knowing my code was covered against defects started to be a big motivator.
To answer your specific question about JavaScript testing, I've been using JasmineBDD[1] for the last 2 years and have found it a joy to use. It really makes testing things easy and has tools that allow you to isolate your tests down to the individual units.
[1] http://pivotal.github.com/jasmine/
Sounds great. What else did you learn?
I wish someone could make a simple service that allows me to set up my web app, set up test parameters that it tests each and every time, and tell me if it failed or not. I want to automate my babysitting.
Their landing page is a little weak but I dig their UI for setting up tests.
http://seleniumhq.org/
There is some reason that your developers aren't engaged in the work. Figure out why they don't care about working code or the user experience and fix that.
If you plug the obvious holes, you won't have fixed your quality problems; you'll just shift them to the places where you won't notice them right away.
I can't agree more with wpietri above. There surely is a non-technical problem at play. That isn't to say that you shouldn't try to automate your babysitting, but if your need for babysitting is that severe, you have other problems.
you can upload a bunch of Selenium tests, indicate when you want to run them and we'll send alerts if a test fails.
TeamCity is good for automating with both kinds of "triggers".
If you have something really long running (eg you make a database and have a two week test), then you may be able to minimize your test (possibly automatically) and use git/hg bisecting to find it.
Fuzz testing (finding holes in your code) can be run separately, and again, you can find the root cause through minimization and bisecting.
Sometimes, it can be frustrating to have to stop and write a unit test before send out something for further review, but computers are more reliable testers than humans are.
Learn, use, and love unit testing, or face the consequences.
I am not automatically testing that, perhaps I am missing something, but automating the steps to log in to Facebook and revoke the token and then also making sure that SendGrid sent the email correctly just seem impractical.
But most of the tests are more fine-grained. So in your example, I'd test the core logic against fake Facebook API responses and a fake outgoing email call. That lets me easily test some of the weirder cases. E.g. if Facebook breaks, will the job skip that user and keep going rather than blowing up?
I wish I wrote tests more. Maybe I'm to impatient. :(