Thanks for the good read, very enlightening. But, I thought the choice of React as an example was interesting since it already provides test utilities. Any specific reason you aren't using these or were they just omitted for simplifying a blog post?
Test Utils with Shallow Rendering support were only added into react in 0.13, so perhaps these methods pre-date that? Test-Utils render into document require jsdom or similar to render into.
It's easy to say "But 0.13 was ages ago", but in reality it was only released back in March, which may seem like an eternity to Hacker News, but in the real world isn't long at all, and not necessarily long enough for businesses to change their working practices if they already have other methods working well for them.
The react test utils work in combination with jsdom. For example, TestUtils.renderIntoDocument will use jsdom as the document.
There is a facebook-sponsored test framework called "jest" that is more akin to mocha or jasmine, but of those three frameworks it is the worst in my opinion. It's absurdly slow, poorly supported, and it mocks all imported/required modules by default which causes me more problems than it solves.
It feels like there's a little bit of comparison mismatch here - I would equate Jasmine to Mocha, as they're both test frameworks, whereas Karma is a test runner.
Mocha comes with a test runner OOTB, but you can also run Mocha tests with Karma.
One of the things I like about Karma is the ability to run the test suite in a number of browsers. We run our suite in Firefox, Chrome and PhantomJS as part of the CI setup.
I wonder if it wouldn't have been possible to set up a complementary test runner which allowed your Jasmine tests to be run with jsdom? That would give you the speed increase, but without needing to port, and would have allowed you to still leverage Karma when needed.
However, a good read and the Karma bootup time is certainly a pain point I've felt!
Of course the comparison between Karma and Mocha is weird, that's why the entire Karma/Jasmine/Phantom combo and Mocha/jsdom is mentioned.
There's one piece missing in the Mocha/jsdom combo actually, the assertion library, and we use `unexpected` for that purpose.
i've been using mocha, just b/c it's the most popular, and tho it's not bad, i dislike that even in programmatic[.] usage, one generally doesn't import tests as a "module" object. somehow it just doesn't jibe w/ notions of code reuse, even if it hasn't become much of a problem in practice. anyway, i got curious about how to do this, started reading the source, and at a cursory glance, it looked like reading in tests as files was baked in pretty deep.
Karma and Mocha don't do the same thing, so this comparison is weird.
Karma is a test runner. Use Karma to launch and run your tests in real browser instances. There is no better tool available for doing this.
Mocha is used to actually create your test suite. It's great at describing how your tests should run, with stuff like before/afterEach, it.only, describe.skip, etc. Use Mocha to describe which tests you want to run and how they should run.
As for file watching (also part of the OP) your bundler should be doing that. Use karma-webpack (or whatever tool you're using to create your bundle).
Finally- probably the best thing about Karma is its wide support in the community. You can find a Karma plugin for just about any other tool you might want to use, including launchers for hosted browser environments like Sauce Labs and BrowserStack.
I agree as well; I was confused as to how they benefited from switching away from Karma as a test runner. It seems that they simply replaced PhantomJS with jsdom, but in the process they lost Karma, which means that they can't run their test suite on other browsers.
Also, from first glance, I'm unable to see a performance improvement by not using Karma. It may be because they're using more tests in Mocha, but the difference isn't huge considering they're losing a lot of functionality.
As you can see from the article, this is not truly a comparison between Karma and Mocha, but I wanted the title to be intentionally provocative, to sparkle some discussion around the topic :)
We use test-automation for real browser testing but prefer to rely on quick and functional unit-testing, thus the choice to go with jsdom.
Regarding the community, you're right, there's a `karma-something` for every need.
However, mocha is pretty popular too, and getting momentum by the day. Also the fact that it simply runs on Node makes it quite easy to obtain what you want without extra plugins (e.g. coverage).
"wanted the title to be intentionally provocative"
Click bait adds no value to the discussion.
There's no either/or when it comes to karma/mocha. As the commenters above mentioned, it's best to use both. Mocha for quick unit testing, karma for in-browser testing.
In-browser testing can be automated to run on a wide range of browsers and platforms using integration tools such as Browser Stack.
By faking the DOM, you remove nerf the greatest advantage of client-side integration testing. Checking functionality under realistic circumstances.
Have you tried Zuul [0]? It does a much better job at this than Karma.
If you want to run automated tests on every browser, you pick up a SauceLabs account, configure the simple zuul.yml file, and you're done. If you wanna run it locally, you run the zuul cli tool and open the link with that browser.
jsdom is great, I have nothing but respect and admiration for its developers. But ultimately, it's not a very good representation of how browser implementations of the DOM really behave, and it's quite lax about letting you do things that won't work in browsers. So often your tests just end up testing that your code works in jsdom. Phantomjs has its own issues but is at least based on a real browser engine.
There are some nice tools that allow you to pipe JS to them for execution. For jsdom, there's jsdom-eval, and for phantom there's ghostface.
Lastly, I don't think there's much point casting about for the exact right assertion library or test harness framework. The important thing is the environment your tests run in, and how your tests are written of course. It doesn't matter if you're calling `assert(realValue, expectedValue)` or `expect(realValue).toBe(expectedValue)`, etc.
19 comments
[ 3.2 ms ] story [ 59.9 ms ] threadhttps://facebook.github.io/react/docs/test-utils.html
It's easy to say "But 0.13 was ages ago", but in reality it was only released back in March, which may seem like an eternity to Hacker News, but in the real world isn't long at all, and not necessarily long enough for businesses to change their working practices if they already have other methods working well for them.
There is a facebook-sponsored test framework called "jest" that is more akin to mocha or jasmine, but of those three frameworks it is the worst in my opinion. It's absurdly slow, poorly supported, and it mocks all imported/required modules by default which causes me more problems than it solves.
One of the things I like about Karma is the ability to run the test suite in a number of browsers. We run our suite in Firefox, Chrome and PhantomJS as part of the CI setup.
I wonder if it wouldn't have been possible to set up a complementary test runner which allowed your Jasmine tests to be run with jsdom? That would give you the speed increase, but without needing to port, and would have allowed you to still leverage Karma when needed.
However, a good read and the Karma bootup time is certainly a pain point I've felt!
There's a great article [0] about tape, which I'd suggest giving a read!
If you want to run your tests on all the browsers, you can use zuul [1], which is also dead simple to use!
[0] https://medium.com/javascript-scene/why-i-use-tape-instead-o...
[1] https://github.com/defunctzombie/zuul
i've been using mocha, just b/c it's the most popular, and tho it's not bad, i dislike that even in programmatic[.] usage, one generally doesn't import tests as a "module" object. somehow it just doesn't jibe w/ notions of code reuse, even if it hasn't become much of a problem in practice. anyway, i got curious about how to do this, started reading the source, and at a cursory glance, it looked like reading in tests as files was baked in pretty deep.
[.]https://github.com/mochajs/mocha/wiki/Using-mocha-programmat...
Karma is a test runner. Use Karma to launch and run your tests in real browser instances. There is no better tool available for doing this.
Mocha is used to actually create your test suite. It's great at describing how your tests should run, with stuff like before/afterEach, it.only, describe.skip, etc. Use Mocha to describe which tests you want to run and how they should run.
You can easily load Mocha onto the page in your karma config using the karma-mocha plugin. Then, one line in your karma config and you're done. https://github.com/rackt/react-router/blob/53623216560e34fbe...
As for file watching (also part of the OP) your bundler should be doing that. Use karma-webpack (or whatever tool you're using to create your bundle).
Finally- probably the best thing about Karma is its wide support in the community. You can find a Karma plugin for just about any other tool you might want to use, including launchers for hosted browser environments like Sauce Labs and BrowserStack.
Also, from first glance, I'm unable to see a performance improvement by not using Karma. It may be because they're using more tests in Mocha, but the difference isn't huge considering they're losing a lot of functionality.
We use test-automation for real browser testing but prefer to rely on quick and functional unit-testing, thus the choice to go with jsdom.
Regarding the community, you're right, there's a `karma-something` for every need. However, mocha is pretty popular too, and getting momentum by the day. Also the fact that it simply runs on Node makes it quite easy to obtain what you want without extra plugins (e.g. coverage).
Click bait adds no value to the discussion.
There's no either/or when it comes to karma/mocha. As the commenters above mentioned, it's best to use both. Mocha for quick unit testing, karma for in-browser testing.
In-browser testing can be automated to run on a wide range of browsers and platforms using integration tools such as Browser Stack.
By faking the DOM, you remove nerf the greatest advantage of client-side integration testing. Checking functionality under realistic circumstances.
Karma shines when you're running integration/e2e tests against _real_ browsers, especially IE.
If you want to run automated tests on every browser, you pick up a SauceLabs account, configure the simple zuul.yml file, and you're done. If you wanna run it locally, you run the zuul cli tool and open the link with that browser.
[0] https://github.com/defunctzombie/zuul
Using Karma and Phantom for unit-testing it's just overhead to me.
There are some nice tools that allow you to pipe JS to them for execution. For jsdom, there's jsdom-eval, and for phantom there's ghostface.
Lastly, I don't think there's much point casting about for the exact right assertion library or test harness framework. The important thing is the environment your tests run in, and how your tests are written of course. It doesn't matter if you're calling `assert(realValue, expectedValue)` or `expect(realValue).toBe(expectedValue)`, etc.