I've been wanting to like TAP, but can't seem to find much straightforward tooling around it. There seem to be tons of libraries for producing it (is a library really needed to spit "ok"/"not ok" to stdout?), a bunch of formatting commands (is it really that useful to swap "ok" with a tick and "not ok" to a cross?), then there are things like Smolder and Jenkins which seem interesting but quite heavyweight.
What I'd really like is a commandline tool for tracking test passes/failures over time, graphing trends, tracking how long tests remain in a failed state, etc. Is there anything like this which doesn't require running a Web server and opening a Web browser?
The testing culture in Perl is among the best I've seen in any language. I miss it when I deal with PHP stuff, which rarely has tests, at all, much less anything resembling complete test coverage and nice smolderish output. Ruby and Python are pretty good, but still seem to be less religious about testing than the stuff that appears on CPAN.
TAP is really nice, though I don't know that it's strictly necessary to build a good suite of tests; it's nice how simple it is (making it easy to use it across languages if you have more than one in your app), but I've found the testing infrastructure to be nice enough on Ruby and Python to not really have to dig into making it do TAP. Everybody has invented their own, and it all works OK.
That said, it'd be really nice if everyone could converge on a common output format, and build super nice tools for consuming it, rather than having a bunch of kinda OK tools built in a dozen different languages. I guess then the problem is that your dev toolchain then requires whatever language you're building in, plus whatever language your reporting tool is built in. And, there's always resistance to that, on almost every project I've worked on. That may be changing with API-based architectures, though...there's probably a business worth building around this problem, though you'd likely be selling to big enterprises who have heterogeneous application stacks (and selling to enterprise kinda sucks).
Is there already a CI tool that'll consume from projects in git and is smart enough to understand the individual test results? I know Jenkins has TAP support, but I haven't gotten around to setting it up yet.
Jenkins TAP plugin maintainer here. Let me know if you have suggestions to improve the current TAP support in Jenkins, or any ideas.
Not much of a Perl programmer, but I try to validate the plug-in against the output of some Perl lib tests, and have been wanting to spend some time adding another view to the plugin to mimic http://www.spurkis.org/TAP-Formatter-HTML/test-output.html
10 comments
[ 6.1 ms ] story [ 38.5 ms ] threadWhat I'd really like is a commandline tool for tracking test passes/failures over time, graphing trends, tracking how long tests remain in a failed state, etc. Is there anything like this which doesn't require running a Web server and opening a Web browser?
http://matrix.cpantesters.org/?dist=Test-Harness%203.36;os=l...
TAP is really nice, though I don't know that it's strictly necessary to build a good suite of tests; it's nice how simple it is (making it easy to use it across languages if you have more than one in your app), but I've found the testing infrastructure to be nice enough on Ruby and Python to not really have to dig into making it do TAP. Everybody has invented their own, and it all works OK.
That said, it'd be really nice if everyone could converge on a common output format, and build super nice tools for consuming it, rather than having a bunch of kinda OK tools built in a dozen different languages. I guess then the problem is that your dev toolchain then requires whatever language you're building in, plus whatever language your reporting tool is built in. And, there's always resistance to that, on almost every project I've worked on. That may be changing with API-based architectures, though...there's probably a business worth building around this problem, though you'd likely be selling to big enterprises who have heterogeneous application stacks (and selling to enterprise kinda sucks).
Is there already a CI tool that'll consume from projects in git and is smart enough to understand the individual test results? I know Jenkins has TAP support, but I haven't gotten around to setting it up yet.
Not much of a Perl programmer, but I try to validate the plug-in against the output of some Perl lib tests, and have been wanting to spend some time adding another view to the plugin to mimic http://www.spurkis.org/TAP-Formatter-HTML/test-output.html
Now with Node.JS the TAP community got some more users, and there are also tools being created (though not really popular I think) like http://www.taptinder.org/ and https://github.com/substack/tape.
Here's an archived copy: https://archive.is/a9MFf