This is very clever, but I don't understand why people choose to write natural language tests, rather than code.
Yes, I've heard of those mythical teams that are rumoured to let their project managers/customers/users write acceptance tests, but I've never actually seen one.
I just find it a bit odd that developers would choose to abstract their tests behind a prose based lookup table. Maybe someone can enlighten me?
With Frank you're free to use lower-level testing frameworks such as RSpec or JUnit too. Most people seem to prefer cucumber, but the client-server architecture of the tool allows you to drive it with whatever testing framework fits your needs.
We work together with our clients to craft the language for acceptance tests in cucumber style. It's quite valuable for us, especially on fixed bid projects, in that it saves us from perception gaps later on.
I wouldn't let a customer write acceptance tests in a vacum, in fact we typically recommend the acceptance tests we think are adequate and they sign off usually with modifications.
As a side note, I'm experimenting with RubyMotion and I really like the rspec clone built in, it would be especially nice to use cucumber with it as well though.
Do you really though? Most cucumber features i see end up being 20 lines of run on sentences, or they end up looking like short rspec declarations.
If you use a descriptive style "spec" testing framework, the output can be similar "A banana should always ...", "A banana when given to a customer should ..." which is just as readable.
I have programming students who have difficulty at writing integration tests, much let alone their future customers. If a customer was to try to write a cucumber test for themselves you would surely have to modify it to make it work correctly right?
This is the only argument i've heard for this style of natural language testing, and i've never heard it from a customer. I've never heard a contractors client come running in from the wings yelling "WE LOVE CUCUMBER, AND COULDN'T LIVE WITHOUT IT!!". I would be interested to hear their point of view.
Sorry to troll on cucumber, I think easier iOS testing and more iOS testing is only a good thing. I'm just personally not a fan of this style tests. Been there done that, now I write my tests in the same language that i'm coding in.
Cucumber is a bit over-used, after having done it for a few years with product owners and testers it can be useful to stimulate conversations around edge cases before someone makes a guess and it ends up being wrong. They are pretty much all integration tests which are expensive to implement and fairly slow to run so the majority of my testing is unit testing.
I found that Calabash does almost the same thing as Frank and has support for more mobile phones
http://blog.lesspainful.com/
Couple of big (agile) projects that I have worked had many "real testers" who weren't really coders (or couldn't really code). To them writing "natural language tests" was more suitable than producing code. The 'lower' level test-libs, that the acceptance tests relied, was either written some of the devs or the 'test-devs' in the test-automation team.
Two non-programming product owners feed work to our five-person dev team via cucumber tests. They own the features directory (we make small mods here and there to get them running or to clarify). They're smart, completely capable of mastering Gherkin (Cucumber's dialect) and writing logical scenarios. It's working pretty well so far, 3 months in. We get the features and just make 'em pass!
Here's a sample:
Feature: Manage Credentials
Background:
Given I am signed in as a superuser
And there are 3 credentials
Scenario: Listing Credentials
When I view the list of credentials
Then I should see all 3 credentials in the list
When I add a 4th credential
And I view the list of credentials
Then I should see all 4 credentials in the list
When I change the name of the 1st credential
And I view the list of credentials
Then I should see that the 1st credential's name has been changed
When I delete the 2nd credential
Then I should see all credentials except the deleted one
We've been using cucumber with Rails at our startup since we started. Once we have clients trained up a little bit, they are able to use and understand our cucumber tests very intuitively.
Non-developer people aren't stupid, they just need a little help to understand. Cucumber lets them be valuable contributors instead of someone to grapple with over acceptance testing.
It drives me nuts that the cucumber team basically said that "Get your client to write the tests" is the one-true-use of cucumber.
Cucumber has another fantastic use, human readable, computer executable tests which are great if you work in enterprise environments where you have human test teams / client human test teams etc.
The usual situation is they have some crappy "test" spreadsheet or (if we are lucky) proprietary systems of various quality.
Using cucumber the test teams can create tests that they can read and test manually but ALSO allows us to test automatically as well.
(Note: In this scenario the much maligned imperative style and web_steps.rb are also very useful since they remove ambiguity)
I introduced Frank to the codebase of a shipping commercial app. Just having the ability to write functional tests without having to deal with all the plumbing was a big win for the whole team, devs and QAs alike. Don't let the Cucumber stuff turn you off; Frank is a valuable tool.
10 comments
[ 4.0 ms ] story [ 31.8 ms ] threadYes, I've heard of those mythical teams that are rumoured to let their project managers/customers/users write acceptance tests, but I've never actually seen one.
I just find it a bit odd that developers would choose to abstract their tests behind a prose based lookup table. Maybe someone can enlighten me?
I wouldn't let a customer write acceptance tests in a vacum, in fact we typically recommend the acceptance tests we think are adequate and they sign off usually with modifications.
As a side note, I'm experimenting with RubyMotion and I really like the rspec clone built in, it would be especially nice to use cucumber with it as well though.
If you use a descriptive style "spec" testing framework, the output can be similar "A banana should always ...", "A banana when given to a customer should ..." which is just as readable.
I have programming students who have difficulty at writing integration tests, much let alone their future customers. If a customer was to try to write a cucumber test for themselves you would surely have to modify it to make it work correctly right?
This is the only argument i've heard for this style of natural language testing, and i've never heard it from a customer. I've never heard a contractors client come running in from the wings yelling "WE LOVE CUCUMBER, AND COULDN'T LIVE WITHOUT IT!!". I would be interested to hear their point of view.
Sorry to troll on cucumber, I think easier iOS testing and more iOS testing is only a good thing. I'm just personally not a fan of this style tests. Been there done that, now I write my tests in the same language that i'm coding in.
I found that Calabash does almost the same thing as Frank and has support for more mobile phones http://blog.lesspainful.com/
Here's a sample:
Feature: Manage Credentials
Non-developer people aren't stupid, they just need a little help to understand. Cucumber lets them be valuable contributors instead of someone to grapple with over acceptance testing.
Cucumber has another fantastic use, human readable, computer executable tests which are great if you work in enterprise environments where you have human test teams / client human test teams etc.
The usual situation is they have some crappy "test" spreadsheet or (if we are lucky) proprietary systems of various quality.
Using cucumber the test teams can create tests that they can read and test manually but ALSO allows us to test automatically as well.
(Note: In this scenario the much maligned imperative style and web_steps.rb are also very useful since they remove ambiguity)