Nice article.. some examples would be helpful in understanding the author's thesis.
"I believe tests should be written in the language most natural for the problem" - I agree with this. However, to assume natural language is the most fit expression I believe is a jump. Natural language is laden in ambiguity to the point where it misses out on creating specifications in a concise way.
I tend to drift towards the field of thought that BDD should be used to constrain the spec - purposely eliminating ambiguity. I think this is the real opportunity in BDD. This article (http://blog.objectmentor.com/articles/2008/11/27/the-truth-a...) illustrates the point well - making BDD specs analogous with a DSL described by finite state machines. "(BDD) connects the human concept of cause and effect, to the software concept of input/process/output".
So, in this sense using BDD tools like cucumber for expressing natural language can possibly bypass the opportunity to constrain the spec by using constrained natural language or DSLs that best describe the problem domain more explicitly.
First off, I think this is pretty well-reasoned. This part struck me as interesting:
"I believe tests should be written in the language most natural for the problem... I have never coded my way through a surfing session."
To take that a step farther, we usually don't write prose-form descriptions of surfing sessions either. Following that, it seems the language most natural to describe a user interacting with an application through a browser is a straight recording of clicks and form interactions (if I understand correctly, this is what Selenium IDE tries to accomplish, although I've had problems with that particular tool), rather than Cucumber's descriptions, which compared to that, are more like code.
This is why I abandoned Cucumber: it seemed to be another way of writing code and left me on the wrong side of the gap between specification and actual interaction. I don't fault anyone who has found it to be valuable—as the poster said, it's possible to write good tests with any tool. Choose one you like.
Surprised that the Robot framework didn't come up here.
Cucumber's domains specific business language approach is great to when you have a larger group of people, and you want to give non-technical staff (product managers) a way to write tests directly instead of requirement documents. Ideally it provides the non-technical people a vocabulary to describe a test, and mapping that vocabulary to the AUT is up to a test engineer.
Selenium can be that lower level of implementation for cucumber, or can describe the test cases itself. As you noted, Selenium provides a simplifying api for DOM parsing and JS execution, which is a pretty reasonable language for describing a test (if not a little low level and requiring the technical chops that cucumber seeks to avoid).
Robot allows the test designer to compound low level actions into higher level ones in a way that is transparent. I think it strikes a pretty good compromise between high level (cucumber) and low level (direct Selenium API calls).
TL;DR'ing myself: you last paragraph makes me think you'd like the robot framework.
Do bear in mind that just writing tests using business language in cucumber doesn't make then run. You still have to use some sort of test library underneath it to power the steps. In my current work I'm using the selenium API to do this, which hopefully is creating readable tests which re-use test execution code that makes updating them to UI changes far easier than just using a recording tool.
Oh, I know, I just was trying to highlight that the underlying library (as far as I know cucumber) is opaque when defining the test, where in robot it's a little less so. Separately, since you're familiar with it - does cucumber allow you to compose new actions from existing actions?
I had a longer post and pared it back... I actually think both frameworks are good in different situations, it just depends how the company is structured and what kind of QA group you have.
FWIW, I have done QA for 11 years, mostly doing automation.
6 comments
[ 3.1 ms ] story [ 26.0 ms ] thread"I believe tests should be written in the language most natural for the problem" - I agree with this. However, to assume natural language is the most fit expression I believe is a jump. Natural language is laden in ambiguity to the point where it misses out on creating specifications in a concise way.
I tend to drift towards the field of thought that BDD should be used to constrain the spec - purposely eliminating ambiguity. I think this is the real opportunity in BDD. This article (http://blog.objectmentor.com/articles/2008/11/27/the-truth-a...) illustrates the point well - making BDD specs analogous with a DSL described by finite state machines. "(BDD) connects the human concept of cause and effect, to the software concept of input/process/output".
So, in this sense using BDD tools like cucumber for expressing natural language can possibly bypass the opportunity to constrain the spec by using constrained natural language or DSLs that best describe the problem domain more explicitly.
"I believe tests should be written in the language most natural for the problem... I have never coded my way through a surfing session."
To take that a step farther, we usually don't write prose-form descriptions of surfing sessions either. Following that, it seems the language most natural to describe a user interacting with an application through a browser is a straight recording of clicks and form interactions (if I understand correctly, this is what Selenium IDE tries to accomplish, although I've had problems with that particular tool), rather than Cucumber's descriptions, which compared to that, are more like code.
This is why I abandoned Cucumber: it seemed to be another way of writing code and left me on the wrong side of the gap between specification and actual interaction. I don't fault anyone who has found it to be valuable—as the poster said, it's possible to write good tests with any tool. Choose one you like.
Cucumber's domains specific business language approach is great to when you have a larger group of people, and you want to give non-technical staff (product managers) a way to write tests directly instead of requirement documents. Ideally it provides the non-technical people a vocabulary to describe a test, and mapping that vocabulary to the AUT is up to a test engineer.
Selenium can be that lower level of implementation for cucumber, or can describe the test cases itself. As you noted, Selenium provides a simplifying api for DOM parsing and JS execution, which is a pretty reasonable language for describing a test (if not a little low level and requiring the technical chops that cucumber seeks to avoid).
Robot allows the test designer to compound low level actions into higher level ones in a way that is transparent. I think it strikes a pretty good compromise between high level (cucumber) and low level (direct Selenium API calls).
TL;DR'ing myself: you last paragraph makes me think you'd like the robot framework.
I had a longer post and pared it back... I actually think both frameworks are good in different situations, it just depends how the company is structured and what kind of QA group you have.
FWIW, I have done QA for 11 years, mostly doing automation.
That thought has a long and troubled history in the world of computers, and I'm not very convinced by it in terms of Cucumber.