Interesting this is being brought up, I used to use this back in 2013 when developing Rails apps, I've been thinking about what a good "standard" spec language would look like in a post-LLM world. I wonder if the cucumber syntax could help here?
In my misspent youth I've tried to introduce Cucumber/Gherkin a couple of different times but it's something with a deceptive promise.
The preconditions are that you have domain experts or business people interested and willing to engage in writing or reviewing these tests. Unless you have this and it's something that those people are going to sustain when the going gets tough you're just making writing tests harder for no real benefit.
Its one of many tools that is founded in one of the two key organizational realizations underlying Agile software development that, unfortunately, is completely ignored everywhere that pretends that they are "doing Agile".
The other one, and the one that is talked about more, is control of the methodology by the individual dev team, rather than as an external dictate from outside.
But the one relevant to Cucumber is that development isn't an activity that the dev team does alone, but a high-contact relationship between the dev team and the people working in the application domain that the software serves.
Like many Agile tools, it is completely broken when divorced from those organizational principles. Unfortunately, while lots of shops are "Agile", those "Agile" implementations are almost invariably completely divorced from those principles, and are the same kind of consultants-sold-this-canned-buzzword-bingo-approach-to-management-and-now-it-is-imposed-on-the-software-team-we-keep-walled-off-from-business thing that the entire Agile movement was a reaction against.
IME this allows the BAs who are writing ticket requirements to also write tests. It works. The dev may have to tweak the tests to get them to compile, but it an org that is willing to spend quite a bit of money on testing (e.g. public sector) this is a format that devs and BAs can read.
I’ve worked with cucumber for a few yeats and only ever seen developers writing Gherkin.
To me it’s just another framework to learn - a failed abstraction. It’s always introduced by idealistic devs who regularly jerks off to conference talks.
Buried in the mess there are a couple of good concepts like it moves away for the 'busy-work' of atomic tests.
But shit like "living documentation" makes me want to vomit.
Are people dying quicker than we can teach this is bullshit? Or are people dying stupid?
The idea non-coders might ever be involved with Cucumber is also fucking retarded.
Last I used it, it was a mess, maybe it's better now. Part of that was those arrogant Selenium motherfuckers.
All this shit is why LLMs are talking over. Fucking retards with their stupid as fuck theory's about auto documentation and Selenium refusing to implement X (i forgot) on some BS 'principal'.
LLMs just get the job done, they might fuck shit up, but they are not arrogant autists about it, and LLMs don't care when they are called useless arrogant autists and double down.
It sounds great on the label, but what you’ll end up with are single-use matchers for almost every line of your cukes, annoyance maintaining the regexes that map your cukes to the actual test implementation, and the “non-technical” people won’t be maintaining the cukes anyway. Sure, let them specify the cukes if they want, but then translate them into regular unit tests or specs and forget maintaining them.
Just like all the other no-code inventions out there, they fail to reckon with the fact that essential complexity isn’t a problem with programming language syntax.
I have ended up writing BDD scenarios as plain code comments with conventions like capital “GIVEN” etc. A simple script to scrape test files and generate a markdown file to browse.
Then you refactor your common helper functions as natural, but you’re not forced to enclose them in steps.
I've always thought this looks cool but then you realize the whole thing is driven by regexs on the "plain english" and I recoil in horror
Given /^a nice new bike$/ do
expect(bike).to be_shiny
end
so I've traded translating business requirements into specs for
trying to regex against business requirements - and probably a lot of back and forth telling people they wrote their gherkin wrong
have never actually tried it for that reason - it just seems worse in every way
Aren’t syntaxes and semantics purposeful? Isn’t knowing the proper syntax and semantic structure part of how we design good software? I don’t understand the desire for “plain language” beyond the notion that ignorant people want to stay ignorant.
Extra layers of busywork that nobody else wants to look at.
If your job is polishing a turd, make the turd shine like a cucumber. I have seen countless hours wasted on making Cuke/Gherkin "pretty" instead of accomplishing anything for the business.
Engineers should refuse to write Gherkin. It's not meant to be for them, it's meant to be for non-engineers to tell engineers (or the systems the engineers make) what they expect will happen. It looks like some flavour of user stories for this reason.
Engineers' responsibility begins at the "code behind" the gherkin layer. Someone needs to enforce this.
Sadly, IMO because gherkin looks like structured language (i.e "code"), and because that's what engineers do, they end up doing it. Either they tricked themselves into thinking they should, or the product team tacitly assumed it was a code thing.
If you're an engineer and you're writing gherkin, you should consider this a "code smell" and consider ditching it. Go for a spec tool one layer closer to your main domain. i.e. just write normal unit & integration tests.
I'm not saying "don't do BDD" or "don't do acceptance tests". You should do that! I'm saying "gherkin should allow product owners/managers to take ownership of reporting on how well the app meets their expectations". But most typically, they don't get it, or they don't want to do it.
I recently worked somewhere where we had dedicated devs-in-test. They wrote most of the e2e tests for the application. They wrote it in Gherkin, which meant that they _also_ wrote the JS/Selenium tests behind them. So they were basically writing two sets of scripts for each test. At some point they explored using a 3rd party cross-device testing platform (lambda test, maybe?) who had their own in-house test script that looked like Gherkin, so it needed a JS/TS code-behind to actually run the tests. So they did it all again!! Such a waste.
I wish the doctest approach won over the BDD approach. I still find myself recreating the doctest approach in miniature...
Specifically I try to make a nice string representation of state, and then tests match off that state.
So for instance I don't want to say expect(x.length).toBe(0) (one syntax, Cucumber goes further in that syntactic direction). Because if the length is 1 then 1!=0 is a very opaque failure. Or if you do expect(x[0].attr).toBe("y") then you've tested one thing but you have to have another test for list length, other attributes, etc. Often you'll leave the others out, expecting them to be ok, but sometimes they aren't...
Investing in a good, stable, matchable string representation will give you a great tool for any number of tests, and it's not just a great tool to verify but also to debug your tests.
I posted this comment in 2013 [1] and I stand by it 12 years later:
Cucumber tries to solve the problem of turning customer requirements into 'real code'. In exchange for that worthwhile benefit, it asks you to implement the most terrible, reg-ex based spaghetti code imaginable.
The problem is that it doesn't solve the original problem AT ALL. And then you are left with terrible reg-ex driven spaghetti code. Like the Jamie Zawinski saying, "now you have two problems".
The lesson here is that software development processes have to pass the 'human nature' test.
The software industry has largely abandoned waterfall development because it just doesn't work well in practice. It doesn't work because people don't know perfectly what they want before they build it. Agile processes usually are much more efficient because they are more closely aligned to how humans solve problems in the real world.
Cucumber suffers from the same issue of being disconnected with reality. In theory, you can find a customer who can give you perfectly written use cases and you can cut-and-paste those into your cukes. In practice, that never, ever works. So let's all stop wasting our time pretending it was a good idea now that it has been shown to not work.
My manager and our qa team like that they can read what we are testing step by step in plain English and reason about it the same way. If they want a new step they often request it in a similar format to the ones we have written, or directly suggest reusing a step from another test.
You dont get that for free, but i disagree with it being a 'mess' of regex...Regex for test steps is usually as simple as a basic string or number capture pattern. It's 100% a skill issue if it ends up 'messy'.
If you need something more complicated than regex you use a data source file for parameters with named columns....Which brings me nicely round to how easy it is for qa or management to look at the data being tested and again, reason about it, make suggestions, spot missing cases, etc.
As a software developer, I have enjoyed occasionally writing Cucumber/Gherkin. However, the roles that were _supposed_ to be writing it did _not_ enjoy it, either refusing to write it or writing unusably-brief and -malformed versions even after extensive training.
This sparks a thought experiment I've been having. In this world where llms can be thought of as the new layer of compilers, things like pickle are likely going to be the main unit of work for humans.
Only now instead of this developing brittle generated tests, it will instead be used by the llm as guidance to generate the actual code and tests.
Before people jump down my throat, I know we are nowhere near that today and I promise I'm not pitching this to my leadership because they would gobble it up too fast.
But for us engineers, I think there is an interesting space for thinking of llms as akin to garbage collection, a feature that allows us to abstract to a slightly higher level of thought. Yes we still need to know how to check under the hood, but this is looking like the right level of precision-flexibility ratio that llms thrive in
Good old cucumber. I remember a discussion over coffee with someone who swore by it back in the day (~2010). They forced customers to sit down and write these things together side by side. Kind of cringe in retrospect. Wonder what happened to them.
Business people don't really care about this stuff. Over the years I realize more and more that we engineers are naive in thinking that business side is concerned with edge cases and race conditions.
Just to give an analogy software people might get better, if you come to a lawyer because, say, you want to buy a house, you are not going to sit down with them and say "given I want to buy a house, when the seller hides water damage costing over $2000, I get to walk away from the deal". You just hope the lawyer is good and will protect you from various edge cases. You have a lot more to deal with than just closing paperwork. You probably are thinking about renovating, moving, getting inspection, etc.
Businesses are just like that with engineering. They don't want to sit down and meticulously analyze every possible edge case. They have other things to do. Especially when stakes are not that high. Most of these errors can probably be resolved with a phone call and a database edit.
I think this is probably for the best. A good engineer will make sure you're standing on a solid ground, and ask the right questions at the right time. They wouldn't need this amount of hand holding. Leave business time to focus on making deals, connections, organizing the whole operation to move forward, etc. Let them give you vague requirements, and crystallize them yourself. It's way better than a micro-managing business that thinks they know exactly how everything should be.
P.S. Also, I'm not sure why everyone is so hung up on regex = bad, it's not like switching to an AST-based language would've made anything better here. Regex is fine imo, just the entire concept isn't.
You can easily build your own clear, simple DSL with Minitest, Pytest, or JUnit - no overthinking needed. Engineers can whip up more readable and reliable tests quickly. Since only engineers are writing tests, there's no need for unnecessary complications.
I am very much for internal DSLs for functional testing, but have never seen the point of external DSLs (it feels wrong that anyone but the implementing team could or should create or maintain them). I passionately believe you should refactor and use abstractions in your test code thought, and in my experience that ends up looking quite like parameterised steps (again, in code rather than English preferably).
I love cucumber but I never follow the pattern to (1) write the feature file, (2) generate partially implemented tests with said file, (3) finish implementing the tests + subject under test.
It's better to (1) write the test, (2) then later once all that's done, extract documentation in a human-friendly format.
I'm a little bit astonished by the comments mentioning regex.
I used Gerkins at work and there was no regex at all.
Once you write it, you can use the sentence for building other tests if you design it for being flexible and variable, i.e. design well enough your code.
Given I'm connected as ___ with password ___.
When I click on the element __.
Then the element ___ is present.
Then the element ___ is present.
[...]
For critical use case, it was enough. For more that's a bad idea: it take way too much maintenance cost.
I've commented elsewhere on lack of buy-in from stakeholders as a huge pitfall.
On a technical level, cucumber is also at odds with the need for a test suite to be easily maintainable. What I mean is that each test (especially e2e tests) will want to do some setup/initialization. This is usually expressed as step definitions.
Over time, an undisciplined team may write several slightly different but effectively identical step definitions. They may also combine multiple steps into bigger steps because usually a spec writer doesn't want to exhaustively define every piece of setup, they just want to write "Given all 200 pieces of input data and mocks are magically in the right place..."
I was able to wrangle the specs into composability using Rule and Background blocks, but at that point we were just programming tests with a shitty layer over the actual code.
37 comments
[ 2.7 ms ] story [ 70.1 ms ] threadThe preconditions are that you have domain experts or business people interested and willing to engage in writing or reviewing these tests. Unless you have this and it's something that those people are going to sustain when the going gets tough you're just making writing tests harder for no real benefit.
The other one, and the one that is talked about more, is control of the methodology by the individual dev team, rather than as an external dictate from outside.
But the one relevant to Cucumber is that development isn't an activity that the dev team does alone, but a high-contact relationship between the dev team and the people working in the application domain that the software serves.
Like many Agile tools, it is completely broken when divorced from those organizational principles. Unfortunately, while lots of shops are "Agile", those "Agile" implementations are almost invariably completely divorced from those principles, and are the same kind of consultants-sold-this-canned-buzzword-bingo-approach-to-management-and-now-it-is-imposed-on-the-software-team-we-keep-walled-off-from-business thing that the entire Agile movement was a reaction against.
To me it’s just another framework to learn - a failed abstraction. It’s always introduced by idealistic devs who regularly jerks off to conference talks.
Buried in the mess there are a couple of good concepts like it moves away for the 'busy-work' of atomic tests.
But shit like "living documentation" makes me want to vomit.
Are people dying quicker than we can teach this is bullshit? Or are people dying stupid?
The idea non-coders might ever be involved with Cucumber is also fucking retarded.
Last I used it, it was a mess, maybe it's better now. Part of that was those arrogant Selenium motherfuckers.
All this shit is why LLMs are talking over. Fucking retards with their stupid as fuck theory's about auto documentation and Selenium refusing to implement X (i forgot) on some BS 'principal'.
LLMs just get the job done, they might fuck shit up, but they are not arrogant autists about it, and LLMs don't care when they are called useless arrogant autists and double down.
Even the US-born engineers cant always agree on how to communicate product ideas!
Just like all the other no-code inventions out there, they fail to reckon with the fact that essential complexity isn’t a problem with programming language syntax.
Then you refactor your common helper functions as natural, but you’re not forced to enclose them in steps.
have never actually tried it for that reason - it just seems worse in every way
If your job is polishing a turd, make the turd shine like a cucumber. I have seen countless hours wasted on making Cuke/Gherkin "pretty" instead of accomplishing anything for the business.
Engineers' responsibility begins at the "code behind" the gherkin layer. Someone needs to enforce this.
Sadly, IMO because gherkin looks like structured language (i.e "code"), and because that's what engineers do, they end up doing it. Either they tricked themselves into thinking they should, or the product team tacitly assumed it was a code thing.
If you're an engineer and you're writing gherkin, you should consider this a "code smell" and consider ditching it. Go for a spec tool one layer closer to your main domain. i.e. just write normal unit & integration tests.
I'm not saying "don't do BDD" or "don't do acceptance tests". You should do that! I'm saying "gherkin should allow product owners/managers to take ownership of reporting on how well the app meets their expectations". But most typically, they don't get it, or they don't want to do it.
I recently worked somewhere where we had dedicated devs-in-test. They wrote most of the e2e tests for the application. They wrote it in Gherkin, which meant that they _also_ wrote the JS/Selenium tests behind them. So they were basically writing two sets of scripts for each test. At some point they explored using a 3rd party cross-device testing platform (lambda test, maybe?) who had their own in-house test script that looked like Gherkin, so it needed a JS/TS code-behind to actually run the tests. So they did it all again!! Such a waste.
Specifically I try to make a nice string representation of state, and then tests match off that state.
So for instance I don't want to say expect(x.length).toBe(0) (one syntax, Cucumber goes further in that syntactic direction). Because if the length is 1 then 1!=0 is a very opaque failure. Or if you do expect(x[0].attr).toBe("y") then you've tested one thing but you have to have another test for list length, other attributes, etc. Often you'll leave the others out, expecting them to be ok, but sometimes they aren't...
Investing in a good, stable, matchable string representation will give you a great tool for any number of tests, and it's not just a great tool to verify but also to debug your tests.
Cucumber was all the rage in 2008, IIRC..
Cucumber tries to solve the problem of turning customer requirements into 'real code'. In exchange for that worthwhile benefit, it asks you to implement the most terrible, reg-ex based spaghetti code imaginable.
The problem is that it doesn't solve the original problem AT ALL. And then you are left with terrible reg-ex driven spaghetti code. Like the Jamie Zawinski saying, "now you have two problems".
The lesson here is that software development processes have to pass the 'human nature' test.
The software industry has largely abandoned waterfall development because it just doesn't work well in practice. It doesn't work because people don't know perfectly what they want before they build it. Agile processes usually are much more efficient because they are more closely aligned to how humans solve problems in the real world.
Cucumber suffers from the same issue of being disconnected with reality. In theory, you can find a customer who can give you perfectly written use cases and you can cut-and-paste those into your cukes. In practice, that never, ever works. So let's all stop wasting our time pretending it was a good idea now that it has been shown to not work.
[1] https://news.ycombinator.com/item?id=6411787#6412391
You dont get that for free, but i disagree with it being a 'mess' of regex...Regex for test steps is usually as simple as a basic string or number capture pattern. It's 100% a skill issue if it ends up 'messy'.
If you need something more complicated than regex you use a data source file for parameters with named columns....Which brings me nicely round to how easy it is for qa or management to look at the data being tested and again, reason about it, make suggestions, spot missing cases, etc.
> The software industry has largely abandoned waterfall development
Variants of waterfall are essential for large, high-risk safety systems. There is no 1SFA dev process.
Only now instead of this developing brittle generated tests, it will instead be used by the llm as guidance to generate the actual code and tests.
Before people jump down my throat, I know we are nowhere near that today and I promise I'm not pitching this to my leadership because they would gobble it up too fast.
But for us engineers, I think there is an interesting space for thinking of llms as akin to garbage collection, a feature that allows us to abstract to a slightly higher level of thought. Yes we still need to know how to check under the hood, but this is looking like the right level of precision-flexibility ratio that llms thrive in
Business people don't really care about this stuff. Over the years I realize more and more that we engineers are naive in thinking that business side is concerned with edge cases and race conditions.
Just to give an analogy software people might get better, if you come to a lawyer because, say, you want to buy a house, you are not going to sit down with them and say "given I want to buy a house, when the seller hides water damage costing over $2000, I get to walk away from the deal". You just hope the lawyer is good and will protect you from various edge cases. You have a lot more to deal with than just closing paperwork. You probably are thinking about renovating, moving, getting inspection, etc.
Businesses are just like that with engineering. They don't want to sit down and meticulously analyze every possible edge case. They have other things to do. Especially when stakes are not that high. Most of these errors can probably be resolved with a phone call and a database edit.
I think this is probably for the best. A good engineer will make sure you're standing on a solid ground, and ask the right questions at the right time. They wouldn't need this amount of hand holding. Leave business time to focus on making deals, connections, organizing the whole operation to move forward, etc. Let them give you vague requirements, and crystallize them yourself. It's way better than a micro-managing business that thinks they know exactly how everything should be.
P.S. Also, I'm not sure why everyone is so hung up on regex = bad, it's not like switching to an AST-based language would've made anything better here. Regex is fine imo, just the entire concept isn't.
It's better to (1) write the test, (2) then later once all that's done, extract documentation in a human-friendly format.
It creates unnecessary work for something that's never to be seen by real stakeholders.
There is a new trend these days using LLM that is similar to Cucumber: Spec Driven Development using AI. You'd be left disappointed again.
Once you write it, you can use the sentence for building other tests if you design it for being flexible and variable, i.e. design well enough your code.
Given I'm connected as ___ with password ___.
When I click on the element __.
Then the element ___ is present.
Then the element ___ is present.
[...]
For critical use case, it was enough. For more that's a bad idea: it take way too much maintenance cost.
On a technical level, cucumber is also at odds with the need for a test suite to be easily maintainable. What I mean is that each test (especially e2e tests) will want to do some setup/initialization. This is usually expressed as step definitions.
Over time, an undisciplined team may write several slightly different but effectively identical step definitions. They may also combine multiple steps into bigger steps because usually a spec writer doesn't want to exhaustively define every piece of setup, they just want to write "Given all 200 pieces of input data and mocks are magically in the right place..."
I was able to wrangle the specs into composability using Rule and Background blocks, but at that point we were just programming tests with a shitty layer over the actual code.