I’ve been having a lot of rants about Cucumber of late, as it’s the new shiny thing for agile teams.  Does anyone else have issues with it?  I’ve asked all of my programmer friends to convince me of its worth, and they’ve all failed so far.  I’ve not seen it adding any value above building a good API (and I see it bringing a lot of negatives relative to other possible approaches).

In my experience, I’m seeing -

- Customers/non-programmers never write the tests (because they have very little interest in specifying everything in given-when-then.  They just want to tell us what they want.  And it doesn’t make much sense to specify everything in that format anyway).
- Customers/non-programmers write the tests but it focuses the test effort on writing those kinds of tests, rather than other testing that seems to add more value.
- The tests are written in English, but what the test actually does depends on how the developers convert the english phrases into code (so there’s no guarantee it tests what the customer intended anyway).
- Avoidance of conversation (ie. tests as contracts).
- Cucumber and the related tools (through the toy examples they provide) encourage developers to put lots of implementation detail into the tests (and sometimes to do a lot more testing through the GUI/http layer rather than pushing some of that testing down).
- Refactoring sucks as we lose IDE support.
- Much heavier test artefacts (when a lot of the teams I work with are already struggling with the weight of their agile test automation).
- A continued focus of tests on what the system does and not why it does it and the business outcome.
- Anecdotally, I’m not seeing better outcomes than people were getting with other approaches.  I realise it may be leading to better designs, but then I’d expect to see improvements somewhere in the process.  That looks like it would require better modelling skills than I’m seeing in most of the cucumber tests on my projects.

Most of the examples that I’ve seen where people are claiming success are fairly small applications.  I’m not seeing the approach scale that well.  Yes, most teams could do write their cucumber tests better, but even then, in my experience, other approaches would be more effective and more efficient.

Any thoughts?  If there’s interest, I’ll try and post some examples of what I think those tests might look like if we pushed them into other forms.

I’ve added a couple more agile haiku. The essence seems to be getting a bit less essential, so I think at some point a refactoring is going to be in order.

Time has passed, and I thought it was time to update my thoughts on what’s critical to successful software testing (and development). While originally, I started noting important ideas for agile teams, Increasingly, I find most of these apply no matter what environment I’m working in. Check them out on my Haiku page.

This tweet was forwarded to me yesterday:

martinfowler: Manual scripted testing should be a human rights violation

It bothered me on a few levels.

Firstly, the simplicity of phrasing around manual and scripted testing. Secondly, that agile developers might view themselves as the saviour of oppressed testers everywhere, and the perpetuation of the concept of tester as helpless victims. And thirdly, criticism of scripted testing is hardly new, and it’s a bit sad that one of the key proponents of test-driven development and design has taken so long to come to this idea.

Still, his comments make a nice soundbite, and what he states is often true, but until he says *why*, I’m not sure if he knows what he’s talking about. It just seems a perpetuation of simplistic binary models of automation and prescriptive testing.

We can do better. There’s much more we should be talking about when discussing manual testing and the conditions of testing in corporate life.

I start with these ideas -

- Let’s put under computer control and evaluation the elements of our tests which are well suited to manipulation and/or evaluation by computers.
- Let’s put under human control and judgement the elements of our tests which are well suited to human manipulation and evaluation.
- Let’s prefer exploratory approaches when we want to find new things and adapt to the information at hand.
- Let’s make sure that we record the details for important things that we don’t want to forget to do, for things that are confusing that we can’t fix.
- Let’s be aware that any test have elements of exploration and prescription.
- Let’s be aware that the testing model we work from biases the kinds of problems we’ll find and the kinds of things we’ll miss.
- Let’s choose appropriate levels of abstraction to help manage the cost of our test design.
- Let’s have as many ways as possible to model our test designs so that we can effectively solve the problem at hand.

In particular, the first two points suggest different ways of doing things that blur the lines between traditional automated and “manual” tests, as discussed by Kaner, Bach and Kohl:

http://www.kohl.ca/blog/archives/000193.html
http://www.satisfice.com/articles/boost.shtml
http://www.satisfice.com/blog/archives/118

On the dehumanising of work, a lot of testers pulled out of frontline business roles were probably already having their human rights violated for far less money than they now make as testers. That doesn’t excuse things, but there aren’t as many testers complaining about the status quo as one might expect. My observation is that the nature of testing as practiced in most places drives the majority of those who might want to take a more creative approach elsewhere.

Standard business practice continues to be to throw money at staff retention problems in skilled roles rather than consider how work might be made to suck less. This isn’t uniquely a software testing problem, and the world is some way from tackling this as an issue.

Mike Cohn’s “User Stories Applied” discusses using the INVEST mnemonic as a guide to writing better user stories. I was recently asked to dig up a reference for it, and found this presentation here, with the section on the mnemonic on pages 47 and 48.

As I read it, I noticed that there’s been a change to one of the letters. Whereas the book uses ‘S’ to denote ‘Small’, now it’s become ‘Sized appropriately’.

I think this is a change for the better, as I noticed that every time I talked someone through the acronym, I would have a long-winded conversation qualifying ‘Small’ as ‘Just small enough but no smaller’. This would come about as I tried to explain the tradeoffs between a story being small enough to estimate with some reasonable certainty, small enough to fit within an iteration, and still ensuring that stories are ‘V- Valuable to the customer’, needing to ensure that user stories continue to express clearly a problem or need of a person.

Overly small stories push us further from the original context of the problem, and thus force us to compensate with an increasing heirarchy of ’super-stories’ to help us focus on the bigger picture. These become more noticable when working in shorter iterations than might be common on a Scrum project, so three cheers for Mike in spending the effort to come up with ‘Sized appropriately’.

In the Yahoo software-testing list, Shrini Kulkarni stated

“…productvity as a term is “bad” for creative work like “testing” or “art”. That makes me to feel that I am a low skilled labour on a manufacturing assembly line (not that – it is a bad profession but that does not represent the kind of job I do or I would to do or I am capable of doing)”

It triggered some thoughts from recent projects, and my reply follows.

What bad things happen when we apply the word ‘productivity’ to testing?

Like it or not, unless we are on a fixed price contract with no deadline (and when is there *not* a deadline?), the person who pays is going to be evaluating the value that we provide relative to the money they spend. This seems to be a productivity measurement, and things don’t usually go well when we ignore it.

We are often forced to balance the artistic/creative part of our work with giving our employer the impression that we are being ‘productive’. My understanding of history is that most artists through history have also had to concern themselves with both productivity and art, especially if they want to make a living.

To me, productivity is the key to defensible testing. It’s the idea that I should be striving to get the best coverage that I can for the least cost.

I think where we go wrong is when we define productivity as output or effort, not ‘value’. Unfortunately, the former definitions are common, and I think this is a huge problem in many places that I work. That is, there is a focus on appearing busy and creating artifacts, and not on gathering the information that would most help the project right now.

There are negatives to the project in terms of our personal productivity if we are forced to ignore those parts of our work which give us the greatest satisfaction. For me, this does include the aspects of testing that I consider creative and artistic, but it also includes other things, such as the satisfaction I get from face-to-face communication, developing relationships, working ethically, collaborating and helping the development of others.

It’s not all about art, but we all have days when our inner artist is challenged. Some days slamming the lid down on the piano and storming off the stage is what’s called for. But mostly, we respect the audience…and the show must go on.

Jonathan Kohl pointed me at a position paper from Brian Marick for the Agile Coach Camp.

If you’re in the middle of adopting agile, it’s well worth a read. You can find it here:
http://wiki.agilecoachcamp.org/
tiki-index.php?page=BrianMarickPositionPaper

During coffee with Agile-coach and all-round excellent guy Shane Clauson, in sympathy with yet another of my what’s-wrong-with-agile rants, he pointed me to this blog post from Jeff Patton:

Don’t know what I want, but I know how to get it

While my opinions diverge on some of what he says must be true, I think the important message that he (and others – Check Alistair Cockburn’s writing on this) make is that it pays to plan to iterate. That is, if you’re on an agile project and you don’t see anyone planning to rework things in response to feedback from using the product, you’re probably in for some disappointment.

I think we frequently fail to give our customers an appropriate expectation when it comes to (agile) software development. Having them read this isn’t a bad start, but you’ll want to figure out how to make this message your own.

I’ve had a few common rants on most of the agile projects I have worked on. Developers bogged down in the detail of stories, while the critical goals of the system wound up ignored, or realising at the last minute that all of the stories built would do nothing useful.

The ideas I came to as a result of the problems I observed were -

- Compensating by starting with extremely high-level stories that defined the critical user and system goals, then progressively breaking these down into tasks.
- Ensuring that the intent and goal is clear from the story title
- Compensating by coaching the testers to ask the critical questions “Why is this feature being added? What problem does it solve and what value does it add?”. The testers would try to ensure that the context was made clear in the story. Links to the higher-level stories in Jira (the tool I most commonly encounter) also helps to provide context.
- Writing high-level acceptance criteria at the top level of stories to help define alternate paths for each goal. These often provided clear boundaries with which stories could be broken down into sub-tasks or sub-stories.
- Evaluating frequently against the high level goals.

I’ve only had a few chances to really try this out, but from a recent project experience with a little more involvement at the early stages of the project, it seemed I was on the right track.
At some point, being aware of Alistair Cockburn’s work, and being a regular reader of his blog, I realised Alistair had probably written about most of this, and I picked up one of his books on use cases. I expected that he’s figured most of this out and more.

Then before I got around to reading the book, it was confirmed for me with this blog post, which I highly recommend taking a look at. Even if you’re not going to apply use cases, it’s worth asking his questions of your project and seeing what you can learn from them.

If you are using use cases, are you avoiding the problems described there?

‘Capturing the essence’, or ‘core’, has been a key theme in some of my work recently, and in several of the books I’ve been reading. So over a drink with Michael Ruschena tonight a couple of these came out as we linked ‘the core’, ‘agile’, and haiku – poetry that captures the essence. I’ve been coaching a team around testing on agile project, so I thought it might be fun to try to capture the essence of that…

I thus present agile testing haiku.

Note: I’m just going to keep adding new ones to this post so that they’re all in the one spot. And increasingly I’m finding these are not just about agile development, but software development generally.
————————————————–

Method it is not
Apply values, principles
To meet your own needs

Planning game is key
How do we know when we’re done?
Check the product goals.

Can’t make it all fit?
You’ll need to take something out
or get more money.

This user story
Does not express your intent.
Why, why, why, why, why?

Push the risk up front
Shorten the feedback cycles
Highest value first

Development, test
Are all part of the same team.
Try working closely.

Good acceptance tests
Set goals but skip one main thing -
Implementation.

We got a green bar,
Why is the customer sad?
Just add a story.

I didn’t do it.
It wasn’t in the story.
Look, new offshored jobs!

Exploratory tests
Help find problems most quickly
but make extra work.

Skilled software testers
Use many different models
Try many angles

Though up-front tests help
We can get stuck in a rut
Go and have a beer

Embrace fuzziness
Precise is not accurate
Band your estimates

Some we can know now,
but some we must build to learn.
Best tell the owner.

Start with a small team
When the hole size becomes known
let everyone dig

Unit, story tests
Why is our build getting slow?
More techniques will help.

How are we going?
If you can’t tell at a glance
have visible charts

Rough iterations?
Preserving uncertainty
may help things improve

Estimate fully
But to manage greatest risk
Take away two thirds

The test strategy…
What information and how,
tied to the vision

Left values or right?
What’s the pairing’s principle?
You’ll need skill at both.

The grind of agile
Sprinting then taking a break
helps you sustain pace

What is knowable?
Predicting or reacting?
Find your middle ground

The requirements.
Stories are only a third.
What about the rest?

Who’s your customer?
You have no ******* idea?
Find out quick or stop.