I’ve been having a lot of rants about Cucumber of late, as it’s the new shiny thing for agile teams.  Does anyone else have issues with it?  I’ve asked all of my programmer friends to convince me of its worth, and they’ve all failed so far.  I’ve not seen it adding any value above building a good API (and I see it bringing a lot of negatives relative to other possible approaches).

In my experience, I’m seeing -

- Customers/non-programmers never write the tests (because they have very little interest in specifying everything in given-when-then.  They just want to tell us what they want.  And it doesn’t make much sense to specify everything in that format anyway).
- Customers/non-programmers write the tests but it focuses the test effort on writing those kinds of tests, rather than other testing that seems to add more value.
- The tests are written in English, but what the test actually does depends on how the developers convert the english phrases into code (so there’s no guarantee it tests what the customer intended anyway).
- Avoidance of conversation (ie. tests as contracts).
- Cucumber and the related tools (through the toy examples they provide) encourage developers to put lots of implementation detail into the tests (and sometimes to do a lot more testing through the GUI/http layer rather than pushing some of that testing down).
- Refactoring sucks as we lose IDE support.
- Much heavier test artefacts (when a lot of the teams I work with are already struggling with the weight of their agile test automation).
- A continued focus of tests on what the system does and not why it does it and the business outcome.
- Anecdotally, I’m not seeing better outcomes than people were getting with other approaches.  I realise it may be leading to better designs, but then I’d expect to see improvements somewhere in the process.  That looks like it would require better modelling skills than I’m seeing in most of the cucumber tests on my projects.

Most of the examples that I’ve seen where people are claiming success are fairly small applications.  I’m not seeing the approach scale that well.  Yes, most teams could do write their cucumber tests better, but even then, in my experience, other approaches would be more effective and more efficient.

Any thoughts?  If there’s interest, I’ll try and post some examples of what I think those tests might look like if we pushed them into other forms.

I’ve added a couple more agile haiku. The essence seems to be getting a bit less essential, so I think at some point a refactoring is going to be in order.

I’ve been helping out testing Jonathan Kohl’s Session Tester over the last few months. While the second release was the first that appeared to meet my needs to a degree, it had a few critical problems that lead to data loss. If I’m taking notes, the most critical feature is that my notes be preserved.

I’m pleased to say that the latest release, version 0.2.1 is much more stable. I’ve had no issues with notes going missing over the last six weeks or so, and can recommend picking it up and taking a look. You can download it at http://sessiontester.openqa.org/download.html

Session Tester is based on James Bach’s Session Based Test Management method. The long-term road map also includes some other exciting ideas. Right now though, this is a product very much in early development. You won’t be able to run your test effort using SBTM as described in the original article. If that’s something you need, or would like to try, you can probably get close to it if you are prepared to spend some time transforming XML documents so that they can feed into the tools provided at satisfice.com.

What it will give you though, and the part that I find of great value right now, is a nice framework for taking notes on your testing, and managing your personal testing process. The tags that it provides, and for which it generates reports, help to remind me of the things that I should be paying attention to. Again, it’s not 100% complete yet, but it’s a good start, and you may find it helps you take better notes when you’re testing.

You may not like the way it does something. That’s OK. The project has an aim of providing open interfaces which will let you modify, transform and hack. If you don’t like its output, you can probably find a way to fix it. Or better yet, you can go straight to the source and tell us how it should be!

The project is looking for another developer to help out right now, so if you’d like to participate in a community project, get in touch with me or Jonathan Kohl.

Time has passed, and I thought it was time to update my thoughts on what’s critical to successful software testing (and development). While originally, I started noting important ideas for agile teams, Increasingly, I find most of these apply no matter what environment I’m working in. Check them out on my Haiku page.

This tweet was forwarded to me yesterday:

martinfowler: Manual scripted testing should be a human rights violation

It bothered me on a few levels.

Firstly, the simplicity of phrasing around manual and scripted testing. Secondly, that agile developers might view themselves as the saviour of oppressed testers everywhere, and the perpetuation of the concept of tester as helpless victims. And thirdly, criticism of scripted testing is hardly new, and it’s a bit sad that one of the key proponents of test-driven development and design has taken so long to come to this idea.

Still, his comments make a nice soundbite, and what he states is often true, but until he says *why*, I’m not sure if he knows what he’s talking about. It just seems a perpetuation of simplistic binary models of automation and prescriptive testing.

We can do better. There’s much more we should be talking about when discussing manual testing and the conditions of testing in corporate life.

I start with these ideas -

- Let’s put under computer control and evaluation the elements of our tests which are well suited to manipulation and/or evaluation by computers.
- Let’s put under human control and judgement the elements of our tests which are well suited to human manipulation and evaluation.
- Let’s prefer exploratory approaches when we want to find new things and adapt to the information at hand.
- Let’s make sure that we record the details for important things that we don’t want to forget to do, for things that are confusing that we can’t fix.
- Let’s be aware that any test have elements of exploration and prescription.
- Let’s be aware that the testing model we work from biases the kinds of problems we’ll find and the kinds of things we’ll miss.
- Let’s choose appropriate levels of abstraction to help manage the cost of our test design.
- Let’s have as many ways as possible to model our test designs so that we can effectively solve the problem at hand.

In particular, the first two points suggest different ways of doing things that blur the lines between traditional automated and “manual” tests, as discussed by Kaner, Bach and Kohl:

http://www.kohl.ca/blog/archives/000193.html
http://www.satisfice.com/articles/boost.shtml
http://www.satisfice.com/blog/archives/118

On the dehumanising of work, a lot of testers pulled out of frontline business roles were probably already having their human rights violated for far less money than they now make as testers. That doesn’t excuse things, but there aren’t as many testers complaining about the status quo as one might expect. My observation is that the nature of testing as practiced in most places drives the majority of those who might want to take a more creative approach elsewhere.

Standard business practice continues to be to throw money at staff retention problems in skilled roles rather than consider how work might be made to suck less. This isn’t uniquely a software testing problem, and the world is some way from tackling this as an issue.

There have been a number of threads I have followed in a few different forums recently where people have discussed requirements, what it means for requirements to be ‘good’, and what it might mean for requirements to be unambiguous. What usually follows is a long-winded back and forth, with no resolution.

At the heart of this deja vu is the fact that one person’s requirements are not necessarily another’s. I resolve this by drawing a clear distinction between specifications and requirements. The distinction may be obvious to some, but in practice it seems to be something we struggle with.

In “Software for dependable systems: Sufficient evidence?”, Daniel Jackson and his co-authors take care to highlight this issue:

“Software systems that are developed specially for a particular client are typically built to meet preagreed requirements, but these requirements are often a long and undifferentiated list of detailed functions.”

“The requirements of a software system should describe the intended effect of the system on its environment and not, more narrowly, the behavior of the system at its interface with the environment, which is the subject of the specification.”

They also take care to point out that the environment of the system includes the software product, plus the humans that use them, and other environmental factors external to the software.

The specifications are not about things that anybody *needs*. Specifications represent the end result of negotiations, conversations, politics and expediency that some group of people thinks represents an understanding that is good enough for now. The specification is at best, our best guess of what’s going to make the world a better place for the numerous people who have a stake in the thing that we’re building. Specifications are a waypoint on the path to something else.

Specifications are never equivalent to requirements in the case of things that will be used by humans.

Specifications apply to the pointy end of a screwdriver that needs to fit into the indented part of some screw.

As testers, testing to specifications is something we do because finding about the requirements is too hard. We do it because that’s what the testers before us did. We do it because the process might be built around specifications documents, and that’s what managers are tracking to. We might reasonably test to spec if our job is just to test a software component that’s on its way to be integrated with something else. However, testing to spec can’t tell us that the system is going to yield the desired benefits.

In the case of the screwdriver, requirements apply to the handle – how it fits your hand, and the hands of others. They apply to whether it gives you enough grip and whether it has a switch to make it only screw or unscrew. They apply to whether it fits enough hands in the world, and whether it can be built to a price that someone is willing to pay. Requirements apply to whether the pointy end of the screwdriver will make do for unscrewing (or screwing in) a screw that is not of the size covered by the specification for the pointy end of our screwdriver.

Requirements are about utility and real human problems, and are fuzzy, and messy, and never fully understood. When our stakeholders ask us questions about testing, though they often don’t phrase it this way, they are usually interested in information with respect to requirements, both explicit and implicit.

Test to see how the product measures up to requirements, and to learn about what the requirements are. That’s the value that you can bring to the project.

I’ve had a few common rants on most of the agile projects I have worked on. Developers bogged down in the detail of stories, while the critical goals of the system wound up ignored, or realising at the last minute that all of the stories built would do nothing useful.

The ideas I came to as a result of the problems I observed were -

- Compensating by starting with extremely high-level stories that defined the critical user and system goals, then progressively breaking these down into tasks.
- Ensuring that the intent and goal is clear from the story title
- Compensating by coaching the testers to ask the critical questions “Why is this feature being added? What problem does it solve and what value does it add?”. The testers would try to ensure that the context was made clear in the story. Links to the higher-level stories in Jira (the tool I most commonly encounter) also helps to provide context.
- Writing high-level acceptance criteria at the top level of stories to help define alternate paths for each goal. These often provided clear boundaries with which stories could be broken down into sub-tasks or sub-stories.
- Evaluating frequently against the high level goals.

I’ve only had a few chances to really try this out, but from a recent project experience with a little more involvement at the early stages of the project, it seemed I was on the right track.
At some point, being aware of Alistair Cockburn’s work, and being a regular reader of his blog, I realised Alistair had probably written about most of this, and I picked up one of his books on use cases. I expected that he’s figured most of this out and more.

Then before I got around to reading the book, it was confirmed for me with this blog post, which I highly recommend taking a look at. Even if you’re not going to apply use cases, it’s worth asking his questions of your project and seeing what you can learn from them.

If you are using use cases, are you avoiding the problems described there?

For the first time, I’m helping run planning sessions for an agile project. Planning has been a bit of a bugbear for me on many of my recent projects, so I’m excited to have a chance to try some things out. So far, it seems to be going well. It’s a short project, so I shouldn’t have to wait too long for feedback on how things went, which is a bonus.

One of the things I noticed is that when in planning or requirements sessions, there’s a tendency to fall into an interview format. That is, we (the development team) ask questions, write things down, ask clarifying questions, and repeat until we think we have enough information to move forward. I think that can be fine sometimes, but it feels like one of the simplest things (but not necessarily the quickest) we can do to help our understanding is to ask our customers to show us how they work. That is, the interview style becomes ‘fly on the wall’ rather than ‘question time’.

As a tester in more traditional projects, I’ve also had occasions where despite the documentation around the project, deep understanding of the domain was difficult to come by, and generally absent from the test team.

On a recent project I had some success combating this by just having the gumption to ask a passing front-line operator “Can I sit with you and watch you work, or do you have time to walk me through the systems you use?”

“Sure! Your desk or mine?” was the answer.

It didn’t hurt me to ask (as I could fit the time in around my other responsibilities), and he was happy to know that some of his concerns might be represented in future. But I know that I don’t always think to do this, or feel that my request will be considered. I learned a lesson though.

So on my current project, it’s time for us to ask the same question of our client. We’re talking through the problem, but the moment the development team is on its own, we keep hitting decisions that can’t be made because our thinking is functional thinking. That is, the client is suggesting features, but we’re not digging deeper to understand the precise intent of those features requests, or the problem that those features will solve. This understanding is critical not just for our testing, but for the whole team to be successful.

Now, we don’t have to work this way. We could just start building the product and let unsatisfying features fall out of the process with our on-site customer, but I believe the amount of up-front thinking and analysis we do is an uncertainty reduction activity. We almost guarantee ourselves a slightly longer project, but I think we reduce the likelihood of having nothing useful to show to the customer at the end of our budgeted time. Critically, I think we have to inform our customers of the tradeoffs we are making around up-front analysis, and how that will impact them.

Excuse me now, I have to go ask someone a question!

‘Capturing the essence’, or ‘core’, has been a key theme in some of my work recently, and in several of the books I’ve been reading. So over a drink with Michael Ruschena tonight a couple of these came out as we linked ‘the core’, ‘agile’, and haiku – poetry that captures the essence. I’ve been coaching a team around testing on agile project, so I thought it might be fun to try to capture the essence of that…

I thus present agile testing haiku.

Note: I’m just going to keep adding new ones to this post so that they’re all in the one spot. And increasingly I’m finding these are not just about agile development, but software development generally.
————————————————–

Method it is not
Apply values, principles
To meet your own needs

Planning game is key
How do we know when we’re done?
Check the product goals.

Can’t make it all fit?
You’ll need to take something out
or get more money.

This user story
Does not express your intent.
Why, why, why, why, why?

Push the risk up front
Shorten the feedback cycles
Highest value first

Development, test
Are all part of the same team.
Try working closely.

Good acceptance tests
Set goals but skip one main thing -
Implementation.

We got a green bar,
Why is the customer sad?
Just add a story.

I didn’t do it.
It wasn’t in the story.
Look, new offshored jobs!

Exploratory tests
Help find problems most quickly
but make extra work.

Skilled software testers
Use many different models
Try many angles

Though up-front tests help
We can get stuck in a rut
Go and have a beer

Embrace fuzziness
Precise is not accurate
Band your estimates

Some we can know now,
but some we must build to learn.
Best tell the owner.

Start with a small team
When the hole size becomes known
let everyone dig

Unit, story tests
Why is our build getting slow?
More techniques will help.

How are we going?
If you can’t tell at a glance
have visible charts

Rough iterations?
Preserving uncertainty
may help things improve

Estimate fully
But to manage greatest risk
Take away two thirds

The test strategy…
What information and how,
tied to the vision

Left values or right?
What’s the pairing’s principle?
You’ll need skill at both.

The grind of agile
Sprinting then taking a break
helps you sustain pace

What is knowable?
Predicting or reacting?
Find your middle ground

The requirements.
Stories are only a third.
What about the rest?

Who’s your customer?
You have no ******* idea?
Find out quick or stop.