Presstimate:

The number you give to get a manager off your back when you’re being hassled to give an estimate; Your best guess of what estimate of effort management will accept, not how long the work will actually take.

Time has passed, and I thought it was time to update my thoughts on what’s critical to successful software testing (and development). While originally, I started noting important ideas for agile teams, Increasingly, I find most of these apply no matter what environment I’m working in. Check them out on my Haiku page.

I spent ages recently googling for references to ‘Diabolical Problems’, about which I thought I had read. Thanks to Matt Heusser’s latest post, I now know I should have instead been googling for Wicked Problems. Just in case I’ve sent anyone off on the same wild goose chase, this post should set them straight. For everyone else, enjoy the reading.

There have been a number of threads I have followed in a few different forums recently where people have discussed requirements, what it means for requirements to be ‘good’, and what it might mean for requirements to be unambiguous. What usually follows is a long-winded back and forth, with no resolution.

At the heart of this deja vu is the fact that one person’s requirements are not necessarily another’s. I resolve this by drawing a clear distinction between specifications and requirements. The distinction may be obvious to some, but in practice it seems to be something we struggle with.

In “Software for dependable systems: Sufficient evidence?”, Daniel Jackson and his co-authors take care to highlight this issue:

“Software systems that are developed specially for a particular client are typically built to meet preagreed requirements, but these requirements are often a long and undifferentiated list of detailed functions.”

“The requirements of a software system should describe the intended effect of the system on its environment and not, more narrowly, the behavior of the system at its interface with the environment, which is the subject of the specification.”

They also take care to point out that the environment of the system includes the software product, plus the humans that use them, and other environmental factors external to the software.

The specifications are not about things that anybody *needs*. Specifications represent the end result of negotiations, conversations, politics and expediency that some group of people thinks represents an understanding that is good enough for now. The specification is at best, our best guess of what’s going to make the world a better place for the numerous people who have a stake in the thing that we’re building. Specifications are a waypoint on the path to something else.

Specifications are never equivalent to requirements in the case of things that will be used by humans.

Specifications apply to the pointy end of a screwdriver that needs to fit into the indented part of some screw.

As testers, testing to specifications is something we do because finding about the requirements is too hard. We do it because that’s what the testers before us did. We do it because the process might be built around specifications documents, and that’s what managers are tracking to. We might reasonably test to spec if our job is just to test a software component that’s on its way to be integrated with something else. However, testing to spec can’t tell us that the system is going to yield the desired benefits.

In the case of the screwdriver, requirements apply to the handle – how it fits your hand, and the hands of others. They apply to whether it gives you enough grip and whether it has a switch to make it only screw or unscrew. They apply to whether it fits enough hands in the world, and whether it can be built to a price that someone is willing to pay. Requirements apply to whether the pointy end of the screwdriver will make do for unscrewing (or screwing in) a screw that is not of the size covered by the specification for the pointy end of our screwdriver.

Requirements are about utility and real human problems, and are fuzzy, and messy, and never fully understood. When our stakeholders ask us questions about testing, though they often don’t phrase it this way, they are usually interested in information with respect to requirements, both explicit and implicit.

Test to see how the product measures up to requirements, and to learn about what the requirements are. That’s the value that you can bring to the project.

An excellent quote from our development lead, James Ladd:

How to test a project might be risky -

- It has people in it.

Thanks to James Bach for inspring this!

A few weeks ago, Designer commented on Software testing, art and productivity. The question got lost in amongst the comment spam, so I thought I’d give my answer a bit more prominently than usual. The question was:

…Many people who want to get a web-developed project don’t even understand the details of work. They just want to have a result and not to make a lot of efforts. How do you think – is there any solution? I think it is wide-spread problem.

If you’re talking about smaller web-based projects, I think you’re right. It can be really difficult to engage clients in the necessary up-front work to help reduce project uncertainty to a reasonable level. It’s a problem given that those with less money to spend have much more business risk in any project they undertake.

I think the second aspect of this is that customers (both internal and external) often come to us with a solution, not a problem. As we build the solution they asked for, the problem becomes clearer and dissatisfaction starts to creep in.

The focus of my work is increasingly on trying to help people build the right thing in the first place. I’m lucky. In my current role, my employer has the conviction that it’s important to make sure the project is heading in the right direction. This means making sure the project team has a shared understanding of the product vision, stakholders, those stakeholders’ goals, priorities and (in the case of a consumer product) the market and opportunities. They also think that it’s OK to bring development to a pause while we get our project bearings.

If you can’t choose the projects you undertake, then I don’t see any easy answers to these problems. And if you can’t convince your customer to be involved appropriately, and to place *some* value on just talking about the problem, it’s a hard road ahead for everyone. I guess the desire to do things ‘right’ is what drives many of us to start our own companies and projects. Without this option though, we can still focus on providing service – helping our customers better understand their problems and pointing out the benefits, costs and risks present in their solution(s). But choose your moments well, and don’t stop dreaming that things can be better than they are.

Mike Cohn’s “User Stories Applied” discusses using the INVEST mnemonic as a guide to writing better user stories. I was recently asked to dig up a reference for it, and found this presentation here, with the section on the mnemonic on pages 47 and 48.

As I read it, I noticed that there’s been a change to one of the letters. Whereas the book uses ‘S’ to denote ‘Small’, now it’s become ‘Sized appropriately’.

I think this is a change for the better, as I noticed that every time I talked someone through the acronym, I would have a long-winded conversation qualifying ‘Small’ as ‘Just small enough but no smaller’. This would come about as I tried to explain the tradeoffs between a story being small enough to estimate with some reasonable certainty, small enough to fit within an iteration, and still ensuring that stories are ‘V- Valuable to the customer’, needing to ensure that user stories continue to express clearly a problem or need of a person.

Overly small stories push us further from the original context of the problem, and thus force us to compensate with an increasing heirarchy of ’super-stories’ to help us focus on the bigger picture. These become more noticable when working in shorter iterations than might be common on a Scrum project, so three cheers for Mike in spending the effort to come up with ‘Sized appropriately’.

Jonathan Kohl pointed me at a position paper from Brian Marick for the Agile Coach Camp.

If you’re in the middle of adopting agile, it’s well worth a read. You can find it here:
http://wiki.agilecoachcamp.org/
tiki-index.php?page=BrianMarickPositionPaper

During coffee with Agile-coach and all-round excellent guy Shane Clauson, in sympathy with yet another of my what’s-wrong-with-agile rants, he pointed me to this blog post from Jeff Patton:

Don’t know what I want, but I know how to get it

While my opinions diverge on some of what he says must be true, I think the important message that he (and others – Check Alistair Cockburn’s writing on this) make is that it pays to plan to iterate. That is, if you’re on an agile project and you don’t see anyone planning to rework things in response to feedback from using the product, you’re probably in for some disappointment.

I think we frequently fail to give our customers an appropriate expectation when it comes to (agile) software development. Having them read this isn’t a bad start, but you’ll want to figure out how to make this message your own.

I’ve had a few common rants on most of the agile projects I have worked on. Developers bogged down in the detail of stories, while the critical goals of the system wound up ignored, or realising at the last minute that all of the stories built would do nothing useful.

The ideas I came to as a result of the problems I observed were -

- Compensating by starting with extremely high-level stories that defined the critical user and system goals, then progressively breaking these down into tasks.
- Ensuring that the intent and goal is clear from the story title
- Compensating by coaching the testers to ask the critical questions “Why is this feature being added? What problem does it solve and what value does it add?”. The testers would try to ensure that the context was made clear in the story. Links to the higher-level stories in Jira (the tool I most commonly encounter) also helps to provide context.
- Writing high-level acceptance criteria at the top level of stories to help define alternate paths for each goal. These often provided clear boundaries with which stories could be broken down into sub-tasks or sub-stories.
- Evaluating frequently against the high level goals.

I’ve only had a few chances to really try this out, but from a recent project experience with a little more involvement at the early stages of the project, it seemed I was on the right track.
At some point, being aware of Alistair Cockburn’s work, and being a regular reader of his blog, I realised Alistair had probably written about most of this, and I picked up one of his books on use cases. I expected that he’s figured most of this out and more.

Then before I got around to reading the book, it was confirmed for me with this blog post, which I highly recommend taking a look at. Even if you’re not going to apply use cases, it’s worth asking his questions of your project and seeing what you can learn from them.

If you are using use cases, are you avoiding the problems described there?