There have been a number of threads I have followed in a few different forums recently where people have discussed requirements, what it means for requirements to be ‘good’, and what it might mean for requirements to be unambiguous. What usually follows is a long-winded back and forth, with no resolution.

At the heart of this deja vu is the fact that one person’s requirements are not necessarily another’s. I resolve this by drawing a clear distinction between specifications and requirements. The distinction may be obvious to some, but in practice it seems to be something we struggle with.

In “Software for dependable systems: Sufficient evidence?”, Daniel Jackson and his co-authors take care to highlight this issue:

“Software systems that are developed specially for a particular client are typically built to meet preagreed requirements, but these requirements are often a long and undifferentiated list of detailed functions.”

“The requirements of a software system should describe the intended effect of the system on its environment and not, more narrowly, the behavior of the system at its interface with the environment, which is the subject of the specification.”

They also take care to point out that the environment of the system includes the software product, plus the humans that use them, and other environmental factors external to the software.

The specifications are not about things that anybody *needs*. Specifications represent the end result of negotiations, conversations, politics and expediency that some group of people thinks represents an understanding that is good enough for now. The specification is at best, our best guess of what’s going to make the world a better place for the numerous people who have a stake in the thing that we’re building. Specifications are a waypoint on the path to something else.

Specifications are never equivalent to requirements in the case of things that will be used by humans.

Specifications apply to the pointy end of a screwdriver that needs to fit into the indented part of some screw.

As testers, testing to specifications is something we do because finding about the requirements is too hard. We do it because that’s what the testers before us did. We do it because the process might be built around specifications documents, and that’s what managers are tracking to. We might reasonably test to spec if our job is just to test a software component that’s on its way to be integrated with something else. However, testing to spec can’t tell us that the system is going to yield the desired benefits.

In the case of the screwdriver, requirements apply to the handle – how it fits your hand, and the hands of others. They apply to whether it gives you enough grip and whether it has a switch to make it only screw or unscrew. They apply to whether it fits enough hands in the world, and whether it can be built to a price that someone is willing to pay. Requirements apply to whether the pointy end of the screwdriver will make do for unscrewing (or screwing in) a screw that is not of the size covered by the specification for the pointy end of our screwdriver.

Requirements are about utility and real human problems, and are fuzzy, and messy, and never fully understood. When our stakeholders ask us questions about testing, though they often don’t phrase it this way, they are usually interested in information with respect to requirements, both explicit and implicit.

Test to see how the product measures up to requirements, and to learn about what the requirements are. That’s the value that you can bring to the project.

In the Yahoo software-testing list, Shrini Kulkarni stated

“…productvity as a term is “bad” for creative work like “testing” or “art”. That makes me to feel that I am a low skilled labour on a manufacturing assembly line (not that – it is a bad profession but that does not represent the kind of job I do or I would to do or I am capable of doing)”

It triggered some thoughts from recent projects, and my reply follows.

What bad things happen when we apply the word ‘productivity’ to testing?

Like it or not, unless we are on a fixed price contract with no deadline (and when is there *not* a deadline?), the person who pays is going to be evaluating the value that we provide relative to the money they spend. This seems to be a productivity measurement, and things don’t usually go well when we ignore it.

We are often forced to balance the artistic/creative part of our work with giving our employer the impression that we are being ‘productive’. My understanding of history is that most artists through history have also had to concern themselves with both productivity and art, especially if they want to make a living.

To me, productivity is the key to defensible testing. It’s the idea that I should be striving to get the best coverage that I can for the least cost.

I think where we go wrong is when we define productivity as output or effort, not ‘value’. Unfortunately, the former definitions are common, and I think this is a huge problem in many places that I work. That is, there is a focus on appearing busy and creating artifacts, and not on gathering the information that would most help the project right now.

There are negatives to the project in terms of our personal productivity if we are forced to ignore those parts of our work which give us the greatest satisfaction. For me, this does include the aspects of testing that I consider creative and artistic, but it also includes other things, such as the satisfaction I get from face-to-face communication, developing relationships, working ethically, collaborating and helping the development of others.

It’s not all about art, but we all have days when our inner artist is challenged. Some days slamming the lid down on the piano and storming off the stage is what’s called for. But mostly, we respect the audience…and the show must go on.

For the first time, I’m helping run planning sessions for an agile project. Planning has been a bit of a bugbear for me on many of my recent projects, so I’m excited to have a chance to try some things out. So far, it seems to be going well. It’s a short project, so I shouldn’t have to wait too long for feedback on how things went, which is a bonus.

One of the things I noticed is that when in planning or requirements sessions, there’s a tendency to fall into an interview format. That is, we (the development team) ask questions, write things down, ask clarifying questions, and repeat until we think we have enough information to move forward. I think that can be fine sometimes, but it feels like one of the simplest things (but not necessarily the quickest) we can do to help our understanding is to ask our customers to show us how they work. That is, the interview style becomes ‘fly on the wall’ rather than ‘question time’.

As a tester in more traditional projects, I’ve also had occasions where despite the documentation around the project, deep understanding of the domain was difficult to come by, and generally absent from the test team.

On a recent project I had some success combating this by just having the gumption to ask a passing front-line operator “Can I sit with you and watch you work, or do you have time to walk me through the systems you use?”

“Sure! Your desk or mine?” was the answer.

It didn’t hurt me to ask (as I could fit the time in around my other responsibilities), and he was happy to know that some of his concerns might be represented in future. But I know that I don’t always think to do this, or feel that my request will be considered. I learned a lesson though.

So on my current project, it’s time for us to ask the same question of our client. We’re talking through the problem, but the moment the development team is on its own, we keep hitting decisions that can’t be made because our thinking is functional thinking. That is, the client is suggesting features, but we’re not digging deeper to understand the precise intent of those features requests, or the problem that those features will solve. This understanding is critical not just for our testing, but for the whole team to be successful.

Now, we don’t have to work this way. We could just start building the product and let unsatisfying features fall out of the process with our on-site customer, but I believe the amount of up-front thinking and analysis we do is an uncertainty reduction activity. We almost guarantee ourselves a slightly longer project, but I think we reduce the likelihood of having nothing useful to show to the customer at the end of our budgeted time. Critically, I think we have to inform our customers of the tradeoffs we are making around up-front analysis, and how that will impact them.

Excuse me now, I have to go ask someone a question!

Alan Page of Microsoft suggests that there is a perfect world of passed and failed tests, and shades of grey that help us provide more useful information. He then asks “What else do you report as test results (to supplement test case pass/fail counts)? What do those results mean?” Read more at:
http://blogs.msdn.com/alanpa/archive/2007/11/07/pass-fail-and-other.aspx

I think Alan intended to say *automated* tests either pass or fail, but I’ll wait for him to clarify. As a human, I’m easily capable of saying ‘I don’t know whether we should be happy with what I’m observing or not’. Or, ‘The test case passed, but I observed other problems that don’t appear to be related to this test case’. Or, ‘I got 9 of the 10 useful pieces of information from that test case, and the 10th piece of information isn’t really that important’. More importantly, the binary ‘pass/fail’ state can also distract from the idea that the tests exist simply to reveal information. That a ‘failed’ test violated my expectation may or not be a problem.

So my preference is to shy away from quantitative metrics, and look to have conversations with stakeholders instead. If I can’t have those conversations, then I’ll broadcast the qualitative information via test reports.

To state it another way, I don’t think that management actually care about what percentage of test cases passed, except that someone somewhere along the way made them think it was a useful proxy metric for something else. I think they want to know ‘Can we feel comfortable shipping?’ or the equivalent for your environment. I think reducing the answer to that question to ‘100 percent of the test cases passed’ really doesn’t help.

Having said that, if the metrics can be collected cheaply enough, and not providing them is going to upset the process police, I’ll happily report them along with some other metrics. But I’ll be sure to supplement them with a story about how our testing has gone – Trends, events over time, threats to the product and how confident I feel in the test effort.

‘Capturing the essence’, or ‘core’, has been a key theme in some of my work recently, and in several of the books I’ve been reading. So over a drink with Michael Ruschena tonight a couple of these came out as we linked ‘the core’, ‘agile’, and haiku – poetry that captures the essence. I’ve been coaching a team around testing on agile project, so I thought it might be fun to try to capture the essence of that…

I thus present agile testing haiku.

Note: I’m just going to keep adding new ones to this post so that they’re all in the one spot. And increasingly I’m finding these are not just about agile development, but software development generally.
————————————————–

Method it is not
Apply values, principles
To meet your own needs

Planning game is key
How do we know when we’re done?
Check the product goals.

Can’t make it all fit?
You’ll need to take something out
or get more money.

This user story
Does not express your intent.
Why, why, why, why, why?

Push the risk up front
Shorten the feedback cycles
Highest value first

Development, test
Are all part of the same team.
Try working closely.

Good acceptance tests
Set goals but skip one main thing -
Implementation.

We got a green bar,
Why is the customer sad?
Just add a story.

I didn’t do it.
It wasn’t in the story.
Look, new offshored jobs!

Exploratory tests
Help find problems most quickly
but make extra work.

Skilled software testers
Use many different models
Try many angles

Though up-front tests help
We can get stuck in a rut
Go and have a beer

Embrace fuzziness
Precise is not accurate
Band your estimates

Some we can know now,
but some we must build to learn.
Best tell the owner.

Start with a small team
When the hole size becomes known
let everyone dig

Unit, story tests
Why is our build getting slow?
More techniques will help.

How are we going?
If you can’t tell at a glance
have visible charts

Rough iterations?
Preserving uncertainty
may help things improve

Estimate fully
But to manage greatest risk
Take away two thirds

The test strategy…
What information and how,
tied to the vision

Left values or right?
What’s the pairing’s principle?
You’ll need skill at both.

The grind of agile
Sprinting then taking a break
helps you sustain pace

What is knowable?
Predicting or reacting?
Find your middle ground

The requirements.
Stories are only a third.
What about the rest?

Who’s your customer?
You have no ******* idea?
Find out quick or stop.

Today, I’m translating the Korean children’s story ‘puppy poo’ for my blog. It turns out that somebody has already translated it, so it’s not the world-first I thought it was going to be. Oh well. I needed the Korean practice anyway. It’s a touching children’s story, about a piece of poo who finds meaning and acceptance fertilising a dandelion.

You can read along with the story and view an animation here:
http://www.doggypoo.co.kr/. The book and movie adaptation can be found on Amazon.

Puppy Poo, by Kwon Jong-saeng. Translated by Jared Quinert.

One cold winter day, a puppy pooed on the side of the road…

A sparrow flew down and began pecking at the poo.

‘Poo! Poo! Geez, you’re dirty!’ said the sparrow.
The poo’s heart was deeply hurt. “I’m dirty?” he asked, to noone in particular.

A clod of dirt nearby began to laugh.
“Why are you laughing?” the poo asked angrily.

“You’re poo, so it’s true. In fact, among all poo, you’re just about the dirtiest”, said the clod. The poo broke into tears.

Along came a farmer with his cow.
“That pile looks fairly fresh”, said the farmer. “That would be great for my vegetable patch”. The poo’s heart rose!

The farmer scooped up the hurtful clod of dirt, leaving the poo alone as the air turned cold. “Seeya, filthy!” shouted the clod as they left.

It became dark, and started to snow.

The snow buried the poo, and he fell into a deep sleep for the winter.
The warm spring came, and the poo woke from his hibernation.

In front of him was a dandelion shoot.
“Who are you?” asked the poo.

“I’m a dandelion. I’m going to bloom into a beautiful star-like flower”
“How are you going to manage that?” asked the poo.
“Well, as long as I get some mulch or compost, I can bloom” whispered the dandelion, with a twinkle in his eye.

“Really?” said the poo, surprised. “I can be mulch or compost!”

The puppy poo happily embraced the dandelion shoot.

Suddenly, the Spring rain came, washing over the poo’s body, breaking him down into fine pieces, entering the earth and fertilising the dandelion.

The day broke, and dazzling sunbeams shone over the dandelion’s bright flower, which had now bloomed. The flower’s scent flew on the Spring breeze.

The poo’s gentle heart had filled the dandelion’s blossom.

———————

Now, the moral of the story is supposed to be about everyone having a purpose. I’m going to offer an alternate moral, based on some recent consulting engagements:

Sometimes, in order to grow, we need to associate with turds. Or be buried up to our neck in poo.

Now, I’m not really sure if that’s one moral or two, but it does describe my reason for trying consultancy in the first place. That is, I needed to be outside my comfort zone in order to grow professionally. There have been lessons, and I’ve enjoyed my year or so of consulting. I’m looking forward to my new role’s balance of internal project work and external consulting, which should be the best of both worlds!

I highly recommend that everyone spend a little time figuring out what fertiliser might be best for their nourishment.

Good luck!

Having worked on the trial project mentioned in Michael Ruschena’s recent blog entry on Project Manifestos, I can say it’s well worth trying out. And when I say ‘trying out’, I mean having the conversations and going through the thought processes required to figure out how you might work together as a team. I find my recent test strategy documents sharing much of the spirit of these documents, being about figuring out how we are going to give ourselves a shot at achieving our aims.

Thanks Michael!

I’ve been sadly busy, finishing a cool project with much learning, and preparing to leave my current employer Revolution IT for Aegeon. This means I get to have another crack at building an army of testing ninjas and sending them out into the world to hopefully make it a better place. Hopefully, that means less blogwebs (the digital form of cobwebs) too.

Ben Kelly, testing martial artist (literally, in this case) and recent blogger threw a question my way which I’ve answered before, but hadn’t really ever had to answer for testers in my team. It went something like this:

Ben: If quality can be described as ‘value to some person (who matters) and testing can be paraphrased as ‘questioning a product in order to evaluate it’, would you describe quality assurance as ‘interpreting the answers to said questions in order to provide information about the quality of said product to someone (who matters)’ ?

I rambled:
Quality assurance should be about ‘Is our process appropriate?’ That is, is it providing the right value to the people that matter.

Ben: Hmm, okay. Quality control is probably more appropriate then.

I rambled some more: I see your point though…One of the ways we assess the success of our development approach is by looking at testing. It’s just not the only way. It would more likely start with questions like:

- ‘What questions didn’t we ask’? That is, what problems escaped into the wild?
- ‘What didn’t we do?’
- ‘What are we doing that is a waste of time, or not helpful?’

Ben: I’m trying to condense the definitions of these terms so that they’re not overly verbose and still retain their meaning – I’m still working on that presentation

Me: So, Quality Assurance is typically assessing the whole software-producing system. Quality control is checking the output of the production process. Quality assurance is checking the process itself. That’s my model, at least.

Ben: That’s inline with what I’m working to. Maybe I don’t need to go into the definitions of these with the new guys right away. Maybe I should let them get their head around the definition of what testing is before I say ‘By the way, testing is not quite the same animal as QA or QC’

Me: It becomes relevant when test groups are called QA and think somehow that they actually do assure quality.
If your team is under that delusion, maybe it’s worth spending some time dispelling it. Testers can provide information about quality. They can’t actually force anyone to do anything with that information.

I want my testers to think:
“I am not assuring quality.”
“I cannot remove bugs.”
“I cannot stop developers creating bugs.”

This is obviously not always true, if we consider our ability to contribute to removal of requirements defects, but that’s not always an area we’re able to contribute to. If we’re not the prime owner of the requirements process, our role is still mostly about pointing out problems and making a compelling case for those problems to be addressed.

It’s potentially problematic for non-testers to think that they have this power though, and it’ s generally bad for the team if testers believe they can do it. Primarily, our role is to provide information to allow others to make good decisions.

Ben: Hmm, I might put it in to my presentation after all.

Me: Yeah, it probably belongs on a ‘What is testing’ slide. It stops your new testers beating themselves up for things that they shouldn’t. It lets your experienced testers defend their practice and not be beaten up by others. Or rather, starts your testers on the path to understanding the scope of their role and communicating that scope to others, setting clearer expectations of what to expect from a tester.

Ben: Something we could stand to do more of here. Excellent. Thanks as always for letting me bother you.

And thanks Ben for bothering me with something that I didn’t realise I needed to. Agile projects change the game a little, but for software, the primary focus of QA is on getting better at producing something that satisfies our customers, through code. Taking on the QA mantle is about telling managers, developers and analysts how to do their job better. And my super powers don’t yet extend that far. I encourage my testers to know their limits too.

Matt Heusser has continued discussion of tester-developer relations, where Jonathan Kohl describes the flipside of testers telling developer’s that their automation code sucks.

Actually there are many scenarios, derived from at least a few properties -

- Does the person doing the automation know/not know that their code sucks.
- Is the code ’suckage’ pointed out by a third party, or the person writing it.
- Is the automation being done by a tester or a developer?

You can plot these out as a truth table and see where you fit. I tend to be automating, saying “I would love some help with this”, but can’t get help or chances to pair with developers. I’ve also observed testers wanting to pair with developers, but developers not wanting to come down to the tester’s skill level or work with their tools (as well as the case I mentioned in my previous post).

I’m sure situations also exist where the testers aren’t interested in working with the developers and learning.

So we have a model that contains -

- the feedback
- the source of the feedback
- awareness of the current state
- receptiveness to feedback
- desire or ability of one of the parties to change the situation.

Has anyone experienced any of the other permutations of these properties?

Matt Heusser’s recent blog entry on Tester/Developer communications prompted me to comment on the dreams of agile projects and tester heaven.

Now, I’ve been tempted to have that conversation, but the conversations I have had instead are much more along these lines:

Me: “Hey dev guy, here’s 20 years of accumulated knowledge describing why the testing approach you are about to take is not going to solve your problem, cost a lot of money, and waste a lot of time”.
Developer: “That’s nice, but we’re going to do it anyway”.

Two months later:

Me: “So, how’d that test approach work for you?”
Developer: “Oh, it completely sucked and was unmaintainable”.

Now, these developers are not stupid people, but it’s difficult for me to understand how things get to this point.

Or at least, it was until recently. Most agile development projects are much like Toyotism (or Lean, if you prefer). Better than how things were in the past, but not fundamentally changing the nature of work. Just as work in manufacturing is still hard, repetitive and largely about lines, so we easily fall into our old factory-style patterns. I believe this is at the heart of Brian Marick’s statement on the current state of Agile project experiences: ‘Whereas the number of people new to Agile who describe their project as “the best project I’ve ever worked on” seems to be declining, and we believe work should be joyful.’

Parallel lines of software development, with variable development and testing tasks performed serially end up being just as unpleasant as the old ways. Except now the moment your testing task takes longer than the development task in the queue behind you, someone’s screaming about the tester backlog.

Of course, it doesn’t have to be this way, but there’s very little incentive for the guy with the upstream work to contribute to solving it. They’re not the ones feeling the pressure. And once those pesky testers are gone, the whole thing actually appears to work much more smoothly. After all, what’s smoother than an assembly line with a single step?

Hint: The last line *is* the answer. But the devil is in the details of what comprises a ’step’. And what Matt describes is a step toward the solution as well.

Stay tuned.