A few weeks ago, Designer commented on Software testing, art and productivity. The question got lost in amongst the comment spam, so I thought I’d give my answer a bit more prominently than usual. The question was:

…Many people who want to get a web-developed project don’t even understand the details of work. They just want to have a result and not to make a lot of efforts. How do you think – is there any solution? I think it is wide-spread problem.

If you’re talking about smaller web-based projects, I think you’re right. It can be really difficult to engage clients in the necessary up-front work to help reduce project uncertainty to a reasonable level. It’s a problem given that those with less money to spend have much more business risk in any project they undertake.

I think the second aspect of this is that customers (both internal and external) often come to us with a solution, not a problem. As we build the solution they asked for, the problem becomes clearer and dissatisfaction starts to creep in.

The focus of my work is increasingly on trying to help people build the right thing in the first place. I’m lucky. In my current role, my employer has the conviction that it’s important to make sure the project is heading in the right direction. This means making sure the project team has a shared understanding of the product vision, stakholders, those stakeholders’ goals, priorities and (in the case of a consumer product) the market and opportunities. They also think that it’s OK to bring development to a pause while we get our project bearings.

If you can’t choose the projects you undertake, then I don’t see any easy answers to these problems. And if you can’t convince your customer to be involved appropriately, and to place *some* value on just talking about the problem, it’s a hard road ahead for everyone. I guess the desire to do things ‘right’ is what drives many of us to start our own companies and projects. Without this option though, we can still focus on providing service – helping our customers better understand their problems and pointing out the benefits, costs and risks present in their solution(s). But choose your moments well, and don’t stop dreaming that things can be better than they are.

In the Yahoo software-testing list, Shrini Kulkarni stated

“…productvity as a term is “bad” for creative work like “testing” or “art”. That makes me to feel that I am a low skilled labour on a manufacturing assembly line (not that – it is a bad profession but that does not represent the kind of job I do or I would to do or I am capable of doing)”

It triggered some thoughts from recent projects, and my reply follows.

What bad things happen when we apply the word ‘productivity’ to testing?

Like it or not, unless we are on a fixed price contract with no deadline (and when is there *not* a deadline?), the person who pays is going to be evaluating the value that we provide relative to the money they spend. This seems to be a productivity measurement, and things don’t usually go well when we ignore it.

We are often forced to balance the artistic/creative part of our work with giving our employer the impression that we are being ‘productive’. My understanding of history is that most artists through history have also had to concern themselves with both productivity and art, especially if they want to make a living.

To me, productivity is the key to defensible testing. It’s the idea that I should be striving to get the best coverage that I can for the least cost.

I think where we go wrong is when we define productivity as output or effort, not ‘value’. Unfortunately, the former definitions are common, and I think this is a huge problem in many places that I work. That is, there is a focus on appearing busy and creating artifacts, and not on gathering the information that would most help the project right now.

There are negatives to the project in terms of our personal productivity if we are forced to ignore those parts of our work which give us the greatest satisfaction. For me, this does include the aspects of testing that I consider creative and artistic, but it also includes other things, such as the satisfaction I get from face-to-face communication, developing relationships, working ethically, collaborating and helping the development of others.

It’s not all about art, but we all have days when our inner artist is challenged. Some days slamming the lid down on the piano and storming off the stage is what’s called for. But mostly, we respect the audience…and the show must go on.

During coffee with Agile-coach and all-round excellent guy Shane Clauson, in sympathy with yet another of my what’s-wrong-with-agile rants, he pointed me to this blog post from Jeff Patton:

Don’t know what I want, but I know how to get it

While my opinions diverge on some of what he says must be true, I think the important message that he (and others – Check Alistair Cockburn’s writing on this) make is that it pays to plan to iterate. That is, if you’re on an agile project and you don’t see anyone planning to rework things in response to feedback from using the product, you’re probably in for some disappointment.

I think we frequently fail to give our customers an appropriate expectation when it comes to (agile) software development. Having them read this isn’t a bad start, but you’ll want to figure out how to make this message your own.

Today I was trolling through my Yahoo mails, and noticed that a spam message had made it through to my inbox. I read through my mail, expecting to just delete the spam message when I eventually got to it. When I did, it struck me as unusual. In addition to the message body, there was the usual link. But it looked like this:

http://www.google.com/search?hl=en&q=inurl:ma…….ls&btnI=I=Im+Feeling+Lucky

I was surprised. It was a link to a Google search. I hovered over the link, and it wasn’t just a clever ploy to make me think I was going to Google. It was actually a spammer exploiting Google’s functionality to redirect someone unwittingly to the spam site.

If you want to see this in action (the link above won’t work), you can try this:

http://www.google.com/search?hl=en&q=inurl:quinert.com&btnI=I=Im+Feeling+Lucky

Now, you can probably filter this just by looking for the “Im+Feeling+Lucky” keyword, as the odds of anyone sending you a legitimate URL containing this are highly unlikely. But it does point to an interesting kind of exploit that we might need to be aware of when we’re developing websites. More generally, when we are testing a web-based product, when we think about security risks, we also need to think about how our website might help spammers.

The most common function of this type that leaps to mind are ‘Email to a friend’ type functions, where using tools like WebScarab, or hand-crafting your own html page, websites can be made to send anonymous or faked emails to anyone you like.

What others can you think of?

I work with James, and he emailed this quote around the office -

“UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things.” (attributed to Doug Gwyn)

I had forgotten about my response until I re-subscribed to his blog just now.

“Unix needs an undo key”

Alan Page of Microsoft suggests that there is a perfect world of passed and failed tests, and shades of grey that help us provide more useful information. He then asks “What else do you report as test results (to supplement test case pass/fail counts)? What do those results mean?” Read more at:
http://blogs.msdn.com/alanpa/archive/2007/11/07/pass-fail-and-other.aspx

I think Alan intended to say *automated* tests either pass or fail, but I’ll wait for him to clarify. As a human, I’m easily capable of saying ‘I don’t know whether we should be happy with what I’m observing or not’. Or, ‘The test case passed, but I observed other problems that don’t appear to be related to this test case’. Or, ‘I got 9 of the 10 useful pieces of information from that test case, and the 10th piece of information isn’t really that important’. More importantly, the binary ‘pass/fail’ state can also distract from the idea that the tests exist simply to reveal information. That a ‘failed’ test violated my expectation may or not be a problem.

So my preference is to shy away from quantitative metrics, and look to have conversations with stakeholders instead. If I can’t have those conversations, then I’ll broadcast the qualitative information via test reports.

To state it another way, I don’t think that management actually care about what percentage of test cases passed, except that someone somewhere along the way made them think it was a useful proxy metric for something else. I think they want to know ‘Can we feel comfortable shipping?’ or the equivalent for your environment. I think reducing the answer to that question to ‘100 percent of the test cases passed’ really doesn’t help.

Having said that, if the metrics can be collected cheaply enough, and not providing them is going to upset the process police, I’ll happily report them along with some other metrics. But I’ll be sure to supplement them with a story about how our testing has gone – Trends, events over time, threats to the product and how confident I feel in the test effort.

‘Capturing the essence’, or ‘core’, has been a key theme in some of my work recently, and in several of the books I’ve been reading. So over a drink with Michael Ruschena tonight a couple of these came out as we linked ‘the core’, ‘agile’, and haiku – poetry that captures the essence. I’ve been coaching a team around testing on agile project, so I thought it might be fun to try to capture the essence of that…

I thus present agile testing haiku.

Note: I’m just going to keep adding new ones to this post so that they’re all in the one spot. And increasingly I’m finding these are not just about agile development, but software development generally.
————————————————–

Method it is not
Apply values, principles
To meet your own needs

Planning game is key
How do we know when we’re done?
Check the product goals.

Can’t make it all fit?
You’ll need to take something out
or get more money.

This user story
Does not express your intent.
Why, why, why, why, why?

Push the risk up front
Shorten the feedback cycles
Highest value first

Development, test
Are all part of the same team.
Try working closely.

Good acceptance tests
Set goals but skip one main thing -
Implementation.

We got a green bar,
Why is the customer sad?
Just add a story.

I didn’t do it.
It wasn’t in the story.
Look, new offshored jobs!

Exploratory tests
Help find problems most quickly
but make extra work.

Skilled software testers
Use many different models
Try many angles

Though up-front tests help
We can get stuck in a rut
Go and have a beer

Embrace fuzziness
Precise is not accurate
Band your estimates

Some we can know now,
but some we must build to learn.
Best tell the owner.

Start with a small team
When the hole size becomes known
let everyone dig

Unit, story tests
Why is our build getting slow?
More techniques will help.

How are we going?
If you can’t tell at a glance
have visible charts

Rough iterations?
Preserving uncertainty
may help things improve

Estimate fully
But to manage greatest risk
Take away two thirds

The test strategy…
What information and how,
tied to the vision

Left values or right?
What’s the pairing’s principle?
You’ll need skill at both.

The grind of agile
Sprinting then taking a break
helps you sustain pace

What is knowable?
Predicting or reacting?
Find your middle ground

The requirements.
Stories are only a third.
What about the rest?

Who’s your customer?
You have no ******* idea?
Find out quick or stop.

You would think that this kind of message would be a thing of the past, but apparently not. Or at least not when buying tickets to major exhibitions in Melbourne.

Having worked on the trial project mentioned in Michael Ruschena’s recent blog entry on Project Manifestos, I can say it’s well worth trying out. And when I say ‘trying out’, I mean having the conversations and going through the thought processes required to figure out how you might work together as a team. I find my recent test strategy documents sharing much of the spirit of these documents, being about figuring out how we are going to give ourselves a shot at achieving our aims.

Thanks Michael!