Presstimate:

The number you give to get a manager off your back when you’re being hassled to give an estimate; Your best guess of what estimate of effort management will accept, not how long the work will actually take.

“In the universe, nothing can be said to be automatic, as nothing can be said to be without design. An imperfect parallel may be found in human inventions; springs may move springs, and wheels, indexes; but the motion and the regulation must be derived from the artist;”

From Elements of Chemical Philosophy Part 1, Vol.1 By Humphry Davy

Of course, the second step will be taking my automation or testing tools course (plug, plug).

A few weeks ago, Designer commented on Software testing, art and productivity. The question got lost in amongst the comment spam, so I thought I’d give my answer a bit more prominently than usual. The question was:

…Many people who want to get a web-developed project don’t even understand the details of work. They just want to have a result and not to make a lot of efforts. How do you think – is there any solution? I think it is wide-spread problem.

If you’re talking about smaller web-based projects, I think you’re right. It can be really difficult to engage clients in the necessary up-front work to help reduce project uncertainty to a reasonable level. It’s a problem given that those with less money to spend have much more business risk in any project they undertake.

I think the second aspect of this is that customers (both internal and external) often come to us with a solution, not a problem. As we build the solution they asked for, the problem becomes clearer and dissatisfaction starts to creep in.

The focus of my work is increasingly on trying to help people build the right thing in the first place. I’m lucky. In my current role, my employer has the conviction that it’s important to make sure the project is heading in the right direction. This means making sure the project team has a shared understanding of the product vision, stakholders, those stakeholders’ goals, priorities and (in the case of a consumer product) the market and opportunities. They also think that it’s OK to bring development to a pause while we get our project bearings.

If you can’t choose the projects you undertake, then I don’t see any easy answers to these problems. And if you can’t convince your customer to be involved appropriately, and to place *some* value on just talking about the problem, it’s a hard road ahead for everyone. I guess the desire to do things ‘right’ is what drives many of us to start our own companies and projects. Without this option though, we can still focus on providing service – helping our customers better understand their problems and pointing out the benefits, costs and risks present in their solution(s). But choose your moments well, and don’t stop dreaming that things can be better than they are.

In the Yahoo software-testing list, Shrini Kulkarni stated

“…productvity as a term is “bad” for creative work like “testing” or “art”. That makes me to feel that I am a low skilled labour on a manufacturing assembly line (not that – it is a bad profession but that does not represent the kind of job I do or I would to do or I am capable of doing)”

It triggered some thoughts from recent projects, and my reply follows.

What bad things happen when we apply the word ‘productivity’ to testing?

Like it or not, unless we are on a fixed price contract with no deadline (and when is there *not* a deadline?), the person who pays is going to be evaluating the value that we provide relative to the money they spend. This seems to be a productivity measurement, and things don’t usually go well when we ignore it.

We are often forced to balance the artistic/creative part of our work with giving our employer the impression that we are being ‘productive’. My understanding of history is that most artists through history have also had to concern themselves with both productivity and art, especially if they want to make a living.

To me, productivity is the key to defensible testing. It’s the idea that I should be striving to get the best coverage that I can for the least cost.

I think where we go wrong is when we define productivity as output or effort, not ‘value’. Unfortunately, the former definitions are common, and I think this is a huge problem in many places that I work. That is, there is a focus on appearing busy and creating artifacts, and not on gathering the information that would most help the project right now.

There are negatives to the project in terms of our personal productivity if we are forced to ignore those parts of our work which give us the greatest satisfaction. For me, this does include the aspects of testing that I consider creative and artistic, but it also includes other things, such as the satisfaction I get from face-to-face communication, developing relationships, working ethically, collaborating and helping the development of others.

It’s not all about art, but we all have days when our inner artist is challenged. Some days slamming the lid down on the piano and storming off the stage is what’s called for. But mostly, we respect the audience…and the show must go on.

For the first time, I’m helping run planning sessions for an agile project. Planning has been a bit of a bugbear for me on many of my recent projects, so I’m excited to have a chance to try some things out. So far, it seems to be going well. It’s a short project, so I shouldn’t have to wait too long for feedback on how things went, which is a bonus.

One of the things I noticed is that when in planning or requirements sessions, there’s a tendency to fall into an interview format. That is, we (the development team) ask questions, write things down, ask clarifying questions, and repeat until we think we have enough information to move forward. I think that can be fine sometimes, but it feels like one of the simplest things (but not necessarily the quickest) we can do to help our understanding is to ask our customers to show us how they work. That is, the interview style becomes ‘fly on the wall’ rather than ‘question time’.

As a tester in more traditional projects, I’ve also had occasions where despite the documentation around the project, deep understanding of the domain was difficult to come by, and generally absent from the test team.

On a recent project I had some success combating this by just having the gumption to ask a passing front-line operator “Can I sit with you and watch you work, or do you have time to walk me through the systems you use?”

“Sure! Your desk or mine?” was the answer.

It didn’t hurt me to ask (as I could fit the time in around my other responsibilities), and he was happy to know that some of his concerns might be represented in future. But I know that I don’t always think to do this, or feel that my request will be considered. I learned a lesson though.

So on my current project, it’s time for us to ask the same question of our client. We’re talking through the problem, but the moment the development team is on its own, we keep hitting decisions that can’t be made because our thinking is functional thinking. That is, the client is suggesting features, but we’re not digging deeper to understand the precise intent of those features requests, or the problem that those features will solve. This understanding is critical not just for our testing, but for the whole team to be successful.

Now, we don’t have to work this way. We could just start building the product and let unsatisfying features fall out of the process with our on-site customer, but I believe the amount of up-front thinking and analysis we do is an uncertainty reduction activity. We almost guarantee ourselves a slightly longer project, but I think we reduce the likelihood of having nothing useful to show to the customer at the end of our budgeted time. Critically, I think we have to inform our customers of the tradeoffs we are making around up-front analysis, and how that will impact them.

Excuse me now, I have to go ask someone a question!

Alan Page of Microsoft suggests that there is a perfect world of passed and failed tests, and shades of grey that help us provide more useful information. He then asks “What else do you report as test results (to supplement test case pass/fail counts)? What do those results mean?” Read more at:
http://blogs.msdn.com/alanpa/archive/2007/11/07/pass-fail-and-other.aspx

I think Alan intended to say *automated* tests either pass or fail, but I’ll wait for him to clarify. As a human, I’m easily capable of saying ‘I don’t know whether we should be happy with what I’m observing or not’. Or, ‘The test case passed, but I observed other problems that don’t appear to be related to this test case’. Or, ‘I got 9 of the 10 useful pieces of information from that test case, and the 10th piece of information isn’t really that important’. More importantly, the binary ‘pass/fail’ state can also distract from the idea that the tests exist simply to reveal information. That a ‘failed’ test violated my expectation may or not be a problem.

So my preference is to shy away from quantitative metrics, and look to have conversations with stakeholders instead. If I can’t have those conversations, then I’ll broadcast the qualitative information via test reports.

To state it another way, I don’t think that management actually care about what percentage of test cases passed, except that someone somewhere along the way made them think it was a useful proxy metric for something else. I think they want to know ‘Can we feel comfortable shipping?’ or the equivalent for your environment. I think reducing the answer to that question to ‘100 percent of the test cases passed’ really doesn’t help.

Having said that, if the metrics can be collected cheaply enough, and not providing them is going to upset the process police, I’ll happily report them along with some other metrics. But I’ll be sure to supplement them with a story about how our testing has gone – Trends, events over time, threats to the product and how confident I feel in the test effort.

Having worked on the trial project mentioned in Michael Ruschena’s recent blog entry on Project Manifestos, I can say it’s well worth trying out. And when I say ‘trying out’, I mean having the conversations and going through the thought processes required to figure out how you might work together as a team. I find my recent test strategy documents sharing much of the spirit of these documents, being about figuring out how we are going to give ourselves a shot at achieving our aims.

Thanks Michael!

Matt Heusser’s recent blog entry on Tester/Developer communications prompted me to comment on the dreams of agile projects and tester heaven.

Now, I’ve been tempted to have that conversation, but the conversations I have had instead are much more along these lines:

Me: “Hey dev guy, here’s 20 years of accumulated knowledge describing why the testing approach you are about to take is not going to solve your problem, cost a lot of money, and waste a lot of time”.
Developer: “That’s nice, but we’re going to do it anyway”.

Two months later:

Me: “So, how’d that test approach work for you?”
Developer: “Oh, it completely sucked and was unmaintainable”.

Now, these developers are not stupid people, but it’s difficult for me to understand how things get to this point.

Or at least, it was until recently. Most agile development projects are much like Toyotism (or Lean, if you prefer). Better than how things were in the past, but not fundamentally changing the nature of work. Just as work in manufacturing is still hard, repetitive and largely about lines, so we easily fall into our old factory-style patterns. I believe this is at the heart of Brian Marick’s statement on the current state of Agile project experiences: ‘Whereas the number of people new to Agile who describe their project as “the best project I’ve ever worked on” seems to be declining, and we believe work should be joyful.’

Parallel lines of software development, with variable development and testing tasks performed serially end up being just as unpleasant as the old ways. Except now the moment your testing task takes longer than the development task in the queue behind you, someone’s screaming about the tester backlog.

Of course, it doesn’t have to be this way, but there’s very little incentive for the guy with the upstream work to contribute to solving it. They’re not the ones feeling the pressure. And once those pesky testers are gone, the whole thing actually appears to work much more smoothly. After all, what’s smoother than an assembly line with a single step?

Hint: The last line *is* the answer. But the devil is in the details of what comprises a ’step’. And what Matt describes is a step toward the solution as well.

Stay tuned.