I’ve recently been on the move a little, and have had a lot of chances to work on test strategy. I generally have historical documents to work from, but decided I should try to come up with a mnemonic device to ensure that I have all of the critical conversations that I need.

One of the most influential things for this was the RUP development case that Michael Ruschena presented when I worked with him at ANZ. This documented the team approach we had agreed upon to deliver a solution to the customer’s needs when working on our first agile project. This was one of the first times I was really conscious of “software development as applied problem solving”, prior to encountering Gerry Weinberg’s work. Paul Szymkowiak also commented that my test strategy reproduces a lot of what would be in the RUP vision artefact. I realised that in many cases, that’s true. Before deciding on the testing mission, what I’m frequently trying to facilitate is consensus on the project or business goals.

So that’s how this helps me. I hope it might help you. To that end, I present the first public version of my test strategy mnemonic – “GRATEDD SCRIPTS”.

• Goals – What are the critical goals of the product? What are the things that absolutely must work?
• Risks – What things are we hoping to avoid? What bad things might happen? What are we doing to address these?
• Approach – How are we going to test this? How we will work together? Who will do what?  Accountabilities may also be considered here.
• Tradeoffs – What tradeoffs we are prepared to make in order to deliver the desired business outcomes?
• Environments – What environments do we have or need? What testing will we do in those environments?
• Dependencies – Is anyone ‘outside’ the project depending on us? Are we depending on anyone? Are there external gates or constrained resources?
• Data – Where will data come from? Are there any special needs?
• Stakeholders – Who has a stake in the software/product/test effort? What are their goals? Who is accountable for what?
• Coverage models – What models will we use to know that we are testing the things we care about? What models will drive test design and test coverage discussions?
• Resources – Who is available to help with testing? What other resources might we need? What’s the budget?
• Information needs – What information needs to come out of the testing process? What decisions does it need to support? What are we trying to learn?
• Prioritisation – What is most important? How will we resolve competing interests, tasks or tests?
• Tooling – Will any special tools be required? Will support be needed to develop these?
• Schedule – What are the important dates and timings?Some of the items overlap, but that’s OK for my brain. Different prompts cause me to consider things in a different light.Ben Kelly proposed ‘Budget’ as a separate item. For myself, I cover this in Resources, but if you work in an environment where you are more hands-on with budgets than I typically am, you might find the mnemonic “B-GRADED SCRIPTTS” to help you with a more explicit ‘B for Budget’ reminder.

  So next time you are thinking about what your test effort should look like, try to work through the above points and see if there is anything you’re missing. Try to come up with your own mnemonic devices too to help you remember things that are important.

Michael has moved to a new city, and is obviously free of social distractions. That, or there’s something physically stopping him from playing World of Warcraft. Anyway, he’s blogging again and his latest entry (http://www.ruschena.org/michael/?p=107) on writing technical requirements is well worth a read.

In the context-driven software testing Yahoo group, there has been an interesting thread on magic numbers. Part of this discussion related to magic numbers for software maintenance investment. While I think you can find plenty of literature that advises a bias towards maintenance, my friend Michael couldn’t find any models that satisfied our burning questions, so he built his own.

Check it out, and thank him for doing my homework

While looking for advice on improving my critical thinking, I came across this article -

http://www.csse.monash.edu.au/~ctwardy/Papers/reasonpaper.pdf

Interestingly, there are a bunch of tools listed here that claim to help with the technique of argument mapping. I haven’t had a chance to try any of these yet. Hopefully, encouraging others to check them out will yield feedback faster than trying them all on my own!

Links to experience reports would be most welcome.

Does your system accept real world data? Does it restrict the lengths of fields and/or prevent certain characters from being entered? How do you know when you are allowing the right kinds of data?

While chatting with colleagues about the NOTAG bug and some of the features of the system we are working on (it is vehicle related), it became clear that many testers never bother to confirm whether field validations on everyday data items are actually appropriate.

For instance, many systems in Australia are developed on the East coast of Australia. Now, I haven’t had to check this out for a while, and my records are buried in some old notes, but the important point is that in Australia’s two most populous states the longest license plate you can have is six characters.

Most systems that I have tested which collected vehicle registration details, happily assumed that this held true for the rest of the country also. And for a while, I think this was true. Western Australia has, however, had nine-character license plates for at least a couple of years now. Some other states allow seven. I know of more than a few systems that probably aren’t going to accept your HIWAY2HEL plate.

So here’s my question: In systems that you are testing, how many of the everyday data fields have you researched to find out whether the specified constraints were actually appropriate?

I’m thinking of things like -

- A phone number. There must be at least a few people out there with global roaming.
- An address.
- An international postcode.
- A surname. On a credit card. Here’s your credit card Mr. Saravanalanganingham. Or not.
- An email address.
- A URL.

While your at it, it might be a good idea to check what characters are valid in that email address or URL before you plan testing for that validation feature. The days when I am no longer allowed to enter my .info email address are thankfully much fewer now.

Google is usually the place to start when you’re looking for this kind of information, but there are other resources too. We also make assumptions about the behaviour of other things, and there are references for those too -

- The printed white pages and yellow pages can be a great source for finding real addresses and names, as can city street maps. The Australian white pages has a bunch of other handy references, including international dialling examples and a list of all postcodes.
- Speaking of postcodes, you can find a complete list of Australian postcodes and cities here.
- Curious to know what you might need to handle in that email or URL field? Read the RFC…
- Want to know who to blame for that website not working on your browser of choice? On another recent project, we couldn’t help but feel that keyboard navigation was a bit clunky on a select list control. There were also differences in IE and Firefox. It turned out that neither browser was actually behaving correctly. How did we know? We looked at the spec…www.w3c.org is your friend.

RFC.net is also a great resource for performance testers, or any time you need to work with the guts of the internet – HTTP and TCP/IP. On a recent project using AJAX, this was a vital resource for troubleshooting caching issues in our application. Mozilla/Firefox’s livehttpheaders extension was a great help too. It monitors the HTTP headers of webpages as they are received by your browser.

One final question: How many of you have been bitten when perfectly legitimate data couldn’t be accepted by a software system you needed to use? Care to share some stories?

I came across someone asking for an answer to the old “How do you test a stapler” question, and in light of my new role, I thought this was a good opportunity to start taking up James Bach’s methodology challenge, using the stapler example as a starting point.

I’m starting with the meta-questions, those which I need to answer *before* I can begin to think about what and how I’m going to test it.

• Who is the stapler for? Who is it *not* for?
• Why do they want a stapler?
• How might the different stapler users know that the stapler has fulfilled their needs? What stapler features are unimportant?
• When does it need to be tested by? What will happen if it’s not done by then?
• How much can I spend to test the stapler?
• Who else can help me test the stapler?
• Does it need to be tested by anyone else before it can be considered ‘ready’?
• Does anyone else need to check the stapler, or approve my work before the stapler can be used by the person who wants the stapler?
• Is there anyone who doesn’t want the stapler to be had?
• What tools am I allowed to use to test the stapler? What tools do I know how to use that might help?
• Who might need to test the stapler at a later date? Who is going to read my tests? What documents do I need to create? Where do they need to live?
• Who shouldn’t be able to use the stapler?
• How critical is the success of the stapler? What might happen if it isn’t tested adequately, or fails to meet the needs of the stakeholders?
• Are there any standards our testing needs to comply with?
• Are there any legal constraints?
• What kind of staples does it need to work with?
• What kind of materials should it be able to staple?
• Does it need to work in conjunction with anything else?

I’m sure I will continue to add to this list and organize it, but it’s a start.

I’ve also found I need to develop a checklist to hit the ground running and ensure that I look far enough forward when my involvement with a project begins. For this, I’m grouping some questions under the heading “Start from the end”. This is a subset of the above questions, but are the questions I think I’m most likely to leave until too late if left to my own devices.

• What does success look like?
• What are the release processes/signoffs? Who are the stakeholders who need to sign off?
• When does it need to ship by? What is depending on it being released on that date? What else does our product depend on?

And so, it begins. If only I’d started earlier!

Developing personas is a well-described technique (see Alan Cooper’s ‘The Inmates are Running the Asylum’ and Mike Cohn’s ‘User Stories Applied’) for considering the different kinds of users of the system we are developing. On a recent project, we began considering the different users who might want to user our product. In the process, I was able to connect it to another recently acquired trick and gain further insight into the user of our system.

The initial features added to our system seemed to be really only considering one type of user. Most of the team felt strongly that there were other important users who were being ignored, as many of the features didn’t seem to work the way we expected them to. They made sense for the targetted user, but made little sense when we thought about how *we* would want to use the system.

This suggested to us two user types – The bargain hunter and the traveller. We imagined the traveller to be someone planning a future trip. They were looking for discounted goods or services that would be available for them to use on holiday, some time in the not-immediate future. We also imagined a bargain hunter, someone for discounted goods and services, with a sense of urgency. This is the kind of shopper who hates to miss out on a bargain. Our system forced us to consider this kind of user as the primary customer.

It seemed that the two had some common objectives, but different needs. We tried to model this -

This begged the question -

What about these customers? Who might they be? We wondered if we could give them a name, and if they were worth considering?

We named them. They were important customer types whose needs had to be taken into account.

The Special Occasioner - A person who is looking for a good deal, but is prepared to take time to find it. Perhaps for a wedding anniversary, or someone who is planning a holiday, but hasn’t firmly decided.
The Last minuter – A person who leaves things to the last minute, has to unexpectedly travel, or just acts on the spur of the moment.

It was at this point that I flashed back to my recent reading of Howard Becker’s ‘Tricks of the trade‘, and his writing on ‘Substruction’. He describes substruction as -

‘…the logical converse of reduction. Reduction puts combinations together for the sake of simplicity. Substruction takes them apart, in the interest of discovery.’

As it turns out, this is exactly what we had just been doing. From analysis of our initial two personas, we were able to derive two more, and on further analysis we expanded this even further.

Here’s the process -

1. Determine the important attributes of a persona. In our case, we started with two, but you may have more.
2. Write down all the combinations of presence and absence of the attributes. This may be a truth table, but Becker’s book provides examples for non-binary attributes.
3. Work out which combinations of present or absent attributes correspond to the personas you have already thought of.
4. See which combinations don’t have a name. Try to name them.

Personas are a great tool. Enhancing the approach with the techniques of substruction gave us a much richer model of our system and the kinds of people who may have an interest in using it, and points to possibilities we may not have considered. Howard Becker’s book, though targetted at sociologists, is a great read for anyone looking to improve their thinking.