This news article where a hairdresser’s client went bald after a power loss in the salon reminded me of a test I frequently forget to run.

Power off your system or server while in the middle of testing and look for problems.  You’ll most frequently find issues if you aim for a power outage -

- in the middle of a network communication (eg. client-server apps or network games).

- in the middle of a multi-step transaction.

- in the middle of disk operations.

The last one’s pretty brutal, but sometimes you need to know your application can survive it gracefully.  Have fun!

I’ve been using the free Squirrel SQL SQL client under windows for a month or so now.  It’s a good tool, though somewhat annoying to get working.  Today it stopped working.  The loading splash screen would display, the progress bar would get about halfway through and then Squirrel would exit without any messages.  I had no desire to recreate all of my connections or reinstall the various drivers again, so I really wanted to fix my installation.

After trolling through forums, there was a suggestion that the problem may have been preferences related. No precise solution was offered, but I began to experiment to see if this was my problem.

First, I found the preferences folder, which lives in Windows’ documents and settings folder (eg. c:\Documents and Settings\). Inside this folder will be Squirrel’s preferences folder, named ‘.squirrel-sql’. I renamed this and restarted Squirrel. Things looked good with the application starting, so it seemed I was looking in the right place. In order to troubleshoot further, I wanted to restore the state of the application, so I renamed the new preferences folder that Squirrel had created and tried to rename the old preferences folder.

No luck! Windows didn’t like me trying to rename the folder back to its original name. I ran Squirrel again, which caused Squirrel to create another preferences folder. I now had three folders – squirrel-sql.old, squirrel-sql.new and the current preferences folder ‘.squirrel-sql’. I opened the old preferences folder, copied the contents and pasted them into the ‘.squirrel-sql’ folder.

Looking inside the preferences folder, I could two folders ‘plugins’, and ‘logs’. I could also see a number of xml files. Now that I had found the broad area I needed to investigate, I wanted to only change one element at a time. As my main objective in resurrecting Squirrel was to not lose my database connections and plugins, I ignored the xml files that were related to these, and looked at the most interestingly named file – ‘prefs.xml’. I renamed this to ‘prefs.xml.bak’ and restarted Squirrel. Still no joy, so I closed Squirrel and restored the original name of the file..

I repeated this step for ’sql_history.xml’, thinking that this file might be dynamic enough to cause problems. Again, Squirrel failed to start correctly.

Next, was a file named SQLAliases23_treeStructure.xml. Suspiciously, this was zero bytes, which seemed odd for something that looked like it was supposed to contain some kind of data structure. I added a ‘.bak’ extension to this and restarted Squirrel again.

Success! I closed Squrrel and I could see that it had recreated the SQLAliases treeStructure file again, this time with data. I restarted Squirrel one more time to make sure that there wasn’t some recurring problem with my database aliases, and it happily started again with my connections and query history intact.

I’ve recently been on the move a little, and have had a lot of chances to work on test strategy. I generally have historical documents to work from, but decided I should try to come up with a mnemonic device to ensure that I have all of the critical conversations that I need.

One of the most influential things for this was the RUP development case that Michael Ruschena presented when I worked with him at ANZ. This documented the team approach we had agreed upon to deliver a solution to the customer’s needs when working on our first agile project. This was one of the first times I was really conscious of “software development as applied problem solving”, prior to encountering Gerry Weinberg’s work. Paul Szymkowiak also commented that my test strategy reproduces a lot of what would be in the RUP vision artefact. I realised that in many cases, that’s true. Before deciding on the testing mission, what I’m frequently trying to facilitate is consensus on the project or business goals.

So that’s how this helps me. I hope it might help you. To that end, I present the first public version of my test strategy mnemonic – “GRATEDD SCRIPTS”.

• Goals – What are the critical goals of the product? What are the things that absolutely must work?
• Risks – What things are we hoping to avoid? What bad things might happen? What are we doing to address these?
• Approach – How are we going to test this? How we will work together? Who will do what?  Accountabilities may also be considered here.
• Tradeoffs – What tradeoffs we are prepared to make in order to deliver the desired business outcomes?
• Environments – What environments do we have or need? What testing will we do in those environments?
• Dependencies – Is anyone ‘outside’ the project depending on us? Are we depending on anyone? Are there external gates or constrained resources?
• Data – Where will data come from? Are there any special needs?
• Stakeholders – Who has a stake in the software/product/test effort? What are their goals? Who is accountable for what?
• Coverage models – What models will we use to know that we are testing the things we care about? What models will drive test design and test coverage discussions?
• Resources – Who is available to help with testing? What other resources might we need? What’s the budget?
• Information needs – What information needs to come out of the testing process? What decisions does it need to support? What are we trying to learn?
• Prioritisation – What is most important? How will we resolve competing interests, tasks or tests?
• Tooling – Will any special tools be required? Will support be needed to develop these?
• Schedule – What are the important dates and timings?Some of the items overlap, but that’s OK for my brain. Different prompts cause me to consider things in a different light.Ben Kelly proposed ‘Budget’ as a separate item. For myself, I cover this in Resources, but if you work in an environment where you are more hands-on with budgets than I typically am, you might find the mnemonic “B-GRADED SCRIPTTS” to help you with a more explicit ‘B for Budget’ reminder.

  So next time you are thinking about what your test effort should look like, try to work through the above points and see if there is anything you’re missing. Try to come up with your own mnemonic devices too to help you remember things that are important.

An excellent quote from our development lead, James Ladd:

How to test a project might be risky -

- It has people in it.

Thanks to James Bach for inspring this!

Many moons ago, when I was a young tester, I worked on the first third party game for Microsoft (Please don’t look for it, it’s terrible). But there was some good to come out of the experience. Windows ‘95 was new shiny, and fraught with danger. To help address some of this, I began collecting test ideas and describing the expected behaviour of ‘good’ Windows applications.

Much of the older documents we created were designed to support an approach that was mostly exploratory. Exploratory testing was supported by coverage models that provided loose charters, and the feedback on tester sessions was verbal.

For the more prescriptive parts of our test effort, documents such as this provided support. Bug catalogues, risk catalogues and checklists were used extensively to ensure that important things were not left untested.

So enjoy the first of these checklists, my Windows ‘95 testing cheat sheet. For anyone testing a Windows game or application, much of this still applies. If you’re testing web applications, this may give you some ideas for how your application should handle events for user-interface controls.

You can find the checklist at http://www.quinert.com/documents/Win95AcceptanceChecklist.pdf.

The concept of inattentional blindness has been, if you’ll excuse the unintended pun, brought to the testing world’s attention lately by Cem Kaner and James Bach. Sajjadul Hakim has written recently on an exploratory testing experience in which he attributed failure to observe a bug to inattentional blindness.

While this may have been the cause, it is worth pointing out that our attention is a fragile thing. Inattentional blindness is but one of the ways that our brains can let us down while testing. Here are a few more:

Attentional blinking

If something grabs our attention, then something else happens almost simultaneously, or just after, we may miss it.

Inhibition of return

If something happens somewhere where we were just looking, we are less likely to see it.

Negative priming

Our brain applies filters to suppress things that are not relevant to the task at hand. These filters can arise through conscious effort or can happen if we perform a task long enough. An example of this is ‘banner blindness’, where things appearing in the spots where website advertising usually appears are highly likely to be ignored.

I became acutely aware of this effect last year. Steve Irwin had just died, and the news came to our team via a phone call to a teammate. I immediately jumped on CNN’s website, looking for confirmation. “There’s no news of it here,” I confidently claimed to the rest of the team. I looked back at the screen, and scrolling across the top of the page was a ticker, proclaiming Irwin’s death as ‘Breaking News’. It’s position at the top of the page, where advertisements usually appear, had caused me to skip straight over it.

Our brain is able to put filters in place to help optimise our performance of common tasks. These effects can last for more than a month after we have stopped performing the task that caused the filters to be put in place, just in case we need to do it again.

Our brain’s processing power is limited. In addition to the above problems, sometimes there’s just too much for it do deal with. It has developed tricks to help us survive lapses of attention, but these are also things that can catch us out as testers. The act of operating the product in order to test it is something which focuses our attention. By doing so, we blind ourselves to some problems.

I play video games with a kind of unfocused stare, and I suspect this actually helped me detect different kinds of problems when testing games, while probably making me blind to others. I find I can’t usually test applications with that protracted stare. I can however observe the application this way by putting the application under computer control, or by observing a playback of a recorded session. Video recorders and Watir scripts are great for this, and give you the chance to find different problems. James Bach also frequently recommends Spector for recording testing sessions.

Strategies

There are plenty of strategies for reducing attentional failures. Mix these up over the lifetime of the project for best effect. Some of them I’ve described already, but others I leave for you to Google -

- Rotate people through projects and tasks to avoid negative priming.
- Negative priming effects are reported to be reduced by alcohol consumption. Have a drink at lunchtime. Tell your boss science says it’s OK!
- Pair up…Maybe even with non-testers or people who have nothing to do with the project.
- Use recording tools and go back and review your testing sessions.
- Automate operation of the product, sit back and observe. I’ve used this technique, and Jonathan Kohl has success stories to tell as well.
- Blink testing. This is a defocusing strategy popularized by Michael Bolton and James Bach. Find different problems in test results, fast, by taking the detail out of them and scrolling through them quickly or zooming out.

If you would like to find out more, check out the book Mind Hacks: Tips & Tricks for Using Your Brain (Hacks). It’s a great primer on your brain, how it works, and how it can let you down. The ‘Developing Intelligence’ blog is also another great source of information and ideas.

In the Yahoo group supporting Cem Kaner’s Black Box Software Testing course, Anil Soni has been describing experiences organising and leading internal training, using the BBST course materials. One point in particular caught my attention:

> The major challenge is to have all the testers together in the same time
> needed for the group interaction.

In a recent role, I had trouble getting all of my testers together for study sessions. Often, the testers felt uncomfortable stopping work to come to a study session. At times, work was a genuine priority. At other times, the things testers were working on could wait an hour or two, but it was difficult for the tester to make that decision.

Sometimes I would have to make the decision for them. That is, tell them “This is more important than the work you are doing right now”. I could do this because I had management support for creating a learning environment.

Organisationally though, we were sending mixed messages. The project manager expressed the opinion “We’re not running a school here”. The development manager countered with “We absolutely are”. These two statements captured the tension between getting today’s job done now, and ensuring that tomorrow’s work can be done better or more efficiently. In one statement, is the view that employees should always be working on something that directly contributes business value. In the other statement, the view that it is vital that time be set aside during work hours expressly for improving the skills of staff and for reflection on past experience.

At this point, I don’t really want to enter into the philosophical debate around whether organisations should be learning organisations are not. However, if you’ve decided that it is important that your staff learn and improve here are a few things that I would consider.

• Make it clear how much time you expect employees to be spending on improving their skills.
• Check that they are spending the necessary time. If not, find out why.
• Keep groups small if possible. This allows scheduling to be more flexible.
• Get a clear statement (or statements) from whoever is in authority that study time is important for the organisation.
• Establish and communicate principles for when work should take priority.
• If participants are having difficulty deciding what is more important, help them.
• If you are going with a less structured informal learning approach, how will you know whether team members are using their study time for study? That is, what feedback mechanisms will you have to help understand and support your team’s learning needs?

On this last point, in one team, a thousand dollar self-education budget was available to each employee per year. They were free to spend this on anything that they wished. However, in a year and a half, almost none of the team had made use of that budget. That’s one example of a feedback mechanism, triggered by the question “Why haven’t you used your training budget?” Regular team meetings, where each team member presented some recent learning were another.

The dominant work culture is about appearing to be busy, and doing ‘real work’. Despite support at the highest level, if other management levels don’t support the company’s learning objectives, employees will be quite nervous about spending time on tasks that their close managers may consider trivialities. This is a significant cultural change, and the forces of history will be working powerfully against you.

I’ve recently come across the Informal Learning blog, and hope to write more on this subject in the future.

Having seen a number of testers posting their heuristics on Testing reflections, I thought I might chime in with one that popped into my head. I’m calling it the SEP heuristic, which is probably all you need to know about it if you are familiar with Douglas Adams’ Hitchhiker’s Guide books.

It is simply this -

- Anything which project members or stakeholders appear to be treating as “somebody else’s problem”, is quite likely not so, and highly likely to bite us firmly on the backside if we continue to treat it as such. Ignore at your peril.

As SEP fields can be beaten by specifically looking for that which is concealed, the countering strategy is simple, but not necessarily easy. The SEP heuristic tells us to look for problems or tasks that are being ignored or thrown in the ‘too hard’ basket and focus energy there sooner rather than later.

I think that early in one’s career, it’s easier to let some things be somebody else’s problem, with minimal personal consequences. I’m finding as I have increasing responsibility, this heuristic becomes increasingly important.