Via Ben Kelly, Rikard Edgren’s brief but dense ‘Little Black Book On Test Design‘ is worth a read.

It’s cheap both in dollars (free) and time (less than 15 minutes if you’re quick).

The topic of interviews came up while chatting with a friend recently, and I noticed that the Dr Dobb’s interview I did a couple of years back is no longer available.  I’m reposting it now to have my own copy and to save me from having to dig it out of the Internet Archive in the future

Originally featured in ‘The book of testing – Thoughts from a Braidy Tester’ at http://www.ddj.com/blog/debugblog/archives/2007/12/five_questions_41.html

by Michael Hunter
December 18, 2007

Five Questions With Jared Quinert

By The Braidy Tester
Posted: December 18, 2007 07:30 AM
Once upon a time Jared Quinert tested console games. Then he led teams of testers of console games. Then he built the testing department for a console game which never actually shipped. Currently he is spreading the context-driven testing love throughout the corporate world as a consultant with Aegeon, one of those Web 2.0 companies.

Here is what Jared has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JQ: I started as a games tester at the beginning of 1995, catching the tail-end of ROM-based games and the transition to CD-based PC and console game titles. Due to the expense of a product recall, those games needed to be rock-solid. Back in those days, our test approach was 100% exploratory, manual testing.

On my first real project, a bug report was thrust into my hand at some point, and I was told to wander around and ask the programmers what they had been working on and what they had changed in the build they had just given us. They would share information about things they thought were complex, risky, or otherwise. I would tell them what kinds of things our team was finding in the product.

After four or five months I was thrown onto a title as test lead, managing one other tester. Suddenly managers were wandering up to me and asking ‘How’s it going? When will we be done? Can we ship it?’ Pretty quickly, I needed to figure out how to answer those questions, and thus I learned about developing coverage models to support exploratory test management.

The company I was at also developed the first third-party Windows game for Microsoft. Microsoft came to us wanting a heavily scripted test approach. Our test group wasn’t even aware that people tried to test things by writing all their test ideas down. We definitely felt we weren’t going to test *our* product that way. After initially rejecting the idea, I did begin to think about what problems they might be trying to solve with their approach. Some of the ideas were adapted and found their way into our regular testing to solve particular problems. So while it was great to get the testing grounding that I did, I’m also grateful that my view of things was challenged fairly early on in my career.

The big things my first job left with were:

• The idea of testing as a service role, and information provider. This came from the relationships we had with developers and producers.
• The idea that exploratory testing, done well, can deliver quality far above what I’ve generally seen using heavily scripted testing approaches.
• The knowledge that if talented people do exploratory testing day in, day out, they get really good at it.
• The idea that face-to-face communication is critical. We moved offices, were split up across multiple floors of the building, and bad things started to happen.
• The knowledge that having skilled testers allows you to travel much lighter.
• The idea that testers have to tell a compelling and realistic story when reporting bugs. We were often representing the concerns of important people who didn’t have a voice on our team: Gamers of different skill levels, with different preferences, publishers and support people.

My understanding now is that not all games testing groups rely on smaller, skilled teams, so I could quite easily have come out of a games testing role with a very different view of testing.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JQ: The big surprise was how useful the skills I developed as a games tester were when I moved to corporate environments. I think this was largely due to the testing philosophy instilled by my first role. I was also surprised by how much worse I perceived the testing in those environments to be. Using exploratory approaches, it was really easy to find bugs that had been lurking in systems for years. Given the potential financial losses involved in many of these systems, I had expected to encounter testing even more rigorous than my games experiences.

Testing for internal audiences I find is a much vaguer experience than consumer product testing. The politics are more complex, and problems are solved in different ways. Learning to be an effective tester in different environments is a huge, probably career-long task.

DDJ: What is the most interesting bug you have seen?
JQ: The most interesting games-testing bugs were usually needle-in-haystack kinds of problems. I remember one of the first bugs I found, which was a crash that happened only after playing seventy percent through a game, then applying an obscure button combination while attacking an enemy which only existed in this one room. I then surprised myself by playing the game back through to that section and reproducing the problem first time. I think that as the programmers later explained to me why it had crashed, I connected the dots and the bug value of null pointers was suddenly burned into my brain.

I think another thing that never ceases to surprise me about good testers, and that’s how much bad luck seems to follow them around when they’re testing. Bugs seemingly fall into their laps. While this may seem like an unfair advantage, I think it needs to be backed up by good investigative skills and the tester’s ability to pull the pieces together and retrace their steps.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
JQ: I see a few challenges. Firstly, that business right now seems particularly keen to reduce the cost of testing. Secondly, developers are becoming interested in testing, and getting better at it. At least in my part of the woods, much of what passes for testing is heavily requirements-based, and fairly clerical. Between outsourcing and enthusiastic developer test automation efforts, traditional tester roles are going to be under increased pressure to deliver value. If we want to have successful, satisfying careers as testers, I think we are going to need to skill up and figure out how to keep adding (enough) value.

The third key challenge is figuring out how to get some experimentation into the workplace and push the boundaries of what we know about testing, in order to test better. Persuading businesses to take on appropriate business risk for an appropriate reward seems like a hard sell from my view on the ground.

DDJ: Is there anything else you would like to say?
JQ: I’ve sort of undertaken a mission to change the testing landscape here in Melbourne, Australia. It’s a slow process, but a community is growing. If anyone feels that they can contribute, I’d love to hear from them!

——————————

Find the original copy at http://web.archive.org/web/20071220220538/www.ddj.com/blog/debugblog/archives/2007/12/five_questions_41.html

The concept of inattentional blindness has been, if you’ll excuse the unintended pun, brought to the testing world’s attention lately by Cem Kaner and James Bach. Sajjadul Hakim has written recently on an exploratory testing experience in which he attributed failure to observe a bug to inattentional blindness.

While this may have been the cause, it is worth pointing out that our attention is a fragile thing. Inattentional blindness is but one of the ways that our brains can let us down while testing. Here are a few more:

Attentional blinking

If something grabs our attention, then something else happens almost simultaneously, or just after, we may miss it.

Inhibition of return

If something happens somewhere where we were just looking, we are less likely to see it.

Negative priming

Our brain applies filters to suppress things that are not relevant to the task at hand. These filters can arise through conscious effort or can happen if we perform a task long enough. An example of this is ‘banner blindness’, where things appearing in the spots where website advertising usually appears are highly likely to be ignored.

I became acutely aware of this effect last year. Steve Irwin had just died, and the news came to our team via a phone call to a teammate. I immediately jumped on CNN’s website, looking for confirmation. “There’s no news of it here,” I confidently claimed to the rest of the team. I looked back at the screen, and scrolling across the top of the page was a ticker, proclaiming Irwin’s death as ‘Breaking News’. It’s position at the top of the page, where advertisements usually appear, had caused me to skip straight over it.

Our brain is able to put filters in place to help optimise our performance of common tasks. These effects can last for more than a month after we have stopped performing the task that caused the filters to be put in place, just in case we need to do it again.

Our brain’s processing power is limited. In addition to the above problems, sometimes there’s just too much for it do deal with. It has developed tricks to help us survive lapses of attention, but these are also things that can catch us out as testers. The act of operating the product in order to test it is something which focuses our attention. By doing so, we blind ourselves to some problems.

I play video games with a kind of unfocused stare, and I suspect this actually helped me detect different kinds of problems when testing games, while probably making me blind to others. I find I can’t usually test applications with that protracted stare. I can however observe the application this way by putting the application under computer control, or by observing a playback of a recorded session. Video recorders and Watir scripts are great for this, and give you the chance to find different problems. James Bach also frequently recommends Spector for recording testing sessions.

Strategies

There are plenty of strategies for reducing attentional failures. Mix these up over the lifetime of the project for best effect. Some of them I’ve described already, but others I leave for you to Google -

- Rotate people through projects and tasks to avoid negative priming.
- Negative priming effects are reported to be reduced by alcohol consumption. Have a drink at lunchtime. Tell your boss science says it’s OK!
- Pair up…Maybe even with non-testers or people who have nothing to do with the project.
- Use recording tools and go back and review your testing sessions.
- Automate operation of the product, sit back and observe. I’ve used this technique, and Jonathan Kohl has success stories to tell as well.
- Blink testing. This is a defocusing strategy popularized by Michael Bolton and James Bach. Find different problems in test results, fast, by taking the detail out of them and scrolling through them quickly or zooming out.

If you would like to find out more, check out the book Mind Hacks: Tips & Tricks for Using Your Brain (Hacks). It’s a great primer on your brain, how it works, and how it can let you down. The ‘Developing Intelligence’ blog is also another great source of information and ideas.