Via Ben Kelly, Rikard Edgren’s brief but dense ‘Little Black Book On Test Design‘ is worth a read.

It’s cheap both in dollars (free) and time (less than 15 minutes if you’re quick).

Alister Scott has posted some code examples on watirmelon.com that show different solutions to a problem that arises for people implementing the Page Object pattern for test automation.

I found Alister’s approach solutions interesting, because as my automation usually models business/user goals and/or domain features/concepts,  I don’t often have my own page abstraction.  See http://gojko.net/2009/10/06/putting-selenium-in-the-right-place/ for an example of the approach I’m thinking of.

At the top level might be something like:


google=Google.new
google.search(search_phrase)


At the next level down, I potentially want to be independent of the user interface, so it doesn’t make sense to organise around page components.  I might have something like:


def search(search_phrase)
enter_search(search_phrase)
submit_search
....
end


When we get to the third (task) level, this is still at a user or business activity level, so the page model still doesn’t make sense for me.  I’ll just use the browser driver directly (or my own driver abstraction):


def submit_search
Driver.button(:value, => "Google Search").click
end


Yes, I could have another level here such as:


def submit_search
GoogleHomepage.search_button.click
end


In most cases though, the task-level UI element has a single purpose, so there’s not much of an issue with violating DRY. I also try to keep the automation stack as small as possible, so can tolerate a little bit of duplication. Regardless, it’s easily factored out into a new method if duplication becomes an issue because methods are common to the domain.

I’ve seen some frameworks that have a really abstract UI driver at the next level (so that they could drive a rich client app, a mobile app and a web app all using the same methods), but I’ve never needed to do that.

I favour a business/domain oriented approach because at the business level, the goals and activities don’t change that much (and I tend to let the automation lag behind my understanding of the problem).  It also encourages me to pay attention to the things that matter and be less implementation-focused.

Today I found out that my post on Watir and TinyMCE let someone dump their Quick Test Pro licences.

A victory for common sense…

Nothing against QTP, but if you’re testing web applications, you should probably be looking at other tools.

According to the dev team, this is the muppet representation of me as a tester -

But as well as straying into Sam’s stern, righteous, “everything’s wrong” attitude on (rare, I hope,) occasions, I think aspects of my testing persona could just as well be represented by some different muppet outlooks:

Animal – The performance and stress tester.  High-volume test automation.

Beaker – The black-thumb tester.  Something baaaad is gunna happen.

Kermit – Maintainer of relationships and manager.

Miss Piggy – Hiiiiiiya!  Throw in some occasional defending of the weak as well.

Swedish chef – Just mix it all up and wait for chaos (and Borking) to ensue.

Gonzo – “Do you really think this will work?” – “No! Isn’t it great?!”

Dr. Bunsen Honeydew – The analytical tester.

Scooter – The gofer, assistant ( to developers).

The Count – Obsessive attention to detail and number wrangling.

Floyd – The creative tester.

Which muppet suits your testing personality?

The topic of interviews came up while chatting with a friend recently, and I noticed that the Dr Dobb’s interview I did a couple of years back is no longer available.  I’m reposting it now to have my own copy and to save me from having to dig it out of the Internet Archive in the future

Originally featured in ‘The book of testing – Thoughts from a Braidy Tester’ at http://www.ddj.com/blog/debugblog/archives/2007/12/five_questions_41.html

by Michael Hunter
December 18, 2007

Five Questions With Jared Quinert

By The Braidy Tester
Posted: December 18, 2007 07:30 AM
Once upon a time Jared Quinert tested console games. Then he led teams of testers of console games. Then he built the testing department for a console game which never actually shipped. Currently he is spreading the context-driven testing love throughout the corporate world as a consultant with Aegeon, one of those Web 2.0 companies.

Here is what Jared has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JQ: I started as a games tester at the beginning of 1995, catching the tail-end of ROM-based games and the transition to CD-based PC and console game titles. Due to the expense of a product recall, those games needed to be rock-solid. Back in those days, our test approach was 100% exploratory, manual testing.

On my first real project, a bug report was thrust into my hand at some point, and I was told to wander around and ask the programmers what they had been working on and what they had changed in the build they had just given us. They would share information about things they thought were complex, risky, or otherwise. I would tell them what kinds of things our team was finding in the product.

After four or five months I was thrown onto a title as test lead, managing one other tester. Suddenly managers were wandering up to me and asking ‘How’s it going? When will we be done? Can we ship it?’ Pretty quickly, I needed to figure out how to answer those questions, and thus I learned about developing coverage models to support exploratory test management.

The company I was at also developed the first third-party Windows game for Microsoft. Microsoft came to us wanting a heavily scripted test approach. Our test group wasn’t even aware that people tried to test things by writing all their test ideas down. We definitely felt we weren’t going to test *our* product that way. After initially rejecting the idea, I did begin to think about what problems they might be trying to solve with their approach. Some of the ideas were adapted and found their way into our regular testing to solve particular problems. So while it was great to get the testing grounding that I did, I’m also grateful that my view of things was challenged fairly early on in my career.

The big things my first job left with were:

• The idea of testing as a service role, and information provider. This came from the relationships we had with developers and producers.
• The idea that exploratory testing, done well, can deliver quality far above what I’ve generally seen using heavily scripted testing approaches.
• The knowledge that if talented people do exploratory testing day in, day out, they get really good at it.
• The idea that face-to-face communication is critical. We moved offices, were split up across multiple floors of the building, and bad things started to happen.
• The knowledge that having skilled testers allows you to travel much lighter.
• The idea that testers have to tell a compelling and realistic story when reporting bugs. We were often representing the concerns of important people who didn’t have a voice on our team: Gamers of different skill levels, with different preferences, publishers and support people.

My understanding now is that not all games testing groups rely on smaller, skilled teams, so I could quite easily have come out of a games testing role with a very different view of testing.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JQ: The big surprise was how useful the skills I developed as a games tester were when I moved to corporate environments. I think this was largely due to the testing philosophy instilled by my first role. I was also surprised by how much worse I perceived the testing in those environments to be. Using exploratory approaches, it was really easy to find bugs that had been lurking in systems for years. Given the potential financial losses involved in many of these systems, I had expected to encounter testing even more rigorous than my games experiences.

Testing for internal audiences I find is a much vaguer experience than consumer product testing. The politics are more complex, and problems are solved in different ways. Learning to be an effective tester in different environments is a huge, probably career-long task.

DDJ: What is the most interesting bug you have seen?
JQ: The most interesting games-testing bugs were usually needle-in-haystack kinds of problems. I remember one of the first bugs I found, which was a crash that happened only after playing seventy percent through a game, then applying an obscure button combination while attacking an enemy which only existed in this one room. I then surprised myself by playing the game back through to that section and reproducing the problem first time. I think that as the programmers later explained to me why it had crashed, I connected the dots and the bug value of null pointers was suddenly burned into my brain.

I think another thing that never ceases to surprise me about good testers, and that’s how much bad luck seems to follow them around when they’re testing. Bugs seemingly fall into their laps. While this may seem like an unfair advantage, I think it needs to be backed up by good investigative skills and the tester’s ability to pull the pieces together and retrace their steps.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
JQ: I see a few challenges. Firstly, that business right now seems particularly keen to reduce the cost of testing. Secondly, developers are becoming interested in testing, and getting better at it. At least in my part of the woods, much of what passes for testing is heavily requirements-based, and fairly clerical. Between outsourcing and enthusiastic developer test automation efforts, traditional tester roles are going to be under increased pressure to deliver value. If we want to have successful, satisfying careers as testers, I think we are going to need to skill up and figure out how to keep adding (enough) value.

The third key challenge is figuring out how to get some experimentation into the workplace and push the boundaries of what we know about testing, in order to test better. Persuading businesses to take on appropriate business risk for an appropriate reward seems like a hard sell from my view on the ground.

DDJ: Is there anything else you would like to say?
JQ: I’ve sort of undertaken a mission to change the testing landscape here in Melbourne, Australia. It’s a slow process, but a community is growing. If anyone feels that they can contribute, I’d love to hear from them!

——————————

Find the original copy at http://web.archive.org/web/20071220220538/www.ddj.com/blog/debugblog/archives/2007/12/five_questions_41.html

When my testing gets technical, there are a lot of things that I only have to grapple with infrequently. Regular expressions are one of those in-again, out-again things for which my expertise varies depending on when you ask me. Today’s Ruby hacking saw me find RegExr which helps you build and test regular expressions, as well as having a lot of ready-to-go samples built into it.  You can also download it as an Air application if you need it locally.

I always seem to end up at regularexpressions.info as well to get help, but had issues with their regular expression tester today.  While the bugs were a hassle, it helped me in a way, because testing your regular expressions with a few different regular expression testers is always a good idea if you think you might need to use them in multiple environments or tools.

Coupled with JEdit or Ruby, regular expressions are a powerful part of your tester toolkit.

What other tools for regular expressions are you using?

When filling out my tax return this year, I was pleased to see that finally, I am no longer ‘IT Analyst – Other’.

I am now a Software Tester!

About time.

I’ve recently been on the move a little, and have had a lot of chances to work on test strategy. I generally have historical documents to work from, but decided I should try to come up with a mnemonic device to ensure that I have all of the critical conversations that I need.

One of the most influential things for this was the RUP development case that Michael Ruschena presented when I worked with him at ANZ. This documented the team approach we had agreed upon to deliver a solution to the customer’s needs when working on our first agile project. This was one of the first times I was really conscious of “software development as applied problem solving”, prior to encountering Gerry Weinberg’s work. Paul Szymkowiak also commented that my test strategy reproduces a lot of what would be in the RUP vision artefact. I realised that in many cases, that’s true. Before deciding on the testing mission, what I’m frequently trying to facilitate is consensus on the project or business goals.

So that’s how this helps me. I hope it might help you. To that end, I present the first public version of my test strategy mnemonic – “GRATEDD SCRIPTS”.

• Goals – What are the critical goals of the product? What are the things that absolutely must work?
• Risks – What things are we hoping to avoid? What bad things might happen? What are we doing to address these?
• Approach – How are we going to test this? How we will work together? Who will do what?  Accountabilities may also be considered here.
• Tradeoffs – What tradeoffs we are prepared to make in order to deliver the desired business outcomes?
• Environments – What environments do we have or need? What testing will we do in those environments?
• Dependencies – Is anyone ‘outside’ the project depending on us? Are we depending on anyone? Are there external gates or constrained resources?
• Data – Where will data come from? Are there any special needs?
• Stakeholders – Who has a stake in the software/product/test effort? What are their goals? Who is accountable for what?
• Coverage models – What models will we use to know that we are testing the things we care about? What models will drive test design and test coverage discussions?
• Resources – Who is available to help with testing? What other resources might we need? What’s the budget?
• Information needs – What information needs to come out of the testing process? What decisions does it need to support? What are we trying to learn?
• Prioritisation – What is most important? How will we resolve competing interests, tasks or tests?
• Tooling – Will any special tools be required? Will support be needed to develop these?
• Schedule – What are the important dates and timings?Some of the items overlap, but that’s OK for my brain. Different prompts cause me to consider things in a different light.Ben Kelly proposed ‘Budget’ as a separate item. For myself, I cover this in Resources, but if you work in an environment where you are more hands-on with budgets than I typically am, you might find the mnemonic “B-GRADED SCRIPTTS” to help you with a more explicit ‘B for Budget’ reminder.

  So next time you are thinking about what your test effort should look like, try to work through the above points and see if there is anything you’re missing. Try to come up with your own mnemonic devices too to help you remember things that are important.

Time has passed, and I thought it was time to update my thoughts on what’s critical to successful software testing (and development). While originally, I started noting important ideas for agile teams, Increasingly, I find most of these apply no matter what environment I’m working in. Check them out on my Haiku page.

Since reading about Celerity, I’ve been excited by the idea that I might be able to take my Watir scripts and run them more quickly by using a browser simulator. It also raised the possibility of taking scripts and running them in different environments and on different platforms, because Celerity runs under JRuby without a browser. It makes http requests directly, and simulates javascript behaviours without actually rendering pages.

I had the hope that this would give me an option to get quicker feedback from my automation. Unfortunately, this hasn’t turned out to be the case for the applications I’m currently testing. Simulation of the javascript on the page seemed to take longer than the real browser. I got quicker feedback by turning off images, blocking certain external hosts and using Internet Explorer. I still hold out hope that Celerity may improve performance, or that the applications I’m testing will be less javascript intensive.

Once I set to getting my Watir scripts working with Celerity, I needed to overcome a few issues. The last of these was the number of differences between Celerity’s behaviour and Watir’s behaviour. Resolving this turned out to be reasonably simple, and the same strategy for getting consistent behaviour between Celerity and Watir driving IE turns out to solve some other problems with differences in behaviour in Watir’s drivers for IE and Firefox.

My first cut at this is by no means complete, with a bunch of unnecessary duplication that can be easily factored out. For now however, it gets the job done, and may be instructive for others.

The strategy is simple -

- Wrap the browser in your own class.
- Dynamically require the appropriate library for the driver you need
- Create your own methods that provide consistent behaviour for the different browsers. In my case, I needed to be able to get the actual html content of the page after it had been modified by scripts running on the page. Each driver (IE, Firefox, Celerity) required a different method to be called to achieve consistency, but using this approach, I can call browser.html and get (almost) the same result from all browsers and drivers.
- Intercept any calls to methods that you haven’t implemented and pass them through to the actual browser object.

This is the first time I’ve touched Ruby’s method_missing method, and it was pretty painless.

There are a few issues getting JRuby up and running with Celerity on windows as well. I’ll describe these shortly in another post. I’ve also recently had issues installing the latest version of Watir, and will detail the solutions to these as well.

Sample code is below. Note that you can actually do without assigning the browser to a global variable, but I provide this example just to show how you can fit this new solution into the ’standard’ Watir approach. If you modify things so that you can specify the driver from the command line, you should be able to run tests using something like -

jruby main.rb –driver :celerity
ruby main.rb –driver :watir_ie
ruby main.rb –driver :watir_ff

----- driver.rb ------
require 'singleton'
require 'rubygems'

class Driver
    include Singleton

    def set_driver(driver)
      @driver=driver
      if @driver==:celerity
        require 'celerity'
        @browser=Celerity::Browser.new
      end
      if @driver==:watir_ie
        require 'watir'
        Watir::Browser.default='ie'
        @browser=Watir::Browser.new
      end
      if @driver==:watir_ff
        require 'watir'
        Watir::Browser.default='firefox'
        @browser=Watir::Browser.new
      end
      return @browser
    end

    def html
      return @browser.document.body.innerhtml if @driver==:watir_ie
      return @browser.xml if @driver==:celerity
      return @browser.html if @driver==:watir_ff
    end

    def initialize
      @browser=nil
    end

    def method_missing(method, *args)
      @browser.send(method, *args)
    end
end

----- main.rb ----
require 'driver.rb'

driver=:watir_ff
#driver=:watir_ie
#driver=:celerity

Driver.instance.set_driver(driver)
$browser=Driver.instance
$browser.goto("http://www.quinert.com/blog/")
puts $browser.html