This news article where a hairdresser’s client went bald after a power loss in the salon reminded me of a test I frequently forget to run.

Power off your system or server while in the middle of testing and look for problems.  You’ll most frequently find issues if you aim for a power outage -

- in the middle of a network communication (eg. client-server apps or network games).

- in the middle of a multi-step transaction.

- in the middle of disk operations.

The last one’s pretty brutal, but sometimes you need to know your application can survive it gracefully.  Have fun!

Continuing the ‘what tool’ theme from last week, today’s topic is ‘Diff’.  

I frequently install windows versions of various Unix command line utilities via the Gnu Utilities for Win32 project.  Diff is particularly handy not just for the programming side of automation, but also for comparing output files from automation as well as database queries.  Occasionally though, I need the niceties of a graphical tool that handles side-by-side comparison of file differences a bit more nicely.  So I installed KDiff3 (http://kdiff3.sourceforge.net/), and it seems pretty good, supporting three-way comparisons.

I also looked at Winmerge which has a portable version.  It seems to have a nicer file diff view, but KDiff has a nicer view of folder differences.  Given that it’s portable, it will go onto my tester toolkit (although there’s some evidence KDiff may be portable enough for my needs).

Are there other diff tools I should know about?

As yet another poor internet soul is scammed by a man pretending to be a woman in online chat rooms, I’m reminded of the sensibility of my number one internet heuristic.

Jared’s first law of online safety is ‘Assume that everyone you are talking to online is a man’.

This has held me in good stead over time, but how relevant is this to testing? Well, it’s strongly related to the testing technique of claims testing. Claims testing is what you’re doing when you are testing a product to specifications or requirements. It is perhaps the most common test approach in large corporate environments I encounter, but it’s also a technique that can be applied in a shallow, context-free way.

James Bach has two points in his Heuristic Test Strategy Model that I find key in describing claims testing -

- Verify that each claim about the product is true.
- If you’re testing from an explicit specification, expect it and the product to be brought into alignment.

Perhaps closer to what we are actually doing is that we usually verify that each claim about the product *can* be true, for some particular situation or situations. We also try to understand for which cases and contexts the claim holds true. And in order to verify the claim, we set about collecting *sufficient* evidence of the truth of the claim, because exhaustive proof is either prohibitively expensive or impossible.

An example I like to use to teach how claims testing should work begins with me making the claim ‘My name is Jared’. I follow with the question ‘How would you set about finding out whether it really is true that my name is Jared?’

The example is not as straightforward as it may seem, because the level of evidence required depends entirely on the purpose for which we need to know.

If I’ve introduced myself online in a chatroom or in person, and you don’t care about any interaction beyond that room, my assertion that my name is Jared may be sufficient.

If you’re the government of Australia, then your standards are a little higher depending on the service you’re providing to me. Sufficient evidence may include my birth certificate or passport, an assortment of historical information, and other knowledge of the details that the government has stored about ‘Jared’.

In other instances, social proof may be OK. The fact that others refer to me as ‘Jared’ may be sufficient evidence, and we can build up a body of evidence based on a history of social interactions.

Or perhaps you’re a hitman paranoid about bumping off the wrong person, so you brute-force the solution. You stalk me for a month and rummage through my mailbox, finally whacking me on the head and rifling through the contents of my house and wallet.

Each one of these approaches may be required for sufficiency, and is appropriate for different contexts and purposes. And we might take similar approaches when we’re testing software. We might simply talk to people who can provide evidence of a claim being met. We might perform a simple confirmatory test for a non-critical function. We might spend a longer time collecting a large body evidence to support a claim being met. It’s important that we think about the importance of the claim and ensure that our approaches to gathering information can be defended under scrutiny.

The real-world analogy to the second point about claims testing – ‘Expect the spec and the product to be brought into alignment’ is perhaps more commonly found in legal circles, and occasionally in the forum posts or blog rants of less careful men who feel their masculinity has been threatened. It tends to involve situations like this. And as in a situation detailed here that more closely parallels software testing, remember that refuting a claim can sometimes have much bigger consequences.

For the first time, I’m helping run planning sessions for an agile project. Planning has been a bit of a bugbear for me on many of my recent projects, so I’m excited to have a chance to try some things out. So far, it seems to be going well. It’s a short project, so I shouldn’t have to wait too long for feedback on how things went, which is a bonus.

One of the things I noticed is that when in planning or requirements sessions, there’s a tendency to fall into an interview format. That is, we (the development team) ask questions, write things down, ask clarifying questions, and repeat until we think we have enough information to move forward. I think that can be fine sometimes, but it feels like one of the simplest things (but not necessarily the quickest) we can do to help our understanding is to ask our customers to show us how they work. That is, the interview style becomes ‘fly on the wall’ rather than ‘question time’.

As a tester in more traditional projects, I’ve also had occasions where despite the documentation around the project, deep understanding of the domain was difficult to come by, and generally absent from the test team.

On a recent project I had some success combating this by just having the gumption to ask a passing front-line operator “Can I sit with you and watch you work, or do you have time to walk me through the systems you use?”

“Sure! Your desk or mine?” was the answer.

It didn’t hurt me to ask (as I could fit the time in around my other responsibilities), and he was happy to know that some of his concerns might be represented in future. But I know that I don’t always think to do this, or feel that my request will be considered. I learned a lesson though.

So on my current project, it’s time for us to ask the same question of our client. We’re talking through the problem, but the moment the development team is on its own, we keep hitting decisions that can’t be made because our thinking is functional thinking. That is, the client is suggesting features, but we’re not digging deeper to understand the precise intent of those features requests, or the problem that those features will solve. This understanding is critical not just for our testing, but for the whole team to be successful.

Now, we don’t have to work this way. We could just start building the product and let unsatisfying features fall out of the process with our on-site customer, but I believe the amount of up-front thinking and analysis we do is an uncertainty reduction activity. We almost guarantee ourselves a slightly longer project, but I think we reduce the likelihood of having nothing useful to show to the customer at the end of our budgeted time. Critically, I think we have to inform our customers of the tradeoffs we are making around up-front analysis, and how that will impact them.

Excuse me now, I have to go ask someone a question!

Matt Heusser is describing his challenge to test various inanimate objects – An egg, a stapler, a salt shaker and a knife. Read it here, but be sure to come back for the rest of the problem.

I’d now like you to spend a few minutes thinking about how you might go about testing an egg.

Now I’d like you to read this and this. (Oh, read this if you can’t read Chinese. Oops). Now ask yourself if there’s anything you’d like to change about your egg-testing strategy. Or another question you think might be worth asking before you start testing the egg.

The questions that leap out at me are -

- Who am I testing this for?
- What is the purpose of the egg?

When testing, we often assume the answer to these questions without checking what our mission really is. This can lead to unwelcome surprises at the pointy end of the project.

Now you can read this and start thinking about your egg testing strategy again! Happy testing…