Claims testing in the wild
As yet another poor internet soul is scammed by a man pretending to be a woman in online chat rooms, I’m reminded of the sensibility of my number one internet heuristic.
Jared’s first law of online safety is ‘Assume that everyone you are talking to online is a man’.
This has held me in good stead over time, but how relevant is this to testing? Well, it’s strongly related to the testing technique of claims testing. Claims testing is what you’re doing when you are testing a product to specifications or requirements. It is perhaps the most common test approach in large corporate environments I encounter, but it’s also a technique that can be applied in a shallow, context-free way.
James Bach has two points in his Heuristic Test Strategy Model that I find key in describing claims testing –
– Verify that each claim about the product is true.
– If you’re testing from an explicit specification, expect it and the product to be brought into alignment.
Perhaps closer to what we are actually doing is that we usually verify that each claim about the product *can* be true, for some particular situation or situations. We also try to understand for which cases and contexts the claim holds true. And in order to verify the claim, we set about collecting *sufficient* evidence of the truth of the claim, because exhaustive proof is either prohibitively expensive or impossible.
An example I like to use to teach how claims testing should work begins with me making the claim ‘My name is Jared’. I follow with the question ‘How would you set about finding out whether it really is true that my name is Jared?’
The example is not as straightforward as it may seem, because the level of evidence required depends entirely on the purpose for which we need to know.
If I’ve introduced myself online in a chatroom or in person, and you don’t care about any interaction beyond that room, my assertion that my name is Jared may be sufficient.
If you’re the government of Australia, then your standards are a little higher depending on the service you’re providing to me. Sufficient evidence may include my birth certificate or passport, an assortment of historical information, and other knowledge of the details that the government has stored about ‘Jared’.
In other instances, social proof may be OK. The fact that others refer to me as ‘Jared’ may be sufficient evidence, and we can build up a body of evidence based on a history of social interactions.
Or perhaps you’re a hitman paranoid about bumping off the wrong person, so you brute-force the solution. You stalk me for a month and rummage through my mailbox, finally whacking me on the head and rifling through the contents of my house and wallet.
Each one of these approaches may be required for sufficiency, and is appropriate for different contexts and purposes. And we might take similar approaches when we’re testing software. We might simply talk to people who can provide evidence of a claim being met. We might perform a simple confirmatory test for a non-critical function. We might spend a longer time collecting a large body evidence to support a claim being met. It’s important that we think about the importance of the claim and ensure that our approaches to gathering information can be defended under scrutiny.
The real-world analogy to the second point about claims testing – ‘Expect the spec and the product to be brought into alignment’ is perhaps more commonly found in legal circles, and occasionally in the forum posts or blog rants of less careful men who feel their masculinity has been threatened. It tends to involve situations like this. And as in a situation detailed here that more closely parallels software testing, remember that refuting a claim can sometimes have much bigger consequences.