I’m back at university, in an effort to force myself to knuckle down and get through the pain of learning Java. It’s been 16 years since I was last there, and here are a few of the things I’ve learned so far -

I’ve learned about the syllabus
Despite the move to Java, very little has changed in the overall teaching approach in 16 years. I’ve heard from another source that someone in my position 15 years ago may have been saying the same thing.

I’ve learned something about gold-plating
University seems to reward you for adding extra features. This doesn’t seem to happen in the business world.

A couple of years ago, I had a graduate developer assigned to my test team to help extend a test tool that I had written. He seemed overly focused on wanting to build a GUI for my tool, when a simple command-line interface was all that was needed to solve our problem. Similarly, with a user base of three, robust error handling was something we could live without. After a couple of semesters of university assignments, I think I understand why this was the case.

I’ve learned something about testing
Lecturers are mostly not embracing some of the not-so-recent trends. I’ve found JUnit tests a boon on assignments where we are incrementally adding new features. Having these tests as I implement existing functions using newly-learned language features, or to add new functionality for new assignments makes things much less stressful. Implement a test per requirement also helps to make sure I get all the marks for the assignment as well. When I suggested a student learn JUnit rather than passing parameters to their classes manually via a GUI interface, lecturers seemed against it, despite the many practical (and career) benefits.

I’ve learned something about learning
University seems really keen to have students build GUIs on everything. Now, GUI building seems like a reasonably complex thing, object-wise. As a beginner, figuring out how to model the domain is hard enough without being forced to put a hacky user interface on top of it.

Learning to model a domain doesn’t start with modelling the user interface.

I’m sure there are more progressive courses out there. I’m just surprised that the take-up of things that are of much benefit to beginning programmers is coming along so slowly. But then we are always making a tradeoff between what we can easily teach and what’s worth teaching. Just look at tester certification.

As I live in Australia, I thought it might be a good idea to allow people to find me on Google using the ‘pages from Australia’ option. To this end, you can now find my website at http://www.software-testing.com.au/ or http://www.quinert.com.au/.

I’ve been sadly busy, finishing a cool project with much learning, and preparing to leave my current employer Revolution IT for Aegeon. This means I get to have another crack at building an army of testing ninjas and sending them out into the world to hopefully make it a better place. Hopefully, that means less blogwebs (the digital form of cobwebs) too.

Ben Kelly, testing martial artist (literally, in this case) and recent blogger threw a question my way which I’ve answered before, but hadn’t really ever had to answer for testers in my team. It went something like this:

Ben: If quality can be described as ‘value to some person (who matters) and testing can be paraphrased as ‘questioning a product in order to evaluate it’, would you describe quality assurance as ‘interpreting the answers to said questions in order to provide information about the quality of said product to someone (who matters)’ ?

I rambled:
Quality assurance should be about ‘Is our process appropriate?’ That is, is it providing the right value to the people that matter.

Ben: Hmm, okay. Quality control is probably more appropriate then.

I rambled some more: I see your point though…One of the ways we assess the success of our development approach is by looking at testing. It’s just not the only way. It would more likely start with questions like:

- ‘What questions didn’t we ask’? That is, what problems escaped into the wild?
- ‘What didn’t we do?’
- ‘What are we doing that is a waste of time, or not helpful?’

Ben: I’m trying to condense the definitions of these terms so that they’re not overly verbose and still retain their meaning – I’m still working on that presentation

Me: So, Quality Assurance is typically assessing the whole software-producing system. Quality control is checking the output of the production process. Quality assurance is checking the process itself. That’s my model, at least.

Ben: That’s inline with what I’m working to. Maybe I don’t need to go into the definitions of these with the new guys right away. Maybe I should let them get their head around the definition of what testing is before I say ‘By the way, testing is not quite the same animal as QA or QC’

Me: It becomes relevant when test groups are called QA and think somehow that they actually do assure quality.
If your team is under that delusion, maybe it’s worth spending some time dispelling it. Testers can provide information about quality. They can’t actually force anyone to do anything with that information.

I want my testers to think:
“I am not assuring quality.”
“I cannot remove bugs.”
“I cannot stop developers creating bugs.”

This is obviously not always true, if we consider our ability to contribute to removal of requirements defects, but that’s not always an area we’re able to contribute to. If we’re not the prime owner of the requirements process, our role is still mostly about pointing out problems and making a compelling case for those problems to be addressed.

It’s potentially problematic for non-testers to think that they have this power though, and it’ s generally bad for the team if testers believe they can do it. Primarily, our role is to provide information to allow others to make good decisions.

Ben: Hmm, I might put it in to my presentation after all.

Me: Yeah, it probably belongs on a ‘What is testing’ slide. It stops your new testers beating themselves up for things that they shouldn’t. It lets your experienced testers defend their practice and not be beaten up by others. Or rather, starts your testers on the path to understanding the scope of their role and communicating that scope to others, setting clearer expectations of what to expect from a tester.

Ben: Something we could stand to do more of here. Excellent. Thanks as always for letting me bother you.

And thanks Ben for bothering me with something that I didn’t realise I needed to. Agile projects change the game a little, but for software, the primary focus of QA is on getting better at producing something that satisfies our customers, through code. Taking on the QA mantle is about telling managers, developers and analysts how to do their job better. And my super powers don’t yet extend that far. I encourage my testers to know their limits too.

After taking the easy route and building some XML check test scripts using Ruby and REXML’s DOM access, I decided that I really didn’t want my computer grinding to a halt for a whole day while it parsed a gig and a half of XML. So it was time to try a streaming parser. Unfortunately, the REXML website seemed to be unavailable. Which led me to this very nice tutorial on Jan Vereecken’s blog:

http://www.janvereecken.com/2007/4/11/event-driven-xml-parser-in-ruby

I’m pretty sure it’s nicer than the one on the REXML site, but I will have to wait and see.

Anyway, thanks Jan!

Matt Heusser has continued discussion of tester-developer relations, where Jonathan Kohl describes the flipside of testers telling developer’s that their automation code sucks.

Actually there are many scenarios, derived from at least a few properties -

- Does the person doing the automation know/not know that their code sucks.
- Is the code ’suckage’ pointed out by a third party, or the person writing it.
- Is the automation being done by a tester or a developer?

You can plot these out as a truth table and see where you fit. I tend to be automating, saying “I would love some help with this”, but can’t get help or chances to pair with developers. I’ve also observed testers wanting to pair with developers, but developers not wanting to come down to the tester’s skill level or work with their tools (as well as the case I mentioned in my previous post).

I’m sure situations also exist where the testers aren’t interested in working with the developers and learning.

So we have a model that contains -

- the feedback
- the source of the feedback
- awareness of the current state
- receptiveness to feedback
- desire or ability of one of the parties to change the situation.

Has anyone experienced any of the other permutations of these properties?

I’ve been building tools for web service testing using Ruby and its SOAP libraries. I hope to write more on this later, but for now, a pointer to a simple problem that took up far too much time.

My test toolkit has three small programs, each providing different services. The first can be passed a list of named test conditions. It queries the database and returns identifiers for data which matches the test condition of interest. This list of identifiers is dumped to a file. The file is passed to the oracle program as input, generating expected results for the items requested. The list of identifiers is also passed to the web service. The output of the oracle and the web service are in the same format, so it’s then a simple case of automatically comparing the two outputs as files, using diff.

I can also use these tools interactively to run ad-hoc queries on the database and web service, so these tools give me a nice interface for exploratory testing, as well as being able to automate and integrate with the build if need be.

The simple batch file to execute all tests looks something like this:

————
‘ 1. Create the list of things to request from the web service
get_test_data_items > datalist.txt

‘ 2. Generate expected results using the list created, and dump the output to a file
generate_expected_results < datalist.txt > expected.txt

‘3. Query web service for the list of items
query_webservice < datalist.txt > actual.txt

‘4. Check that actual matches expected using Unix
diff actual.txt expected.txt
————
Note that the first three commands are ruby scripts, so windows kindly lets me omit the ‘.rb’ extension.

It turns out that this is a bad thing. Letting windows figure out the file association means that the command line fails to send the specified file as standard input to the ruby script.

The telltale error message for this problem is this:

D:/test/query_webservice.rb:32:in `gets’: Bad file descriptor (Errno::EBADF)
from D:/test/query_webservice.rb:32

The simple workaround is to bypass the ruby file association and explicitly invoke ruby:

ruby get_test_data_items.rb > datalist.txt
ruby generate_expected_results.rb < datalist.txt > expected.txt
ruby query_webservice.rb < datalist.txt > actual.txt

Now it all works.

There’s a more detailed description of the problem here: http://mail.python.org/pipermail/python-bugs-list/2004-August/024920.html

As a side note, I’m calling ‘diff’ from the Gnu utilities for Win32 (http://unxutils.sourceforge.net/) package, a collection of unix utilities to make the windows command line a little friendlier. Laziness is what got me into this problem in the first place. In the spirit of laziness, I’ve also installed a bash shell for Windows. By configuring bash to keep a history of the last 5000 commands, I get automatic logging of my test activities as well.

Matt Heusser’s recent blog entry on Tester/Developer communications prompted me to comment on the dreams of agile projects and tester heaven.

Now, I’ve been tempted to have that conversation, but the conversations I have had instead are much more along these lines:

Me: “Hey dev guy, here’s 20 years of accumulated knowledge describing why the testing approach you are about to take is not going to solve your problem, cost a lot of money, and waste a lot of time”.
Developer: “That’s nice, but we’re going to do it anyway”.

Two months later:

Me: “So, how’d that test approach work for you?”
Developer: “Oh, it completely sucked and was unmaintainable”.

Now, these developers are not stupid people, but it’s difficult for me to understand how things get to this point.

Or at least, it was until recently. Most agile development projects are much like Toyotism (or Lean, if you prefer). Better than how things were in the past, but not fundamentally changing the nature of work. Just as work in manufacturing is still hard, repetitive and largely about lines, so we easily fall into our old factory-style patterns. I believe this is at the heart of Brian Marick’s statement on the current state of Agile project experiences: ‘Whereas the number of people new to Agile who describe their project as “the best project I’ve ever worked on” seems to be declining, and we believe work should be joyful.’

Parallel lines of software development, with variable development and testing tasks performed serially end up being just as unpleasant as the old ways. Except now the moment your testing task takes longer than the development task in the queue behind you, someone’s screaming about the tester backlog.

Of course, it doesn’t have to be this way, but there’s very little incentive for the guy with the upstream work to contribute to solving it. They’re not the ones feeling the pressure. And once those pesky testers are gone, the whole thing actually appears to work much more smoothly. After all, what’s smoother than an assembly line with a single step?

Hint: The last line *is* the answer. But the devil is in the details of what comprises a ’step’. And what Matt describes is a step toward the solution as well.

Stay tuned.

Here’s a cool little tester-valuable link courtesy of Mike Kelly’s Rational Functional Tester tutorials:

http://labs.google.com/sets

Generate test data, or ideas if you’re stuck.

James Bach is going to be in Melbourne early June, teaching his Rapid Software Testing course. If you’re in Melbourne, don’t miss out. If you’re not in Melbourne, try to get there anyway!

Details can be found at http://www.softed.com.au/Courses/rst.aspx

The first MAST lunch will be taking place on Tuesday, May 1st, 12.30pm. The venue will be Blue Train Cafe at Southbank (http://www.bluetrain.com.au). If you want to attend, but have not responded, please do so ASAP.