Paul Szymkowiak and I will be running a one-day workshop on tools for testers, as part of the STANZ conference. The current run on August 13th is full, but you can register your interest for future courses via SoftEd, or drop me a line.

Details of the course are at http://www.softed.com/stanz/speakersandsessions.htm#tst

I’ll have the presentation slides here on the website shortly!

Alister Scott, an Australian tester located in Brisbane has started a blog. There are a few Watir samples, and it’s always nice when a test automator puts their code up for scrutiny. I know my plans to do the same have taken far too long.

Check his writing out at http://watirmelon.wordpress.com/

In response to Matt’s comment on the robustness of my Tiny MCE Watir solution, I’d like to point out that the main threat to the solution’s value for me is that I can no longer run the automation in the background. Send_keys seems to need the IE window to be the topmost window. A lot of my Watir use is for tool-assisted testing rather than complete automation, so being able to leave a script running, go and do other things, then come back when the script has finished is a bit of a problem for me.

Anyway, I’m sure if you are running scripts as part of a build, or have a separate machine that handles all of your test execution, this is going to be fine for you. For me, it’s not quite what I needed. Just remember, quality equals value to some person…

I’ve spent far too much of the last day and a bit trying to get Watir to identify and manipulate the TinyMCE rich text edit control. Hopefully the title of this post will allow anyone experiencing similar problems to Google my solution!

TinyMCE does a lot of javascript nastiness (from Watir’s perspective), turning a text area HTML tag into a funky iframe WYSIWYG text editor. Watir seems to have trouble doing much with it, but I was finally able to get an acceptable (for now) solution:


#Substitute your TinyMCE editor instance for 'mce_editor_0'. It will probably usually be OK though.

$ie.ie.Document.parentWindow.execScript("tf=document.
getElementById(\"mce_editor_0\");tf.contentWindow.focus()")
$ie.send_keys("Hello there")


It’s not nice, but it sure beats yesterday’s solution:

20.times{$ie.send_keys("{TAB}")}
$ie.send_keys("Hello there")


After taking the easy route and building some XML check test scripts using Ruby and REXML’s DOM access, I decided that I really didn’t want my computer grinding to a halt for a whole day while it parsed a gig and a half of XML. So it was time to try a streaming parser. Unfortunately, the REXML website seemed to be unavailable. Which led me to this very nice tutorial on Jan Vereecken’s blog:

http://www.janvereecken.com/2007/4/11/event-driven-xml-parser-in-ruby

I’m pretty sure it’s nicer than the one on the REXML site, but I will have to wait and see.

Anyway, thanks Jan!

I’ve been building tools for web service testing using Ruby and its SOAP libraries. I hope to write more on this later, but for now, a pointer to a simple problem that took up far too much time.

My test toolkit has three small programs, each providing different services. The first can be passed a list of named test conditions. It queries the database and returns identifiers for data which matches the test condition of interest. This list of identifiers is dumped to a file. The file is passed to the oracle program as input, generating expected results for the items requested. The list of identifiers is also passed to the web service. The output of the oracle and the web service are in the same format, so it’s then a simple case of automatically comparing the two outputs as files, using diff.

I can also use these tools interactively to run ad-hoc queries on the database and web service, so these tools give me a nice interface for exploratory testing, as well as being able to automate and integrate with the build if need be.

The simple batch file to execute all tests looks something like this:

————
‘ 1. Create the list of things to request from the web service
get_test_data_items > datalist.txt

‘ 2. Generate expected results using the list created, and dump the output to a file
generate_expected_results < datalist.txt > expected.txt

‘3. Query web service for the list of items
query_webservice < datalist.txt > actual.txt

‘4. Check that actual matches expected using Unix
diff actual.txt expected.txt
————
Note that the first three commands are ruby scripts, so windows kindly lets me omit the ‘.rb’ extension.

It turns out that this is a bad thing. Letting windows figure out the file association means that the command line fails to send the specified file as standard input to the ruby script.

The telltale error message for this problem is this:

D:/test/query_webservice.rb:32:in `gets’: Bad file descriptor (Errno::EBADF)
from D:/test/query_webservice.rb:32

The simple workaround is to bypass the ruby file association and explicitly invoke ruby:

ruby get_test_data_items.rb > datalist.txt
ruby generate_expected_results.rb < datalist.txt > expected.txt
ruby query_webservice.rb < datalist.txt > actual.txt

Now it all works.

There’s a more detailed description of the problem here: http://mail.python.org/pipermail/python-bugs-list/2004-August/024920.html

As a side note, I’m calling ‘diff’ from the Gnu utilities for Win32 (http://unxutils.sourceforge.net/) package, a collection of unix utilities to make the windows command line a little friendlier. Laziness is what got me into this problem in the first place. In the spirit of laziness, I’ve also installed a bash shell for Windows. By configuring bash to keep a history of the last 5000 commands, I get automatic logging of my test activities as well.

I thought I was done with ranting about automation tools for a while, but I couldn’t resist this quote from my former boss’ blog:

“Tools that let programmers create software by manipulating icons and graphics shapes on screen have a long and sometimes successful history… But these have generally served as layers of shortcuts on top of the same old text0based code, and sooner or later, to fix any really hard problems, the programmer would end up elbow-deep in that code anyway.”

There’s more good software-development food for thought in his full post, so why not check it out here?

Someone recently suggested to me that the selection of VBScript for an automation language is because it’s easier for testers. This is slightly rant-y, but I promise there are a few helpful tips in here just to make up for a mild dose of language-warring.

Exhibit A.

I can understand that there’s a chance that a tester (if they came from a business role) might have some familiarity with VBScript as a result of writing office macros, but really…

Which would you rather write (and read)?

Dim newList
newListNextItem = 0

For i=1 to ubound(list)
    Redim newList(newListNextItem)
    newList(newListSize) = list(i).getROProperty(“text”)
    newListNextItem = newListNextItem+1
Next i

OR

list.each do | item |
  newList.push item.text
end

Of course, you can (and probably should) implement your own class to do something like the above. Below is how things might look if do.

Set newList = new dynamicList
while list.hasNextItem do
  newList.add(list.nextItem)
end while

You will, however, quickly encounter the issue detailed in exhibit C.

Exhibit B.

Is it easier to have to change the way that you assign a variable based upon the type of the thing being assigned?


a="Fred"
Set b=New Fred
c=a
Set c=b


OR


a="Fred"
b=Fred.new
c=a
c=b


The net effect of this is to have functions which need to be aware of exactly what kind of thing they are returning, then have them do this:

If isObject(a) then
   Set functionReturnValue = a
Else
   FunctionReturnValue = a
End if

Exhibit C.

We include a reference to file B from file A

File A contains:


Set a = New a


File B contains:


Class A
End Class


For some reason this doesn’t work, so just to check, we try a different method to include our Class definition. Now we are told that the class already exists.

But wait, if you somehow thought to create a special factory function to give you an instance of the class, it all works.

So now we change File B to:

Function newA
   Set newA = New A
End Function

Class A
End Class

Now why didn’t I think of that?

Rant off. I’ll be working with different languages and tools for a while now…But do we really have to suffer this as testers? I’ve never spent so much time in a debugger, ever. Productivity tool? I leave that to you to decide.

And in case you didn’t read this the first time around, discover the joys of Visual Basic over here:

http://adamv.com/dev/articles/hatevbs/vbscript
Terry Horwath has also done some further investigation of the namespace issue (Exhibit C) over at SQA Forums.

I have been riding the ups and downs as I transfer my previous test automation framework learnings to one of the big vendor automation tools. I’d resisted criticism, but today I have to say, loudly, it sucks.

Rather than waste my energy, I simply direct you to read this (http://adamv.com/dev/articles/hatevbs/) (make sure you follow the link at the end). It refers to ASP, but I found almost all of this applies.

The joy of unproductivity tools…

How to implement a company-wide test automation framework effort and fail (usually).

1. Choose your test automation tools without ever thinking about what problem you are trying to solve, or what tests are important to automate.
2. Choose the tool that will look best on your resume. Refuse to acknowledge the test results from any tool which would not increase your market value.
3. Have non-testers and people with no automation skill choose a tool and mandate its use for the entire organisation, without regard for the differing needs of diverse teams.
4. Have your test automators duplicate all of the automation that your development team is already performing in another tool.
5. Hire in skilled test automators to build automation suites, then have them hand over all of their automation code to a test team with no automation skills.
6. Ensure your skilled automators never get to work with each other, and build their frameworks in totally different ways.
7. Ensure that all of your test automators (who are unaware of each other) write tests to cover the functions that other automators are working on.
8. Make sure that there is no easily accessible central place for automation code to live.
9. Avoid creating and communicating standards for versioning, coding and merging of changes to shared automation code.
10. Avoid talking about test strategy. Ever.
11. Only execute tests that your tool can easily perform.
12. Have your regular test team automate every test they can think of.