In my experience, I’m seeing -

- Customers/non-programmers never write the tests (because they have very little interest in specifying everything in given-when-then.  They just want to tell us what they want.  And it doesn’t make much sense to specify everything in that format anyway).
- Customers/non-programmers write the tests but it focuses the test effort on writing those kinds of tests, rather than other testing that seems to add more value.
- The tests are written in English, but what the test actually does depends on how the developers convert the english phrases into code (so there’s no guarantee it tests what the customer intended anyway).
- Avoidance of conversation (ie. tests as contracts).
- Cucumber and the related tools (through the toy examples they provide) encourage developers to put lots of implementation detail into the tests (and sometimes to do a lot more testing through the GUI/http layer rather than pushing some of that testing down).
- Refactoring sucks as we lose IDE support.
- Much heavier test artefacts (when a lot of the teams I work with are already struggling with the weight of their agile test automation).
- A continued focus of tests on what the system does and not why it does it and the business outcome.
- Anecdotally, I’m not seeing better outcomes than people were getting with other approaches.  I realise it may be leading to better designs, but then I’d expect to see improvements somewhere in the process.  That looks like it would require better modelling skills than I’m seeing in most of the cucumber tests on my projects.

Most of the examples that I’ve seen where people are claiming success are fairly small applications.  I’m not seeing the approach scale that well.  Yes, most teams could do write their cucumber tests better, but even then, in my experience, other approaches would be more effective and more efficient.

Any thoughts?  If there’s interest, I’ll try and post some examples of what I think those tests might look like if we pushed them into other forms.

The first is Webmetrics‘ Global Watch Script recorder, available at http://www.webmetrics.com/products/script_recorder.html

The second, is CubicTest.  This uses SeleniumIDE to record the test, but can generate a script for Watir.

(Update) A third is the Testwise Watir Recorder, a Firefox plugin which I looked at a while back, but am unsure of it’s current state.

These will only get you so far, and have their own limitations, the most obvious of which is some corporate applications which only work in Internet Explorer. If a recorder is what you need though, there are a few options and these may get you further.  At any rate, hopefully they’re now easier for people to find.

This isn’t exactly a new approach, as commercial tools have had this feature for a long time as a means of creating custom objects.  So after playing with Sikuli, I wondered whether I could take advantage of it as a library to augment my Watir scripts in Ruby.

It turns out, the answer is ‘yes’, with a caveat.  You need to use JRuby (although you could probably do it in Ruby using the Ruby-Java bridge – It just looked a lot harder), and you also need to use Watir-Webdriver, a new implementation of Watir’s API which is used in a number of other automation frameworks.

Below is a simple example script and instructions to get you started.  It navigates to a website, then clicks on the flash control there. I’ve so far only tested this on Windows.  It should work on OSX and Linux, but perhaps not quite so easily (I’m waiting on some feedback).  Check the instructions for ‘Install OpenCV’ at the page with instructions for calling Sikuli from other tools.

Sikuli can be used to automate Flash components, any challenging AJAX elements of your web application, to dismiss pop-ups or probably even to inspect visual elements of the page (though I’d want to do this minimally).  It’s a little slow, but an interesting and immediately useful add-on to Watir or your favourite java-based testing tool.


#Install Java, or install the JRuby/JRE bundle at the next step
#Install JRuby 1.5.1 - http://jruby.org/download
#Install Sikuli - http://groups.csail.mit.edu/uid/sikuli/download.shtml
#Install watir-webdriver (eg. jgem install watir-webdriver)
#Update ssh - jgem install jruby-openssl
#Copy sikuli-script.jar to \jruby-1.5.1\lib
#Get the test image
#Download http://www.software-testing.com.au/images/flashmap_middleeast.png and put it in the image folder as below
#See http://sikuli.org/trac/wiki/How to use Sikuli Script in your JAVA programs for examples

require ‘rubygems’
require ‘watir-webdriver’
require ‘java’

java_import “edu.mit.csail.uid.SikuliScript”
java_import “edu.mit.csail.uid.Region”
java_import “edu.mit.csail.uid.Screen”

start_page=’http://www.lonelyplanet.com’
image_folder=”c:/sikuli_icons/”

$screen=Screen.new
$browser=Watir::Browser.new :ie
$browser.goto start_page

$screen.click(“#{image_folder}flashmap_middleeast.png”,0)

Power off your system or server while in the middle of testing and look for problems.  You’ll most frequently find issues if you aim for a power outage -

- in the middle of a network communication (eg. client-server apps or network games).

- in the middle of a multi-step transaction.

- in the middle of disk operations.

The last one’s pretty brutal, but sometimes you need to know your application can survive it gracefully.  Have fun!

The best example of this in the Watir space is probably here – http://wiki.openqa.org/display/WTR/Pop+Ups.  I recently lost half a day because apparently Watir’s ‘click_no_wait’ doesn’t work in the release version of the 1.8.6 Windows one-click installer.  Of course, you have to use the beta release that came out after the 1.8.6 release, which is a bit trickier to find now that the one-click installer is not really supported (there’s a new Windows installer).

Let’s hope things start to stabilise again soon.

Though not supported out of the box, you can download the tool that makes it happen at Microsoft’s sysinternals website.

I found it via this site.

Once you’ve downloaded it, add it to your path and then simply:

junction <linkname> <existing directory>

So if you had a folder c:\Program Files\Apache Software Foundation\tomcat and you wanted to reference it as c:\tomcat, you would run:

junction c:\tomcat c:\Program Files\Apache Software Foundation\tomcat

Windows 7 and Vista apparently support this via the ‘mklink’ command.

Safe from hackers, safe from users.  Movable type takes Captcha to the next level!

However, before it could work, I had to figure out that the AutoIT dll wasn’t correctly registered.  If you try the solution above and find it to be behaving mysteriously, then try the steps on Watir.com’s FAQ page.

A victory for common sense…

Nothing against QTP, but if you’re testing web applications, you should probably be looking at other tools.