Tim seems to be pretty busy now that he’s a free agent, so I’m looking forward to trying this new(ish) gem he announced recently for querying browser performance stats.

I can’t believe it’s been as long as it has since I last posted.  This has mostly been because I’ve been busy preparing to become a dad.  I’m a dad as of two weeks ago, and have a new job, so as things settle down I hope to get through my backlog of posts, mostly on technical topics and tools.

Alister Scott has posted some code examples on watirmelon.com that show different solutions to a problem that arises for people implementing the Page Object pattern for test automation.

I found Alister’s approach solutions interesting, because as my automation usually models business/user goals and/or domain features/concepts,  I don’t often have my own page abstraction.  See http://gojko.net/2009/10/06/putting-selenium-in-the-right-place/ for an example of the approach I’m thinking of.

At the top level might be something like:


google=Google.new
google.search(search_phrase)


At the next level down, I potentially want to be independent of the user interface, so it doesn’t make sense to organise around page components.  I might have something like:


def search(search_phrase)
enter_search(search_phrase)
submit_search
....
end


When we get to the third (task) level, this is still at a user or business activity level, so the page model still doesn’t make sense for me.  I’ll just use the browser driver directly (or my own driver abstraction):


def submit_search
Driver.button(:value, => "Google Search").click
end


Yes, I could have another level here such as:


def submit_search
GoogleHomepage.search_button.click
end


In most cases though, the task-level UI element has a single purpose, so there’s not much of an issue with violating DRY. I also try to keep the automation stack as small as possible, so can tolerate a little bit of duplication. Regardless, it’s easily factored out into a new method if duplication becomes an issue because methods are common to the domain.

I’ve seen some frameworks that have a really abstract UI driver at the next level (so that they could drive a rich client app, a mobile app and a web app all using the same methods), but I’ve never needed to do that.

I favour a business/domain oriented approach because at the business level, the goals and activities don’t change that much (and I tend to let the automation lag behind my understanding of the problem).  It also encourages me to pay attention to the things that matter and be less implementation-focused.

I’m hacking away at a tool to give me better JMeter results, and I needed to see progress as I iterated through the data I’d imported into MySQL.  The progress bar described in my earlier post can be made to work with a simple addition:

class Sequel::Dataset
  alias :size :count
end

This adds the necessary ’size’ method to Sequel’s dataset and everything’s happy.

I’ve been having a lot of rants about Cucumber of late, as it’s the new shiny thing for agile teams.  Does anyone else have issues with it?  I’ve asked all of my programmer friends to convince me of its worth, and they’ve all failed so far.  I’ve not seen it adding any value above building a good API (and I see it bringing a lot of negatives relative to other possible approaches).

In my experience, I’m seeing -

- Customers/non-programmers never write the tests (because they have very little interest in specifying everything in given-when-then.  They just want to tell us what they want.  And it doesn’t make much sense to specify everything in that format anyway).
- Customers/non-programmers write the tests but it focuses the test effort on writing those kinds of tests, rather than other testing that seems to add more value.
- The tests are written in English, but what the test actually does depends on how the developers convert the english phrases into code (so there’s no guarantee it tests what the customer intended anyway).
- Avoidance of conversation (ie. tests as contracts).
- Cucumber and the related tools (through the toy examples they provide) encourage developers to put lots of implementation detail into the tests (and sometimes to do a lot more testing through the GUI/http layer rather than pushing some of that testing down).
- Refactoring sucks as we lose IDE support.
- Much heavier test artefacts (when a lot of the teams I work with are already struggling with the weight of their agile test automation).
- A continued focus of tests on what the system does and not why it does it and the business outcome.
- Anecdotally, I’m not seeing better outcomes than people were getting with other approaches.  I realise it may be leading to better designs, but then I’d expect to see improvements somewhere in the process.  That looks like it would require better modelling skills than I’m seeing in most of the cucumber tests on my projects.

Most of the examples that I’ve seen where people are claiming success are fairly small applications.  I’m not seeing the approach scale that well.  Yes, most teams could do write their cucumber tests better, but even then, in my experience, other approaches would be more effective and more efficient.

Any thoughts?  If there’s interest, I’ll try and post some examples of what I think those tests might look like if we pushed them into other forms.

As much as I love using Ruby, it’s become more frustrating over time.  Developers are feeling this, and as a tester, I’ve had my share as well.  This article, passed on by James Ladd sums things up for me (for testing, as well as development).

The best example of this in the Watir space is probably here – http://wiki.openqa.org/display/WTR/Pop+Ups.  I recently lost half a day because apparently Watir’s ‘click_no_wait’ doesn’t work in the release version of the 1.8.6 Windows one-click installer.  Of course, you have to use the beta release that came out after the 1.8.6 release, which is a bit trickier to find now that the one-click installer is not really supported (there’s a new Windows installer).

Let’s hope things start to stabilise again soon.

A good testing habit when working with web apps is to monitor the log files of servers as you test.  In some cases this is easy, especially where there’s a single application server.  With the trend toward more service-oriented architectures, and server clusters for high-traffic applications, the environments I’m working in tend to have many log files spread over multiple servers and folders.  My old friend, ‘tail -f error.log’ becomes a bit more difficult.

I’d heard about Ruby’s SSH library quite a while back, but only recently have my spare time and my memory conspired to work together. The library lets you connect to a server via SSH, execute commands and see the result. So today I quickly hacked out a script to let me watch multiple (Linux) log files at the same time.  This might work for other environments that support SSH, but I only have it connecting to Linux servers at this time.

Code is below, very hacky, and tailored to my needs.  If you have any questions, leave a comment, but hopefully this helps someone get started!

To install, use the instructions for NET::SSH version one at http://net-ssh.rubyforge.org/ssh/v1/index.html. This will actually install version two though, so you’ll need the docs and examples from http://net-ssh.rubyforge.org/sftp/v2/api/index.html

require 'net/ssh'

class Logfile
  def initialize(name,server,filename)
    @name=name
    @server=server
    @filename=filename
    @current_file_size=0
    @ssh_server='server'
    @username='username'
    @password={:password=>'password'}
  end

  #You'll need to customise this method so that it generates the
  #full path of the log file you're interested in.  This one is for Splunk logs.

  def full_log_path
    today=Time.now
    day=today.day.to_s.rjust(2,'0')
    month=today.month.to_s.rjust(2,'0')
    year=today.year.to_s
    todays_log_folder="#{year}-#{month}-#{day}"
    full_path="/var/log/#{@server}/#{todays_log_folder}/#{@filename}"
    return full_path
  end

  def get_new_lines
    Net::SSH.start(@log_server,@username,@password) do |session|
      new_file_size=session.exec!("wc -l #{full_log_path}").split(" ")[0].to_i
      lines_to_get=new_file_size-@current_file_size

      # Don't generally need to get everything if the error logs are being flooded
      lines_to_get=100 if lines_to_get > 1000 

      new_logs=session.exec!("tail -n #{lines_to_get.to_s} #{full_log_path}")
      @current_file_size=new_file_size
      return new_logs
    end
  end

end

logfiles=[]
logfiles.push Logfile.new(:log_name1,"log_app_folder1","log_file_name1")
logfiles.push Logfile.new(:log_name2,"log_app_folder2","log_file_name2")

5.times do
  logfiles.each do | logfile |
    changes=logfile.get_new_lines
    puts changes if !changes.nil?
    $stdout.flush
  end
  sleep 10
end

When I’m running tests and tools, I frequently want to know how far through I am. I’d built a class to help with this a while back, but Corey Goldberg’s recent post on a python version prompted me to post this Ruby version.

First, you’ll need to save this to progress_bar.rb:

class Progress_bar

  attr_accessor :items_to_do, :items_done

  def initialize(items_to_do, items_done=0)
    reset(items_to_do, items_done)
  end

  def percent_complete
    return (@items_complete*1.0/@items_to_do*1.0)*100
  end

  def advance(steps_to_advance=1)
    @items_complete+=steps_to_advance
  end

  def reset(items_to_do, items_done=0)
    @items_to_do=items_to_do
    @items_complete=items_done
  end

  def report
    $stderr.print "\r#{progress_bar} #{@items_complete} of #{@items_to_do} done"
  end

  def progress_bar
    complete_bar=(percent_complete/2.0).floor
    incomplete_bar=((100-percent_complete)/2.0).ceil
    return "[#{"*"*complete_bar}#{"-"*incomplete_bar}]"
  end
end

You can then mix this in to so that when you iterate through collections, you can report on your progress. Handy for data-driven tests:

require 'progress_bar.rb'

module Enumerable
  def each_with_progress
    progress=Progress_bar.new(self.size)

    self.each do |item |
      progress.advance
      yield item
      progress.report
    end
  end
end

# Examples:

b={1=>"a",2=>"b"}

puts "Iterating through hash"
b.each_with_progress do | val |
  # do stuff
  sleep 2
end

puts
puts "Iterating through array"
c=[1,2,3,4]
c.each_with_progress do | val |
  # do stuff
  sleep 2
end

puts "Manual control..."
pb=Progress_bar.new(40,20)
5.times do
  pb.advance
  pb.report
  sleep 1
end

5.times do
  pb.advance(2)
  sleep 1
  pb.report
end

If you run from the command line, your output will look something like this:

[*******************************************-------] 35 of 40 done

This should all appear on a single line, although anything sent to standard output will force this to go to a new line. I never managed to get ANSI terminal display libraries working.

Enjoy!

After taking the easy route and building some XML check test scripts using Ruby and REXML’s DOM access, I decided that I really didn’t want my computer grinding to a halt for a whole day while it parsed a gig and a half of XML. So it was time to try a streaming parser. Unfortunately, the REXML website seemed to be unavailable. Which led me to this very nice tutorial on Jan Vereecken’s blog:

http://www.janvereecken.com/2007/4/11/event-driven-xml-parser-in-ruby

I’m pretty sure it’s nicer than the one on the REXML site, but I will have to wait and see.

Anyway, thanks Jan!

I’ve been building tools for web service testing using Ruby and its SOAP libraries. I hope to write more on this later, but for now, a pointer to a simple problem that took up far too much time.

My test toolkit has three small programs, each providing different services. The first can be passed a list of named test conditions. It queries the database and returns identifiers for data which matches the test condition of interest. This list of identifiers is dumped to a file. The file is passed to the oracle program as input, generating expected results for the items requested. The list of identifiers is also passed to the web service. The output of the oracle and the web service are in the same format, so it’s then a simple case of automatically comparing the two outputs as files, using diff.

I can also use these tools interactively to run ad-hoc queries on the database and web service, so these tools give me a nice interface for exploratory testing, as well as being able to automate and integrate with the build if need be.

The simple batch file to execute all tests looks something like this:

————
‘ 1. Create the list of things to request from the web service
get_test_data_items > datalist.txt

‘ 2. Generate expected results using the list created, and dump the output to a file
generate_expected_results < datalist.txt > expected.txt

‘3. Query web service for the list of items
query_webservice < datalist.txt > actual.txt

‘4. Check that actual matches expected using Unix
diff actual.txt expected.txt
————
Note that the first three commands are ruby scripts, so windows kindly lets me omit the ‘.rb’ extension.

It turns out that this is a bad thing. Letting windows figure out the file association means that the command line fails to send the specified file as standard input to the ruby script.

The telltale error message for this problem is this:

D:/test/query_webservice.rb:32:in `gets’: Bad file descriptor (Errno::EBADF)
from D:/test/query_webservice.rb:32

The simple workaround is to bypass the ruby file association and explicitly invoke ruby:

ruby get_test_data_items.rb > datalist.txt
ruby generate_expected_results.rb < datalist.txt > expected.txt
ruby query_webservice.rb < datalist.txt > actual.txt

Now it all works.

There’s a more detailed description of the problem here: http://mail.python.org/pipermail/python-bugs-list/2004-August/024920.html

As a side note, I’m calling ‘diff’ from the Gnu utilities for Win32 (http://unxutils.sourceforge.net/) package, a collection of unix utilities to make the windows command line a little friendlier. Laziness is what got me into this problem in the first place. In the spirit of laziness, I’ve also installed a bash shell for Windows. By configuring bash to keep a history of the last 5000 commands, I get automatic logging of my test activities as well.

Someone recently suggested to me that the selection of VBScript for an automation language is because it’s easier for testers. This is slightly rant-y, but I promise there are a few helpful tips in here just to make up for a mild dose of language-warring.

Exhibit A.

I can understand that there’s a chance that a tester (if they came from a business role) might have some familiarity with VBScript as a result of writing office macros, but really…

Which would you rather write (and read)?

Dim newList
newListNextItem = 0

For i=1 to ubound(list)
    Redim newList(newListNextItem)
    newList(newListSize) = list(i).getROProperty(“text”)
    newListNextItem = newListNextItem+1
Next i

OR

list.each do | item |
  newList.push item.text
end

Of course, you can (and probably should) implement your own class to do something like the above. Below is how things might look if do.

Set newList = new dynamicList
while list.hasNextItem do
  newList.add(list.nextItem)
end while

You will, however, quickly encounter the issue detailed in exhibit C.

Exhibit B.

Is it easier to have to change the way that you assign a variable based upon the type of the thing being assigned?


a="Fred"
Set b=New Fred
c=a
Set c=b


OR


a="Fred"
b=Fred.new
c=a
c=b


The net effect of this is to have functions which need to be aware of exactly what kind of thing they are returning, then have them do this:

If isObject(a) then
   Set functionReturnValue = a
Else
   FunctionReturnValue = a
End if

Exhibit C.

We include a reference to file B from file A

File A contains:


Set a = New a


File B contains:


Class A
End Class


For some reason this doesn’t work, so just to check, we try a different method to include our Class definition. Now we are told that the class already exists.

But wait, if you somehow thought to create a special factory function to give you an instance of the class, it all works.

So now we change File B to:

Function newA
   Set newA = New A
End Function

Class A
End Class

Now why didn’t I think of that?

Rant off. I’ll be working with different languages and tools for a while now…But do we really have to suffer this as testers? I’ve never spent so much time in a debugger, ever. Productivity tool? I leave that to you to decide.

And in case you didn’t read this the first time around, discover the joys of Visual Basic over here:

http://adamv.com/dev/articles/hatevbs/vbscript
Terry Horwath has also done some further investigation of the namespace issue (Exhibit C) over at SQA Forums.