{"id":480,"date":"2010-08-16T11:35:15","date_gmt":"2010-08-16T01:35:15","guid":{"rendered":"\/blog\/?p=480"},"modified":"2011-07-05T11:55:17","modified_gmt":"2011-07-05T01:55:17","slug":"automating-flash-ajax-popups-and-more-using-ruby-watir-and-sikuli","status":"publish","type":"post","link":"http:\/\/www.software-testing.com.au\/blog\/2010\/08\/16\/automating-flash-ajax-popups-and-more-using-ruby-watir-and-sikuli\/","title":{"rendered":"Automating Flash, AJAX, Popups and more using Ruby, Watir and Sikuli"},"content":{"rendered":"

Jonathan Kohl<\/a> pointed me at Sikuli<\/a>, a Python-based tool for automating applications using image recognition.\u00a0 Unlike most tools, which attempt to identify objects via public APIs, Sikuli looks at the pixels on the screen and attempts to identify objects based on how they look.<\/p>\n

This isn’t exactly a new approach, as commercial tools have had this feature for a long time as a means of creating custom objects.\u00a0 So after playing with Sikuli, I wondered whether I could take advantage of it as a library to augment my Watir scripts in Ruby.<\/p>\n

It turns out, the answer is ‘yes’, with a caveat.\u00a0 You need to use JRuby (although you could probably do it in Ruby using the Ruby-Java bridge – It just looked a lot harder), and you also need to use Watir-Webdriver, a new implementation of Watir’s API which is used in a number of other automation frameworks.<\/p>\n

Below is a simple example script and instructions to get you started.\u00a0 It navigates to a website, then clicks on the flash control there. I’ve so far only tested this on Windows.\u00a0 It should work on OSX and Linux, but perhaps not quite so easily (I’m waiting on some feedback).\u00a0 Check the instructions for ‘Install OpenCV’ at the page with instructions for calling Sikuli from other tools<\/a>.<\/p>\n

Sikuli can be used to automate Flash components, any challenging AJAX elements of your web application, to dismiss pop-ups or probably even to inspect visual elements of the page (though I’d want to do this minimally).\u00a0 It’s a little slow, but an interesting and immediately useful add-on to Watir or your favourite java-based testing tool.<\/p>\n


\n#Install Java, or install the JRuby\/JRE bundle at the next step
\n#Install JRuby 1.5.1 - http:\/\/jruby.org\/download
\n#Install Sikuli - http:\/\/sikuli.org\/download.shtml
\n#Install watir-webdriver (eg. jgem install watir-webdriver)
\n#Update ssh - jgem install jruby-openssl
\n#Copy sikuli-script.jar to \\jruby-1.5.1\\lib
\n#Get the test image
\n#Download http:\/\/www.software-testing.com.au\/images\/flashmap_middleeast.png and put it in the image folder as below
\n#See http:\/\/sikuli.org\/trac\/wiki\/How to use Sikuli Script in your JAVA programs for examples<\/code><\/p>\n

require ‘rubygems’
\nrequire ‘watir-webdriver’
\nrequire ‘java’<\/p>\n

java_import “org.sikuli.script.SikuliScript”
\njava_import “org.sikuli.script.Region”
\njava_import “org.sikuli.script.Screen”<\/p>\n

start_page=’http:\/\/www.lonelyplanet.com’
\nimage_folder=”c:\/sikuli_icons\/”<\/p>\n

$screen=Screen.new
\n$browser=Watir::Browser.new :ie
\n$browser.goto start_page<\/p>\n

$screen.click(“#{image_folder}flashmap_middleeast.png”,0)<\/p>\n

http:\/\/sikuli.org\/download.shtml<\/div>\n","protected":false},"excerpt":{"rendered":"

Jonathan Kohl pointed me at Sikuli, a Python-based tool for automating applications using image recognition.\u00a0 Unlike most tools, which attempt to identify objects via public APIs, Sikuli looks at the pixels on the screen and attempts to identify objects based on how they look. This isn’t exactly a new approach, as commercial tools have had […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[92,34,36,90],"tags":[98,96,99,100,95,97,139],"class_list":["post-480","post","type-post","status-publish","format-standard","hentry","category-technical-testing","category-test-automation","category-test-tools","category-tool-assisted-testing","tag-ajax","tag-flash","tag-pop-ups","tag-popups","tag-sikuli","tag-silverlight","tag-watir"],"_links":{"self":[{"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/posts\/480","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/comments?post=480"}],"version-history":[{"count":8,"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/posts\/480\/revisions"}],"predecessor-version":[{"id":602,"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/posts\/480\/revisions\/602"}],"wp:attachment":[{"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/media?parent=480"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/categories?post=480"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/tags?post=480"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}