Illiteration - Jared's Testing Times - Software Testing Blog

Running Watir cross-browser using Internet Explorer, Firefox and Celerity

Thu, 2 Jul 2009 19:19:06 +0600

Since reading about Celerity, I've been excited by the idea that I might be able to take my Watir scripts and run them more quickly by using a browser simulator. It also raised the possibility of taking scripts and running them in different environments and on different platforms, because Celerity runs under JRuby without a browser. It makes http requests directly, and simulates javascript behaviours without actually rendering pages.

I had the hope that this would give me an option to get quicker feedback from my automation. Unfortunately, this hasn't turned out to be the case for the applications I'm currently testing. Simulation of the javascript on the page seemed to take longer than the real browser. I got quicker feedback by turning off images, blocking certain external hosts and using Internet Explorer. I still hold out hope that Celerity may improve performance, or that the applications I'm testing will be less javascript intensive.

Once I set to getting my Watir scripts working with Celerity, I needed to overcome a few issues. The last of these was the number of differences between Celerity's behaviour and Watir's behaviour. Resolving this turned out to be reasonably simple, and the same strategy for getting consistent behaviour between Celerity and Watir driving IE turns out to solve some other problems with differences in behaviour in Watir's drivers for IE and Firefox.

My first cut at this is by no means complete, with a bunch of unnecessary duplication that can be easily factored out. For now however, it gets the job done, and may be instructive for others.

The strategy is simple -

- Wrap the browser in your own class.
- Dynamically require the appropriate library for the driver you need
- Create your own methods that provide consistent behaviour for the different browsers. In my case, I needed to be able to get the actual html content of the page after it had been modified by scripts running on the page. Each driver (IE, Firefox, Celerity) required a different method to be called to achieve consistency, but using this approach, I can call browser.html and get (almost) the same result from all browsers and drivers.
- Intercept any calls to methods that you haven't implemented and pass them through to the actual browser object.

This is the first time I've touched Ruby's method_missing method, and it was pretty painless.

There are a few issues getting JRuby up and running with Celerity on windows as well. I'll describe these shortly in another post. I've also recently had issues installing the latest version of Watir, and will detail the solutions to these as well.

Sample code is below. Note that you can actually do without assigning the browser to a global variable, but I provide this example just to show how you can fit this new solution into the 'standard' Watir approach. If you modify things so that you can specify the driver from the command line, you should be able to run tests using something like -

jruby main.rb --driver :celerity
ruby main.rb --driver :watir_ie
ruby main.rb --driver :watir_ff



----- driver.rb ------

require 'singleton'

require 'rubygems'



class Driver

    include Singleton



    def set_driver(driver)

      @driver=driver

      if @driver==:celerity

        require 'celerity'

        @browser=Celerity::Browser.new

      end

      if @driver==:watir_ie

        require 'watir'

        Watir::Browser.default='ie'

        @browser=Watir::Browser.new

      end

      if @driver==:watir_ff

        require 'watir'

        Watir::Browser.default='firefox'

        @browser=Watir::Browser.new

      end

      return @browser

    end



    def html

      return @browser.document.body.innerhtml if @driver==:watir_ie

      return @browser.xml if @driver==:celerity

      return @browser.html if @driver==:watir_ff

    end



    def initialize

      @browser=nil

    end

    

    def method_missing(method, *args)

      @browser.send(method, *args)

    end

end



----- main.rb ----

require 'driver.rb'



driver=:watir_ff

#driver=:watir_ie

#driver=:celerity



Driver.instance.set_driver(driver)

$browser=Driver.instance

$browser.goto("http://www.quinert.com/blog/")

puts $browser.html

Test automation models, dehumanising testers and Agile dysfunctions

Tue, 23 Jun 2009 08:00:14 +0600

This tweet was forwarded to me yesterday:

martinfowler: Manual scripted testing should be a human rights violation

It bothered me on a few levels.

Firstly, the simplicity of phrasing around manual and scripted testing. Secondly, that agile developers might view themselves as the saviour of oppressed testers everywhere, and the perpetuation of the concept of tester as helpless victims. And thirdly, criticism of scripted testing is hardly new, and it's a bit sad that one of the key proponents of test-driven development and design has taken so long to come to this idea.

Still, his comments make a nice soundbite, and what he states is often true, but until he says *why*, I'm not sure if he knows what he's talking about. It just seems a perpetuation of simplistic binary models of automation and prescriptive testing.

We can do better. There's much more we should be talking about when discussing manual testing and the conditions of testing in corporate life.

I start with these ideas -

- Let's put under computer control and evaluation the elements of our tests which are well suited to manipulation and/or evaluation by computers.
- Let's put under human control and judgement the elements of our tests which are well suited to human manipulation and evaluation.
- Let's prefer exploratory approaches when we want to find new things and adapt to the information at hand.
- Let's make sure that we record the details for important things that we don't want to forget to do, for things that are confusing that we can't fix.
- Let's be aware that any test have elements of exploration and prescription.
- Let's be aware that the testing model we work from biases the kinds of problems we'll find and the kinds of things we'll miss.
- Let's choose appropriate levels of abstraction to help manage the cost of our test design.
- Let's have as many ways as possible to model our test designs so that we can effectively solve the problem at hand.

In particular, the first two points suggest different ways of doing things that blur the lines between traditional automated and "manual" tests, as discussed by Kaner, Bach and Kohl:

http://www.kohl.ca/blog/archives/000193.html
http://www.satisfice.com/articles/boost.shtml
http://www.satisfice.com/blog/archives/118

On the dehumanising of work, a lot of testers pulled out of frontline business roles were probably already having their human rights violated for far less money than they now make as testers. That doesn't excuse things, but there aren't as many testers complaining about the status quo as one might expect. My observation is that the nature of testing as practiced in most places drives the majority of those who might want to take a more creative approach elsewhere.

Standard business practice continues to be to throw money at staff retention problems in skilled roles rather than consider how work might be made to suck less. This isn't uniquely a software testing problem, and the world is some way from tackling this as an issue.

Trying out this new-fangled twitter thing

Wed, 17 Jun 2009 11:24:45 +0600

As I seem to be somewhat time poor, I thought I might be able to squeeze in some blogging-light at http://twitter.com/xflibble.

Chinese comment spammers win

Mon, 25 May 2009 18:04:53 +0600

Unfortunately, you won't be able to post a comment until I summon the energy to move my blog to something that has adequate anti-spam controls. Feel free to email me your thoughts if I post anything comment-worthy in the meantime.

150 spam comments in a day is just a bit much.

Tools for simulating page load times and throttling bandwidth

Mon, 4 May 2009 08:08:00 +0600

I needed to do some page-weight testing on a new site, and found these handy plugins at http://www.uselessapplications.com

IEThrottle is the Internet Explorer version, and Firefox Throttle is, unsurprisngly, the Firefox version.

In Internet Explorer, for me at least, the plugin was hidden in the IE 7 tab bar, next to tools:

In Firefox, you'll see it at the bottom of the page:

The Onion predicts the future once again!

Wed, 4 Feb 2009 07:13:41 +0500

The Onion shows off their system-thinking skills once more.

Compare this article on predictive sentence completion with The Onion's Mac Wheel features at around the one minute mark in the video at http://www.theonion.com/content/video/apple_introduces_revolutionary.

Other clairvoyant articles are here and here. Find your own here.

My diabolical problem becomes simply a wicked one

Tue, 28 Oct 2008 10:47:44 +0600

I spent ages recently googling for references to 'Diabolical Problems', about which I thought I had read. Thanks to Matt Heusser's latest post, I now know I should have instead been googling for Wicked Problems. Just in case I've sent anyone off on the same wild goose chase, this post should set them straight. For everyone else, enjoy the reading.

Claims testing in the wild

Fri, 3 Oct 2008 04:04:52 +0600

As yet another poor internet soul is scammed by a man pretending to be a woman in online chat rooms, I'm reminded of the sensibility of my number one internet heuristic.

Jared's first law of online safety is 'Assume that everyone you are talking to online is a man'.

This has held me in good stead over time, but how relevant is this to testing? Well, it's strongly related to the testing technique of claims testing. Claims testing is what you're doing when you are testing a product to specifications or requirements. It is perhaps the most common test approach in large corporate environments I encounter, but it's also a technique that can be applied in a shallow, context-free way.

James Bach has two points in his Heuristic Test Strategy Model that I find key in describing claims testing -

- Verify that each claim about the product is true.
- If you’re testing from an explicit specification, expect it and the product to be brought into alignment.

Perhaps closer to what we are actually doing is that we usually verify that each claim about the product *can* be true, for some particular situation or situations. We also try to understand for which cases and contexts the claim holds true. And in order to verify the claim, we set about collecting *sufficient* evidence of the truth of the claim, because exhaustive proof is either prohibitively expensive or impossible.

An example I like to use to teach how claims testing should work begins with me making the claim 'My name is Jared'. I follow with the question 'How would you set about finding out whether it really is true that my name is Jared?'

The example is not as straightforward as it may seem, because the level of evidence required depends entirely on the purpose for which we need to know.

If I've introduced myself online in a chatroom or in person, and you don't care about any interaction beyond that room, my assertion that my name is Jared may be sufficient.

If you're the government of Australia, then your standards are a little higher depending on the service you're providing to me. Sufficient evidence may include my birth certificate or passport, an assortment of historical information, and other knowledge of the details that the government has stored about 'Jared'.

In other instances, social proof may be OK. The fact that others refer to me as 'Jared' may be sufficient evidence, and we can build up a body of evidence based on a history of social interactions.

Or perhaps you're a hitman paranoid about bumping off the wrong person, so you brute-force the solution. You stalk me for a month and rummage through my mailbox, finally whacking me on the head and rifling through the contents of my house and wallet.

Each one of these approaches may be required for sufficiency, and is appropriate for different contexts and purposes. And we might take similar approaches when we're testing software. We might simply talk to people who can provide evidence of a claim being met. We might perform a simple confirmatory test for a non-critical function. We might spend a longer time collecting a large body evidence to support a claim being met. It's important that we think about the importance of the claim and ensure that our approaches to gathering information can be defended under scrutiny.

The real-world analogy to the second point about claims testing - 'Expect the spec and the product to be brought into alignment' is perhaps more commonly found in legal circles, and occasionally in the forum posts or blog rants of less careful men who feel their masculinity has been threatened. It tends to involve situations like this. And as in a situation detailed here that more closely parallels software testing, remember that refuting a claim can sometimes have much bigger consequences.

Odd Asian fascinations and self-study resources

Thu, 18 Sep 2008 18:22:36 +0600

I don't normally like to mix up entertainment with the software posts, but this one does have a vague relationship to the 'Puppy poo' story. It also invokes the PC Engine game of yore, 'Toilet kids'. And it's a nice interactive flash animation.

Sit back, relax, and enjoy Kkukku and Yaya in
'ppung ppung ppung eungga hagi'.

I found this while following a link to study resources on Korean Class 101. The free podcasts they have are excellent and the forums are helpful. The website family also has other languages in the same conversational style, in case you're looking for something other than Korean. You can find the podcasts via iTunes, or on their website. I've been enjoying their services for free, it's nice to give something back.

Requirements and specifications: What's the difference and what's it to you?

Tue, 16 Sep 2008 18:20:37 +0600

There have been a number of threads I have followed in a few different forums recently where people have discussed requirements, what it means for requirements to be 'good', and what it might mean for requirements to be unambiguous. What usually follows is a long-winded back and forth, with no resolution.

At the heart of this deja vu is the fact that one person's requirements are not necessarily another's. I resolve this by drawing a clear distinction between specifications and requirements. The distinction may be obvious to some, but in practice it seems to be something we struggle with.

In "Software for dependable systems: Sufficient evidence?", Daniel Jackson and his co-authors take care to highlight this issue:

"Software systems that are developed specially for a particular client are typically built to meet preagreed requirements, but these requirements are often a long and undifferentiated list of detailed functions."

"The requirements of a software system should describe the intended effect of the system on its environment and not, more narrowly, the behavior of the system at its interface with the environment, which is the subject of the specification."

They also take care to point out that the environment of the system includes the software product, plus the humans that use them, and other environmental factors external to the software.

The specifications are not about things that anybody *needs*. Specifications represent the end result of negotiations, conversations, politics and expediency that some group of people thinks represents an understanding that is good enough for now. The specification is at best, our best guess of what's going to make the world a better place for the numerous people who have a stake in the thing that we're building. Specifications are a waypoint on the path to something else.

Specifications are never equivalent to requirements in the case of things that will be used by humans.

Specifications apply to the pointy end of a screwdriver that needs to fit into the indented part of some screw.

As testers, testing to specifications is something we do because finding about the requirements is too hard. We do it because that's what the testers before us did. We do it because the process might be built around specifications documents, and that's what managers are tracking to. We might reasonably test to spec if our job is just to test a software component that's on its way to be integrated with something else. However, testing to spec can't tell us that the system is going to yield the desired benefits.

In the case of the screwdriver, requirements apply to the handle - how it fits your hand, and the hands of others. They apply to whether it gives you enough grip and whether it has a switch to make it only screw or unscrew. They apply to whether it fits enough hands in the world, and whether it can be built to a price that someone is willing to pay. Requirements apply to whether the pointy end of the screwdriver will make do for unscrewing (or screwing in) a screw that is not of the size covered by the specification for the pointy end of our screwdriver.

Requirements are about utility and real human problems, and are fuzzy, and messy, and never fully understood. When our stakeholders ask us questions about testing, though they often don't phrase it this way, they are usually interested in information with respect to requirements, both explicit and implicit.

Test to see how the product measures up to requirements, and to learn about what the requirements are. That's the value that you can bring to the project.