A colleague of mine, Kyle Sun, spent some time implementing a Watir and web-driver based performance monitoring system. In this post I thought I would share some of the generic results on Waitr.
The problem we were trying to solve was that we needed to monitor load performance of actions on our site. These actions included routines such as loading the home page, logging in, loading a specific page of the site, etc. The real crux of the issue was that we had to perform these actions behind a VPN and also while the site was under a tight lock and key. Thus public solutions such as Keynote and others were expensive and would not work.
We turned to Waitr (http://www.watir.com) which is an automation test suite built in Ruby. It allows you to simulate actions through a browser – exactly as if a user was performing them.
Out of the box the installation was fairly straight forward. Download the package, ensure Ruby is installed, install Watir as a ruby gem. In fact the command we used looked something like:
gem install watir --no-rdoc --no-ri
Although we ran into no issues, I would assume if you did run into issues during the install phase, then most likely it would relate to Ruby. Or more specifically, issues regarding ruby versions and or specific gems that might clash with Watir. I can’t provide any data – but If you run into issues I would troubleshoot those paths first.
As of this writing the Watir installation document can be found here: http://watir.com/installation/
Once you are installed you are ready to make your test script. Those instructions can be found here: http://watir.com/examples/
In a nutshell, we made a script that looks roughly like the following:
Load up Ruby and Watir
require 'rubygems' require 'watir'
Create a browser object
browser = Watir::Browser.new
Next – you basically write a script that dictates to your browser object what you want it to do. In our case we wanted to first test out how long it would take to load a generic URL.
From here you can do all sorts of things, enter text in a text box, load different pages, submit forms, etc… The Watir community has many great examples and docs and I don’t dare want to try to encapsulate that work here.
Now is the fun part – using this for performance monitoring. This was done simply by adding timestamps throughout our scripts. For example – after we loaded the browser we started a timer. We would then log checkpoints as we moved throughout our script to dictate how much time each step took. At the end we simply logged the data to a database and graphed the result.
If you choose this route there are a few recommendations we offer.
First – make sure to start the timers after the browser object is created. The time spent before and during the browser creation is really not related to your site. That is similar to the user turning the computer on and loading FireFox or IE.
Second – clearly define your tests in terms of caching. For example: if you want to test the page load of your site on a clear cache – then start the timer immediately after the browser is created. If you want to simulate a returning user – then create your browser object, process a goto command on the browser object, start the timer, then process a new goto command. The two scenarios above should represent two different user experiences, and in turn, two different load times. The creation of the browser object simulates a browser with cleared cache. Loading the page, leaving and then returning in essence simulates a returning user with data in their cache.
Third – be realistic when doing competitive analysis. It is a common need to compare a site that is similar to your own. The difficulty can be in making sure that the comparisons are apples to apples and not apples to oranges. It can be very easy to tweak a test favorably in your direction – even though this provides no benefit to your users.
For example: let’s imagine that you wanted to compare load times of your site compared to your competitor. A common approach would be to set the test up to report back once the DOM is fully loaded for each site. Although this approach can be very helpful – it may be misleading. Imagine if your site loads an empty DOM tree, then loads everything via AJAX. It could be 5 seconds before your login box appears whereas your test reports 1 second because your DOM had finished within 1 second. Our advice is to find specific elements that represent the ability of interaction. Then report on those. For example: report when the text box elements for the login form are available – rather then when the DOM tree loads.
So – in conclusion our experiences have been very good here. A nice mature open source project that has a strong community and is easy to setup. A quick script to simulate a browser experience. Viola – you have a behind the VPN performance testing solution.