Automating Browsers with Ruby and Watir-webdriver

0
26278
Automation technology
Lightweight and simple to use, Watir is a browser driver that is great for automating modern browsers. It is built on Ruby but it also works well with other testing frameworks supported by Ruby. In this article, the author describes the installation of Watir and its use in retrieving the results of the CBSE Board exams.

If necessity is the mother of invention, then boredom is the grandmother of automation. Automation refers to the process of performing a task without any attention or input required from the person performing that task. I am referring to tasks that are quite trivial, while at the same time boring, like getting results from a website based on a query or a user selection. Most of the time, things like this can easily be done by Web scraping or vendor APIs, but we can also instruct browsers to do these systematically. In this article, we will use Ruby as the commanding force and Watir-webdriver (Watir stands for Web automation testing in Ruby) as the bridge between Ruby and the browser. Although Watir is mainly used for Web application testing, we can also use it to create useful scripts. And here’s a small note: after you finish this article, you may want to get deeper into Watir, and while exploring the Web, you will usually see both Watir and Watir-webdriver; don’t get confused. The difference lies in the browser that is driven. Watir only works/drives with Internet Explorer on Windows, while Watir-webdriver drives Firefox and Chrome on Windows, Mac and Linux.

Figure_1
Figure 1: Firefox instance
Figure_2
Figure 2: Error message

Installation
Let’s get started. Since this set-up requires the use of Ruby, the first obvious step is to check for Ruby. If Ruby is installed on your system then your development environment is already prepared (which is no easy task); otherwise, install Ruby. Depending on your operating system and distribution, the procedure’s steps may differ. When in doubt, refer to the installation documentation available at https://www.ruby-lang.org/en/documentation/installation/. But whichever way you choose, always remember to use version management tools like rbenv, rvm or pik. For Linux and Mac, the best way to install Ruby is through rbenv or rvm and for Windows, it is pik (no longer maintained) or Uru (though, I would advise you to stick with UNIX systems, as managing Ruby in Windows is difficult compared to Mac and Linux). Check the ‘References’ section for links of detailed tutorials. Since we have prepared our development environment, we can proceed to install libraries for Watir or ‘Gems’ as we call them in the Ruby world. Open the terminal or the equivalent to install Gems.
To install Watir-webdriver, use the following command:

gem install watir-webdriver

If the following commands throw errors due to lack of privileges (usually encountered on Ubuntu), prefix the command with sudo:

sudo gem install watir-webdriver # Use this if previous command doesn’t work

To install Watir, issue the following command:

gem install watir # Prefix with sudo if necessary

Fire up irb or pry (whichever suits you) to load the Watir-webdriver gem:

require ‘watir-webdriver’

Now, to invoke a new browser, use the following command:

browser = Watir::Browser.new :firefox # Default is firefox

Now, let’s do it for Chrome, as follows:

browser = Watir::Browser.new :chrome

You may get an error message similar to what’s shown in Figure 2.
The error message says that it can’t find the chromedriver executable, which is required to drive Chrome. This can be fixed by downloading the latest Chrome driver from the http://chromedriver.storage.googleapis.com/index.html and adding the executable to our PATH (which specifies the list of directories to be searched to find the program). Download the latest chromedriver, which at the time of writing this article was 2.13. The latest version can be checked by visiting the link: http://chromedriver.storage.googleapis.com/LATEST_RELEASE. Make sure to download the correct executable for your system. Since I am running the 64-bit version of Chrome on Linux, I selected chromedriver_linux64.zip. Download the latest version (2.13) and extract it into a folder named driver. The following commands will do the trick:

whet http://chromedriver.storage.googleapis.com/2.13/chromedriver_linux64.zip unzip chromedriver_linux64.zip -d driver

Now, we need to add a PATH for the Chrome driver, as follows:

echo “export PATH=$PATH:$HOME/driver” >>. .bashrc

Since we now have browser support, we can try to do a few things. From now on, we will call our browser slacko (our virtual pet). Save this script as test_1.rb or whatever name you like.

require ‘watir-webdriver’ #Include the gem
slacko = Watir::Browser.new :chrome #You can also use firefox
slacko.goto “google.co.in” #Loads the google.com
slacko.text_field(:name => ‘q’).set “open source” #Sets the query field (having class name ‘q’) to open source
slacko.button(:name => ‘btnG’).click
puts slack.title #Prints the title of the page
slacko.screenshot.save “open.png” #Saves the screenshot of the page to open.png”

Run the script with the following command:

ruby test_1.rb # substitute with the name of the script

Finding class attributes like names or an ID can be done using developer tools, since almost all major browsers come with one. In Chrome, doing this is as simple as clicking the Inspect element from the right-click menu.
Let’s apply this to do something useful.

Figure_3
Figure 3: The output
Figure_4
Figure 4: Inspecting elements

Result retriever
Watir can be used to automate the retrieval of results for an examination from a large pool of numbers (usually roll numbers). CBSE is one of the boards for education in India and the result day breaks many hearts. Let’s write a simple script to retrieve the exam’s results list for a given range of numbers.
Disclaimer: The following scripts are presented for educational purposes only. While using them, use the resultant data with utmost care and responsibility, and respect the privacy of others. Neither the author nor OSFY are responsible for any misuse.
For the sake of simplicity, let’s target the 2013 results of CBSE Class XII.
Here’s the complete script:

require ‘watir-webdriver’
range = (1..90) # Replace numbers as per use
slacko = Watir::Browser.new :firefox #Makes a firefox pet
slacko.goto “ http://resultsarchives.nic.in/cbseresults/cbseresults2013/class12/cbse122013.htm” #Loads the target page
range.each do |number|
slacko.text_field(:name => ‘regno’).set number #Sets the input with the current number in range
slacko.button(:name => “B1”).click #Clicks the submit button
slacko.screenshot.save “#{number}.png” #Saves the screenshot of result as number.png
slacko.back #Loads the previous page
end

If there are no errors and the script runs fine, you will see a list of screenshots of results arranged by roll numbers. Since you will have a lot of data, use it carefully and don’t use it to harm anybody. This could be particularly useful for schools that want to digitally archive the results of their students.

Going headless
The results retrieval script is good but browsers use resources. We can replace Firefox or Chrome with a headless browser (one with no graphical interface) like phantomjs so as to save memory. Headless browsers like phantomjs can accomplish the same task with less resource consumption and also support screenshots.
Installing phantomjs is as easy as downloading the binary and placing it in the path.
The following commands will come handy:

#For 64 bit Linux systems
wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-x86_64.tar.bz2tar xvjf phantomjs-1.9.8-linux-x86_64
cd phantomjs-1.9.8-linux-x86_64/bin/
sudo ln -sf phantomjs /usr/local/share/phantomjs
sudo ln -sf phantomjs /usr/local/bin/phantomjs
sudo ln -sf phantomjs /usr/bin/phantomjs

If all goes right, a new phantomjs instance can be created by the following code:

require ‘watir-webdriver’
slacko = Watir::Browser.new :phantomjs
#Do crazy things with slacko :)

References
[1] Ruby Installation Documentation: https://www.ruby-lang.org/en/documentation/installation/
[2] Ruby Installer and Pik for Windows (Pik is no longer maintained, so proceed with care): http://misheska.com/blog/2013/03/10/using-pik-to-manage-multiple-versions-of-ruby-on-windows/
[3] Uru Wiki: https://bitbucket.org/jonforums/uru/wiki/Home
[4] Rbenv on Mac and Linux: http://misheska.com/blog/2013/06/15/using-rbenv-to-manage-multiple-versions-of-ruby/
[5] Link to download the Chrome driver: http://chromedriver.storage.googleapis.com/index.html
[6] 1 Watir cheat sheets and references: https://awetest.zendesk.com/hc/en-us/articles/201883796-Watir-Webdriver-Cheatsheet
[6]. 2 https://github.com/watir/watir/wiki/Cheat-Sheet
[7] Phantomjs installation instructions: https://gist.github.com/julionc/7476620
[8] CBSE link used for demonstration: http://resultsarchives.nic.in/cbseresults/cbseresults2013/class12/cbse122013.htm
[9] Phantomjs: http://phantomjs.org/download.html

LEAVE A REPLY

Please enter your comment!
Please enter your name here