Test Automation Frameworks

In the test engineering team, one of the choices we have to make often is to pick the best automation tool for a particular project. In terms of automation frameworks, one size does not fit all. Each tool/framework has its own strengths, which make it suitable to be used for specific types of projects. In this blog post, I'd like to introduce to you (in no particular order) the various frameworks we use for test automation and the reasons they were chosen for that particular application.

Robot Framework

Robot Framework is a generic test automation/execution framework. It's generic in the sense that it doesn't provide any real automation capabilities itself, but has pluggable underlying libraries that can be used for most automation needs. There is a Selenium library for browser automation, Appium library for mobile and a Requests library for HTTP web services testing. The strengths of Robot Framework are in its ease of use, flexibility and extensibility. You can create your tests in easy to read text (or markdown or HTML) format and you can use the lower level keywords provided by the libraries to create higher level keywords that describe your application better. This is particularly beneficial for test engineers because it lowers the automation learning curve for other team members, like Product Managers and Developers. Test Engineers can provide the keywords in such a way that writing tests becomes easy and just a matter of picking up the right keywords and providing right arguments.

We use Robot Framework for some of our front end Rails projects. These projects have mostly design and text changes and are tested using Selenium library. Once the features of these applications are described appropriately in keywords, it's easy to create or update tests. For example, here is a test that opens our home page, clicks on a link and validates that the "Apply Now" button in present.

*** Test Cases ***
| Validate About Page
| | [Documentation]             | Validate content in 'About Payoff' page
| | [Tags]                      | about
| | Open Browser To Home Page
| | Click Element               | link=About Payoff
| | Title Should Be             | About Us \| Next-Generation Financial Services \| Payoff
| | Page should contain element | id=btn-apply-now-get-started
| | [Teardown]                  | Close Browser

Example Robot Framework Test Case

Capybara/Rspec

For testing our loan UI application we use Capybara, a web automation framework. We use it with Selenium WebDriver to run the tests in a real browser, but it can also be used with headless browsers like capybara-webkit or poltergeist. While Robot Framework works great for validating simpler UI applications, we need a more robust programming environment for testing loan application because of multiple steps and conditionals involved. We also have an external data source for this application, and it gets tedious to write data driven tests with multiple steps in Robot Framework.

Capybara has a nice DSL to write expressive UI tests. Another advantage of writing tests in Capybara is that we can use our internally developed gems (or external gems, if needed) to enhance the automated tests. For example, we use feature flags to disable/enable certain functionality in our applications and to test them, we might use these gems to turn the feature flags off or on, run the tests and then toggle them back to their original state.

Here's a snippet of a typical feature test written in Capybara:

feature 'Loan application flow' do
  scenario 'complete and submit an application for approval' do
    # homepage
    visit '/'
    # start application
    click_button('the-payoff-loan-apply-now')
    # loan amount
    expect(page).to have_content('How much do you owe on your credit cards?')
    fill_in('loan-amount', :with => '10000')
    click_button('linear-continue')
    # FICO score
    expect(page).to have_content('What\'s your FICO Score?')
    select('Excellent (720+)', :from => 'credit_score_bracket')
    click_button('linear-continue')
    # name
    expect(page).to have_content('What\'s your name?')
  end
end

Feature Test in Capybara

Karma/Jasmine

Karma is a javascript test runner for unit testing of javascript in web applications. The tests can be described in frameworks like Jasmine. It is primarily used by our front end developers. Karma integrates with our development workflow as the tests are included as part of the rails package and run every time a build runs in our CI environment.

Airborne/Rspec

Airborne is a gem that makes it easy to write webservice API tests using Rspec. The reasons for using this gem as opposed to others is to make the tests and validations as descriptive as possible. We also have some APIs that have a lot of endpoints or intermediate states to test, so it especially important to make it easy to add new tests. 

Here's a snippet of hypothetical tests for httpbin.org:

describe 'httpbin api service' do
  before(:all) do
    @url = 'http://httpbin.org'
  end
  it 'get returns original url in json response' do
    get @url + '/get'
    expect_json(url: 'http://httpbin.org/get')
    expect_json_sizes(args: 0)
  end
  it 'get with url params returns the params in json response' do
    get @url + '/get?spec=true'
    expect_json_sizes(args: 1)
    expect_json('args', spec: 'true')
  end
  it 'post returns data in json response' do
    post @url + '/post', { :spec => 'true' }
    expect_json('json', spec: 'true')
  end
end

API Tests using Airborne

TestNG

TestNG is a Java based test framework, which provides capabilities to write automated tests in Java. So for example, browser automation tests can be written using Selenium WebDriver's Java bindings, or for REST based services using libraries like rest-assured. We use TestNG for testing some of our internal REST services, specifically because TestNG makes it easy to use external data source such as CSV or Excel file using DataProviders. It also integrates nicely with build tools like maven and CI tools like Jenkins and Bamboo.

Overall, TestNG works well for automation projects in Java but to leverage our internal toolset and Ruby expertise, we are using Rspec + Capybara or Airborne for automation.

Galen Framework

We have recently started investing some time in Galen Framework for testing the UI/UX of our front-end applications. This framework promises to solve the widespread problem of testing the layout of web applications on multiple display resolutions. So far, the framework looks promising and we are continuing to automate the layout testing of our marketing websites. We plan on partnering with our UX designers so that they can use this tool to test the layout as they make changes to the applications.

Galen Framework uses Selenium as the underlying automation tool, and the layout specs are written in its own spec language. In the below layout spec, we are verifying some elements on our marketing home page on 2 different layouts (desktop and mobile) with specific display resolutions:

@objects
    how-payoff-works        id hdr-how-payoff-works
    rates-fees              id hdr-rates-and-fees
    about-payoff            id hdr-about-payoff
= Verify Marketing Landing page Elements =
    @on desktop
        how-payoff-works:
            text matches "How Payoff Works"
            width ~131px
            height ~70px
            aligned horizontally all how-payoff-works 2px
        menu-button:
            absent
    @on mobile
        how-payoff-works:
            text matches "How Payoff Works"
            width 345 to 360 px
            height ~40px
            aligned vertically all how-payoff-works 2px
        menu-button:
            text matches "Menu"
            width ~81px
            height ~40px
            aligned horizontally all login-mobile 2px

Galen Framework Layout Spec

Galen Framework can handle differences between a mobile or desktop screen, like cases where an element is visible in one and not the other. For example, the menu-button above is not visible on a desktop screen resolution and so, it can be specified to be absent on that.

The interaction with UI can be automated using Java (or Javascript) and as mentioned earlier, it uses Selenium WebDriver for that. Here's an example that use the above spec to validate the layout:

test("Desktop Landing Page Elements", function(){
    var driver = session.get("driver");
    marketingPage.waitForIt();
    checkLayout(driver, "./specs/marketingPage.gspec", "desktop");
});

Galen Framework Test

Appium

Appium is a mobile automation framework that allows writing tests using a common API for both iOS and Android platforms. Since it uses WebDriver protocol, tests are written similarly as browser tests on a desktop using WebDriver. Currently, we don't have a use case for native app testing so we are content with using Appium for testing our websites on mobile browsers. Once we do, we will explore other tools/frameworks like UIAutomator for Android, UI Automation for iOS or Calabash for both Android and iOS etc.

Appendix: Selenium WebDriver

I couldn't end this post on Test Automation frameworks without talking more about Selenium WebDriver, which is the underlying tool for browser automation. There was a time in distant history when Mercury's WinRunner and later, QuickTest Pro were the tools of choice for UI Automation. At that point, the term Continuous Integration hadn't been coined by Martin Fowler and the idea of Test Automation for most companies was to have some QA engineers write some scripts on the side and run them manually to make repeated, time consuming tests execute faster. QuickTest Pro was a commercial tool and cost quite a bit, AND it only supported Internet Explorer on Windows. Nevertheless, it was good for what it did, which was to reduce a bit of manual testing overhead for repeated regression testing. It also integrated well with Mercury's TestDirector or Quality Center for test management, so you could envision tying together requirements gathering, test planning, test execution and defect reporting within one tool.

But as IE lost ground to Firefox and Chrome and mobile browsing grew more popular, and as HP acquired Mercury, QuickTest Pro started falling behind. More and more teams wanted to adopt agile methodologies so HP's tools seemed very heavy to use and didn't fit in an agile environment. Around that time, Selenium started gaining popularity. It was open source, so test engineers could easily prototype an automation solution with it and convince their teams to start using it. Support for testing on different browsers was a huge bonus, as well as its easy integration with continuous integration tools. The open source community behind it continued improving it with support from companies like Mozilla, Google and ThoughtWorks. The ability to write automated tests in any of the popular languages and Selenium's integration with a number of frameworks like TestNG, Junit, Capybara, Cucumber, Watir etc. added to its appeal.

Paul Hammant wrote a blog post with his thoughts on why QTP was not relevant anymore. He has some graphs showing the popularity of Selenium vs. QTP on job site indeed.com. Those are a bit old (2011) so I pulled up a similar graph myself, and it is very telling:

Job Trends from Indeed.com - Relative Growth (UFT vs QTP vs quicktest vs Selenium)

Appendix: Apache JMeter

While this post is about functional automation, I want to briefly mention Apache JMeter, which is the tool we use for load/performance testing. In a future post, I'll describe how we use it and how it integrates in our continuous deployment process.