Watai - Integration Testing for Web Applications and Components

Matti Schneider, @matti_sg, mattischneider.fr

Watai (Web Application Testing Automation Infrastructure) is an open-source, declarative full-stack web testing framework. It is both a test runner engine and a set of architectural patterns that help front-end ops-conscious [1] developers create maintainable and solid end-to-end tests. These tests are written as text files that simply describe the components of the application and how the user is expected to interact with them. Watai uses them to automate web application navigation through actual browsers, checking whether such interactions present the expected information to the end users along the way, including through asynchronous operations. This allows automating demonstration of features, detecting regressions when updating code and, since the same tests can be run in almost any browser [2], easily checking cross-browser functional support.

Web Site: https://github.com/MattiSG/Watai/
Version tested: 0.7.
System requirements: Node.js ≥ 0.8, NPM ≥ 2, Java ≥ 1.4 and installed browsers.
License & Pricing: Free & open-source (AGPL).
Support: Community, through GitHub issues and wiki.

Context and vocabulary

"Integration" testing is a testing level in which a combination of individual software modules is tested as a group. In the context of web applications, it means testing the result of combining client- and server-side code, and accessing it through a browser.

Based on this definition, you might have already heard about what I call here "integration testing" under a different name. You might have read about "customer", "end-to-end" (also abbreviated as "E2E"), "system", "acceptance", "validation" or even "functional" testing. These are many synonyms for almost the same reality. For consistency, I will only refer to it in this document as "integration testing".

Watai is a tool for integration testing that focuses on making it easy for developers to write user-oriented tests that consider the system under test as a black box, yet are maintainable enough to be worth investing in. These tests are simple JavaScript files that are parsed by Watai and transformed into Selenium (WebDriver) instructions. By using WebDriver to automate browsers, it makes it easy to play tests in any browser, and thus to fight against platform discrepancies, usually referred to as "cross-browser issues".

Example

We will present a test for the DuckDuckGo search engine. This engine is just like Google, except that it doesn't track you, and that it has a nice disambiguation feature: the "Zero Click Info box". Let's imagine we are engineers at DuckDuckGo, and we want to test if a Zero Click Info box is shown for an acronym: "TTIP".

Why are people around the world using this acronym? Are they talking about a chemical compound? Or about a US-EU free trade agreement negotiated in secret? The result page should help our users get a quick answer!

 

Based on the above requirements, we can formalize the following testing scenario:

1. Go to duckduckgo.com (English version).
2. Type "TTIP" in the search field.
3. Submit the search.
4. Ensure that the text "Transatlantic Trade" is presented in the "meanings" header ribbon.

The test suite automating the above scenario is already available at https://github.com/MattiSG/Watai/tree/master/example/DuckDuckGo. We see a Configuration file, two Component files, a Scenario and a Fixture file. These are all the parts of a Watai test.

Configuration

The config.js file exports a JavaScript object with two mandatory key-value pairs:

• baseURL, that describes the URL at which the test should start.
• browser, that defines which browser should be used to run the test.

Components

Components are defined in files ending in Component.js. They represent a graphical component of the tested web application. A component provides a mapping from DOM elements, identified through selectors, to testing-environment attributes.

Open the ZeroClickComponent.js file: it is a simple JavaScript object property.

// example/DuckDuckGo/ZeroClickComponent.js
meanings: '#zci-meanings'   // this is an element definition

Our header definition is here as an element. It maps a meaningful, testable name to a CSS selector describing how this element is retrievable in the DOM. We will now be able to retrieve the content of the Meanings section of the Zero Click Info box.

Interface components in modern web apps are not only sets of elements. They can also offer complex actions to the user, triggered by basic interaction events such as clicks or keystrokes on specific elements. This is why a component is a bit more than a simple DOM map: it also abstracts action sequences and makes them callable directly through a meaningful name.

Open the SearchBarComponent.js file: there are once again elements, but there is also an action defined.

// example/DuckDuckGo/SearchBarComponent.js
field:        'input[name=q]',            // this is an element
submitButton: '#search_button_homepage',  // this is an element too

searchFor: function searchFor(term) {     // this is an action
    return  this.setField(term)()
                .then(this.submit());
}

The searchFor key is a JavaScript function that takes a parameter and sequences two basic steps: typing the passed string in a field, and clicking a submit button. With this action, we will be able to search for an ambiguous term, as our scenario needs.

Fixtures

We then need to define what we want to search for, and what we expect to be presented with. Open the ZeroClickFixture.js file: this is where we define such fixture.

// example/DuckDuckGo/ZeroClickFixture.js
query = 'TTIP'
expandedAcronym = /Transatlantic Trade/

You can notice that we are inputting a specific string, but we only expect results to match a given RegExp rather than defining exactly what they will be. This improves the resilience of our tests.

Using fixtures files to store values is optional, but it is recommended as a way to decouple concerns even more. In our example, if we wanted to test another ambiguous term, there is no hesitation where to define it. You have simply to open the fixture file, replace "TTIP" with the new term, and restart the test. A fixture file is pure JavaScript code, and has to end in "Fixture.js". All the variables defined inside will be made available to all components and scenarios in the same suite.

Scenarios

So, we have now defined all needed elements, actions and fixtures for our scenario. However, we have not yet defined how these pieces fit together, nor what we are going to check. This is what a scenario is for. Let's look at the 1 - ZeroClickScenario.js file.

// example/DuckDuckGo/1 - ZeroClickScenario.js
description: 'Looking up an ambiguous term should make a Zero Click Info box appear.',

steps: [
    SearchBarComponent.searchFor(query),
    { 'ZeroClickComponent.meanings': expandedAcronym }
]

Just like a component, a scenario is series of object properties, this time with two mandatory keys. The first is description, mapping to a String presenting what expected behavior you are testing. The second expected key in a scenario is steps. Its value must be an array.

In our example, the first line is simply a call to the action we defined in the SearchBarComponent: we are going to searchFor the term we defined in the fixture: query. You can define as many steps in a scenario as you want. A step can be, like here, an action, a custom function, or a state definition. All these steps will always be executed in order.

We define how our components will be used with scenario steps, but we still don't know how to actually check behavior! Assertions in Watai are implicit. That is, you won't have to assert anything, unless you want to. You simply define an expected state, and the actual state of the current web page will be checked against it.

A state definition is simply a hash where keys reference some elements, and the corresponding values are matched against the values of the targeted DOM elements. The only tricky thing is that you can't have an actual reference as a hash key, only its name. So, remember to quote keys when defining states.

Executing the suite

To execute a test suite locally, you will first have to install dependencies. You will need Node.js (installable from nodejs.org), and a Java runtime (java.com/download). Then, simply run `npm install --global watai selenium-server` at a command prompt, and start the Selenium server by running `selenium &`. You can then try the suite we examined above by downloading it as a ZIP (github.com/MattiSG/Watai/archive/master.zip) and running it with `watai <path/to/downloaded/archive/example/DuckDuckGo>`.

A dynamic progress indicator will be displayed in your terminal, and you will be able to follow through the test suite. Several views are available, including a more verbose one, suitable for continuous integration, a WebSocket-based one that allows interaction with external views. The exit code [3] will always be non-0 if a test fails, allowing you to integrate with any pipeline.

You are now ready to write your own test suites, using this example and the thorough reference documentation available at github.com/MattiSG/Watai/wiki!

A gentle reminder

All levels of testing are important, and doing tests only at one level is not a proper software testing strategy. You should have unit tests for your application, both on the server- and on the client-side. If your application is complex, adding functional-level tests is a good thing. Integration tests validate expected behavior, and Watai largely reduces their barrier to entry, but they still cannot help in pinpointing the origin of a failure as much as unit tests can.

References

1. Front-end ops: http://smashingmagazine.com/2013/06/front-end-ops/
2. The supported browsers are the ones supported by WebDriver, i.e. almost all: http://docs.seleniumhq.org/about/platforms.jsp
3. https://github.com/MattiSG/Watai/wiki/Exit-codes

---

More Software Testing Knowledge

Software Testing Magazine

Functional Testing Tools

---

Click here to view the complete list of tools reviews

This article was originally published in the Winter 2015 issue of Methods & Tools