PESTT - PESTT Educational Software Testing Tool for Java

Francisco Martins, Rui Gameiro, LaSIGE & University of Lisbon, Faculty of Sciences.

PESTT is an open source Eclipse plug-in for unit testing of Java methods. It started as a tool specially tailored for teaching how to test software, but can be very well used in other scenarios. The current version supports the testing of methods based on control flow graph coverage criteria. Its suits well the teaching of software testing classes, because of its careful designed UI and its smoothly integration with the Eclipse IDE. The testing concepts are well organized: a view for the control flow graph (automatically generated from the method's source code), a view with the supported coverage criteria arranged in such a way that it is easy to visualize which criteria subsumes others, a view for manipulating test requirements and its explanation both in terms of the control flow graph and of the source code, a view for the test paths, another for coverage statistics, yet another for data flow information, and all these views integrated together to give as much information as possible to the test engineer. It provides integration with JUnit and reconstructs run paths of each test method, computing statistics either for each test method or for the complete test set. PESTT distinguishes from other coverage analysis testing tools because it is not just a node and branch coverage analyzer. Among other things, it supports many other "more powerful" coverage criteria and assists the test engineer in the various tasks of the testing activity.

Web Site: http://pestt.github.com
Eclipse update site: http://pestt.github.com/updateSite
Version Tested: 0.5
System Requirements: Eclipse 3.7 or greater
License & Pricing: Eclipse Public License
Support: The PESTT team will handle issues reported either by submitting a GitHub issue

PESTT by Example

This section illustrates how PESTT can assist in the various activities of the testing process through the description of several use case scenarios for unit testing based on Control Flow Graphs (CFGs)-a representation, using graph notation, of all paths that might be traversed through a program during its execution. Consider the following method for determining the average of the even elements of a vector of integers, ignoring the first n elements.

// @pre: v.length > 0
public double evenAverageExceptN (int [] v, int n) {
  int sum = 0;
  int count = 0;
  for (int i = 0; i < v.length; i++)
    if (v[i] % 2 == 0) 
      if (n == 0) {
        sum += v[i];
        count++;
      } else
          n--;
  return (double) sum / count;
}

Control Flow Graph Generation

The first task to perform is to draw the CFG of the method. It is usually a quick and easy task to do, but as other tasks that we will see below, it is tedious and repetitive. PESTT fully automates this task by the clicking of a button from the toolbar (). The selected method's CFG appears in the Graph view (right) as illustrated below.

CFG for method evenAverageExceptN

Note that the graph edges are decorated with information about guards (for loops and branch commands), indication of break or continue, etc. Sometimes this information makes difficult the reading of the CFG (for example, when guards are extensive), but the test engineer has the ability to remove it partially or completely.

An interesting feature is the ability to link CFG nodes with the corresponding source code instructions and vice-versa. It is of special importance in the scenarios described below, in particular, when the graph is complicated and the mental correspondence starts becoming blurred.

Selecting a coverage criteria and computing the requirements set

Tests based on graphs implicitly group input space values according to the execution paths they originate; those that originate the same execution path are considered identical for testing purposes. But is it possible to test all execution paths of a method? In general it is not the case, since, for instance, a method with a loop has an infinite number of execution paths. Thus, the methodology chosen is to follow a coverage criterion-a collection of rules that defines the set of requirements to be tested. In the case of tests based on CFGs, these requirements represent paths in the CFG that must be traversed by the execution of the method. The literature defines several testing coverage criteria as, for example, visiting all nodes in the graph, which amounts to execute all instructions of the method, or visiting all graph edges, which is equivalent to exercise the method in order to traverse all its branches.

The test engineer needs to determine the set of test requirements based on a coverage criterion that she finds appropriate to test the method. Some of these criteria, e.g., prime paths or all definition-uses, are cumbersome to compute and it is easy to incur in errors during its calculation.

PESTT supports most of the common coverage criteria based on CFGs. The following figure shows the criteria that PESTT offers, organized in a directed graph according to its "power"; if a criterion has one edge directed to another, for example, from Edge Coverage to Node Coverage, it means that any set of tests that satisfies the requirements produced by Edge Coverage also satisfies those produced by Node Coverage. In this situation it is said that Edge Coverage subsumes Node Coverage.

CFG coverage criteria supported by PESTT

To generate the test requirements set the test engineer chooses a coverage criterion, for instance, Prime Path Coverage, from the Graph Coverage Criteria view (beside the Graph view). Then, simply pressing the refresh button from the Structural Coverage Criteria view (bottom), the set of requirements appear in the left part of this view (refer to the figure above). When selecting a test requirement, it is drawn in the CFG and, if the link to the source code is active, the instructions corresponding to this path are highlighted, thus identifying better which code needs to be tested to satisfy this requirement.

Planning the tests to perform

Devising a test case involves determining the input values needed to run the method under test and to identify its expected outcome. (For complicated tests, e.g., testing embedded applications, extra information might be needed, but it is not the case for the running example).

The test-planning task aims at realizing which tests need to be performed to cover all test requirements. Each test represents a valid path in the CFG from the starting node (displayed in a green background) to one of the finish nodes (in red), denoted test path. A test path satisfies a test requirement if the latter is a sub path of the former. It is said in this case that the test path tours the requirement. A test path may tour more than one test requirement.

It is up to the test engineer to design the test case, determine the test path, and identify which requirements it satisfies. She should repeat this activity until all requirements are covered, i.e., until she achieves 100% coverage level, which is not always possible (c.f., below section on infeasible tests).

PESTT assists in test planning by allowing the test engineer to add paths and by computing the coverage level both for the added path and for all paths. The central part of the Structural Coverage Criteria view is reserved for test paths. This area is intended not only for the planning of test cases, but also for gathering the results of its execution (see below paragraph on test implementation). PESTT makes available operations for adding, removing, and editing test paths.

When a test path is selected, it is drawn in the Graph view and if the source code link is active it highlights executed instructions in green and those not run in red. For the running example, a set of test paths that guarantees 100% coverage level is given in the table below.

Id	Test paths
1	[0, 1, 3, 4, 5, 6, 1, 3, 4, 5, 6, 1, 2]
2	[0, 1, 3, 4, 7, 6, 1, 3, 4, 7, 6, 1, 2]
3	[0, 1, 3, 6, 1, 3, 4, 7, 6, 1, 3, 4, 5, 6, 1, 3, 6, 1, 2]

Infeasible Tests

Test requirements are generated from all valid CFG paths prescribed by the rules of the chosen coverage criterion. This means that there may be syntactically valid paths that are not possible to obtain from any run of the method. These paths-not semantically admissible-are denoted as infeasible requirements. The running example contains two infeasible paths, although due to different reasons. Path [0, 1, 2] is infeasible because the method's precondition (v.length > 0) requires that vector v must have at least one element.

The second situation relates to the method's semantics. Even though [5, 6, 1, 3, 4, 7] is a prime path, corresponding to the execution of the then branch (when n==0), and, in the following loop iteration, executing the else branch, it is not semantically valid, since when variable n reaches zero its value remains the same until the end of the loop. An infeasible path is never going to be satisfied by any test, which prevents from reaching 100% coverage. In PESTT we can identify infeasible paths by selecting a test requirement and marking it as such. For now, this information is being recorded in method's javadoc and serves both for test documentation and for identifying these paths as infeasible in future method tests.

Sometimes it might be desirable to test sub paths of an infeasible path. PESTT provides operations to add, remove, and edit test requirements in order to manage the requirements set manually. This information is also recorded in the method's javadoc and is automatically used in further testing.

 

Writing Tests and Test Scripts

After identifying the test paths that maximize the coverage level, possibly identifying infeasible requirements, the test engineer can write the test cases. Ideally, test execution should be automated by writing test scripts, for instance, in JUnit format. As a result test (re-)execution is going to be automatic. Below is a Java class that satisfies prime paths coverage criterion for the running example. PESTT, for now, does not generate tests from the requirements. This process is left entirely for the test engineer.

class EvenAverageTest{

       @Test
       public void twoEvenNZero() {
                EvenAverage o = new EvenAverage ();
                System.out.println(o.evenAverageExceptN (new int [] {2,4}, 0));
        }

        @Test
        public void twoEvenNThree () {
                EvenAverage o = new EvenAverage ();
                System.out.println(o.evenAverageExceptN (new int [] {2,4}, 3));
        }

        @Test
        public void oddEvenEvenOddNOne() {
                EvenAverage o = new EvenAverage ();
                System.out.println(o.evenAverageExceptN (new int [] {1, 2, 4, 5}, 1));
        }
}

Test Execution

PESTT supports integration with JUnit. When running tests, JUnit presents a report identifying the tests that have executed successfully and the ones that have failed. Determining the path executed by each test is critical to compute the coverage level and to find out which test requirements were satisfied. PESTT automates this task and provides statistics on the coverage level, as well as the requirements that have been met.

PESTT provides a specific launch environment to run JUnit tests, to keep track of the path executed by each test, and to be able to compute the coverage level and other statistics. The environment is similar to that of JUnit, presents the same JUnit views, but instrument class byte code so it can, afterwards, identify the run path of each test. It uses the CFG to derive a script for code instrumentation. Thereafter, it analyses the result and reconstructs the path executed by each test method. From there, it determines the satisfied requirements, coverage level, and further statistics. This is one of the features that distinguish PESTT from other coverage analysis tools: the ability to handle coverage criteria other then node and edge coverage.

When selecting a test path, PESTT shows whether each test requirement have been satisfied (in green and with state ) or not (in red and with state ). Infeasible test requirements are marked as as well, but are colored blue to distinguish from those that have been met. The right hand side of the view contains self-explanatory test statistics for the selected test path. The final step consists in writing a set of tests that have the execution paths identified previously (refer to the table and the Java class above).

By adding more than one test path it appears a total option (as a path) that shows statistics for all test paths as illustrated below.

It is also possible to set different tour types, checking sub path alternatives. To set it, press the visit type button () and select the Tour, Sidetrip, or Detour option. For example, test path [0, 1, 3, 4, 5, 8, 7, 1, 2] does not tour requirement [0, 1, 2] (the requirement is not its sub path), but it sidetrips and detours the requirement. Notice that [1, 3, 4, 5, 6, 7, 1] corresponds to a loop iteration.

PESTT has many more features, like data flow views, that we leave for the interested reader to find out by trying the tool. Until mid 2013 we plan to extend PESTT in two directions: address mutation testing and implement refactoring patterns to rewrite classes in such a way that they can be unit tested; unfortunately, much of the software written today is not testable. We want PESTT to detect and provide assistance in refactoring such source code.

---

Unit Testing Articles

Writing Testable Code

More Java and Software Testing Knowledge

Java Tutorials

Software Testing Magazine

Unit Testing Tools Directory

---

This article was originally published in the Fall 2012 issue of Methods & Tools

Click here to view the complete list of tools reviews