Click here to view the complete list of tools reviews

This article was originally published in the Winter 2011 issue of Methods & Tools

---

Sureassert Exemplars: Java Unit Testing Without Unit Tests

Nathan Dolan, http://www.sureassert.com/uc/about-us/

Sureassert UC is a new multi-faceted annotation-driven tool that seeks to cut the expense, increase the effectiveness and simplify the process of unit testing for Java projects. It introduces specification-driven, declarative tests called Exemplars along with the capability to perform continuous test execution and coverage reporting within the Eclipse IDE.

Web Site: www.sureassert.com
Version tested: 1.3.0
System requirements: Any Eclipse 3.4+ distro
License & Pricing: Open source (Annotations Library); free (Engine for Eclipse)
Support: via forums, guide and specs at www.sureassert.com

Unit Testing and the Importance of Specification

Unit testing applied correctly reduces delivery risk and increases maintainability by providing a regression safety net. Having been standard practice for some considerable time however, writing comprehensive unit test suites (typically in JUnit) remains an expensive exercise and one that is often poorly applied in the real world. For the purposes of the article, it’s important to differentiate the term "unit tests" from longer-running tests that depend on the presence of external resources (such as integration tests and functional tests). The term refers to the smallest testable part of an application, i.e. in Java, isolating and testing methods.

Writing poorly named classes full of undocumented, poorly named methods introduces maintainability issues. Establishing an effective project vocabulary and semantic model is important. Building on this, methods should include a basic specification that describes what the method does (but not how), what it returns, what parameters it expects and the exceptions it may throw. Additionally it may describe pre and post conditions of the method, invariants, and it may summarise example usage scenarios. This is commonly achieved using javadoc.

If you doubt the importance of specifying classes and methods, step back and consider how pleased you would be if all the APIs on which your code was dependent (like the JDK, Spring, Commons Utils, etc) had their class and method specifications (javadocs) removed. This is not just about readability, it is about reducing complexity by allowing the reader to abstract and conceptualize, which makes code more extensible and maintainable thereby reducing cost and risk of change.

How does this relate to unit testing? Generally, unit tests test methods. If a method doesn’t have a specification, it is perhaps valid to ask what the test is actually testing. Unit tests should assert that the contract of each method holds under varying conditions. Ideally, the contract between class/method and consumer is that the consumer adheres to the specification. If no contract is defined for a method, consumers may use it as they please and subsequently the results are undefined. Asserting a method works without defining the boundaries of what "works" means has little value other than in trivial software that has few consumers or inter-dependencies.

As a brief aside: It has been claimed that unit tests themselves can form the specification. Confusing what can sometimes work for user acceptance testing with unit testing is a mistake. Consuming an API with reference to its suite of unit tests does not help abstract or conceptualize: it just gives the consumer unnecessary puzzles to solve. However, unit tests can form part of the specification. At this point let’s introduce Sureassert Exemplars.

An Exemplar encapsulates a single test thread targeted at a specific method and enables Declarative Unit Testing. Traditionally, Java unit tests are written as classes containing methods that invoke a "method under test". The test code fundamentally does three things:

1. Create or otherwise retrieve an object (the "object under test"),
2. Execute a method on the object under test with typical inputs, and
3. Assert one or more expected results on the method’s returned value, thrown exception, the object under test’s state, and/or the state of some other affected object.

It typically might also:

1. Perform some initialization and/or clean-up; although this is more typical and necessary within integration and functional test classes rather than unit tests as these more often deal with resources external to the object under test;
2. Setup stubs or mocks to force external dependencies to behave in a manner determined by the unit test, and
3. Assert the behaviour of the method under test, e.g. what methods have been invoked by the method under test and in what way.

This being the case, why code it? Why not declare the object under test, test inputs and expected results as part of the method specification itself? After-all, providing example execution scenarios often forms part of the worded specification. For example, take the String.substring specification:

public String substring(int beginIndex, int endIndex);

Returns a new string that is a substring of this string. The substring begins at the specified beginIndex and extends to the character at index endIndex - 1. Thus the length of the substring is endIndex-beginIndex.

Examples: "hamburger".substring(4, 8) returns "urge"

"smiles".substring(1, 5) returns "mile"

Parameters:

beginIndex the beginning index, inclusive.

endIndex the ending index, exclusive.

Returns: the specified substring.

Throws: IndexOutOfBoundsException - if the beginIndex is negative, or endIndex is larger than the length of this String object, or beginIndex is larger than endIndex.

Introducing Exemplars

Let’s use Sureassert Exemplars to show the same thing. To define them, annotate the method-under-test:

@Exemplars(set={

@Exemplar(instance="'hamburger')", args={"4", "8"}, expect="'urge'"),

@Exemplar(instance="'smiles'", args={"1", "5"}, expect="'mile'") })

public String substring(int beginIndex, int endIndex);

Each Exemplar attribute takes a string, but it is parsed by Sureassert as a SIN expression.

These match Java very closely, the main notable differences are the use of ' instead of " (as escaping /" wouldn’t be pretty), and the ability to use SIN types. These are prefixed special types designed to simplify object creation. For instance to create a new map of strings to integers for use as an argument, you could specify: args="m:'k1'=2, 'k2'=4,'k3'=8". Expressions can get field values (including private fields), refer to the object-under-test or parameters of the method-under-test, and make any constructor or method call (including private methods) on any object or class. This means that if you need to, you can do things like build test data and assert expected results in test utility code called from an Exemplar.

Returning to the example, we could also assert the exception scenarios:

@Exemplar(instance="'negtest'", args={"-1", "5"},

expectexception="StringIndexOutOfBoundsException"),

@Exemplar(instance="'negtest'", args={"1", "8"},

expectexception="StringIndexOutOfBoundsException"),

@Exemplar(instance="'negtest'", args={"3", "2"},

expectexception="StringIndexOutOfBoundsException")

Most Exemplar properties have shorthand alternatives for brevity:

@Exemplar(i="'negtest'", a={"-1", "5"}, ee="StringIndexOutOfBoundsException")

This is a very simple example. There are 16 exemplar properties as of version 1.3.0 providing a wealth of testing options including stubbing, behaviour verification, naming, setting up, tearing down, default class generation, templating and more. There are also several other annotations for doing things like integrating JUnits and defining "test double" classes. Crucially, you can define Exemplars on interfaces which are then inherited.

How are Exemplars run? Simple: just install Sureassert in Eclipse, enable it on your project and save your code (Ctrl+S). The Sureassert engine integrates with the Eclipse build process to execute all those Exemplars and integrated JUnit tests that are affected by the code changes you’ve made. Errors are reported just like compilation errors, except using a purple rather than a red cross. This includes nested dependencies on methods and field values. Effectively, you never need to run tests: those that need to be, are run whenever you change anything. On top of this, Sureassert also includes integrated test coverage reporting which is also attached to the Eclipse incremental build process.

Let’s move on to a discussion on test isolation, and how using Sureassert can help prevent "design for testability" from being a negative influence on your software architecture.

Test Isolation and Designing for Testability

There are two broad camps with regard to unit testing style and how best to isolate the code-under-test from its dependencies. A behaviour-driven approach dictates that unit tests should assert how the method interacts with other code, and that state "becomes a side-issue". A state-assertion approach dictates that tests should assert the state of affected objects and that the mechanism by which this state was reached is unimportant.

It’s unlikely either approach is better than the other across all types of software. Regardless of approach, effective unit tests assert that the behaviour of the method-under-test matches its contract, under varying conditions. For example, let’s say a method’s contract states it changes a given external object’s state (say it adds a Message to a notional Queue). The test can assert that the object’s state is changed (the Queue now contains the Message), or it can assert that a method is invoked on the Queue in a way that is known by virtue of its own contract to have this effect (e.g. addMessage was called on the Queue with the Message). The latter has indirection through a dependency on another method’s contract, but enables mocking to simplify isolation. The former is direct and can simplify the test but requires dependencies are replaced with stubs which complicates establishing the testing context.

Advocates of a behaviour-driven style would prefer contracts that define interactions, not state changes, in order to reduce the indirection. Those in the other camp might say the opposite. Either of these decisions might be seen as dictating software architecture just to fit a unit testing approach. Regardless of the approach taken, for non-trivial software what is most important is that there are specifications acting as code contracts, they are exercised by the tests, and no "assumptions" are made about what a method does.

The problem with considering "design for testability" in isolation is that non-functional aspects play-off against each-other. It would not be prudent to design overly for testability at the greater expense of maintainability or delivery cost. Let’s introduce how using Sureassert can help prevent testability being a negative influence on your software architecture by simplifying test isolation. The screen-grab below shows some code in Eclipse with Sureassert enabled:

Exemplars in the Real World

In the example above we have a Game class that registers the highest of a given list of scores with an external GameLeaderboardService that is retrieved from a ServiceBroker class defined by the project (not shown). It hasn’t really been designed for testability – the ServiceBroker has a static method and isn’t injected. While it may be better to inject a singleton ServiceBroker instance, this may not be desirable or even possible. We shouldn’t have to make a decision like this simply to achieve testability. With Sureassert we can easily stub or mock the call; in fact there are several different techniques available for doing so.

We place an Exemplar on the constructor and define a method stub that replaces the ServiceBroker.getServiceImpl call with a named instance of GameLeaderboardImpl called "leaderboard". The named instance will have been created by another Exemplar defined in GameLeaderboardImpl (not shown). We give our Exemplar the name "game1", which assigns the object created by the constructor this name (i.e. we create another named instance within our testing context). By testing the constructor, we create an instance that can then be used by other Exemplars. Sureassert manages dependencies between these named instances automatically and understands when it needs to re-run dependant Exemplars. You can control whether to re-use or create new test instances (just postfix the name with a bang "!").

It’s worth pointing out that rather than declaring a method stub, we could have declared a test double class of ServiceBrokerFacade which would have automatically replaced the doubled class whenever running Exemplars or integrated JUnits. Or we could have used an in-line source stub to replace the getServiceImpl code with something else in the testing context. This gives you the flexibility to employ the best test isolation techniques for your project, all integrated into Sureassert and all without writing any test code.

Moving on to the registerHighScore method, we’ve defined two Exemplars. The first tests the precondition that the list must be non-null by passing null as an argument and asserting that an IllegalArgumentException is thrown via the property expectexception. The second uses the l: prefix (List SIN Type) to quickly create an ArrayList populated with three Integers for the method argument. It then performs behaviour verification using the verify property to ensure that the GameLeaderboardServiceImpl.registerScore method is called with ‘user1’ and 17 (the highest of scores passed).

Continuous Test Execution and Feedback

You may have noticed the green ticks in the left border – this is how you know the Exemplars have been executed successfully. They appear as soon as the code is saved. Mouse-over them in Eclipse and we’d see details of the tests run. If the tests had failed, we’d have seen purple crosses instead, the mouse-over would show what went wrong, and the file and project would be marked as having errors in the Package Explorer and Problems views (just like compiler errors). You might have also noticed the three lines of code shaded red in the catch block at the end of the example. This is Sureassert’s continuous in-line test coverage reporting at work – they’re red because this code isn’t covered by any Exemplars or JUnits.

Hopefully this article has provided a useful introduction to Sureassert UC and the concepts that inspired its creation. If you’re interested in learning more, there’s a full guide and downloads available at the website: www.sureassert.com.

---

Unit Testing Articles

Writing Testable Code

More Software Testing and Java Knowledge

Unit Testing Tools

Java Tutorials and Videos

Software Testing Magazine

Software Testing Books

Growing Object-Oriented Software, Guided by Tests

Implementing Automated Testing

---

Click here to view the complete list of tools reviews