Click here to view the complete list of tools reviews

This article was originally published in the Summer 2010 issue of Methods & Tools

---

VoodooMock: Mock Objects Framework for C++

Shlomo Matichin

VoodooMock is a mock object framework that fully automates the repetitive tasks associated with writing mock objects for C++ unit tests. Similar to other frameworks, VoodooMock provides an expectation storage and verification engine. But unlike other frameworks, VoodooMock also automates the generation of stubs, and rerouting calls to the stubs, without requiring any macros or unnecessary interfaces. VoodooMock is available under the GPL, and is successfully deployed in at least three production projects.

Web Site: http://sourceforge.net/projects/voodoo-mock
Version Tested: 0.3
System Requirements: Python 2.5 or higher
License and Pricing: GPL
Support: Through the SourceForge project homepage
Installation: just unzip
Documentation: contained inside the package
Configuration: works out of the box

Introduction

When writing unit tests based on mock objects, there are two main implementation details to address: Interception, or how method calls are forwarded to the mock object, and the Expectation engine, or how to store and verify expected events.

Other frameworks do not handle interception. The test writer usually implements interception by using an abstract class inheritance ("interface class") and a factory (even when there is only one class that inherits the interface). The factory is replaced at the unit test with a stub that returns a stub object instead. The stub object forwards the calls to an expectation engine. All this code is written manually. It also has a demoralizing effect of testing everything through as much existing interfaces as possible, to avoid this type of repetition.

VoodooMock addresses interception differently. A pre-build phase of the test suite build system, the VoodooMock parser generates a parallel directory hierarchy of the project headers, each replaced with an automatically generated stub for all the classes and functions of the original header, the voodoo header. The voodoo header contains ifdef directives to allow including the original header instead. So by just defining some macros before the original code is included in the test, the tester can select which classes are to be replaced by stubs. The stubs in turn, contain full implementation for forwarding all access to the expectation engine.

A drawback for this approach, is that the build system must enable this pre-build support, and compile each test-suite into a different binary file (the stub and real class cannot coexist in the same binary, having the same name and namespace). A test harness that runs all these separate binaries is also recommended. The VoodooMock project at sourceforge contains a sample harness written in python, and a Jamrules file (‘Jam’ is a ‘Make’ like language), as a reference.

However, this approach eliminates the need for virtual interfaces, or any other real code modifications (e.g., no macros are needed). One project using VoodooMock was a kernel mode driver, where unnecessary virtual interfaces were unwelcome.

VoodooMock’s expectation engine uses Scenario objects, which contain a half textual half object oriented description of what all the stubs in the test are expected to verify, and in what order. A tester will spend most of his time writing these scenarios.

VoodooMock is called voodoo, because it uses bad practices, hacks, and black magic to implement all of its features (i.e., globals, god objects, void pointers, and abusing the header inclusion order). Testing code is allowed to contain hacks to make the test work, but VoodooMock’s approach isolates the hacks into the generated code, and out of the test code. Generation of the stub code also means that once a hack is used, it is consistent across all stubs in the test suite. A lot of hacks were designed in order to allow full support of inheritance and templates. For example: an inheriting class can be tested as a different unit than its ancestor, which is stubbed.

A Short Tutorial

The following example provides a quick tutorial of using VoodooMock. The example contains two classes: Number, which is just an object wrapper for unsigned, and Sum, which performs addition on two Number s. The contents of the file Number.h:

#ifndef __NUMBER_H__
#define __NUMBER_H__
class Number { public:
Number( unsigned value ) : _value( value ) {} unsigned value() const { return _value; } private: unsigned _value; };
#endif

And the content of Sum.h:

#ifndef __SUM_H__
#define __SUM_H__
#include <Number.h>
class Sum { public:
Sum( const Number & first , const Number & second ) :
_result( first.value() + second.value() )
{
}
unsigned result() const { return _result; } private: unsigned _result; };
#endif

A test for the unit that contains only the class Sum (stubbing the Number class), can be:

#include <stdio.h>
#define VOODOO_EXPECT_Number_h
#include "Sum.h"
using namespace VoodooCommon::Expect; class TestFailed {};
int main()
{
Scenario scenario;
scenario <<
new CallReturnValue< unsigned >( "Fake Number 1::value" , 100 ) <<
new CallReturnValue< unsigned >( "Fake Number 2::value" , 200 );
FakeND_Number number1( "Fake Number 1" );
FakeND_Number number2( "Fake Number 2" );
if ( Sum( number1 , number2 ).result() != 300 )
throw TestFailed();
scenario.assertFinished();
printf( "OK!\n" );

Lets examine the code in detail. First, the following two lines:

#define VOODOO_EXPECT_include_Number_h
#include "Sum.h"

The first lines tells the preprocessor, that if Number.h gets included, use the stub code, instead of including the real Number.h. The macro name is derived from the header file name, by the VoodooMock stub tree generator. The second line includes Sum.h, which actually includes the stub Sum.h first, and since VOODOO EXPECT include Sum.h is not defined in this test, the stub simply includes the original Sum.h. Sum.h includes Number.h, and this is when the macro defined in the first line makes a difference. Note: the macro name is created from the relative path of the header file from the project root. In this case, the folder ‘include’ is the prefix to the header file name. The test starts with the following lines:

Scenario scenario;
>scenario <<
new CallReturnValue< unsigned >( "Fake Number 1::value" , 100 ) <<
new CallReturnValue< unsigned >( "Fake Number 2::value" , 200 );

This snippet creates a scenario object, and fills it with the following two events: First, the method ‘value’ of a stub object with the name ‘Fake Number 1’ will be called, with no parameters. It will return an unsigned, with the value 100. Then, the method ‘value’ of a stub object with name ‘Fake Number 2’ will be called without parameters. This time it will return 200. The next section explains in detail about the scenario object and what it can store. The code continues:

FakeND_Number number1( "Fake Number 1" );
FakeND_Number number2( "Fake Number 2" );

The above snippet creates two Number objects, bypassing the construction phase, and naming them in the process. More about the ‘FakeND ’ prefix in the next section. Note that the objects are created after the story already references them. The example continues:

if ( Sum( number1 , number2 ).result() != 300 )
throw TestFailed(); // But C++ test suites usually write this as 
// TS_ASSERT_EQUALS( Sum( number1, number2 ).result(), 300 );

This snippets actually calls Sum, first constructing it over the two stub objects, and then calling the ‘result’ method, and finally asserts the result. The constructor Sum::Sum first calls the ‘value’ method of the object ‘number1’, which has the textual name ‘Fake Number 1’, that matches the first event in the only alive scenario object. Therefore the return value is fetched from that event, and the scenario object advances to the second event. Sum::Sum continues to call the method ‘value’ of the second object, matching the second event, and completing the scenario.

The last line of interest:

scenario.assertFinished();

This line verifies that the scenario object ‘scenario’ has finished matching all of its events. VoodooMock configuration file defines how this assertion is implemented (which is usually just TS ASSERT).

Summary: this example shows how to:

• Create stub objects without using the constructor.
• Create a scenario recording, run it, and verify it was complete.
• Assert the unit gave a correct result under that scenario.

Features Of The Expectation Engine

This section describes the different capabilities of the VoodooMock expectation engine, without giving any source code examples. An expectation is one of the following "events":

• Construction or Destruction of a stub class
• Call to a stub object method
• Call to a global function

The expectation engine is responsible for storing, running and verifying "recordings" of expectations. Each event has a textual representation (e.g. "Construction of std::list<unsigned>"). When an event occurs (e.g., a stub class is constructed), the engine searches for matching expectation in the alive Scenario and Always objects (see next subsection). In order to match, the expectation textual representation must match the event textual representation, plus the parameters must also pass their verification (stored in the expectation - see the ‘Parameters’ subsection).

The features of the expectation engine are described in the following subsections.

Scenarios

Scenario objects store expectations, meant to run one by one and only once (e.g. first create the file, write to it, then close it). Before a Scenario object is destroyed, one of the following methods must be called: ‘assertFinished’ will assert all the expectations were matched, or ‘assertNotFinished’ otherwise.

Several Scenario objects can coexist, representing independent "recordings" of expectations. If both Scenario objects currently point to two expectations matching the current event (in both textual name, and parameters verification), the Scenario object constructed last takes precedence. Only the return value from that Scenario object will be used, and only it will advance to the next expectation. This behavior is useful in test driven development: first write a "good" scenario, and when writing the "bad" cases, use the "good" scenario, but only "override" the failing event.

‘Always’ objects are lists of unordered expectations, that can be called zero or more times (e.g., gettimeofday and logging). Several Always objects can coexist, together with Scenario objects. Scenario objects always takes precedence over Always objects. When in conflict, the last Always object created take precedence over other Always objects. If the expectation conflict is inside a single Always object, the expectation last added to the object takes precedence. This behavior provides a similar facility for "good" and "bad" scenarios: create a single Always object for the "good" test, and fail each call one by one in other tests.

‘ExpectationList ’ is a container of expectations, that is not connected to the expectation engine (i.e., expectations stored in it will not be matched). It allows creating "subscenarios" to be added into a Scenario or an Always as a group (used for dividing scenario building into functions).

All three containers support simple editing. Editing a scenario helps writing readable tests: most of the tests will only describe the "difference" from one of the major testing scenarios. The editing primitives include:

• Iterating containers
• Deleting or replacing expectations
• Insert ExpectationList s

Expectations

Expectations are what the tester spends most of his writing time. Each expectation matches an event the tested unit must or can do. VoodooMock is shipped with the following expectations:

• Construction expectation: ‘scenario << Construction<class>( "Name to give new object" );’
• Destruction expectation: ‘scenario << Destruction( "Name of object being destroyed" );’
• Call to a global function: ‘scenario << CallReturnVoid( "Namespace1::Namespace2::function" );’
• Call to a stub object method: ‘scenario << CallReturnVoid( "Stub objects name::method name" );’

Once a call matches an expectation, the expectation contains instruction as to what to provide as a return value. Storing the return value in the expectation also raises the question of scope for the return value stored. VoodooMock provides several calls expectations that only differ between them in the way the return value is stored. A call expectation looks like this: ‘CallReturn[ReturnValueScope]( "Stub object::method", value )’. Simple examples include:

• ‘CallReturnVoid’ is for calls returning void.
• ‘CallReturnValue’ copies the return value once when creating the expectation, and a second time when the call return. This works best for immediate values (e.g., integers).
• ‘CallReturnReference’ stores a reference to the provided return value, nly copies it when the call return. This works best for functions like ‘gettimeofday’ returning a global variable representing "now".
• ‘CallReturnAuto’ receives a pointer, copies what it references as the return value, but also deletes the pointer when the scenario is destroyed.
• ‘CallReturnCallback’ fetches the return value from a given functor.

Implementing additional call expectations is a matter of inheriting from the Expectation hierarchy.

Each expectation can also contain one or more "hooks"; functors to run after they are matched. The hook notation is similar to the parameters notation (see below).

When no expectation matches an event (e.g., the unit called a stub method which were not supposed to be called at this point), VoodooMock fails the test suite with a message containing the next expectation of every Scenario object, by line number created, and index inside the Scenario.

Parameters

Except for destruction, each expectation also describes the parameters it expects, and how to verify their correctness. For example:

const Serializable * secondObject; scenario <<
new Construction< DataFile >( "The Data File") <<
new EqualsValue< const std::string >("C:\\\\data.dat" ) <<
new Ignore< int >() <<
new CallReturnVoid( "The Data File::serialize" )<< new Named< const Serializable >
( "First Object" )<< new Destruction( "The Data File" );

Each parameter verifier is strongly typed. It will not match a different type. Overloading is supported by expecting different parameter types. Note: when using default values for parameters, the default values must also be verified. ‘EqualsValue’ verifies the parameter is equal to the value expected. Like return values, the issue of the scope for the stored value is addressed by providing multiple such verifiers (e.g., ‘EqualsReference’, ‘EqualsAuto’ and so on). The ‘Ignore’ verifier just checks the type of the parameter, but does not access it. The ‘Named’ verifier assumes the type provided is a stub class and verifies the name of the object. There are more parameters verifiers, and new ones are easily implemented.

Naming

Object names help writing readable tests. For example, a stub file object called ‘Log File’ and a second one called ‘Data File’. Each stub class exports a method ‘voodooInstanceName’ as a getter for the object name.

Whenever a stub object is copy constructed, the new object will be named "Copy of [first objects name]". This is useful for covering exactly which object are copied, and how many times. In case where the tester wishes to ignore wither the object was copied or not, VoodooMock provides a similar version for every expectation type that ignores the "Copy of" prefix (or prefixes). For example: ‘CallOrCopyOfReturnVoid( "Data File::write" )’ will also match the event "Call to Copy of Data File::write", and "Call to Copy of Copy of Data File::write".

Almost every unit, apart from creating objects and receiving them through calls, also accept them as arguments to the interface it exports. VoodooMock provides a feature to create stub objects inside the test suite without calling their normal constructor, and without passing through the expectation engine (and therefore no expectation for their construction is required). It does so by creating another class alongside every stub class, prefixed by ‘Fake ’ (e.g., for a class called ‘Connection’, another class called ‘Fake Connection’ will be generated). When constructing such an object, the test code provides the textual name for the object.

A ‘Fake ’ object can be passed to the unit whereever the original class is used (implemented by derivation, and without adding new members, to allow copy construction).

When a ‘Fake’ object is destroyed, the destruction event still goes through the expectation engine, and is therefore appropriate when handing over the ownership of an object to the unit, since it’s destruction is part of the unit’s behavior. If the unit does not own the object (e.g., receives it by reference), a second prefix, ‘FakeND ’ is more useful: the destruction event also does not invoke the expectation engine.

Externals

Stubbing OS provided global function presents two new problems: First, the OpenCCore parser (the C++ parser used by VoodooMock) can not handle OS level header files (which usually contain compiler specific macros). But even worse, the include file hierarchy is never trivial (e.g., ‘windows.h’ includes other files which actually define the system calls).

VoodooMock has a feature specifically designed for this case: provide an "alternative" header to parse, and generate stubs according to. The fake header file then always includes the original, and implements the stubs, in another names-pace, named ‘External’. For example, the alternative header file for ‘windows.h’ might look like this:

HANDLE WINAPI CreateFileA(
__in LPCTSTR lpFileName,
__in DWORD dwDesiredAccess,
__in DWORD dwShareMode,
__in_opt LPSECURITY_ATTRIBUTES lpSecurityAttributes,
__in DWORD dwCreationDisposition,
__in DWORD dwFlagsAndAttributes,
__in_opt HANDLE hTemplateFile
);
BOOL WINAPI CloseHandle(
__in HANDLE hObject
);

And the real code must be modified, from using ‘CloseHandle(...)’, to ‘External::CloseHandle(...)’. Now the only thing that remains is to define ‘External’ as a macro that expands to an empty string when compiling the real product.

Programmable Mock Objects

VoodooMock has a second implementation to every stubbed class, that allows complete interception of all events into methods. This implementation is faster, and more suitable for crunching tests. For more information, refer to the VoodooMock documentation.

Methodology

VoodooMock was design with the eXtreme Programming methodology in mind. This section details how one XP team used it effectively.

Selecting a unit size is an art by itself. Experiment with different unit sizes, since there is no single answer to the unit size issue. Making each class a unit provides excellent coverage with ease, but will force testing wrapper and adapter classes, and will not cover integration between classes.

When writing a test suite, use a setup and teardown routines to build and destroy the ‘Fake ’ dependencies, and to create the "normal" Always object and it’s expectations.

Scenarios tend to grow quite rapidly. Split them into smaller subscenarios by using functions (either return an ExpectationList from a function, or append to an existing scenario). Practicing this correctly creates readable tests.

Carefully consider if an expectation should be inside a Scenario object, or an Always object. As a rule of thumb, global getters and locking should reside inside an Always, and the core unit behavior should be a Scenario. When in doubt, prefer to use Always. Consider the following case: the unit writes a string to a file. Should the test check only the file content, or how many times it might have been overridden or appended? The answer of course is "it depends on the requirements". But consider that the first alternative will not require changing the test when refactoring to use some other formatting engine, which might decide to split the write to several smaller ones.

Finally, reuse scenarios. Make the good case the first test, then only override it (more than one way) at the failure point, to cover failures. Split stories into simple smaller stories and reuse these "building blocks" to create your good cases. Same as in software development.

Set Up

VoodooMock uses a binary python module that wraps the Open C Core parser. The official release includes this module in binary form for Python 2.6 under windows. Other platforms will require building the module. The VoodooMock distribution includes the sources for rebuilding it, and makefiles for GCC and MinGW (inside the archive named wrapocccore.tbz2). Warning: The OpenCCore parser still has many bugs.

An example VoodooMock generator command line invocation:

tools\\voodoo\\multi.py --input=cpp --output=voodoo
--exclude=cpp\\client\\Core.h --ignore="\\bIN\\b" 
--ignore="\\bOUT\\b" --ignore="\\bINOUT\\b" 
--ignore="\\bOPTIONAL\\b" --only-if-new

The ‘–input’ argument points to the first directory to scan for header files. The ‘– output’ argument points to a directory where the header files mirror tree will be created. ‘–exclude’ excludes a file from being mirrored. This is useful when the original code contains include magic, or when the OpenCCore parser crashes on a specific file. ‘–exclude’ might be repeated as many times as necessary. ‘–only-if-new’ will only regenerate stub header files if the modification time stamp on the original header file is newer (similar to make). A must when using a system that detects header file dependencies. ‘–ignore’ is a list of regular expressions to erase from the code before parsing it. Since the OpenCCore parser parses the files before preprocessing, any macros or annotations must be removed before a valid C++ syntax remains.

The last thing to verify is that the directory ‘voodoo’ precedes the directory ‘cpp’ in the include path, when building the tests.

Conclusion

A good unit test is one which reflects the requirements, and is readable. A bad unit test just reflects the code itself, and is implemented in a 400-lines long test case functions.

VoodooMock’s influence on unit testing mentality is similar to the influence script languages have on software development: It solves the little problems so the coder can focus on progressing rapidly. However, without the commitment to produce high quality code, script languages sometimes just help the coder accumulate a big pile of crap, quite rapidly as well. Using VoodooMock without this commitment for high quality tests, has the same effects. Its common to see a "mirroring" test, where each line in the scenario just rewrites a line of code.

As a personal advice: VoodooMock is only one tool, out of many I have written over the years to help me test code more efficiently. I have found it rewarding to keep developing my working environments and coding tools. Note that while VoodooMock is a complete package by itself, I warmly recommend working on extending and adapting it to your needs. My must have customization list includes: stack trace on expectation verification failure (redefine VOODOO TS ASSERT in VoodooConfiguration.h), a shortcut for running a single test and rerun all tests on each build.

---

More Software Testing Content

• Software Testing Magazine
• Software Testing Videos and Tutorials
• Functional Testing Tools
• Software Testing and Quality Portal

---

Click here to view the complete list of tools reviews