Cucumber - Behavior Driven Development for Ruby

Richard Lawrence, Humanizing Work, http://www.humanizingwork.com/

Cucumber is an open source tool to support the Agile testing approach of Behavior Driven Development (BDD) with plain text specifications and unobtrusive automation in Ruby. Alternative implementations of Cucumber exist for Java, .NET, and several other platforms.

Web Site: https://github.com/cucumber
Version tested: 1.1.2
System requirements: Windows, OSX, Linux with Ruby 1.8.7 or later
License & Pricing: Free, MIT license
Support: Issue tracker and wiki at https://github.com/cucumber/cucumber, mailing list at http://groups.google.com/group/cukes

Bertrand Russell once wrote, "Everything is vague to a degree you do not realize till you have tried to make it precise." Herein lies one of the core problems of software development. Developing software is fundamentally an exercise in making the vague and unknown - the stuff of wishes, ideas, and conversations - sufficiently precise to make a machine behave properly.

Behavior Driven Development (or BDD) is an approach to software development designed to address just this problem. How do we discover what our software ought to do, specify it clearly, and validate that the software does and continues to do what we intend? With BDD, we begin development of each feature with the desired new behavior, specifying that behavior with concrete examples, and making those examples executable as automated tests. Only after we have a failing test - a desired behavior currently unsatisfied by the system - do we consider implementing the behavior.

When Dan North originally proposed BDD, it was an answer to issues he had doing and teaching Test-Driven Development. At this time, the focus was at the class and method level. Later, BDD grew to encompass requirements and analysis, and the emphasis moved to describing behavior at the system level. (See http://dannorth.net/introducing-bdd/ for more BDD background.)

Today, there are several tools that support BDD. I have used and taught a handful of them, including FitNesse, JBehave (the original BDD tool), Concordion, and Cucumber. Of the BDD tools I have worked with, Cucumber is by far my favorite.

Cucumber was originally a Ruby tool. It grew out of the Ruby unit-level BDD framework rspec. The Ruby version of Cucumber is still the reference implementation and the most widely used, but there are now versions for Java, .NET, JavaScript, as well as several other languages and platforms. I'll introduce the Ruby version and then briefly compare it with the Java and .NET versions.

Installation

Cucumber is installed with Ruby's package manager, RubyGems. Assuming you already have a current version of Ruby (1.8.7 or 1.9.3 as of this writing), to install Cucumber simply open a command window and run

gem install cucumber

This will install Cucumber along with its dependencies.

Note: If you intend to use Cucumber under Ruby on Rails 3.x, you'll want to install Cucumber using Bundler instead. See the Cucumber wiki for details.

Features and Scenarios

Specifications written for Cucumber have two parts: feature files containing scenarios expressed in the Gherkin language and Ruby files containing unobtrusive automation for the steps in the scenarios.

Feature files begin with a feature title and description:

  Feature: Medical Provider Search
    In order to avoid paying out-of-network fees for 
    medical care, insurance policy holders (aka members) 
    want to find medical providers covered by their 
    insurance for particular specialties, locations, etc.

This would be followed by a number of scenarios to elaborate how the feature ought to behave. Scenarios are generated as product people, developers, and testers discuss the feature. They'll ask questions like, "What's an example of a simple search?" "What's the next most important example?" and "What kind of change to the context would produce different results from that same action?"

Teams typically discuss scenarios in natural language first. Then, they express the scenarios more precisely using Gherkin's scenario structure:

  Given <some initial context>
  When <an event occurs>
  Then <ensure some outcomes>

Each line in a scenario is called a step. There may be more than one step of a particular type, in which case the keyword "And" or "But" is used instead of "Given", "When", or "Then". If no initial context setup is required, there may be no "Given" step.

After we add a few scenarios, the feature for the Medical Provider Search might look like:

  Feature: Medical Provider Search
    In order to avoid paying out-of-network fees for 
    medical care, insurance policy holders (aka members) 
    want to find medical providers covered by their 
    insurance for particular specialties, locations, etc.
    
    Background:
      Given I'm logged in as a member
      
    Scenario: Policy Holder Can Get to the Search Form
      When I click "Find a Provider"
      Then I should see the provider search form
    
    Scenario: Empty Search
      Given no available providers
      When I search without specifying any search criteria
      Then the results should indicate, "No matching providers found"
  
    Scenario: Search by Specialty and Location
      Given the following providers:
        | Name  | Provider Type | Specialty        | ZIP   |
        | Jones | Doctor        | General          | 90010 |
        | Smith | Doctor        | Endocrinology    | 90010 |
        | Khan  | Therapist     | Physical Therapy | 90010 |
        | Cho   | Doctor        | Endocrinology    | 80113 |
      When I search for a provider with the criteria:
        | Provider Type     | Doctor        |
        | Specialty         | Endocrinology |
        | ZIP               | 90010         |
        | Search Radius     | 5 miles       |
      Then the results should include only provider Smith

The Background section describes any common context to be established before each scenario. The first scenario doesn't need any additional context, so it only includes When and Then steps, while the other scenarios set up context regarding the available providers. The third scenario contains more complex data; it uses tables to structure that data in a readable way.

Step Definitions

Cucumber scenarios become automated tests with the addition of what are called step definitions. A step definition is a block of code associated with one or more steps by a regular expression (or, in simple cases, a string). Two step definitions for the scenarios above might look like this:

  Given "I'm logged in as a member" do
    visit home_page
    fill_in 'Username', :with=> 'testmember'
    fill_in 'Password', :with=> 'Passw0rd'
    click_button 'Sign in'
  end
  
  Given /^the following providers?:$/ do |providers_table|
    Provider.delete_all
    providers_table.hashes.each do |row|
      Provider.create :last_name=> row['Name'], 
        :provider_type=> 
          ProviderType.find_by_name(row['Provider Type']),
        :specialty=> Specialty.find_by_name(row['Specialty']),
        :zip=> row['ZIP']
    end
  end

The body of each step definition is just Ruby code. The first example uses the web application driver library Capybara to interact with a web page. The second example uses Rails' ActiveRecord to set up particular providers in the database. We could just as easily run a command line application, work with the file system, or call a REST service if we needed to.

Capture groups in the regular expression become arguments to the step definition. For example, if we wanted to extend the first step definition to support multiple roles, we might modify it to look like the following:

  Given /^I'm logged in as an? (.+)$/ do |role|
    credentials = {
      'member'=> { 
        :username=> 'testmember', :password=> 'Passw0rd' 
      },
      'admin'=> { 
        :username=> 'testadmin', :password=> 'secret123' 
      }
    }
    unless credentials.keys.include?(role)
      throw "Unknown role: #{role}"
    end
    
    visit home_page
    fill_in :username, :with=> credentials[role][:username]
    fill_in :password, :with=> credentials[role][:password]
    click_button 'Sign in'
  end

Running Scenarios and Integrating with Other Tools

Automated tests are only useful if you can run them. The main interface for running Cucumber scenarios is the cucumber command line executable that installs with the Cucumber gem.

Like many Ruby tools, Cucumber is opinionated: it assumes you organize your features, step definitions, and supporting code in a particular way. You can override the defaults, but life is easier if you don't. The standard directory structure is:

features - Contains feature files, which all have a .feature extension. May contain subdirectories to organize feature files.

features/step_definitions - Contains step definition files, which are Ruby code and have a .rb extension.

features/support - Contains supporting Ruby code. Files in support load before those in step_definitions, which makes it useful for such things as environment configuration (commonly done in a file called env.rb).

In a command window, simply navigate to the directory above your features directory (your project directory in a Rails application) and run cucumber to use the default options. Cucumber will attempt to run every scenario in every feature file in the features directory, looking for matching step definitions in step_definitions. When it finds a match, it will execute that step definition. When it can't find a match, it will suggest code you could use to create a matching step definition. Passing steps are colored green, failing steps red, and undefined and pending steps yellow. When a step fails or is undefined, Cucumber skips to the next scenario and colors the skipped steps cyan.

You can specify many options on the command line beyond the defaults. For example, suppose you're working on a story that includes scenarios in two feature files. You might tag those scenarios with @current to say, "These are the scenarios I'm currently working on." On the command line, you could run cucumber --tags @current to run just those scenarios. Another example: You might want to run Cucumber with different output. You could specify cucumber --format html --out output/report.html to get a nice looking HTML report. Or you might use --format junit to get a JUnit-style report to integrate with a continuous integration server. There are too many options to list here; run cucumber --help to see them.

While most users run Cucumber from the command line, Cucumber does integrate with editors such as TextMate and JetBrains RubyMine. Both the TextMate bundle and RubyMine plugin provide syntax highlighting, formatting, code completion, and the ability to run features and scenarios inside the editor.

Why I Like Cucumber

Cucumber is optimized for BDD. It supports a particular set of interactions between team members and stakeholders. It's not optimized for testing (though I think it does that part of its job sufficiently well).

When I see explanations of new tools aiming to "fix Cucumber's problems" (e.g. Spinach and Turnip in recent weeks), I note that what the authors consider Cucumber's problems are often things I consider Cucumber's strengths. I infer that they want Cucumber to be a different kind of tool for a different kind of purpose.

So, what do I like about Cucumber?

Separation of examples and automation - Cucumber's unobtrusive automation means product people are more likely to engage with features and scenarios than if code were mixed in. Whether or not product people actually write scenarios, they should be able to read, verify, and adjust them.

Some structure but not too much - The Given-When-Then syntax of Cucumber scenarios imposes some structure on scenarios while leaving plenty of room for teams to grow their own language to describe their system. Similarly, the use of regular expressions for mapping between steps and step definitions leaves you room for flexibility in language but naturally limits how much flexibility you employ, lest the regular expressions become unreadable.

Pressure to grow a ubiquitous language - One of the apparent weaknesses of Cucumber as a test automation tool is that step definitions are all global. This quickly reveals, via ambiguous step matches, whether you use the same words to mean more than one thing in your domain. This pressure, plus the precision of specification by concrete examples, helps a team grow a precise language to express their domain.

Just enough syntax - The Gherkin language walks a fine line between natural language and a programming language. The Background and Scenario Outline features support simple refactoring to remove excess duplication. But more complex structures such as procedure calls and includes are left out because they would hurt readability for non-programmers.

Potential Drawbacks

Extra overhead - Compared to writing tests in a general-purpose language like Ruby, Cucumber introduces extra overhead. If you're not going to have the BDD-style interactions, if you're not going to have non-programmers read scenarios, and if you're disciplined enough to use domain language in your tests, you may prefer to write functional tests in test/unit or rspec.

Regular expressions - I consider regular expressions to be a strong point of Cucumber, but many people, even developers, are uncomfortable with them. If you use Cucumber, you'll find it hard to stay away from regular expressions, so this may be a reason to choose a different tool. To overcome this, I wrote an article and cheat sheet highlighting the most useful subset of regular expressions for Cucumber users.

See http://www.richardlawrence.info/2011/08/23/cucumber-regular-expressions-cheat-sheet/.

Cucumber for Other Platforms

Cucumber projects are available for other platforms beyond Ruby. Some use Ruby Cucumber with a bridge into the target language (e.g. cuke4php and cuke4lua). Others use the Gherkin parser but implement everything else in the target language.

For the Java platform, cucumber-jvm is a pure Java implementation of Cucumber. As of this writing, it's still under development. While cucumber-jvm fully supports Gherkin, it's missing many of the runtime features in Ruby Cucumber. See https://github.com/cucumber/cucumber-jvm.

Features and scenarios in cucumber-jvm look exactly the same as shown above - they still use Gherkin. A step definition in Java looks like this:

@Given("^the following providers?:$")
public void createProviders(cucumber.table.Table providersTable) {
  // create the providers in Java
}

Various other JVM languages are supported. The same step definition in Groovy, for example:

Given(~"^the following providers?:$") { 
  cucumber.table.Table providersTable ->
    // create the providers in Groovy
}

Cucumber-jvm scenarios can be run from the command line or with JUnit.

SpecFlow is a pure .NET implementation of Cucumber, again based on the Gherkin parser, with integration into Visual Studio 2008 and 2010. See https://www.specflow.org/. Step definitions in C# look much like those in Java:

[Given(@"^the following providers?:$")]
public void CreateProviders(SpecFlow.Table providersTable)
{
  // create the providers in C#
}

Scenarios in SpecFlow are generated into NUnit or MSTest unit tests in the background and run with whatever unit test runner you prefer. Many SpecFlow users use the Visual Studio unit test runner provided by JetBrains ReSharper.

Because Cucumber tests typically interact with the target application out-of-process and because the Ruby language and libraries are nice to work with, many teams use the Ruby version of Cucumber to test Java or .NET applications.

The Bottom Line

Teams using or considering the approach variously known as Behavior Driven Development, Acceptance Test Driven Development, or Specification by Example should look at Cucumber. It supports this approach better than any tool I have used. Teams simply looking for a functional test automation tool may find that other, more technical test frameworks fit their needs better.

---

More Software Testing Resources

• Software Testing Magazine
• Software Testing Tutorials and Videos

---

Click here to view the complete list of tools reviews

This article was originally published in the Winter 2011 issue of Methods & Tools