|
Software Development Magazine - Project Management, Programming, Software Testing |
|
Scrum Expert - Articles, tools, videos, news and other resources on Agile, Scrum and Kanban |
Cuke_sniffer - Static Analysis for Cucumber
Robert Cochran, @cochrarj
cuke_sniffer is a free open source static analysis tool for Cucumber that identifies smells and issues in a project. Cucumber is a tool that executes plain-text functional descriptions as automated tests in a behavior-driven development (BDD) style.
Web Site: https://github.com/r-cochran/cuke_sniffer
Version tested: 0.0.8
System Requirements: Windows, OSX, Linux with Ruby 1.9.3 or later
License & Pricing: Free, MIT license
Support: Issue tracker and wiki at https://github.com/r-cochran/cuke_sniffer
Cucumber was introduced to the software profession as a way for teams to explore and refine their practice of Behavior Driven Development(BDD). Product Owners, Developers, and Testers are given the flexibility to discuss what the system needs to do in business terms in the context of the problem being solved. However, too often teams see Cucumber as an automation solution and treat it as such; creating scenarios that are long and drawn out with reused steps instead of short and concise stories. As much as we value Cucumber as a communications tool to bring the team and business together, it is still code and should be held to a quality standard amongst the team.
cuke_sniffer was born out of the frustration of dealing with unhealthy Cucumber projects that are hard to read and maintain. The ultimate goals are to educate and to empower people who want to use Cucumber. In order to understand the problem we have to see how it happened and think about what could have been done better. Was it the junior team member who didn't know better? Was it the team member who only cared about a passing test and started to copy and paste everything? Was it me in my learning of a feature in Cucumber and over using it, just to feel the pain of it later? (I'm looking at you nested step definitions).
With the freedom to communicate anything in a story, people tend to paint themselves into a corner of anti-patterns and abuse. Like any code base, Cucumber projects can accumulate technical debt and begin to rot. Some projects just build on top of that rot with little consideration of the difficulties they will have to answer to in the future.
The concepts of using Cucumber are simple enough. You have a Feature file that has scenarios that tell a story in a given/when/then order which is linked to code that tests the application. Here is an example of what you might find in a basic introduction to Cucumber:
Feature: Buying a sandwich Scenario: As a customer I can buy a sandwich with my credit card Given I have ordered a sandwich When I pay with my credit card Then I get my sandwich
Cucumber can be pretty simple to use. Now, anyone who has been using Cucumber in an enterprise or team environment can tell you that the Scenarios don't always stay this simple. The business now needs to specify what credit cards can be accepted, what sides can be sold with the sandwich, and what drinks are offered. It would be nice to identify all the scenarios when the feature is written but these enhancements will happen over the life of the project. Your nice feature for buying a sandwich with a credit card might transform into something like this:
Feature: Buying a sandwich @wip Scenario Outline: As a customer I can buy a sandwich with my credit card #Given I have ordered a sandwich When I have ordered a grilled cheese sandwich And select a root beer from the drink drop down And check the pickle check box And chilli cheese fries #When I pay with my credit card Given I pay with mycredit card When I click the submit button Then I get my sandwich And the card is billed automatically Examples: | card_name | #1 | Visa | #2 | Master Card | #3 | American Express | #4 #| Debit | #5 ... | Discover | #25
It's not pretty and as much as we might not like to believe it, there is probably a scenario like this in our project. We might be able to forgive that it is clearly a work in progress, but the @wip tag is often forgotten when the scenario is completed. The scenario is out of the Given/When/Then order. There are entirely too many steps to manage; if a scenario is more than 3-5 steps long it is too complex or can be more concise. There are steps that are commented out and serve no purpose. Our scenarios should tell a story and not specify directly what the system needs to look like or react to. Implementation words like drop down, check box, and button distract from the story. The examples table is way too large and for some reason Debit is no longer a valid example. Without the review of our Cucumber projects we might simply miss or ignore these problems.
cuke_sniffer is my attempt to automate the feedback and identification of these issues as well as give a representation of the size and health of a Cucumber project. To help quantify the health of the various parts of Cucumber I chose to use a golf like scoring system, more points mean more areas to improve. Unlike golf, the score of a project is a means to monitor growth and improvement of a project and really can't be compared against others.
When cuke_sniffer is used against the ugly example we are automatically informed about several of the issues I mentioned before and more. cuke_sniffer will give a description of the issue found as well as the number of times it was found.
Improvements to make:
(2) Commented step.
(1) Scenario Outline with too many examples.
(1) Implementation word used: drop down.
(1) Given/When/Then used multiple times in the same Scenario.
(1) Implementation word used: click.
(1) Scenario steps out of Given/When/Then order.
(1) Scenario with too many steps.
(1) Commented example.
(1) Implementation word used: button.
(1) Tag appears on all scenarios.
With this information about our scenario we can begin to identify what needs the most improvement. We can educate ourselves on why the rule fired and how to prevent causing more smells in the project. With just knowing about the problem and being mindful not to cause it to continue you have given yourself a fighting chance of cleaning up the project. The improvements are identified when cuke_sniffer goes over your Features/Scenarios/Step Definitions/Hooks files and executes rules against them.
Rules are divided up into 4 categories: information, warning, error, and fatal. Information rules are things that are minor issues that can be situational from team to team and are meant only as advice for improvement of the projects use of Cucumber. Warnings are issues that impact the readability or the usability for parts of your cucumber project. Errors are meant to be a way to help defend projects from debugging problems that are difficult to catch. Lastly, Fatal issues are things that will prevent your project from even executing. The full list of rules can be found on the cuke_sniffer github wiki: https://github.com/r-cochran/cuke_sniffer/wiki/Rules-list.
cuke_sniffer allows for the customization of these rules. If you don't agree with a rule, it can simply be turned off. Maybe the phrasing of the rule doesn't make sense and you want to rewrite it. All components of a rule are customizable. If you think of a new rule that is not found in cuke_sniffer you are able to write the rule and include it. If you do find yourself writing rules let the cuke_sniffer team know and we can include it in future releases to share your work with others! For more details on customizing or writing rules consult the cuke_sniffer github wiki: https://github.com/r-cochran/cuke_sniffer/wiki/Customizing-Rules.
Just because a rule fired against a Scenario does not automatically mean it is bad. cuke_sniffer specifies a customizable threshold of points that must be met for a Cucumber object to be considered bad. Customizing this threshold allows for you to slowly slide the bar up or down for your Cucumber objects. At the start of improving your Cucumber project you might slide the threshold higher to set a goal of making all Cucumber objects good. As time goes on you might slide the threshold down to continue doing improvements.
When cuke_sniffer is ran it will give you the list of improvements as well as the score but it will also give you a breakdown of the various Cucumber objects. You will learn about what your worst and best scores are as well as the average of your Cucumber objects. Additionally, you will be given an idea of what percentage of your Cucumber objects are good or bad.
While Features and Scenarios are the most visible and problematic objects in Cucumber, Step Definitions are where we spend most of our time and they have unique problems. Steps Definitions have the inherit intention of being reused for multiple scenarios. If I am describing how a certain feature works with several scenarios I am most likely going to reuse my setup(Given) or action(When) steps. However, as a developer I will want to reuse any existing Step Definitions. Suppose I find a Step Definition that I did not write that looks like it matches exactly what I want to do. Hooray! I don't have to do more work! I run my test and find that it fails on the step that I reused.
Now I have a problem, I have a Step Definition that doesn't work for my use case and I am left with several options. I could just not use it and write an entire new Step Definition from scrap or I could spend the time to debug the existing Step Definition and hope that my changes don't break anything else. The real reason for the Step Definition not working could be that it never worked or that it was replaced with another Step Definition. It would be very nice if the Step Definition was removed after no one was using it anymore.
cuke_sniffer will catalog all Step Definition calls in a project and identify any that would be considered dead. With this information we can now quickly remove or spot check the Step Definition. By removing the dead Step Definitions we greatly reduce the amount of noise in our Step Definition files and prevent the same scene to play out again. If the decision is to debug the Step Definition we can now have confidence that the changes will not break other tests.
The first time you look at the cuke_sniffer results for your project you might feel overwhelmed. Remember that you don't have to fix everything at once; incremental change and understanding of the problem can often be better than just fixing the problem.
Understand why an improvement would be suggested and try and not to cause more of that issue to occur. Take a report from the start of the week and compare it to a report from the end of the week. Did the total for that improvement go up? Find areas where the improvement is suggested and make the fixes one at a time. Before too long you will be happier with the health of your Cucumber project.
