Build a Strong Testing Foundation with BDD and Testaify

TABLE OF CONTENTS

• Why is Behavior Driven Development (BDD) not used by most agile teams?
  
  • BDD Arrives
  • Early Days
  • BDD Adoption
  • BDD is Hard

• Behavior Driven Development and Testaify: A Perfect Pair for Quality Product Development

  • Business Facing & Support Programming
  • Business Facing & Critique Product
  • BDD & Testaify

Why is Behavior Driven Development (BDD) not used by most agile teams?

My journey in Agile software development began in 2004 and has been a testament to the evolution of testing methodologies. Before behavior driven development, I navigated through FIT (Framework for Integrated Test) using Fitnesse as the tool. This era was marked by peculiar tables like Column Fixtures, which posed a challenge. Fitnesse, in particular, was a poorly developed and unreliable tool. However, these obstacles didn't discourage us from creating thousands of Fitnesse tests at Ultimate Software, a testament to our resilience and commitment to improving our testing practices.

BDD Arrives

I was thrilled when behavior driven development arrived. I love the Gherkin DSL (Domain-Specific Language). Dan North created JBehave (Java) as the first implementation of BDD. Interestingly enough, the Ruby community initially embraced BDD faster than anyone else. Instead of creating weird tables, we can now express our tests in English, obviously, within the constraints of the Gherkin DSL. If you are unfamiliar with behavior driven development, all scenarios follow this basic structure:

• “Given” describes the preconditions for the scenario.
• “When” describes the action under test.
• “Then” describes the expected outcomes.

You can also use And and But keywords to join Given, When, or Then steps together more readably.

Here is an example:

1. Given I have $100 in my checking account

2. And I have $10 in my savings account

3. When I transfer $50 from my checking account to my savings account

4. Then I have $50 in my checking account

5. And I have $60 in my savings account

That is a behavior driven development scenario. You can group several scenarios into a single feature file. I am not going to explain all the aspects of BDD. Instead, I will provide you with some useful links:

• BDD in Action book: https://www.amazon.com/Action-Second-John-Ferguson-Smart/dp/1617297534
• The SpecFlow Gherkin reference: https://docs.specflow.org/projects/specflow/en/latest/Gherkin/Gherkin-Reference.html
• The Cucumber Gherkin reference: https://cucumber.io/docs/gherkin/reference/

Early Days

However, the early days of behavior driven development were challenging. The only frameworks available were for Java and Ruby programming languages, which posed a problem for our Microsoft.NET shop. We could experiment with BDD, but our options were limited. The lack of comprehensive tooling and resources for BDD in different programming languages was a significant hurdle. Finally, we could fully leverage BDD in our product development when Cucumber was ported to .NET (SpecFlow), which opened up new possibilities for us.

A few years back, I was the co-founder of ArgusQ. We built a crowdsourcing platform to test ideas and specifications. While our customers enjoyed the product, we did not make any money. Eventually, we started discussing a pivot. Much discussion focused on BDD and how to build a tool to help its adoption. My co-founder, Fernando, loves BDD. I do, too. We used BDD to build ArgusQ, but the data kept pointing to a tiny market.

At some point, we decided not to work on the pivot and instead dissolved the company. Still, I kept teaching behavior driven development everywhere I went. I always looked for indications of how BDD was progressing in the marketplace. Everything I found indicated that our first observations were correct: low adoption.

BDD Adoption

For example, VersionOne, a provider of Agile software development tools, has published a 'State of Agile' annual report for some time. In their 13th edition, published in 2019, you can find this chart:

SOURCE: Dux Diligens. (2019, May 7). 13th Annual State of Agile Report (p. 10). Retrieved from http://www.duxdiligens.com/wp-content/uploads/2019/09/13th-annual-state-of-agile-report_7_May_2019.pdf

Behavior driven development is second to last, just above emergent design. I do not know if you mentioned “emergent design” before your development team. I had, and you get back blank faces with no idea what you are talking about. In other words, BDD was doing a little better than a deer in the headlights practice.

Are things any better now? Maybe. The software application tools vendor SmartBear has been publishing an annual report called 'State of Software Quality/Testing' for some time. In their 2022 edition, you can find this chart about BDD adoption:

SOURCE: SmartBear. (2022). The State of Quality Testing 2022 [Ebook] (p. 25). Retrieved from https://smartbear.com/resources/ebooks/state-of-quality-testing/

The message from SmartBear is that behavior driven development is found in larger organizations, but the percentage is 24%. Also, the report only has two years of data.

Besides, most of the expansion in the use of BDD is for GUI test automation with Selenium. This is one of the 'mutations' of BDD, where it is used in a way it was not originally intended for. That is a terrible use of behavior driven development. Harmful mutations in the world of practices and processes do not cause something to die out like in biology. Agile has several of those ugly mutations. [There is no other way to explain SAFe and its adoption. By the way, SAFe is not Agile. Never let anyone tell you otherwise.]

BDD is Hard

While I am a staunch advocate of BDD and believe in its excellence, I have witnessed teams struggle with its adoption. I have often questioned my teaching methods, thinking I was doing a poor job. However, while watching a BDD video recording of Liz Keogh explaining BDD and the Cynefin framework, I realized that BDD is indeed challenging and supposed to be. Acknowledging this difficulty is the first step towards overcoming it.

Cynefin (pronounced kuh-nev-in – do not ask me why; I do not speak Welsh) is a decision-making framework. Cynefin means habitat. Dave Snowden created Cynefin in 1999. Cynefin offers five decision-making domains. The following diagram describes the domains:

SOURCE: Agility11. (2020, January 7). Understanding Complexity to Make Better Decisions: Cynefin Demystified. Retrieved from https://www.agility11.com/blog/2020/1/7/understanding-complexity-to-make-better-decisions-cynefin-demystified

As Liz explains in this short video, BDD is for complicated stuff. To paraphrase Liz, if you try to write a BDD scenario and cannot finish it, that is complicated. The only choice is to have a conversation with your teammates and figure out how to get the answers you need so you can complete your BDD scenario. That means embracing uncertainty. Most people feel very uncomfortable with uncertainty. That is why we have these BDD mutations, like using it to drive your Selenium scripts.

I have two reasons behavior driven development has been on my mind in the last few weeks. While discussing the Testaify roadmap, we discussed implementing use case testing by capturing requirements using NLP (Natural Language Processing). One idea was to focus on behavior driven development for the first iteration. The other reason was the emergence of a new AI startup called purgo.ai. Their offering depends on bdd.

Maybe the time for behavior driven development is finally here. I hope that is the case.

Behavior Driven Development and Testaify: A Perfect Pair for Quality Product Development

If you have read my previous blog posts, you know how much I like Brian Marick’s Test Matrix. If you do not remember the matrix, here it is:

The matrix has four quadrants. Quadrant 1 is for Technology Facing and Support Programming testing. Quadrant 2 is for Business Facing and Support Programming testing. Quadrant 3 is for Business Facing and Critique Product testing. Finally, Quadrant 4 is for Technology Facing and Critique Product testing. In this blog post, we will focus on quadrants 2 and 3.

Let’s start with quadrant 2:

Business Facing & Support Programming

The second quadrant describes the tests that help you understand the capabilities you need to develop. The critical difference between Q2 and Q1 is that these tests arise from the business perspective. These were called customer tests in the early days of XP (eXtreme Programming). Then came ATDD (Acceptance Test Driven Development), usually implemented using FIT or Fitnesse. This quadrant has evolved even further to Behavior Driven Development (BDD). Essentially, BDD replaced ATDD.

The BDD tests use the Gherkin language and read something like this:

1. Given I have $100 in my checking account

2. And I have $10 in my savings account

3. When I transfer $50 from my checking account to my savings account

4. Then I have $50 in my checking account

5. And I have $60 in my savings account

Such tests are examples of business rules within a specific business domain. In essence, you are creating tests to define your business requirements. You do this before you write any line of code. These tests become the acceptance criteria for your user stories. This quadrant is essential to implement a Shift Left strategy and build quality in product development.

From a software testing methodologies perspective, this is where you implement use case testing. All the test cases that cover all your use cases are here. These tests serve as examples of what you want to build. By using BDD, it is straightforward to include these tests in your CI/CD pipeline so you can get regular feedback during the development process regarding the core functionality of your application.

Now, let’s move on to the quadrant 3:

Business Facing & Critique Product

Most of the testing in the third quadrant is about the functionality and usability of the application. This quadrant has all our regression tests. We verified that we did not break anything by adding the new functionality. This quadrant is also where we validate the usability of the product. If there’s enough time before release, we go into exploratory testing mode to find omissions and mistakes.

The third quadrant is the birthplace of GUI test automation tools and test case management solutions; you’re trying to speed up testing to cover more of your product. However, this quadrant is very expensive, as we created a new role for developing test automation scripts and added tools to support this effort.

From a software testing methodologies perspective, we use many techniques to break the product (critique). These include equivalence class partitioning, boundary value analysis, state transition coverage, etc. QA practitioners have long used these techniques to test software products. These techniques allow you to evaluate the product based on well-known failure types. In this quadrant, QA people spend a lot of time, as their imaginations go wild, trying to find new combinations to test.

Our Testaify MVP (Minimum Viable Product) will include four of these techniques:

• State Transition Coverage
• Basis Path Coverage
• Equivalence Class Partitioning
• Boundary Value Analysis

That puts Testaify's first version at the center of quadrant 3. We can discover the application as is. We can test it using these techniques and report our findings quickly. Our objective is to critique the product. The main difference is that we can do it hundreds of times faster than the QA status quo. We do not depend on fragile test automation scripts and can discover an application a hundred times faster than a human. Better than that, we can complete the whole testing cycle over four hundred times faster than a QA team. Even better than that, we can design and execute thousands of test cases in minutes or hours, depending on the size of your application. We can integrate it with your CI/CD pipeline and provide regular feedback on every build.

BDD & Testaify

While agile testing significantly improved testing by introducing the idea of using tests to drive development, Testaify is set to revolutionize testing by dramatically increasing the speed of the testing process to unprecedented levels. As we've mentioned before, it's crucial to implement all four agile testing quadrants to create a comprehensive testing strategy. BDD provides a successful approach to handling quadrant 2. Testaify is poised to forever change how teams implement Quadrant 3 and, eventually, Quadrant 4.

As Testaify evolves, we envision adding support to ingest your BDD scenarios. This potential future integration of BDD scenarios into Testaify will further enhance the testing process, making it even more efficient and effective.

The Future of Software Testing is on the Horizon! QA and Devs, get ready for the exciting changes that Testaify will bring!