A Comprehensive Agile Testing Strategy: Does one exist?

Do Agile teams know what testing needs to happen and when?

In a recent blog post, we talked about the trend of “Testing without Testers.” In the last decade, more organizations have assembled development teams without people in a dedicated testing role.

We shared some examples and some of the key challenges with this trend. However, we did not discuss an underlying assumption. That assumption is a comprehensive testing strategy. In other words, many teams assume they know what testing needs to happen and when. That “Agile comprehensive testing strategy” does not require people with a dedicated testing role.

Brian Marick’s Agile Testing Quadrants

At the center of Agile testing, you find Brian Marick’s Agile testing matrix:

Quadrant 1: Support programming / Technology facing

The first quadrant concerns supporting programming from a technology-facing perspective. Traditionally, this has meant unit testing. While Agilistas will push for practices like test-driven development (TDD), only a small percentage of software engineers will use it. Still, old-fashioned unit testing using white box testing techniques has existed longer than Agile.

Quadrant 2: Support programming / Business facing

The second quadrant concerns supporting programming from a business-facing perspective. This quadrant approach underwent several iterations in the early years of Agile software development. It is generally considered the quadrant for behavior-driven development (BDD). By the way, if you genuinely implement the BDD process, quadrant 2 happens before quadrant 1. In BDD, the scenarios drive the unit tests. 

From a software testing perspective, quadrant 2 defines your acceptance testing suite. Usually, that means using use case testing and ensuring you have BDD scenarios for the primary and alternative paths associated with the specific story or feature you are working on. In essence, you define your minimum regression suite in this quadrant. In the Agile worldview, that regression suite must be fully automated.

Quadrant 3: Critique Product / Business facing

The third quadrant is about critiquing the product from a business-facing perspective. Before Agile, most of the functional testing happened in this quadrant. More advanced organizations will have a lot of GUI test automation. In very few organizations, usability testing will occur in this quadrant. Most QA teams lived in this quadrant. The prevalent Agile approaches to this quadrant represent a radical departure from the traditional testing strategy. These approaches are why many modern Agile teams do not have people in a testing role. In theory, Agile embraces exploratory testing, and this quadrant is the place for it, but in practice, few teams practice exploratory testing.

Quadrant 4: Critique Product / Technology facing

The fourth quadrant concerns critiquing the product from a technology-facing perspective. It means conducting all the complex testing needed for everything inside the performance and security umbrellas. Only organizations with enough resources will build these teams. In many instances, this quadrant never gets implemented. It does not help that many performance and security testing tools are costly. I was lucky as I had the opportunity to manage the performance testing team at Ultimate Software.

Fundamentally, you must implement all four Brian Marick Agile Testing Matrix quadrants to create a comprehensive testing strategy. What kind of testing strategy patterns exist? We will discuss some we have seen during our careers.

Agile Testing Strategy Patterns

The most extreme pattern I have seen is the one pushed by ThoughtWorks many years ago at Ultimate Software. At the time, their philosophy insisted that all code must be written using pair programming and that only unit tests are needed. In other words, only the first quadrant matters, and we will deal with the rest as it emerges. With this strategy, you do not need testers. It does not take long to realize that this strategy will not work.

This Agile orthodoxy was prevalent in the first decade of the 21st century. But even then, ThoughtWorks employees sought alternatives to this TDD-only approach.

The second pattern I see widely adopted with minor differences is where you shift left and primarily focus on quadrants 1 and 2. These teams will have unit and integration tests in their CI/CD pipelines. The best teams implementing this approach will use BDD in their development process, but this testing strategy has many variants. Some will focus more on unit tests over integration tests (testing pyramid), or vice versa, with more integration than unit tests (testing trophy). Some will even use BDD for UI tests.

These teams struggle with production defects. Because no one spends time in quadrants 3 and 4, many production issues come up. Again, these teams believe they do not need someone in a testing role, as all their unit and integration tests are automated and defined before the coding starts. As long as the developers get with the product and UX teams, they have everything they need, and so it goes with this testing strategy.

Another issue with this testing strategy pattern is in quadrant 2. If no team member understands use case testing well, they create an incomplete set of BDD scenarios and miss use case paths. If left unchecked, this is one of the most challenging defects in production.

A third testing strategy widely used today is accepting that quadrants 3 and 4 are shift-right endeavors, and you need a way to control the rollout. A popular approach is using feature flags. You release the new feature to a subset of your customers and monitor issues. If those customers are experiencing problems, you do not activate the new feature for the rest of your user population.

I worked with a team that used this approach, which the development team called the warranty period. In essence, the release happens in stages. As defects arise, they get fixed and deployed as quickly as possible.

We described this testing strategy as implementing a shift-left approach (quadrants 1 and 2) and a shift-right approach for the rest (monitoring production for quadrants 3 and 4 issues). Many Agile teams assume this pattern is the Agile way when they develop their testing strategy.

Part 2 of this post will analyze the challenges with these Agile testing strategy patterns.