How to Choose Software Testing Metrics

Part 1: Context, Guidance, and Preparation 

In a recent post, I talked about software testing metrics. I shared an example of how DeepSeek generated a list of over 40 metrics but did not include one of the most important ones. I started thinking about how you pick what to measure. This blog post is about my thoughts on this subject.

Context Matters

The first thing you need to keep in mind is the context. Not long ago, The Pragmatic Engineer newsletter included an overview of a small company that developed a popular video game. The company did minimal testing: no unit tests and no code reviews. Some software engineering absolutists will immediately criticize this development team.

It makes sense that they did not invest significantly in software testing. First, they are building a game. No one will die if the game stops working. My son might disagree, but boredom does not cause death. Second, it was a one-time release. While some video games become franchises and you must keep releasing new versions, most indie games are one-time releases.

The context of your application significantly impacts the quality standards you must meet.

Guidance in Decision-Making: The Cynefin Framework (pronounced kuh-nev-in)

SOURCE: Agility11. (2020, January 7). Understanding Complexity to Make Better Decisions: Cynefin Demystified. Retrieved from https://www.agility11.com/blog/2020/1/7/understanding-complexity-to-make-better-decisions-cynefin-demystified

While Cynefin is a decision-making framework, the first thing it does is help you identify your context. In this case, the domain you are in right now. How do you do that? Liz Keogh has a heuristic she uses to help you get to that answer. It depends on the answer to the question: “Who has done this before?” Here are the potential answers:

Complex domain

1. Nobody has ever done it before.

2. Someone outside the organization (probably a competitor) has done it before.

Complicated domain

3. Someone in the company has done it before.

4. Someone in the team has done it before.

Obvious domain

5. We all know how to do it.

Answer 1 puts you in the obvious domain, precisely the context for this indie game developer team. This small team knows the domain well and has developed many games before. They used what they learned in computer science.

Preparation is Key: Defining Your Testing Strategy

My background is working with B2B applications. In that context, we cannot use the same quality standards as the game developers. Unlike this gaming development team, most B2B development teams live in the complicated domain and might get into the complex domain sometimes. In that context, the first thing you need to do is determine your testing strategy. How broad is that software testing strategy? I like to start with Brian Marick’s Agile testing matrix.

The matrix has four quadrants: Quadrant 1 is for Technology-Facing and Support Programming testing, Quadrant 2 is for Business-Facing and Support Programming testing, Quadrant 3 is for Business-Facing and Critique Product testing, and Quadrant 4 is for Technology-Facing and Critique Product testing.

Consider embracing good practices for each of the quadrants for most B2B applications. Most B2B applications have large codebases with millions of lines of code, so having unit tests for such a large codebase is a good idea.

For quadrant 2, I usually recommend BDD (Behavior-Driven Development). It allows us to define the essential use cases for each feature in a language understood by different groups involved in the software development process. BDD also provides guidance about which unit tests we need.

For quadrant 3, you do need to answer several questions. For example, what do you plan to do about usability and accessibility testing? Are you required to cover more than just use case testing from a functional perspective? The answers to these and other related questions will determine what you need to measure.

For quadrant 4, you must determine what you must cover for security and performance. You might also have to add some compliance testing depending on your unique context.

In Part II, we will discuss an example based on my previous experience and the metrics that make the most sense for that context.