Continuous Comprehensive Testing: Essential Perspectives

Always Be Testing

Continuous testing, a concept well understood at this point, is a powerful tool that plays a pivotal role in the software development lifecycle. With automated pipelines, you can always be testing. This approach is not just about making decisions and taking different paths based on the feedback you receive but also about risk reduction. It's a powerful tool for expanding your scope to enhance product quality. While continuous testing enables development teams to deliver faster, it's important to note that it doesn’t eliminate the need for a comprehensive testing strategy. 

Comprehensive Testing Perspectives

To achieve a comprehensive testing strategy and a systematic approach to testing that covers all aspects of the software, you need to deeply understand the different testing types. These differences in the type of testing you are executing are a question of perspective. What kind of questions do I need to ask the system now?

To achieve comprehensive testing, you must ask enough questions from different perspectives. While I discussed how to use Brian Marick’s Test Matrix to guide you in the past, you can also consider the four essential perspectives that form the foundation of comprehensive testing.

Functional Perspective

The testing community uses functional and non-functional testing to cover all these perspectives. I find that short-sighted and dangerous. Even a term like functional testing has its problems. Let’s take a look at a couple of definitions. In the famous Boris Beizer book Software Testing Techniques, he defines functional tests as tests based on a comparison of a component’s actual behavior with its specified behavior without regard to the component’s implementation. He adds that this is also called black-box testing (a method where the internal structure or design of the system is not known to the tester.)

Beizer is not the only one. In his book “Testing Object-Oriented Systems,” Robert Binder defines it as the development and execution of test cases developed by analysis of requirements or specifications. No knowledge about the internal structure of the program is used to identify test cases (syn. black box testing).

If that definition is correct, this type of testing only happens when you have a specification. I can tell you that is not correct.

Let’s look at a real-life example. Using boundary value analysis, we test a field that captures the date for a car model. We used a date before the building of the first car. We get an ugly error as the system does not know what to do with this input. Is that functional testing? Is that reliability testing because I received an unexpected error? Because no one wrote the requirement about this scenario, no specification exists. I used a well-known “black box” testing technique. The answer you get is that your test explicitly discovered a missing requirement. That’s a convenient answer. 

I prefer a broader perspective instead of only tying functional testing to requirements or specifications. I want to know if it matches the business domains and associated expectations. It does not matter if this information is captured in the requirements documentation. A broader perspective in functional testing is essential to ensure the system meets the business domain expectations.

If you start with the functional perspective, we have testing methodologies that are very well-defined today and allow you to ask precise questions. You can ask field-level questions or go broader, like a business process. From a functional perspective, your core question is, “Is the system doing what it is supposed to be doing from a business domain perspective?” If what it’s doing diverges from what it’s supposed to do, you’ve got a valid finding. 

When you consider other types of testing, you’re talking about many different perspectives, each with additional questions and a different set of inquiries you need to pursue. Take load testing, for example, which falls within the performance testing umbrella. You can’t ask the same questions you would ask about a specific data input field or business process because you wouldn’t get a helpful answer. So, let’s analyze other perspectives. 

Performance Perspective

For performance testing, your question is, “Is the system behaving or interacting responsively for a user?” Assuming it’s a human on the other side. The system could be interacting with another system via an API. But what we’re talking about here is response time. While the functional perspective asks, “Is this process behaving correctly?” the performance perspective asks, “Is this process fast and stable?” The process duration must be short enough to feel like you’re having a seamless conversation with the system vs. waiting so long that you move on or feel frustrated. The future of AI assistance will make most interactions feel like you’re having a seamless conversation. Does this matter? Yes. If I ask the system a question and don’t get an answer for one full minute, that’s very problematic. Humans have clearly defined limits on how long they’re willing to wait. 

Security Perspective

When we say we want to do comprehensive testing, we mean we need to cover the fundamental, core perspectives to the point that we have a precise, quantifiable measurement of where we are in the context of each perspective. The Security perspective asks questions from another unique perspective: “Is this system going to protect my information? Will this system preserve my privacy? Can the system defend itself against attempts to steal the information within it? Is this system vulnerable to common security threats such as SQL injection or cross-site scripting?” That means a lot of different tests at multiple levels in your system.

Usability Perspective

Usability is another essential perspective. Now you’re asking, “How usable is this system? Is the user interface intuitive and easy to navigate?” These qualifying questions are very different from functional testing: "Does the system do what it’s supposed to do?” or load testing, which is “Does the system perform within a set time parameter?” Just because the process is functioning correctly, I can execute it quickly, and my data is secure doesn’t mean the system is viable. As a user, if it is challenging for me to figure out the steps I need to take, I won’t use the product. If the product is confusing, then the product is dead to me. And now, what if I’m color-blind? Or visually impaired? Accessibility testing is a subset of usability testing as you ensure your users can use your product.

CCT is No Longer a Pipedream

The testing approaches you take depend on the perspectives you need to address, but you’re still testing. You’re just asking and answering very different questions. To get to a place that is at least acceptable from a comprehensive testing perspective, you have to have enough data from each one of these perspectives. Comprehensive testing means that you have to cover all four essential perspectives. Only then can you deliver a viable solution, instilling confidence in your testing strategies.

Continuous testing is already a well-known practice, but comprehensive testing has been a pipedream for years. You need to combine them to achieve continuous comprehensive testing, the CCT star we use as our guide at Testaify.