Intersecting Software Testing Principles & AI/ML Innovation

There are "Seven" Software Testing Principles

For many years, people have been writing about software testing. One of the most popular blog posts is the famous seven software testing principles. These testing principles are inside the ISTQB (International Software Testing Qualifications Board) materials and in dozens of blog posts by many people and organizations within the software testing industry.

Here is the list of the seven software testing principles and their explanation (italics = text from different sources about the principles):

Testing Principle #1. Testing shows the presence of defects

Testing contributes to finding and fixing defects. However, this process doesn't mean there aren't any bugs in the product. This principle, which helps to set stakeholder expectations, means you shouldn't guarantee the software is error-free.

Testing Principle #2. Exhaustive testing is not possible

"Combinations of inputs and preconditions" are multiple. Testing is, therefore, a set of planned and scheduled activities with time and cost clearly defined. As testing them all is impossible, a test strategy is required to set out test objectives and prioritize test execution according to the risk analysis performed beforehand.

Testing Principle #3. Early testing

Early testing is the key to identifying defects in the requirements or design phase as soon as possible. It's much easier and less expensive to fix bugs in the early stages of testing than at the end of the software development lifecycle.

Testing Principle #4. Defect clustering

When a defect is found in a specific area of a software product, it becomes a potential cluster (a hotspot) with knock-on effects on related code areas. In other words, a few modules contain the most defects discovered during pre-release testing or show the most operational failures.

Testing Principle #5. Pesticide paradox

Tests must evolve and ensure they're still effective in finding undisclosed defects. The pesticide paradox principle is related to the defect clustering principle. Over time, when a hotspot has been fixed, the dynamic or static acceptance tests that are too often repeated will no longer reveal defects. Therefore, these would need continuous review so the focus can vary and be applied elsewhere. 

Testing Principle #6. Testing is context-dependent

Criticality and way of testing are both related to the software/system context. If you compare military software to an e-commerce website, it's evident that neither will be similarly tested. 

Testing Principle #7. Absence-of-errors fallacy

Testing is performed to assess whether the software/system fits its purpose and user expectations. Finding no defects does not mean that it fulfills its initial requirements. 

Software Testing Principles Questions

While the seven software testing principles have been widely accepted, it's important to question their validity. Are these principles truly fundamental truths that guide our approach to software testing, or are they simply observations and advice?

Are the seven software testing principles all actual principles? I know several executives who will read some of these software testing principles and see them as excuses instead of principles. Personally, I find some of these software testing principles interesting observations, and some are good advice. Some feel redundant or repetitive, and a couple sound very generic.

Aside from their validity as principles, we also need to consider how these principles can adapt to the changing landscape of technology. With the rapid advancements in AI/ML, it's crucial to assess how these innovations impact our understanding and application of these principles. Can they still hold in this new context?

In the second part of this series, we will examine the software testing principles and see if they are affected by AI/ML.