Do we need test case reduction in a Testaify AI/ML world?

TABLE OF CONTENTS

• Part 1: How Testing Techniques Help Testing Efficiency
  
  • Boundary Value Analysis Example
  • Software Testing Techniques & Test Case Reduction

Part 1: How Testing Techniques Help Testing Efficiency

Recently, we delved into the practicality of Testaify's AI/ML capabilities by examining the ROI (Return On Investment) data. We focused on a real-world scenario, testing a CRM web application. In just 54 minutes, Testaify efficiently handled over 1,600 test cases for the application. Despite the potential for generating and executing more tests, we limited the number of AI worker bees to ten, showcasing the practical application of Testaify's AI/ML in a controlled environment.

The impressive results of Testaify's AI/ML capabilities sparked a discussion on the importance of test reduction techniques. Testaify leverages well-established software testing methodologies to generate test cases, highlighting their crucial role in software testing. For instance, boundary value analysis is one such technique that optimizes tests by focusing on high-risk areas that are prone to bugs or high-value sections that provide essential services. In this context, the coding implementation of a boundary is a prime example of a high-risk area.

Boundary Value Analysis Example

Let’s go over an example of boundary value analysis. Let's say you have a user story for the POS (Point of Sale) system your team supports that states you want to implement the new City of Chicago Expensive Items Tax. By the way, this is a made-up tax. I hope it is not accurate, but you never know about governments.

The City of Chicago Expensive Items Tax requires retailers to collect a 0.03% tax on items worth $900.00 or more. The boundary is $900.00. If you identify the taxed and non-taxed domains for this boundary, you need at least two test cases. One test case will have an item with a price tag of $899.99 (non-taxed), and the other will have a price tag of $900.00 (taxed). The first should result in a $0.00 tax for the City of Chicago Expensive Items Tax. The second item should calculate a tax of $0.27 to comply with the City of Chicago Expensive Items Tax.

I am sure some testing people wonder if I selected the correct precision for the decimal places in my tests. It is a made-up example, and humans think of currency in terms of two decimal places. Most computer systems use more decimal places, at least four or five, to store currency data.

Because it is a Black-box test, you cannot see the code; some software testing sources suggest adding a third test case with a value of $900.01 to confirm the code implements the boundary correctly. In other words, we want to check the tax is calculated for higher values besides $900.00.

For my example, it does not matter if we have two or three test cases. What matters is that software testing techniques like boundary value analysis exist to help you target your testing towards the highest value areas. In theory, I could write hundreds of tests using multiple values from the taxed and non-taxed domains, but many of those tests will provide no additional value.

Software Testing Techniques & Test Case Reduction

Because software testing methodologies are designed to enhance the efficiency of your testing, they offer a second advantage. They enable you to reduce the number of test cases. If you recall my previous blog post about the biggest challenge in software testing, you'll remember that the number of possible combinations can lead to an astronomical number of test cases. In other words, exhaustive testing is simply not feasible. You cannot test all the combinations. This is where software testing techniques step in, providing a structured approach to reduce the number of test cases without compromising on the quality of your testing.

Boundary value analysis tells you the minimum number of test cases to test boundaries. Use case testing also tells you the minimum number of test cases you need to test a specific use case (requirement). In that case, you must create at least one test case for the main path (main success scenario) and one for each alternative path (extensions) in the use case.

The same is true for other software testing techniques like State Transition Coverage. Most of them tell you the minimum number of test cases you need to achieve a specific goal, like covering all the state transition paths or all the use case paths. One technique is used to identify the maximum number of test cases to target. That is pairwise or all-pairs testing. 

Does AI/ML impact the use of pairwise as a test case reduction technique? Find out here.