Using Pairwise Testing to determine how many tests you need

TABLE OF CONTENTS

• Do we need test case reduction in a Testaify AI/ML world?
  • What is Pairwise Testing?
  • Testaify and the Power of AI/ML

Do we need test case reduction in a Testaify AI/ML world?

In Part I, we discussed how software testing techniques help your testing become more efficient. They provide two benefits: one is that you can focus on the higher-risk and more valuable areas, and second, they enable you to reduce the number of test cases. Specifically, we discussed using pairwise testing to devise a target for the number of tests.

What is Pairwise Testing?

Pairwise tells you the number of test cases you need to test all the pairs. The intention is to focus on issues caused by one parameter (i.e., field) or the interaction between two parameters. It is rare for issues to happen because of the interaction between three or more parameters.

First, pairwise testing does not guarantee 100% test coverage, as some software testing websites claim. Yes, a website article states that pairwise testing gives you 100% test coverage. It does not. It is a shame that software testing sources do not adequately edit their content to meet higher standards.

While the intention of pairwise is the same as other testing techniques, some people use it to come up with a target of how many test cases to create. Pairwise is handy when dealing with complex interfaces with multiple parameters. Let’s look at an example.

Source: https://help.e-builder.net/Content/add_company.htm

The following screenshot shows nine text fields (two of them are required), three dropdown fields, nine checkboxes, and five buttons. Let’s assume you need to test three domains for the non-required text fields. Also, let’s use two domains for the required text boxes. Let's say the list of countries in the Country dropdown is 25. The number for the State/Province dropdown is 78. For some reason, there is a second Country dropdown; let’s keep it the same at 25. It does seem like those two dropdown fields are connected somehow, but it is not clear from this image. Every checkbox has two options: checked or unchecked. You can decide to press a button or not (two values). To test all these combinations, you will need:

2 x 25 x 3 x 3 x 3 x 78 x 25 x 3 x 2 x 3 x 3 x 3 x 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 =
6,987,202,560,000 test cases

That is almost 7 trillion test cases. Obviously, no one is going to do that. The typical shortcut to figure out all pairs is to multiply the two largest numbers. You multiply 78 by 25 in this case and get 1,950 test cases. That is still a large number, but a manageable one. Some organizations will use that number as a guidepost regarding the number of test cases they need to test the specific functionality properly. The problem is that most development teams need help to create and automate 1,950 test cases to test one test page in a two-week sprint. Well, they can now get that help. Enter AI/ML.

Testaify and the Power of AI/ML

As I mentioned at the beginning of this post, we recently tested a CRM application. The session included the application's discovery, designing more than 1,600 test cases, and executing those test cases. The CRM application does contain more than one page. Let's say it is 30 pages (the actual number is close to this). If we assume an even distribution of test cases per page (which is never true, but to keep things simple, we are doing that here), then we created and executed more than 53 test cases per page. Remember, all of that took 54 minutes.

Imagine we ask the system to generate more tests and let it run for 8 hours (nighttime while we sleep). Testaify will create and execute over 12,800 test cases, which is over 426 test cases per page. Suddenly, we are getting closer to one thousand test cases per page.

Also, there is a second dimension to Testaify, our AI worker bees. By our default configuration, we only use ten. If we double the number of AI worker bees, we can double the number of test cases if the application under test can handle that many concurrent users.

So, if we double the number of test cases per hour from 1,600 to 3,200 by doubling our AI worker bees and running for 24 hours, we can generate and execute 76,800 test cases, translating to an average of 2,560 test cases per page.

Every time I taught pairwise testing, the testers always reacted negatively to the number generated even with pairwise. I will hear comments like: “1,950 test cases are too many for one page.” Thanks to Testaify, that should go away. Welcome to the Testing Revolution!

Did you miss the intro to this blog? Find it here: Do we need test case reduction in a Testaify AI/ML world?