Exploratory Testing: From Origins to AI-Driven Future

What is Exploratory Testing?

I first heard the term exploratory testing at the Agile conference, probably in 2005 or later. I remember sitting at a table with Brian Marick, who explained agile testing to us (Ultimate Software attendees).

I remember initially being curious about exploratory testing, but I found the term quirky. Before explaining why I reacted this way, let’s understand where the term comes from and discuss some criticisms. 

Exploratory Testing Origins

There is general agreement that the first time the term appeared in print was in the “Testing Computer Software” book by Cem Kaner, Jack Falk, and Hung Quoc Nguyen.

Finding the term doesn’t take long in the book's second edition. On page 6, you can see the following two sentences:

• Rather than gambling away the planning time, try some exploratory tests - whatever comes to mind.
• Always write down what you do and what happens when you run exploratory tests.

The reason you need to conduct exploratory testing, according to this book, is stated in this sentence: No matter how many test cases or how many types you’ve created, you will run out of formally planned tests. In other words, you will never be able to execute all the tests you want, but after you finish your planned tests, you should keep going and continue testing (exploratory testing).

In 2003, Cem Kaner and James Bach collaborated on a paper titled “Exploratory Testing Explained.” Bach's definition of exploratory testing is the most widely used: Exploratory testing is simultaneous learning, test design, and test execution.

If you know Kaner and Bach, then you know exploratory testing comes from the Context-Driven school of testing. I covered the different schools here if you want to familiarize yourself with them. 

Agile and Exploratory Testing

While the agile testing school is most famous for its embrace of test automation with practices like Continuous Integration (CI), Test-Driven Development (TDD), and Behavior-Driven Development (BDD), it also includes exploratory testing.

One of the most comprehensive exploratory testing books is Elisabeth Hendrickson’s “Explore It!” Elisabeth Hendrickson is a well-known agile thought leader who is obsessed with testing. In her book, she modifies Bach’s definition of exploratory testing to simultaneously designing and executing tests to learn about the system, using your insights from the last experiment to inform the next.

I used "practice" to describe exploratory testing, but others use other terms, like "style," to refer to it. In the agile world, practice is usually more common, as agile software development is a set of principles, practices, and rituals teams follow. Also, exploratory testing falls within a specific quadrant in Marick’s test matrix. Here is Marick’s matrix for reference:

Exploratory testing falls in quadrant 3, the business-facing critique of the product. As you saw from the introduction of the concept, exploratory testing is manual testing with no defined test plan. It requires a product you can interact with, so it falls within the third quadrant.

While theoretically, you can conduct some fourth-quadrant tests in an exploratory fashion; this practice is mainly for functional testing. Explanations of exploratory testing include references to free jazz (in case you wonder why I think it is quirky).

Generally, it requires the following:

• To report on your findings, you must capture the steps you took to find a specific defect. Many practitioners use screen and keyboard recording software during exploratory testing sessions.
• While you can repeat some scripted tests to get things going during your exploratory testing session, the objective is to go where your scripted tests do not go. As the name suggests, you want to explore new possibilities and expose unknown risks and their consequences. In other words, you want to avoid repeating what is in your automated regression testing suite during your exploratory testing session.
• Because the number of combinations you can explore is way beyond what you have time to cover, it is recommended that you set a time limit for your exploratory testing session.

Many agile teams conduct exploratory testing sessions as part of their agile testing process. Whether they do it well or not is a subject for another day. The practice has detractors. 

Exploratory Testing Criticisms

Criticism 1: Nothing New; We Already Do It

When I introduced exploratory testing to our QA team at Ultimate Software, I received the following response: “We already do that. That is our job.” In other words, many of our QA team members conducted manual testing. Sometimes scripted, they often conducted tests they designed on the fly. The criticism is that exploratory testing is nothing new; everyone in software testing has been doing it for decades.

From the perspective of our system thinkers at Ultimate Software, exploratory testing was another example of some testing personality trying to say they invented something new when, in reality, it was always there.

There is some truth to this Ultimate Software QA team complaint, but there is one crucial difference. The QA team never limited their exploratory testing sessions. They spoke about their manual testing, which took months to complete. As we moved to implementing an agile methodology like Scrum, they had to get used to shorter iterations (2 weeks). They also had to accept that it was only possible to continue working the way they were working by adding more automated testing. That also meant that exploratory testing could not be the only thing they do anymore. They had to adjust and spend more time in quadrant 2, helping the team develop the essential test cases to validate each user story.

Criticism 2: Another Meaningless Term

There is another criticism embedded in the previous one. Many people online say exploratory testing is just testing. While you are following a less formal process (formal means less documentation in this instance), you are still implementing the testing process. You are discovering the system, designing test cases, and executing them. Finally, if you find something to report, you do so. The four steps are covered.

That criticism is partially correct. Exploratory testing does not add anything new. There are no new steps or techniques. Testing is more complex than simply following a set of steps repeatedly. Elisabeth Hendrickson provides an answer in her book. She said there are two sides to testing: checking and exploring.

• Your scripted tests cover checking, which defines and executes every time to answer the question: Does the software behave as intended under the conditions it is supposed to be able to handle? She says this is weaving the net.
• Exploring involves scouting areas that the net doesn’t cover to determine whether there are any other risks. In other words, what are the areas the net does not cover? Are there any issues in these unexplored parts of the system?

She says that to discover additional surprises, repeatability will not help—variation will. You have to go beyond your scripted tests without repeating yourself.

I prefer to look at it from Brian Marick’s Agile Testing Matrix perspective. You need repeatable tests that can provide you with quick feedback to help you support the development effort. Those are quadrants 1 and 2. However, you must also increase your test coverage by adding additional tests to expose risks. That testing is in quadrant 3 and 4. Functional exploratory testing, the most common form of exploratory testing, is a product critique from a business perspective (quadrant 3).

I prefer to use something other than the term checking, which diminishes the work of building scripted tests. I agree that you need both sides to have a comprehensive test strategy, but I find Marick’s Matrix a more helpful tool for defining that strategy.

The final criticism is the reason I find the practice a little quirky. 

Criticism 3: An excuse not to change

While no thought leader has ever said that exploratory testing is the only practice, many QA teams have used it as a reason to keep their work the same. An example is a QA team member saying they had just returned from a testing conference where one of the speakers noted that exploratory testing is the most important testing style.

I was not present at the conference, so I don't know what the thought leader said. Still, I doubt they said it is the only thing you should do. Humans tend to hear what they want to hear. We listen to ideas that validate our own. Worse, we sometimes transform what we hear into what we want to hear. More importantly, we prefer to avoid changes.

However, it does not help that the people selling a specific idea or practice tend to overemphasize its importance. Exploratory testing is the main contribution of the context-driven school, so I am not surprised they will overemphasize it.

Exploratory Testing is One Practice

Exploratory and scripted testing are testing; there is no such thing as checking. We support development with automated testing (TDD, BDD) and critique the product with practices like exploratory testing. 

Exploratory testing is not superior to any other practice. We have been doing it all along because, at its core, it is just how we design tests. It is one more implementation of the scientific method.

Exploratory testing is an essential part of the testing toolbox. As stated before, we want a comprehensive testing strategy, which means implementing all four quadrants of the testing matrix. Exploratory testing is one of the most common ways, together with usability testing, of implementing quadrant 3.

Testaify and Exploratory Testing

For us at Testaify, the most exciting questions about exploratory testing revolve around how AI/ML affects it. Will AI/ML enhance it? Will AI/ML have no impact on exploratory testing? Will AI/ML tools make it obsolete? 

At the core, all testing must follow this process:

Exploratory testing is no exception to that. You must discover and understand what you are testing before designing the tests. You must design the tests before you execute them. Finally, you need to execute the tests before evaluating its findings.

The critical difference is that the time between each one of the steps in exploratory testing is minimal. While James Bach and Elisabeth Hendrickson used the word “simultaneous” to describe exploratory testing, it is closer to micro-iterative testing. You start discovering a product section, and an idea pops up. That idea becomes a couple of test cases you want to implement. As soon as you define the test case in your mind, you execute it to see what happens. The whole cycle could take a minute or less.

Testing Challenges and Exploratory Testing

At Testaify, we believe the fundamental challenge in testing is the vast number of combinations required to test a product exhaustively. Three serious issues affect the majority of software development teams, making that challenge even worse.

To improve software testing significantly, we must address three essential issues.

• Designing comprehensive test suites is difficult and time-consuming. Most people in the industry do not know all the testing methodologies needed to create them.
• Human-generated test automation wastes time and money. It is fragile and built on a foundation of incomplete or poorly designed tests.
• For too long, the information about the product’s quality has been scattered or sometimes is completely missing. Most companies make release decisions based on limited information and sometimes on one person's opinion.

Is exploratory testing a solution to these issues? Not really. One revealing quote in Elisabeth Hendrickson’s book “Explore It!” is the following: The more familiar you are with test design, the better you are able to design good experiments on the fly.

That line comes after a paragraph discussing software testing methods and mentioning books like Glenford Myers’ “The Art of Software Testing,” Boris Beizer’s “Software Testing Techniques,” and Lee Copeland’s “A Practioner’s Guide to Software Test Design.” Those books covered the software testing techniques essential to designing test cases.

In other words, you must learn the software testing techniques to become proficient at exploratory testing. If that statement is true, then only a minority of software testing practitioners should use a practice like exploratory testing. Our own experience and surveys show that most people in the software testing industry know four or fewer of the software testing techniques. Less than a handful of people at our 50+ Ultimate Sofware QA team knew all the software testing techniques. If that matches the rest of the world, less than 10% can conduct exploratory testing sessions effectively.

As mentioned, some software testing thought leaders compare exploratory testing to free jazz. The problem is that they must tell you that a few highly talented musicians can play free jazz. Only a few people have the skills of Ornette Coleman or John Coltrane.

Autonomous Testing to the Rescue!

The advent of AI/ML allows the creation of software testing tools like Testaify that can implement the whole testing process, use all the software testing techniques to design test cases, and automatically execute them without human intervention. That is autonomous testing.

All of a sudden, the capability of a few becomes available to everyone by using autonomous software testing tools like Testaify. Thanks to Testaify’s Discovery, Design, and Execution engines, we can fully implement the testing cycle. But the secret weapon is the Testaify Oracle. The Testaify Oracle analyzes all the test sessions executed by your team and every other customer to improve future test sessions.

Testaify is constantly learning and improving with every test session. It also has the advantage of knowing every testing technique we want, including complex and time-consuming techniques like state-transition coverage (also known as state transition testing, state diagrams, etc.).

If Testaify does the following tasks with every test session, does that mean Testaify is doing exploratory testing? 

• Discover the application
• Applies the software testing techniques to design test cases
• Generate appropriate test data for your specific domain
• Execute each one of these test cases
• Reports back the findings with detailed steps for easy reproduction
• And learns from each test session

Again, does that mean Testaify is doing exploratory testing? That is not precisely the case, as Testaify does not follow the approach of designing and executing the test case immediately. Also, discovery takes some time. After designing test cases, a little time passes before the engine starts executing them.

Is Testaify better at exploratory testing than most people? Testaify does several things that a person conducting an exploratory test session cannot do. It discovers and tests the whole application in every test session. It generates hundreds of test cases and executes them very quickly. It provides detailed findings a lot faster than any tester. In other words, Testaify can significantly increase the test coverage in ways no exploratory tester can.

Can Testaify help you improve your exploratory testing sessions? Yes, it can. The objective is to cover the gaps in your testing net. An experienced QA system thinker will be able to review the discovery model generated by Testaify and review the test cases. That should provide guidance regarding which gaps are still unexplored. That will massively improve the effectiveness of your exploratory testing sessions as you now have an effective testing net that includes your automated unit and integration tests plus Testaify. Finally, you will be able to spend time on those obscure areas of the products that only your power users tend to break.

The Future of Testing, including Exploratory Testing, is Here!

We see a fascinating future for all software development teams. No matter what stage your team is in, Testaify will significantly help you.

• If your team is still struggling to implement agile, has no test automation, and takes an ad hoc approach to manual testing, Testaify will allow you to jump forward. You can use Testaify to significantly increase your test coverage without a massive investment in test automation. The impact will be so significant that you will catch up with your competitors and surpass your customers' expectations.
• Suppose your team is doing agile but has no time to Shift Left and implement BDD. Your great QA system thinkers spend most of the sprint conducting exploratory testing because your net still has too many holes. It is time to free them by bringing Testaify to supercharge your exploratory and regression testing, letting your system thinkers shift left, and helping the team build quality into your coding process. Testaify will take care of quadrant 3, and your QA system thinkers can move left to quadrant 2 to help you successfully implement BDD.
• You have a modern agile micro team with a CI/CD pipeline and many unit and integration tests that allow you to deploy to production multiple times daily. Because you are a modern agile team, you have no testers, as every test is automated by the developers. You have APM and security tools in the CI/CD pipeline and production. Still, you spend most of the week fixing bugs reported by your customers. Your product manager is complaining about the lack of new features. Just because you do not have testers does not mean you do not need testing. You are covering quadrants 1 and 2, but without quadrants 3 and 4, you will never have a continuous comprehensive testing (CCT) strategy. Testaify will increase your test coverage and plug the holes in your testing net. Let’s get back to adding and enhancing features.

The next generation of AI solutions is here! Finally, autonomous testing is arriving! The Future of Testing is closer than you think!