A Comprehensive Overview of Regression Testing

Regression Testing from Inception

While there are numerous well-known software testing books, such as “Software Testing Techniques,” “Testing Object-Oriented Systems,” or “Testing Computer Software,” One book that often goes unmentioned is Louise Tamres's “Introducing Software Testing.” This book stands out for its simplicity, directness, and practicality, particularly in its clear definition of regression testing. 

Regression Testing Definition

In her book, Louise defines regression testing as reusing a subset of previously executed tests on new versions of the application. This definition is straightforward and practical. Let’s compare with Cem Kaner’s definition in his “Testing Computer Software” book: regression testing is fundamental work done by glass box and black box testers. Talk about using buzzwords that only practitioners can understand. Kaner then goes on for four paragraphs to say the same thing Louise Tamres achieves in one sentence. I am exaggerating, but not by much.

Regression testing is the last stop before release. As such, it increases over time with new features introduced. In theory, regression testing can be one test executed to validate a specific fix to a defect. As Louise Tamres said in her book, it is usually a subset of tests or the whole suite of tests. In mature software development organizations, regression testing includes more than functional testing. It can consist of performance, security, usability, etc. You need regression testing every time the software changes. That could be a major new feature or a minor software update to address one defect.

Because regression testing is the last step in the software development process before release, it is also the most costly development step to find defects. In the waterfall model, regression testing only happens several months after development starts. At that point, any software bugs found will undergo a process to decide whether to fix them now or later. The introduction of agile and lean practices in software development has changed this approach considerably.

Regression Testing Strategy

Regression testing is not just a step in the software development process but a cornerstone of a comprehensive software development process and testing strategy. Software development would be risky without a robust regression testing strategy. It is a vital component of the testing strategy, particularly when it comes to releasing a new version of the product. If your product did not pass regression testing, it should not go to production.

The evolution of regression testing strategies has been significant over the years. Similar to what Pettichord did with the software testing schools I discussed in this blog post, we can categorize the evolution of regression testing into three distinct ages, each representing a significant shift in practices.

Regression Testing Through the Ages

The Manual Regression Testing Age

Many quality assurance professionals I worked with in the 1990s did manual regression testing. Some will be surprised, but there are still companies out there that do mostly manual regression testing. I had the opportunity to meet one of those companies recently.

In the early days, developers did manual regression testing, if they did it at all. Teams did not have project management frameworks, so everybody did what they thought was the best way to do it. It was an ad hoc approach to regression testing.

The emergence of formal approaches for software testing came from the first three testing schools: Analytical, Factory, and Quality Assurance. These schools introduce software testing techniques (Analytical school), a formal requirements phase (Factory school), and different types of testing and stages of testing, such as regression testing (Quality Assurance school).

To implement manual regression testing, you need a comprehensive test case inventory. When I joined e-Builder in 2016, the Quality Assurance team had an inventory of around 1,000 test cases. However, I found out they had not executed a full regression suite run in a long time. They release once per week, and the team will need five weeks to run the full regression testing suite. They picked a number of those test cases (less than 200) and executed them every week. They refer to this suite as their smoke test inventory.

As you can imagine, many software bugs escaped detection. That is unsurprising, as the e-Builder application, e-Builder Enterprise, has around 1,100 web pages. That means even the regression suite had fewer tests than web pages in the application. Manual regression testing is not scalable because it takes a long time. To make it faster, you need to hire more people, which is costly.

While manual regression testing is a common practice, it has drawbacks. It is not scalable and can compromise the quality of your product. Due to its time-consuming nature, many enterprise software organizations tend to release new versions with a list of known software bugs. This normalization of releasing a product with known defects is not acceptable. It's time for a change in the way we approach regression testing.

At e-Builder, we addressed the scalability and quality issues of manual regression testing by transitioning to the second age of regression testing, which involved automated testing.

The Automated Regression Testing Age

The birth of automated testing is a direct response to the challenges posed by manual regression testing. As you have seen from the experience at e-Builder, manual regression testing not only increases the time to test but also considerably limits your test coverage. Before I joined e-Builder, they tried to address this issue twice without success. In one example, they hired a consultant who wrote a test automation framework. The problem was that he wrote the framework on a programming language the team had never used and needed to learn.

First, we used a crowdsourcing service to help the QA team. The smoke test suite will be executed by many people simultaneously, allowing us to speed up regression testing in the short run. That was a transitional strategy.

At the same time, we hired a company to build a test automation suite using a test automation framework based on Selenium that we can code in C# (e-Builder is a .NET shop). It took nearly a year, but we automated the whole regression suite (1,000 tests). Once again, we ran the entire suite. We did it in a few days instead of five weeks, demonstrating the significant time and cost savings that automated regression testing can bring.

Interestingly enough, having an automated regression suite did not remove the issue of regressions. We still had escaped defects, which was not a surprise. As we mentioned before, we had fewer tests than web pages. To confirm the issue, we started to measure the code coverage of the automated regression testing suite. We found it was less than a third of the codebase.

The first generation of test automation tools and frameworks focused on automating UI-based tests (what people in the industry will call black box testing). Because it is a lot harder to automate UI-based tests, these automated tests tend to be fragile and require a lot of maintenance. In essence, the industry created a new job. The companies implementing test automation had no choice but to significantly increase their QA teams by hiring automation developers to write and maintain all their test scripts.

As discussed in a previous article, we focused heavily on increasing our test automation coverage at Ultimate Software. The manual regression testing effort for our product took months. We implemented Scrum and created cross-functional teams that each own their areas of responsibility. We asked the teams to target a test automation coverage of 80%. In other words, we want more than 80% of the tests to be automated. We met the goal in less than twelve months. It had a significant impact on our regression testing. We met daily during that time to track the progress. The pattern was clear; we would hit 80% in a matter of days, and then we would need five more weeks to complete the manual regression tests.

Eventually, we reached over 90% test automation coverage, but reaching higher became too complicated and expensive. Besides, at this point, we have increased our QA budget considerably. We had to double our test environments to run and develop test automation scripts for such an extensive portfolio of products. We had over 300,000 automated tests. Most of them were UI-based test automation scripts. Eventually, our test automation codebase was larger than our product codebase.

We did achieve a 60% reduction in regression testing time, but we ended up with an inverted pyramid or ice cream cone if you are familiar with those agile testing terms. We had more than doubled our budget. In other words, the automated regression testing age did help to reduce the time and increase the test coverage, but it came with a huge bill associated with it. We had a new role; we needed more infrastructure and were breaking the bank.

To resolve this problem, we had to adopt a different approach to test automation. Enter the age of agile regression testing.

The Agile Regression Testing Age

Three central ideas from agile have had an incredible impact on software development. The first is that tests can serve as examples and drive development (Test First).

Test First Approach

In 2001, a group of software practitioners published the Agile Manifesto. Brian Marick was one of the software practitioners present at the meeting while writing the Agile Manifesto. Brian was the only representative of the software testing community present. If you have read my blog posts, you know I am about to mention Marick’s Test Matrix. The Test Matrix is the foundation of what we call Agile Testing. Let’s take a look at the matrix one more time:

Before agile, our understanding of testing focused only on quadrants 3 and 4. The focus was on critiquing the product, looking at software only from the perspective of the final product. The first two regression testing ages (Manual and Automated) focused on quadrants 3 and 4. While quadrants 3 and 4 are essential, we needed to do more about supporting programming.

Quadrant 1: Test-Driven Development

The emergence of test-driven development (TDD) directly leads us to the first quadrant. Thanks to the XP (Extreme Programming) community, TDD became a well-known practice in software development. The concept of writing small isolated tests to verify functions or methods of the code existed already. Writing the unit tests before writing the actual code was revolutionary. In other words, write tests to drive the code you need to write. The “Red-Green-Refactor” TDD cycle changes how software engineers think about how to write code. The following diagram illustrates the process:

Quadrant 2: Behavior-Driven Development

Quadrant 2 expands on that idea and introduces the concept of using higher-level tests to drive development. It took some iterations from customer tests (XP community) to ATDD (Fit and Fitnesse) and eventually BDD (Behavior-Driven Development).

Like in TDD, in BDD, you write your tests before you write any code. The difference is that you focus on higher-level tests on the core business domain. These tests are not isolated. Another difference is that BDD is a collaborative effort between team members instead of an individual developer task.

Let’s take a look at the BDD process:

Source: https://www.slideshare.net/slideshow/behaviour-driven-development-bdd-closing-the-loop-on-a-great-fiori-ux/74927727#13 

As you can see, the development team creates the test scenarios together. The scenarios look something like this:

1. Given I have $100 in my checking account

2. And I have $10 in my savings account

3. When I transfer $50 from my checking account to my savings account

4. Then I have $50 in my checking account

5. And I have $60 in my savings account

Related scenarios formed one feature file. We have made use cases testable.

Some might be wondering what this has to do with regression testing. It revolutionizes regression testing by providing a new template for an automated regression testing strategy. Specifically, you can implement an approach that reduces the dependency on UI-based automated testing. The diagram below shows the first model:

This diagram is the famous testing pyramid. Instead of using only UI-based test automation, you focus on building unit tests for your codebase. Unit tests are fast and execute in seconds.

After unit tests, we have service tests, primarily BDD tests. These two types are below the UI tests. As such, they are faster and more reliable than UI-based tests. One of the many great things about Ultimate Software is that you get to work with some of the world's top experts. Because we worked with Mike Cohn for our Scrum implementation, I knew about the testing pyramid years before he mentioned it in his book “Succeeding with Agile: Software Development Using Scrum,” published in October 2009.

Since the emergence of the testing pyramid in the late 2000s, other models have emerged. I recommend the use of the testing trophy model these days. The trophy model looks like this:

The testing trophy model focuses on the integration tests (BDD tests) instead of the unit tests. The architecture of your product usually determines the model to use. The testing trophy model became popular with the emergence of microservices architecture. The testing pyramid emerged when the most common architecture was the n-tier monolithic architecture. Still, these models provide a faster and more cost-effective test automation strategy, essential for another of the three crucial ideas behind agile software development.

Continuous Integration: Working Software at All Times

One of the most important agile practices and revolutionary ideas is continuous integration (CI). Grady Booch coined the term in 1994, and the XP community made it part of their standard practices. The idea is to integrate all software changes in the codebase and validate that they did not break something as soon as possible. In other words, test your product every time there is any change to the software.

You need an automated build and automated tests to implement something like this. You also need tests that run very fast and environments to run those tests. Because you need fast tests, the first are unit tests. Those are the fastest automated tests you can have.

The following image from GitLab describes the CI/CD pipeline:

Source: https://about.gitlab.com/blog/2018/02/14/setting-up-gitlab-ci-for-android-projects/ 

Let’s focus on the CI section. First, you must commit the code with the software updates you want to add to the product. Second, the build system builds and deploys the product in an environment. Third, you run the unit tests. If any unit tests fail, you stop the process and send a message or signal to the team. At that point, we have a broken product. If you follow lean practices like “stop the line,” the team should stop and figure out what failed and why, and they should fix it. The fix should trigger another build.

If all unit tests pass, then it moves on to run the integration tests. As you can see in the diagram, there are three dots for integration tests. The reason is that you will have many different tests. In essence, you are using the testing pyramid as your model. The last set of tests will be your broadest and slowest tests. They might be UI-based tests, but you should limit them to keep a fast pipeline. If all those tests pass, you have a working product; theoretically, you can release it to production. In other words, you have a fully automated regression testing suite.

Customer Collaboration and Responding to Change

The third essential idea behind agile is to implement a process where you collaborate with your customers. You need to be able to work with them and react quickly to their changing needs. That requires continuous integration (CI) and more. The community handles that by introducing continuous delivery or deployment (CD). It also requires working in short iterations so we can get feedback quickly. Finally, it requires us to streamline the relationship between development and operations. That is the birth of DevOps.

Let’s start with continuous delivery or deployment (CD). In the diagram from GitLab, CD is the phase following continuous integration (CI). Scrum is the most recognized agile development methodology in the marketplace. As such, most people started their experience with agile with the assumption of two-week sprints. In other words, development teams will learn how to develop and deploy to production every two weeks. Deploying to production often aims to get customer feedback quickly and adjust accordingly.

While collaborating with customers is well-meaning, it is not practical. In many companies, the number of customers is too high to collaborate with all of them. Some companies will implement old approaches like customer advisory boards or regional groups. A better approach is to monitor the behavior and adapt using strategies like canary deployments, feature flags, etc. Today, you can directly interact with users while they use the application.

CD emerges as the need for a scalable approach increases. After completing tests, the pipeline moves towards a staging process. At Rocket Matter, we deployed the release to a small group of customers in production before we rolled it out to the rest of the customers.

DevOps And Regression Testing

These approaches reduce the pressure on the regression testing stage by providing a buffer to share the release or new feature with a select group of customers and validate before sharing it. These changes increase the working relationship between the development and operations teams. These developments brought us DevOps:

DevOps is a natural evolution for those who have implemented agile since the early 2000s and are familiar with Lean thinking. An agile regression testing suite becomes a must-have for any organization implementing DevOps.

Organizations using a DevOps model shift right the quadrant 4 work. Part of implementing continuous monitoring is tracking security and performance issues. The following diagram from Atlassian illustrates the perspective of testing within a modern CI/CD pipeline:

Source: https://community.atlassian.com/t5/App-Central/Software-testing-types-in-your-Continuous-Delivery-pipeline/ba-p/1797693 

While agile is the state of the art in software development today, it is less widespread than most people think. Even today, less than 18% of companies can meet the DORA metrics at the elite level, according to the State of DevOps Report 2023 results. Because these reports are surveys, they tend to inflate the number as those who perform well answer the survey, and those who do not perform as well tend to ignore it.

Another area for improvement with this approach is its complete dependency on automation. From a software testing perspective, agile has a mixed history of delivering high-quality software. We have seen this shift towards assembling agile teams that do not include QA team members without a comprehensive testing strategy. Many agile teams tend to shift right quadrants 3 and 4. That could be dangerous as it deemphasizes the importance of testing aspects, focusing on a product critique.

In our experience, most agile teams need help developing a comprehensive testing strategy. I worked with teams that only had unit tests in the CI/CD pipeline. Even agile teams with a reasonable suite of functional tests usually have no performance tests in their pipeline. They worked with the assumption that their APM (Application Performance Monitoring) tool would tell them if there was any issue during staging. This approach is the source of many application outages in production. The load in staging is rarely sufficient to validate the production environment.

From Inception to Innovation: Testaify and Regression Testing

The advent of AI/ML is already impacting coding and testing. If you have read our series on the AI software testing tools market, you know most of that impact is targeted at helping with test automation scripts—specifically, the UI-based automation described in the second age of regression testing.

Testaify is different as it helps any team, whether conducting manual, automated, or agile regression testing. Testaify goes through the complete testing cycle:

Testaify provides unique benefits for each one of these categories.

Testaify and Manual Regression Testing

The impact of autonomous testing is quite significant for manual regression testing. For example, a team like the one at e-Builder, which took five weeks to execute a 1,000-test case regression suite, will be able to complete regression testing in a matter of days, including time to fix the issues found. According to our internal return on investment analysis, Testaify will help the team speed up by a factor of four (400% faster). But that is not the only improvement.

Because Testaify discovers the whole application with every test session, the new suite of tests will have considerably higher coverage than the existing suite, most likely by at least a factor of three (300% more coverage). Testaify includes all validated findings (defects) in all future test sessions, improving your regression test suite.

Finally, the savings are incredible as AI/ML will allow the organization to rightsize its QA team and avoid spending money on test automation developers for UI-based test automation.

Testaify and UI-based Automated Regression Testing

The improvements are considerable for QA teams living in a world of UI-based test automation scripts. Creating and maintaining test automation scripts takes a long time. Suddenly, an AI/ML offering will take over that work. Testaify will discover, design, and execute thousands of test cases in minutes or hours.

Instead of investing in some of the old self-healing AI tools that will give you, at best, a 20% improvement in your test automation execution, you can replace it entirely with Testaify. Testaify is an AI solution that allows technology executives to rethink their testing strategy holistically. Testaify will significantly reduce the need for an extensive test automation test suite. It will save the organization time and money.

At the same time, it will help the team increase test coverage. UI-based automation rarely crosses 50% test coverage because it is expensive. Testaify achieves considerably higher test coverage as it creates tests for all the web pages discovered in your application.

As the need for UI-based test automation development diminishes, the organization can save time and money on testing.

Testaify and Agile Regression Testing

Even for the few organizations that live in the DevOps nirvana, Testaify will plug a significant gap. Instead of shifting all the work for quadrants 3 and 4 right, these teams can use Testaify’s API and add it to their CI/CD pipeline. The difficult task of achieving continuous comprehensive testing (CCT) can now shift left. Adding Testaify to their pipelines will allow these teams to increase their test coverage without adding the fragility of typical UI-based test automation scripts.

Most of those teams used the testing trophy model. In that context, Testaify can take over all the system testing in their pipelines. These teams will gain additional time to work on new features instead of developing end-to-end system tests.

As Testaify expands its capabilities to include performance, usability, and security, these teams will achieve outstanding quality and avoid the quality pitfalls associated with agile teams.

The Future of Testing is closer than you think! The Age of Autonomous Testing is coming soon!