Test Automation Definition, Strategies and Challenges

What I’ve Learned About Test Automation Strategies Over My Career

After my recent post about why I do not discuss test automation more, I noticed I have not compiled my thoughts. While I mentioned test automation extensively in many posts, no single post is dedicated to the subject.

This post does not provide a comprehensive overview of test automation. It is an opinionated overview of test automation strategies. I will cover aspects of test automation on which I have spent most of my career.

Test Automation Definition

Wikipedia defines test automation as the use of software separate from the software being tested to control the execution of tests and the comparison of actual outcomes with predicted outcomes. First, this definition only applies to software test automation as it says software being tested. Interestingly, it mentions the control of test execution as part of the definition.

There is the software under test (SUT), which is usually the product you are selling, and the test automation software, which helps you run tests. The objective of software test automation is to accelerate testing.

Test Automation Strategies

Test Automation Before Agile

I did not spend much time on test automation until I started working at Ultimate Software. As mentioned in my previous blog post, I joined the automation department, which included the test automation team. The team focused on automating functional tests using WinRunner and QuickTest Pro (QTP) tools.

Mercury Interactive created those tools to automate tests through the graphical user interface (GUI). The team used WinRunner for the Windows desktop application and QTP for the web application. At the time, we had a GUI-only test automation strategy. These tools allowed you to generate user interface events and validate the behavior of the SUT.

Agile Early Days

Around 2004, Ultimate Software started its Agile journey by implementing Scrum. We were working with Mike Cohn at the time. Mike introduced us to his test automation pyramid.

As the diagram above shows, the testing pyramid is a test automation strategy. The pyramid states that you should automate most of your testing at the unit level and the least at the GUI level. After years of using WinRunner and QTP, we did not match this strategy. Introducing TDD and ATDD practices did allow us to start moving in this direction, but the truth is that we were still an ice cream cone (see below).

While we added below UI tests either as unit tests or integration tests, the bulk of our inventory was GUI end-to-end scripts. Plus, we still have a lot of manual testing, even if we call it “exploratory testing.” Before we can release any product, we still spend a lot of time “exploring” the system.

Agile after Ultimate Software

After I left Ultimate Software, I started avoiding some of the practices we used there. One key move was to adopt BDD instead of ATDD and TDD. Agile was evolving from continuous integration to continuous deployment and delivery. The emergence of the cloud brought new architectural patterns like microservices. In that new world, alternatives to the testing pyramid emerged, such as the testing trophy (see diagram below).

While the trophy model still requires you to focus most of your test automation efforts below the UI, it moved the bulk of that work to the services layer (integration testing) instead of the units. The API became king, making it the most crucial layer to focus testing.

Test Automation Challenges

While each of these test automation strategies improves upon the previous ones, they all share a few challenges.

• Test automation is expensive, no matter which strategy you select. Several factors contribute to that cost. 
  • The dedicated resources (test automation engineers) are the most expensive of test automation endeavors.
  • The infrastructure is costly as teams build complicated CI/CD pipelines trying to automate and execute enough tests to meet the quality criteria.
  • Finally, maintenance: You write software to test software and must eventually maintain it.
• Test automation requires significant tests to have a decent return on investment. As stated before, the objective of test automation is to accelerate testing. Running them provides little value if you only have a minimal number of automated tests. If your automated test suite does not cover all the core use cases and reaches at least 60% code coverage, it provides limited help. You will continue to require a sizeable manual testing effort before every release.

The arrival of AI tools did help in areas like script maintenance. The first generation of AI tools handled particular use cases to improve the status quo. At Testaify, we believe the new generation of autonomous testing solutions will significantly impact test automation's main challenges. But that is a subject for another post.

The Future of Testing is Coming!