Test Automation: Focus on Strategy, Not Scripts

"Why don’t you talk more about Test Automation?"

I have been getting this question more often in the last five years: “Why don’t you talk about test automation?”. The funny thing is that I talked about test automation quite frequently. One day, the real question hit me: My conversation about test automation was not what my team wanted to discuss. The disconnect struck me when I reflected on a conversation with a test architect that occurred two years ago. He did not want to talk about test automation strategies; he wanted to talk about test automation frameworks and tools. We were talking about two different subjects.

So, the real question is, why don’t I discuss test automation frameworks and tools more? I think the answer is twofold. But before I explain the reasons, I must provide some context. First, I will discuss my experience with test automation.

UI Test Automation at Ultimate Software: 2001-2004 before Agile

I got a job at Ultimate Software in 2001 as a team lead. My tiny team (just me and one other person) was responsible for supporting the infrastructure needs of the Automation department. The department included a functional test automation team and a performance testing team.

The performance team infrastructure required more time and resources than the functional testing team, so I spent time with them for the first few months. I let my one team member, who had been at the company longer than me, take care of the functional testing team's needs. A few months later, the manager fired him.

While I was rebuilding a 200-database environment for the performance team, he spent three days trying to install a new build on a small test environment for the functional team. At the end of the third day, my boss, the department manager, let him go.

The following day, I felt terrible that I had not done my job as a leader and coached my one team member. In a way, I left him alone, and he failed. I assumed that he knew what he was doing. 

At the time, I was not interested in functional test automation and dreaded spending time on it. Instead, I worked with the performance testing team. However, my team was gone, and the test environment was not ready yet. It was the first time I had to work on one of the functional test automation team environments, so I reviewed the documentation to ensure I was not missing any steps. I completed the task in 30 minutes. At that moment, I realized letting him go was the right call.

The functional test automation team used Mercury Interactive’s WinRunner to create GUI-based test automation scripts for the desktop application and QuickTest Pro for the web application. I started to spend more time with them, learning about the tools. In those days, vendors sold functional test automation tools as record and playback tools. Nothing was further from the truth.

From the beginning, it was clear that this was a programming job. You had to write code to achieve what you wanted adequately. It was a lot harder than most people realize. The whole approach bothered me. We had a team of four people writing and running the test automation scripts and, yes, running the scripts. They were so unreliable that you had to babysit them and make adjustments on the fly.

After becoming manager, I started working with members of the performance testing team to simplify the creation of functional test automation scripts in WinRunner. It was not easy, and the results were still disappointing. We made some progress in making the scripts more stable, but not enough to stop babysitting them. These were the days of UI keyword-driven frameworks.

UI Test Automation at Ultimate Software after Agile (2005 and beyond)

As the company started implementing Scrum across the product development organization, we started re-organizing into cross-functional teams. I became the Director of Quality Assurance. We split the functional test automation team as part of the transition to Agile. Also, the application was becoming more web-based with limited changes to the desktop application. As such, the effort moved to QuickTest Pro (QTP). While QTP used Visual Basic instead of a proprietary language like WinRunner, the people working on test automation remained the same.

At the same time, we started looking at simplifying our tests. Eventually, we built a web test automation framework before Selenium became popular. We were not the only ones doing this, as a few open-source projects, including Selenium, were competing for attention.

My focus also moved to implementing the agile test automation pyramid. We spent time and resources getting software engineers to write unit tests. We introduced Fitnesse as a solution to below the UI tests. We started putting in place continuous integration pipelines for the development teams.

Once again, my interest in UI test automation moved to the back. I started talking more about test automation strategies (testing pyramid) than the specific challenges with UI test automation.

UI Test Automation with Selenium

As the move to the cloud accelerated, the focus was on web-based applications. Selenium won the race to become the default UI test automation framework as time passed. Its large ecosystem and flexibility made it the default choice for most organizations.

At the same time, my perspective regarding test automation strategies changed. I moved away from the testing pyramid and its overreliance on unit tests towards the testing trophy and its emphasis on integration testing.

I guided my teams in using BDD to communicate requirements for the different user stories they were working on. I advised my teams that only the main success scenarios (primary use case paths) should be automated through the UI with Selenium.

Even with Selenium, UI test scripts tend to be fragile. Instead of running them manually, as in the past, test automation engineers wait for the CI/CD pipeline results and review each failed Selenium script one at a time to determine whether the failure is valid.

Why don’t I discuss UI test automation frameworks and tools more often?

As I mentioned, there are two main reasons. After reading the long history, you can guess them. One is that we do not have a good solution to the problem of UI test automation. Hiring staff to write and run test automation scripts is expensive. We have made progress in easing this problem, but I ask myself: Are we resolving the most critical issue in software testing? The answer is no. It is the quality of tests that worries us the most.

The second reason is simply a personal bias. I get bored quickly with small programming tasks, and most discussions about test automation frameworks involve minutiae. I do not enjoy that conversation.

Besides, at Testaify, we believe there is a better answer to handcrafting UI test automation scripts. AI/ML provides a new paradigm for software testing that makes UI test automation a non-issue.

Stay Tuned! The Future of Testing is Coming!