Testing without Testers

Is testing without testers a damaging trend?

Early Examples

The first examples I saw involved agile development teams. These teams are assembled inside mature Agile organizations and have all the typical trappings of a modern Agile team: CI/CD pipelines, unit and integration testing, and many practices like Behavior-Driven Development (BDD). They also automate their regression suite.

Early examples were tiny teams with as little as three developers. These micro teams accompany the emergence of microservices architecture. Other organizations built teams with software engineers, UX designers, and product managers but no testers.

These early examples were in Big Tech and Silicon Valley companies with strong infrastructure support and a history of implementing agile software development practices.

Testing as Cost Center

Testing and its value have long been sources of contention. This dilemma is behind many organizations outsourcing software testing. To paraphrase an entrepreneur I met many years ago, “You do not know what QA is doing, but you always need it, and every year, you need more of it.” His tone was one of frustration.

I discovered the “testing without testers” trend was spreading through outsourcing. I was having dinner with a friend who owns a nearshore company (outsourcing to Latin America). During dinner, he shared that some recent clients were asking for teams of developers, but they did not want to pay for testers. They expected that the developers would be responsible for testing the product.

The One That Hit Home

The event that hit home and significantly impacted me happened in the summer of 2024. As many of you know, I worked at Ultimate Software for many years. My time there was the best of my career. Ultimate Software had the best culture of any organization in the United States and was one of the top ten in the world.

The changes started in earnest after Ultimate Software was acquired and eventually merged with Kronos to become UKG. New leaders came in who did not know anything about building great company cultures or much about how to develop software. Some of those leaders have drunk the Kool-aid regarding building teams without QA. In July of 2024, in one of the worst executed layoffs in history, UKG let go of thousands of employees. One of the hardest-hit teams was the QA team at Ultimate Software. People with deep expertise in the domain let go because they had no “technical skills.”

Trend Origins

In the early years of the 21st century, most software development organizations had a dedicated QA team. In most instances, this meant many people were conducting manual testing, and most new people in QA were writing GUI-based test automation scripts.

While outsourcing changes the location of many QA team members, it has little effect on the organizational structure of software development teams. The significant change came with Agile. As Agile software development became more prevalent, most organizations started implementing Scrum, one of the project management frameworks within the Agile umbrella.

Scrum asked teams to change the way they develop software significantly. It requires teams to limit their size to between 5 and 9. It requires cross-functional teams. While the original Scrum papers talked about one-month sprints, the most popular iteration size in Scrum is the two-week sprint. In that period, the team expects to plan one sprint, implement it, and release it to production.

As teams started using the Scrum framework, many realized they needed to make more changes to their operations to successfully build and ship software in a two-week sprint. 

QA & The Agile Shock

A lot of what we call today Agile testing came up as potential answers to this problem. If you read our posts, you know that ideas like Shift Left are part of Brian Marick’s Agile Testing Quadrants:

Together with the strong emphasis on automated testing brought by concepts like the testing pyramid and testing trophy:

Agile testing encourages a “build quality in” mentality. This approach means that instead of relying on post-development testing to catch bugs, the focus is on writing small, fast, automated tests that provide quick feedback during development. This approach moves teams away from their GUI-based test automation strategies and toward more developer testing, like unit testing.

This revolution created an uncomfortable environment for well-established QA teams. Before Agile, most QA teams consisted of people conducting manual testing. Manual testing was not an option if you were trying to release every two weeks. While Agile testing includes references to practices like exploratory testing, its primary focus has been on test automation.

It also puts a lot of pressure on people who build their careers around GUI-based test automation tools. In Agile, we want to do less of that kind of test automation.

Testing without Testers from Big Tech and others

In 2014, Microsoft retired its famous SDET (Software Development Engineer in Test) role. Like many of these roles in Big Tech, this one focused on automation. They are not the only ones making these organizational changes. Most of the 7 Big Tech companies have taken this path, except Apple and Amazon.

However, it is not only Big Tech that is making these changes. A survey of 47 organizations published by The Pragmatic Engineer newsletter (this article might be behind a paywall) shows the pattern spreading. Here is a quote from a 150-person marketing agency:

We frequently deliver small pieces of software behind feature flags using continuous deployment. Before merging, we run unit and integration tests – we have a lot of them! Ideally, the team should do some level of pair testing together, but I admit this is frequently skipped.

Testing without Testers: Challenges

Still, even in this survey, you can see the challenges with this approach. I found the following quotes quite revealing:

We now find bugs in prod that we have to fix immediately after a production launch.

There should be a healthy amount of upfront design/thinking/questioning at the beginning of a project. Evaluating and testing with users comes at the end. In practice, both of these tend to be forgotten, and this is where most issues creep in!

Many agile teams focus only on quadrants 1 and 2 of Marick’s Agile Testing Matrix, forgetting that more complex testing is needed to cover quadrants 3 and 4. Generally, we see an overcorrection towards the left side of the matrix and underinvestment in the right side quadrants.

Overall, we are very concerned with this trend. Before Agile, most QA teams focused only on quadrants 3 and 4, but Agile has shifted that to quadrants 1 and 2. The problem is that we ignore the fact that the matrix provides a framework for comprehensive testing. You can only achieve that by addressing all four quadrants. Neither the agile testing pyramid nor the testing trophy says you have no UI tests, but that is what many agile teams without testers now believe.

At Testaify, we believe in a comprehensive testing strategy that covers all four quadrants. That is why our product's first focus is quadrant 3. We will never have a complete functional testing strategy without UI tests. Our AI solution can help you achieve this goal.

You can have testing without testers but cannot do it well without a comprehensive testing strategy.