The Uncomfortable Truth About Test Automation

3. How intelligently is the regression scope selected?

4. How quickly can test data be created and refreshed?

5. How traceable are requirements to test coverage?

The post The Future of QA: How Generative and Agentic AI Are Transforming Quality Engineering appeared first on Sandhata.

You didn’t put money into test automation so that your QA engineers could spend Tuesday afternoon working out why a locator broke, and you almost certainly didn’t do it so that release day would turn into a group investigation into which failures are actually real. You did it because you wanted to move faster, catch problems earlier, cover more ground, and get to a release decision without that low hum of anxiety sitting in the background.

Somewhere along the way, though, quite a lot of teams have found that automation quietly became another thing that needs looking after. Regression suites take longer every quarter, test data expires at the worst possible moment, API contracts shift underneath you, UI changes take out a dozen locators at once, and failure reports still need a human being to sit down and interpret them before anyone can act on them.

That is not automation letting you down, and it’s worth saying that plainly before anyone starts blaming their framework or the people who built it. What has actually happened is that software delivery has moved on, and tooling that only follows instructions can no longer keep pace with the complexity of what it’s being asked to check. Quality engineering now needs something that can weigh things up, adjust when the application changes, work out what matters most on a given day, and improve over time instead of slowly decaying.

That is the gap Generative AI and Agentic AI are starting to fill, and it’s worth understanding what they genuinely do before deciding what to do about them.

A Release Day You’ve Probably Sat Through

It’s late in the cycle, the build has passed development review, the deployment window is a few hours away, and everyone is waiting on the regression report to land. When it does land, there are failures, and the room does that thing where nobody says anything for a moment because everyone already knows how the next two hours are going to go.

Some of those tests failed because a locator changed, some because the test data had gone stale, some because an API response now returns a field in a slightly different shape, and a handful might be genuine defects that nobody has spotted yet. The trouble is that at this exact moment, nobody knows which is which, so the team starts the familiar routine of rerunning the failures, digging through logs, comparing screenshots, checking whether the environment was healthy, refreshing data, and scanning recent commits for anything that looks relevant.

By the time all of that is done and the team finally understands what the failures mean, the problem was never really test execution. The problem was confidence, and confidence is the thing that traditional automation at scale struggles hardest to deliver, because it can give you a great many more tests without giving you any more clarity about what to do next.

Where Traditional Automation Runs Out of Road

Automation frameworks have earned their place, and there’s no sense pretending otherwise. Selenium, Playwright, Cypress, Appium and Rest Assured have taken enormous amounts of manual execution off people’s plates and made continuous testing a realistic proposition for organisations that previously spent whole weeks on regression cycles.

The difficulty is that these tools are brilliant at one specific thing, which is executing a script that somebody has already written, and they were never designed to do the harder work that sits either side of that. They don’t understand the context of a change, they can’t adapt when the application shifts underneath them, they have no opinion about which tests matter most for a particular release, and they certainly can’t look at a wall of red and tell you which failures deserve your attention first.

As the systems being tested get more interconnected, the cost of keeping the automation upright climbs steadily, and teams end up spending a larger and larger share of every sprint fixing scripts, managing data, chasing flaky tests and trying to trim regression suites that keep growing regardless. It rarely shows up as a crisis, which is precisely why it goes unaddressed for so long, but it drains real capacity from people you hired to think about quality rather than to babysit tooling.

The bottleneck in modern QA has quietly moved somewhere else entirely, because the constraint is no longer how much you can execute in a given window but how intelligently you can decide what’s worth executing in the first place and what the results are really telling you.

The Quiet Tax You’re Already Paying

Every broken locator, unstable test, out-of-date data set and ambiguous failure report takes a small bite out of your engineering capacity, and because each individual bite is so small, almost nobody adds them up. It builds gradually and unremarkably, in ways that look something like this:

• Half an hour gone investigating a flaky test that turns out to be fine
• Two hours updating locators after a front-end change that took ten minutes to make
• Most of a morning preparing test data for a scenario you’ll run once
• A release pushed back because nobody can say with confidence whether the failures are real
• A regression suite that keeps getting bigger without anyone feeling any better about shipping

None of these is worth escalating on its own, and that’s rather the point, because collectively they push the whole quality function into a reactive posture where the team is always responding to the last thing that broke rather than getting ahead of the next one. It’s the reason so many organisations will tell you, quite accurately, that they have automated their testing but have not automated their confidence.

From Test Automation to Autonomous Quality Engineering

The way out of this is not simply more automation, because more of the same thing produces more of the same maintenance burden, and anyone who has doubled the size of a regression suite already knows how that story ends. What’s emerging instead is something closer to autonomous quality engineering, where the system doesn’t just run the tests you gave it but helps you design scenarios, generate scripts, prepare data, run validations, make sense of failures, recommend a sensible regression scope and keep improving coverage as the product changes.

That shift is happening in three fairly distinct stages, and most organisations we speak to are somewhere between the first and the second.

Stage one: rule-based automation

This is where most teams live today, running predefined scripts that engineers wrote and engineers maintain, which works perfectly well for stable and repeatable scenarios but demands human intervention every single time the application changes in a way the script didn’t anticipate.

Stage two: generative AI

Here the speed of creation changes dramatically, because the system can produce test cases, automation scripts, SQL queries, API validations, synthetic data, assertions and documentation from natural language descriptions and technical specifications, which removes a great deal of the typing and boilerplate that used to fill an engineer’s week.

Stage three: agentic AI

This is where reasoning and orchestration enter the picture, with agents that can plan a testing workflow, carry out the tasks in it, look at what came back, work out why something failed, adapt when the application has moved and recommend what should happen next. It’s the difference between having an assistant who does what you ask and having a colleague who tells you what’s worth doing.

Generative AI: Taking the Grunt Work Off Your Plate

Generative AI is already doing useful work in QA teams today, and the applications are refreshingly unglamorous, which is usually a good sign that something is real rather than a demo. Teams are using it to turn requirements into test cases, build API automation directly from OpenAPI or Swagger specifications, write SQL and generate test data, produce assertion and validation logic, explain failures in plain language, generate automation code for Selenium, Playwright, Cypress and Rest Assured, tidy up test documentation and traceability, and find the coverage gaps that nobody had time to go looking for.

Scripts certainly appear faster, but the part that tends to surprise people is what their QA engineers start doing with the hours they get back, because risk analysis, exploratory testing, business validation and quality strategy are all things that need experienced people thinking hard, and those are exactly the things that get squeezed out when the week fills up with maintenance.

Generative AI doesn’t reduce the need for QA expertise in the slightest. It just means that expertise gets applied to the problems where it makes a difference.

Agentic AI: Deciding What Deserves Testing at All

Agentic AI takes this further by adding judgement to the mix, and it’s worth being concrete about what that looks like in practice rather than talking about it in the abstract. A testing agent can read a set of business requirements, work out which services are affected, find the related APIs, build end-to-end scenarios, run them, read the logs, form a view on the likely root cause of a failure, repair locators that have drifted, write up a defect summary that a developer can actually use, and tell you which slice of the regression suite is worth running for this particular change.

That changes the shape of the conversation your team has before a release. The question stops being how many tests you managed to automate this quarter and becomes something more useful, which is how intelligently you can decide what to test, when to test it, and how much confidence is enough to press the button. Getting a defensible answer to that last question is the real prize here, and it’s the thing that no amount of additional test execution has ever quite managed to deliver.

So Why Is Everyone Moving So Slowly?

Most QA leaders we talk to don’t need convincing that the problem exists, because they’re living inside it. They know maintenance is eating their capacity, they know the regression suite has become unwieldy, and they know that release confidence still comes down to somebody’s judgement at eleven o’clock at night. Change still happens slowly, and there are usually three reasons for it.

The investment you’ve already made

Years of work have gone into building the current framework, and rethinking it feels like putting a functioning thing at risk for an uncertain gain, which is a perfectly reasonable instinct. The good news is that AI-driven QA doesn’t ask you to throw any of it away, because it sits alongside what you already have and improves the design, maintenance, analysis and optimisation of the tests you’re already running.

The governance question

AI-generated tests and recommendations need reviewing, and leaders are right to ask about accuracy, data privacy and whether they’ll be able to explain a decision to an auditor six months later. These are legitimate concerns rather than obstacles, and the answer isn’t to wait until they resolve themselves but to build in the controls, the traceability and the human oversight from the beginning rather than bolting them on afterwards.

The trouble with proving return

Teams often struggle to build a business case, and the usual reason is that they’ve tried to price up a transformation of everything at once, which is both hard to justify and hard to deliver. It works far better to pick one area where the friction is genuinely painful, whether that’s regression optimisation, test data generation, failure analysis or self-healing automation, and get a measurable result there. One improvement you can point to will do more for the next conversation than any amount of strategy documentation.

Five Questions Worth Asking Before You Buy Anything

If you want a quick read on whether your QA function is in a position to get value from AI, these five questions will tell you most of what you need to know, and you can answer them from data you almost certainly already have.

How much of each sprint goes on maintaining automation?

If a meaningful chunk of every sprint disappears into fixing locators, updating scripts and stabilising flaky tests, then AI-assisted maintenance is likely to pay for itself quickly, because that work is repetitive, pattern-driven and well suited to being handled by something other than a person.

How often do regression failures need a human investigation?

If your team routinely spends hours working out whether a failure is a genuine defect, an environment problem, a data problem or just noise, then automated root cause analysis will take a large and very visible cost out of your release cycle.

How thoughtfully is the regression scope chosen?

If every release triggers the same enormous suite regardless of what changed, then you are almost certainly over-testing the parts of the system that haven’t moved and under-testing the parts that have, which is an expensive way to end up with less assurance than you think you have.

How quickly can you get fresh test data?

If preparing data is what holds up execution, or if it quietly limits which scenarios anyone bothers to write, then generated data will improve both your speed and the range of situations you’re able to cover.

How well can you trace requirements to coverage?

If answering that question means somebody sitting down with a spreadsheet, then you’re finding your coverage gaps far later than you should be, and this is an area where AI can surface missing scenarios and weak spots early enough for it to matter.

The aim of this exercise isn’t to find as many places as possible to apply AI. It’s to find the one process where the friction is worst and fix that properly, because a real improvement in a painful area will teach you more about what works in your organisation than a broad rollout ever will.

Why It’s Worth Starting Now

The commercial argument for AI in quality engineering has very little to do with tests running faster, which is a nice side effect rather than the main event. The argument that matters is that your release decisions get better, because the people making them have clearer information, fewer false alarms to wade through and a much better sense of where the actual risk sits.

Along the way you get the practical benefits you’d expect, which include less manual effort, more efficient regression cycles, defects surfacing earlier, faster triage, wider coverage and a release process that people trust. The more durable gain is that your QA team stops spending its time on repetitive maintenance and starts spending it on quality strategy, and that’s a change in what the function is for rather than just a change in how quickly it works.

Organisations that move on this early won’t simply have automated more than everyone else. They’ll have built a quality capability that adapts as the product does, and in a market where speed, reliability and customer trust are so closely tied together, that turns into a genuine advantage rather than an operational nicety.

What This Isn’t

It’s worth being direct about the things AI-powered QA is not, because there’s plenty of noise in this space and some of it sets expectations that nobody can meet. This isn’t about replacing testers, it isn’t about accepting machine-generated scripts without looking at them, it isn’t about taking engineering judgement out of release decisions, and it certainly isn’t about automating things simply because the technology now makes it possible to do so.

The sensible division of labour is that AI handles the high-volume, repetitive and pattern-driven work where it’s genuinely good, while your engineers keep hold of risk assessment, business validation, customer experience, domain interpretation, ethical judgement and the final call on whether something ships. That’s not a compromise position, it’s just what each side is actually good at, and pretending otherwise tends to end badly for everyone involved.

Where This Goes Next

Generative AI will keep getting better at helping teams create test assets, and agentic AI will increasingly shape how testing workflows are executed, analysed, repaired and optimised, which means the day-to-day experience of working in QA is going to look quite different in a few years’ time. The direction of travel is away from a function whose main job is finding defects before production, and towards one that provides continuous intelligence about quality to everybody who needs it.

In practice that means fewer repetitive scripts to write and maintain, less noise to sift through, faster analysis when something does fail, more sensible decisions about what to regression test, automation that survives contact with a changing application, and a much closer relationship between what your technical quality metrics say and what your business actually cares about.

Quality engineering has always been about more than catching bugs before customers do, but the tooling hasn’t previously made that easy to demonstrate. What’s changing now is that it’s becoming possible to build a quality function that lets an organisation keep innovating quickly without quietly trading away the trust it has spent years earning.

The Short Version

Traditional automation helped teams execute faster, generative AI is helping them create faster, and agentic AI is starting to help them decide, adapt and orchestrate their testing far more intelligently than a script ever could. Each stage builds on the one before it rather than replacing it, which is why none of this requires you to tear down what you’ve already built.

The next generation of software excellence isn’t going to come from automation on its own, because we’ve collectively spent a decade proving that more scripts do not automatically produce more confidence. It’s going to come from QA engineers and AI-powered systems working on the parts of the problem that each of them handles best.

None of this ends with AI replacing QA professionals, whatever the more excitable corners of the internet would have you believe. The people who work out how to use it well are the ones who will define what quality engineering looks like over the next decade, and that is a considerably more interesting place to be than spending your Tuesday afternoons fixing locators.

Book a free discovery call. 

The post The Future of QA: How Generative and Agentic AI Are Changing Quality Engineering appeared first on Sandhata.