It's Thursday afternoon and the production release is scheduled for Monday. The team runs the test suite: 40 tests fail. Nobody touched those screens; someone moved a button and renamed a field. The next two days go into fixing selectors instead of testing what actually changed in this release.
Automated software testing has been around for decades, but AI changed who writes the tests and how much it costs to keep them working. Today an assistant can write unit tests from existing code or fix a test that broke because someone moved a button. That saves real work. What the tool doesn't do is decide what's worth testing, and it won't notice when a business rule was misunderstood from day one. This article looks at which parts of automated QA you can hand to AI, where you still need human judgment, and what to ask a vendor who offers you "AI-powered testing".
What the data says about AI in testing
The World Quality Report 2025 by Capgemini, Sogeti and OpenText, published in November 2025 and based on a survey of more than 2,000 executives in 22 countries, paints a fairly sober picture:
- 89% of organizations are piloting or using generative AI in their quality processes, but only 15% have it deployed across the whole enterprise.
- The average reported productivity gain is 19%, although a third of organizations saw minimal improvement.
- The main obstacles are data privacy risks (67%), integration complexity (64%) and hallucinations or reliability concerns (60%).
- Usage is shifting from analyzing outputs (defect reports) toward designing test cases and refining requirements.
Our take: almost everyone is experimenting, few have it working properly, and what separates the two groups has more to do with process than with the tool they picked.
What you can automate with AI today
AI does best where the result can be checked quickly. The test compiles and passes, or it doesn't.
Unit tests from existing code
In the .NET world, Microsoft documents GitHub Copilot testing for .NET, which generates tests in a separate project, builds them, tries to fix errors and runs them in Test Explorer. It requires Visual Studio 2026 version 18.3 or later and a paid GitHub Copilot subscription, and it works with MSTest, NUnit and xUnit. Most coding assistants offer something similar for other languages, with less integration.
End-to-end tests for web applications
Playwright ships with three test agents: one explores the app and writes a test plan in Markdown, another turns that plan into executable tests, and a third runs the suite and repairs failing tests. Microsoft also publishes guidance on using Playwright with AI assistants for Power Platform apps, and its recommended workflow ends with a plain instruction: review the generated test before you commit it.
Maintaining regression tests
This is the opening scene. When a test fails because a selector or a load time changed, an agent can propose the fix and rerun it. That's where the most interesting savings are, because keeping the suite alive is exactly what makes many teams give up on automation after a few months.
Test data and edge cases
AI is good at suggesting combinations nobody wrote down: Chilean RUT numbers ending in K, zero amounts, end-of-month dates, text with accents and ñ.
Where you still need human judgment
An AI-generated test checks what the code does. It has no idea what the code should do. If the rule for calculating a sales commission is implemented wrong, the AI can write a perfectly tidy test that confirms the bug.
That's why some decisions shouldn't be delegated:
| Task | Can AI handle it? | Notes |
|---|---|---|
| Generate unit tests for existing functions | Yes, with review | Check that assertions test business rules, not just "it doesn't crash" |
| Propose an end-to-end test plan | Partly | Useful as a draft; someone who knows the process has to prioritize |
| Fix tests broken by UI changes | Yes, with review | Watch for an agent that "fixes" a test by hiding a real bug |
| Define acceptance criteria | No | That's a conversation with the business |
| Decide which risks to cover first | No | It depends on what each failure costs the operation |
| Security and permissions testing | Only as support | You need someone who thinks like an attacker |
| Exploratory and usability testing | No | AI doesn't get frustrated with a confusing screen |
Be careful with the agent that repairs tests. Playwright's own documentation says that if the agent believes the functionality is broken, it may mark the test as skipped. If nobody reviews those skipped tests, the suite stays green and the bug still reaches production.
The risk people tend to overlook: data
The World Quality Report lists data privacy as the top obstacle, and in Chile it has a concrete angle. To generate tests, the tool needs to see code, schemas and often data. If your test environment is a copy of production with real RUT numbers, addresses and salaries, that data can end up in the context of an external model.
Chile's Law 21.719 on personal data protection makes this more than a best practice. We cover it in detail in Real data in test environments and Law 21.719. In practice, the rule is to test with synthetic or anonymized data and to know which AI tools can access the repository and on what terms.
What to ask a development vendor
If you're hiring custom development and the vendor says they use AI for QA, these questions separate a real practice from a sales line:
- What kinds of tests do you deliver, and in what proportion? Unit, integration and end-to-end. Ask to see the suite, not just the coverage percentage.
- Who reviews the tests the AI generates? There should be a named person, and the review should be visible in the pull request.
- Do the tests run on every change? If they aren't wired into a CI/CD pipeline, they'll stop running within a few weeks.
- What data do you test with? The acceptable answer is synthetic or anonymized data, never a straight copy of production.
- Which AI tools see the code, and under what terms of use? It's worth knowing whether the vendor uses business plans or personal accounts.
- Are the tests part of the deliverable? When the project ends, the suite should live in your repository, documented and runnable by another team.
That last point ties into what to ask for at each stage of a project, which we explain in Software design and development: stages and deliverables.
Where to start
If your system has no automated tests today, don't start with agents. You'd end up with hundreds of tests nobody understands. This order has worked for us:
- Pick the three or four flows that hurt most when they fail (invoicing, payroll calculation, dispatch, whatever applies).
- Write the acceptance criteria for those flows together with the business team.
- Use AI to generate tests for those flows and review them one by one.
- Wire them into the pipeline so they run on every change.
- Only then expand coverage and add maintenance agents.
Frequently asked questions
Does AI replace the QA team?
No. It takes repetitive work off their plate, like writing basic tests or fixing selectors, but someone still has to decide what to test, review what the tool produces and do exploratory testing.
Are AI-generated tests reliable?
They are when someone reviews them. A generated test can confirm incorrect behavior if the business rule was implemented wrong. Human review in the pull request is what makes them worth having.
Can I use production data so the AI can generate tests?
It's not a good idea. Under Law 21.719, exposing real personal data in test environments or external tools is a legal risk as well as a technical one. Work with synthetic or anonymized data instead.
Quality from the first sprint
At Disrupsoft we build custom web and mobile applications, with GitHub and CI/CD as part of everyday work. If you're evaluating a project and want to plan from the start how it will be tested, let's talk about your custom development.
