AI GxP validation for regulated life sciences
Test Automation

When an Automated Test Fails: From Red Result to Deviation

A failing automated test is the moment automation either proves itself or embarrasses you. What the run has to capture, when a failure becomes a deviation, and why a re-run that passes is not an explanation.

Cybroscape Technologies4 min read

Key takeaway

A failing automated test is the moment automation either proves itself or embarrasses you. What the run has to capture, when a failure becomes a deviation, and why a re-run that passes is not an explanation.

Automated testing is easy to sell on the days everything passes. The day it matters is the first red result on a system somebody is waiting to release.

What happens next decides whether your automation is a control or a formality — and most of it is determined before the failure, by what the run was built to capture.

A failure is a finding, not an error

The instinct on a red result is to ask whether the test is broken. Sometimes it is. But treating "the test is wrong" as the default explanation is how automation quietly stops being evidence of anything.

The discipline is to treat every failure as a finding about the system until you have shown otherwise, and to record the showing. Three outcomes, all legitimate, all documented:

  • The system is wrong. Raise a deviation. This is the outcome the test existed for.
  • The test is wrong. The expected result was specified incorrectly. That is a change to an approved protocol, which goes through change control and re-approval — not an edit.
  • The environment was wrong. The test ran against the wrong version, or a dependency was down. Fix the environment, re-run, and record both runs.

Notice that all three leave a record. The failure mode to design against is the fourth, undocumented one: somebody re-runs it, it passes, and nobody writes anything down.

Why a passing re-run is not an explanation

This is the most common bad habit in automated validation, and it is worth being direct about. A test that failed and then passed has told you something important: the system is not behaving consistently. That is a finding in its own right, and often a more serious one than a clean failure.

Intermittent failures usually mean timing, concurrency, state left behind by a previous test, or an unreliable dependency. Every one of those is a real defect in a regulated system, and every one of them will eventually happen to a user rather than to a test.

The rule that holds up under review: a re-run never replaces a failed run. Both are part of the record, and the second one does not close the first. Closing it requires an explanation of why the results differed.

What the run has to have captured

An investigation is only as good as what the run kept, and you cannot go back and collect it afterwards. At minimum, every step of a failing run should have left behind:

  • The expected and the actual value, verbatim, not a summary.
  • The system version under test and the environment it ran in.
  • The protocol version executed, and who approved it.
  • A timestamp for each step, not only the run.
  • Evidence at the point of failure — a screenshot, the relevant log region — captured automatically rather than on request.

There is more on this in what an automated run has to capture. The short version: if the investigation needs a detail the run did not record, you are reconstructing rather than investigating.

When it becomes a deviation, and what that costs

If the system behaved other than specified, that is a deviation regardless of how the failure was found. Automation does not lower the bar; it raises the number of findings, which is the point.

Teams new to this are often alarmed by the initial spike in deviations. It is worth saying out loud to your quality group before you start: the deviations were always there. You were finding fewer of them. A rise in the first cycle, followed by a fall, is the shape of the thing working. See CSA services for how this sits within a risk-based approach.

Where to go next

Explore Cybroscape GxP Copilot for AI-native validation, Cybroscape TraceDraft for source-traceable clinical documentation, or book a demo to see either on your own data.

High riskFailure could affect patient safety or product quality
Scripted testingWritten steps, expected results, an independent reviewer
Step-by-step results, screenshots and signatures
Medium riskFailure is noticed and contained
Unscripted testingExploratory or ad-hoc testing against the intended use
A record of what was tested, by whom, and the result
Low riskStandard function, supplier-tested
Supplier evidenceThe vendor’s own testing, assessed and accepted
The assessment and the reason it is enough
Computer Software Assurance in one picture: test effort follows the risk of each requirement. Based on FDA’s Computer Software Assurance guidance.
See it on your own systemBring a URS or a system. We run GxP Copilot against it live on the call.
automated test failure gxptest failure deviationvalidation deviation handlingoq failure investigationcapa from test failure

Keep reading

All articles
Test AutomationAI Drafts, a Human Approves, a Machine Runs: Three Roles in Automated ValidationMost debate about AI in validation collapses because nobody separates who writes the test from who approves it from what executes it. Keep those three apart and the regulatory question gets much simpler. · 4 min readTest AutomationAutomated Execution in an Inspection: The Questions You Will Be AskedInspectors are not opposed to automation. They are opposed to evidence they cannot follow. The questions that come up, and what a good answer looks like with the record open in front of you. · 4 min readTest AutomationAutomating IQ: Verifying an Installation When There Is No User InterfaceOQ and PQ automation gets all the attention because there is a screen to click. IQ is mostly files, versions, services and permissions, which is why it is still done by hand. It does not have to be. · 4 min readTest AutomationOff by Default: Controlling the Blast Radius of Validation AutomationAutomation that can reach anything is automation you have to validate against everything. Shipping it switched off, and behind an explicit flag, is a scope decision more than a security one. · 4 min readTest AutomationWhy a GxP Test Engine Should Not Be Able to Run Arbitrary CommandsThe fastest way to automate a protocol is to let the engine run a shell command. It is also the fastest way to lose the argument with an inspector. Why a closed vocabulary of test steps is the safer design, and what you give up for it. · 5 min readTest AutomationWhat an Automated Execution Has to Capture to Count as EvidenceA green tick is not evidence. The specific record an automated run needs to leave behind so that someone reading it in three years can tell what was tested, against what, and by whose authority. · 4 min readTest AutomationGxP Test Automation: The Complete Technical Guide Under CSA in 2026How to build a GxP test automation strategy under Computer Software Assurance: what to automate, what not to, how to document it, and how to keep it defensible under inspection. · 6 min readTest AutomationGxP Test Automation and 21 CFR Part 11: The Audit Trail Requirements That Most Teams MissThe Part 11 requirements that apply specifically to automated test execution (audit trail, e-signature, result attribution, and data integrity) with a defensible record structure. · 5 min read

Bring a system. We'll show you the package.

Book a demo, or ask a validation lead a quick question on WhatsApp.

  • We reply within one business day
  • We run your URS live on the call
  • You get a clear plan for your system
Chat on WhatsApp

Book a demo

Four fields. The right specialist replies within one business day.

Your details stay private. We never share them.