Automated testing is easy to sell on the days everything passes. The day it matters is the first red result on a system somebody is waiting to release.
What happens next decides whether your automation is a control or a formality — and most of it is determined before the failure, by what the run was built to capture.
A failure is a finding, not an error
The instinct on a red result is to ask whether the test is broken. Sometimes it is. But treating "the test is wrong" as the default explanation is how automation quietly stops being evidence of anything.
The discipline is to treat every failure as a finding about the system until you have shown otherwise, and to record the showing. Three outcomes, all legitimate, all documented:
- The system is wrong. Raise a deviation. This is the outcome the test existed for.
- The test is wrong. The expected result was specified incorrectly. That is a change to an approved protocol, which goes through change control and re-approval — not an edit.
- The environment was wrong. The test ran against the wrong version, or a dependency was down. Fix the environment, re-run, and record both runs.
Notice that all three leave a record. The failure mode to design against is the fourth, undocumented one: somebody re-runs it, it passes, and nobody writes anything down.
Why a passing re-run is not an explanation
This is the most common bad habit in automated validation, and it is worth being direct about. A test that failed and then passed has told you something important: the system is not behaving consistently. That is a finding in its own right, and often a more serious one than a clean failure.
Intermittent failures usually mean timing, concurrency, state left behind by a previous test, or an unreliable dependency. Every one of those is a real defect in a regulated system, and every one of them will eventually happen to a user rather than to a test.
The rule that holds up under review: a re-run never replaces a failed run. Both are part of the record, and the second one does not close the first. Closing it requires an explanation of why the results differed.
What the run has to have captured
An investigation is only as good as what the run kept, and you cannot go back and collect it afterwards. At minimum, every step of a failing run should have left behind:
- The expected and the actual value, verbatim, not a summary.
- The system version under test and the environment it ran in.
- The protocol version executed, and who approved it.
- A timestamp for each step, not only the run.
- Evidence at the point of failure — a screenshot, the relevant log region — captured automatically rather than on request.
There is more on this in what an automated run has to capture. The short version: if the investigation needs a detail the run did not record, you are reconstructing rather than investigating.
When it becomes a deviation, and what that costs
If the system behaved other than specified, that is a deviation regardless of how the failure was found. Automation does not lower the bar; it raises the number of findings, which is the point.
Teams new to this are often alarmed by the initial spike in deviations. It is worth saying out loud to your quality group before you start: the deviations were always there. You were finding fewer of them. A rise in the first cycle, followed by a fall, is the shape of the thing working. See CSA services for how this sits within a risk-based approach.
Where to go next
Explore GxP Copilot for AI-native validation, TraceDraft for source-traceable clinical documentation, or book a demo to see either on your own data.