When a test step fails: handling incidents during validation execution

Not every failure is a defect

When a test step does not produce the expected result, there are four broad explanations, and they need different handling:

• A system defect. The system does not do what it should. This needs a fix, a retest and an assessment of whether it affects anything else.
• A script error. The expected result was wrong, or the step was written badly. The system is fine; the script needs correcting.
• A tester error. The step was performed incorrectly. Repeat it properly and record what happened.
• An environment problem. Test data missing, an interface down, a configuration not yet applied. Fix the environment and rerun.

Recording everything as a defect inflates the numbers and hides the real ones. Recording a real defect as a script error is worse, because it changes the test to fit the system.

Record it where it happened

The incident record should be created at the step where the problem occurred, with what the tester saw, captured evidence and the time. Reconstructing incidents afterwards from memory produces records that read well and help nobody.

Then classify it, ideally by someone other than the tester, and decide what happens next. That decision, and the reasoning, is the part an inspector will read.

Correcting a script mid-execution

This is the area most likely to raise questions. Changing an expected result after seeing the actual result looks, from the outside, like making the test pass. Sometimes it is entirely legitimate, because the script was wrong. The way to make it defensible is to record the original text, the change, who approved it, and why the new expected result is correct, with reference to the requirement or specification rather than to what the system happened to do.

Assess the impact

For real defects, the fix is only half the work. Ask what else the defect, or the fix, could have affected. A calculation error found in one report may exist in others that use the same logic. A fix to one workflow may change behaviour elsewhere. Decide which tests need repeating and write down why the rest are safe.

What the numbers tell you

Incident data is useful beyond the project. A high share of script errors means test scripts are being written too far from the system, or too quickly. A high share of environment problems means test environments are not under control. A cluster of defects in one area suggests a design problem rather than a run of bad luck.

Look at these patterns at the end of each project and feed them into how the next one is planned. Over time, a falling rate of script and environment incidents is one of the clearest signs a validation practice is improving.

Closing out

Before the summary report is signed, every incident should be closed or explicitly carried forward with a justification. Open incidents accepted for go-live need a documented risk assessment and an owner. The summary should say plainly how many incidents there were, of what kind, and how they were resolved.

Automated execution changes some of this. Our piece on turning an automated test failure into a deviation covers how that should work, and test case generation shows how we write tests that make classification easier.