When Not to Automate a Validation Protocol

Six cases where manual is the better answer

• You will run it once. A one-off migration verification, a decommissioning check. Automation pays back over repetition; with no repetition there is nothing to pay it back.
• The test requires judgement. "The report is legible and the layout is suitable for the intended user" is a real acceptance criterion and a machine cannot assess it. Automate the numbers in the report; let a person judge the report.
• The interface is about to change. Automating against a UI that is mid-redesign buys you a maintenance commitment and no stability.
• Setting up the state costs more than the test. Some tests need a specific, elaborate data condition. If building that state reliably is most of the work, the automation is mostly a fixture, and fixtures rot.
• It is a physical or environmental check. Cabling, labelling, temperature, the location of a machine. Somebody has to look.
• The system is out of support or leaving. Effort spent automating a system you are retiring in a year is effort not spent on its replacement.

The honest arithmetic

A simple frame that survives challenge: automation is worth it when build cost plus lifetime maintenance is less than manual execution cost times the number of executions you will actually perform.

Two numbers get under-estimated in that comparison, reliably. Maintenance is the first — an automated test is code, and code needs updating whenever the system changes. The second is the number of executions, which people set optimistically at the point of maximum enthusiasm.

A practical rule of thumb: if you cannot name at least five future executions with a straight face, write it as a scripted manual protocol and move on.

Saying no in a way that survives review

The risk of declining to automate is that it reads as a shortcut. It does not have to, provided the reasoning is written down at the time rather than defended afterwards.

In the validation plan, record which protocols are automated and which are not, with a reason for each. "Executed manually; single execution expected as part of decommissioning" is a complete justification. So is "executed manually; acceptance criteria require assessment of report legibility".

What does not survive review is silence — a plan where some things happen to be automated and others happen not to be, with nothing recording why.

The middle ground people forget

It is not binary. A great deal of the value in automation comes from partial application, which the all-or-nothing framing tends to obscure:

• Automate the setup, execute by hand. Get the system into the required state reliably, then let a person perform the test and judge the result.
• Automate the evidence capture. A person drives; the tool records timestamps, versions and screenshots consistently. This removes the transcription errors without touching the judgement.
• Automate the repeatable subset. Most protocols have a handful of steps that are pure assertion and a few that need a person. Split them rather than forcing the whole protocol one way.