Automation bias is the tendency to accept a machine's output without checking it properly, especially when it is usually right. It is well documented in aviation and medicine, and it is the quiet risk in every "AI drafts, a human approves" process.
A reviewer handed fluent, plausible text and a queue of fifty more will approve most of it. That is not negligence. Checking each statement means opening the source and comparing, again and again. Unless the process makes checking easy, the human review exists on paper only. Real review is the heart of GxP AI validation.
Why better models make it worse
As the AI improves, errors become rarer and harder to spot, and reviewers learn to trust it. The errors that remain are the convincing ones. Review quality has to be designed to hold up as the model gets better, not to rely on errors being obvious.
Design the review so checking is the easy path
- Show the source beside the draft. Each statement should link to the record it came from, so checking takes a click, not a search.
- Flag what the source does not support. Mark statements without a source so the reviewer's attention goes where the risk is.
- Separate drafting from approval. The person who ran the AI should not be the one who approves its output.
- Make the signature mean something. Re-authentication at signing, with the meaning of the signature recorded.
Measure whether review is real
Track how often reviewers change AI output and how long they spend. An approval rate close to 100% with very short review times is a warning sign, not a success.
One direct test: from time to time, insert a known error into a draft under controlled conditions and see whether review catches it. Document the method, tell reviewers that such checks happen, and use the results to improve the process, not to blame individuals.
Keep the AI out of the decisions that matter most
The surest defence is not to ask a reviewer to catch an AI's judgement at all. Severity, root cause, risk classification and release decisions should be made by people, or calculated from stated rules, with the AI helping with the writing around them. See who decides in GxP AI and what the audit trail must record.
Where to go next
Explore GxP Copilot for AI-native validation, TraceDraft for source-traceable clinical documentation, or book a demo to see either on your own data.