Four factors for judging AI risk in GMP: uncertainty, complexity, model influence, detectability

Model influence: what moves downstream

Start here, because it decides most of the answer and it is the one people conflate with capability.

The question is not how sophisticated the model is. It is what changes as a result of its output. An AI drafting a protocol that a qualified reviewer approves line by line carries different influence from one scoring a deviation that routes an investigation, which carries different influence again from one contributing to a release decision.

Low influence is not a loophole. It is the reason most AI adoption in regulated life sciences is defensible at all: the overwhelming majority of it drafts work that a human then owns. Saying so explicitly, with the reasoning recorded, is what lets you concentrate assurance effort where influence is genuinely high. AI Risk Assessment

Detectability: would you find out?

The factor most often left out, and usually the one that changes the conclusion.

A wrong output that a downstream control catches immediately is a different risk from one that enters a record and surfaces eighteen months later during an inspection. Classical risk management has always accounted for detectability; AI assessments frequently drop it, which is how systems with good controls end up rated identically to systems with none.

Ask it concretely. If this output were wrong, what would notice — a person, a reconciliation, a downstream check, a periodic review, or nothing? And how long would it take? A fluent, plausible, wrong statement is the hard case precisely because it has low detectability: an obvious error gets caught, and a confident one does not.

This is also why source traceability earns its place in document generation. A statement a reviewer can verify in place against its source has materially higher detectability than one requiring them to go and find the source, and the difference shows up on page 180 where attention is thinnest.

Uncertainty: does the system know when it does not know?

Models differ in whether they signal low confidence and in whether anything is done with the signal.

A system that behaves differently when uncertain — routing to a human, declining to answer, flagging the output — is meaningfully safer than one that produces its best guess with uniform confidence. The second case is common, and it pushes the entire burden onto review.

So two questions. Does the system express uncertainty in a way the workflow can act on? And does the workflow act on it, or is the signal displayed and ignored? A confidence score that nobody routes on is telemetry, not a control.

Related and worth separating: uncertainty about the output versus uncertainty about the input. A model given data outside the distribution it was evaluated against is operating beyond its qualified range, which is a situation conventional validation has language for. the AI validation test set

Complexity: can you explain it to the person who has to defend it?

Complexity matters less for its own sake than for what it does to the other three factors. A system nobody can explain is a system whose failure modes nobody can enumerate, which makes detectability unassessable and uncertainty unknowable.

The practical test is not whether the architecture is interpretable in an academic sense. It is whether the people accountable for the output can describe what the system does, what it is approved for, where it is known to be weak, and what would make it fail. If they cannot, the risk assessment is a formality regardless of its conclusion.

Complexity also compounds across a workflow. A single model with a reviewed output is tractable. A sequence of steps each conditioned on the last, with a different path each run, is not — reviewing the final output no longer tells you the process was sound. agentic AI systems covers where that line sits.

Using the four together

The combinations are more informative than the individual ratings, and three recur.

• High influence, low detectability. The case that warrants the deepest assurance available — the output matters and nothing downstream would catch it being wrong. If you find one of these, it is the priority.
• High influence, high detectability. Manageable, and often manageable cheaply, because the detection is doing real work. Document the downstream control as the mitigation it is rather than leaving it implicit.
• Low influence, high complexity. Where over-assessment happens. A system nobody understands, drafting something a human rewrites anyway, attracts scrutiny out of proportion to what depends on it.

This is ICH Q9 applied rather than replaced, which is the framing the workshop emphasised throughout. The factors are not a new methodology — they are a better hazard list for an unfamiliar case.

What it looks like written down

For each AI use case, four short statements and a conclusion that follows from them. What depends on this output. What would detect it being wrong, and how fast. Whether the system signals uncertainty and whether the workflow acts on it. Whether the accountable people can explain what it does and where it is weak.

Then the assurance approach, proportionate to those answers, with the reasoning recorded. The reasoning is what an inspector asks for — not whether you followed a framework, but how you decided. A light approach with documented reasoning is defensible; the same approach without it reads as an unassessed system, and the difference is entirely in the writing.

AI governance is where this fits in a governance model, The risk-based control strategy Annex 22 will ask for covers the wider control strategy, and book a demo if you want a use case assessed against these four.