Agentic AI in regulated environments: patterns that hold up in audit

Why autonomy is the problem, not capability

A single-output system is straightforward to govern: one input, one output, one review. An agentic system takes a sequence of steps, each one conditioned on the last, and the sequence differs between runs.

That creates two difficulties. Reviewing the final output no longer tells you the process was sound, because an acceptable answer can be reached through steps that were not. And reproducibility weakens — a system that takes a different path each time is difficult to qualify in the way GxP expects, because the thing you qualified is not quite the thing that runs.

The draft EU GMP Annex 22, published for consultation in July 2025 and not yet in force, points away from dynamic and probabilistic models in critical GMP applications for roughly this reason. EMA has indicated the scope may change following consultation, so this is unsettled rather than decided — but the underlying concern is not going to disappear, because it is a real one.

The pattern that works: bounded steps, human gates

Decompose the goal into a fixed sequence of steps defined in advance, each with a defined input and output, and put a human gate where a step produces something that will enter a regulated record.

The agent does the work within each step. It does not choose the steps, and it does not decide whether the output of one step is good enough to proceed. That decision is the gate, and the gate is a qualified person.

This is less impressive than an agent given a goal, and it is the version that can be validated. You can state what each step is for, evaluate each step against acceptance criteria, and demonstrate that nothing reached a record without a signature. Human-in-the-Loop reviews is how the gate is enforced in our product — architecturally, so that output cannot progress without it, rather than by a procedure asking people to review.

Keep agents out of evidence generation

This is the line we hold most firmly and it is worth stating plainly: an AI system should not execute the test that produces the evidence that the system works.

If an agent runs a validation test and reports a pass, the evidence is only as good as the agent's own correctness, which is the thing being established. It is circular, and an inspector will see it immediately.

Execution of tests should be deterministic. A test that runs the same way every time, produces the same artefacts, and whose own behaviour is qualified, is evidence. An agent deciding how to exercise a system and judging the outcome is not. AI drafting the test case and a human approving it, then a deterministic engine running it — that sequence keeps the evidence clean. why AI should not run validation tests and closed-vocabulary test automation go into why.

What has to be recorded

For an agentic workflow the audit trail requirement is heavier than for a single-output system, because the process is part of what needs to be demonstrable.

• Which steps ran, in what order, and with what inputs — not just the final output.
• What each step produced, retained rather than discarded once consumed by the next step.
• Where a human gate was reached, who passed it, and what they saw at the time.
• Where the agent failed, retried, or took an alternative path, and why.

That last one is routinely absent. A system that silently retries a failed step presents a clean record of a process that did not go cleanly, and the absence of the retry is itself a data integrity issue.

Tool access is the risk surface

An agent is only as constrained as the things it can do. If it can write to a validated system, it can affect a regulated record without a human in the path, whatever the design intended.

So the practical control is narrow permissions rather than careful instructions. Read access where reading is sufficient. Write access only to draft states that cannot progress without a signature. No access at all to approved records, audit trails or configuration that controls a pass or fail decision.

This is the same reasoning as segregation of duties applied to a non-human participant, and it should be tested the way segregation of duties is tested — by attempting the action and confirming refusal, not by reading the permission matrix.

Questions to ask any vendor selling agentic AI

Including us.

• Is the sequence of steps fixed and defined, or chosen by the model at run time?
• Where are the human gates, and can output reach an approved state without passing one? Ask them to try it.
• Does the agent generate the evidence that the system works? If so, why is that not circular?
• What can it write to, and what happens if it attempts to write to an approved record?
• Is the full step sequence retained, including retries and failures?
• What happens to the process when the agent is unavailable?

agentic AI systems is our position in more detail, automation off by default covers why we default automation to off, and book a demo if you want an agentic design reviewed before it is built rather than after.