GxP AI validation is documented evidence that an AI system does the specific job you rely on it for, reliably enough for the risk involved, inside a process where a qualified person stays accountable for the decision. That's the whole idea. Everything below is how you actually produce that evidence.
If you've validated software before, most of this will feel familiar — intended use, risk, testing, change control. But AI breaks a few assumptions classic validation quietly relies on, and those are exactly where teams get caught out. We'll go through what's different, what regulators expect, and a step-by-step method we use ourselves.
Why validating AI is different from classic CSV
Classic computer system validation assumes a few things that AI doesn't always honour:
- Same input, same output. Traditional software is deterministic. Generative AI can give a different answer to the same question tomorrow.
- Behaviour comes from code. With AI, a lot of the behaviour comes from data — training data, and the documents you feed it. Change the data and you change the behaviour.
- It only changes when you release. AI can change when a vendor swaps the model underneath, or when you edit a prompt or template.
- Wrong answers look wrong. AI can be fluent, confident and incorrect. That's harder to catch than an error message.
None of this makes AI unvalidatable. It means you test differently: with representative real data, repeat runs, deliberate edge cases, clear acceptance criteria, and a plan for what happens after go-live. Compare the two approaches in CSA vs CSV for AI validation.
What regulators expect
There isn't one AI rulebook yet, but the direction is consistent across regulators.
- EU GMP Annex 22 (draft). The first GMP text written specifically for AI. For critical GMP uses it expects static models with consistent output, a written intended use, test data that is representative and independent of training data, acceptance criteria at least as good as the process being replaced, explainability, and human review. It does not expect generative or continuously learning models in critical decisions. It was a consultation draft at the time of writing — check current status. More in Annex 22 requirements.
- EU Annex 11 and 21 CFR Part 11. Everything that applies to computerised systems still applies: audit trails, access control, electronic signatures, data integrity. See Part 11 compliance.
- FDA. The Computer Software Assurance guidance sets the risk-based mindset. FDA's January 2025 draft guidance on AI supporting regulatory decisions for drugs and biologics adds a risk-based credibility assessment built around the model's context of use — its formal scope is narrower than internal tooling, but the thinking applies widely.
- GAMP. GAMP 5 Second Edition covers AI and machine learning, and ISPE has since published dedicated GAMP guidance on AI.
Put simply: know exactly what the AI is for, test it on honest data against criteria you set in advance, keep a person accountable, and keep watching it. Regulator-by-regulator detail is in FDA vs EMA vs MHRA on AI.
The eight-step method
1. Write the intended use
One page, plain words. The task, the inputs, the output, who reviews it, the decision it feeds — and what it must never be used for. Everything else depends on this. How to write one: AI intended use statement.
2. Assess risk and set the GAMP category
If the output is wrong, what's the worst realistic outcome, and would a person catch it first? That sets the rigour. An AI feature you only configure usually sits around Category 4; a model you trained or tuned on your own data behaves like Category 5. See GxP risk assessment.
3. Assess the supplier and the data
For vendor AI: how model changes are controlled and notified, whether your data is used for training, where it's processed. For your own model: where the training data came from and whether it represents real use. See qualifying an AI vendor.
4. Set acceptance criteria before testing
Tie them to the cost of a mistake and to how well today's manual process performs. Separate limits for missed items, false alarms and confident-wrong answers. Get QA to approve them first. See acceptance criteria for AI.
5. Build an honest test set
Real material including the messy cases, agreed correct answers prepared in advance, and nothing the model was trained on. See building the test set.
6. Test — including the uncomfortable cases
Run the test set against the criteria. Repeat key cases to check consistency. Feed it out-of-scope inputs and check it says it isn't sure instead of guessing. Record every failure — a failure you remove afterwards is exactly what an inspector looks for.
7. Design the human oversight, and write it into SOPs
Who reviews, what they check, how you know the review is real and not a rubber stamp, and what the fallback is when the AI is unavailable. Train reviewers on how this AI fails. See SOPs and training for AI.
8. Release, monitor and control change
Validation doesn't end at go-live. Monitor performance against limits, record the model version with every output, and name AI-specific change triggers — new model, prompt edits, new input types. See change control and revalidation and continuous monitoring.
The deliverables
A typical AI validation package. Size each one by risk — a low-risk drafting aid doesn't need the same weight as something feeding a quality decision.
- Intended use statement
- Risk assessment and GAMP category rationale
- Supplier assessment (or model development record, for your own model)
- Test set record: sources, ground truth, independence from training data
- Acceptance criteria with rationale, approved before testing
- Test protocol and report, including repeatability and out-of-scope results
- Human oversight design, SOP updates and training records
- Monitoring plan and AI-specific change control triggers
- Validation summary report
Mistakes that fail inspections
- Testing on the vendor's demo documents. Clean inputs predict nothing about your messy ones.
- A round-number accuracy target with no reason behind it. The first question will be "why that number?"
- Letting the AI decide. If an AI output goes into a GMP-critical decision without a person approving it, you have an Annex 22 problem, not a validation problem.
- Letting the AI run and judge the tests. Test evidence has to be repeatable. See why AI should draft tests, not run them.
- Validating once and forgetting. A silent model update can expire your validation without anyone noticing.
- No recorded errors. An AI with zero logged mistakes suggests nobody is looking, not that it's perfect.
A good dry run before any inspection: 14 questions inspectors ask about AI.
How we approach it
Our own rule is simple: the AI proposes, a person decides. Every AI output carries its source and a confidence signal, and anything uncertain goes to a human by default. When AI helps with testing, it only drafts steps from a fixed list of approved actions; a qualified person approves them; and a plain, non-AI engine runs exactly what was approved. No AI is involved at run time, and a failed step becomes a deviation like any other.
That's how GxP Copilot is built, and it's the same method we use when we validate AI for clients through AI validation services. For the wider landscape of AI in regulated work, see GxP AI.
Where to go next
Explore GxP Copilot for AI-native validation, TraceDraft for source-traceable clinical documentation, or book a demo to see either on your own data.