Testing an LLM for Hallucinations Before It Touches GxP Work

Define what counts as a hallucination for your use

Be specific. For a tool that drafts from source documents, a useful definition is: any statement of fact that the provided sources do not support. That includes invented numbers, wrong dates, a correct fact attributed to the wrong batch, and a summary that changes the meaning of the source.

Decide in advance which kinds are critical. A wrong lot number in a deviation is worse than an awkward sentence, and your limits should reflect that.

Build a test set from your own documents

Use real inputs from the process the tool will support, with the sources attached and the correct answers agreed by people who know the work. Include the hard cases: incomplete records, conflicting sources, questions the sources cannot answer. A good tool should say it does not know rather than fill the gap.

Check claim by claim, not answer by answer

Break each output into individual factual statements and check each one against the source. Marking a whole answer "mostly right" hides exactly the errors that matter. Record every unsupported statement, its type and whether it was critical.

Run each test more than once. Generated text can change between runs, so a single pass understates the problem.

Set the limit before you look

Agree the acceptable rate of unsupported statements, with a lower limit for critical ones, before testing starts. If the tool cannot meet it, narrow the use case or strengthen the review step rather than relaxing the limit.

Keep measuring in use

Reviewers will correct errors after go-live. Capture those corrections as data: how often, what kind, and in which documents. A rising rate is your signal to investigate, and it is the evidence an inspector will ask for.

Under the current draft of EU GMP Annex 22, generative models are kept out of critical GMP decisions, and use elsewhere relies on a human in the loop. The draft is still under consultation and may change, as EMA's workshop report suggests, but the testing above is worth doing whatever the final text says. For the wider picture, see validating LLMs under Part 11.