AI GxP validation for regulated life sciences
Test Automation

Why a GxP Test Engine Should Not Be Able to Run Arbitrary Commands

The fastest way to automate a protocol is to let the engine run a shell command. It is also the fastest way to lose the argument with an inspector. Why a closed vocabulary of test steps is the safer design, and what you give up for it.

Cybroscape Technologies5 min read

Key takeaway

The fastest way to automate a protocol is to let the engine run a shell command. It is also the fastest way to lose the argument with an inspector. Why a closed vocabulary of test steps is the safer design, and what you give up for it.

There is a shortcut in every test automation tool, and it is always the same one: let the test step run a command. Give the engine a shell and it can do anything the system can do, so it can test anything. Nobody has to write a new step type ever again.

It is also the design that loses you the argument in an inspection, and it is worth understanding exactly why before you accept it in a tool you buy or build.

What a shell in the engine actually costs you

Three things, in rising order of seriousness.

  • The test becomes unreviewable. An approver looking at a protocol step that reads run: ./verify.sh --env prod has approved a filename, not a test. What the script does at run time is not what they signed.
  • The test can change without change control. The protocol is approved and frozen; the script it calls is a file on a disk that anybody with access can edit. Your approved protocol now executes something nobody approved.
  • The engine can do things a test should never do. A test that can run commands can write to the system under test, clean up after itself, or quietly fix the thing it was supposed to be checking. Once that is possible, every passing result carries an asterisk.

None of these are hypothetical failure modes of a badly-behaved team. They are properties of the design. A control you have to trust people not to use is not a control.

The alternative: a closed vocabulary

The other approach is to decide, in advance, the complete list of things a test step is allowed to be — and to make anything outside that list impossible rather than discouraged.

A workable vocabulary for GxP protocols is smaller than people expect:

  • Navigate to a named screen or endpoint.
  • Enter a value into a named field.
  • Click a named control.
  • Assert that a named thing is present, absent, equal to, or within a range.
  • Check a file exists, has a given checksum, or has given permissions.
  • Check a service is running, or reports a given version.
  • Capture a screenshot or a named log region as evidence.

Every one of those is fully described by its parameters. An approver reading the step knows precisely what will happen, because there is nothing else it can do. That is the entire point.

What you give up, honestly

You give up the long tail. Roughly one test in ten wants something the vocabulary does not have, and you now have two options rather than one: extend the vocabulary through your own change control, or execute that test by hand.

Both are slower than writing a script. That is the trade, and it is worth stating plainly rather than pretending the closed design is free. What you get back is that the other nine in ten are reviewable by a quality person who does not read code, and that no approved protocol can change underneath you.

The vocabulary also grows in the right direction. When a step type is added deliberately, it gets specified, tested and documented once, and then every protocol can use it. A shell gives you a thousand bespoke scripts, each of which is somebody's private understanding of what the test does.

How to assess a tool on this

If you are buying rather than building, the question to ask a vendor is blunt: can a test step execute code that is not part of your product?

  • If the answer is yes with no qualification, the tool is a scripting harness with a compliance skin. It can still be used, but the controls now have to live in your procedures and your access model, and you should plan the validation accordingly.
  • If the answer is yes but only via a plugin the vendor ships and qualifies, that is a reasonable middle. Ask which plugins, and what their qualification covers.
  • If the answer is no, ask what happens when a protocol needs something the vocabulary lacks. A tool with no honest answer to that has simply moved the problem.

We took the closed route in Cybroscape GxP Copilot, with no general command step by design, and the constraint has been productive more often than it has been annoying — mostly because it forces the question "what is this step actually verifying?" at design time rather than at review.

Where to go next

Explore Cybroscape GxP Copilot for AI-native validation, Cybroscape TraceDraft for source-traceable clinical documentation, or book a demo to see either on your own data.

High riskFailure could affect patient safety or product quality
Scripted testingWritten steps, expected results, an independent reviewer
Step-by-step results, screenshots and signatures
Medium riskFailure is noticed and contained
Unscripted testingExploratory or ad-hoc testing against the intended use
A record of what was tested, by whom, and the result
Low riskStandard function, supplier-tested
Supplier evidenceThe vendor’s own testing, assessed and accepted
The assessment and the reason it is enough
Computer Software Assurance in one picture: test effort follows the risk of each requirement. Based on FDA’s Computer Software Assurance guidance.
See it on your own systemBring a URS or a system. We run GxP Copilot against it live on the call.
gxp test automation securityclosed vocabulary test stepsautomated validation executiontest engine design gxpcsa automation

Keep reading

All articles
Test AutomationAutomating IQ/OQ/PQ Protocols: What Works, What Fails, and What Inspectors ExpectWhere test automation adds genuine value in IQ/OQ/PQ execution, where it introduces new compliance risk, and how to build an audit-ready automated protocol record. · 5 min readTest AutomationGxP Test Automation: The Complete Technical Guide Under CSA in 2026How to build a GxP test automation strategy under Computer Software Assurance: what to automate, what not to, how to document it, and how to keep it defensible under inspection. · 6 min readTest AutomationGxP Test Automation and 21 CFR Part 11: The Audit Trail Requirements That Most Teams MissThe Part 11 requirements that apply specifically to automated test execution (audit trail, e-signature, result attribution, and data integrity) with a defensible record structure. · 5 min readTest Automation7 Best GxP Validation Test Automation Tools (2026): CSA, UI, API & Equipment AdaptersA technical comparison of GxP test automation tools, including Cybroscape GxP Copilot (12 closed-vocabulary adapters across Web, API, SQL, OPC UA & data loggers), Tricentis Tosca, Vera, Playwright and Selenium, scored on 21 CFR Part 11 evidence, maintenance cost, and FDA CSA alignment. · 6 min readTest AutomationYour Test Automation Tool Is Also a GxP SystemIf a tool produces the evidence that another system is validated, the tool is in scope. The recursive problem nobody wants, and the proportionate way through it. · 4 min readTest AutomationRe-Running a Protocol After a Change: How to Scope Regression HonestlyAutomation makes re-running everything cheap, which is exactly why teams stop thinking about what actually needed re-running. Scoping regression on risk rather than on convenience. · 4 min readTest AutomationRisk-Based Test Automation in GxP: How to Build a Defensible Automation StrategyHow to apply per-requirement risk scores to drive test automation decisions: what gets scripted automation, what gets exploratory execution, and what borrows vendor evidence. · 4 min readTest AutomationWhen Not to Automate a Validation ProtocolThe honest counterweight to every automation pitch. Six situations where a scripted manual execution is the better answer, and how to say so in a way that survives review. · 4 min read

Bring a system. We'll show you the package.

Book a demo, or ask a validation lead a quick question on WhatsApp.

  • We reply within one business day
  • We run your URS live on the call
  • You get a clear plan for your system
Chat on WhatsApp

Book a demo

Four fields. The right specialist replies within one business day.

Your details stay private. We never share them.