Clinical AI validation
Independent validation of a diagnostic AI before procurement
How an independent evaluation of a diagnostic AI tool is scoped, run and reported so a health-system buyer can make a safe, evidenced decision.
Illustrative example. This is a representative worked example of how we structure this kind of work — not a specific client engagement. It contains no client names, confidential information or achieved metrics. Real client work is confidential and shared only anonymised, with permission, under NDA.
The challenge
A digital-health company has built an AI tool that flags a clinical finding from routine data, and a prospective NHS buyer has asked for independent evidence that it is safe and performs as claimed — not the vendor’s own marketing figures. The company needs an evaluation that a clinical-safety officer and a procurement panel will both accept, without exposing patient data unnecessarily or over-claiming a result that will not hold in the buyer’s population.
Approach
How the work is structured
The evaluation is designed before any data is touched, with the measures of success agreed up front so the result is meaningful rather than retrofitted.
- Define intended use and claims. Pin down exactly what the tool does, for whom and in what setting, and translate the marketing claims into testable performance questions.
- Pre-register the protocol. Agree the reference standard, the test population, the primary and subgroup metrics and the acceptance thresholds in writing, before seeing results.
- Evaluate performance and bias. Measure performance against the reference standard on an appropriate test set, and — critically — break results down by subgroup to surface any bias or blind spots.
- Assemble the assurance pack. Package the findings, limitations and evidence into a form a clinical-safety case and a procurement panel can rely on.
Result
What a good result looks like — and how it is measured
The deliverable is an independent, procurement-ready evaluation report — its value is in being trustworthy, which means stating limits as clearly as strengths.
- Performance against the pre-agreed reference standard (e.g. sensitivity, specificity, predictive values) on a defined test set
- Subgroup performance, to reveal where the tool is weaker and for whom
- A plain statement of what the evidence does NOT show, and the conditions under which it would not transfer
- Evidence mapped to the buyer’s assurance requirements (DTAC domains)
Transferability
Would this transfer to your setting?
A result only transfers when the buyer’s population and workflow resemble the test conditions. The report states those conditions explicitly, so a buyer can judge relevance to their own setting rather than assuming a headline figure applies to them.
Answers
Clinical AI validation: frequently asked questions
Is this a real client engagement?
No. This is an illustrative worked example that shows how we structure and report an independent validation. It contains no client-identifying information and no fabricated results. Real client work is confidential and shared only anonymised, with permission, under NDA.
Why pre-register the protocol?
Agreeing the measures and thresholds before seeing results prevents the outcome being retrofitted to flatter the tool. It is the single biggest thing that makes an evaluation credible to a clinical-safety officer.
Want proof relevant to your project?
Tell us your sector and challenge; we'll share matched, anonymised examples and references.