Clinical AI evaluation and safety

In plain words

Reading an AI paper is critical appraisal, the same skill you use for any clinical study, with a few extra questions. The site's Paper checklist page lists 15 questions to guide you, ready to print. The core question is always: does this give me good reason to believe the tool would help my patients, in my setting?

Healthcare example (illustrative)

A paper reports that an AI model reads chest X-rays as well as radiologists. Working through the questions, you notice the images came from a single hospital, the "correct answers" were set by one reader, the comparison was made on images alone without the clinical history radiologists normally have, and the model was never tested at another site. The headline claim may be true for that dataset and still say little about your department.

A little more technical

A good appraisal checklist asks about:

• the clinical problem, and whether solving it would matter;
• the data, the population and how the outcome labels were decided;
• the comparator (usual care, clinicians, another model);
• how the model was validated, and whether external validation was done;
• which metrics were reported, including calibration, not just AUC;
• possible bias and which groups might be served worse;
• clinical utility and generalisability to other settings;
• conflicts of interest;
• whether using the tool could actually change care.

Don't learn this yet

You don't need to re-analyse a paper's data or judge its model architecture. Focus on the questions above; they catch most serious problems.

Go deeper (optional)

• AI in Medicine I (BMIF 202) #30
• PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods #191
• Evaluations of AI Applications in Healthcare #21

In plain words

These are checklists written by expert groups. Most are reporting guidelines: they list what an AI study should tell readers so its methods and results can be judged. One, PROBAST+AI, is an appraisal tool for judging a study's quality and risk of bias. You don't have to memorise them; knowing which one fits which kind of study tells you what a good paper should contain.

Healthcare example (illustrative)

Imagine you are reviewing three papers for a journal club. One develops a model predicting ICU readmission: TRIPOD+AI is the relevant reporting guideline, and PROBAST+AI helps judge its risk of bias. One tests an AI tool's diagnostic accuracy for fractures on X-ray: STARD-AI applies. One is a randomised trial of an AI alert in the emergency department: CONSORT-AI applies, and SPIRIT-AI covers its protocol.

A little more technical

As the resource list records them:

• TRIPOD+AI (#22): a 27-item checklist for prediction models built with regression or machine learning; it supersedes TRIPOD 2015.
• PROBAST+AI (#23): quality, risk-of-bias and applicability assessment for prediction models.
• STARD-AI (#24): keeps STARD 2015 and adds new or modified items for AI diagnostic-accuracy studies.
• CONSORT-AI (#25): reporting randomised trials of AI interventions. SPIRIT-AI (#26): protocols for such trials.
• TRIPOD-LLM (#27): reporting for studies of large language models, separate in scope from TRIPOD+AI.
• CLAIM and DECIDE-AI have not yet been verified for the resource list, so they are not described here.

Don't learn this yet

Don't read every checklist item by item. Learn which guideline matches which study type, and check whether a paper says it followed one.

Go deeper (optional)

• TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods #190
• PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods #191
• The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence #192
• Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension #193
• Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension #194
• The TRIPOD-LLM reporting guideline for studies using large language models #234

In plain words

"Bias" in clinical AI means a systematic error that makes a model less accurate, or less fair, for some patients or situations. It can enter at every stage: which patients are in the data, how outcomes are labelled, how the model is built, how it is tested and how people use it. Naming the types helps you look for them.

Healthcare example (illustrative)

Imagine a model that estimates which patients need a specialist review, trained using past referral decisions as the "right answer". If some groups were historically referred less often for the same illness, the model learns to under-refer them too. It is accurate at copying past practice and unfair to the patients that practice already served poorly.

A little more technical

Common types:

• Selection or sampling bias: the training data do not represent the patients the model will see.
• Label or measurement bias: the outcome used for training is a poor or unequal stand-in for the real clinical truth, or is measured differently across groups.
• Historical bias: the data faithfully record past inequities.
• Spectrum bias: the model is tested on unusually clear-cut cases, so accuracy looks better than it will in practice.
• Deployment bias: the tool is used for a purpose or population it was not designed for.

A useful check is whether a paper reports performance separately for relevant subgroups.

Don't learn this yet

Skip the mathematical definitions of fairness and the debates between them. Learn to ask who is in the data, who set the labels, and who might be served worse.

Go deeper (optional)

• PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods #191

In plain words

Drift is when a model that worked when it was deployed gradually gets worse, because the world around it changes. Patients change, treatments change, equipment and documentation change, and the model does not notice. A model that passed validation once is not guaranteed to keep working.

Healthcare example (illustrative)

Imagine a hospital deploys a model predicting deterioration on surgical wards. Over the following period, the hospital introduces a new early-warning observation chart, a new laboratory analyser and an enhanced-recovery pathway that changes post-operative care. Each change alters the data the model sees. Nobody changes the model, but its alerts gradually become less accurate and its risk estimates less well calibrated.

A little more technical

Drift can be in the inputs (the patient mix or how values are measured), in the relationship between inputs and outcome (because practice changes), or in the outcome itself (a new definition or coding rule). The response is ongoing monitoring: regularly checking discrimination, calibration and alert rates against local outcomes, with a named person responsible and a plan for recalibrating, retraining or switching the model off. One argument in the literature is that one-off external validation should give way to repeated local validation (see #114).

Don't learn this yet

You don't need to know statistical drift-detection methods. Learn to ask of any deployed tool: who is monitoring it, how often, and what happens if performance falls?

Go deeper (optional)

• All models are local: time to replace external validation with recurrent local validation #114

In plain words

Automation bias is the tendency to accept a computer's output without enough checking, and to miss problems the computer did not flag. Alert fatigue is what happens when a system produces so many alerts, many of them false, that people start ignoring or overriding them, including the important ones. Both are human-factors problems: the model may be fine, and patients can still be harmed.

Healthcare example (illustrative)

• Automation bias: on a busy night, a junior doctor accepts an AI-drafted handover summary without re-reading the notes, and a pending blood result the summary left out is not chased.
• Alert fatigue: a sepsis alert in an emergency department fires for many patients who turn out not to have sepsis. Staff learn to click through it, and one day they click through a true case.

A little more technical

Automation bias is worse when people are busy, tired, less experienced or trust the system strongly, and when the output looks confident and polished. Alert fatigue grows when a model's positive predictive value is low, so most alerts are false alarms. Protections include showing the evidence behind an output, designing workflows where disagreeing with the AI is easy, choosing alert thresholds with front-line staff, and tracking override rates. The safe pattern stays the same: AI output, human verification, clinical judgement, then action.

Don't learn this yet

You don't need the human-factors research literature. Watch for these two effects in yourself and your team whenever an AI tool is introduced.

In plain words

Medicine learns a great deal from things that go wrong: morbidity and mortality meetings, incident reports, published case reports. The same approach works for AI. This roadmap plans a library of documented AI failures, each explained in plain words with what went wrong and what would have caught it earlier. Studying real failures makes the ideas in this track concrete.

Healthcare example (illustrative)

Imagine a hospital buys a deterioration-prediction tool from a vendor, switches it on and trusts the vendor's reported performance. Nobody checks it locally. Months later, an audit finds it misses many true cases and fires many false alarms. A failure-case write-up would trace the chain: no local validation, poor calibration in this population, low positive predictive value, alert fatigue, and no one monitoring performance.

A little more technical

A good failure case records the intended use, the data and validation behind the tool, what happened in practice, which evaluation step was skipped or misleading, and what monitoring or governance would have helped. One example in the resource list is a proprietary sepsis prediction model deployed at many US hospitals: an external validation found poor discrimination and calibration (#111), an accompanying editorial explains for clinicians why vendor models need local validation (#112), a later study in two emergency departments showed the earlier findings persisting (#113), and a further paper uses it among its examples to argue for ongoing local validation (#114).

Don't learn this yet

You don't need to memorise the numbers from these studies. Take away the pattern: a tool's performance elsewhere is not its performance here.

Go deeper (optional)

• External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients #199
• The Epic Sepsis Model Falls Short — The Importance of External Validation #112
• External validation of the Epic sepsis predictive model in 2 county emergency departments #200
• All models are local: time to replace external validation with recurrent local validation #114