Data and statistics for clinical AI

In plain words

An electronic health record (EHR, also called EMR) is not one document but many linked tables: demographics, admissions, observations, blood results, prescriptions, procedures, diagnoses and free-text notes. Each entry has a time stamp, so a patient's record is really a timeline. Any AI model built on hospital data is built on these tables, with all their gaps and quirks.

Healthcare example (illustrative)

Imagine a critical-care team wants to predict which patients will need re-intubation. The ventilator settings sit in one table, the arterial blood gases in another, the sedation chart in a third and the extubation decision only in a free-text note. Before any modelling, someone has to line these up on the same timeline for each patient, and decide what to do when a value was never recorded.

A little more technical

Structured data (coded fields, numbers, dates) is easy for software to read. Unstructured data (notes, letters, reports) holds much of the clinical reasoning but needs extra processing, such as natural language processing, before a model can use it. Turning raw records into model inputs is called feature construction: for example, "highest lactate in the last six hours" is a feature built from many individual results. The choices made here quietly shape what the model learns.

Don't learn this yet

You don't need to know how any particular hospital system stores its tables, or to write database queries (SQL). Understand the idea of the patient timeline and the difference between structured and unstructured data.

Go deeper (optional)

• Introduction to Clinical Data #13
• AI Integration in Healthcare Patient Data (part of Digital Healthcare Informatics and AI Specialization) #14
• Introduction to Healthcare (course 1 of AI in Healthcare) #12

In plain words

These are shared standards that let different systems describe and exchange health data in the same way. Some are codes for clinical concepts (ICD for diagnoses, SNOMED for clinical terms in general, LOINC for laboratory tests and observations). Some are formats for exchanging data (HL7 and its modern version FHIR for records, DICOM for medical images). OMOP is a common data model: a shared layout that lets researchers analyse data from many hospitals with the same code.

Healthcare example (illustrative)

A radiology AI tool receives chest CT images from the scanner in DICOM format. The emergency department's system sends it the patient's details and request through an HL7 or FHIR message. If a research network later wants to study outcomes across several hospitals, each hospital's data might be converted to OMOP, with its local lab names mapped to LOINC codes and diagnoses to standard vocabularies, so one analysis runs everywhere.

A little more technical

FHIR breaks health information into small building blocks called "resources" (a patient, an observation, a medication) that can be requested and combined. The FHIR specification is open (see #103), and #105 notes that the older R4 version is still the one most live implementations profile against. Mapping local codes to standard ones is where many errors creep in: a lab test mapped to the wrong code silently corrupts every model trained on it.

Don't learn this yet

You don't need to read the specifications or learn the code systems by heart. Know what each standard is for, and ask of any dataset or tool: which standards does it use, and who checked the mappings?

Go deeper (optional)

• 6.S897/HST.956 lecture notes, Lecture 1 (clinical data and standards) #19
• FHIR Overview (road map for first-time readers) #104
• The Book of OHDSI (open textbook) #106
• HL7 FHIR R5 specification (home) #103

In plain words

Three data problems quietly ruin models. Poor data quality: values that are missing, wrong, duplicated or recorded inconsistently. Leakage: the model is accidentally given information during training that would not be available at the moment it is meant to make its prediction, so it looks brilliant in testing and fails in real use. Shift: the patients, practices or recording habits change between the data the model learned from and the place it is used.

Healthcare example (illustrative)

• Leakage: a model to predict sepsis on the ward learns that "blood cultures were sent" and "broad-spectrum antibiotics were started" mean sepsis. It scores highly in testing, but by the time those orders exist the clinician has already recognised sepsis, so the model adds nothing.
• Shift: a model trained on an adult general-medicine population is used in a surgical high-dependency unit, where the patients, observations and treatments differ.

A little more technical

Missing data is rarely random in medicine: a test is often missing because a clinician judged it unnecessary, which is itself information. Leakage often hides in time stamps, in codes entered after discharge, or when the same patient appears in both the training and the test data. Shift (also called dataset shift) can be in the patients, in how outcomes are defined, or in equipment and documentation. Each one means a reported result may not hold where you work.

Don't learn this yet

You don't need to learn methods for filling in missing values or statistical tests for shift. Learn to ask: "Could the model have seen the answer early?" and "Were these patients like mine?"

Go deeper (optional)

• AI Integration in Healthcare Patient Data (part of Digital Healthcare Informatics and AI Specialization) #14
• The Book of OHDSI (open textbook) #106

In plain words

Sensitivity is the proportion of people with the condition that the test (or model) correctly flags. Specificity is the proportion of people without the condition that it correctly clears. The same ideas apply to AI as to any diagnostic test. What a clinician usually wants to know at the bedside is the positive predictive value (PPV): if the model flags this patient, how likely is it that they really have the condition?

Healthcare example (illustrative)

Imagine an emergency department AI tool that flags possible pulmonary embolism. High sensitivity means it rarely misses a true embolism; high specificity means it rarely flags patients who do not have one. If pulmonary embolism is uncommon among the patients it screens, even a tool with good specificity will produce many false alarms, so most flagged patients will not have the condition. The PPV tells you this; sensitivity and specificity alone do not.

A little more technical

These numbers come from a confusion matrix, a two-by-two table of true positives, false positives, true negatives and false negatives. Most AI models output a score, not a yes/no, so someone chooses a cut-off (threshold); moving it trades sensitivity against specificity. PPV and negative predictive value (NPV) depend on how common the condition is in the population tested, so they change from one setting to another even when the model does not.

Don't learn this yet

Skip the formulas for likelihood ratios and confidence intervals for now. Get comfortable reading a two-by-two table and asking what threshold was used.

Go deeper (optional)

• StatQuest video index (Confusion Matrix; Sensitivity & Specificity; Cross-validation; Bias-Variance; Logistic regression) #110

In plain words

A ROC curve shows how a model's sensitivity and false-positive rate change as you move its threshold from strict to lenient. The area under the ROC curve (AUC, or C-statistic) summarises this in one number: roughly, how often the model gives a higher score to a patient with the condition than to one without. A precision–recall (PR) curve and its area (PR-AUC) do something similar but focus on how many flagged patients are true cases, which matters more when the condition is rare.

Healthcare example (illustrative)

Two teams report models to predict post-operative delirium after major surgery, and both have similar ROC AUCs. Delirium is relatively uncommon in their populations. When you look at the precision–recall curves, one model keeps precision reasonable at useful sensitivity and the other floods the ward with false alerts. The ROC AUC alone would not have shown the difference.

A little more technical

The ROC curve is built by computing a confusion matrix at many thresholds and plotting each result. AUC measures ranking (discrimination) only: it says nothing about whether the predicted risks are correct in size (that is calibration), or whether using the model helps patients (that is clinical utility). A model can have a respectable AUC and still be useless at the threshold a hospital actually uses.

Don't learn this yet

You don't need to calculate AUCs or compare them statistically. Know what the curve shows, why PR curves matter for rare outcomes, and that AUC is not the whole story.

Go deeper (optional)

• ROC and AUC, Clearly Explained! (single video) #109
• StatQuest video index (Confusion Matrix; Sensitivity & Specificity; Cross-validation; Bias-Variance; Logistic regression) #110

In plain words

Calibration asks whether a model's predicted risks match what actually happens. If a well-calibrated model says a group of patients each have a "20% risk", about one in five of them should go on to have the outcome. A model can rank patients well (good discrimination) but still give risks that are consistently too high or too low.

Healthcare example (illustrative)

Imagine a risk model for post-operative acute kidney injury, developed in a hospital where the complication was common. It is used in another hospital where it is less common. The model still ranks patients sensibly, but it overestimates everyone's risk. Anaesthetists who take the numbers at face value might delay surgery or change plans for patients who were at low risk all along.

A little more technical

Calibration is usually shown as a calibration plot: predicted risk on one axis, observed rate on the other, with a perfect model along the diagonal. Summary measures include the calibration slope and intercept (whether predictions are too extreme or too high or low overall). Calibration often drifts when a model moves to a new population or as practice changes over time, and it can sometimes be corrected locally without rebuilding the model (recalibration).

Don't learn this yet

You don't need to learn the statistical tests for calibration. Learn to look for a calibration plot in any paper that reports risk numbers, and be wary when there is none.

In plain words

Decision curve analysis asks a practical question: would using this model to make decisions do more good than harm, compared with treating everyone or treating no one? It weighs the benefit of catching true cases against the harm of acting on false alarms. The result is shown as "net benefit" across a range of thresholds at which a clinician might choose to act.

Healthcare example (illustrative)

A general-medicine team considers a model that identifies patients at high risk of venous thromboembolism, to target extra prevention. A decision curve compares three strategies: give extra prevention to everyone, to no one, or only to those the model flags. If the model's curve sits above the other two across the range of thresholds clinicians find reasonable, the model may be useful; if not, a simpler strategy is as good or better.

A little more technical

The threshold reflects how a clinician values a missed case against an unnecessary intervention: a low threshold means "I would act even at modest risk". Net benefit counts true positives and subtracts false positives weighted by that threshold. Unlike AUC, a decision curve links model performance to a clinical decision, which is why it is a useful bridge between statistics and clinical utility. It still does not prove patient benefit; that needs a prospective study.

Don't learn this yet

You don't need to compute net benefit. Understand what the curve compares and recognise one when a paper includes it.

In plain words

Validation means testing a model on data it did not learn from. There is a ladder of increasingly convincing tests. Internal validation tests it on held-back data from the same source. External validation tests it on data from a different hospital or population. Temporal validation tests it on later patients, to see if it still works as time passes. Prospective validation tests it in real time, on new patients, often in the actual workflow.

Healthcare example (illustrative)

A team builds a model to predict difficult laryngoscopy from pre-operative assessment data at their own hospital. Testing on a held-back portion of their own patients is internal validation. Testing on records from a hospital in another region is external. Testing on their own patients from a later period is temporal. Running it silently alongside anaesthetists' usual assessments, and comparing predictions with what actually happens, is a step towards prospective validation.

A little more technical

Internal methods include splitting the data or cross-validation, where the data are divided into parts and each part takes a turn as the test set. Performance usually falls as you move up the ladder, because new sites and later periods bring dataset shift. Prospective studies, and ideally trials, are needed to show that using the model changes care or outcomes; accuracy on past records does not show that. Track 3 builds on this with evaluation and monitoring after deployment.

Don't learn this yet

You don't need to know how to run cross-validation or design a trial. When you read about a model, ask which rung of the ladder it has reached.