W3 · Diagnostics, prediction and external evidence

Workshop W3, teaching units 09–12: predictive values at two prevalences, a baseline against an alternative model, calibration and threshold with subgroup denominators, and a journal club on external validation.
Author

R. Heller

Part II · Testing and comparing models · Level: application · Teaching units: 09–12 (4 × 45 min) · Required reading: core readings of B06 and B09 (about 20 minutes) · Lab: Lab W3

Guiding question

What does an alert mean for the next patient when prevalence changes, and who carries the errors of the threshold we choose?

Learning objectives

By the end of this workshop you can

  1. fill in a two-by-two table for 1,000 examinations from prevalence, sensitivity and specificity and compute the predictive values by hand and in R;
  2. compare a pre-specified logistic baseline with a tuned alternative on fixed test identifiers, keeping all pre-processing and tuning inside the training data;
  3. read a calibration plot and explain why a good AUC does not guarantee usable risk estimates;
  4. propose a threshold and report errors by subgroup with denominators and confidence intervals, naming the group that is disadvantaged and the information still missing;
  5. separate evidence on diagnostic accuracy from evidence on patient benefit when reading a published study.

Preparation

Teaching units

Unit Task Minutes Observable evidence
09 For 1,000 examinations with 5% or 20% prevalence and an assumed sensitivity and specificity of 90%, determine all frequencies by hand, then in R (Lab W3, Part A). 45 Two two-by-two tables. PPV 45/140 ≈ 32.1% and 180/260 ≈ 69.2%. All numbers are teaching assumptions.
10 In R, compare a simple logistic baseline with an alternative model on synthetic admission data; pre-processing and tuning use training data only (Part B). 45 Fixed test identifiers, the baseline and a results table with the test denominator.
11 Move a threshold. Discuss ranking, calibration and false alerts per shift. Report errors by group with denominators and uncertainty (Part C). 45 A decision for an example threshold, with the disadvantaged group named and the missing information stated.
12 Journal club: examine a diagnostic accuracy study or an external model validation. Keep evidence on accuracy and evidence on patient benefit apart. 45 A critique naming reference standard and test population, and a proposal for the next type of study.

Two-by-two tables for the room (unit 09)

The instructor first covers the right-hand column. Explain to the person next to you who belongs in the denominator, then reproduce the numbers in R.

Table 1: Two referral settings with the same assumed sensitivity and specificity of 90%. Assumption; no product is described.
Per 1,000 examined Fracture present No fracture Positive predictive value
5% prevalence, alert positive 45 95 45/140 = 32.1%
5% prevalence, alert negative 5 855 —
20% prevalence, alert positive 180 80 180/260 = 69.2%
20% prevalence, alert negative 20 720 —

Journal club card (unit 12)

Work in pairs on one study: the external validation by Wong et al. (2021) or a diagnostic accuracy study of your choice. Use the reporting guideline that fits the study type: STARD-AI for diagnostic accuracy, TRIPOD+AI for prediction models, PROBAST+AI for risk of bias and applicability, DECIDE-AI for early clinical evaluation.

Question Your notes
Recruitment: who was included, where, when, and who was left out?
Reference standard: how was the true condition defined, and was it applied to everyone?
Analysis: unit of analysis, denominators, handling of missing data, calibration reported?
Generalisation: which population, setting and time does the result transfer to?
Decision: what does the study show about accuracy, and what about patient benefit?
One missing reporting item and the question you would ask the authors
The next type of study you would propose

Materials

Four perspectives in this workshop

  • Ethics. A threshold distributes errors. In the synthetic cohort, older patients receive both more missed fractures and more unnecessary follow-up at the same threshold. Name them, with denominators, before choosing.
  • Regulation. A lower PPV alone is no reason to reject a tool, and a higher AUC is no proof of benefit. Regulatory evaluation asks for clinical evidence related to the intended purpose (W7).
  • Data protection. The test identifiers are stored with the data so that results can be reproduced without sharing more data than needed.
  • Security. A model that is silently replaced by another version would invalidate the results table; the model version belongs in the model passport.

Transition to W4

You now have a reproducible comparison table and an error pattern. W4 asks whether such a table measures what it claims to measure when images, animals and measurement devices are involved.

Further reading

Sources