Model passport

A one-page model passport: the empty template and a completed teaching example for the fracture-alert baseline from Lab W3.
Author

R. Heller

A model passport is one page that lets a person outside the group judge what a model was built for, on which data it was tested and where its evidence ends. It travels with the model: every new version gets an updated passport. Every entry is either a fact with its source, a number with its denominator, or an open point together with the information that would close it.

Template

Copy the table into your project and fill in every row. Leave nothing blank: write open and what is missing.

Field Entry
Name and version
Intended purpose (one sentence: who uses it, for whom, to do what)
Decision supported and what happens without the model
Target population and setting
Data source and status (synthetic, published, released for teaching)
Unit of analysis and independent denominator
Time zero and prediction horizon
Outcome and reference standard
Predictors (available at time zero?)
Baseline for comparison
Model and tuning (on which data?)
Validation design (split, test identifiers, external data?)
Performance with uncertainty (discrimination, calibration, errors at the threshold as x/n)
Errors by subgroup (with denominators)
Counter-check (what was changed, what was predicted, what happened)
Not tested (populations, devices, time periods)
Limits of the claim
Reproducibility (script, start command, seed, R version)
Open points, the information that would close each, and the responsible role

Worked teaching example: fracture-alert baseline

All numbers come from the synthetic data of Lab W3 and are computed when this page is rendered.

Field Entry
Name and version Fracture-alert baseline, teaching version 1.0 (logistic regression, four predictors).
Intended purpose For adults with a suspected wrist fracture in the emergency department, prompt a second review of the radiograph before discharge. Teaching purpose only; not authorised for use in care.
Decision supported Second review of the radiograph yes/no. Without the model: usual single reading and routine follow-up.
Target population and setting Adults (18–95 years) presenting with wrist pain after a fall; one fictitious emergency department.
Data source and status Synthetic cohort of 2,000 admissions generated in r/w3-diagnostic-accuracy.R with seed 20260309.
Unit of analysis and independent denominator One admission per person; test set 600 admissions with 84 fractures.
Time zero and prediction horizon Time zero: clinical examination before imaging. Outcome: fracture according to the reference standard.
Outcome and reference standard Fracture yes/no; reference standard assumed complete for everyone (assumption; in practice follow-up imaging).
Predictors Age, fall on outstretched hand, snuffbox tenderness, swelling; all available at time zero.
Baseline Null model with the training prevalence (Brier score 0.121).
Model and tuning Logistic regression, no tuning. Alternative for comparison: elastic net with interactions, penalty tuned by five-fold cross-validation inside the 1400 training admissions.
Validation design One fixed random split (1400 training, 600 test); test identifiers stored with the data; zero shared identifiers. Internal validation only.
Performance with uncertainty AUC 0.858; Brier score 0.091; calibration slope 0.86 (overestimates risk in the highest risk group). At a threshold of 20%: detects 60/84 = 71.4% (60.5% to 80.8%) of fractures; false alerts in 103/516 = 20.0% (16.6% to 23.7%) of people without fracture.
Errors by subgroup Detected fractures, age 18–74: 35/46 = 76.1% (61.2% to 87.4%); age 75 and older: 25/38 = 65.8% (48.6% to 80.4%). False alerts, age 75 and older: 43/131 = 32.8% (24.9% to 41.6%).
Counter-check The elastic net, predicted to rank clearly better, improved the AUC by only 0.024 on the same test admissions.
Not tested Other hospitals, other imaging devices, children, time periods after development, effect on patient outcomes.
Limits of the claim Internal, synthetic validation of discrimination and calibration. No statement about benefit for patients or workload.
Reproducibility Rscript r/w3-diagnostic-accuracy.R in a fresh session; target values checked with stopifnot(); R 4.6.1.
Open points (1) Performance in the target hospital: needs a local, prospective accuracy study (research lead). (2) Lower sensitivity in older patients: needs real local data on this subgroup (clinical lead). (3) Intended purpose for care: needs regulatory classification (manufacturer’s regulatory affairs; see W7).

How the passport is used in the course

  • W4, unit 16: first version for your own project; the other group checks whether every number has its denominator.
  • W6, unit 24: another group reproduces one number of the passport from a fresh R session.
  • W8: the passport is one of the three documents you defend; the rubric criterion “R analysis, validation and uncertainty” is scored mainly from it.

Further reading