auc brier ipa
development_apparent 0.7606899 0.1909777 0.1791146
external 0.7399078 0.1375266 0.1099784
Course 4 — #courses
Note
Testing labs use the main template: Hypothesis → Visualise → Assumptions → Conduct → Conclude.
Survival ML from Session 4; calibration from Course 1.
Validation is the point at which a prediction model either earns its keep or does not. Internal validation — CV or the bootstrap on the training data — controls optimism from the fitting step. External validation applies the fixed model to an independent cohort and measures discrimination, calibration, and clinical utility. Time- dependent Brier scores and IPA (1 − Brier/Brier_null) summarise the cost of a probabilistic prediction at a given horizon, combining discrimination and calibration into a single number per time.
Real biomedical models routinely lose performance on external cohorts. Changes in case mix, measurement, baseline risk, or predictor effects can affect performance in different ways. This lab changes the covariate distribution while keeping the conditional event model fixed.
TRIPOD-AI asks you to report all three — discrimination, calibration, clinical utility — on both internal and external data, and to disclose the shift between them. A reader who only sees AUC on internal data has no basis for deploying the model.
A Cox model fit on a development cohort retains its calibration and discrimination on an external cohort with shifted covariate distribution.
Censoring is independent of both event time and covariates in this simulation. We use marginal inverse-probability-of-censoring weights (IPCW), require positive censoring survival at the horizon, assume Cox proportional hazards, and use no time-varying covariates. Subjects censored before the horizon have unknown outcomes and are not coded as event-free.
Fit on dev; predict survival at a fixed horizon on both cohorts.
auc brier ipa
development_apparent 0.7606899 0.1909777 0.1791146
external 0.7399078 0.1375266 0.1099784
A calibration plot at the horizon.
At horizon 3, the independent simulated external cohort had cumulative/dynamic AUC 0.74, IPCW Brier score 0.138, and IPA 0.11. Apparent development values were AUC 0.76, Brier 0.191, and IPA 0.18. Development estimates reuse training data and therefore do not constitute internal validation.
Only the distribution of X changes here; the conditional event-time model is unchanged. Such covariate shift need not cause miscalibration. Brier, AUC, and the cohort-specific null model used for IPA may change with case mix even when predictions remain calibrated. The grouped calibration plot is descriptive and has sampling uncertainty; it does not establish a systematic direction of error in every simulation.
pec or riskRegression handle this properly.R version 4.6.1 (2026-06-24)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 26.04.1 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.32.so; LAPACK version 3.12.0
locale:
[1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
[4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
[7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
time zone: UTC
tzcode source: system (glibc)
attached base packages:
[1] stats graphics grDevices datasets utils methods base
other attached packages:
[1] pROC_1.19.1 survival_3.8-6 lubridate_1.9.5 forcats_1.0.1
[5] stringr_1.6.0 dplyr_1.2.1 purrr_1.2.2 readr_2.2.0
[9] tidyr_1.3.2 tibble_3.3.1 ggplot2_4.0.3 tidyverse_2.0.0
loaded via a namespace (and not attached):
[1] Matrix_1.7-5 gtable_0.3.6 jsonlite_2.0.0 compiler_4.6.1
[5] renv_1.3.1 Rcpp_1.1.2 tidyselect_1.2.1 splines_4.6.1
[9] scales_1.4.0 yaml_2.3.12 fastmap_1.2.0 lattice_0.22-9
[13] R6_2.6.1 labeling_0.4.3 generics_0.1.4 knitr_1.52
[17] pillar_1.11.1 RColorBrewer_1.1-3 tzdb_0.5.0 rlang_1.3.0
[21] stringi_1.8.9 xfun_0.62 S7_0.2.2 otel_0.2.0
[25] timechange_0.4.0 cli_3.6.6 withr_3.0.3 magrittr_2.0.5
[29] digest_0.6.39 grid_4.6.1 hms_1.1.4 lifecycle_1.0.5
[33] vctrs_0.7.3 evaluate_1.0.5 glue_1.8.1 farver_2.1.2
[37] rmarkdown_2.32 tools_4.6.1 pkgconfig_2.0.3 htmltools_0.5.9