Week 3, Session 5 — Time-dependent Brier, IPA, external validation

Course 4 — #courses

R. Heller

Note

Testing labs use the main template: Hypothesis → Visualise → Assumptions → Conduct → Conclude.

Learning objectives

  • Compute a time-dependent Brier score and the Index of Prediction Accuracy (IPA).
  • Simulate an external-validation cohort with distribution shift and evaluate a fitted model on it.
  • Describe why a model can have a high internal AUC and a bad external IPA.

Prerequisites

Survival ML from Session 4; calibration from Course 1.

Background

Validation is the point at which a prediction model either earns its keep or does not. Internal validation — CV or the bootstrap on the training data — controls optimism from the fitting step. External validation applies the fixed model to an independent cohort and measures discrimination, calibration, and clinical utility. Time- dependent Brier scores and IPA (1 − Brier/Brier_null) summarise the cost of a probabilistic prediction at a given horizon, combining discrimination and calibration into a single number per time.

Real biomedical models routinely lose performance on external cohorts. Changes in case mix, measurement, baseline risk, or predictor effects can affect performance in different ways. This lab changes the covariate distribution while keeping the conditional event model fixed.

TRIPOD-AI asks you to report all three — discrimination, calibration, clinical utility — on both internal and external data, and to disclose the shift between them. A reader who only sees AUC on internal data has no basis for deploying the model.

Setup

1. Hypothesis

A Cox model fit on a development cohort retains its calibration and discrimination on an external cohort with shifted covariate distribution.

2. Visualise

3. Assumptions

Censoring is independent of both event time and covariates in this simulation. We use marginal inverse-probability-of-censoring weights (IPCW), require positive censoring survival at the horizon, assume Cox proportional hazards, and use no time-varying covariates. Subjects censored before the horizon have unknown outcomes and are not coded as event-free.

4. Conduct

Fit on dev; predict survival at a fixed horizon on both cohorts.

                           auc     brier       ipa
development_apparent 0.7606899 0.1909777 0.1791146
external             0.7399078 0.1375266 0.1099784

A calibration plot at the horizon.

5. Concluding statement

At horizon 3, the independent simulated external cohort had cumulative/dynamic AUC 0.74, IPCW Brier score 0.138, and IPA 0.11. Apparent development values were AUC 0.76, Brier 0.191, and IPA 0.18. Development estimates reuse training data and therefore do not constitute internal validation.

Only the distribution of X changes here; the conditional event-time model is unchanged. Such covariate shift need not cause miscalibration. Brier, AUC, and the cohort-specific null model used for IPA may change with case mix even when predictions remain calibrated. The grouped calibration plot is descriptive and has sampling uncertainty; it does not establish a systematic direction of error in every simulation.

Common pitfalls

  • Reporting AUC as the only discrimination metric on an external cohort.
  • Ignoring censoring when computing a Brier score at a horizon; pec or riskRegression handle this properly.
  • Recalibrating on external data and then reporting the resulting performance as external validation.

Further reading

Session info

R version 4.6.1 (2026-06-24)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 26.04.1 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.32.so;  LAPACK version 3.12.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

time zone: UTC
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
 [1] pROC_1.19.1     survival_3.8-6  lubridate_1.9.5 forcats_1.0.1  
 [5] stringr_1.6.0   dplyr_1.2.1     purrr_1.2.2     readr_2.2.0    
 [9] tidyr_1.3.2     tibble_3.3.1    ggplot2_4.0.3   tidyverse_2.0.0

loaded via a namespace (and not attached):
 [1] Matrix_1.7-5       gtable_0.3.6       jsonlite_2.0.0     compiler_4.6.1    
 [5] renv_1.3.1         Rcpp_1.1.2         tidyselect_1.2.1   splines_4.6.1     
 [9] scales_1.4.0       yaml_2.3.12        fastmap_1.2.0      lattice_0.22-9    
[13] R6_2.6.1           labeling_0.4.3     generics_0.1.4     knitr_1.52        
[17] pillar_1.11.1      RColorBrewer_1.1-3 tzdb_0.5.0         rlang_1.3.0       
[21] stringi_1.8.9      xfun_0.62          S7_0.2.2           otel_0.2.0        
[25] timechange_0.4.0   cli_3.6.6          withr_3.0.3        magrittr_2.0.5    
[29] digest_0.6.39      grid_4.6.1         hms_1.1.4          lifecycle_1.0.5   
[33] vctrs_0.7.3        evaluate_1.0.5     glue_1.8.1         farver_2.1.2      
[37] rmarkdown_2.32     tools_4.6.1        pkgconfig_2.0.3    htmltools_0.5.9