Week 3, Session 3 — Biomarker statistics (Youden, NRI, decision curves)

Course 4 — #courses

R. Heller

Note

Testing labs use the main template: Hypothesis → Visualise → Assumptions → Conduct → Conclude.

Learning objectives

  • Compute ROC-AUC, Youden’s index, sensitivity, and specificity at the optimal cut-point.
  • Compute a net reclassification index between two predictive models.
  • Construct a decision curve and interpret net benefit at a range of threshold probabilities.

Prerequisites

Binary classification and ROC curves from Course 1; logistic regression.

Background

A candidate biomarker is not useful until it is tied to a decision. ROC-AUC summarises discrimination across all thresholds but is insensitive to where on the curve the action happens. Youden’s index (sensitivity + specificity − 1) picks the threshold that maximises the equal-weighted sum. The net reclassification index (NRI) quantifies whether a new model reclassifies cases and non-cases in the correct direction relative to a baseline. Decision curve analysis plots net benefit as a function of the threshold probability, and lets a reader compare strategies (“treat all”, “treat none”, “treat by model”) across a clinically relevant range.

Discrimination, calibration, and net benefit are three complementary axes. A biomarker with high AUC that is poorly calibrated can produce harmful decisions; a perfectly calibrated biomarker with low AUC gives no useful ranking. Reporting all three keeps the evaluation honest.

Decision curves are not hypothesis tests. They are a principled way to put a clinical question — what is the harm-to-benefit ratio of acting on this prediction? — into the analysis, and to see how the answer depends on that ratio.

Setup

1. Hypothesis

Can a logistic model on Pima.tr (glucose, BMI, age) distinguish diabetic from non-diabetic patients well enough to support a screening decision?

2. Visualise

3. Assumptions

Independence of observations; probability of diabetes is monotone in the linear predictor; no missingness.

4. Conduct

Fit a simple logistic regression and compute discrimination.

Area under the curve: 0.8355
  threshold sensitivity specificity   youden
1  0.267804   0.8676471   0.6969697 1.564617

NRI against a glucose-only baseline.

   nri_cases nri_noncases    nri_total 
   0.2941176    0.2878788    0.5819964 

A manual decision curve.

5. Concluding statement

A logistic model using glucose, BMI, and age discriminated diabetic from non-diabetic patients in MASS::Pima.tr with AUC 0.835. The Youden-optimal cut-point occurred at a predicted probability of 0.27. Adding BMI and age to a glucose-only baseline produced an NRI of 0.58; the decision curve showed net benefit above “treat all” for threshold probabilities from roughly 0.15 to 0.5.

Decision curves give the clinical context: if the decision to intervene at, say, p = 0.2 is under discussion, the model is useful; at p = 0.05 or p = 0.7, it is barely distinguishable from treat-all or treat-none.

Common pitfalls

  • Reporting AUC without calibration or decision curves.
  • Computing NRI with a categorical risk cut-point and failing to disclose the cut-off.
  • Using the same data to develop and evaluate the biomarker (apparent performance).

Further reading

  • Pencina MJ, D’Agostino RB Sr, et al. (2008), Evaluating the added predictive ability of a new marker.
  • Vickers AJ, Elkin EB (2006), Decision curve analysis.

Session info

R version 4.6.1 (2026-06-24)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 26.04.1 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.32.so;  LAPACK version 3.12.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

time zone: UTC
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
 [1] MASS_7.3-65     pROC_1.19.1     lubridate_1.9.5 forcats_1.0.1  
 [5] stringr_1.6.0   dplyr_1.2.1     purrr_1.2.2     readr_2.2.0    
 [9] tidyr_1.3.2     tibble_3.3.1    ggplot2_4.0.3   tidyverse_2.0.0

loaded via a namespace (and not attached):
 [1] gtable_0.3.6       jsonlite_2.0.0     compiler_4.6.1     renv_1.3.1        
 [5] Rcpp_1.1.2         tidyselect_1.2.1   scales_1.4.0       yaml_2.3.12       
 [9] fastmap_1.2.0      R6_2.6.1           labeling_0.4.3     generics_0.1.4    
[13] knitr_1.52         pillar_1.11.1      RColorBrewer_1.1-3 tzdb_0.5.0        
[17] rlang_1.3.0        stringi_1.8.9      xfun_0.62          S7_0.2.2          
[21] otel_0.2.0         timechange_0.4.0   cli_3.6.6          withr_3.0.3       
[25] magrittr_2.0.5     digest_0.6.39      grid_4.6.1         hms_1.1.4         
[29] lifecycle_1.0.5    vctrs_0.7.3        evaluate_1.0.5     glue_1.8.1        
[33] farver_2.1.2       rmarkdown_2.32     tools_4.6.1        pkgconfig_2.0.3   
[37] htmltools_0.5.9