[1] 1.206876
Course 4 — #courses
Note
Workflow labs use the variant template: Goal → Approach → Execution → Check → Report.
Course 1 material on sampling variability; comfort with caret or base-R model fitting.
Every predictive model reports an error on the data it was fit to. That number is almost always optimistic, because the fit has had a chance to memorise noise as well as signal. Cross-validation and the bootstrap are two resampling strategies that simulate the experience of scoring the model on data it has not seen. They are not interchangeable: they make different trade-offs between bias and variance, and they behave differently under high-dimensional settings.
A particular failure mode that recurs in the applied literature is double dipping: using the same held-out fold both to choose a hyperparameter and to report the final error. The fold is then no longer held out, and the reported error slides back toward the training error. Nested cross-validation is the clean solution: the inner loop picks hyperparameters, the outer loop reports error.
The bootstrap gives a different window onto generalisation. The plain bootstrap resamples with replacement; the .632 and .632+ variants combine apparent and out-of-bag errors. The .632+ correction adapts to estimated overfitting. Its bias and variance depend on the learner, loss, and data-generating process; no estimator is uniformly best.
A practical note: when you write up a modelling study, the generalisation error is the headline number. If you tuned, you must disclose how. If the inner tuning cost was hidden in the outer fold, a careful reader will notice that the reported error is too good and cannot be reproduced. Nested CV is not a cosmetic fix; it is the difference between a number that generalises and a number that does not.
Estimate the generalisation error of a lasso on a moderate p, small n simulated regression problem, first naively, then with a proper nested cross-validation.
We simulate n = 120 observations with p = 40 features, of which only 5 are true signal. We will (i) fit a single cv.glmnet, (ii) report the CV-minimising error, then (iii) wrap that in an outer 5-fold loop to show that the inner CV error is optimistic.
Naive single-layer CV.
[1] 1.206876
Nested CV. The inner cv.glmnet picks lambda; the outer 5 folds produce the reported MSE.
naive nested
1.206876 1.048175
Bootstrap .632+ error as a cross-check.
apparent oob boot632 boot632plus
0.73569350 1.22831749 1.04703186 1.05010707
relative_overfit nested
0.02657819 1.04817523
On simulated data (n = 120, p = 40, five true signals), a lasso with tuning parameter chosen by 5-fold CV achieved an estimated MSE of 1.05 under nested 5 × 5 CV. The minimum single-layer CV estimate was 1.21; bootstrap .632 and .632+ estimates were 1.05 and 1.05, respectively.
The minimum inner-CV error is subject to selection optimism; its ordering relative to the outer estimate can vary in a single simulation. Outer CV keeps tuning separate from evaluation, but remains subject to sampling and fold variability. Here B = 50 keeps the demonstration small; assess Monte Carlo stability with more replicates before using bootstrap results in a study.
The .632+ calculation uses squared-error loss and the bounded correction in Efron and Tibshirani (1997), equations 26 and 31–32. Original observations are weighted equally in the out-of-bag estimate, and bootstrap duplicates are grouped in the inner tuning folds.
R version 4.6.1 (2026-06-24)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 26.04.1 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.32.so; LAPACK version 3.12.0
locale:
[1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
[4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
[7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
time zone: UTC
tzcode source: system (glibc)
attached base packages:
[1] stats graphics grDevices datasets utils methods base
other attached packages:
[1] glmnet_5.2 Matrix_1.7-5 lubridate_1.9.5 forcats_1.0.1
[5] stringr_1.6.0 dplyr_1.2.1 purrr_1.2.2 readr_2.2.0
[9] tidyr_1.3.2 tibble_3.3.1 ggplot2_4.0.3 tidyverse_2.0.0
loaded via a namespace (and not attached):
[1] generics_0.1.4 renv_1.3.1 shape_1.4.6.1 stringi_1.8.9
[5] lattice_0.22-9 hms_1.1.4 digest_0.6.39 magrittr_2.0.5
[9] evaluate_1.0.5 grid_4.6.1 timechange_0.4.0 RColorBrewer_1.1-3
[13] iterators_1.0.14 fastmap_1.2.0 foreach_1.5.2 jsonlite_2.0.0
[17] survival_3.8-6 scales_1.4.0 codetools_0.2-20 cli_3.6.6
[21] rlang_1.3.0 splines_4.6.1 withr_3.0.3 yaml_2.3.12
[25] otel_0.2.0 tools_4.6.1 tzdb_0.5.0 vctrs_0.7.3
[29] R6_2.6.1 lifecycle_1.0.5 pkgconfig_2.0.3 pillar_1.11.1
[33] gtable_0.3.6 glue_1.8.1 Rcpp_1.1.2 xfun_0.62
[37] tidyselect_1.2.1 knitr_1.52 farver_2.1.2 htmltools_0.5.9
[41] rmarkdown_2.32 labeling_0.4.3 compiler_4.6.1 S7_0.2.2