Screen every engine on one cohort

Fits each available engine to the same data with the same preprocessing and the same resamples, and returns a leaderboard. This is the whole bench: glass-box and black-box models scored side by side, so the price of interpretability is a number rather than an opinion.

Usage

sdt_workflowset(
  data,
  target = c("triage", "therapy", "called_triage", "called_therapy"),
  engines = NULL,
  predictors = NULL,
  tune = TRUE,
  grid = 5L,
  folds = 5L,
  prop = 0.75,
  parallel = FALSE,
  workers = NULL,
  seed = NULL,
  verbose = TRUE
)

Arguments

data: A cohort or data frame of casualties.
target: Outcome to predict. "triage" and "therapy" are the true values on the card; "called_triage" and "called_therapy" are what a participant assigned, as produced by sdt_add_decisions(). The true values are close to a deterministic function of the card, so every engine scores near 1 on them and the comparison is uninformative. The participant's call is the target worth benchmarking.
engines: Which engines to screen. Defaults to all of them.
predictors: Character vector of predictor columns. Defaults to the card fields, excluding identifiers and every outcome column, so a model for the participant's call cannot simply read the true category.
tune: If TRUE, tune hyperparameters over a grid; if FALSE, fit the specification's defaults once.
grid: Number of candidate hyperparameter settings.
folds: Number of cross-validation folds, stratified on the target.
prop: Proportion of the data used for training in the initial split.
parallel: If TRUE, fit resamples and grid points in parallel. Defaults to FALSE: a package that claims every core without being asked is a bad guest on a shared machine, and a sequential run needs no thought about RNG streams. It pays off on a tuned fit or an engine screen and costs a little on a single untuned one. The backend is set up for the duration of the call and the caller's plan is restored on exit.
workers: Number of parallel workers, or NULL to choose one: a core short of the machine, capped at 8.
seed: Optional integer seed, applied with withr::local_seed().
verbose: If TRUE, report progress via cli.

Value

A tibble::tibble() with one row per engine and metric: engine, class_box, .metric, .estimator, .estimate, ranked with the best roc_auc first. Carries a skipped attribute listing engines that could not be fitted here, and a models attribute holding the fitted sdt_model objects.

Details

Engines whose package is not installed are skipped, and named in the skipped attribute rather than silently dropped. So is the transformer when keras3 is installed but has no working backend. The transformer is fitted after every other engine, so Python and its backend are not loaded into the session before their parallel workers are started.

Examples

co <- sdt_synthetic_cohort(n = 150, seed = 4)
needed <- c("parsnip", "recipes", "workflows", "rsample", "tune",
            "yardstick", "rpart")
if (all(vapply(needed, requireNamespace, logical(1), quietly = TRUE))) {
  sdt_workflowset(co, target = "triage", engines = "tree",
                  tune = FALSE, seed = 1, verbose = FALSE)
}
#> # A tibble: 4 × 5
#>   engine class_box .metric  .estimator .estimate
#>   <chr>  <chr>     <chr>    <chr>          <dbl>
#> 1 tree   glass     accuracy multiclass         1
#> 2 tree   glass     f_meas   macro              1
#> 3 tree   glass     kap      multiclass         1
#> 4 tree   glass     roc_auc  hand_till          1

Usage

Arguments

Value

Details

See also

Examples