Score how faithfully a simulation reproduces triage error

Compares simulated triage calls against observed ones per true category, and reports two ratios for every statistic: the level and the between-team standard deviation, each as observed over simulated, each judged inside the same 0.90-1.15 band the rest of the package uses. Getting the average error rate right while making every team err identically is the failure this function exists to catch, so the level alone is never enough.

Usage

sdt_error_fidelity(
  sim,
  observed = NULL,
  truth = NULL,
  estimate = NULL,
  group = NULL,
  card = NULL,
  cards = NULL,
  min_coverage = NULL,
  n_boot = 200L,
  seed = NULL
)

Arguments

sim: Simulated calls: an sdt_simulated_triage object from sdt_simulate_triage(), or a data frame with one row per (team, card) carrying a true and a called category.
observed: The observed calls, in the same shape. Optional when sim carries its fitted model, in which case the model's own fitting frame is used.
truth, estimate, group, card: Column names. NULL takes them from the fitted model where there is one, and otherwise from the usual candidates.
cards: Optional character vector fixing the matched frame's cards. NULL chooses them, as described above.
min_coverage: Minimum number of observed teams a card must be seen by to enter the matched frame. NULL chooses it from the data.
n_boot: Bootstrap replicates for the intervals. 0 skips them.
seed: Optional integer seed, for a reproducible bootstrap.

Value

An S3 object of class sdt_error_fidelity, a tibble with one row per (statistic, measure): statistic, category, quantity, measure ("level" or "sd"), observed, simulated, ratio, ci_lower, ci_upper, verdict, control, n_obs, n_sim, location_z, overlap, the four percentile columns, reason, n_cards, n_obs_rows, n_sim_rows and n_sim_reps. Carries band, location_tol, frame, structural, group_rates and n_boot attributes. The column set is a superset of what sdt_plot_fidelity() needs, so the table can be drawn with it.

Why this is not an argument to `sdt_fidelity()`

sdt_fidelity() scores unconditional per-group rates, and enforces composition matching on the value: a group is dropped for a statistic unless its non-missing cells cover every shared card. A conditional statistic – the hit rate among the red cards – is undefined on every card that is not red, so under that rule every group is dropped on both sides. Handed such a statistic, sdt_fidelity() does not fail loudly; it returns the row with n_obs = 0, n_obs_rows = 0, an NA ratio and the reason "not evaluable", which is the correct answer to the question it was asked and no answer at all to this one. The conditional case also needs a different denominator per category, a per-category composition check, and two ratios per statistic rather than one. What the two share is the verdict: the band and the location tolerance are read from sdt_fidelity()'s own helpers, so they cannot drift apart.

What is scored

For each true category, and for the pooled "overall" row, the hit rate, the over-triage rate and the under-triage rate. Direction is the clinically meaningful part: over-triage spends a scarce resource, under-triage can cost a life, and a single accuracy figure hides which one a team is doing.

Two pairs are zero by construction and are excluded rather than failed: over-triage on the most severe category and under-triage on the least, since there is nothing above red or below green to move to. They are listed in the structural attribute. Note also that for those same two categories the remaining error rate is the exact complement of the hit rate, so red under and green over carry the same information as red hit and green hit; they are reported because the direction of error is what a reader is looking for, not because they are independent evidence.

The matched frame

Real play is truncated by the clock, so observed teams saw different subsets of the deck while every simulated team is dealt all of it. Keeping only the teams that saw the whole intersection would therefore keep the handful that finished, so the card set is chosen the way sdt_fit_behaviour() chooses it: the threshold that maximises cards times complete teams, with ties going to the frame that keeps more teams. Pass cards to fix the set by hand, or min_coverage to fix the threshold.

The two sides must then agree about which true category each kept card carries. A category left with fewer than three cards is warned about: a team's rate there can take only a handful of values, so its between-team SD is as much a property of the deck as of the teams.

Examples

set.seed(1)
cards <- sprintf("c%02d", seq_len(12))
truth <- rep(c("red", "yellow", "green"), each = 4)
d <- expand.grid(card = cards, group_id = sprintf("g%02d", seq_len(20)),
                 stringsAsFactors = FALSE)
d$session_id <- ifelse(d$group_id < "g11", "s1", "s2")
d$triage <- truth[match(d$card, cards)]
b <- stats::rnorm(20, sd = 1)
p <- stats::plogis(1 + b[match(d$group_id, unique(d$group_id))])
hit <- stats::runif(nrow(d)) < p
wrong <- vapply(seq_len(nrow(d)), function(i) {
  sample(setdiff(sdt_triage_levels(), d$triage[[i]]), 1)
}, character(1))
d$called_triage <- ifelse(hit, d$triage, wrong)
if (requireNamespace("lme4", quietly = TRUE)) {
  fit <- sdt_fit_triage_error(d, verbose = FALSE)
  sim <- sdt_simulate_triage(fit, n_groups = 20, seed = 1)
  sdt_error_fidelity(sim, n_boot = 50, seed = 1)
}
#> Warning: ! 1 cell fitted a zero team variance: "green.split".
#> ✖ Simulated teams will be identical there.
#> ℹ That is a finding about the cell, not a failure of the fit.
#> # Triage error fidelity: 11 of 20 ratios outside 0.90-1.15
#> # Matched frame:         12 cards (red 4 / yellow 4 / green 4), 20 observed vs
#> #   20 simulated teams x 1 replicate
#> # Excluded:              red over, green under (zero by construction)
#>    statistic         category quantity measure observed simulated ratio ci_lower
#>  * <chr>             <chr>    <chr>    <chr>      <dbl>     <dbl> <dbl>    <dbl>
#>  1 green hit (sd)    green    hit      sd         0.269    0.228  1.18     0.616
#>  2 green over (sd)   green    over     sd         0.269    0.228  1.18     0.654
#>  3 overall hit (sd)  overall  hit      sd         0.212    0.120  1.77     1.17 
#>  4 overall over (sd) overall  over     sd         0.105    0.0823 1.28     0.944
#>  5 overall under (l… overall  under    level      0.125    0.0875 1.43     0.752
#>  6 overall under (s… overall  under    sd         0.125    0.106  1.18     0.781
#>  7 red under (level) red      under    level      0.25     0.188  1.33     0.636
#>  8 yellow hit (sd)   yellow   hit      sd         0.238    0.172  1.39     0.997
#>  9 yellow over (lev… yellow   over     level      0.075    0.0625 1.2      0.334
#> 10 yellow under (le… yellow   under    level      0.125    0.075  1.67     0.571
#> 11 yellow under (sd) yellow   under    sd         0.190    0.118  1.62     1    
#> 12 green hit (level) green    hit      level      0.75     0.775  0.968    0.758
#> 13 green over (leve… green    over     level      0.25     0.225  1.11     0.593
#> 14 overall hit (lev… overall  hit      level      0.767    0.817  0.939    0.801
#> 15 overall over (le… overall  over     level      0.108    0.0958 1.13     0.608
#> 16 red hit (level)   red      hit      level      0.75     0.812  0.923    0.785
#> 17 red hit (sd)      red      hit      sd         0.281    0.280  1.01     0.738
#> 18 red under (sd)    red      under    sd         0.281    0.280  1.01     0.750
#> 19 yellow hit (leve… yellow   hit      level      0.8      0.862  0.928    0.796
#> 20 yellow over (sd)  yellow   over     sd         0.143    0.138  1.04     0.481
#> # ℹ 16 more variables: ci_upper <dbl>, verdict <chr>, control <lgl>,
#> #   n_obs <int>, n_sim <int>, location_z <dbl>, overlap <dbl>,
#> #   observed_p10 <dbl>, observed_p90 <dbl>, simulated_p10 <dbl>,
#> #   simulated_p90 <dbl>, reason <chr>, n_cards <int>, n_obs_rows <int>,
#> #   n_sim_rows <int>, n_sim_reps <int>