Score how faithful a simulation is to the real thing

Compares simulated groups against observed groups on the statistics that matter for a tabletop exercise, and reports one row per statistic. The headline is the dispersion ratio: the between-group standard deviation of a behavioural rate, observed divided by simulated. A simulator that reproduces the average behaviour but makes every team look the same fails here, which is the whole point of the function.

Usage

sdt_fidelity(x, observed = NULL, statistics = NULL, n_boot = 200L, seed = NULL)

Arguments

x: The simulated groups: an sdt_simulated_groups object from sdt_simulate_groups(), a game_run from sdt_simulate() (converted with sdt_run_behaviour()), or a plain data frame with one row per (group, card).
observed: The observed groups, in the same shape. Optional when x carries its fitted model, in which case the model's own training frame is used.
statistics: Which statistics to score. NULL (the default) scores every behavioural column the two frames share. A character vector selects columns by name. A named list of functions defines derived statistics; each is called with a data frame and must return one logical or numeric value per row, and each is applied to both frames, so the columns it uses must exist in both. A shared character or factor column is expanded into one statistic per level, named column=level.
n_boot: Bootstrap replicates for the interval on ratio. 0 skips the bootstrap and returns NA bounds.
seed: Optional integer seed, for a reproducible bootstrap.

Value

A tibble with one row per statistic, ordered with the failures first: statistic, observed, simulated, ratio, ci_lower, ci_upper, verdict, control, n_obs, n_sim, and then observed_mean, simulated_mean, location_z, overlap, the four percentile columns, reason, n_obs_rows, n_sim_rows, n_obs_missing, n_sim_missing and n_sim_reps. n_obs and n_sim count groups – the sample size of each standard deviation, with n_sim counted per simulated replicate – while the _rows columns count the (group, card) episodes behind them and the _missing columns count the episodes that statistic could not be computed on, so a rate built on a partly missing column cannot pass unnoticed.

What is compared

For each statistic, each group gets one number – the share of its patient cards on which the statistic was true. The spread of those group-level rates is the quantity of interest, and the columns report:

observed, simulated: the between-group SD of the group rates.
ratio: observed / simulated. Inf when the simulator produces no between-group spread at all, which is a real and reportable result.
ci_lower, ci_upper: a percentile bootstrap interval on ratio, resampling groups on each side with the simulated cohort held fixed. It answers "how much would this move with a different sample of teams?".
rep_lower, rep_upper, rep_covers_one: the interval across replicates – observed / SD(replicate) for each simulated cohort, at the 2.5th and 97.5th percentiles. It answers the other question, "how much would this move on a different RUN of the simulator?", which is the one that dominates for a rare behaviour and which the bootstrap cannot see. Needs at least ten replicates; NA below that.
observed_mean, simulated_mean, location_z: the level check. location_z is the difference in pooled rates expressed in observed between-group SDs, so a model can also fail by getting the level wrong.
overlap: the overlapping coefficient of the two distributions of group-level rates, from 0 (disjoint) to 1 (identical). Together with the observed_p10/observed_p90/simulated_p10/simulated_p90 columns it shows how far the simulated teams reach, rather than only how wide they are on average.
verdict: "PASS" only when the dispersion ratio lies inside 0.90 and 1.15 and abs(location_z) is at most 0.25. reason names whichever check failed.

The full per-group rates behind the summary are attached as the "group_rates" attribute of the result, and the frame the comparison ran on as "frame".

The composition-matched frame

Between-group dispersion is only meaningful when every group is scored on the same cards. sdt_fidelity() enforces that: it intersects the card sets of the two frames, then keeps only groups that saw every card in the intersection, and records what it dropped in the "frame" attribute. On the real course corpus, skipping this step reports no team signal at all, because each truncated group saw a different deck and that composition variance swamps the behavioural one.

Negative controls

Outcomes fixed by the patient card rather than chosen by the team – died and ventilated by default – are marked control = TRUE. They are the pipeline's own alarm, and they read backwards from the other rows: a control is expected to pass even for a simulator with no team term at all, so a failing control means the frame or the pipeline is wrong, not the model. Check the matched frame before you believe anything else in the table. The one exception is a simulator with no team-level randomness at all: the discrete-event engine reads died and ventilated off the patient card and never redraws them, so its controls are dead flat and fail too. That is a finding about the engine – it has no team layer of any kind – rather than about the frame, and the way to tell the two apart is that a broken frame moves the controls a little while a missing team layer flattens them completely.

What the bootstrap resamples

Whole groups, with replacement, independently on each side, and the ratio is recomputed from the resampled group rates. The group is the unit the dispersion is a statement about and the unit that is exchangeable in both frames; resampling individual (group, card) episodes instead would break the very clustering being measured and would report an interval far too narrow. Where the simulated frame carries several replicates, each is resampled within itself and the resulting SDs averaged, exactly as the point estimate is formed.

Examples

set.seed(1)
cards <- sprintf("c%02d", 1:6)
# Ten observed groups that genuinely differ from one another.
obs <- expand.grid(group_id = sprintf("g%02d", 1:10), card = cards,
                   stringsAsFactors = FALSE)
team <- stats::setNames(stats::runif(10, 0.1, 0.9), sprintf("g%02d", 1:10))
obs$surgery <- stats::runif(nrow(obs)) < team[obs$group_id]

# A simulator that gets the average right and the spread badly wrong.
sim <- expand.grid(sim_group = sprintf("s%02d", 1:10), card = cards,
                   stringsAsFactors = FALSE)
sim$surgery <- stats::runif(nrow(sim)) < mean(team)

sdt_fidelity(sim, obs, n_boot = 50, seed = 1)
#> # A tibble: 1 × 34
#>   statistic observed simulated ratio ci_lower ci_upper verdict control n_obs
#>   <chr>        <dbl>     <dbl> <dbl>    <dbl>    <dbl> <chr>   <lgl>   <int>
#> 1 surgery      0.375     0.238  1.57    0.972     2.40 FAIL    FALSE      10
#> # ℹ 25 more variables: n_sim <int>, observed_mean <dbl>, simulated_mean <dbl>,
#> #   location_z <dbl>, overlap <dbl>, observed_p10 <dbl>, observed_p90 <dbl>,
#> #   simulated_p10 <dbl>, simulated_p90 <dbl>, sd_p <dbl>, rep_lower <dbl>,
#> #   rep_upper <dbl>, rep_covers_one <lgl>, observed_floor <dbl>,
#> #   simulated_floor <dbl>, observed_between <dbl>, simulated_between <dbl>,
#> #   ratio_between <dbl>, ci_overlaps_band <lgl>, reason <chr>,
#> #   n_obs_rows <int>, n_sim_rows <int>, n_obs_missing <int>, …