Agreement between two labellings of the same casualties — sdt_consistency

Cohen's kappa between two sets of labels for the same units: a participant's calls against the card's true category, one participant against another, or a model's predictions across a repeated scenario. Kappa rather than raw agreement, because on a cohort that is 44% red, two people who both guess "red" every time agree 44% of the time while sharing no judgement at all.

Usage

sdt_consistency_score(truth, estimate, levels = NULL)

Arguments

truth: Reference labels.
estimate: Labels to compare against truth.
levels: Optional level set. Defaults to the union of both inputs, so a level nobody used does not distort the expected agreement.

Value

A one-row tibble::tibble() with .metric, .estimator, .estimate and n.

Examples

if (requireNamespace("yardstick", quietly = TRUE)) {
  truth <- c("red", "red", "yellow", "green", "green")
  called <- c("red", "yellow", "yellow", "green", "green")
  sdt_consistency_score(truth, called)
}
#> # A tibble: 1 × 4
#>   .metric .estimator .estimate     n
#>   <chr>   <chr>          <dbl> <int>
#> 1 kap     multiclass     0.706     5