Score triage calls against the truth

Hit rate per sighting category and overall, with the confusion matrix, and with the two directions of error kept apart. This is the function that only becomes answerable once a gold standard exists: without one, the same arithmetic measures inter-team agreement.

Usage

sdt_triage_accuracy(x, truth = "triage", estimate = "called_triage", by = NULL)

Arguments

x

A data frame carrying both labels.

truth

Column holding the true category. With a cohort that has been through sdt_apply_gold_standard(), this is "triage".

estimate

Column holding the called category.

by

Optional character vector of columns to split by, for example "group" or c("file", "group"). NULL, the default, pools everything.

Value

An sdt_triage_accuracy object: a tibble::tibble() subclass with the by columns, then category (each triage level plus "overall"), n, n_hit, hit_rate, n_over, over_rate, n_under, under_rate and kappa. The confusion attribute holds the matrix (truth in rows, estimate in columns), or a named list of them when by is given; meta records the columns used and how many rows were unusable.

Direction matters

A triage error is not symmetric. On the severity order red > yellow > green, calling a casualty more urgent than they are is an over-triage: it wastes a scarce resource. Calling them less urgent is an under-triage: it can cost a life. A single accuracy number hides which of the two a team is doing, so both rates are reported alongside the hit rate, per category and overall.

Cohen's kappa is reported on the overall row: it is a property of the whole table, and repeating it on every category row would state it four times without adding anything.

See also

sdt_read_gold_standard(), sdt_apply_gold_standard().

Examples

calls <- data.frame(
  triage = c("red", "red", "yellow", "green", "green"),
  called_triage = c("red", "yellow", "yellow", "yellow", "green")
)
acc <- sdt_triage_accuracy(calls)
acc
#> # Triage accuracy: triage vs called_triage, n = 5
#> # Overall:         hit 0.600 over 0.200 under 0.200 kappa 0.444
#>   category     n n_hit hit_rate n_over over_rate n_under under_rate  kappa
#>   <chr>    <int> <int>    <dbl>  <int>     <dbl>   <int>      <dbl>  <dbl>
#> 1 red          2     1      0.5      0       0         1        0.5 NA    
#> 2 yellow       1     1      1        0       0         0        0   NA    
#> 3 green        2     1      0.5      1       0.5       0        0   NA    
#> 4 overall      5     3      0.6      1       0.2       1        0.2  0.444
attr(acc, "confusion")
#>         estimate
#> truth    red yellow green
#>   red      1      1     0
#>   yellow   0      1     0
#>   green    0      1     1