Describe a table without hiding what is missing

A descriptive table over a sdt_participants register, a sdt_course_composition table, or any data frame. Every variable reports its counts, its missingness and the summary its type deserves – counts and percents for a categorical variable, mean, SD, median, IQR and range for a numeric one – optionally split by a grouping column.

Usage

sdt_describe(
  x,
  by = NULL,
  vars = NULL,
  na_report = c("row", "column", "footnote"),
  digits = 2
)

# S3 method for class 'sdt_describe'
format(x, ..., max_groups = 8L, max_levels = 12L)

# S3 method for class 'sdt_describe'
print(x, ..., max_groups = 8L, max_levels = 12L)

Arguments

x: A sdt_participants object, a sdt_course_composition object, or any data frame.
by: Optional single column name to split by. NULL, the default, pools every row. Rows whose grouping value is missing form their own group rather than disappearing.
vars: Optional character vector of columns to describe. Defaults to a curated set for the two classes this package produces, and to every non-indicator column otherwise.
na_report: Where to surface missingness: "row", "column" or "footnote". It cannot be switched off.
digits: Decimal places for the printed statistics. Percentages use one fewer, since a percent already carries two significant figures more than the proportion it comes from. The returned tibble is never rounded.
...: These dots are for future extensions and must be empty.
max_groups: Groups to print before the rest are summarised in one line. The object keeps every group either way.
max_levels: Levels to print per categorical variable.

Value

A sdt_describe object: a tibble::tibble() subclass with one row per group, variable and level. Two families of counts share every row and are named apart so they cannot be confused. n, n_level_observed and n_level_imputed belong to the row – to one level of one categorical variable, or to the values of one numeric variable. n_observed, n_imputed, n_missing, n_valid and n_records belong to the variable, and are repeated on each of its rows, so that n_observed + n_imputed + n_missing == n_records holds on every row of the table. The three percents, pct_missing and the numeric statistics complete it. tibble::as_tibble() returns it bare; sdt_describe_render() returns the display-ready shape.

Every percent states its denominator

A categorical level carries three shares, and all three are in the returned tibble whatever the report style:

pct_observed – over answered rows only. Its denominator excludes both missing rows and filled ones.
pct_valid – over rows that carry a value. Its denominator excludes missing rows and includes filled ones.
pct_total – over every row in the group. Its denominator includes everything, missing rows included.

pct_valid and pct_total are the two a reader usually means by "the percent", and they differ exactly when something is missing. pct_observed differs from pct_valid exactly when something was imputed.

Missingness cannot be switched off

na_report decides where missingness is surfaced, never whether it is:

"row" – each categorical variable gains a (missing) row, and each numeric variable a missing line of its own.
"column" – the missing count and percent appear beside every row.
"footnote" – they are collected under each block, complete variables named as such so silence is never ambiguous.

n_missing and pct_missing are columns of the returned tibble in all three cases.

Imputed values are flagged, not absorbed

Where the input carries indicators – .missing_<field> and .imputed_<field> from sdt_read_participants(), <covariate>_missing and n_<covariate> from sdt_course_composition() – each row is placed in exactly one of three states: observed, imputed or missing. Imputed rows are counted separately, never as observations, and numeric variables carry mean_observed, sd_observed and median_observed alongside the all-values statistics so both are visible at once.

For a variable with a fill rule, n_imputed + n_missing is the number of rows nobody answered: some of them were filled and some were not, and the two are reported apart rather than added up for the reader. That is why title_level reports no missing values and 99 imputed ones on the TDSC register – an empty title box is filled with none, and the fill is visible in every share it touches.

A value that is present in a row whose own missing indicator is set is not an observation. It is counted as imputed when an imputation indicator or a zero denominator (n_<covariate> of 0) says it was filled, and as missing otherwise. That zero-denominator rule is what catches sdt_course_composition(impute = "median"), which fills without flagging: the fill is reported as imputed anyway, and named as unflagged.

Level, not group

The participant register has no group column, so a composition descriptive split by course is a between-course description with as many points as there are courses. There is no split by group to be had.

Examples

d <- data.frame(
  grade = c("consultant", "resident", "consultant", NA),
  years = c(11, 2, 7, NA),
  .missing_grade = c(FALSE, FALSE, FALSE, TRUE),
  .imputed_grade = c(FALSE, FALSE, FALSE, FALSE)
)
sdt_describe(d, vars = c("grade", "years"))
#> <sdt_describe> 2 variables over 4 records
#> Denominators: val% excludes missing, all% includes it; obs% excludes fills too.
#> Missingness: reported as its own row, and never suppressed.
#> 
#>   grade                      n     imp    val%    all%
#>     consultant               2       0    66.7    50.0
#>     resident                 1       0    33.3    25.0
#>     (missing)                1       -       -    25.0
#>   years
#>     n 3   imputed 0
#>     missing 1 of 4 (25.0%)
#>     mean 6.67 (SD 4.51)   median 7.00 [4.50, 9.00]
#>     range 2.00 to 11.00
tibble::as_tibble(sdt_describe(d, vars = "grade"))
#> # A tibble: 3 × 28
#>   group variable type   level missing_row     n n_level_observed n_level_imputed
#>   <chr> <chr>    <chr>  <chr> <lgl>       <int>            <int>           <int>
#> 1 (all) grade    categ… cons… FALSE           2                2               0
#> 2 (all) grade    categ… resi… FALSE           1                1               0
#> 3 (all) grade    categ… (mis… TRUE            1                0               0
#> # ℹ 20 more variables: n_observed <int>, n_imputed <int>, n_missing <int>,
#> #   n_valid <int>, n_records <int>, pct_observed <dbl>, pct_valid <dbl>,
#> #   pct_total <dbl>, pct_missing <dbl>, mean <dbl>, sd <dbl>, median <dbl>,
#> #   q25 <dbl>, q75 <dbl>, iqr <dbl>, min <dbl>, max <dbl>, mean_observed <dbl>,
#> #   sd_observed <dbl>, median_observed <dbl>