Audit corpus coverage against a ground-truth reference
Source:R/coverage-audit.R
sm_coverage_audit.RdComputes recall, precision, and F1 of an sm_corpus against an external
ground-truth reference (a manual tracker, an ORCID works set, an
institutional-repository export, or another sm_corpus). This is the core
primitive for a coverage / completeness audit: "how much of what we should
have did we actually capture, and how much of what we captured is real?"
Matching is by normalised DOI with a fuzzy-title fallback (see
sm_reconcile() for the shared matching engine). Full match provenance is
retained so every decision can be inspected.
Usage
sm_coverage_audit(
corpus,
reference,
by = NULL,
match = c("doi_then_title", "doi", "title"),
threshold = 0.9,
index_table = NULL,
call = rlang::caller_env()
)
# S3 method for class 'sm_coverage'
print(x, ...)
# S3 method for class 'sm_coverage'
summary(object, ...)
# S3 method for class 'sm_coverage'
autoplot(object, dim = NULL, ...)Arguments
- corpus
An
sm_corpusobject (the corpus under audit).- reference
The ground truth: an
sm_corpusor a data frame with at least a DOI and/or title column (column names matched case-insensitively againstdoi/diandtitle/ti/display_name, the year againstyear/py/publication_year). ADateor date-time year column gives its calendar year, and a year outside 1000-2999 (such as a day count) counts as missing. Rows are identified by the first of the columnsut(Web of Science accession numbers),id,work_id,reference_idandref_idwhose values are unique and never missing, and otherwise by their position (R000001, ...), with a warning when such a column had to be passed over. In a data frame with Web of Science field tags (such asDI,TI,PYorUT) a column namedIDholds the Keywords Plus and is never the identifier.- by
Optional character vector of breakdown dimensions. A name is either one of the named dimensions
"year","source"and"affiliation", written in lower case (see Details), or a column: of the reference data frame (matched case-insensitively) or of theworkstable of ansm_corpus(the exact name first, then case-insensitively). Each name is resolved on both sides; a name found on one side only gives levels whose columns for the other side areNA. A name that is neither a named dimension nor a column on either side is an error listing the available columns. The breakdowns label each dimension with the name as given.- match
Matching strategy:
"doi_then_title"(default),"doi", or"title".- threshold
Minimum Jaro-Winkler title similarity (
[0, 1]) to accept a title match. Default0.9. When the optionalstringdistpackage is not installed, the title fallback degrades to normalised exact matching. Titles only match between records published at most one year apart (when both years are known) and never between two records with different DOIs.- index_table
Optional journal index master list (same contract as
sm_journal_in_index(); document expected columns there). When supplied, the audit additionally reports, per corpus record, whether its journal is indexable – so "is this journal in the index" and "is this record captured" are assessed together. WhenNULL(default) the result is identical to before.- call
Caller environment for error reporting.
- x
An
sm_coverageobject.- ...
Ignored.
- object
An
sm_coverageobject.- dim
Which breakdown dimension to plot (defaults to the first available). If no breakdowns exist, a recall/precision summary bar is drawn.
Value
An sm_coverage S3 object (a list) with components:
- recall
Reference records found in the corpus / total reference records. Each reference row is looked up on its own, so rows that report the same item (one paper reported by two units) are all found.
- precision
Matched corpus records / total corpus records. Each corpus record pairs with at most one reference row.
- f1
Harmonic mean of recall and precision.
- n_corpus, n_reference
Integer record counts.
- n_matched
Reference records found in the corpus (the numerator of recall).
- n_corpus_matched
Corpus records paired with a reference record (the numerator of precision); equal to
n_matchedunless reference rows share a corpus record.- n_corpus_only, n_reference_only
Unmatched counts.
- matches
Tibble with one row per corpus record:
corpus_id,reference_id,match_type("doi"/"title"/"none"),match_score. Whenindex_tableis supplied, alsoissn,indexable(logical), andindexed_title.- reference_matches
Tibble with one row per reference record:
reference_id,corpus_id(the corpus record it was found in),match_typeandmatch_score; the recall-side counterpart ofmatches.- corpus_only
Tibble of corpus records absent from the reference.
- reference_only
Tibble of reference records absent from the corpus.
- breakdowns
A single flat tibble (
dimension,level,n_reference,n_matched,recall) across allbydimensions, from the reference side. Usesm_coverage_breakdowns()to access/filter it, also with the corpus-side counts and precision.- breakdowns_nested
The legacy named list of per-dimension tibbles (
slice,n_reference,n_matched,recall). Retained for one release; preferbreakdowns.- indexability
When
index_tableis supplied, a summary tibble of indexable vs non-indexable record counts; otherwiseNULL.
print returns x invisibly.
summary returns a one-row tibble of headline metrics: recall,
precision, f1, n_corpus, n_reference, n_matched,
n_corpus_only, n_reference_only and n_corpus_matched, so that
n_matched + n_reference_only gives n_reference and
n_corpus_matched + n_corpus_only gives n_corpus.
autoplot returns a ggplot object.
Details
Named dimensions. "year" uses the publication years. "source" uses
the source names of an sm_corpus (sources$display_name, or
works$source_id when the corpus has no source table) and, in a data
frame, the first of the columns source, source_title, journal, so
and source_id, in this order (names matched case-insensitively); a
source_id column holds ids, whose levels do not join the names of the
corpus side. "affiliation" uses, in a data frame, the first of the
columns affiliation, affiliations, institution, country and
country_code, in this order. A data-frame cell holds one value, so an
item reported by two units takes two rows; a list column gives each of
its values. Missing and blank values have no level. A named dimension
that neither side can resolve gives a warning and no breakdown rows.
Only "affiliation" joins a tracker's units with the institutions of the
corpus: by = "institution" names the column itself, which the corpus
side has only if works has a column of that name.
Affiliation levels of a corpus. For an sm_corpus (the corpus under
audit, or a reference given as one), "affiliation" labels the
authorships from the first of these authorships columns that has a
value in at least one row: institution_match (from
sm_affiliation_match()), institution_name (from
sm_attribute_institution()), institution_id (named by
corpus$institutions$display_name when that table lists the id, else
the id itself) and, as a last resort, country_code. One column serves
every row, so the levels of a side are all of one kind, and an authorship
without a value in that column has no level. A message names the column
used on each side, and a warning says when a side falls back to country
codes, whose levels do not join institution names. Because
sm_attribute_institution() runs sm_affiliation_match() first,
institution_name is used only when institution_match is absent or
empty; the Details of sm_attribute_institution() show how to break the
coverage down by its vocabulary instead.
sm_count(level = "institution") reads the same columns in the
same order, but row by row, and counts the raw affiliation string where
none has a value instead of falling back to countries; up to scimapR
0.4.0 it used, for every row, the first of institution_id,
institution_match and the raw strings (matched with
sm_affiliation_dict) that had a value in any row.
Full counting. A record belongs to every level of its values, so a
work with authors at two institutions counts in both slices, and the
per-level counts of a dimension can add up to more than n_corpus or
n_reference. The overall recall and precision count each record once.
Examples
corpus <- sm_example_corpus(n_works = 30, seed = 1)
# Pretend the reference is the corpus minus a few works, plus an extra one
ref <- corpus$works[1:25, c("work_id", "doi", "title", "year")]
cov <- sm_coverage_audit(corpus, ref, by = "year")
cov
#>
#> ── <sm_coverage> ───────────────────────────────────────────────────────────────
#> Match strategy: doi_then_title (title threshold 0.9)
#> Recall: 1 (25/25 reference records found)
#> Precision: 0.8333 (25/30 corpus records in reference)
#> F1: 0.9091
#>
#> Corpus-only: 5 Reference-only: 0
#>
#>
#> ── Worst-covered slices by year
#> 2015: recall 1 (1/1)
#> 2016: recall 1 (2/2)
#> 2017: recall 1 (2/2)
#> 2018: recall 1 (2/2)
#> 2020: recall 1 (6/6)
# Any column of the reference or of the works is a dimension too
ref$unit <- rep(c("Unit A", "Unit B"), length.out = nrow(ref))
cov <- sm_coverage_audit(corpus, ref, by = c("unit", "type"), match = "doi")
sm_coverage_breakdowns(cov)
#> # A tibble: 5 × 8
#> dimension level n_reference n_matched recall n_corpus precision f1
#> <chr> <chr> <int> <int> <dbl> <int> <dbl> <dbl>
#> 1 unit Unit A 13 13 1 NA NA NA
#> 2 unit Unit B 12 12 1 NA NA NA
#> 3 type journal-artic… NA NA NA 18 0.833 NA
#> 4 type letter NA NA NA 6 1 NA
#> 5 type review NA NA NA 6 0.667 NA