Skip to contents

Build a typed, tibble-of-tibbles corpus container for bibliometric analysis. The corpus is the central data structure in scimapR, holding works, authors, authorships, institutions, sources, references, concepts, embeddings, provenance, screening decisions, and metadata.

Usage

sm_corpus(
  works,
  authors = NULL,
  authorships = NULL,
  institutions = NULL,
  sources = NULL,
  references = NULL,
  concepts = NULL,
  embeddings = NULL,
  provenance = NULL,
  screening = NULL,
  metadata = list(),
  citations_by_year = NULL
)

# S3 method for class 'sm_corpus'
x[i, ...]

# S3 method for class 'sm_corpus'
length(x)

# S3 method for class 'sm_corpus'
dim(x)

# S3 method for class 'sm_corpus'
as_tibble(x, ...)

# S3 method for class 'sm_corpus'
as.data.frame(x, ...)

# S3 method for class 'sm_corpus'
print(x, ...)

# S3 method for class 'sm_corpus'
format(x, ...)

# S3 method for class 'sm_corpus'
summary(object, ...)

# S3 method for class 'sm_corpus'
str(object, ...)

Arguments

works

A tibble of works (publications). See Details for schema. Common spellings of oa_status are renamed using sm_oa_statuses(), with a message; unrecognised values are kept and reported as unknown.

authors

A tibble of authors. If NULL, empty.

authorships

A tibble linking works to authors. If NULL, empty.

institutions

A tibble of institutions. If NULL, empty.

sources

A tibble of publication sources/journals. If NULL, empty.

references

A tibble of cited references. If NULL, empty.

concepts

A tibble of concepts/keywords. If NULL, empty. Legacy and common spellings in its vocabulary column (such as "keywords-author") are renamed to the names of sm_concept_vocabularies(), with a message. Values that name no known vocabulary are kept as written and listed in the same message, each time a corpus is built from them; name a vocabulary of your own "custom".

embeddings

A numeric matrix of work embeddings, or NULL.

provenance

A tibble tracking data lineage. If NULL, empty.

screening

A tibble of screening decisions. If NULL, empty.

metadata

A list of corpus-level metadata.

citations_by_year

NULL (default: the corpus has no yearly citation counts) or a tibble of yearly citation counts: work_id, year and cited_by_count, one row per work and year. Whole numbers are stored as integers. See Details.

x, object

An sm_corpus object.

i

Row index for subsetting.

...

Ignored.

Value

An sm_corpus S3 object.

  • [: An sm_corpus with the selected works and only the authors, institutions and sources they refer to. Counts and the corpus hash describe the subset, including when all works are selected.

  • length(): Number of works (integer).

  • dim(): Integer vector of length 2 (rows, columns of works table).

  • as_tibble(): The works tibble.

  • as.data.frame(): The works table as a data frame.

Details

IDs are unique within a corpus; sm_bind_corpora() gives the second corpus new IDs where they collide.

A corpus is a list of tables that share identifiers: work_id links every table to works, author_id links authorships to authors, institution_id links them to institutions and source_id links works to sources. The tables below distinguish the standard schema columns (as in the empty tables) from optional columns that particular readers and functions write or read. A schema value may be NA. sm_corpus() adds missing schema columns to works (mostly NA); it takes other supplied tables as given. validate_sm_corpus() checks the required list elements, that works is a tibble with work_id, that metadata is a list, and the yearly citation schema below. It does not require or check every standard column of the other tables. Use sm_validate() for further data checks.

works: one row per work. Standard schema:

ColumnTypeContent
work_idcharacterIdentifier, unique within the corpus
doicharacterDOI, lower case, without https://doi.org/
title, abstractcharacterTitle and abstract
yearintegerPublication year
typecharacterDocument type, as the source names it
source_idcharacterThe journal or other source (sources)
cited_by_countintegerCitations, as the source counts them
oa_statuscharacterOne of sm_oa_statuses(), or NA
languagecharacterLanguage
pmid, arxiv_id, openalex_idcharacterOther identifiers
is_retractedlogicalRetraction flag
retraction_dateDateDate of the retraction
last_refreshedPOSIXctWhen the record was read or fetched

Optional: quartile, jif_quartile or sjr_quartile (journal quartiles such as "Q1", for sm_its(outcome = "share_q1")); numeric impact columns cnci, fwci, fnci, rcr, cnci_value, ncs, mncs and field_citation_ratio (searched in that order by sm_its() and sm_metric_summary()); field (for sm_metric_fnci(classification = "manual")); citation_mature and cnci_provisional (sm_citation_maturity()); cluster_id (the clustering functions); cluster_label and cluster_label_basis, a cluster's label and how it was made (sm_cluster_label()). The OpenAlex readers and sm_enrich_openalex_impact() write these impact columns:

ColumnTypeContent
fwcidoubleField-weighted citation impact from OpenAlex
citation_percentiledoubleOpenAlex citation percentile, 0 to 1
is_top_10pctlogicalAmong the 10% most cited of field and year
is_top_1pctlogicalAmong the 1% most cited of field and year
citations_as_ofDateDate of these values and the yearly counts

The openalexR engine of sm_fetch_openalex() has no percentile, so its citation_percentile, is_top_10pct and is_top_1pct are NA. citations_as_of is the day of the OpenAlex record's updated_date, or of the fetch when the record has none; sm_read_openalex_json() leaves it NA for records without updated_date. sm_citation_window() reads it to tell complete citation windows from partial ones. A corpus restored with sm_corpus_from_tables() can hold it as a day count (days since 1970-01-01, as SQLite keeps dates) or as ISO 8601 text; sm_citation_window() and sm_enrich_openalex_impact() read both as the date they stand for.

authors: one row per author. Standard schema:

ColumnTypeContent
author_idcharacterIdentifier, unique within the corpus
orcidcharacterORCID iD
display_namecharacterName
display_name_alternativeslist of characterOther spellings
inferred_gender, gender_methodcharacterGender and its method
gender_confidencedoubleConfidence of inferred_gender

display_name_alternatives holds, for example, the names as a Web of Science file writes them, and sm_audit_gender(method = "manual") reads the gender columns. Optional: none so far.

authorships: one row per work, author and affiliation, so an author with several addresses in a work has several rows, which share position and is_corresponding. Standard schema:

ColumnTypeContent
work_id, author_idcharacterThe work and the author
positionintegerPlace in the author list, from 1 (NA: unknown)
is_correspondinglogicalCorresponding author: see below
institution_idcharacterThe institution (institutions)
raw_affiliationcharacterThe address as the source writes it
country_codecharacterISO 3166-1 alpha-2 country code

is_corresponding is TRUE for the authors the source names as corresponding, FALSE for the other authors of such a work and NA when the source does not say who the corresponding author is. sm_read_wos() and as_sm_corpus() read it from the RP field, sm_read_openalex_json() and sm_fetch_openalex() from OpenAlex (a work whose authors are all marked false takes its corresponding authors from corresponding_author_ids, and counts as unknown when that lists none of them), sm_read_dimensions() from its Corresponding Authors column and sm_fetch_biorxiv() from the bioRxiv API; the other readers and fetchers write NA. Up to scimapR 0.4.0 they flagged the first author instead. A name that fits several authors of a work equally (namesakes with the same surname and initials) and that the source does not tell apart leaves them NA. Functions that select corresponding authors take TRUE only, and the leadership outcome of sm_its() leaves works whose corresponding author is unknown out of its shares.

Optional: institution_match, match_method, match_signal and match_evidence (sm_affiliation_match()); institution_name (sm_attribute_institution()); email (exported as EM by sm_to_bibliometrix()).

institutions: one row per institution. Standard schema:

ColumnTypeContent
institution_idcharacterIdentifier, as in authorships
rorcharacterROR id
display_namecharacterName
country_codecharacterISO 3166-1 alpha-2 country code
region, income_tiercharacterWorld region and income group
typecharacterKind of institution, such as education or healthcare

Optional: none so far.

sources: one row per journal or other source. Standard schema:

ColumnTypeContent
source_idcharacterIdentifier, as in works
issn_lcharacterLinking ISSN
issnlist of characterAll ISSNs of the source
display_namecharacterName
typecharacterKind of source, such as journal
is_oalogicalOpen-access source
publisher, publisher_countrycharacterPublisher and its country

Optional: abbreviation and iso_abbreviation (character; the Web of Science abbreviations J9 and JI, written by sm_read_wos() and the bibliometrix conversion and exported by sm_to_bibliometrix()).

references: one row per cited reference. Standard schema:

ColumnTypeContent
work_idcharacterThe citing work
ref_indexintegerPlace in the work's reference list
cited_work_idcharacterCited work in works; see linking below
cited_doicharacterDOI of the cited work
cited_rawcharacterThe reference as the source writes it

Reference linking: readers, fetchers, sm_bind_corpora(), sm_dedupe() and sm_link_references() set cited_work_id from cited_doi or an OpenAlex id in cited_raw when the cited work is in works.

Optional: none so far.

concepts: one row per keyword, category or concept of a work. Standard schema:

ColumnTypeContent
work_idcharacterThe work
concept_idcharacterIdentifier in its vocabulary
concept_namecharacterThe keyword, category or concept
levelintegerLevel in the hierarchy of the vocabulary
scoredoubleWeight given by the source
vocabularycharacterOne of sm_concept_vocabularies()

For MeSH headings score is 1 for a major topic and 0 otherwise. Optional: none so far.

embeddings: NULL or a numeric matrix with one row per embedded work, named by its work_id, and one column per dimension. Works may have no row: works without text to embed (sm_embed_works()), works missing from a cache (sm_embed_load()) or works for which a service has no vector (sm_enrich_specter()). The clustering functions give them no cluster (NA), and sm_network_semantic() and sm_plot_landscape() leave them out. metadata$embedding records the model (see below).

provenance: one row per work and source. Standard schema:

ColumnTypeContent
work_idcharacterThe work
sourcecharacterWhere the record comes from, such as "wos"
source_id_externalcharacterIts identifier at the source
fetch_datePOSIXctWhen it was read or fetched
querycharacterThe query or file it came from
enginecharacterThe parser or client that read it
scimapR_versioncharacterThe scimapR version that read it
prompt_hashcharacterHash of a prompt behind the record

source_id_external is, for example, the Web of Science accession number (UT); prompt_hash is NA for records that no prompt produced. Optional:

ColumnTypeContent
kindcharacterThe kind of step that wrote the row (see below)
filtercharacterThe filter of a search, else NA
n_maxintegerThe maximum number of works of a search

kind is "api_query" for the searches of sm_fetch_openalex(), sm_fetch_pubmed(), sm_fetch_crossref() and sm_corpus_for_question(), which also write filter and n_max; "file" for the file readers, whose query holds the file relative to the working directory when it lies below it, else its name; and "enrichment" for the enrichers, such as sm_enrich_openalex_impact(), whose rows have the source "openalex_impact". sm_certificate() records the kind of each step, inferred from the source, query and engine for rows without kind, and sm_rebuild_from_cert() replays only searches, never a file read or an enrichment.

screening: one row per screening decision. Standard schema:

ColumnTypeContent
work_idcharacterThe work
stagecharacterStage, such as "title-abstract" or "full-text"
decisioncharacter"include", "exclude" or "uncertain"
reasoncharacterReason for the decision
confidencedoubleConfidence of the decision
sourcecharacterWho or what decided
decided_atPOSIXctWhen

Optional: model records the chat-provider model for the decision. The legacy decision "unclear" is read as "uncertain".

citations_by_year: optional (NULL when the corpus has no yearly citation counts); one row per work and citation year, as the OpenAlex readers, sm_enrich_openalex_impact() and sm_corpus_from_tables() write it:

ColumnTypeContent
work_idcharacterThe cited work
yearintegerThe year of the citations
cited_by_countintegerCitations received in that year, 0 or more

work_id and year may not be missing, and each pair appears once. OpenAlex gives the counts of the ten most recent years and leaves out the years without citations, so a missing year counts 0 for a work with yearly counts (see works$citations_as_of and sm_citation_window()). Subsetting keeps the rows of the kept works, sm_bind_corpora() binds them and sm_hash_corpus() includes them when there are any.

metrics: an optional legacy table read by the impact resolver when no impact column in works has values. It is not created by sm_corpus() or included in the standard empty schema:

ColumnTypeContent
work_idcharacterThe work
Impact columnnumericOne of the impact names above, or value

The first named impact column, in the same order as for works and then value, is used when it contains a nonmissing value. Prefer impact columns in works for new corpora.

metadata: a list. Standard entries unless marked optional:

EntryTypeContent
scimapR_versioncharacterThe scimapR version that built the corpus
build_datePOSIXctWhen the corpus was built
corpus_hashcharacterHash of the content (sm_hash_corpus())
is_lockedlogicalWhether sm_lock() locked the corpus
last_refreshtimestampLast refresh without a failed request
n_identifiedintegerOptional count before search deduplication
question_idcharacterId of its sm_question()
fetch_incompletelistOptional failed-fetch report (see below)

n_identified includes records removed by question-search deduplication or its result limit.

last_refresh advances only when sm_refresh() refreshed at least one work and no request failed. A recovered partial fetch has optional fetch_incomplete (list) with source, n_fetched, n_expected and reason; see the fetchers' Failed requests sections.

question_id is set by sm_corpus_for_question(). Optional entries locked_at (POSIXct), lock_reason (character) and lock_hash (character) record when, why and which content sm_lock() locked. Subsets remain locked but discard lock_hash. Other optional entries include those added by readers, such as source_file and reader; hash_version (integer), the algorithm of corpus_hash (sm_hash_corpus()), which sm_snapshot() writes (a snapshot without it used version 1); and local_inputs, a tibble of the local files the corpus was built from, with their SHA-256 hashes, which sm_register_input() and sm_corpus_from_tables() (inputs) write, sm_certificate() records and sm_verify_certificate() checks (see sm_register_input() for its columns). local_inputs is NULL until inputs are registered or supplied. build_date and last_refresh may also be numbers of seconds since 1970-01-01 UTC, as corpora restored from tables store them, ISO 8601 text or dates; sm_certificate() and the citation-maturity cut-offs read all of these. print() also reads these forms for last_refresh and shows YYYY-MM-DD HH:MM UTC. NULL and a single missing value print as "never"; unsupported values and vectors use format().

Optional: embedding, the record of the model behind embeddings, a list that sm_embed_works(), sm_enrich_specter(), sm_fetch_semantic_scholar(), sm_embed_load() (from the cache of sm_embed_save()) and sm_bind_corpora() write and sm_certificate() reports:

EntryTypeContent
modelcharacterShort model name, "mixed" for several
model_idcharacterHugging Face id or Semantic Scholar field
base_model_idcharacterBase model of an adapter, else NA
revisioncharacterCommit of the model snapshot, or NA
base_revisioncharacterCommit of the base model, or NA
backendcharacterThe Python package or the service used
textcharacterThe text embedded, such as "title_abstract"
prefixcharacterInput prefix, else ""
versionslistVersions of the Python packages used
modelscharacterThe model ids behind the rows
n_works, n_dimensionsintegerSize of the matrix when made
createdPOSIXctWhen the record was written (UTC)

model_id is the Hugging Face id of the model (for "specter2" that of the adapter, whose base model is base_model_id) or the Semantic Scholar field of the vectors ("embedding.specter_v2" or "embedding.specter_v1"), whose revision is the model version that Semantic Scholar reports. backend is "sentence-transformers", "adapters", "semantic-scholar" or "mixed"; a mix lists its models in models. prefix is "query: " for "multilingual-e5". A record whose n_dimensions differs from the width of embeddings, or whose n_works is smaller than its number of rows, no longer describes the matrix and counts as no record. Embeddings without a record, such as those made by earlier versions of scimapR, have a model that is not recorded: sm_certificate() reports it as "unknown", and the functions that combine embeddings combine them with recorded ones only when asked to (mix_models).

Examples

corpus <- sm_corpus(
  works = tibble::tibble(
    work_id = "W000000001",
    doi = "10.1234/example",
    title = "Example Work",
    abstract = "An example abstract.",
    year = 2024L,
    type = "journal-article",
    source_id = NA_character_,
    cited_by_count = 0L,
    oa_status = "closed",
    language = "en",
    pmid = NA_character_,
    arxiv_id = NA_character_,
    openalex_id = NA_character_,
    is_retracted = FALSE,
    retraction_date = NA_real_,
    last_refreshed = Sys.time()
  )
)
print(corpus)
#> 
#> ── <sm_corpus> ─────────────────────────────────────────────────────────────────
#> Works: 1 | Authors: 0 | Institutions: 0
#> Years: 2024 - 2024
#> Sources (journals): 0
#> Embeddings: none
#> Status: Unlocked (last refreshed: never)