Create an sm_corpus object
Source:R/corpus-class.R, R/corpus-coerce.R, R/corpus-print.R
sm_corpus.RdBuild a typed, tibble-of-tibbles corpus container for bibliometric analysis. The corpus is the central data structure in scimapR, holding works, authors, authorships, institutions, sources, references, concepts, embeddings, provenance, screening decisions, and metadata.
Usage
sm_corpus(
works,
authors = NULL,
authorships = NULL,
institutions = NULL,
sources = NULL,
references = NULL,
concepts = NULL,
embeddings = NULL,
provenance = NULL,
screening = NULL,
metadata = list(),
citations_by_year = NULL
)
# S3 method for class 'sm_corpus'
x[i, ...]
# S3 method for class 'sm_corpus'
length(x)
# S3 method for class 'sm_corpus'
dim(x)
# S3 method for class 'sm_corpus'
as_tibble(x, ...)
# S3 method for class 'sm_corpus'
as.data.frame(x, ...)
# S3 method for class 'sm_corpus'
print(x, ...)
# S3 method for class 'sm_corpus'
format(x, ...)
# S3 method for class 'sm_corpus'
summary(object, ...)
# S3 method for class 'sm_corpus'
str(object, ...)Arguments
- works
A tibble of works (publications). See Details for schema. Common spellings of
oa_statusare renamed usingsm_oa_statuses(), with a message; unrecognised values are kept and reported as unknown.A tibble of authors. If
NULL, empty.A tibble linking works to authors. If
NULL, empty.- institutions
A tibble of institutions. If
NULL, empty.- sources
A tibble of publication sources/journals. If
NULL, empty.- references
A tibble of cited references. If
NULL, empty.- concepts
A tibble of concepts/keywords. If
NULL, empty. Legacy and common spellings in itsvocabularycolumn (such as"keywords-author") are renamed to the names ofsm_concept_vocabularies(), with a message. Values that name no known vocabulary are kept as written and listed in the same message, each time a corpus is built from them; name a vocabulary of your own"custom".- embeddings
A numeric matrix of work embeddings, or
NULL.- provenance
A tibble tracking data lineage. If
NULL, empty.- screening
A tibble of screening decisions. If
NULL, empty.- metadata
A list of corpus-level metadata.
- citations_by_year
NULL(default: the corpus has no yearly citation counts) or a tibble of yearly citation counts:work_id,yearandcited_by_count, one row per work and year. Whole numbers are stored as integers. See Details.- x, object
An
sm_corpusobject.- i
Row index for subsetting.
- ...
Ignored.
Value
An sm_corpus S3 object.
[: Ansm_corpuswith the selected works and only the authors, institutions and sources they refer to. Counts and the corpus hash describe the subset, including when all works are selected.length(): Number of works (integer).dim(): Integer vector of length 2 (rows, columns of works table).
as_tibble(): The works tibble.as.data.frame(): The works table as a data frame.
Details
IDs are unique within a corpus; sm_bind_corpora() gives the second
corpus new IDs where they collide.
A corpus is a list of tables that share identifiers: work_id links
every table to works, author_id links authorships to authors,
institution_id links them to institutions and source_id links
works to sources. The tables below distinguish the standard schema
columns (as in the empty tables) from optional columns that particular
readers and functions write or read. A schema value may be NA.
sm_corpus() adds missing schema columns to works (mostly NA);
it takes other supplied tables as given. validate_sm_corpus() checks
the required list elements, that works is a tibble with work_id,
that metadata is a list, and the yearly citation schema below. It does
not require or check every standard column of the other tables. Use
sm_validate() for further data checks.
works: one row per work. Standard schema:
| Column | Type | Content |
work_id | character | Identifier, unique within the corpus |
doi | character | DOI, lower case, without https://doi.org/ |
title, abstract | character | Title and abstract |
year | integer | Publication year |
type | character | Document type, as the source names it |
source_id | character | The journal or other source (sources) |
cited_by_count | integer | Citations, as the source counts them |
oa_status | character | One of sm_oa_statuses(), or NA |
language | character | Language |
pmid, arxiv_id, openalex_id | character | Other identifiers |
is_retracted | logical | Retraction flag |
retraction_date | Date | Date of the retraction |
last_refreshed | POSIXct | When the record was read or fetched |
Optional: quartile, jif_quartile or sjr_quartile (journal
quartiles such as "Q1", for sm_its(outcome = "share_q1")); numeric
impact columns cnci, fwci, fnci, rcr, cnci_value, ncs, mncs
and field_citation_ratio (searched in that order by sm_its() and
sm_metric_summary()); field (for
sm_metric_fnci(classification = "manual")); citation_mature and
cnci_provisional (sm_citation_maturity()); cluster_id (the
clustering functions); cluster_label and cluster_label_basis, a
cluster's label and how it was made (sm_cluster_label()). The
OpenAlex readers and sm_enrich_openalex_impact() write these impact
columns:
| Column | Type | Content |
fwci | double | Field-weighted citation impact from OpenAlex |
citation_percentile | double | OpenAlex citation percentile, 0 to 1 |
is_top_10pct | logical | Among the 10% most cited of field and year |
is_top_1pct | logical | Among the 1% most cited of field and year |
citations_as_of | Date | Date of these values and the yearly counts |
The openalexR engine of sm_fetch_openalex() has no percentile, so its
citation_percentile, is_top_10pct and is_top_1pct are NA.
citations_as_of is the day of the OpenAlex record's updated_date, or
of the fetch when the record has none; sm_read_openalex_json() leaves
it NA for records without updated_date. sm_citation_window() reads
it to tell complete citation windows from partial ones. A corpus
restored with sm_corpus_from_tables() can hold it as a day count (days
since 1970-01-01, as SQLite keeps dates) or as ISO 8601 text;
sm_citation_window() and sm_enrich_openalex_impact() read both as
the date they stand for.
authors: one row per author. Standard schema:
| Column | Type | Content |
author_id | character | Identifier, unique within the corpus |
orcid | character | ORCID iD |
display_name | character | Name |
display_name_alternatives | list of character | Other spellings |
inferred_gender, gender_method | character | Gender and its method |
gender_confidence | double | Confidence of inferred_gender |
display_name_alternatives holds, for example, the names as a Web of
Science file writes them, and sm_audit_gender(method = "manual")
reads the gender columns. Optional: none so far.
authorships: one row per work, author and affiliation, so an author
with several addresses in a work has several rows, which share
position and is_corresponding. Standard schema:
| Column | Type | Content |
work_id, author_id | character | The work and the author |
position | integer | Place in the author list, from 1 (NA: unknown) |
is_corresponding | logical | Corresponding author: see below |
institution_id | character | The institution (institutions) |
raw_affiliation | character | The address as the source writes it |
country_code | character | ISO 3166-1 alpha-2 country code |
is_corresponding is TRUE for the authors the source names as
corresponding, FALSE for the other authors of such a work and NA
when the source does not say who the corresponding author is.
sm_read_wos() and as_sm_corpus() read it from the RP field,
sm_read_openalex_json() and sm_fetch_openalex() from OpenAlex (a
work whose authors are all marked false takes its corresponding
authors from corresponding_author_ids, and counts as unknown when
that lists none of them), sm_read_dimensions() from its Corresponding
Authors column and sm_fetch_biorxiv() from the bioRxiv API; the other
readers and fetchers write NA. Up to scimapR 0.4.0 they flagged the
first author instead. A name that fits several authors of a work
equally (namesakes with the same surname and initials) and that the
source does not tell apart leaves them NA. Functions that select
corresponding authors take TRUE only, and the leadership outcome of
sm_its() leaves works whose corresponding author is unknown out of
its shares.
Optional: institution_match, match_method, match_signal and
match_evidence (sm_affiliation_match()); institution_name
(sm_attribute_institution()); email (exported as EM by
sm_to_bibliometrix()).
institutions: one row per institution. Standard schema:
| Column | Type | Content |
institution_id | character | Identifier, as in authorships |
ror | character | ROR id |
display_name | character | Name |
country_code | character | ISO 3166-1 alpha-2 country code |
region, income_tier | character | World region and income group |
type | character | Kind of institution, such as education or healthcare |
Optional: none so far.
sources: one row per journal or other source. Standard schema:
| Column | Type | Content |
source_id | character | Identifier, as in works |
issn_l | character | Linking ISSN |
issn | list of character | All ISSNs of the source |
display_name | character | Name |
type | character | Kind of source, such as journal |
is_oa | logical | Open-access source |
publisher, publisher_country | character | Publisher and its country |
Optional: abbreviation and iso_abbreviation (character; the Web of
Science abbreviations J9 and JI, written by sm_read_wos() and the
bibliometrix conversion and exported by sm_to_bibliometrix()).
references: one row per cited reference. Standard schema:
| Column | Type | Content |
work_id | character | The citing work |
ref_index | integer | Place in the work's reference list |
cited_work_id | character | Cited work in works; see linking below |
cited_doi | character | DOI of the cited work |
cited_raw | character | The reference as the source writes it |
Reference linking: readers, fetchers, sm_bind_corpora(), sm_dedupe()
and sm_link_references() set cited_work_id from cited_doi or an
OpenAlex id in cited_raw when the cited work is in works.
Optional: none so far.
concepts: one row per keyword, category or concept of a work.
Standard schema:
| Column | Type | Content |
work_id | character | The work |
concept_id | character | Identifier in its vocabulary |
concept_name | character | The keyword, category or concept |
level | integer | Level in the hierarchy of the vocabulary |
score | double | Weight given by the source |
vocabulary | character | One of sm_concept_vocabularies() |
For MeSH headings score is 1 for a major topic and 0 otherwise.
Optional: none so far.
embeddings: NULL or a numeric matrix with one row per embedded
work, named by its work_id, and one column per dimension. Works may
have no row: works without text to embed (sm_embed_works()), works
missing from a cache (sm_embed_load()) or works for which a service
has no vector (sm_enrich_specter()). The clustering functions give
them no cluster (NA), and sm_network_semantic() and
sm_plot_landscape() leave them out. metadata$embedding records the
model (see below).
provenance: one row per work and source. Standard schema:
| Column | Type | Content |
work_id | character | The work |
source | character | Where the record comes from, such as "wos" |
source_id_external | character | Its identifier at the source |
fetch_date | POSIXct | When it was read or fetched |
query | character | The query or file it came from |
engine | character | The parser or client that read it |
scimapR_version | character | The scimapR version that read it |
prompt_hash | character | Hash of a prompt behind the record |
source_id_external is, for example, the Web of Science accession
number (UT); prompt_hash is NA for records that no prompt
produced. Optional:
| Column | Type | Content |
kind | character | The kind of step that wrote the row (see below) |
filter | character | The filter of a search, else NA |
n_max | integer | The maximum number of works of a search |
kind is "api_query" for the searches of sm_fetch_openalex(),
sm_fetch_pubmed(), sm_fetch_crossref() and
sm_corpus_for_question(), which also write filter and n_max;
"file" for the file readers, whose query holds the file relative to
the working directory when it lies below it, else its name; and
"enrichment" for the enrichers, such as sm_enrich_openalex_impact(),
whose rows have the source "openalex_impact". sm_certificate()
records the kind of each step, inferred from the source, query and
engine for rows without kind, and sm_rebuild_from_cert() replays
only searches, never a file read or an enrichment.
screening: one row per screening decision. Standard schema:
| Column | Type | Content |
work_id | character | The work |
stage | character | Stage, such as "title-abstract" or "full-text" |
decision | character | "include", "exclude" or "uncertain" |
reason | character | Reason for the decision |
confidence | double | Confidence of the decision |
source | character | Who or what decided |
decided_at | POSIXct | When |
Optional: model records the chat-provider model for the decision.
The legacy decision "unclear" is read as "uncertain".
citations_by_year: optional (NULL when the corpus has no yearly
citation counts); one row per work and citation year, as the OpenAlex
readers, sm_enrich_openalex_impact() and sm_corpus_from_tables()
write it:
| Column | Type | Content |
work_id | character | The cited work |
year | integer | The year of the citations |
cited_by_count | integer | Citations received in that year, 0 or more |
work_id and year may not be missing, and each pair appears once.
OpenAlex gives the counts of the ten most recent years and leaves out
the years without citations, so a missing year counts 0 for a work with
yearly counts (see works$citations_as_of and sm_citation_window()).
Subsetting keeps the rows of the kept works, sm_bind_corpora() binds
them and sm_hash_corpus() includes them when there are any.
metrics: an optional legacy table read by the impact resolver
when no impact column in works has values. It is not created by
sm_corpus() or included in the standard empty schema:
| Column | Type | Content |
work_id | character | The work |
| Impact column | numeric | One of the impact names above, or value |
The first named impact column, in the same order as for works and
then value, is used when it contains a nonmissing value. Prefer
impact columns in works for new corpora.
metadata: a list. Standard entries unless marked optional:
| Entry | Type | Content |
scimapR_version | character | The scimapR version that built the corpus |
build_date | POSIXct | When the corpus was built |
corpus_hash | character | Hash of the content (sm_hash_corpus()) |
is_locked | logical | Whether sm_lock() locked the corpus |
last_refresh | timestamp | Last refresh without a failed request |
n_identified | integer | Optional count before search deduplication |
question_id | character | Id of its sm_question() |
fetch_incomplete | list | Optional failed-fetch report (see below) |
n_identified includes records removed by question-search deduplication
or its result limit.
last_refresh advances only when sm_refresh() refreshed at least one
work and no request failed. A recovered partial fetch has optional
fetch_incomplete (list) with source, n_fetched, n_expected and
reason; see the fetchers' Failed requests sections.
question_id is set by sm_corpus_for_question(). Optional entries
locked_at (POSIXct), lock_reason (character) and lock_hash
(character) record when, why and which content sm_lock() locked.
Subsets remain locked but discard lock_hash.
Other optional entries include those added by readers, such as
source_file and reader;
hash_version (integer), the algorithm of corpus_hash
(sm_hash_corpus()), which sm_snapshot() writes (a snapshot without
it used version 1); and local_inputs, a tibble of the local files the
corpus was built from, with their SHA-256 hashes, which
sm_register_input() and sm_corpus_from_tables() (inputs) write,
sm_certificate() records and sm_verify_certificate() checks (see
sm_register_input() for its columns). local_inputs is NULL until
inputs are registered or supplied.
build_date and last_refresh may also be numbers of seconds since
1970-01-01 UTC, as corpora restored from tables store them, ISO 8601
text or dates; sm_certificate() and the citation-maturity cut-offs
read all of these. print() also reads these forms for last_refresh
and shows YYYY-MM-DD HH:MM UTC. NULL and a single missing value
print as "never"; unsupported values and vectors use format().
Optional: embedding, the record of the model behind embeddings, a
list that sm_embed_works(), sm_enrich_specter(),
sm_fetch_semantic_scholar(), sm_embed_load() (from the cache of
sm_embed_save()) and sm_bind_corpora() write and
sm_certificate() reports:
| Entry | Type | Content |
model | character | Short model name, "mixed" for several |
model_id | character | Hugging Face id or Semantic Scholar field |
base_model_id | character | Base model of an adapter, else NA |
revision | character | Commit of the model snapshot, or NA |
base_revision | character | Commit of the base model, or NA |
backend | character | The Python package or the service used |
text | character | The text embedded, such as "title_abstract" |
prefix | character | Input prefix, else "" |
versions | list | Versions of the Python packages used |
models | character | The model ids behind the rows |
n_works, n_dimensions | integer | Size of the matrix when made |
created | POSIXct | When the record was written (UTC) |
model_id is the Hugging Face id of the model (for "specter2" that
of the adapter, whose base model is base_model_id) or the Semantic
Scholar field of the vectors ("embedding.specter_v2" or
"embedding.specter_v1"), whose revision is the model version that
Semantic Scholar reports. backend is "sentence-transformers",
"adapters", "semantic-scholar" or "mixed"; a mix lists its models
in models. prefix is "query: " for "multilingual-e5".
A record whose n_dimensions differs from the width of embeddings,
or whose n_works is smaller than its number of rows, no longer
describes the matrix and counts as no record. Embeddings without a
record, such as those made by earlier versions of scimapR, have a model
that is not recorded: sm_certificate() reports it as "unknown", and
the functions that combine embeddings combine them with recorded ones
only when asked to (mix_models).
Examples
corpus <- sm_corpus(
works = tibble::tibble(
work_id = "W000000001",
doi = "10.1234/example",
title = "Example Work",
abstract = "An example abstract.",
year = 2024L,
type = "journal-article",
source_id = NA_character_,
cited_by_count = 0L,
oa_status = "closed",
language = "en",
pmid = NA_character_,
arxiv_id = NA_character_,
openalex_id = NA_character_,
is_retracted = FALSE,
retraction_date = NA_real_,
last_refreshed = Sys.time()
)
)
print(corpus)
#>
#> ── <sm_corpus> ─────────────────────────────────────────────────────────────────
#> Works: 1 | Authors: 0 | Institutions: 0
#> Years: 2024 - 2024
#> Sources (journals): 0
#> Embeddings: none
#> Status: Unlocked (last refreshed: never)