Fetch pre-computed SPECTER v2 embeddings from the
Semantic Scholar API for works in
the corpus and attach them as the embeddings matrix.
SPECTER embeddings are 768-dimensional vectors useful for computing document similarity, clustering, and visualisation. No Python installation is required – this function retrieves pre-computed vectors from the API.
This enricher is idempotent: works that already have embeddings from the same source are skipped, so a second call fills only the gaps.
Usage
sm_enrich_specter(
corpus,
api_key = Sys.getenv("SEMANTIC_SCHOLAR_API_KEY"),
replace = FALSE,
mix_models = FALSE,
verbose = TRUE,
call = rlang::caller_env()
)Arguments
- corpus
An
sm_corpusobject.- api_key
Semantic Scholar API key. Read from
SEMANTIC_SCHOLAR_API_KEYenv var by default.- replace
Logical; replace existing embeddings of another model (or of an unrecorded one) by SPECTER v2 vectors? Defaults to
FALSE. See Details.- mix_models
Logical; keep existing embeddings of another model (or of an unrecorded one) and add SPECTER v2 vectors for the other works? Defaults to
FALSE. See Details.- verbose
Print progress messages?
- call
Caller environment for error reporting.
Value
An sm_corpus object with an embeddings matrix, its record
in metadata$embedding, plus new provenance rows of the kind
"enrichment", which sm_rebuild_from_cert() does not replay.
Details
The function asks Semantic Scholar for its SPECTER v2 vectors (the API
field embedding.specter_v2; the API's default is SPECTER v1) and
records them in corpus$metadata$embedding (see sm_corpus()) with
the model id "embedding.specter_v2" and, as revision, the model
version that Semantic Scholar reports. Semantic Scholar does not say
which revision of the model or which text it used, so these vectors
count as a model of their own, also next to embeddings from
sm_embed_works(model = "specter2").
A corpus that already has embeddings of another model, or of a model
that is not recorded (such as embeddings made by earlier versions of
scimapR), would get a matrix whose rows come from two models, and
similarities between such rows mean nothing. The function therefore
stops (class scimapR_embedding_model_mismatch) unless you choose:
replace = TRUE discards the existing embeddings, with a message, and
fetches SPECTER v2 vectors for every work with a DOI; mix_models = TRUE keeps them and adds SPECTER v2 vectors for the works without an
embedding, and the record then names the model "mixed". A mix needs
embeddings of the same width.
Requests go out in batches of 100 DOIs. A batch that Semantic Scholar
answers with HTTP 429 (too many requests) is tried once more after a
short wait; a batch that still fails is skipped, and one warning (class
scimapR_specter_failed) gives the number of failed batches.
Examples
# \donttest{
# Queries the Semantic Scholar web API; without an API key the shared
# rate limit can refuse the request.
if (requireNamespace("curl", quietly = TRUE) && curl::has_internet()) {
corpus <- sm_corpus(works = tibble::tibble(
work_id = c("W1", "W2"),
doi = c("10.1016/j.joi.2017.08.007", "10.1371/journal.pbio.1002541")
))
enriched <- try(sm_enrich_specter(corpus))
if (is_sm_corpus(enriched)) enriched$metadata$embedding$model_id
}
#> ℹ Fetching SPECTER embeddings for 2 works...
#> ℹ SPECTER: 2 / 2 works processed.
#> Warning: 1 of 1 Semantic Scholar request failed, so their works got no SPECTER vector.
#> ✖ Semantic Scholar request failed: HTTP 429.
#> ℹ Semantic Scholar limits requests without an API key more strictly; set
#> `SEMANTIC_SCHOLAR_API_KEY`.
#> ℹ No SPECTER embeddings retrieved.
#> NULL
# }
if (FALSE) { # \dontrun{
# Not run: sm_embed_works() needs Python with 'sentence-transformers'.
# a corpus with embeddings of another model
corpus <- sm_embed_works(corpus, model = "mpnet")
corpus <- sm_enrich_specter(corpus, replace = TRUE)
} # }