library(reticulate)
library(spacyr)
library(dplyr)
library(tibble)
source("R/use-spacy.R")
within_tolerance <- function(observed, expected, tolerance = 1e-3) {
abs(observed - expected) < tolerance
}
pipeline <- use_project_spacy()
spacy_py <- import("spacy", delay_load = FALSE)
nlp <- spacy_py$load("en_core_web_sm")
pipeline_table <- tibble(
item = c("pipeline", "static vector rows", "static vector columns"),
value = c(
pipeline$model,
as.character(nlp$vocab$vectors$shape[[1]]),
as.character(nlp$vocab$vectors$shape[[2]])
)
)Reading words in context
Compare one word vector with one token occurrence vector
similarity
contextual embeddings
spaCy
Learn how spaCy returns a context-sensitive token representation and why two occurrences of the same word can have different vectors.
A student underlines two occurrences of bank in a short text. The first names a financial institution. The second names land beside a river.
The spelling is identical. The surrounding words make the two occurrences different.
A contextualized word representation assigns a vector to each token occurrence, an individual instance of a word or punctuation mark. A word type groups occurrences with the same spelling. A contextual model can assign different vectors to occurrences of the same word within a document.
TipWhat you will learn
This lesson shows how to:
- start the fixed local spaCy pipeline through the project helper
- confirm that the small pipeline has no static word-vector table
- read the token tensor produced for one text
- compare token occurrence vectors with an all-pair baseline
- examine same-sense controls
- separate contextual encoding from word-sense identification.
Load the local spaCy pipeline
spaCy is a Python library. This project uses it through R/use-spacy.R. The helper selects a fixed Python environment and prevents downloads during rendering.
| Item | Value |
|---|---|
| pipeline | en_core_web_sm |
| static vector rows | 0 |
| static vector columns | 0 |
The zero-row vector table matters. This small pipeline does not store one reusable vector for every word. It still produces an internal tensor, or array of numbers, for the tokens it processes.
Extract token occurrence vectors
A tensor is an array of numbers. For this text, spaCy returns one row per token and 96 columns per token.
text <- "The bank raised interest rates. He sat on the river bank and watched."
doc <- nlp(text)
tokens <- vapply(
seq_len(length(doc)) - 1L,
function(i) doc[i]$text,
character(1)
)
tensor <- py_to_r(doc$tensor)
bank_positions <- which(tokens == "bank")
the_positions <- which(tokens %in% c("The", "the"))
token_table <- tibble(
position = seq_along(tokens),
token = tokens,
target = case_when(
position %in% bank_positions ~ "bank occurrence",
position %in% the_positions ~ "the occurrence",
TRUE ~ "other token"
)
)| Position | Token | Role in this lesson |
|---|---|---|
| 1 | The | the occurrence |
| 2 | bank | bank occurrence |
| 3 | raised | other token |
| 4 | interest | other token |
| 5 | rates | other token |
| 6 | . | other token |
| 7 | He | other token |
| 8 | sat | other token |
| 9 | on | other token |
| 10 | the | the occurrence |
| 11 | river | other token |
| 12 | bank | bank occurrence |
| 13 | and | other token |
| 14 | watched | other token |
| 15 | . | other token |
The two bank tokens are at positions 2 and 12. They share spelling, but they do not share the same tensor row.
Compare occurrence vectors with a baseline
Cosine similarity compares vector direction. A score of 1 would mean the two rows point in exactly the same direction. The two bank occurrence vectors score about 0.3466.
cosine <- function(first, second) {
sum(first * second) / sqrt(sum(first^2) * sum(second^2))
}
all_pair_index <- which(
upper.tri(matrix(NA_real_, nrow = length(tokens), ncol = length(tokens))),
arr.ind = TRUE
)
all_pair_cosines <- apply(
all_pair_index,
1,
function(pair) cosine(tensor[pair[1], ], tensor[pair[2], ])
)
bank_cosine <- cosine(tensor[bank_positions[1], ], tensor[bank_positions[2], ])
the_cosine <- cosine(tensor[the_positions[1], ], tensor[the_positions[2], ])
bank_percentile <- mean(all_pair_cosines <= bank_cosine)
the_percentile <- mean(all_pair_cosines <= the_cosine)
all_pair_summary <- tibble(
statistic = c("mean", "median", "5th percentile", "95th percentile"),
cosine = c(
mean(all_pair_cosines),
median(all_pair_cosines),
as.numeric(quantile(all_pair_cosines, 0.05)),
as.numeric(quantile(all_pair_cosines, 0.95))
)
)
comparison_table <- tibble(
comparison = c("bank / bank", "the / the"),
first_position = c(bank_positions[1], the_positions[1]),
second_position = c(bank_positions[2], the_positions[2]),
cosine_similarity = c(bank_cosine, the_cosine),
percentile_of_all_pairs = c(bank_percentile, the_percentile)
)| All-pair baseline | Cosine |
|---|---|
| mean | 0.0476 |
| median | -0.0074 |
| 5th percentile | -0.1594 |
| 95th percentile | 0.4475 |
| Comparison | First position | Second position | Cosine similarity | Percentile of all token pairs |
|---|---|---|---|---|
| bank / bank | 2 | 12 | 0.3466 | 0.914 |
| the / the | 1 | 10 | 0.5322 | 0.971 |
The bank pair scores around the 91st percentile of the all-pair baseline for the same text. The encoder assigns different vectors to individual occurrences. The two bank rows remain closer than most token pairs in this short text.
Examine same-sense controls
The stronger claim would be that the distance isolates word sense. The controls below examine that claim through separate changes to sense and grammatical role.
tensor_for_text <- function(text) {
doc <- nlp(text)
tokens <- vapply(
seq_len(length(doc)) - 1L,
function(i) doc[i]$text,
character(1)
)
list(tokens = tokens, tensor = py_to_r(doc$tensor))
}
bank_pair_cosine <- function(text) {
parsed <- tensor_for_text(text)
positions <- which(parsed$tokens == "bank")
if (length(positions) != 2L) {
stop("Expected exactly two bank tokens.", call. = FALSE)
}
cosine(parsed$tensor[positions[1], ], parsed$tensor[positions[2], ])
}
bank_control_sentences <- tibble(
comparison = c(
"different senses, different roles",
"same financial sense, same role",
"same river sense",
"same financial sense, different roles"
),
text = c(
"The bank raised interest rates. He sat on the river bank and watched.",
"The bank raised interest rates. The bank approved the loan.",
"He sat on the river bank and watched. She walked along the muddy bank.",
"The bank approved the loan. She walked into the bank."
),
cosine_similarity = c(
bank_pair_cosine(text[1]),
bank_pair_cosine(text[2]),
bank_pair_cosine(text[3]),
bank_pair_cosine(text[4])
)
)
role_gap <- abs(
bank_control_sentences$cosine_similarity[bank_control_sentences$comparison == "different senses, different roles"] -
bank_control_sentences$cosine_similarity[bank_control_sentences$comparison == "same financial sense, different roles"]
)| Comparison | Sentences | Cosine similarity |
|---|---|---|
| different senses, different roles | The bank raised interest rates. He sat on the river bank and watched. | 0.3466 |
| same financial sense, same role | The bank raised interest rates. The bank approved the loan. | 0.9422 |
| same river sense | He sat on the river bank and watched. She walked along the muddy bank. | 0.5900 |
| same financial sense, different roles | The bank approved the loan. She walked into the bank. | 0.3447 |
The same financial sense in different grammatical roles scores about 0.3447. Different senses in different roles score almost the same, about 0.3466. The same financial sense in the same role scores about 0.9422. In these four examples, the grammatical-role contrast is at least as large as the dictionary-sense contrast. These examples illustrate the representation, not a word-sense evaluation. There is no benchmark or independent sense labeling here.
Contrast with one vector per word
The word2vec lesson uses a saved model with one trained vector for each retained word type. A one-vector-per-word model cannot give river bank and finance bank two rows in the same sentence. A contextual pipeline can.
Different vectors do not show that the model finds the intended dictionary sense. The en_core_web_sm tok2vec layer, short for token-to-vector, converts each token into numbers. The tagger, parser, and named-entity recognizer use this shared encoder. Training targets those tasks, not word-sense identification.
A representation can change with context without finding the intended word sense. This encoder belongs to a small pipeline, not a large language model.
final_checks <- tibble(
check = c("token rows", "tensor dimensions", "bank cosine rounded", "same-role financial cosine rounded", "static vector rows"),
value = c(
as.character(nrow(tensor)),
paste(ncol(tensor), "columns"),
sprintf("%.4f", bank_cosine),
sprintf("%.4f", bank_control_sentences$cosine_similarity[2]),
as.character(nlp$vocab$vectors$shape[[1]])
)
)| Check | Value |
|---|---|
| token rows | 15 |
| tensor dimensions | 96 columns |
| bank cosine rounded | 0.3466 |
| same-role financial cosine rounded | 0.9422 |
| static vector rows | 0 |
What to remember
- A contextualized representation gives one vector to each token occurrence.
- The pinned small spaCy pipeline has 0 rows in its static vector table.
- The example text produces a 15 by 96 tensor.
- The two
bankoccurrence vectors score about 0.3466, above most token pairs in that text. - In these four examples, grammatical role matters at least as much as the dictionary-sense contrast. These examples are not a word-sense benchmark.