Training word vectors

Examine whether word neighbours indicate meaningful similarity

similarity
word embeddings
inaugural addresses
Learn how models train distributed word representations from local context. Examine why neighbour lists from a small corpus can be misleading.

A curator wants to tag a speech archive without reading every paragraph. She asks which words have similar vectors to freedom, union, and peace in the collection.

A table of neighbours can appear meaningful. However, a word-vector model can return a nearest neighbour for any word it retains.

A distributed word representation is one numeric vector per word, learned from nearby words. This lesson loads one saved local model. It compares a clearly related pair with arbitrary pairs.

TipWhat you will learn

This lesson shows how to:

  • read a saved word2vec skip-gram model with a fixed version
  • examine the preprocessing and training settings stored with the model
  • explain why the page loads a saved fitted model instead of training during rendering
  • compare an expected related pair with random pairs
  • summarize the full distribution of cosine similarities.

Load one saved word2vec model

Word2vec comes from work by Mikolov and colleagues on learning word vectors from nearby words. The skip-gram version uses the current word to predict nearby words within a specified window. The R package splits input on whitespace. Before training, convert text to lowercase. Remove punctuation and numbers.

library(dplyr)
library(tibble)
library(stringr)
library(readr)
library(quanteda)
library(word2vec)
library(ggplot2)

source("R/inaugural-corpus.R")

within_tolerance <- function(observed, expected, tolerance = 1e-4) {
  abs(observed - expected) < tolerance
}

metadata <- read_csv(
  "data/word2vec/word2vec-metadata.csv",
  na = character(),
  col_types = cols(
    field = col_character(),
    value = col_character()
  )
)
metadata_values <- setNames(metadata$value, metadata$field)

paragraphs <- inaugural_paragraphs()
training_tokens <- tokens(
  paragraphs$paragraph,
  remove_punct = TRUE,
  remove_numbers = TRUE
) |>
  tokens_tolower()

model <- word2vec::read.word2vec("data/word2vec/inaugural-word2vec.bin")
embedding_matrix <- as.matrix(model)
punctuation_rows <- rownames(embedding_matrix)[grepl("[[:punct:]]", rownames(embedding_matrix))]

training_summary <- tibble(
  item = c(
    "training documents",
    "tokens after cleanup",
    "vocabulary rows",
    "dimensions",
    "punctuation rows",
    "unique word pairs"
  ),
  value = c(
    metadata_values[["training_documents"]],
    metadata_values[["training_tokens"]],
    metadata_values[["vocabulary_rows"]],
    metadata_values[["dimensions"]],
    metadata_values[["punctuation_rows"]],
    metadata_values[["unique_pairs"]]
  )
)

hyperparameter_table <- tibble(
  setting = c("algorithm", "dimensions", "window", "iterations", "minimum count", "threads"),
  value = metadata_values[c("algorithm", "dimensions", "window", "iterations", "minimum_count", "threads")]
)

model_fingerprint <- tibble(
  field = c("generator", "model SHA-256", "reproducibility note"),
  value = metadata_values[c("generator", "sha256_model", "reproducibility")]
)
Training text and vocabulary checks stored with the saved word2vec model
Item Value
training documents 1377
tokens after cleanup 134419
vocabulary rows 2709
dimensions 50
punctuation rows 1
unique word pairs 3667986
Hyperparameters for the pinned local word2vec skip-gram model
Setting Value
algorithm skip-gram
dimensions 50
window 5
iterations 5
minimum count 5
threads 1
Provenance for the committed word2vec model
Field Value
generator data-raw/build-word2vec-model.R
model SHA-256 b182100ecb33cb339f16fdf5df5cea783dd113843a3a89e7e65393b921e448dd
reproducibility note word2vec is not bitwise reproducible on this package version. Across five fits with the same seed the neighbour order changed every time, so this model is trained once here and committed.

The metadata records 1,377 training documents, 134,419 cleaned tokens, 2,709 vocabulary rows, and 3,667,986 unique word pairs. It also records one punctuation row, the trainer’s </s> marker. Preprocessing removed punctuation before training, so forms such as peace. and peace, do not become separate entries from peace.

Repeated fits used the same training code, seed, and threads = 1L. Each fit produced neighbour lists in a different order. Similarity values changed by about 0.03 here, but the ranking changed. This page loads one saved fit. The metadata records its SHA-256 fingerprint, which identifies the file contents. Keeping a fixed model file lets readers reproduce the exact printed neighbours.

Show the similarity distribution

The distribution below uses every unique word pair in the cleaned vocabulary. It also marks the freedom / liberty pair with a vertical line.

normalized_embeddings <- embedding_matrix / sqrt(rowSums(embedding_matrix^2))
word_similarity_matrix <- tcrossprod(normalized_embeddings)
diag(word_similarity_matrix) <- NA_real_
word_similarities <- word_similarity_matrix[upper.tri(word_similarity_matrix)]
freedom_liberty_percentile <- mean(word_similarities <= freedom_liberty_cosine)

similarity_summary <- tibble(
  statistic = c("word pairs", "median", "95th percentile", "freedom / liberty percentile"),
  value = c(
    length(word_similarities),
    median(word_similarities),
    as.numeric(quantile(word_similarities, 0.95)),
    freedom_liberty_percentile
  )
)
Distribution of all pairwise word-vector cosine similarities
Statistic Value
word pairs 3667986
median 0.794
95th percentile 0.935
freedom / liberty percentile 0.444

freedom / liberty is at the 44th percentile of the full pairwise distribution in this saved model. The clearly related pair has a score below the median for all pairs.

Show the plotting code 
source("R/lesson-figures.R")

plot_similarities <- tibble(cosine_similarity = word_similarities)

ggplot(plot_similarities, aes(x = cosine_similarity)) +
  geom_histogram(
    bins = 40,
    boundary = 0,
    fill = lesson_colours[["context"]],
    colour = lesson_colours[["paper"]]
  ) +
  geom_vline(
    xintercept = freedom_liberty_cosine,
    linewidth = 0.9,
    linetype = "22",
    colour = lesson_colours[["contrast"]]
  ) +
  annotate(
    "text",
    x = freedom_liberty_cosine,
    y = Inf,
    label = sprintf("freedom / liberty\n%.2f", freedom_liberty_cosine),
    hjust = 1.08,
    vjust = 1.4,
    lineheight = 0.95,
    family = lesson_font,
    size = 3.6,
    colour = lesson_text_colours[["contrast"]]
  ) +
  scale_y_continuous(labels = scales::label_comma()) +
  labs(
    title = "Many arbitrary word pairs score above a clearly related pair",
    subtitle = "Cosine similarity for every pair of words kept by the pinned word2vec model",
    x = "Cosine similarity",
    y = "Word pairs"
  ) +
  theme_lesson(grid = "y")
A histogram shows cosine similarities concentrated toward high positive values. A dashed line marks freedom and liberty at 0.77. The line is left of the median, not near the right edge.
Figure 1: Pairwise cosine similarities among words kept by the pinned local word2vec model.

Published embeddings use billions of training tokens. A larger training corpus provides more contexts for distinguishing meaningful neighbours from accidental neighbours.

What to remember

  • A word2vec model gives one vector to each kept word.
  • The pinned model used skip-gram, 50 dimensions, window 5, 5 iterations, minimum count 5, and one thread.
  • The word2vec package expects cleaned text before splitting on whitespace.
  • Reproducing exact neighbour lists requires a fixed model file.
  • In this saved fit, even freedom / liberty scores below the median of all pairwise similarities.

The curator uses the neighbour table to select passages for reading. The table does not establish the archive’s themes.

Sources