Counting nearby words with n-grams

See why phrase counts get sparse fast

phrases and entities
n-grams
workforce research
Learn how bigrams and trigrams count adjacent tokens in the Riverton sentences, why most of them appear only once, and why a PMI score without a count floor ranks unique pairs first.

Repeated words such as training stand out in the Riverton Workforce Lab’s short job-board lines and flyer headings. The harder question is whether neighbouring word pairs tell a clearer story.

The coordinator tries a simple phrase count before trusting it. If most phrases appear once, the list may describe these 28 sentences closely while helping less with new documents.

Note

The Riverton Workforce Lab, its job board, and its training flyer are fictional and were created for teaching.

TipWhat you will learn

This lesson shows how to:

  • define an n-gram as a run of adjacent tokens;
  • build bigrams and trigrams with tidytext;
  • count repeated n-grams in the Riverton sentences;
  • measure how many n-grams appear once;
  • score repeated bigrams with pointwise mutual information; and
  • explain why phrase vocabularies can generalise badly from small data.

Load the sentences

This setup chunk reads the sentence file, prepares small tables, builds n-grams with tidytext, and draws one summary chart. A tibble is a table that prints its size and column types.

library(readr)
library(dplyr)
library(tibble)
library(tidyr)
library(stringr)
library(tidytext)
library(ggplot2)

sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

source_counts <- sentences |>
  mutate(source = if_else(document_id == "F001", "training flyer", "job details")) |>
  count(source, name = "rows")

knitr::kable(
  source_counts,
  col.names = c("Source", "Rows"),
  caption = "The 28 Riverton sentences by source",
  row.names = FALSE
)
The 28 Riverton sentences by source
Source Rows
job details 22
training flyer 6

The file is small enough to inspect by eye. That convenience is also the reason its phrase counts should be treated as local examples.

Build adjacent-word phrases

An n-gram is a run of n adjacent tokens. A unigram has one token, a bigram has two, and a trigram has three. These counts use the same token rule as Lesson 15: lowercase the text and drop punctuation. Because each row here is one sentence, no n-gram spans two sentences; a file with whole documents in rows would need a different check.

make_ngrams <- function(n) {
  if (identical(n, 1L)) {
    sentences |>
      select(sentence_id, text) |>
      unnest_tokens(output = ngram, input = text, token = "words") |>
      filter(!is.na(ngram))
  } else {
    sentences |>
      select(sentence_id, text) |>
      unnest_tokens(output = ngram, input = text, token = "ngrams", n = n) |>
      filter(!is.na(ngram))
  }
}

short_for_trigrams <- make_ngrams(1L) |>
  count(sentence_id, name = "tokens") |>
  filter(tokens < 3L)

bigrams <- make_ngrams(2L) |>
  count(ngram, name = "frequency", sort = TRUE)
trigrams <- make_ngrams(3L) |>
  count(ngram, name = "frequency", sort = TRUE)

top_phrases <- bind_rows(
  bigrams |>
    slice_head(n = 8) |>
    mutate(kind = "bigram"),
  trigrams |>
    slice_head(n = 8) |>
    mutate(kind = "trigram")
) |>
  select(kind, ngram, frequency)

knitr::kable(
  top_phrases,
  col.names = c("N-gram type", "N-gram", "Frequency"),
  caption = "The most frequent bigrams and trigrams in the 28 sentences",
  row.names = FALSE
)
The most frequent bigrams and trigrams in the 28 sentences
N-gram type N-gram Frequency
bigram is required 4
bigram training is 3
bigram are required 2
bigram experience is 2
bigram is preferred 2
bigram is provided 2
bigram no prior 2
bigram shifts are 2
trigram training is provided 2
trigram 12 week training 1
trigram a daytime schedule 1
trigram a high school 1
trigram a medical records 1
trigram a paid apprenticeship 1
trigram a portfolio is 1
trigram a valid driver’s 1

The sentence Evening classes has only two tokens, so it has no trigram. The code drops that empty result before counting. The most frequent bigram is is required, with 4 appearances, and the most frequent trigram is training is provided, with 2.

Measure the sparsity

Sparsity means that many possible items have no examples or only one example. The general reason is arithmetic, not this particular table. Each new word adds at most one new word type, but it can also add a new pair and a new triple; the set of pairs a vocabulary could form grows faster than the vocabulary itself. The Riverton counts below illustrate that argument rather than prove it.

unigrams <- make_ngrams(1L) |>
  count(ngram, name = "frequency", sort = TRUE)

ngram_counts <- list(unigrams, bigrams, trigrams)

ngram_summary <- tibble(
  n = 1:3,
  name = c("unigrams", "bigrams", "trigrams"),
  distinct_items = vapply(ngram_counts, nrow, integer(1)),
  seen_once = vapply(
    ngram_counts,
    \(counts) sum(counts$frequency == 1L),
    integer(1)
  ),
  seen_once_label = str_c(seen_once, "/", distinct_items)
)

knitr::kable(
  ngram_summary,
  col.names = c("n", "Unit", "Distinct units", "Seen once", "Seen-once fraction"),
  caption = "Distinct n-grams and one-time n-grams in the Riverton sentences",
  row.names = FALSE
)
Distinct n-grams and one-time n-grams in the Riverton sentences
n Unit Distinct units Seen once Seen-once fraction
1 unigrams 96 72 72/96
2 bigrams 112 102 102/112
3 trigrams 96 95 95/96

The 28 sentences contain 96 distinct unigrams, 112 distinct bigrams, and 96 distinct trigrams. The seen-once fractions are 72/96 for unigrams, 102/112 for bigrams, and 95/96 for trigrams. These numbers describe only this teaching corpus.

plot_data <- ngram_summary |>
  transmute(
    name,
    seen_once_share = seen_once / distinct_items,
    seen_once_label
  )

ggplot(plot_data, aes(x = name, y = seen_once_share)) +
  geom_col(fill = "#4C78A8") +
  geom_text(aes(label = seen_once_label), vjust = -0.4) +
  labs(
    x = NULL,
    y = "Share seen once"
  ) +
  scale_y_continuous(labels = \(value) str_c(round(value * 100), "%"), limits = c(0, 1)) +
  theme_minimal()
Bar chart with three bars for unigrams, bigrams, and trigrams. The seen-once share is high for every n-gram size and highest for trigrams.
Figure 1: Share of distinct n-grams seen once in the 28 Riverton sentences.

The chart focuses on the quantity the table asks the reader to compare: how much of each list appears only once.

Score pairs against chance

Raw frequency likes common words. Pointwise mutual information, or PMI, asks whether two adjacent words appear more often than they would if each word were chosen independently. For a pair (w1, w2):

PMI = log2( P(w1, w2) / (P(w1) * P(w2)) )

The probabilities here are token shares in these 28 sentences: 153 unigram tokens and 125 bigram tokens. A pair of words that each appear once, and that appear together once, gets a high score because the denominator is tiny. That is a property of the formula, not evidence that the pair is a stable phrase.

unigram_tokens <- make_ngrams(1L)
bigram_tokens <- make_ngrams(2L)
unigram_n <- nrow(unigram_tokens)
bigram_n <- nrow(bigram_tokens)

unigram_counts <- unigram_tokens |>
  count(ngram, name = "word_n")

pmi_bigrams <- bigram_tokens |>
  separate_wider_delim(
    ngram,
    delim = " ",
    names = c("word1", "word2"),
    too_many = "merge"
  ) |>
  count(word1, word2, name = "pair_n") |>
  left_join(unigram_counts, by = join_by(word1 == ngram)) |>
  rename(word1_n = word_n) |>
  left_join(unigram_counts, by = join_by(word2 == ngram)) |>
  rename(word2_n = word_n) |>
  mutate(
    ngram = str_c(word1, word2, sep = " "),
    pmi = log2(
      (pair_n / bigram_n) /
        ((word1_n / unigram_n) * (word2_n / unigram_n))
    )
  )

hapax_singletons <- pmi_bigrams |>
  filter(pair_n == 1L, word1_n == 1L, word2_n == 1L) |>
  arrange(ngram)

repeated_pmi <- pmi_bigrams |>
  filter(pair_n >= 2L) |>
  arrange(desc(pmi), desc(pair_n), ngram)

hapax_singleton_pmi <- log2(
  (1 / bigram_n) / ((1 / unigram_n) * (1 / unigram_n))
)

is_required <- pmi_bigrams |>
  filter(ngram == "is required")

knitr::kable(
  hapax_singletons |>
    slice_head(n = 3) |>
    transmute(
      ngram,
      pair_n,
      pmi = round(pmi, 2)
    ),
  col.names = c("Bigram", "Pair count", "PMI"),
  caption = "Three one-time bigrams whose words also appear only once",
  row.names = FALSE
)
Three one-time bigrams whose words also appear only once
Bigram Pair count PMI
50 pounds 1 7.55
able to 1 7.55
an essential 1 7.55

Thirty-seven of the 102 one-time bigrams are pairs of words that each appear once. They all receive PMI 7.55. The most frequent bigram, is required (4 times), scores 3.04 because is appears 13 times and required appears 7 times. A floor of two pair counts leaves ten bigrams.

repeated_pmi_table <- repeated_pmi |>
  transmute(
    ngram,
    pair_n,
    pmi = round(pmi, 2)
  )

knitr::kable(
  repeated_pmi_table,
  col.names = c("Bigram", "Pair count", "PMI"),
  caption = "Bigrams seen at least twice, ranked by PMI",
  row.names = FALSE
)
Bigrams seen at least twice, ranked by PMI
Bigram Pair count PMI
no prior 2 6.55
shifts are 2 5.55
the job 2 5.23
is preferred 2 3.85
is provided 2 3.85
work is 2 3.85
are required 2 3.74
training is 3 3.11
is required 4 3.04
experience is 2 2.85

With that floor, no prior leads and is required falls to ninth. Both lists still describe 28 teaching sentences. PMI does not repair sparsity; it changes which sparse pairs look interesting. For matches in running text, Lesson 25 uses a concordance instead of a score.

What to remember

  • An n-gram counts adjacent tokens after a chosen token rule.
  • The Riverton rows contain 112 distinct bigrams and 96 distinct trigrams.
  • A two-token sentence contributes no trigram.
  • PMI without a count floor ranks unique word pairs first.
  • Counts from 28 sentences illustrate sparsity; they do not estimate phrase use in workforce writing.

Report the repeated phrases as local clues, then stop. The useful result here is a small list of repeats paired with a warning about sparse phrase counts.

Sources