Finding distinctive words and phrases

Compare tf-idf with tagged phrase keywords

signals and discovery
keyword extraction
inaugural addresses
Learn how keyword extraction finds distinctive words and tagged phrases without establishing their importance.

Mara has a folder of 60 inaugural addresses and a blank label card for a small library display. The card has room for a few words, so a full reading is too much for the first pass.

A word list could help. However, the scoring rule may favour a rare name or unusual phrase rather than a useful description of the speech.

TipWhat you will learn

This lesson shows how to:

  • explain tf-idf in plain language.
  • compute distinctive words for speeches.
  • execute a tagged phrase keyword method with UDPipe.
  • compare word and phrase outputs on the same word form.
  • find whether the observed overlap exceeds a random baseline.

Load the speeches

The helper returns one row per paragraph. Here a paragraph is a block of at least 25 words. The corpus has 1,377 paragraphs from 60 speeches by 40 presidents.

quanteda records only a surname for each speech. Four surnames each occur for two people, so counting surnames would give 36. That count would merge John Adams with John Quincy Adams. The shared helper creates a full name to distinguish those people.

library(dplyr)
library(tidyr)
library(tibble)
library(stringr)
library(tidytext)
library(tokenizers)
library(udpipe)
library(purrr)

source("R/inaugural-corpus.R")
source("R/permutation-null.R")

paragraphs <- inaugural_paragraphs()

speeches <- paragraphs |>
  summarise(
    text = paste(paragraph, collapse = "\n"),
    paragraph_count = n(),
    .by = c(speech_id, year, president, party)
  )

corpus_summary <- tibble(
  measure = c("paragraphs", "speeches", "presidents"),
  value = c(
    nrow(paragraphs),
    n_distinct(paragraphs$speech_id),
    n_distinct(paragraphs$president)
  )
)
The inaugural-address corpus used in this lesson
Measure Value
paragraphs 1377
speeches 60
presidents 40

The next sections examine three speeches: 1789-Washington, 1865-Lincoln, and 2021-Biden. The code still computes tf-idf against all 60 speeches because distinctiveness depends on the comparison set.

Score distinctive words with tf-idf

Tf-idf combines two quantities. Term frequency is a word’s share of tokens in one speech. Inverse document frequency uses the ratio of all documents to documents that contain the word. It takes the logarithm of that ratio. Words that appear in many speeches therefore receive less weight. A word has a high score when it is common in one speech and uncommon in the corpus.

speech_words <- paragraphs |>
  select(speech_id, year, president, paragraph) |>
  unnest_tokens(word, paragraph) |>
  filter(str_detect(word, "^[a-z]+$"), !word %in% stop_words$word)

word_counts <- speech_words |>
  count(speech_id, year, president, word, name = "n")

tfidf_scores <- word_counts |>
  bind_tf_idf(word, speech_id, n) |>
  arrange(speech_id, desc(tf_idf), word)

selected_ids <- c("1789-Washington", "1865-Lincoln", "2021-Biden")

tfidf_top <- tfidf_scores |>
  filter(speech_id %in% selected_ids) |>
  arrange(desc(tf_idf), word, .by_group = TRUE) |>
  slice_head(n = 5, by = c(speech_id, year, president)) |>
  mutate(tf_idf = round(tf_idf, 4)) |>
  select(speech_id, word, n, tf_idf)
Top tf-idf words in three selected speeches
Speech Word Count tf-idf
1865-Lincoln offenses 3 0.0551
1865-Lincoln woe 3 0.0551
1865-Lincoln offense 2 0.0305
1865-Lincoln wills 2 0.0305
1865-Lincoln answered 2 0.0269
2021-Biden saint 2 0.0434
2021-Biden augustine 1 0.0297
2021-Biden block 1 0.0297
2021-Biden demonization 1 0.0297
2021-Biden distrusting 1 0.0297
1789-Washington immutable 2 0.0130
1789-Washington impressions 2 0.0130
1789-Washington providential 2 0.0130
1789-Washington qualifications 2 0.0115
1789-Washington peculiarly 2 0.0104

The 2021 list includes augustine, block, demonization, and distrusting, each counted once. A once-only word can rank high when it is rare across the comparison set. A high score therefore does not establish importance.

Score tagged phrases with RAKE

The second method uses the UDPipe tagger from the part-of-speech lesson. A lemma is a dictionary form such as answer for answered. RAKE, short for Rapid Automatic Keyword Extraction, finds sequences of relevant lemmas. Here the relevant lemmas are adjectives, common nouns, and proper nouns after stopword removal. This linguistic rule can return phrases such as actual expenditure or capitol dome.

tagger <- udpipe_load_model("data/treebank/en_ewt-500-tagger.udpipe")

vertical <- speeches$text |>
  # A stated pattern rather than `tokenize_words()`. That function splits on ICU
  # word boundaries, and ICU versions differ between machines, so the token
  # stream handed to the tagger was not the same on a Linux runner as on a
  # laptop. Words, numbers, and single punctuation marks are kept as separate
  # tokens, which is what the vertical tokenizer expects.
  str_extract_all("[A-Za-z]+(?:'[A-Za-z]+)*|[0-9]+|[[:punct:]]") |>
  vapply(\(tokens) paste(tokens, collapse = "\n"), character(1))

annotated_all <- udpipe_annotate(
  tagger,
  x = vertical,
  doc_id = speeches$speech_id,
  tokenizer = "vertical",
  tagger = "default",
  parser = "none"
) |>
  as.data.frame() |>
  as_tibble() |>
  mutate(
    lemma_clean = str_to_lower(lemma),
    relevant = upos %in% c("ADJ", "NOUN", "PROPN") &
      str_detect(lemma_clean, "^[a-z]+$") &
      !lemma_clean %in% stop_words$word
  )

rake_top_20 <- speeches$speech_id |>
  map(\(id) {
    one_speech <- annotated_all |>
      filter(doc_id == id)

    keywords_rake(
      one_speech,
      term = "lemma_clean",
      group = "doc_id",
      relevant = one_speech$relevant,
      ngram_max = 3,
      n_min = 1
    ) |>
      as_tibble() |>
      arrange(desc(rake), keyword) |>
      slice_head(n = 20) |>
      mutate(speech_id = id, .before = 1)
  }) |>
  list_rbind()

rake_top <- rake_top_20 |>
  filter(speech_id %in% selected_ids) |>
  slice_head(n = 5, by = speech_id) |>
  mutate(rake = round(rake, 2)) |>
  select(speech_id, keyword, ngram, freq, rake)

tagger_error <- annotated_all |>
  filter(doc_id == "1789-Washington", str_to_lower(token) %in% c("auspiciously", "commence")) |>
  select(token, lemma, upos)
Top RAKE phrase keywords in the same three speeches
Speech Tagged phrase Words Frequency RAKE score
1789-Washington actual expenditure 2 1 2.00
1789-Washington affectionate sensibility 2 1 2.00
1789-Washington ardent love 2 1 2.00
1789-Washington arduous struggle 2 1 2.00
1789-Washington auspiciously commence 2 1 2.00
1865-Lincoln impending civil war 3 1 4.25
1865-Lincoln american slavery 2 1 2.00
1865-Lincoln chiefly depend 2 1 2.00
1865-Lincoln divide effect 2 1 2.00
1865-Lincoln divine attribute 2 1 2.00
2021-Biden brave woman 2 1 2.00
2021-Biden capitol dome 2 1 2.00
2021-Biden civil war 2 1 2.00
2021-Biden common object 2 1 2.00
2021-Biden constant struggle 2 1 2.00

RAKE favours phrases that contain tagged content words. The table also shows a tagger error. UDPipe gives auspiciously an adjective label and commence a noun label, so auspiciously commence appears in the list. The list requires inspection before anyone chooses a final label.

Put both methods on the same word form

The two methods answer different questions. Tf-idf scores individual words across speeches. RAKE scores tagged word sequences within a speech. A comparison also requires matching word forms. Raw forms such as offenses, wills, and answered differ from RAKE lemmas such as offense, will, and answer.

The last comparison in this section uses a frequency-weighted random baseline. For each speech, the code draws two random lists of 20 lemmas and counts the shared words. A lemma that occurs more often in the speech is more likely to be drawn. The mean count across speeches is one random result, and the code makes 500 of them.

The p-value adds 1 to the number of random results at least as large as the observed result. It then divides that sum by 501, the number of random results plus 1.

comparison <- tfidf_top |>
  summarise(tfidf_words = list(word), .by = speech_id) |>
  left_join(
    rake_top |>
      summarise(
        rake_phrases = paste(keyword, collapse = " | "),
        rake_words = list(unique(unlist(str_split(keyword, " ")))),
        .by = speech_id
      ),
    by = join_by(speech_id)
  ) |>
  rowwise() |>
  mutate(overlap_count = sum(tfidf_words %in% rake_words)) |>
  ungroup() |>
  transmute(
    speech_id,
    tfidf_words = vapply(tfidf_words, paste, collapse = ", ", FUN.VALUE = character(1)),
    rake_phrases,
    overlap_count
  ) |>
  arrange(match(speech_id, selected_ids))

raw_tfidf_top_20 <- tfidf_scores |>
  arrange(desc(tf_idf), word, .by_group = TRUE) |>
  slice_head(n = 20, by = speech_id) |>
  summarise(tfidf_words = list(word), .by = speech_id)

rake_words_20 <- rake_top_20 |>
  summarise(
    rake_words = list(unique(unlist(str_split(keyword, " ")))),
    .by = speech_id
  )

raw_overlap_20 <- raw_tfidf_top_20 |>
  left_join(rake_words_20, by = join_by(speech_id)) |>
  rowwise() |>
  mutate(shared_words = sum(tfidf_words %in% rake_words)) |>
  ungroup()

lemma_counts <- annotated_all |>
  filter(str_detect(lemma_clean, "^[a-z]+$"), !lemma_clean %in% stop_words$word) |>
  count(doc_id, lemma_clean, name = "n") |>
  rename(speech_id = doc_id, word = lemma_clean)

lemma_tfidf_top_20 <- lemma_counts |>
  bind_tf_idf(word, speech_id, n) |>
  arrange(speech_id, desc(tf_idf), word) |>
  slice_head(n = 20, by = speech_id) |>
  summarise(tfidf_words = list(word), .by = speech_id)

normalized_overlap_20 <- lemma_tfidf_top_20 |>
  left_join(rake_words_20, by = join_by(speech_id)) |>
  rowwise() |>
  mutate(shared_words = sum(tfidf_words %in% rake_words)) |>
  ungroup()

overlap_summary <- bind_rows(
  raw_overlap_20 |>
    summarise(
      match_basis = "Raw word forms",
      speeches = n(),
      mean_shared_words = round(mean(shared_words), 2),
      median_shared_words = median(shared_words),
      speeches_with_zero = sum(shared_words == 0L),
      maximum_shared_words = max(shared_words)
    ),
  normalized_overlap_20 |>
    summarise(
      match_basis = "Common lemmas",
      speeches = n(),
      mean_shared_words = round(mean(shared_words), 2),
      median_shared_words = median(shared_words),
      speeches_with_zero = sum(shared_words == 0L),
      maximum_shared_words = max(shared_words)
    )
)

frequency_lists <- lemma_counts |>
  summarise(words = list(word), weights = list(n), .by = speech_id)

random_overlap_stat <- function() {
  map2_dbl(frequency_lists$words, frequency_lists$weights, \(words, weights) {
    draw_size <- min(20L, length(words))
    first_draw <- sample(words, size = draw_size, replace = FALSE, prob = weights)
    second_draw <- sample(words, size = draw_size, replace = FALSE, prob = weights)
    sum(first_draw %in% second_draw)
  }) |>
    mean()
}

overlap_null <- permutation_null(
  observed = mean(normalized_overlap_20$shared_words),
  replicate_fn = random_overlap_stat,
  replicates = 500L,
  seed = 5301L,
  alternative = "greater"
)

overlap_null_display <- overlap_null |>
  mutate(across(c(observed, null_mean, null_low, null_high, p_value), \(x) round(x, 3)))
Top-5 overlap before normalizing word forms
Speech Top tf-idf words Top RAKE phrases Shared raw words
1789-Washington immutable, impressions, providential, qualifications, peculiarly actual expenditure | affectionate sensibility | ardent love | arduous struggle | auspiciously commence 0
1865-Lincoln offenses, woe, offense, wills, answered impending civil war | american slavery | chiefly depend | divide effect | divine attribute 0
2021-Biden saint, augustine, block, demonization, distrusting brave woman | capitol dome | civil war | common object | constant struggle 0
Overlap after widening both keyword lists to 20 items across all speeches
Match basis Speeches Mean shared words Median shared words Zero-overlap speeches Maximum shared words
Raw word forms 60 1.80 1 14 9
Common lemmas 60 2.27 2 9 10
Frequency-weighted random baseline for normalized top-20 overlap
Observed Null mean Null low Null high p-value Replicates Alternative
2.267 1.621 1.383 1.851 0.002 500 greater

With 20 items per list, raw forms share 1.80 words on average. Lemmas share 2.27. The raw count understated agreement because it treated offenses and offense as different strings.

Random frequency-weighted pairs of depth-20 lists from the same speech averaged 1.62 shared words. The central 90% of the 500 random results ranged from 1.38 to 1.85. No random result reached the observed 2.27, so p = 1/501, which rounds to 0.002. That is the smallest p-value possible with 500 random results. The observed normalized overlap exceeds that baseline, but the practical overlap remains small.

Keyword extractors always return a list. The comparison above removes disagreement caused by different word forms. It then measures overlap between random lists that preserve the speech’s word-frequency weights. This comparison adds evidence about the list beyond the algorithm’s output alone.

What to remember

  • Tf-idf scores words that are distinctive within a comparison set.
  • A high tf-idf score is not proof that a word is central to a speech.
  • RAKE uses tagged lemma sequences, so its phrase candidates still need reading.
  • Across all 60 speeches, raw top-20 overlap is 1.80 words on average. Lemma overlap is 2.27.
  • Normalized overlap exceeds the frequency-weighted random baseline, but the methods still select mostly different words.

Use keyword lists to select text for closer reading. A person must still choose the words for a label card, index, or exhibit note.

Sources