Reading topic-model output carefully

Topic counts, LDA fitting, and random starts

signals and discovery

topic modeling

inaugural addresses

Learn how LDA groups co-occurring words, how diagnostics inform the topic count, and why no single metric chooses k for you.

Priya receives five numbered topics and a request to name them before lunch. Each table row contains a few common words and a number.

The table can encourage interpretations that exceed the evidence. A topic model produces topics even when those topics vary across fits. The analyst must compare results from different random starts.

What you will learn

This lesson shows how to:

explain Latent Dirichlet Allocation in everyday language.
build a paragraph-level document-feature matrix.
compare candidate topic counts with three diagnostics.
distinguish an LDA model from the algorithm used to fit it.
fit a five-topic LDA model with explicit seeds.
match topics across runs by their top terms.
compare observed agreement with a random-label null.

Build the matrix

Latent Dirichlet Allocation, usually shortened to LDA, groups words that tend to appear in the same documents. Here each document is one paragraph. A document-feature matrix stores document rows and word columns. The value k is the number of topics the analyst asks the model to find.

library(dplyr)
library(tibble)
library(stringr)
library(purrr)
library(readr)
library(quanteda)
library(topicmodels)

source("R/inaugural-corpus.R")
source("R/permutation-null.R")

paragraphs <- inaugural_paragraphs()

paragraph_tokens <- tokens(
  paragraphs$paragraph,
  remove_punct = TRUE,
  remove_numbers = TRUE
) |>
  tokens_tolower() |>
  tokens_remove(stopwords("en"))

paragraph_dfm <- dfm(paragraph_tokens)
trimmed_dfm <- dfm_trim(paragraph_dfm, min_termfreq = 5)
lda_input <- convert(trimmed_dfm, to = "topicmodels")

matrix_summary <- tibble(
  measure = c("documents", "features", "nonzero term counts"),
  value = c(ndoc(trimmed_dfm), nfeat(trimmed_dfm), sum(trimmed_dfm))
)

The trimmed paragraph matrix for LDA
Measure	Value
documents	1377
features	2587
nonzero term counts	52934

Trimming keeps words that appear at least five times in the paragraph corpus. The matrix has 1,377 paragraph documents and 2,587 word features.

Paragraphs give the model 1,377 documents instead of 60 whole speeches. Each document contains fewer words, so the model calculates an estimate of its topic mixture from less evidence. Document size is therefore part of model design. This modest corpus describes these speeches rather than political language generally.

Compare candidate topic counts

There is no single correct value of k. Fewer topics create broad groups. More topics divide the same vocabulary into smaller groups. The appropriate value depends on the planned analysis.

The committed diagnostic study fits k = 3 through k = 8 under three random starts. It builds the vocabulary from training speeches and evaluates perplexity on held-out speeches. Three measures ask different questions:

Held-out perplexity measures how poorly the fitted model predicts word counts in held-out speeches. Lower is better.
Semantic coherence asks whether a topic’s most probable words appear together in the training paragraphs. Higher values, which are less negative here, are better.
Adjusted top-word exclusivity measures whether those words concentrate in one topic rather than occur across all topics. Higher is better. The adjustment removes the (1/k) share expected under equal distribution.

topic_diagnostics <- read_csv(
  "data/inaugural/topic-k-diagnostics.csv",
  na = character(),
  col_types = cols(
    k = col_integer(),
    seeds = col_integer(),
    coherence_mean = col_double(),
    coherence_low = col_double(),
    coherence_high = col_double(),
    exclusivity_mean = col_double(),
    exclusivity_low = col_double(),
    exclusivity_high = col_double(),
    perplexity_mean = col_double(),
    perplexity_low = col_double(),
    perplexity_high = col_double(),
    coherence_rank = col_integer(),
    exclusivity_rank = col_integer(),
    perplexity_rank = col_integer(),
    selected = col_logical(),
    selection_reason = col_character()
  )
)

diagnostic_display <- topic_diagnostics |>
  transmute(
    k,
    coherence = round(coherence_mean, 3),
    coherence_rank,
    adjusted_exclusivity = round(exclusivity_mean, 3),
    exclusivity_rank,
    heldout_perplexity = round(perplexity_mean, 1),
    perplexity_rank,
    used_below = if_else(selected, "yes", "no")
  )

selected_diagnostics <- topic_diagnostics |>
  filter(selected)

three_topic_diagnostics <- topic_diagnostics |>
  filter(k == 3L)

coherence_lead <- (
  selected_diagnostics$coherence_mean -
    three_topic_diagnostics$coherence_mean
)
selected_coherence_spread <- (
  selected_diagnostics$coherence_high -
    selected_diagnostics$coherence_low
)

Three diagnostics give different preferred topic counts
k	Coherence	Coherence rank	Adjusted exclusivity	Exclusivity rank	Held-out perplexity	Perplexity rank	Used below
3	-1.623	4	0.274	1	1084.9	1	no
4	-1.618	3	0.196	2	1086.0	2	no
5	-1.603	1	0.157	3	1087.0	3	yes
6	-1.617	2	0.131	4	1087.6	4	no
7	-1.631	5	0.117	5	1088.3	5	no
8	-1.642	6	0.101	6	1088.9	6	no

The diagnostics do not agree. Three topics have the best held-out perplexity and adjusted exclusivity. Five topics have the best mean coherence. This lesson uses five because it gives coherence priority for this teaching example. Three would also be a defensible choice if prediction or top-word concentration mattered more. The later seed check examines only k = 5, so it cannot choose between five and three.

The coherence lead over three topics is only 0.02, while the three five-topic starts span 0.095. Rank 1 does not establish a clear advantage here. These ranges come from three starts on one speech-grouped split, not confidence intervals or a validated optimum for another corpus.

Do not choose k because its word table is easiest to name. That choice uses the output to select the model retrospectively. Set the diagnostics and their priority before reading topic tables. Then read the topics. People must examine the results because predictive fit and automated coherence do not guarantee useful or understandable topics.

The model and fitting method are separate choices

The topicmodels package can fit LDA in two ways. Variational expectation maximization, or VEM, approximates the model’s probability distributions. This lesson uses VEM. Gibbs sampling draws samples from those distributions across successive iterations. The analyst chooses the iteration count, warm-up, and number of retained samples. Both methods fit LDA but can return different answers.

Other topic models use different assumptions. LDA cannot represent a tendency for some topics to appear together in the same documents. A correlated topic model can represent that tendency. A structural topic model can use document information such as year to calculate estimates of topic prevalence or content. Seeded and supervised topic models add analyst guidance or an outcome. These models answer different questions and are not necessary for every analysis.

Fit five topics three times

The analyst chooses k = 5. The seed in control = list(seed = ...) fixes the random starting values used by the LDA fitting algorithm. It does not choose k, and it does not prove that the fitted topics are the only possible solution.

The committed diagnostic study used training and held-out speeches to inform that choice. The three objects below are new full-corpus fits for inspection at the selected resolution. They are not the diagnostic models.

top_terms <- function(model, n = 10L) {
  beta <- posterior(model)$terms
  bind_rows(lapply(seq_len(nrow(beta)), function(topic) {
    ordered <- sort(beta[topic, ], decreasing = TRUE)
    tibble(
      topic = topic,
      term = names(ordered)[seq_len(n)],
      beta = as.numeric(ordered[seq_len(n)])
    )
  }))
}

set.seed(42)
lda_a <- LDA(
  lda_input,
  k = 5,
  method = "VEM",
  control = list(seed = 42)
)

set.seed(7)
lda_b <- LDA(
  lda_input,
  k = 5,
  method = "VEM",
  control = list(seed = 7)
)

set.seed(2024)
lda_c <- LDA(
  lda_input,
  k = 5,
  method = "VEM",
  control = list(seed = 2024)
)

terms_a <- top_terms(lda_a, n = 10L)
terms_b <- top_terms(lda_b, n = 10L)
terms_c <- top_terms(lda_c, n = 10L)

topics_a <- terms_a |>
  slice_head(n = 8, by = topic) |>
  summarise(top_terms = paste(term, collapse = ", "), .by = topic)

Top terms from a five-topic LDA model with seed 42
Topic number	Top terms
1	country, power, can, us, must, government, people, new
2	people, can, government, us, great, upon, states, shall
3	can, may, country, government, people, us, nation, one
4	free, public, states, nation, us, power, upon, world
5	must, government, people, upon, public, nations, peace, shall

The row names identify topics by number. The words are high-probability terms under this fitted model, not validated descriptions of a subject.

Match topics across random starts

Topic numbers are arbitrary. Topic 1 in one run does not have to be topic 1 in another run. The comparison below matches topics by the largest total overlap in their top-10 term sets.

A permutation null computes the comparison again after shuffling topic assignments across top-term rows. The p-value adds 1 to the number of shuffled runs that match or exceed the observed overlap. It then divides that sum by the number of shuffled runs plus 1.

permutations <- function(x) {
  if (length(x) == 1L) {
    return(list(x))
  }

  seq_along(x) |>
    map(\(i) {
      permutations(x[-i]) |>
        map(\(p) c(x[i], p))
    }) |>
    list_flatten()
}

best_topic_match <- function(terms_first, terms_second) {
  topics <- sort(unique(terms_first$topic))
  first_sets <- topics |>
    map(\(topic_id) terms_first$term[terms_first$topic == topic_id])
  second_sets <- topics |>
    map(\(topic_id) terms_second$term[terms_second$topic == topic_id])

  candidate_orders <- permutations(seq_along(topics))
  candidate_scores <- candidate_orders |>
    map_int(\(order) {
      seq_along(topics) |>
        map_int(\(i) length(intersect(first_sets[[i]], second_sets[[order[i]]]))) |>
        sum()
    })

  best_order <- candidate_orders[[which.max(candidate_scores)]]

  tibble(
    topic_first = topics,
    topic_second = topics[best_order],
    shared_top_10_terms = seq_along(topics) |>
      map_int(\(i) length(intersect(first_sets[[i]], second_sets[[best_order[i]]])))
  )
}

seed_matches <- bind_rows(
  best_topic_match(terms_a, terms_b) |>
    mutate(comparison = "seed 42 vs 7", .before = 1),
  best_topic_match(terms_a, terms_c) |>
    mutate(comparison = "seed 42 vs 2024", .before = 1),
  best_topic_match(terms_b, terms_c) |>
    mutate(comparison = "seed 7 vs 2024", .before = 1)
)

seed_summary <- seed_matches |>
  summarise(
    mean_shared_terms = round(mean(shared_top_10_terms), 2),
    minimum_shared_terms = min(shared_top_10_terms),
    maximum_shared_terms = max(shared_top_10_terms),
    total_shared_terms = sum(shared_top_10_terms),
    .by = comparison
  )

within_fit_overlap <- expand_grid(
  topic_first = sort(unique(terms_a$topic)),
  topic_second = sort(unique(terms_a$topic))
) |>
  filter(topic_first < topic_second) |>
  mutate(
    shared_top_10_terms = map2_int(
      topic_first,
      topic_second,
      \(first_topic, second_topic) {
        length(intersect(
          terms_a$term[terms_a$topic == first_topic],
          terms_a$term[terms_a$topic == second_topic]
        ))
      }
    )
  )
within_fit_mean <- mean(within_fit_overlap$shared_top_10_terms)

seed_null <- permutation_null(
  observed = mean(seed_matches$shared_top_10_terms[seed_matches$comparison == "seed 42 vs 7"]),
  replicate_fn = function() {
    shuffled_terms <- terms_b |>
      mutate(topic = sample(topic, n(), replace = FALSE))

    mean(best_topic_match(terms_a, shuffled_terms)$shared_top_10_terms)
  },
  replicates = 500L,
  seed = 5502L,
  alternative = "greater"
)

seed_null_display <- seed_null |>
  mutate(across(c(observed, null_mean, null_low, null_high, p_value), \(x) round(x, 3)))

Best-matched topic overlaps across three five-topic runs
Comparison	Topic in first run	Matched topic in second run	Shared top-10 terms
seed 42 vs 7	1	2	6
seed 42 vs 7	2	5	4
seed 42 vs 7	3	1	6
seed 42 vs 7	4	3	5
seed 42 vs 7	5	4	7
seed 42 vs 2024	1	2	4
seed 42 vs 2024	2	3	7
seed 42 vs 2024	3	1	6
seed 42 vs 2024	4	5	5
seed 42 vs 2024	5	4	5
seed 7 vs 2024	1	1	6
seed 7 vs 2024	2	3	6
seed 7 vs 2024	3	5	6
seed 7 vs 2024	4	2	4
seed 7 vs 2024	5	4	3

Topic overlap after matching arbitrary topic numbers
Comparison	Mean shared top-10 terms	Minimum	Maximum	Total shared terms
seed 42 vs 7	5.6	4	7	28
seed 42 vs 2024	5.4	4	7	27
seed 7 vs 2024	5.0	3	6	25

Frequency-preserving allocation null for seed 42 vs seed 7 overlap
Observed	Null mean	Null low	Null high	p-value	Replicates	Alternative
5.6	4.901	4.4	5.4	0.03	500	greater

The three full-corpus five-topic fits show modest repeatability. Matched topics share 5.6, 5.4, and 5.0 of their top 10 terms on average across the three pairwise comparisons. Distinct topics within the seed-42 fit already share 3.9 terms on average.

The null keeps the same 50 top-term rows, which contain repeated high-frequency terms. It randomly allocates those rows across five equal-sized topics. Its mean is 4.901, compared with observed overlap 5.6 and p = 0.03. This is limited evidence of repeatability for these three k = 5 fits. It does not establish that five topics are more stable than another topic count.

Changing k changes the question

A four-topic fit provides a sensitivity check rather than the same model as a five-topic fit. After k changes, topic numbers no longer identify corresponding topics across models.

set.seed(42)
lda_4 <- LDA(
  lda_input,
  k = 4,
  method = "VEM",
  control = list(seed = 42)
)

topics_4 <- top_terms(lda_4, n = 8L) |>
  summarise(
    top_terms = paste(term, collapse = ", "),
    top_term = first(term),
    .by = topic
  )

shared_first_terms <- length(intersect(topics_a$top_terms |> str_extract("^[^,]+"), topics_4$top_term))

Top terms from a four-topic LDA model with seed 42
Topic number	Top terms
1	must, government, people, can, us, power, country, now
2	people, government, can, upon, great, us, states, shall
3	government, can, may, people, country, us, nation, world
4	public, free, upon, must, nation, states, good, world

A small comparison after changing k
Shared first terms with the seed-42 five-topic fit
2

The first terms for the four-topic model are must, people, government, and public. Two are also first terms in the five-topic seed-42 fit. This count does not give an estimate of stability because a different k specifies a different modeling target.

What to remember

LDA groups words that co-occur across paragraph documents.
This lesson used 1,377 paragraph documents and 2,587 features.
No diagnostic supplies a single correct k. The choice depends on the intended resolution and purpose.
Perplexity, coherence, and exclusivity can favor different topic counts.
Three full-corpus k = 5 fits showed modest repeatability. That check did not compare topic counts.
VEM and Gibbs sampling fit LDA through different algorithms. Other topic-model families answer different questions.
A different k requires new topic labels.
A topic name is a reading aid, not a finding produced by the model.

Keep the topic numbers with their model settings. Reassess the interpretation after model settings change.