Selecting sentences as a summary

Compare TextRank with simple null models

signals and discovery
extractive summarization
inaugural addresses
Learn how extractive summarization ranks existing sentences and why chance baselines still matter.

Jon has ten minutes before a reading-group meeting and one speech still open on his laptop. He does not need a new paragraph written for him. He needs to know which existing sentences deserve a first look.

Extractive summarization selects sentences already present in a text. It does not write new sentences. Readers can therefore compare each selected sentence with the source.

TipWhat you will learn

This lesson shows how to:

  • explain extractive summarization.
  • build the two input tables required by textrank_sentences().
  • rank sentences from one speech.
  • compare the result with random-position baselines.
  • state what cannot be measured without a reference summary.

Build the sentence and term tables

TextRank is a graph-ranking algorithm. Each sentence is a node, an item in a graph. Two nodes connect when their sentences share terms. The function needs a data table with textrank_id and sentence. It also needs a terminology table with textrank_id and term.

library(dplyr)
library(tidyr)
library(tibble)
library(stringr)
library(tidytext)
library(tokenizers)
library(textrank)

source("R/inaugural-corpus.R")
source("R/permutation-null.R")

paragraphs <- inaugural_paragraphs()

speeches <- paragraphs |>
  summarise(text = paste(paragraph, collapse = "\n"), .by = c(speech_id, year, president, party))

speech <- speeches |>
  filter(speech_id == "1933-Roosevelt")

sentence_vec <- tokenize_sentences(
  speech$text,
  strip_punct = FALSE,
  simplify = TRUE
)

sentence_data <- tibble(
  sentence_position = seq_along(sentence_vec),
  textrank_id = sprintf("s%02d", sentence_position),
  sentence = sentence_vec
) |>
  mutate(sentence_words = str_count(sentence, "[A-Za-z]+"))

terminology <- sentence_data |>
  select(textrank_id, sentence) |>
  unnest_tokens(term, sentence) |>
  filter(str_detect(term, "^[a-z]+$"), !term %in% stop_words$word)

input_summary <- tibble(
  table = c("data", "terminology", "distinct terms"),
  rows = c(nrow(sentence_data), nrow(terminology), n_distinct(terminology$term))
)
Sentence and term inputs for TextRank
Input Rows
data 81
terminology 700
distinct terms 511

The data table has 81 sentences. The terminology table has 700 sentence-term rows and 511 distinct terms after stopwords are removed.

Rank the sentences

The algorithm scores each sentence from its connections to other sentences. A sentence can rank high because it shares terms with many sentences. Connections to other high-ranked sentences can also increase its rank.

textrank_result <- textrank_sentences(
  data = sentence_data |>
    select(textrank_id, sentence),
  terminology = terminology |>
    select(textrank_id, term)
)

ranked_sentences <- textrank_result$sentences |>
  as_tibble() |>
  left_join(
    sentence_data |>
      select(textrank_id, sentence_position, sentence_words),
    by = join_by(textrank_id)
  ) |>
  arrange(desc(textrank), sentence_position)

textrank_summary <- ranked_sentences |>
  slice_head(n = 3) |>
  transmute(
    rank = row_number(),
    sentence_position,
    sentence_words,
    score = round(textrank, 4),
    sentence
  )
The three highest-ranked sentences in the 1933 speech
Rank Sentence position Sentence words TextRank score Selected sentence
1 6 30 0.0328 In every dark hour of our national life a leadership of frankness and vigor has met with that understanding and support of the people themselves which is essential to victory.
2 67 24 0.0270 I am prepared under my constitutional duty to recommend the measures that a stricken nation in the midst of a stricken world may require.
3 60 25 0.0239 With this pledge taken, I assume unhesitatingly the leadership of this great army of our people dedicated to a disciplined attack upon our common problems.

The selected sentences come from positions 6, 67, and 60. Their positions differ from the speech opening. A visible difference does not establish that the selection is unusual.

Ask what random positions do

The corpus has no reference summary for this speech, so this page cannot report accuracy. It can show the selected sentences and their positions. It can also compare those positions with random sentence selections.

If human reference summaries existed, an overlap measure such as ROUGE could compare selected words or word sequences with those references. ROUGE would measure overlap, not whether the summary is complete, factual, or useful to a particular reader.

A lead baseline selects the first sentences. News openings often contain the main facts, so summaries that use those openings can perform well. Here the baseline provides only a comparison.

A permutation null gives the same statistic after random shuffling. The p-value adds 1 to the number of shuffled selections at least as extreme as the observed selection. It then divides that sum by the number of shuffled selections plus 1.

lead_baseline <- sentence_data |>
  slice_head(n = 3) |>
  transmute(
    baseline_rank = row_number(),
    sentence_position,
    sentence
  )

position_check <- tibble(
  method = c("TextRank", "Lead baseline"),
  selected_positions = c(
    paste(textrank_summary$sentence_position, collapse = ", "),
    paste(lead_baseline$sentence_position, collapse = ", ")
  ),
  earliest_position = c(
    min(textrank_summary$sentence_position),
    min(lead_baseline$sentence_position)
  ),
  latest_position = c(
    max(textrank_summary$sentence_position),
    max(lead_baseline$sentence_position)
  )
)

overlap_count <- sum(
  textrank_summary$sentence_position %in% lead_baseline$sentence_position
)

lead_overlap_null <- permutation_null(
  observed = overlap_count,
  replicate_fn = function() sum(sample(sentence_data$sentence_position, 3) %in% lead_baseline$sentence_position),
  replicates = 1000L,
  seed = 5401L,
  alternative = "less"
)

spread_observed <- max(textrank_summary$sentence_position) - min(textrank_summary$sentence_position)

spread_null <- permutation_null(
  observed = spread_observed,
  replicate_fn = function() {
    random_positions <- sample(sentence_data$sentence_position, 3)
    max(random_positions) - min(random_positions)
  },
  replicates = 1000L,
  seed = 5402L,
  alternative = "greater"
)

mean_length_observed <- mean(textrank_summary$sentence_words)

length_null <- permutation_null(
  observed = mean_length_observed,
  replicate_fn = function() mean(sentence_data$sentence_words[sample(sentence_data$sentence_position, 3)]),
  replicates = 1000L,
  seed = 5403L,
  alternative = "greater"
)

null_checks <- bind_rows(
  lead_overlap_null |> mutate(statistic = "Overlap with lead-3"),
  spread_null |> mutate(statistic = "Position range"),
  length_null |> mutate(statistic = "Mean selected sentence words")
) |>
  select(statistic, observed, null_mean, null_low, null_high, p_value, replicates, alternative) |>
  mutate(across(c(observed, null_mean, null_low, null_high, p_value), \(x) round(x, 3)))
The first three sentences as a lead baseline
Lead rank Sentence position Selected sentence
1 1 I am certain that my fellow Americans expect that on my induction into the Presidency I will address them with a candor and a decision which the present situation of our Nation impels.
2 2 This is preeminently the time to speak the truth, the whole truth, frankly and boldly.
3 3 Nor need we shrink from honestly facing conditions in our country today.
A position check for the extracted sentences
Method Selected positions Earliest Latest
TextRank 6, 67, 60 6 67
Lead baseline 1, 2, 3 1 3
Random-position checks for the TextRank selection
Statistic Observed Null mean Null low Null high p-value Replicates Alternative
Overlap with lead-3 0.000 0.113 0.000 1.00 0.889 1000 less
Position range 61.000 41.263 12.000 70.00 0.189 1000 greater
Mean selected sentence words 26.333 22.761 11.667 38.35 0.297 1000 greater

The two selections share 0 sentence positions. Random three-sentence selections from an 81-sentence speech usually share no positions with sentences 1 to 3. In 1,000 shuffles, the null mean was 0.113. The p-value for unusually low overlap was 0.889.

Other checks examine whether TextRank selects widely separated or longer sentences. The observed range was 61 sentence positions, against a random mean of 41.263 (p = 0.189). The selected sentences averaged 26.33 words, against a random mean of 22.761 (p = 0.297). Neither result exceeds its chance baseline convincingly. TextRank supplies sentences for a person to examine, not evidence that these three sentences form a better summary.

Summarizers always return sentences. A null comparison measures the same statistic after random selection replaces the algorithm’s selection. Without that comparison, a clear list can appear more meaningful than the evidence warrants.

These three checks do not establish that TextRank produces a better summary. The method executes and returns readable output, but the evidence does not justify unattended summarization of this material. A person must read the selected sentences and judge their adequacy.

What to remember

  • Extractive summarization selects sentences already in the text.
  • textrank_sentences() needs a sentence table and a term table.
  • The 1933 speech produced 81 sentence rows and 700 sentence-term rows.
  • Zero overlap with the lead-3 baseline is what random selection usually produces here.
  • Without a reference summary, accuracy is not available.

Use the ranked sentences as a reading aid. Do not interpret their ranks as scores for the speech’s meaning.

Sources