Skip to main content

Writing a shorter account

Examine what a generated summary preserves and changes

Learn how abstractive summarization differs from sentence selection and why shorter text still needs comparison with its source.

Rosa prepares the morning brief for a neighborhood archive. She receives two service notes. Each is longer than the one sentence her colleagues need. A shorter sentence must still preserve the source’s meaning.

Abstractive summarization generates wording for a shorter account of a source. The output may reuse some source phrases, but it is not limited to selecting whole sentences. Lesson 54 used sentence selection. This lesson asks a local language model to write a new sentence. The lesson then compares the candidate with the original record.

TipWhat you will learn

This lesson shows how to:

• keep source records, prompts, and outputs linked to persistent IDs
• count the model’s input tokens before generation
• compare generated summaries with a first-sentence baseline
• separate compression from source evidence and useful coverage
• examine names, numbers, and explicit limits without treating these checks as proof
• leave factual and editorial approval to a recorded human check.

Load the local model

Course setup downloads model files before rendering at the revision recorded in data/nlg-models.csv. The code below uses huggingfaceR to load that local version. R/use-nlg.R sets offline mode. If model files are missing, the lesson stops instead of starting a network download.

suppressPackageStartupMessages({
  library(dplyr)
  library(huggingfaceR)
  library(purrr)
  library(reticulate)
  library(stringr)
  library(tibble)
  library(tidyr)
})

source("R/use-nlg.R")

summary_model <- load_nlg_pipeline(
  "qwen_1_5b_instruct",
  "text-generation"
)

model_record <- summary_model$metadata |>
  select(model_id, revision, license, task_scope)

Pinned local model used for the generated summaries

Model	Revision	License	Course task
Qwen/Qwen2.5-1.5B-Instruct	989aa7980e4cf806f80c7fef2b1adb7bc71aa306	Apache-2.0	abstractive summarization;paraphrasing;long-text generation;question answering;chatbot dialogue

The model card documents an instruction-tuned causal language model. That description and license do not establish that its summaries are accurate.

Keep the source beside the brief

The author wrote these fictional teaching records for this lesson. They are not a random sample, a held-out benchmark, or evidence about archive services elsewhere.

Each brief states information that matters to the reader. SUM-01 needs the service, Tuesday schedule, and boundary on volunteer help. SUM-02 needs the event, photo-identification rule, and restricted-storage limit.

summary_sources <- tribble(
  ~source_id, ~brief, ~source_text,
  "SUM-01",
  paste(
    "One sentence for staff; retain the job-search service, Tuesday help",
    "schedule, and boundary on volunteer help."
  ),
  paste(
    "The Riverton library opens a new job-search room on Monday.",
    "The room has 12 computers.",
    "Volunteers offer free help on Tuesdays, but they do not complete",
    "applications for visitors.",
    "The pilot runs for six weeks."
  ),
  "SUM-02",
  paste(
    "One sentence for visitors; retain the catalog workshop,",
    "photo-identification rule, and restricted-storage limit."
  ),
  paste(
    "The neighborhood archive will open Room 5 on 14 September for a",
    "catalog workshop led by Maya Chen.",
    "Registration is free, but visitors must bring photo identification.",
    "The session does not include access to restricted storage."
  )
)

Constructed source records and their summary briefs

Source ID	Brief	Source text
SUM-01	One sentence for staff; retain the job-search service, Tuesday help schedule, and boundary on volunteer help.	The Riverton library opens a new job-search room on Monday. The room has 12 computers. Volunteers offer free help on Tuesdays, but they do not complete applications for visitors. The pilot runs for six weeks.
SUM-02	One sentence for visitors; retain the catalog workshop, photo-identification rule, and restricted-storage limit.	The neighborhood archive will open Room 5 on 14 September for a catalog workshop led by Maya Chen. Registration is free, but visitors must bring photo identification. The session does not include access to restricted storage.

Declare required and optional facts

The author fixes the checklist before generation. A missing required item requires candidate revision. Optional clues remain useful during source checks, but their omission alone does not violate the brief.

summary_checks <- tribble(
  ~source_id, ~check_id, ~needed_information, ~required, ~pattern,
  "SUM-01", "F1", "job-search service", TRUE,
  "\\bjob[ -]search\\b",
  "SUM-01", "F2", "Tuesday help", TRUE,
  "\\bTuesday\\b",
  "SUM-01", "F3", "volunteers do not complete applications", TRUE,
  "\\b(not|no|never|without|do not|does not)\\b",
  "SUM-01", "F4", "12 computers", FALSE,
  "\\b12\\b",
  "SUM-01", "F5", "six-week pilot", FALSE,
  "\\b(six|6)[ -]week",
  "SUM-02", "F6", "catalog workshop", TRUE,
  "\\bworkshop\\b",
  "SUM-02", "F7", "photo identification", TRUE,
  "\\b(photo|identification|ID)\\b",
  "SUM-02", "F8", "no restricted-storage access", TRUE,
  "\\b(not|no|without|does not)\\b",
  "SUM-02", "F9", "Room 5", FALSE,
  "\\bRoom 5\\b",
  "SUM-02", "F10", "14 September", FALSE,
  "\\b14\\b.*\\bSeptember\\b",
  "SUM-02", "F11", "Maya Chen", FALSE,
  "\\bMaya Chen\\b",
  "SUM-02", "F12", "free registration", FALSE,
  "\\bfree\\b"
)

Required facts and optional clues for source checks declared before generation

Source ID	Check	Predeclared information	Required by brief
SUM-01	F1	job-search service	TRUE
SUM-01	F2	Tuesday help	TRUE
SUM-01	F3	volunteers do not complete applications	TRUE
SUM-01	F4	12 computers	FALSE
SUM-01	F5	six-week pilot	FALSE
SUM-02	F6	catalog workshop	TRUE
SUM-02	F7	photo identification	TRUE
SUM-02	F8	no restricted-storage access	TRUE
SUM-02	F9	Room 5	FALSE
SUM-02	F10	14 September	FALSE
SUM-02	F11	Maya Chen	FALSE
SUM-02	F12	free registration	FALSE

Examine the input limit before generation

A model token is a tokenizer unit, not necessarily a word. The prompt includes the instruction, brief, and source text, so all three count against the input budget. This demonstration uses a 256-token limit and prohibits truncation.

summary_system <- paste(
  "Write one sentence for the stated brief.",
  "Use only information in the source.",
  "Do not add a cause, recommendation, or new event."
)

summary_inputs <- summary_sources |>
  mutate(
    user_prompt = paste0(
      "BRIEF: ", brief,
      "\nSOURCE ID: ", source_id,
      "\nSOURCE: ", source_text
    ),
    model_prompt = map2_chr(
      summary_system,
      user_prompt,
      \(system, user) {
        nlg_chat_prompt(
          summary_model$tokenizer,
          system,
          user
        )
      }
    ),
    input_tokens = map_int(
      model_prompt,
      \(prompt) nlg_token_count(summary_model$tokenizer, prompt)
    )
  )

Tokenizer counts before summarization

Source ID	Complete prompt tokens
SUM-01	120
SUM-02	117

The workflow does not remove text to fit the limit. A longer document would need a stated policy before the model call. Options include separate sections or hierarchical summarization, which summarizes sections before combining their summaries.

Generate the candidate summaries

Greedy decoding chooses the highest-scoring next token at each step. The output limit is 72 new tokens. The call returns the generated text and token-level stopping information from the local model.

summary_results <- summary_inputs |>
  mutate(
    generation = map(
      model_prompt,
      \(prompt) {
        nlg_generate(
          summary_model,
          prompt,
          max_new_tokens = 72L,
          repetition_penalty = 1.05
        )
      }
    ),
    generated_summary = map_chr(generation, "text"),
    output_tokens = map_int(generation, "output_tokens"),
    ended_by_eos = map_lgl(generation, "ended_by_eos"),
    hit_token_cap = map_lgl(generation, "hit_token_cap"),
    first_sentence_baseline = map_chr(
      str_split(source_text, "(?<=[.!?])\\s+"),
      \(sentences) sentences[[1]]
    )
  )

Generated summaries beside an extractive comparator

Source ID	Generated abstractive candidate	First-sentence teaching baseline
SUM-01	The Riverton library will continue to provide a job-search room with 12 computers, offer Tuesday volunteer help, and maintain a boundary on volunteer assistance during the pilot program’s six-week run.	The Riverton library opens a new job-search room on Monday.
SUM-02	Visitors are encouraged to attend the catalog workshop, participate in photo-identification, and adhere to the restricted-storage limit at the neighborhood archive’s opening on 14 September.	The neighborhood archive will open Room 5 on 14 September for a catalog workshop led by Maya Chen.

The first-sentence baseline is available before model generation. The comparison does not match output lengths. It shows the difference between selecting source wording and generating a new account.

Measure shortening without calling it truth

The next table counts whitespace-separated words. That rule is easy to examine, but it is different from the model tokenizer used for the generation budget. A compression ratio is candidate length divided by source length. A ratio below 1 means the candidate is shorter. The ratio says nothing about whether the candidate kept the needed facts.

count_words <- function(text) {
  str_count(str_squish(text), "\\S+")
}

compression <- summary_results |>
  transmute(
    source_id,
    source_words = map_int(source_text, count_words),
    generated_words = map_int(generated_summary, count_words),
    first_sentence_words = map_int(
      first_sentence_baseline,
      count_words
    ),
    generated_to_source = generated_words / source_words
  )

Length diagnostics use whitespace-separated words

Source ID	Source words	Generated words	Baseline words	Generated/source ratio
SUM-01	35	30	10	0.86
SUM-02	36	25	18	0.69

The model can produce a short sentence by selecting relevant content, omitting necessary content, or both. Length cannot distinguish those cases.

Examine facts chosen before generation

Rosa writes a checklist before reading the output. A regular expression is a pattern for matching text. The expressions below find visible strings related to that checklist. A missing match suggests a possible omission to examine. A match does not establish the correct role or limits of a name, number, or negative statement.

summary_flags <- summary_checks |>
  left_join(
    summary_results |>
      select(source_id, generated_summary),
    by = "source_id"
  ) |>
  mutate(
    text_flag = str_detect(
      generated_summary,
      regex(pattern, ignore_case = TRUE)
    )
  ) |>
  select(
    source_id,
    check_id,
    needed_information,
    required,
    text_flag
  )

summary_actions <- summary_flags |>
  group_by(source_id) |>
  summarise(
    decision = list(nlg_required_fact_action(required, text_flag)),
    optional_clues_missing = sum(!required & !text_flag),
    .groups = "drop"
  ) |>
  mutate(
    missing_required = map_int(decision, "missing_required"),
    action = if_else(
      map_lgl(decision, "revision_needed"),
      "revise",
      "send to human source review"
    )
  ) |>
  select(
    source_id,
    missing_required,
    optional_clues_missing,
    action
  )

String-match flags for required facts and optional clues for source checks

Source ID	Check	Information chosen before generation	Required by brief	Surface flag present
SUM-01	F1	job-search service	TRUE	TRUE
SUM-01	F2	Tuesday help	TRUE	TRUE
SUM-01	F3	volunteers do not complete applications	TRUE	FALSE
SUM-01	F4	12 computers	FALSE	TRUE
SUM-01	F5	six-week pilot	FALSE	TRUE
SUM-02	F6	catalog workshop	TRUE	TRUE
SUM-02	F7	photo identification	TRUE	TRUE
SUM-02	F8	no restricted-storage access	TRUE	FALSE
SUM-02	F9	Room 5	FALSE	FALSE
SUM-02	F10	14 September	FALSE	TRUE
SUM-02	F11	Maya Chen	FALSE	FALSE
SUM-02	F12	free registration	FALSE	FALSE

Only missing required facts trigger automatic revision

Source ID	Missing required facts	Missing optional clues	Next action
SUM-01	1	0	revise
SUM-02	1	3	revise

This check is intentionally incomplete. It cannot detect a changed relation, an incorrect number association, a cause absent from the source, or a fluent generalization. Those errors require comparison of the source and candidate.

Compare each summary with the volunteer and storage rules in its source. For SUM-01, volunteers offer Tuesday help but do not complete applications for visitors. For SUM-02, the session does not include access to restricted storage. A general statement about limits does not preserve either rule. The string checks can miss changes in these relationships.

N-gram overlap metrics such as ROUGE compare word sequences with a reference under a specified normalization rule. Normalization standardizes text before comparison. These metrics do not measure the share of claims with evidence in the source.

Record the limit of automated checks

Execution can establish that a model executed, stayed within the configured limits, and returned text. It cannot grant factual or editorial approval.

summary_review <- tibble(
  review = c(
    "local model execution",
    "prompt input within 256 tokens",
    "output stopped at EOS or token cap",
    "required-fact surface screen",
    "human checks of source evidence",
    "human checks of usefulness and prose"
  ),
  status = c(
    sprintf(
      "completed: %d candidates",
      sum(nzchar(summary_results$generated_summary))
    ),
    sprintf(
      "within budget: %d/%d candidates",
      sum(summary_inputs$input_tokens <= 256L),
      nrow(summary_inputs)
    ),
    sprintf(
      "recorded: %d/%d candidates",
      sum(summary_results$ended_by_eos | summary_results$hit_token_cap),
      nrow(summary_results)
    ),
    sprintf(
      "held: %d/%d candidates missing required facts",
      sum(summary_actions$missing_required > 0L),
      nrow(summary_actions)
    ),
    "pending",
    "pending"
  )
)

Automatic execution checks remain separate from human checks

Check stage	Status
local model execution	completed: 2 candidates
prompt input within 256 tokens	within budget: 2/2 candidates
output stopped at EOS or token cap	recorded: 2/2 candidates
required-fact surface screen	held: 2/2 candidates missing required facts
human checks of source evidence	pending
human checks of usefulness and prose	pending

Before publication, a reader must compare each summary claim with its source. The reader must find omissions of essential information. The reader must reject additions without source evidence. A fluent one-sentence result is only a candidate.

What to remember

• Abstractive summarization generates wording. Extractive summarization selects source text.
• Count the complete prompt with the model tokenizer.
• Disclose any truncation.
• Compression describes length, not coverage or source evidence.
• Examine omissions and additions against a brief written before generation.
• Names, numbers, and negation flags are prompts for human checks, not semantic proof.
• Keep model execution separate from human factual and editorial approval.

Sources

• Maynez et al. on faithfulness and factuality in abstractive summarization
• Lin on ROUGE and reference n-gram overlap
• Pagnoni et al. on classes of factual errors in summaries
• Qwen2.5-1.5B-Instruct model card
• Transformers text-generation controls