Writing a report from evidence

Choose evidence before generating sentences

natural language generation

report writing

data to text

Learn how to write a short report from results with recorded evidence. Keep the source reference for every factual sentence.

Nia finished the next-token study. The archive team wants a short report, not a folder of CSV files and not a guess about why one model scored better.

Report writing is a natural language generation application. A data-to-text workflow selects content with source evidence and orders the claims. It converts the claims into sentences and compares the result with its sources. Templates or a language model can produce the sentences. Neither method makes a claim correct.

What you will learn

This lesson shows how to:

state a report’s audience, purpose, and exclusions
keep one evidence row for each reported value
separate content selection, document planning, and sentence realization
generate a deterministic report with glue
reject fluent claims without evidence
keep factual checks separate from human prose checks.

Start with the reporting decision

The audience is the archive team. The team must decide whether the validation-selected trigram improves on the unigram baseline on the test speeches. The report need not explain how people write or why one context length performs best. It also need not predict results for another corpus.

The brief limits the report before sentence generation:

include the validation choice and untouched-test comparison
state the tokenizer, corpus, and unknown-token limit
exclude causes, transfer claims, and claims about factual writing quality.

Build an evidence ledger

An evidence ledger is a table that gives each usable result an ID. Each report sentence can refer to source records. This approach establishes sources before writing, rather than adding them afterward.

library(dplyr)
library(tibble)
library(stringr)
library(readr)
library(glue)

validation_results <- read_csv(
  "data/inaugural/next-token-validation.csv",
  na = c("", "NA"),
  col_types = cols(
    split = col_character(),
    model = col_character(),
    context_tokens = col_integer(),
    interpolation_strength = col_double(),
    next_token_rows = col_integer(),
    oov_target_share = col_double(),
    seen_context_share = col_double(),
    mean_log_loss = col_double(),
    perplexity = col_double(),
    selected = col_logical(),
    selection_reason = col_character()
  )
)
test_results <- read_csv(
  "data/inaugural/next-token-test.csv",
  na = c("", "NA"),
  col_types = cols(
    split = col_character(),
    model = col_character(),
    context_tokens = col_integer(),
    interpolation_strength = col_double(),
    next_token_rows = col_integer(),
    oov_target_share = col_double(),
    seen_context_share = col_double(),
    mean_log_loss = col_double(),
    perplexity = col_double(),
    top_1_accuracy = col_double(),
    role = col_character()
  )
)
test_by_speech <- read_csv(
  "data/inaugural/next-token-test-by-speech.csv",
  na = character(),
  col_types = cols(
    speech_id = col_character(),
    next_token_rows = col_integer(),
    model = col_character(),
    perplexity = col_double()
  )
)

validation_trigram <- validation_results |>
  filter(selected)
test_uniform <- test_results |>
  filter(model == "uniform")
test_unigram <- test_results |>
  filter(model == "unigram")
test_trigram <- test_results |>
  filter(model == "trigram")

perplexity_reduction <- 1 -
  test_trigram$perplexity / test_unigram$perplexity

paired_test <- test_by_speech |>
  select(speech_id, model, perplexity) |>
  tidyr::pivot_wider(
    names_from = model,
    values_from = perplexity
  ) |>
  summarise(
    test_speeches = n(),
    trigram_wins = sum(trigram < unigram),
    ties = sum(trigram == unigram),
    trigram_losses = sum(trigram > unigram)
  )

evidence_ledger <- tribble(
  ~evidence_id, ~source_artifact, ~split, ~measure, ~value, ~unit, ~limit,
  "E1", "next-token-validation.csv", "validation", "trigram perplexity",
  validation_trigram$perplexity, "perplexity",
  "trigram with interpolation strength 100, minimum across 13 candidates",
  "E2", "next-token-test.csv", "test", "uniform perplexity",
  test_uniform$perplexity, "perplexity",
  "2,500 equally likely vocabulary items",
  "E3", "next-token-test.csv", "test", "unigram perplexity",
  test_unigram$perplexity, "perplexity",
  "predeclared baseline under the same tokens",
  "E4", "next-token-test.csv", "test", "trigram perplexity",
  test_trigram$perplexity, "perplexity",
  "selected on validation speeches",
  "E5", "next-token-test.csv", "test", "trigram top-1 accuracy",
  test_trigram$top_1_accuracy, "share",
  "one highest-probability guess per row",
  "E6", "next-token-test-by-speech.csv", "test", "speech-level wins",
  paired_test$trigram_wins, "speeches",
  "paired against the unigram on the same test speeches",
  "E7", "next-token-test.csv", "test", "targets mapped to `<unk>`",
  test_trigram$oov_target_share, "share",
  "the original unseen token is not recovered",
  "E8", "data-raw/build-next-token-study.R", "study", "token rule",
  NA_real_, "setup",
  "lowercase ASCII alphabetic tokens, training-only vocabulary",
  "E9", "R/inaugural-corpus.R", "study", "corpus grain",
  NA_real_, "setup",
  "reconstructed inaugural paragraphs, paragraph-bounded contexts"
)

Evidence records available to the report
Evidence ID	Source artifact	Split	Measure	Value	Unit	Limit
E1	next-token-validation.csv	validation	trigram perplexity	133.3	perplexity	trigram with interpolation strength 100, minimum across 13 candidates
E2	next-token-test.csv	test	uniform perplexity	2500.0	perplexity	2,500 equally likely vocabulary items
E3	next-token-test.csv	test	unigram perplexity	240.7	perplexity	predeclared baseline under the same tokens
E4	next-token-test.csv	test	trigram perplexity	145.3	perplexity	selected on validation speeches
E5	next-token-test.csv	test	trigram top-1 accuracy	19.2%	share	one highest-probability guess per row
E6	next-token-test-by-speech.csv	test	speech-level wins	13	speeches	paired against the unigram on the same test speeches
E7	next-token-test.csv	test	targets mapped to `<unk>`	11.7%	share	the original unseen token is not recovered
E8	data-raw/build-next-token-study.R	study	token rule	documented	setup	lowercase ASCII alphabetic tokens, training-only vocabulary
E9	R/inaugural-corpus.R	study	corpus grain	documented	setup	reconstructed inaugural paragraphs, paragraph-bounded contexts

The records contain more than numbers. Each records the split, measure, unit, and limit. E1 contains no final test results. E2 through E4 are comparable because they use the same test rows, tokenizer, and vocabulary.

Decide which claims belong

Content determination chooses the facts the report will include. A fluent sentence can still fail here.

candidate_claims <- tribble(
  ~claim_id, ~claim, ~evidence_ids, ~status, ~reason,
  "C1",
  glue(
    "The trigram with interpolation strength 100 had the lowest ",
    "validation perplexity ({round(validation_trigram$perplexity, 1)})."
  ),
  "E1",
  "supported",
  "describes the recorded selection result",
  "C2",
  glue(
    "On test speeches, trigram perplexity was ",
    "{round(test_trigram$perplexity, 1)} versus ",
    "{round(test_unigram$perplexity, 1)} for the unigram baseline."
  ),
  "E3;E4",
  "supported",
  "compares the same unit and test rows",
  "C3",
  glue(
    "The trigram's highest-probability token matched ",
    "{sprintf('%.1f%%', 100 * test_trigram$top_1_accuracy)} ",
    "of test rows."
  ),
  "E5",
  "supported",
  "reports the decision rule separately from perplexity",
  "C4",
  glue(
    "The trigram reduced test perplexity by ",
    "{sprintf('%.1f%%', 100 * perplexity_reduction)} ",
    "relative to the unigram baseline."
  ),
  "E3;E4",
  "supported",
  "derives a comparison from two test perplexities",
  "C5",
  glue(
    "The trigram had lower perplexity in ",
    "{paired_test$trigram_wins} of {paired_test$test_speeches} ",
    "paired test speeches."
  ),
  "E6",
  "supported",
  "uses the paired speech-level comparison",
  "C6",
  paste(
    "The study uses lowercase alphabetic tokens from reconstructed",
    "inaugural paragraphs."
  ),
  "E8;E9",
  "supported",
  "states the token rule and source grain",
  "C7",
  glue(
    "{sprintf('%.1f%%', 100 * test_trigram$oov_target_share)} ",
    "of test targets were mapped to an unknown-token bucket."
  ),
  "E7",
  "supported",
  "states the recorded unknown-token limit",
  "U1",
  "The trigram understands presidential language.",
  NA_character_,
  "reject",
  "perplexity does not measure understanding",
  "U2",
  "Using a trigram will improve report writing.",
  NA_character_,
  "reject",
  "the study did not evaluate reports"
)

The workflow examines candidate claims before document planning
Claim ID	Candidate sentence	Evidence IDs	Decision	Reason
C1	The trigram with interpolation strength 100 had the lowest validation perplexity (133.3).	E1	supported	describes the recorded selection result
C2	On test speeches, trigram perplexity was 145.3 versus 240.7 for the unigram baseline.	E3;E4	supported	compares the same unit and test rows
C3	The trigram’s highest-probability token matched 19.2% of test rows.	E5	supported	reports the decision rule separately from perplexity
C4	The trigram reduced test perplexity by 39.6% relative to the unigram baseline.	E3;E4	supported	derives a comparison from two test perplexities
C5	The trigram had lower perplexity in 13 of 13 paired test speeches.	E6	supported	uses the paired speech-level comparison
C6	The study uses lowercase alphabetic tokens from reconstructed inaugural paragraphs.	E8;E9	supported	states the token rule and source grain
C7	11.7% of test targets were mapped to an unknown-token bucket.	E7	supported	states the recorded unknown-token limit
U1	The trigram understands presidential language.	NA	reject	perplexity does not measure understanding
U2	Using a trigram will improve report writing.	NA	reject	the study did not evaluate reports

The rejected sentences sound reasonable. They still exceed the evidence. Perplexity measures next-token probability under one setup. The study contains no test of understanding and no written-report outcome.

Plan the document before writing it

Document planning sets the order and purpose of the sentences. Realization turns values with recorded evidence into wording. The plan below starts with the decision, gives the untouched-test result, and ends with the limit.

report_plan <- tribble(
  ~sentence_id, ~role, ~claim_ids, ~evidence_ids,
  "S1", "selection", "C1", "E1",
  "S2", "test comparison", "C2;C4;C5", "E3;E4;E6",
  "S3", "top-choice result", "C3", "E5",
  "S4", "scope and unknown-token limit", "C6;C7", "E7;E8;E9"
)

report_sentences <- report_plan |>
  mutate(
    text = case_when(
      sentence_id == "S1" ~ glue(
        "The study selected a trigram with interpolation strength 100. It had ",
        "the lowest validation perplexity ",
        "({round(validation_trigram$perplexity, 1)})."
      ),
      sentence_id == "S2" ~ glue(
        "On {paired_test$test_speeches} untouched test speeches, ",
        "trigram perplexity was ",
        "{round(test_trigram$perplexity, 1)} versus ",
        "{round(test_unigram$perplexity, 1)} for the unigram baseline. ",
        "This was a reduction of {sprintf('%.1f%%', 100 * perplexity_reduction)}. ",
        "The trigram had lower perplexity in {paired_test$trigram_wins} of ",
        "{paired_test$test_speeches} speeches."
      ),
      sentence_id == "S3" ~ glue(
        "The trigram's highest-probability token matched ",
        "{sprintf('%.1f%%', 100 * test_trigram$top_1_accuracy)} ",
        "of test rows."
      ),
      sentence_id == "S4" ~ glue(
        "These results describe lowercase alphabetic tokens from reconstructed ",
        "inaugural paragraphs. Vocabulary mapping places {sprintf('%.1f%%', 100 * test_trigram$oov_target_share)} ",
        "of test targets in an unknown-token category."
      )
    )
  )

report_text <- str_c(
  report_sentences$text,
  collapse = "\n\n"
)

Every factual report sentence retains its claim and evidence IDs
Report unit	Role	Claim IDs	Evidence IDs	Generated text
S1	selection	C1	E1	The study selected a trigram with interpolation strength 100. It had the lowest validation perplexity (133.3).
S2	test comparison	C2;C4;C5	E3;E4;E6	On 13 untouched test speeches, trigram perplexity was 145.3 versus 240.7 for the unigram baseline. This was a reduction of 39.6%. The trigram had lower perplexity in 13 of 13 speeches.
S3	top-choice result	C3	E5	The trigram’s highest-probability token matched 19.2% of test rows.
S4	scope and unknown-token limit	C6;C7	E7;E8;E9	These results describe lowercase alphabetic tokens from reconstructed inaugural paragraphs. Vocabulary mapping places 11.7% of test targets in an unknown-token category.

The trace table stays with the report draft. If a value changes, use the claim or evidence ID to find the corresponding sentence.

Render the report from evidence

This chunk writes the four planned report units as prose. The generation is deterministic: the same evidence and templates produce the same text.

cat(
  "### Next-token study report\n\n",
  report_text,
  "\n",
  sep = ""
)

Next-token study report

The study selected a trigram with interpolation strength 100. It had the lowest validation perplexity (133.3).

On 13 untouched test speeches, trigram perplexity was 145.3 versus 240.7 for the unigram baseline. This was a reduction of 39.6%. The trigram had lower perplexity in 13 of 13 speeches.

The trigram’s highest-probability token matched 19.2% of test rows.

These results describe lowercase alphabetic tokens from reconstructed inaugural paragraphs. Vocabulary mapping places 11.7% of test targets in an unknown-token category.

Examine facts and prose separately

The automated checks compare known numbers and IDs with the evidence ledger. They also search for prohibited claims. They cannot decide whether the report is clear enough for the archive team or whether that team should act on it.

review_status <- tibble(
  review = c(
    names(automated_checks),
    "human factual checks",
    "human prose checks"
  ),
  status = c(
    unname(if_else(automated_checks, "passed", "failed")),
    "pending",
    "pending"
  )
)

Automated checks do not substitute for human approval
Check stage	Status
claim and evidence IDs resolve	passed
numbers match evidence	passed
claims without evidence excluded	passed
human factual checks	pending
human prose checks	pending

For a factual check, a reader compares each sentence with the source records. The reader looks for added causes, populations, or claims that results apply elsewhere. For a prose check, a reader examines order, clarity, tone, and usefulness for the named audience. Approval in one check does not establish approval in the other.

Use a language model for draft wording

A language model can draft alternate wording after the author finalizes the evidence ledger and report plan. If you use the draft, save the prompt, model, version, and output. Reject any number, entity, cause, citation, or evidence ID absent from the ledger. Do the factual and prose checks again.

This lesson does not call a live model. Credentials, model updates, and random output would make the rendered page hard to reproduce. A plausible continuation would still not supply evidence for its own claims.

What to remember

State the audience and decision before generating prose.
Select claims with source evidence before arranging sentences.
Keep evidence IDs, units, splits, and limits with each claim.
Templates improve reproducibility. They do not establish truth.
A language model may draft wording, but new claims require evidence.
Automated factual checks and human prose checks answer different questions.