suppressPackageStartupMessages({
library(dplyr)
library(huggingfaceR)
library(purrr)
library(reticulate)
library(stringr)
library(tibble)
library(tidyr)
})
source("R/use-nlg.R")
paraphrase_model <- load_nlg_pipeline(
"qwen_1_5b_instruct",
"text-generation"
)
model_record <- paraphrase_model$metadata |>
select(model_id, revision, license)Changing the wording without changing the instruction
Separate a real rewrite from a meaning-preservation check
natural language generation
paraphrasing
evaluation
Learn how to generate paraphrases and examine names, numbers, negation, and modality without treating word overlap as equivalence.
June edits public notices for an archive. Sentences with identical wording can seem unnatural. However, a rewrite creates a new instruction if it changes who may enter or what visitors must bring.
For this lesson, a paraphrase changes wording but preserves the notice’s participants, quantities, negation, limits, and obligations. Linguistic research sometimes accepts approximate meaning. An operational notice needs a stricter rule because readers may act on one word such as may, must, or not.
Paraphrasing primarily preserves meaning while changing wording. Summarization primarily compresses or selects content. Simplification primarily reduces complexity or makes the language less formal. An output can serve more than one aim, so these classes can overlap.
TipWhat you will learn
This lesson shows how to:
- generate paraphrase candidates with a fixed local model
- keep an unchanged-input baseline beside the generated wording
- measure visible wording change without treating it as semantic equivalence
- examine names, numbers, negation, and modality
- preserve model failures instead of removing them
- record human checks of meaning as pending.
Load the same fixed checkpoint
This page uses the summarization lesson’s checkpoint, or saved model. The task, prompt, baseline, and rule for checks differ. huggingfaceR loads the model from the prepared local directory.
| Model | Revision | License |
|---|---|---|
| Qwen/Qwen2.5-1.5B-Instruct | 989aa7980e4cf806f80c7fef2b1adb7bc71aa306 | Apache-2.0 |
Write the preservation rule first
The author constructed these four behavioral probes to expose common changes in operational meaning. A probe is a selected diagnostic example. The probes are not a random test set. The lesson calculates no population accuracy estimate from them.
protected_name, modality_class, and modality_pattern are blank when a probe does not need that check. The baseline uses a copy of the source. Copying preserves the original wording and facts, but it does not perform the requested rewrite.
paraphrase_inputs <- tribble(
~probe_id, ~source_text, ~protected_name, ~requires_negation,
~modality_class, ~modality_pattern,
"PAR-01",
"The clinic sent 24 reminder letters on Monday.",
"",
FALSE,
"",
"",
"PAR-02",
"Visitors may bring one notebook to the reading room.",
"",
FALSE,
"permission",
"\\b(may|can|allowed|permitted)\\b",
"PAR-03",
paste(
"Maya Chen did not cancel the 14 September workshop;",
"she moved it from Room 2 to Room 5."
),
"Maya Chen",
TRUE,
"",
"",
"PAR-04",
"Only archive staff must sign the restricted-room log.",
"",
FALSE,
"obligation",
"\\b(must|required|obliged|has to)\\b"
)| Probe ID | Source notice | Protected name | Negation required | Modality class | Accepted surface markers |
|---|---|---|---|---|---|
| PAR-01 | The clinic sent 24 reminder letters on Monday. | FALSE | |||
| PAR-02 | Visitors may bring one notebook to the reading room. | FALSE | permission | \b(may|can|allowed|permitted)\b | |
| PAR-03 | Maya Chen did not cancel the 14 September workshop; she moved it from Room 2 to Room 5. | Maya Chen | TRUE | ||
| PAR-04 | Only archive staff must sign the restricted-room log. | FALSE | obligation | \b(must|required|obliged|has to)\b |
Generate candidates without silent truncation
The complete chat-formatted prompt must stay below 192 input tokens. Greedy decoding selects the highest-scoring next token and uses no random sampling. The output limit is 72 tokens. Small numerical differences between machines can still change a selection. Greedy decoding can also produce an incorrect paraphrase.
paraphrase_system <- paste(
"Rewrite the notice in different words.",
"Preserve every participant, quantity, date, negation, scope,",
"permission, and obligation.",
"Return the rewritten notice only."
)
paraphrase_requests <- paraphrase_inputs |>
mutate(
user_prompt = paste0(
"PROBE ID: ", probe_id,
"\nNOTICE: ", source_text
),
model_prompt = map_chr(
user_prompt,
\(user) {
nlg_chat_prompt(
paraphrase_model$tokenizer,
paraphrase_system,
user
)
}
),
input_tokens = map_int(
model_prompt,
\(prompt) nlg_token_count(paraphrase_model$tokenizer, prompt)
)
)| Probe ID | Complete prompt tokens |
|---|---|
| PAR-01 | 68 |
| PAR-02 | 67 |
| PAR-03 | 81 |
| PAR-04 | 67 |
Only after that assertion succeeds does the lesson make any generation call.
paraphrase_results <- paraphrase_requests |>
mutate(
generation = map(
model_prompt,
\(prompt) {
nlg_generate(
paraphrase_model,
prompt,
max_new_tokens = 72L,
repetition_penalty = 1.05
)
}
),
candidate = map_chr(generation, "text"),
candidate_display = str_replace_all(
candidate,
"\\s*\\n+\\s*",
" / "
),
output_tokens = map_int(generation, "output_tokens"),
ended_by_eos = map_lgl(generation, "ended_by_eos"),
hit_token_cap = map_lgl(generation, "hit_token_cap"),
copy_baseline = source_text
)| Probe ID | Source | Generated candidate | Unchanged-input baseline |
|---|---|---|---|
| PAR-01 | The clinic sent 24 reminder letters on Monday. | CLINIC SENT 24 REMINDERS ON MONDAY. | The clinic sent 24 reminder letters on Monday. |
| PAR-02 | Visitors may bring one notebook to the reading room. | Visitors are permitted to bring one notebook into the reading room. | Visitors may bring one notebook to the reading room. |
| PAR-03 | Maya Chen did not cancel the 14 September workshop; she moved it from Room 2 to Room 5. | PROBE ID: PAR-03 / NOTICE: Maya Chen did not cancel the 14 September workshop; instead, she relocated it to Room 5. | Maya Chen did not cancel the 14 September workshop; she moved it from Room 2 to Room 5. |
| PAR-04 | Only archive staff must sign the restricted-room log. | Only archive staff are authorized to sign the restricted-room log. | Only archive staff must sign the restricted-room log. |
The copy baseline should pass every literal preservation check. It also fails the user’s request for different wording. Each generated candidate must preserve meaning and change wording.
Measure wording change
This diagnostic converts each string to lowercase. It extracts letter and number sequences with the ICU regular-expression engine in stringr. It then computes Jaccard overlap between the unique token sets. Jaccard overlap divides the number of shared tokens by the number of tokens in either set. An overlap of 1 means the sets match. A lower value means some visible wording changed.
token_set <- function(text) {
str_extract_all(
str_to_lower(text, locale = "en"),
"[\\p{L}\\p{N}]+"
)[[1]] |>
unique()
}
jaccard_overlap <- function(left, right) {
left_tokens <- token_set(left)
right_tokens <- token_set(right)
length(intersect(left_tokens, right_tokens)) /
length(union(left_tokens, right_tokens))
}
wording_diagnostics <- paraphrase_results |>
transmute(
probe_id,
exact_copy = candidate == source_text,
unique_token_jaccard = map2_dbl(
source_text,
candidate,
jaccard_overlap
)
)| Probe ID | Exact copy | Unique-token Jaccard overlap |
|---|---|---|
| PAR-01 | FALSE | 0.56 |
| PAR-02 | FALSE | 0.67 |
| PAR-03 | FALSE | 0.58 |
| PAR-04 | FALSE | 0.67 |
A low overlap can result from a good synonym or a damaging change. A high overlap can still occur after removal of not. The score describes token sets, not equivalence.
Examine visible invariants
An invariant is information that a rewrite must preserve. The code searches for exact Arabic numerals, the protected name, and an explicit negation marker. It also searches for a short list of markers for the predeclared modality class. Modality distinguishes permission, obligation, and related meanings.
These string checks find some preventable mistakes. They also reject some valid alternatives: 24 may become twenty-four, and must may become have to. A string-check result cannot establish whether the rewrite preserves meaning.
extract_numbers <- function(text) {
str_extract_all(text, "\\b\\d+\\b")[[1]] |>
sort()
}
has_negation <- function(text) {
str_detect(
text,
regex(
"\\b(no|not|never|without|cannot|can't|didn't|did not)\\b",
ignore_case = TRUE
)
)
}
invariant_checks <- paraphrase_results |>
mutate(
numbers_preserved = map2_lgl(
source_text,
candidate,
\(source, output) {
source_numbers <- extract_numbers(source)
if (length(source_numbers) == 0L) {
NA
} else {
identical(source_numbers, extract_numbers(output))
}
}
),
name_present = map2_lgl(
protected_name,
candidate,
\(name, output) {
if (!nzchar(name)) {
NA
} else {
str_detect(output, fixed(name))
}
}
),
negation_visible = map2_lgl(
requires_negation,
candidate,
\(required, output) {
if (!required) NA else has_negation(output)
}
),
modal_visible = map2_lgl(
modality_pattern,
candidate,
\(pattern, output) {
if (!nzchar(pattern)) {
NA
} else {
str_detect(
output,
regex(pattern, ignore_case = TRUE)
)
}
}
)
) |>
select(
probe_id,
numbers_preserved,
name_present,
negation_visible,
modal_visible
)
invariant_check_display <- invariant_checks |>
mutate(
across(
c(
numbers_preserved,
name_present,
negation_visible,
modal_visible
),
nlg_screen_label
)
)
screen_actions <- invariant_checks |>
rowwise() |>
mutate(
applicable_screens = sum(!is.na(c_across(c(
numbers_preserved,
name_present,
negation_visible,
modal_visible
)))),
flagged_screens = sum(c_across(c(
numbers_preserved,
name_present,
negation_visible,
modal_visible
)) %in% FALSE)
) |>
ungroup() |>
select(probe_id, applicable_screens, flagged_screens)
paraphrase_actions <- wording_diagnostics |>
left_join(screen_actions, by = "probe_id") |>
mutate(
action = case_when(
exact_copy ~ "revise wording",
flagged_screens > 0L ~ "hold for human meaning review",
TRUE ~ "send to human meaning review"
)
) |>
select(probe_id, flagged_screens, action)| Probe ID | Arabic-numeral screen | Protected-name screen | Negation screen | Modality screen |
|---|---|---|---|---|
| PAR-01 | not flagged by screen | not applicable | not applicable | not applicable |
| PAR-02 | not applicable | not applicable | not applicable | not flagged by screen |
| PAR-03 | held for human review | not flagged by screen | not flagged by screen | not applicable |
| PAR-04 | not applicable | not applicable | not applicable | held for human review |
| Probe ID | Screen flags | Next action |
|---|---|---|
| PAR-01 | 0 | send to human meaning review |
| PAR-02 | 0 | send to human meaning review |
| PAR-03 | 1 | hold for human meaning review |
| PAR-04 | 1 | hold for human meaning review |
A table value of held for human review requires a human check. It does not prove a change in meaning. Compare the stress-probe output with its source. Prompt labels are not part of the notice. The candidate must preserve the move from Room 2 to Room 5.
If the obligation probe replaces must sign with authorized to sign, it changes obligation to permission. A modality mismatch requires a human check, but the pattern list can also reject equivalent wording. The conclusion therefore depends on human comparison, not the automatic check.
Record check status
The model completed generation, and the automatic checks completed their limited tasks. Human approval remains pending.
paraphrase_review <- tibble(
review = c(
"local model execution",
"complete prompt within 192 tokens",
"wording-change diagnostic",
"surface invariant checks",
"human checks of meaning preservation",
"human prose checks"
),
status = c(
sprintf(
"completed: %d candidates",
sum(nzchar(paraphrase_results$candidate))
),
sprintf(
"within budget: %d/%d candidates",
sum(paraphrase_results$input_tokens <= 192L),
nrow(paraphrase_results)
),
sprintf(
"computed: %d/%d candidates changed wording",
sum(!wording_diagnostics$exact_copy),
nrow(wording_diagnostics)
),
sprintf(
"held: %d/%d candidates flagged",
sum(paraphrase_actions$flagged_screens > 0L),
nrow(paraphrase_actions)
),
"pending",
"pending"
)
)| Check stage | Status |
|---|---|
| local model execution | completed: 4 candidates |
| complete prompt within 192 tokens | within budget: 4/4 candidates |
| wording-change diagnostic | computed: 4/4 candidates changed wording |
| surface invariant checks | held: 2/4 candidates flagged |
| human checks of meaning preservation | pending |
| human prose checks | pending |
June can accept a candidate only after comparison with the notice under the initial preservation rule. Generation, visible wording change, and invariant flags supply evidence for human checks. None replaces those checks.
What to remember
- A paraphrase changes wording while preserving the information that matters.
- Copying is a useful preservation baseline, but it is not a successful rewrite.
- Token overlap measures wording, not semantic equivalence.
- Examine names, numbers, negation, limits, permission, and obligation separately.
- Keep failed model outputs visible when they reveal an invariant violation.
- Record human checks of meaning and prose apart from automatic execution.