Creating additional training examples

Make sure augmented text still meets its label definition

training data
augmentation
workforce research
Learn how to create workforce text variants and examine whether they still meet the intended label definition.

The reviewer comparison preserved two disputed cases rather than treating a majority as truth. Synthetic examples present a different risk. The Riverton Workforce Lab can increase row counts without adding observed evidence. Additional sentences may help a model learn. However, a changed sentence can lose the meaning that justified its label.

For this exercise, the team writes five fictional variants about training and schedules. These variants cannot establish a real course’s schedule or paid training offer.

Textual data augmentation creates new training examples through changes or additions to existing text. Augmentation changes a dataset. It does not collect new evidence about the labor market.

Note

All sentences and organizations in this lesson are fictional. Augmented rows are synthetic and remain marked as such.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • distinguish a new text variant from a new observation.
  • create controlled template variants.
  • use cmapr job titles without adding unsupported claims.
  • prepare a classification-check payload with huggingfaceR.
  • reject candidates that no longer meet their label definitions.
  • examine duplication, label balance, and effects of augmentation.

Start with one purpose

The team wants more examples of explicit training offers. The examples will not describe real employers or give an estimate of training-offer frequency. The next chunk loads helpers for files, tables, strings, CMap examples, and model-check payloads. It then reads the file into a tibble with declared column types.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
library(cmapr)
library(huggingfaceR)

workforce_sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)
training_examples <- workforce_sentences |>
  filter(reference_label == "training")
The seven training examples before augmentation
Sentence ID Reviewed training example
s001 Paid 12-week training is provided.
s006 The employer pays for certification training.
s012 Each new hire receives a mentor.
s015 On-the-job training is provided.
s016 This is a paid apprenticeship.
s021 Product training is included.
s025 Paid training stipend

Reviewers assigned the source labels under the manual annotation codebook. The augmentation step must preserve those label definitions.

Use job titles as template values

cmapr includes a ten-row teaching fixture with job-title transitions. This example uses three titles as vocabulary only. It does not use transition weights or interpret title changes as promotions.

cmap_titles <- read_csv(
  cmap_example_data(),
  col_types = cols(
    sector = col_character(),
    region = col_character(),
    job_title_from = col_character(),
    job_title_to = col_character(),
    transition_weighted_count = col_double(),
    si_from = col_double(),
    si_to = col_double()
  )
)
selected_titles <- cmap_titles$job_title_from[c(3L, 5L, 8L)]

selected_titles
[1] "Data Analyst"      "Medical Assistant" "Junior Developer"

The titles come from the cmapr package’s teaching fixture. That file lacks upstream record identifiers. This lesson therefore cannot authenticate the three strings as extracts from the full CMap dataset. The following sentences are fictional.

Create controlled variants

The template states the same kind of claim for each title.

source("R/workforce-codebook.R")
augmentation_codebook <- workforce_codebook()

template_variants <- tibble(
  source_sentence_id = "s001",
  source_family_id = "training-offer-s001",
  template_id = "training-pathway-v1",
  vocabulary_source = "cmapr example_transitions.csv",
  vocabulary_row = c(3L, 5L, 8L),
  vocabulary_commit =
    "eb2d0b3810e07782aab4a81fba1f539133f2ba6a",
  text = sprintf(
    "The %s pathway includes paid training.",
    str_to_lower(selected_titles)
  ),
  label = "training",
  synthetic = TRUE,
  method = "controlled template",
  codebook_version = augmentation_codebook$version,
  codebook_hash = augmentation_codebook$hash,
  review_status = "requires_human_review",
  split = "training_candidate"
)

schedule_variants <- tibble(
  source_sentence_id = "s003",
  source_family_id = "schedule-s003",
  template_id = "class-schedule-v1",
  vocabulary_source = "author-created template",
  vocabulary_row = NA_integer_,
  vocabulary_commit = NA_character_,
  text = sprintf(
    "%s schedules are available.",
    c("Morning", "Weekend")
  ),
  label = "schedule",
  synthetic = TRUE,
  method = "controlled template",
  codebook_version = augmentation_codebook$version,
  codebook_hash = augmentation_codebook$hash,
  review_status = "requires_human_review",
  split = "training_candidate"
)

augmented_rows <- bind_rows(
  template_variants,
  schedule_variants
)
augmentation_label_order <- c(
  "training",
  "requirement",
  "schedule",
  "skill",
  "other"
)
augmentation_counts <- tibble(label = augmentation_label_order) |>
  left_join(
    augmented_rows |>
      count(label, name = "n"),
    by = join_by(label)
  ) |>
  mutate(n = coalesce(n, 0L))
source_families <- augmented_rows |>
  distinct(source_family_id) |>
  arrange(source_family_id)
Five controlled augmentation candidates
Source sentence Source family Template Vocabulary source Vocabulary row Vocabulary commit Synthetic text Proposed label Synthetic Method Codebook version Codebook hash Review status Intended split
s001 training-offer-s001 training-pathway-v1 cmapr example_transitions.csv 3 eb2d0b3810e07782aab4a81fba1f539133f2ba6a The data analyst pathway includes paid training. training TRUE controlled template 1.1.0 7f4213ccadd8c8e36c5017dbce74c02bc519c9e18dd04908a9873af10206400d requires_human_review training_candidate
s001 training-offer-s001 training-pathway-v1 cmapr example_transitions.csv 5 eb2d0b3810e07782aab4a81fba1f539133f2ba6a The medical assistant pathway includes paid training. training TRUE controlled template 1.1.0 7f4213ccadd8c8e36c5017dbce74c02bc519c9e18dd04908a9873af10206400d requires_human_review training_candidate
s001 training-offer-s001 training-pathway-v1 cmapr example_transitions.csv 8 eb2d0b3810e07782aab4a81fba1f539133f2ba6a The junior developer pathway includes paid training. training TRUE controlled template 1.1.0 7f4213ccadd8c8e36c5017dbce74c02bc519c9e18dd04908a9873af10206400d requires_human_review training_candidate
s003 schedule-s003 class-schedule-v1 author-created template NA NA Morning schedules are available. schedule TRUE controlled template 1.1.0 7f4213ccadd8c8e36c5017dbce74c02bc519c9e18dd04908a9873af10206400d requires_human_review training_candidate
s003 schedule-s003 class-schedule-v1 author-created template NA NA Weekend schedules are available. schedule TRUE controlled template 1.1.0 7f4213ccadd8c8e36c5017dbce74c02bc519c9e18dd04908a9873af10206400d requires_human_review training_candidate

The rows retain source family, template, vocabulary source, and applicable package commit. They also retain codebook, synthetic status, intended split, and pending review status. All examples derived from s001 or s003 must remain outside the evaluation set with their source families. The five rows remain candidates until a person examines their wording, usefulness, and consistency with the codebook.

Examine the candidates and reject some

Examine unsuitable candidates, not just usable output. The Lab added two unsuitable candidates that a template could produce. It sent all seven candidates to a reviewer.

risky_variants <- tibble(
  source_sentence_id = c("s001", "s015"),
  source_family_id = c(
    "training-offer-s001",
    "training-offer-s015"
  ),
  template_id = c(
    "training-pathway-v1",
    "training-restatement-v1"
  ),
  vocabulary_source = c(
    "cmapr example_transitions.csv",
    "author-created template"
  ),
  vocabulary_row = c(6L, NA_integer_),
  vocabulary_commit = c(
    "eb2d0b3810e07782aab4a81fba1f539133f2ba6a",
    NA_character_
  ),
  text = c(
    "The financial analyst pathway may include paid training.",
    "On-the-job training is provided."
  ),
  label = "training",
  synthetic = TRUE,
  method = "controlled template",
  codebook_version = augmentation_codebook$version,
  codebook_hash = augmentation_codebook$hash,
  review_status = "requires_human_review",
  split = "training_candidate"
)

candidate_rows <- bind_rows(augmented_rows, risky_variants)

review_decisions <- tibble(
  text = candidate_rows$text,
  decision = c(
    rep("accept", 5L),
    "reject",
    "reject"
  ),
  reason = c(
    rep("States an offer without hedging; label holds", 3L),
    rep("States when schedules are available; label holds", 2L),
    "\"may include\" removes the explicit offer the label requires",
    "Repeats a reviewed sentence word for word"
  )
)

reviewed_candidates <- candidate_rows |>
  left_join(review_decisions, by = join_by(text)) |>
  mutate(
    duplicates_reviewed_text = text %in%
      workforce_sentences$text,
    hedged = str_detect(text, "\\b(may|might|could)\\b"),
    review_status = case_when(
      decision == "accept" ~ "accepted",
      decision == "reject" ~ "rejected",
      TRUE ~ review_status
    )
  )

accepted_rows <- reviewed_candidates |>
  filter(decision == "accept")
rejected_rows <- reviewed_candidates |>
  filter(decision == "reject")
Seven candidates, five accepted, two rejected
Candidate text Proposed label Decision Reason
The data analyst pathway includes paid training. training accept States an offer without hedging; label holds
The medical assistant pathway includes paid training. training accept States an offer without hedging; label holds
The junior developer pathway includes paid training. training accept States an offer without hedging; label holds
Morning schedules are available. schedule accept States when schedules are available; label holds
Weekend schedules are available. schedule accept States when schedules are available; label holds
The financial analyst pathway may include paid training. training reject “may include” removes the explicit offer the label requires
On-the-job training is provided. training reject Repeats a reviewed sentence word for word

Neither rejection follows from the template alone.

The hedged sentence is grammatical and plausible but does not meet its label definition. The codebook’s training label requires an explicit offer. The phrase “may include” states a possibility. Using that example for training would teach the model to treat uncertain language as an offer, contrary to the annotation rule.

The repeated sentence exactly matches a reviewed row and adds no information. If training used the copy and evaluation used the original, evaluation would measure recall of a training string.

Two rejections out of seven describe the rejection rate, not a defect in the checks. Record the checks for every candidate and the reasons for each rejection.

Ask whether the accepted rows changed anything

The five accepted rows pass the stated criteria. To measure their usefulness, run the downstream task with and without those rows.

The following simple classifier detects training offers. It uses terms present in training examples and absent from other categories. It learns from the five complete postings J001 through J005. Evaluation uses the ten units from J006 and the flyer. No augmented row derives from those evaluation sources.

labeled_pool <- workforce_sentences |>
  filter(document_id %in% sprintf("J%03d", 1:5)) |>
  select(text, label = reference_label)
held_out <- workforce_sentences |>
  filter(document_id %in% c("J006", "F001")) |>
  select(sentence_id, text, label = reference_label)
accepted_source_ids <- unique(accepted_rows$source_sentence_id)

to_words <- function(text) {
  text |>
    str_to_lower() |>
    str_replace_all("[^a-z0-9 ]", " ") |>
    str_squish() |>
    str_split(" ")
}

distinctive_terms <- function(examples) {
  training_words <- unique(unlist(
    to_words(examples$text[examples$label == "training"])
  ))
  other_words <- unique(unlist(
    to_words(examples$text[examples$label != "training"])
  ))

  sort(setdiff(
    training_words[str_length(training_words) > 2],
    other_words
  ))
}

predict_training <- function(terms, text) {
  map_lgl(
    to_words(text),
    \(row_words) any(row_words %in% terms)
  )
}

without_augmentation <- distinctive_terms(labeled_pool)
with_augmentation <- distinctive_terms(
  bind_rows(
    labeled_pool,
    accepted_rows |>
      select(text, label)
  )
)

is_training <- held_out$label == "training"
ablation <- tibble(
  condition = c("without augmentation", "with augmentation"),
  distinctive_terms = c(
    length(without_augmentation),
    length(with_augmentation)
  ),
  correct = c(
    sum(predict_training(without_augmentation, held_out$text) == is_training),
    sum(predict_training(with_augmentation, held_out$text) == is_training)
  ),
  items = nrow(held_out)
)
The same held-out units judged with and without the augmented rows
Condition Distinctive terms Correct decisions Items judged
without augmentation 13 10 10
with augmentation 18 10 10

setdiff() lists the terms that exist only in the augmented vocabulary:

new_terms <- setdiff(with_augmentation, without_augmentation)

new_terms
[1] "analyst"   "assistant" "developer" "junior"    "pathway"

The augmented rows added five vocabulary terms without changing any held-out decision. Both conditions produce all ten correct decisions.

Four new terms come from job titles, and the fifth is pathway. All five come from the template, not new observations of employer wording. The classifier already had the useful term training.

This comparison does not establish that augmentation is useless. Ten items with two positive cases cannot detect a small improvement. A stronger model with more data might use the additional rows differently. Here, five additional rows increased the training pool by more than a quarter without changing the result. Report that absence of improvement.

Prepare a model check without claiming a result

huggingfaceR can prepare the payload for a zero-shot classification request. This code records the candidate labels and texts, but it does not call a model or return predictions.

label_check_payload <-
  hf_zero_shot_classification_payload(
    augmented_rows$text,
    candidate_labels = c(
      "training",
      "requirement",
      "schedule",
      "skill",
      "other"
    ),
    multi_label = FALSE
  )

str(label_check_payload)
List of 2
 $ inputs    : chr [1:5] "The data analyst pathway includes paid training." "The medical assistant pathway includes paid training." "The junior developer pathway includes paid training." "Morning schedules are available." ...
 $ parameters:List of 2
  ..$ candidate_labels: chr [1:5] "training" "requirement" "schedule" "skill" ...
  ..$ multi_label     : logi FALSE

A future model call could select candidates for further checks. A model label would not prove valid augmentation. The same codebook and human checks still apply. An actual request would require a named model and provider. It would transmit the text under the provider’s terms. This code selects neither and makes no network request.

A small edit can reverse the claim

Word replacement is risky when it changes negation or obligation. str_to_lower() makes the keyword checks ignore case. str_detect() asks whether each pattern is present.

meaning_check <- tibble(
  version = c("source", "unsafe edit"),
  text = c(
    "Paid training is provided.",
    "Paid training is not provided."
  )
) |>
  mutate(
    lower_text = str_to_lower(text),
    keyword_only_match =
      str_detect(lower_text, fixed("paid")) &
        str_detect(lower_text, fixed("training")) &
        str_detect(lower_text, fixed("provided")),
    negation_aware_offer =
      keyword_only_match &
        !str_detect(lower_text, "\\bnot\\b")
  ) |>
  select(-lower_text)
One added word reverses the training offer
Version Text Keyword-only match Negation-aware offer rule
source Paid training is provided. TRUE TRUE
unsafe edit Paid training is not provided. TRUE FALSE

Both sentences satisfy the keyword-only check. The small negation rule separates this pair, but it is not a general language-understanding method. Label preservation still requires a check of meaning.

Augmentation can amplify a mistake

Examine:

  1. whether the label and important details remain correct.
  2. whether the wording is grammatical in the intended language.
  3. whether duplicates occur in both training and evaluation sets.
  4. whether templates make one class unnaturally easy to recognize.
  5. whether generated text adds stereotypes or private information.
  6. whether one language or job family receives most synthetic rows.
  7. how conclusions change when synthetic examples are removed.

Large language models can generate many variants, but quantity does not resolve these questions. Repeated training on generated text can reduce variation in a model’s representation of the underlying distribution.

What to remember

  • Augmented text is synthetic data, not a newly observed case.
  • Record the source, method, label, and synthetic status.
  • A familiar word does not guarantee that meaning stayed the same.
  • Examine every candidate. Keep rejected candidates with their rejection reasons.
  • Keep augmented rows out of evaluation data derived from their source.
  • Compare results with and without synthetic examples.
  • Report unchanged results as readily as improvements.

The Lab ends with five accepted rows, two rejected rows, and no observed downstream difference. The rows retain their synthetic status and generation method. The next lesson examines keyword rules on the designed data. It also examines cases with negation or misleading word matches.

Sources