Training models

Fit a text model only after the example set is large enough

model development
training models
text classification
Learn why a 28-sentence pilot cannot evaluate a model and fit penalized logistic regression on inaugural paragraphs.

Maya’s folder of sentence labels helped the team agree on its annotation categories. Those categories cover training, scheduling, skills, requirements, and other workforce details.

The team now needs a predictor. Data that help people learn a codebook can still be too limited for model evaluation.

This lesson uses the Riverton set only for a pilot check. Model development uses the larger public corpus of presidential inaugural paragraphs bundled with quanteda.

Note

The Riverton Workforce Lab, its job board, and its training flyer are fictional teaching materials. The inaugural addresses are works of the United States government. This lesson treats them as public domain text packaged by quanteda.

TipWhat you will learn

This lesson shows how to:

  • explain when a labeled set is too small for a test of a model.
  • use a larger teaching corpus after the labeling pilot.
  • distinguish a recipe, model specification, workflow, fit, and prediction.
  • compare candidate models without using the final test set.
  • fit the selected penalized logistic regression model.
  • report a live score with a baseline and limits attached.

Stop at the pilot set

A training set is the labeled material a model learns from. A test set is labeled material reserved for scoring. In a small test set, one correct or incorrect prediction can greatly affect the score.

suppressPackageStartupMessages({
  library(readr)
  library(dplyr)
  library(tibble)
  library(rsample)
  library(recipes)
  library(textrecipes)
  library(parsnip)
  library(workflows)
  library(yardstick)
  library(glmnet)
})

riverton_sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

set.seed(4300)
riverton_split <- group_initial_split(
  riverton_sentences,
  group = document_id,
  prop = 0.75
)
riverton_training <- training(riverton_split)
riverton_testing <- testing(riverton_split)

riverton_label_counts <- riverton_sentences |>
  count(reference_label, name = "sentences") |>
  arrange(desc(sentences), reference_label)

split_size <- tibble(
  part = c("training", "test"),
  rows = c(nrow(riverton_training), nrow(riverton_testing))
)
The Riverton pilot labels are too few for a model score
Label Sentences
requirement 9
training 7
other 5
schedule 5
skill 2
Keeping documents whole leaves only four test sentences
Split part Rows
training 24
test 4

The split is deterministic because the seed is 4300, and it keeps all rows from a source document together. It leaves 4 test sentences from one document, containing 4 of the five labels. A single changed answer would move accuracy by about 25 percentage points. A test set this small cannot give a reliable estimate of prediction performance.

Use a larger corpus

The shared helper reads the larger corpus from quanteda::data_corpus_inaugural. Each row contains one paragraph of at least 25 words. The era outcome distinguishes addresses before 1900 from addresses in 1900 or later.

source("R/inaugural-corpus.R")

paragraphs <- inaugural_paragraphs()

corpus_summary <- tibble(
  measure = c("paragraphs", "speeches", "people", "surnames"),
  value = c(
    nrow(paragraphs),
    n_distinct(paragraphs$speech_id),
    n_distinct(paragraphs$president),
    n_distinct(paragraphs$surname)
  )
)

era_counts <- paragraphs |>
  group_by(era) |>
  summarise(
    paragraphs = n(),
    median_words = median(paragraph_words),
    .groups = "drop"
  )
The inaugural paragraph corpus used for model training
Measure Value
paragraphs 1377
speeches 60
people 40
surnames 36
Paragraphs by era
Era Paragraphs Median paragraph words
before 1900 485 112
1900 or later 892 54

The material has 1,377 paragraphs from 60 speeches by 40 people. The helper retains both the full name in president and the surname. Grouping by surname alone would merge different people with the same family name.

Paragraphs before 1900 have median length 112. Later paragraphs have median length 54.

This corpus does not represent all political speech or all historical writing. It gives the lesson enough held-out paragraphs to examine a specific claim about these speeches.

Keep the final test set out of model selection

The split keeps each speech whole. Paragraphs from one inaugural address cannot appear in both training and test sets. The seed is 4301 for the split.

The test speeches provide an estimate of performance after model selection. Selection based on test scores would reuse final evaluation evidence. This internal holdout comes from a corpus used throughout the section. It does not confirm performance on new text from another source.

Inside the training set, cross-validation makes several temporary train-and-check splits. Each temporary check is called a fold. The folds are often called CV folds. They are grouped by speech here, so every paragraph from one address stays together. Five folds put 9 or 10 speeches in each held-out set while leaving both era labels represented.

baselines <- read_csv(
  "data/inaugural/baselines.csv",
  na = character(),
  col_types = cols(
    task = col_character(),
    outcome = col_character(),
    rows = col_integer(),
    speeches = col_integer(),
    people = col_integer(),
    largest_class = col_character(),
    baseline_accuracy = col_double()
  )
)

model_comparison <- read_csv(
  "data/inaugural/model-comparison.csv",
  na = "",
  col_types = cols(
    rank = col_integer(),
    candidate = col_character(),
    family = col_character(),
    penalty = col_double(),
    mtry = col_integer(),
    min_n = col_integer(),
    mean_bal_accuracy = col_double(),
    std_err_bal_accuracy = col_double(),
    mean_accuracy = col_double(),
    std_err_accuracy = col_double(),
    selected = col_logical(),
    selection_reason = col_character()
  )
)

fold_summary <- read_csv(
  "data/inaugural/model-comparison-folds.csv",
  na = character(),
  col_types = cols(
    fold = col_character(),
    analysis_speeches = col_integer(),
    assessment_speeches = col_integer(),
    assessment_paragraphs = col_integer(),
    before_1900 = col_integer(),
    later_1900 = col_integer()
  )
)

set.seed(4301)
era_split <- group_initial_split(
  paragraphs,
  group = speech_id,
  prop = 0.75
)
era_training <- training(era_split)
era_testing <- testing(era_split)

set.seed(4310)
era_folds <- group_vfold_cv(
  era_training,
  group = speech_id,
  v = 5,
  strata = era,
  balance = "groups"
)

fold_group_overlap <- vapply(
  era_folds$splits,
  \(split) {
    length(intersect(
      unique(analysis(split)$speech_id),
      unique(assessment(split)$speech_id)
    ))
  },
  integer(1)
)
Five grouped folds inside the training speeches
Fold Training speeches Held-out speeches Held-out paragraphs Before 1900 1900 or later
Resample1 36 10 244 51 193
Resample2 37 9 254 55 199
Resample3 37 9 150 42 108
Resample4 37 9 228 90 138
Resample5 37 9 166 52 114

Stratification distributes early and later speeches across the folds. It does not equalize paragraph counts. The table shows that the paragraph-level class mix still varies, which is one reason balanced accuracy is the primary measure.

Compare a small set of candidates

This is not a search across every model R can fit. The comparison asks whether a different kind of decision rule changes the answer:

  • Ridge logistic regression retains every word feature but reduces the magnitudes of fitted weights toward zero.
  • Lasso logistic regression uses the same linear model but can set some word weights exactly to zero.
  • A random forest combines many branching rules and can learn relationships that are not linear.

Tuning parameters control how each candidate learns. The comparison uses four ridge penalties from 0.01 to 10 and four lasso penalties from 0.0001 to 0.1. Their fitted penalty paths use different scales. One shared grid would include duplicate or intercept-only fits without adding useful comparisons.

The forest uses six combinations of features per split (mtry) and minimum rows per branch (min_n). The builder in data-raw/build-model-comparison.R fits all 14 settings: four ridge, four lasso, and six forest. It never reads the final test speeches.

Balanced accuracy is the primary selection measure because it gives the two era classes equal weight. Raw accuracy appears beside it so the effect of the uneven class counts stays visible.

The table shows the two leading ridge settings and the leading lasso and forest settings. The ridge means differ, but their fold-to-fold standard errors overlap. Five related folds do not provide a confidence interval for other corpora.

comparison_display <- model_comparison |>
  filter(selected | rank %in% c(2L, 3L, 5L)) |>
  mutate(
    setting = case_when(
      candidate == "random forest" ~
        paste0("mtry ", mtry, "; min_n ", min_n),
      TRUE ~ paste0("penalty ", penalty)
    ),
    chosen = if_else(selected, "yes", "no")
  ) |>
  select(
    candidate,
    setting,
    mean_bal_accuracy,
    std_err_bal_accuracy,
    mean_accuracy,
    chosen
  )

selected_candidate <- model_comparison |>
  filter(selected)
Leading settings from 14 candidates compared on training folds
Candidate Setting Mean balanced accuracy Standard error Mean accuracy Chosen
ridge logistic penalty 0.01 0.8008 0.0266 0.8188 yes
ridge logistic penalty 0.1 0.7901 0.0343 0.8320 no
lasso logistic penalty 0.001 0.7809 0.0274 0.8013 no
random forest mtry 100; min_n 20 0.7761 0.0300 0.8437 no

Ridge with penalty 0.01 has the highest mean balanced accuracy, 0.8008. The fold-to-fold standard errors overlap those of the nearby candidates, so the table does not establish the best model for every corpus. The lesson retains ridge because it ranks first here and provides a linear model. Lesson 45 examines its coefficients.

The selected penalty is the lowest ridge value tried. Less shrinkage, or reduction in coefficient magnitudes, helped among these four settings. That does not establish 0.01 as the best possible penalty. A complete search would extend the grid before making that claim.

The random forest is a useful check rather than a required upgrade. Its best raw accuracy is competitive, but its lower balanced accuracy shows that more correct paragraphs can still mean less even performance across classes.

Build the selected workflow

A tidymodels pipeline has four parts:

  1. A recipe is a set of instructions that converts raw text into numeric model columns. Each step_*() line adds one instruction.
  2. A model specification names the model family and its settings before fitting.
  3. An engine names the package that performs the calculations. Here that package is glmnet.
  4. A workflow keeps the recipe and model specification together. Fitting uses training data to estimate model weights.

The recipe converts text into model columns. Tf-idf means term frequency times inverse document frequency. A word gets more weight in a paragraph when it is common there and rare elsewhere. The recipe first removes stop words such as the and of. It then retains the 500 most frequent remaining tokens.

The selected model is logistic regression, a linear classifier for categories. The penalty reduces coefficient magnitudes toward zero to limit the model’s response to incidental patterns in the training text.

text_recipe <- recipe(era ~ paragraph, data = era_training) |>
  step_tokenize(paragraph) |>
  step_stopwords(paragraph) |>
  step_tokenfilter(paragraph, max_tokens = 500) |>
  step_tfidf(paragraph)

text_model <- logistic_reg(penalty = 0.01, mixture = 0) |>
  set_engine("glmnet")

era_workflow <- workflow() |>
  add_recipe(text_recipe) |>
  add_model(text_model)

set.seed(4302)
era_fitted <- fit(era_workflow, data = era_training)

era_predictions <- predict(era_fitted, era_testing) |>
  bind_cols(era_testing |> select(era))

live_metrics <- metric_set(accuracy, bal_accuracy)(
  era_predictions,
  truth = era,
  estimate = .pred_class
)
live_accuracy <- live_metrics |>
  filter(.metric == "accuracy") |>
  pull(.estimate)
live_bal_accuracy <- live_metrics |>
  filter(.metric == "bal_accuracy") |>
  pull(.estimate)

corpus_baseline <- baselines |>
  filter(task == "era") |>
  pull(baseline_accuracy)

training_majority <- era_training |>
  count(era, sort = TRUE) |>
  slice(1) |>
  pull(era)

split_baseline_accuracy <- mean(era_testing$era == training_majority)

score_table <- tibble(
  measure = c(
    "Final test accuracy",
    "Final test balanced accuracy",
    "Same-split training-majority rule",
    "Corpus largest-class baseline"
  ),
  value = c(
    live_accuracy,
    live_bal_accuracy,
    split_baseline_accuracy,
    corpus_baseline
  )
)
One held-out test after candidate selection
Measure Value
Final test accuracy 0.8209
Final test balanced accuracy 0.8361
Same-split training-majority rule 0.4179
Corpus largest-class baseline 0.6478

Only after the candidate and penalty are fixed do we use the 14 final test speeches. The model scores 0.8209 accuracy and 0.8361 balanced accuracy. The majority label learned from training would score 0.4179 on this exact test set. That rate is below 0.5 because the training majority is 1900 or later, while this test split contains more before 1900 paragraphs.

Keeping speeches whole does not preserve the corpus class mix. The corpus-wide largest-class baseline is 0.6478, but the same-split comparison is the one a deployable rule could use.

What the fit did not prove

A fitted model contains a fixed recipe and learned weights. It gives labels to new rows with the same structure as the training rows. It does not establish that the label is appropriate or that the corpus represents other text. It also cannot guarantee performance under a harder split. The next lesson examines sensitivity to repeated splits.

What to remember

  • A pilot corpus can teach labeling without being large enough for a test of a model.
  • A grouped test set with four rows cannot measure much.
  • A speech-held-out split keeps one address from leaking across both sides.
  • Grouped folds compare candidates without using the final test set.
  • A recipe, model specification, engine, and workflow have different jobs.
  • Tf-idf gives more weight to words that are common in one paragraph and rare in the rest.
  • A model score needs a baseline beside it.

A trained classifier requires further evaluation. Its score must remain useful under a more demanding test.

Sources