Explaining models

Examine the words a fitted text model learned to weight

model development
explanation
text classification
Learn how to read linear model coefficients and why a plausible explanation can still belong to a weak model.

A reviewer asks why the model gave a paragraph the label 1900 or later. She needs to examine the words that affected its score.

A plausible explanation does not establish model quality. Even a weak model can produce an apparently convincing coefficient table.

This lesson fits the same model family for two tasks. Grouped folds in lesson 43 placed ridge logistic regression with penalty 0.01 among the leading candidates. The final test provided separate evaluation evidence. The era design has useful evaluation evidence, but the party design with these features and this penalty does not.

TipWhat you will learn

This lesson shows how to:

  • extract coefficients from a fitted workflow.
  • read positive and negative coefficients in plain language.
  • examine terms weighted by an era classifier.
  • mask terms from a weak political classifier.
  • keep explanation separate from evidence that a model is right.

Fit the era model for inspection

A coefficient is a learned weight. In this linear model, a positive coefficient increases the prediction score for the second class, 1900 or later. A negative coefficient increases the score for the first class, before 1900. Coefficient size describes the fitted weight on a tf-idf feature.

This direction depends on outcome order. The shared corpus helper places before 1900 first and 1900 or later second. The second class is the target of this binary glmnet fit.

Ranking coefficients by raw magnitude does not prove exactly what the model relied on. The features are correlated, and ridge regression spreads weight across related words. A different penalty could change the ranking. Read the list as heavily weighted terms in this fitted model, not as fixed meanings for historical periods.

glmnet standardizes predictors while fitting but returns coefficients on the original tf-idf scale. A rare term with little spread can therefore receive a larger raw coefficient than a common term with the same standardized effect. Multiplying each coefficient by its feature standard deviation would produce a different ranking.

The model below uses every paragraph in the corpus. Teams often want explanations for the final model they deploy after training on all available data. This full-corpus fit therefore has no evaluation score.

The evaluation lesson’s 0.8457 repeated-holdout mean describes different fits. Those models used three quarters of the speeches for training and the rest for evaluation. This page does not measure the performance of the full-corpus model. Similar-looking coefficients could occur in either a strong or weak fit.

The folds selected the model family and penalty before this refit. Refitting on all rows changes the exact coefficients, so the table explains the refitted model rather than reproducing the coefficients from any one fold.

suppressPackageStartupMessages({
  library(readr)
  library(dplyr)
  library(tibble)
  library(stringr)
  library(recipes)
  library(textrecipes)
  library(parsnip)
  library(workflows)
  library(broom)
  library(glmnet)
})

source("R/inaugural-corpus.R")
paragraphs <- inaugural_paragraphs()

make_text_workflow <- function(formula, training_data) {
  recipe(formula, data = training_data) |>
    step_tokenize(paragraph) |>
    step_stopwords(paragraph) |>
    step_tokenfilter(paragraph, max_tokens = 500) |>
    step_tfidf(paragraph) |>
    workflow() |>
    add_model(
      logistic_reg(penalty = 0.01, mixture = 0) |>
        set_engine("glmnet")
    )
}

set.seed(4500)
era_fitted <- make_text_workflow(era ~ paragraph, paragraphs) |>
  fit(data = paragraphs)

era_coefficients <- tidy(extract_fit_parsnip(era_fitted)) |>
  mutate(
    word = str_remove(term, "^tfidf_paragraph_"),
    direction = if_else(estimate >= 0, "1900 or later", "before 1900")
  )

check_words <- c("attention", "object", "effect", "placed", "parts", "opinion")
checked_coefficients <- era_coefficients |>
  filter(word %in% check_words) |>
  arrange(word)
Selected large era-model coefficients
Term Coefficient Class with increased score
attention -9.85 before 1900
effect -7.50 before 1900
object -12.82 before 1900
opinion -10.39 before 1900
parts -9.93 before 1900
placed -15.30 before 1900

The fitted era model has 501 terms. This count includes the intercept, the constant term. The listed words increase the score for before 1900 in this fit. They are formal-register signals in this corpus, not summaries of what any period was about.

Read both directions

The largest positive coefficients increase the score for 1900 or later. The largest negative coefficients increase the score for before 1900. This table describes fitted weights rather than word meanings.

era_extremes <- bind_rows(
  era_coefficients |>
    filter(term != "(Intercept)") |>
    slice_max(estimate, n = 8) |>
    arrange(desc(estimate)),
  era_coefficients |>
    filter(term != "(Intercept)") |>
    slice_min(estimate, n = 8) |>
    arrange(estimate)
) |>
  mutate(estimate = round(estimate, 2)) |>
  select(word, direction, estimate)

sensitive_terms <- c("slavery", "slave", "race", "war")
Strongest positive and negative coefficients in the era model
Term Class with increased score Coefficient
conditions 1900 or later 8.30
created 1900 or later 8.22
cost 1900 or later 7.64
mere 1900 or later 7.63
action 1900 or later 7.19
america 1900 or later 7.17
economic 1900 or later 7.17
way 1900 or later 7.13
intercourse before 1900 -16.01
placed before 1900 -15.30
object before 1900 -12.82
debt before 1900 -12.25
limits before 1900 -11.20
condition before 1900 -10.61
blessings before 1900 -10.45
constitution before 1900 -10.39

With all other features unchanged, a larger tf-idf value on a positive term increases the score for 1900 or later. A larger value on a negative term increases the score for before 1900. The linear model adds these weighted contributions.

Explain the weak model too

The party model did not improve on the deployable largest-class rule at the paragraph or speech split. Its balanced accuracy was near chance. The same extraction code still produces coefficients. A clear coefficient table therefore does not establish that a model is useful.

study <- read_csv(
  "data/inaugural/split-study.csv",
  na = c("", "NA"),
  col_types = cols(
    task = col_character(),
    split_scheme = col_character(),
    replicate = col_integer(),
    accuracy = col_double(),
    bal_accuracy = col_double(),
    train_majority_accuracy = col_double(),
    test_majority_rate = col_double(),
    length_rule_accuracy = col_double(),
    test_rows = col_integer(),
    test_speeches = col_integer()
  )
)

party_rows <- paragraphs |>
  filter(party %in% c("Democratic", "Republican")) |>
  mutate(party = factor(party))

set.seed(4503)
party_fitted <- make_text_workflow(party ~ paragraph, party_rows) |>
  fit(data = party_rows)

party_coefficients <- tidy(extract_fit_parsnip(party_fitted)) |>
  mutate(
    word = str_remove(term, "^tfidf_paragraph_"),
    direction = if_else(estimate >= 0, levels(party_rows$party)[2], levels(party_rows$party)[1])
  )

party_extremes <- party_coefficients |>
  filter(term != "(Intercept)") |>
  slice_max(abs(estimate), n = 10) |>
  arrange(desc(abs(estimate))) |>
  mutate(
    term = paste("term", row_number()),
    estimate = round(estimate, 2)
  ) |>
  select(term, direction, estimate)

party_summary <- study |>
  filter(task == "party") |>
  summarise(
    mean_accuracy = mean(accuracy),
    mean_bal_accuracy = mean(bal_accuracy),
    mean_train_majority = mean(train_majority_accuracy),
    .by = split_scheme
  )

party_paragraph <- party_summary |>
  filter(split_scheme == "paragraph")
party_speech <- party_summary |>
  filter(split_scheme == "speech")
party_president <- party_summary |>
  filter(split_scheme == "president")
Large coefficients from the party model that did not pass evaluation
Masked term Class with increased score Coefficient
term 1 Democratic -13.27
term 2 Democratic -12.71
term 3 Democratic -11.05
term 4 Republican 10.26
term 5 Republican 9.79
term 6 Republican 9.72
term 7 Republican 9.65
term 8 Republican 9.63
term 9 Republican 9.50
term 10 Republican 9.44

The lesson deliberately withholds party terms. The model scores 0.5514 at the speech split against a deployable baseline of 0.6563. Its person-split balanced accuracy is 0.509.

Displaying the terms could encourage political interpretations that the model’s evaluation does not justify. Masking preserves the teaching point. The table can look convincing even when the model performs poorly.

What to remember

  • A coefficient is a learned weight that affects the score for a class.
  • Coefficients show fitted weights after preprocessing, not proof of causation.
  • A coefficient is not proof that the model is right.
  • The era explanation matters because the era model performed better than simple baselines.
  • The party explanation is a warning because this model did not learn party.

If the score fails, the explanation should make you more cautious, not more comfortable.

Sources