Sorting messages by purpose

Intent prediction and label design

classification
intent classification
workforce research
Learn how intent classification routes messages and why the label schema controls what the system can do.

Luis manages the shared inbox while the front desk is closed. One message asks for a password reset. Another asks whether a class accepts GED applicants. A third reports a suspicious recruiter.

Routing the message is useful only if the label matches a real next step. A worker can reset access, read application status, answer eligibility questions, record a problem, or return a call. The label set does not describe actions outside that list.

Note

The Riverton Workforce Lab, its job board, and its training flyer are fictional teaching materials.

A dataset written by the classifier’s author cannot establish whether the method works generally. This lesson demonstrates calculations with the invented inbox. It does not give an estimate of intake-system performance for real residents.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • explain intent classification in plain language.
  • fit a small multi-class text model.
  • examine multi-class probability output.
  • find messages that need more than one routing label.
  • explain why coarser taxonomies can make accuracy look better.

Load the intent labels

An intent is what the sender wants the system to do. Each message in this fixed teaching dataset has exactly one recorded intent, even when a person might find several.

readr reads the file, while dplyr and tidyr reshape results. purrr selects probability columns, and stringr evaluates simple rules. tidymodels, textrecipes, and glmnet fit the model.

library(readr)
library(dplyr)
library(tibble)
library(tidyr)
library(purrr)
library(stringr)
library(ggplot2)
library(tidymodels)
library(textrecipes)
library(glmnet)

inbox <- read_csv(
  "data/riverton/riverton-inbox.csv",
  na = character(),
  col_types = cols(
    message_id = col_character(),
    text = col_character(),
    is_spam = col_logical(),
    intent = col_character(),
    author_note = col_character()
  )
) |>
  mutate(intent = factor(intent))

intent_counts <- inbox |>
  count(intent, name = "messages")
Intent labels in the invented inbox
Recorded intent Messages
ask_eligibility 12
check_status 13
report_problem 13
request_callback 11
reset_access 11

The label schema states the distinctions that the Lab can use. If staff cannot act on a distinction, adding its label creates disagreement without improving service.

Fit a multi-class model

Multi-class classification chooses one label from more than two possible labels. The seed before this fit is 5002. The model demonstrates the calculations for a multi-class prediction.

intent_recipe <- recipe(intent ~ text, data = inbox) |>
  step_tokenize(text) |>
  step_tokenfilter(text, max_tokens = 100) |>
  step_tfidf(text)

mechanics_rows <- inbox

intent_model <- multinom_reg(penalty = 0.01, mixture = 1) |>
  set_engine("glmnet") |>
  set_mode("classification")

set.seed(5002)
intent_fit <- workflow() |>
  add_recipe(intent_recipe) |>
  add_model(intent_model) |>
  # The lesson reports nothing performance-related from these rows; it shows the
  # shape of a multi-class fit and states plainly why it scores nothing.
  fit(data = mechanics_rows) # resubstitution-ok: no score is reported

intent_predictions <- mechanics_rows |>
  select(message_id, text, intent) |>
  bind_cols(predict(intent_fit, mechanics_rows, type = "prob")) |>
  bind_cols(predict(intent_fit, mechanics_rows))

probability_columns <- names(intent_predictions) |>
  keep(\(column) str_starts(column, ".pred_") & column != ".pred_class")

probability_check <- intent_predictions |>
  transmute(
    message_id,
    probability_sum = rowSums(pick(all_of(probability_columns)))
  )

probability_display <- intent_predictions |>
  slice_head(n = 3) |>
  select(message_id, all_of(probability_columns)) |>
  rename_with(
    \(column) str_remove(column, "^\\.pred_"),
    all_of(probability_columns)
  )

predicted_distribution <- intent_predictions |>
  count(.pred_class, name = "messages") |>
  arrange(.pred_class)
Five class probabilities for three fitted-row examples
Message Ask eligibility Read application status Report problem Request callback Reset access
M001 0.014 0.940 0.039 0.003 0.004
M002 0.061 0.881 0.007 0.022 0.029
M003 0.016 0.934 0.034 0.007 0.009
Predicted-class distribution from the fitted multi-class model
Predicted intent Messages
ask_eligibility 12
check_status 13
report_problem 13
request_callback 11
reset_access 11

The fitted model has five classes. It returns five probabilities per message, one per class, with a row sum of 1. The model predicts the same 60 rows it uses for fitting. The table therefore demonstrates output structure rather than performance. This lesson reports no performance score.

The classifier’s author also wrote all 60 messages. A score on those messages would describe this fixture rather than the method. Independent evaluation requires messages from other authors and labels from reviewers who did not build the model. It also requires a holdout set excluded from fitting.

Find gaps in the taxonomy

A taxonomy specifies the labels and the rules for choosing them. Label design can be harder than model fitting. If the taxonomy does not match the work, accurate label predictions can still send messages to the wrong service.

boundary_messages <- inbox |>
  filter(message_id %in% c("M034", "M055", "M044")) |>
  mutate(
    other_plausible_action = case_when(
      message_id == "M034" ~ "ask_eligibility or fraud review",
      message_id == "M055" ~ "ask_eligibility or fraud review",
      message_id == "M044" ~ "none without the missing request"
    ),
    why_it_strains_the_schema = case_when(
      message_id == "M034" ~
        "reports a suspicious listing but also mentions training and bank access",
      message_id == "M055" ~
        "asks whether an offer is real and reports a recruiter asking for ID",
      message_id == "M044" ~
        "gives call instructions without a full service request"
    )
  ) |>
  select(message_id, intent, other_plausible_action, why_it_strains_the_schema, text)
Boundary cases for the intent taxonomy
Message Recorded intent Other plausible action Why one label is hard Text
M034 report_problem ask_eligibility or fraud review reports a suspicious listing but also mentions training and bank access A listing says paid training, but the attachment asks for my bank password.
M044 request_callback none without the missing request gives call instructions without a full service request Please leave a voicemail if I miss the call.
M055 report_problem ask_eligibility or fraud review asks whether an offer is real and reports a recruiter asking for ID I am not sure whether this offer is real; the recruiter wants a photo of my ID.

Messages M034 and M055 genuinely fit two actions. A staff member may both record a problem and answer whether the offer is safe. If the Lab has a separate queue for fraud checks, these two messages should probably go there. If it does not, the worker must choose the nearest available label and record the information that the label omits.

M044 fits none of the main service questions by itself. It gives a call preference without the reason for the call. An intake system needs the missing purpose before it can route the message. Lesson 11, “Working with several annotators,” explains why multiple reviewers should examine boundary cases before choosing reference labels.

An other label covers messages that the designer did not anticipate. Staff must examine those messages to find new service needs.

Broader labels can increase measured accuracy

Combining intents into broader queues can increase the score for unchanged predictions. The scoring task becomes easier without better classification. This lesson does not compute that score. A rule fitted to these same 60 messages would provide another resubstitution result, a score calculated on its training examples.

Broader categories may be appropriate when the organization cannot act on finer distinctions. A report must not present the resulting increase in accuracy as better understanding.

What to remember

  • Intent classification routes text by what the sender wants.
  • The intent list must match actions the system can take.
  • Probability output shows the shape of a multi-class prediction, not its quality.
  • Boundary cases need human adjudication and clearer instructions.
  • Coarser categories can raise measured accuracy by making the task easier.

Sources