library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
all_sentences <- read_csv(
"data/workforce/workforce_sentences.csv",
col_types = cols(
sentence_id = col_character(),
document_id = col_character(),
source_line = col_character(),
text = col_character(),
reference_label = col_character(),
uncertainty = col_character(),
annotator_id = col_character(),
rationale = col_character(),
codebook_version = col_character(),
codebook_hash = col_character(),
derived = col_character(),
transformation = col_character()
)
)
annotation_view <- all_sentences |>
select(
sentence_id,
document_id,
text,
reference_label
)
initial_labels <- annotation_view |>
filter(document_id %in% sprintf("J%03d", 1:5))
annotation_pool <- annotation_view |>
filter(document_id %in% c("J006", "F001")) |>
select(sentence_id, document_id, text)
held_back_labels <- annotation_view |>
filter(document_id %in% c("J006", "F001")) |>
select(sentence_id, reference_label)Choosing which examples to label next
Use active learning without confusing uncertainty with truth
training data
active learning
workforce research
Learn how a small model can rank unlabeled workforce text for people to label.
A finished codebook covers training, requirements, schedules, skills, and other workforce text. Five complete job postings form the labeled starting set. A sixth posting and the flyer form an unlabeled pool. The team could read the pool in file order. Instead, it examines whether a small model can find cases that its current evidence does not distinguish clearly.
Active learning uses successive model updates to help select items for human labeling. The model does not supply the answer. It selects examples under a rule, and a person applies the codebook. Here, the rule measures uncertainty about training, not a sentence’s importance to a worker.
Note
The sentences and research team are fictional. The deliberately small model makes the selection logic visible. It is not suitable for workforce decisions.
TipWhat you will learn
By the end of this lesson, you will be able to:
- separate an initial labeled set from an annotation pool.
- explain uncertainty sampling.
- rank items by distance from a 50% model score.
- reveal labels only after selection.
- show that a selected subset can increase or decrease an estimate.
- name the biases that active learning can introduce.
Hide the answers in the annotation pool
The teaching file contains reviewed labels for comparison with results. During selection, the model receives labels from documents J001 through J005. The complete J006 posting and F001 flyer form the annotation pool. Keeping documents intact prevents training and selection from using different parts of the same posting.
The code uses readr for files and dplyr and tibble for tables. It uses purrr for successive checks and stringr for text matching.
| Sentence ID | Document ID | Text |
|---|---|---|
| s019 | J006 | Two years of customer service experience are required. |
| s020 | J006 | Weekend shifts are required. |
| s021 | J006 | Product training is included. |
| s022 | J006 | Clear written communication is an essential skill. |
| s023 | F001 | RIVERTON SKILLS OPEN HOUSE |
| s024 | F001 | DATA SUPPORT CERTIFICATE |
| s025 | F001 | Paid training stipend |
| s026 | F001 | Evening classes |
| s027 | F001 | No prior experience required |
| s028 | F001 | Apply by October 15 |
The code stores pool labels separately and excludes them from selection. The selection calculation therefore cannot use answers unavailable in a real annotation round.
Fit a small keyword model
The example uses a Bernoulli naive Bayes model. It records whether selected terms appear, then compares how often those terms occur in training and non-training sentences. Half-count smoothing prevents an observed term from receiving a conditional probability of exactly zero or one.
A reader can examine the model’s calculations, but its assumptions are strong. It treats the selected terms as conditionally independent and ignores word order, context, and most of the sentence.
The candidate terms come from the codebook and labeled examples, not the held-back pool. The code removes candidates absent from the labeled set. The shorthand \(keyword) defines a small function with one input named keyword.
Without that removal, half-count smoothing could make an unseen term appear more common in the smaller class. The difference would reflect only the smaller denominator.
candidate_keywords <- c(
"training",
"certificate",
"apprenticeship",
"mentor",
"stipend",
"classes",
"required",
"preferred",
"license",
"diploma",
"portfolio",
"shift",
"schedule",
"skill",
"spreadsheet",
"communication",
"experience"
)
make_features <- function(text, terms) {
normalized_text <- str_to_lower(text)
feature_columns <- terms |>
map(\(keyword) {
str_detect(
normalized_text,
sprintf("\\b%s\\w*\\b", keyword)
)
})
matrix(
unlist(feature_columns, use.names = FALSE),
nrow = length(text),
ncol = length(terms),
dimnames = list(NULL, terms)
)
}
candidate_features <- make_features(
initial_labels$text,
candidate_keywords
)
keywords <- candidate_keywords[
colSums(candidate_features) > 0L
]
training_features <- candidate_features[
,
keywords,
drop = FALSE
]
is_training <- initial_labels$reference_label == "training"
smoothing <- 0.5
training_prior <- (
sum(is_training) + smoothing
) / (
length(is_training) + 2 * smoothing
)
training_term_count <- colSums(
training_features[is_training, , drop = FALSE]
)
other_term_count <- colSums(
training_features[!is_training, , drop = FALSE]
)
empirical_rate_training <- training_term_count /
sum(is_training)
empirical_rate_other <- other_term_count /
sum(!is_training)
term_rate_training <- (
training_term_count + smoothing
) / (
sum(is_training) + 2 * smoothing
)
term_rate_other <- (
other_term_count + smoothing
) / (
sum(!is_training) + 2 * smoothing
)
direction_is_preserved <- sign(
term_rate_training - term_rate_other
) == sign(
empirical_rate_training -
empirical_rate_other
)
score_training <- function(text) {
features <- make_features(text, keywords)
log_odds <- log(
training_prior / (1 - training_prior)
) +
rowSums(
sweep(
features,
2,
log(term_rate_training / term_rate_other),
"*"
) +
sweep(
!features,
2,
log(
(1 - term_rate_training) /
(1 - term_rate_other)
),
"*"
)
)
stats::plogis(log_odds)
}
training_scores <- score_training(annotation_pool$text)| Term | Training rows | Other rows | Smoothed training rate | Smoothed other rate |
|---|---|---|---|---|
| training | 3 | 0 | 0.583 | 0.036 |
| certificate | 0 | 1 | 0.083 | 0.107 |
| apprenticeship | 1 | 0 | 0.250 | 0.036 |
| mentor | 1 | 0 | 0.250 | 0.036 |
| required | 0 | 4 | 0.083 | 0.321 |
| preferred | 0 | 2 | 0.083 | 0.179 |
| license | 0 | 1 | 0.083 | 0.107 |
| diploma | 0 | 1 | 0.083 | 0.107 |
| portfolio | 0 | 1 | 0.083 | 0.107 |
| shift | 0 | 1 | 0.083 | 0.107 |
| schedule | 0 | 2 | 0.083 | 0.179 |
| skill | 0 | 1 | 0.083 | 0.107 |
| spreadsheet | 0 | 1 | 0.083 | 0.107 |
| experience | 0 | 2 | 0.083 | 0.179 |
Half-count smoothing prevents probabilities of zero and one. The assertion requires each smoothed difference to have the same sign as the observed difference. A term absent from training rows cannot become positive training evidence merely because that class is smaller.
The score used next is the model’s estimate under this small dataset and model. It is not a validated probability that the label is correct.
Select the closest scores
Uncertainty sampling often selects scores closest to 0.5 in a binary task. The model is least decisive there. We calculate each score’s distance from 0.5 and choose the two smallest distances. The model vocabulary contains only terms observed in the labeled documents. Words found only in the pool cannot acquire artificial evidence through smoothing.
selection_table <- tibble(
sentence_id = annotation_pool$sentence_id,
text = annotation_pool$text,
training_score = training_scores,
distance_from_half = abs(
training_scores - 0.5
)
) |>
arrange(distance_from_half, sentence_id)
selected_for_review <- selection_table |>
slice_head(n = 2)| Sentence ID | Text | Training score | Distance from 0.5 |
|---|---|---|---|
| s026 | Evening classes | 0.194 | 0.306 |
| s028 | Apply by October 15 | 0.194 | 0.306 |
| s022 | Clear written communication is an essential skill. | 0.154 | 0.346 |
| s023 | RIVERTON SKILLS OPEN HOUSE | 0.154 | 0.346 |
| s024 | DATA SUPPORT CERTIFICATE | 0.154 | 0.346 |
| s021 | Product training is included. | 0.901 | 0.401 |
| s025 | Paid training stipend | 0.901 | 0.401 |
| s020 | Weekend shifts are required. | 0.034 | 0.466 |
| s019 | Two years of customer service experience are required. | 0.019 | 0.481 |
| s027 | No prior experience required | 0.019 | 0.481 |
The first two rows are the next annotation candidates. Neither contains a feature from the labeled set, so both receive the same score. This exposes a limit of uncertainty-only selection. A score can reflect missing vocabulary rather than a difficult distinction between labels.
People still assign the labels
Only after selection do we join the held-back reference labels. This simulates the annotator returning completed work.
reviewed_selection <- selected_for_review |>
select(sentence_id, text, training_score) |>
left_join(
held_back_labels,
by = join_by(sentence_id),
multiple = "error"
)
newly_labeled <- selected_for_review |>
select(sentence_id) |>
left_join(
annotation_pool,
by = join_by(sentence_id),
multiple = "error"
) |>
left_join(
held_back_labels,
by = join_by(sentence_id),
multiple = "error"
)
updated_labels <- bind_rows(
initial_labels,
newly_labeled
)| Sentence ID | Text | Training score | Human-applied label |
|---|---|---|---|
| s026 | Evening classes | 0.194 | schedule |
| s028 | Apply by October 15 | 0.194 | other |
The two sentences receive schedule and other. Human labeling adds distinctions that the binary training detector cannot express. The tie shows why a practical selection policy may combine uncertainty with diversity and explicit coverage goals.
Uncertainty is only one selection rule
A model can be confidently wrong. Selecting only uncertain examples may also overlook:
- communities or languages absent from the initial labels.
- rare cases with incorrect but confident model scores.
- duplicate examples that consume the annotation budget.
- important harms that occur infrequently.
- examples far from the model’s training distribution.
A stronger plan combines uncertainty with random audit samples, diversity, coverage targets, and deliberate tests of known failure modes. Keep a document-disjoint evaluation set that active learning never uses for feature design, selection, or training.
Selection changes the estimate, and not in a fixed direction
Model-selected examples do not necessarily represent the whole pool. Here, selection bias means systematic error in a pool-level estimate caused by the item-selection process. A measurement on those examples therefore describes the selected subset, not the pool. Selection does not always increase measured error. The direction depends on which examples the rule selects.
The comparison measures the gap between each model score and the human label. It compares three sets: the two uncertainty-selected items, the two most decisive predictions, and the whole pool.
pool_outcomes <- annotation_pool |>
left_join(
held_back_labels,
by = join_by(sentence_id),
multiple = "error"
) |>
mutate(
training_score = training_scores,
is_training = as.integer(reference_label == "training"),
score_gap = abs(training_score - is_training),
predicted_training = training_score > 0.5
)
most_decisive <- selection_table |>
arrange(desc(distance_from_half), sentence_id) |>
slice_head(n = 2)
gap_for <- function(ids) {
mean(
pool_outcomes$score_gap[pool_outcomes$sentence_id %in% ids]
)
}
selection_comparison <- tibble(
evaluated_on = c(
"two items the uncertainty rule chose",
"two items the model was most decisive about",
"all ten pool items"
),
items = c(2L, 2L, nrow(pool_outcomes)),
mean_score_gap = c(
gap_for(selected_for_review$sentence_id),
gap_for(most_decisive$sentence_id),
gap_for(pool_outcomes$sentence_id)
)
)
pool_accuracy <- mean(
pool_outcomes$predicted_training ==
(pool_outcomes$is_training == 1L)
)| Measured on | Items | Mean gap between score and label |
|---|---|---|
| two items the uncertainty rule chose | 2 | 0.194 |
| two items the model was most decisive about | 2 | 0.019 |
| all ten pool items | 10 | 0.112 |
The pool value is 0.112. Uncertainty selection increased the measured gap to 0.194. Confidence selection reduced it to 0.019. The model, ten items, and quantity remained the same. Only the selection rule changed, producing opposite changes in the estimate.
Selection can increase or decrease measured error. Selecting difficult cases can reduce a score, while selecting easy cases can increase it. Selecting a class that the model handles well can make a weak model appear stronger. Each estimate describes the selected subset.
Keep a fixed, randomly selected evaluation set. Exclude its documents from active-learning feature design, selection, and training. Use that set for pool-level estimates.
This classifier places all ten pool items on the correct side of 0.5. That result has strict limits. The exercise uses ten items, vocabulary from the labeled documents, and no independent test set.
What to remember
- Active learning selects items for human labeling. It does not replace people.
- Hide pool labels until after selection.
- A score near 0.5 is uncertain only under the current model.
- Confidence is not the same as correctness or social importance.
- A measurement on selected items can exceed or be less than the pool value.
- Audit random, rare, and underrepresented cases alongside uncertainty.
The selection record accounts for two new human decisions but does not expand the fictional notices’ coverage. Selection cannot repair source coverage. Before combining records, the Lab examines three outside resources for information relevant to the same questions.