Labeling text with written rules

Measure rule coverage, conflicts, and mistakes

training data
weak supervision
workforce research
Learn how rule-based weak labels can expand workforce training data without becoming ground truth.

Five fictional sentence variants remain marked as synthetic. For the rule test, the Riverton Workforce Lab returns to its 28 reviewed fictional text units. Several phrases appear repeatedly: training, required, shift, and skill. The team writes rules from those patterns, then examines whether familiar words preserve distinctions a worker may use when comparing notices.

Rule-based training data uses written conditions to assign labels. These proposed labels are often called weak labels because a rule is an imperfect source of supervision. The team will measure where its rules label, abstain, conflict, and disagree with human judgments.

Note

The sentences and team are fictional. The author designed the rules after examining this teaching dataset. Results on these rows are not an independent performance estimate.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • express labeling rules as text patterns.
  • preserve individual votes and conflicts.
  • calculate coverage, conflict, and abstention.
  • report each rule’s own coverage and error.
  • compare usable weak labels with reference labels.
  • run rule checks on cases outside their design examples.

Write rules that can abstain

Each rule searches for words associated with one codebook category. A rule returns TRUE when it votes for its category and FALSE otherwise. The next chunk loads file, table, reshaping, repetition, string, and agreement helpers. It then reads the file with declared column types. map() makes one TRUE/FALSE list per rule. as_tibble() converts those lists into columns.

library(readr)
library(dplyr)
library(tibble)
library(tidyr)
library(purrr)
library(stringr)
library(foundryR)

workforce_sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

rule_patterns <- c(
  training = str_c(
    "training",
    "certificat",
    "apprenticeship",
    "mentor",
    sep = "|"
  ),
  requirement = str_c(
    "required",
    "preferred",
    "license",
    "diploma",
    "portfolio",
    "must be able",
    sep = "|"
  ),
  schedule = "schedule|shift|classes",
  skill = "skill|spreadsheet|communication"
)
rule_labels <- names(rule_patterns)
sentence_text_lower <- str_to_lower(workforce_sentences$text)

rule_votes <- bind_cols(
  tibble(sentence_id = workforce_sentences$sentence_id),
  as_tibble(
    map(rule_patterns, \(pattern) {
      str_detect(sentence_text_lower, pattern)
    })
  )
)

head(rule_votes, 8)
# A tibble: 8 × 5
  sentence_id training requirement schedule skill
  <chr>       <lgl>    <lgl>       <lgl>    <lgl>
1 s001        TRUE     FALSE       FALSE    FALSE
2 s002        FALSE    TRUE        FALSE    FALSE
3 s003        FALSE    FALSE       TRUE     FALSE
4 s004        FALSE    FALSE       FALSE    TRUE 
5 s005        FALSE    TRUE        FALSE    FALSE
6 s006        TRUE     FALSE       FALSE    FALSE
7 s007        FALSE    FALSE       TRUE     FALSE
8 s008        FALSE    TRUE        FALSE    FALSE

The rule-vote tibble preserves one column per rule. Combining the votes too early would hide why a label was proposed.

Mark abstentions and conflicts

A row can receive:

  • one vote, which produces a usable weak label.
  • no votes, which produces an abstention.
  • several votes, which produces a conflict.
rule_counts <- rule_votes |>
  mutate(
    rule_count = as.integer(rowSums(rule_votes[rule_labels]))
  ) |>
  select(sentence_id, rule_count)
voted_labels <- rule_votes |>
  pivot_longer(
    cols = all_of(rule_labels),
    names_to = "weak_label",
    values_to = "rule_vote"
  ) |>
  filter(rule_vote) |>
  summarise(
    voted_label = first(weak_label),
    .by = sentence_id
  )

weak_results <- workforce_sentences |>
  select(sentence_id, text, reference_label) |>
  left_join(rule_counts, by = join_by(sentence_id)) |>
  left_join(voted_labels, by = join_by(sentence_id)) |>
  mutate(
    weak_label = case_when(
      rule_count == 0L ~ "abstain",
      rule_count > 1L ~ "conflict",
      TRUE ~ voted_label
    )
  ) |>
  select(-voted_label)

result_order <- c(
  "abstain",
  "conflict",
  "requirement",
  "schedule",
  "skill",
  "training"
)
result_counts <- tibble(weak_label = result_order) |>
  left_join(
    weak_results |>
      count(weak_label, name = "rows"),
    by = join_by(weak_label)
  ) |>
  mutate(rows = coalesce(rows, 0L))
Five text units with no rule vote or conflicting votes
Sentence ID Text Reference label Rule votes Weak-label result
s011 Remote work is available two days each week. other 0 abstain
s013 A medical records certificate is preferred. requirement 2 conflict
s018 The work is outdoors and includes local travel. other 0 abstain
s020 Weekend shifts are required. schedule 2 conflict
s028 Apply by October 15 other 0 abstain

Three sentences receive no vote. Two receive conflicting votes. Sentence s013, for example, contains both certificate and preferred, so the training and requirement rules disagree.

Report coverage before accuracy

Coverage is the share of rows receiving at least one vote. Conflict rate is the share receiving more than one. Usable coverage counts rows with exactly one vote.

rule_summary <- tibble(
  measure = c(
    "coverage",
    "conflict rate",
    "usable coverage"
  ),
  value = c(
    mean(weak_results$rule_count > 0L),
    mean(weak_results$rule_count > 1L),
    mean(weak_results$rule_count == 1L)
  )
)
Coverage and conflict in the teaching dataset
Measure Share of 28 text units
coverage 0.893
conflict rate 0.071
usable coverage 0.821

Coverage alone does not measure correctness. A rule can label every row and be wrong every time.

Compare usable labels with human judgments

The 23 rows with one rule vote include two errors. We can summarize that comparison with foundryR.

agreement_data <- weak_results |>
  filter(rule_count == 1L) |>
  select(weak_label, reference_label)
error_sentences <- weak_results |>
  filter(
    rule_count == 1L,
    weak_label != reference_label
  )
error_sentence_ids <- error_sentences$sentence_id
rule_agreement <- foundryR::foundry_agreement(
  agreement_data,
  estimate = "weak_label",
  truth = "reference_label"
)
In-sample agreement for usable weak labels
Measure Value Compared text units
accuracy 0.9130435 23
precision_macro 0.8854167 23
recall_macro 0.8000000 23
f1_macro 0.7466667 23
cohen_kappa 0.8811370 23
krippendorff_alpha 0.8832685 23

The rules label 21 of the 23 usable rows the same way as the reference annotation. They incorrectly label RIVERTON SKILLS OPEN HOUSE as skill and DATA SUPPORT CERTIFICATE as training. The trigger words are present, but the headings alone do not make either claim.

The full accounting keeps that error beside unresolved rows.

all_row_accounting <- tibble(
  outcome = c(
    "usable and matches reference",
    "usable and differs from reference",
    "unresolved: abstain or conflict"
  ),
  rows = c(
    weak_results |>
      filter(
        rule_count == 1L,
        weak_label == reference_label
      ) |>
      nrow(),
    weak_results |>
      filter(
        rule_count == 1L,
        weak_label != reference_label
      ) |>
      nrow(),
    weak_results |>
      filter(rule_count != 1L) |>
      nrow()
  )
)
Accounting for all 28 rule results
Outcome Text units
usable and matches reference 21
usable and differs from reference 2
unresolved: abstain or conflict 5

These are in-sample results. The author wrote the rules after reading the sentences. The rules may therefore encode those examples rather than the broader concept.

Examine each rule separately

The 21-of-23 figure for all rules conceals differences between individual rules. Each rule has separate coverage and error counts. Errors can differ between rows with one matching rule and rows with several matching rules.

reference_labels <- workforce_sentences$reference_label
votes_per_row <- as.integer(
  rowSums(rule_votes[rule_labels])
)

rule_diagnostics <- map(
  rule_labels,
  \(rule_name) {
    fired <- rule_votes[[rule_name]]
    fired_alone <- fired & votes_per_row == 1L
    matches_reference <- reference_labels == rule_name

    tibble(
      rule = rule_name,
      rows_fired = sum(fired),
      rows_fired_alone = sum(fired_alone),
      alone_and_correct = sum(fired_alone & matches_reference),
      correct_whenever_fired = sum(fired & matches_reference)
    )
  }
) |>
  list_rbind() |>
  mutate(
    precision_when_alone = alone_and_correct / rows_fired_alone
  )
Each rule’s coverage and error, separated
Rule Matching rows Rows matching only this rule Correct labels when only this rule matched Precision when only this rule matched
training 9 8 7 0.88
requirement 10 8 8 1.00
schedule 5 4 4 1.00
skill 3 3 2 0.67

The requirement and schedule rules correctly labeled every row where each was the only matching rule. The training rule made one error in eight such rows. The skill rule matched only three rows and made one error. Precision of two thirds on three rows indicates a problem, not established performance on new data.

In-sample results with few matches provide little evidence of performance on new data. Such results cannot show whether absent cases would cause errors.

Separate conflicts from correlated same-label votes

Several rules can match one row with conflicting labels or repeated evidence. Those cases have different consequences. The following example contains opposing-label conflicts. It does not give an estimate of correlation or demonstrate confidence inflation.

vote_matrix <- as.matrix(rule_votes[rule_labels])
rule_pairs <- expand.grid(
  first = rule_labels,
  second = rule_labels,
  stringsAsFactors = FALSE
) |>
  as_tibble() |>
  filter(first < second) |>
  mutate(
    fired_together = map2_int(
      first,
      second,
      \(left, right) {
        sum(vote_matrix[, left] & vote_matrix[, right])
      }
    )
  ) |>
  arrange(desc(fired_together), first, second)
Rows with matches from each rule pair
Rule Other rule Rows matching both rules
requirement schedule 1
requirement training 1
requirement skill 0
schedule skill 0
schedule training 0
skill training 0

Only two rule pairs match the same row, producing the two observed conflicts. In s013, certificate matches the training pattern and preferred matches the requirement pattern. In s020, the required shift matches both requirement and schedule patterns.

These overlaps are conflicts, not duplicate votes for one label. Correlated same-label votes occur when two rules tend to propose one label from shared evidence. Two training rules with nearly identical keyword lists would provide an example. A label model can give estimates of rule accuracy and dependence under stronger data and modeling assumptions. This table does not supply that evidence.

This lesson does not fit a label model. Four rules, 28 rows, and two conflicting overlaps provide insufficient information for accuracy and correlation estimates. The lesson measures rule agreement, conflict, and abstention only.

Examine a conflict policy

A team that needs a label for every row usually writes a priority order. The following example applies one such order.

priority_order <- c(
  "schedule",
  "requirement",
  "training",
  "skill"
)

resolve_by_priority <- function(row_votes) {
  chosen <- priority_order[priority_order %in% rule_labels[row_votes]]

  if (length(chosen) == 0L) {
    NA_character_
  } else {
    chosen[1]
  }
}

conflict_rows <- which(votes_per_row > 1L)
conflict_outcomes <- tibble(
  sentence_id = rule_votes$sentence_id[conflict_rows],
  reference_label = reference_labels[conflict_rows],
  resolved_label = map_chr(
    conflict_rows,
    \(row) resolve_by_priority(vote_matrix[row, ])
  )
) |>
  mutate(policy_agrees = resolved_label == reference_label)

abstain_rows <- which(votes_per_row == 0L)
abstain_outcomes <- tibble(
  sentence_id = rule_votes$sentence_id[abstain_rows],
  reference_label = reference_labels[abstain_rows]
)
What a priority order does with the two conflicts
Sentence ID Reference label Priority policy label Policy agrees
s013 requirement requirement TRUE
s020 schedule schedule TRUE
Reference labels for the three rows with no rule vote
Sentence ID Reference label
s011 other
s018 other
s028 other

The policy resolves both conflicts correctly, but this provides little performance evidence. The author chose the order after reading the codebook’s timing-priority rule. Evaluation used only two rows. Those two successes do not establish performance on new data.

All three rows with complete abstention have the reference label other. No rule targets that category, so abstention is appropriate. A separate keyword rule for other would be less appropriate. The codebook assigns other in the absence of the four other categories, not by particular words.

The accounting includes all 28 rows. Twenty-one usable labels match the reference, and two differ. A stated policy resolves two conflicts. Three abstentions have reference labels outside the rule set. No rows remain unreported.

Examine cases outside the rule-design examples

Before another batch, the team writes four fictional adversarial unit tests. The team chooses expected labels after rule design to expose known rule limitations. These cases are not an independent evaluation sample.

stress_cases <- tibble(
  case_id = sprintf("challenge-%d", 1:4),
  text = c(
    "The training department is hiring an accountant.",
    "Employees learn from experienced coworkers.",
    "Certification is not required.",
    "Flexible hours are available."
  ),
  reference_label = c(
    "other",
    "training",
    "requirement",
    "schedule"
  ),
  reference_status = "designed challenge expectation"
)

stress_text_lower <- str_to_lower(stress_cases$text)
stress_votes <- bind_cols(
  tibble(case_id = stress_cases$case_id),
  as_tibble(
    map(rule_patterns, \(pattern) {
      str_detect(stress_text_lower, pattern)
    })
  )
)
stress_counts <- stress_votes |>
  mutate(
    rule_count = as.integer(rowSums(stress_votes[rule_labels]))
  ) |>
  select(case_id, rule_count)
stress_voted_labels <- stress_votes |>
  pivot_longer(
    cols = all_of(rule_labels),
    names_to = "weak_label",
    values_to = "rule_vote"
  ) |>
  filter(rule_vote) |>
  summarise(
    voted_label = first(weak_label),
    .by = case_id
  )

stress_results <- stress_cases |>
  left_join(stress_counts, by = join_by(case_id)) |>
  left_join(stress_voted_labels, by = join_by(case_id)) |>
  mutate(
    weak_label = case_when(
      rule_count == 0L ~ "abstain",
      rule_count > 1L ~ "conflict",
      TRUE ~ voted_label
    )
  ) |>
  select(-rule_count, -voted_label)
Four designed adversarial failures
Challenge ID Challenge text Reference label Reference status Rule result
challenge-1 The training department is hiring an accountant. other designed challenge expectation training
challenge-2 Employees learn from experienced coworkers. training designed challenge expectation abstain
challenge-3 Certification is not required. requirement designed challenge expectation conflict
challenge-4 Flexible hours are available. schedule designed challenge expectation abstain

training department triggers a false positive. Employees learn and Flexible hours use words the rules do not cover. Certification is not required creates a conflict. These failures reveal missing context, synonyms, and negation.

Keep weak labels weak

When applying rules to more data:

  1. version every pattern and codebook.
  2. preserve each rule vote and abstention.
  3. measure coverage and conflict by source, language, and group.
  4. reserve independently reviewed data for evaluation.
  5. select a sample of apparent successes for human audit.
  6. run checks for negation, polysemy, and unseen wording.
  7. compare conclusions with and without weak labels.

Rules can provide useful training signals. They do not create facts, repair a source’s systematic errors, or remove the need for people who understand the domain.

What to remember

  • A weak label is a rule’s proposal, not ground truth.
  • Rules should permit abstention and show conflicts.
  • Coverage, conflict, and agreement answer different questions.
  • Report each rule separately. A score for all rules hides weaker rules.
  • Rules that share trigger words are not independent votes.
  • In-sample success can reflect rules written around the examples.
  • Designed challenge cases expose known limitations.
  • Independent evaluation measures rule performance on new data.

The investigation asks what each fictional notice explicitly states about training, schedules, requirements, and skills. The Lab can trace each answer through the source, OCR checks, codebook, individual labels, synthetic status, and rule votes. Those records do not make the notices representative or establish rule performance on new data. They make limits and unresolved cases available for inspection.

Sources