Finding phrases with readable rules

Evaluate a credential pattern before using its results

phrases and entities
rule-based matching
workforce research
Learn how simple string rules find credential phrases, how concordances help inspection, and why rules need maintenance.

Credential requirements are easy to miss in a short job-board feed. The Riverton Workforce Lab starts with a readable rule, then examines whether the rule answers the same question a reviewer would ask.

Readability does not establish correctness. A pattern can find real mentions, miss differently worded mentions, or match the same word with a different meaning.

Note

The author created the fictional Riverton Workforce Lab, its job board, and its training flyer for teaching.

TipWhat you will learn

After this lesson, you should be able to:

  • write small phrase rules with stringr.
  • extract the matched credential word.
  • apply a written hand-marking rule for required credentials.
  • count false positives and false negatives without treating the counts as a rate.
  • use a concordance to examine matches in context.

Write the first rules

The setup chunk reads the Riverton sentences, builds rule-result tables, handles regular expressions, and prepares a concordance. In stringr, regular expressions use the ICU engine and its rules for boundaries and letter classes.

library(readr)
library(dplyr)
library(tibble)
library(tidyr)
library(stringr)
library(quanteda)

sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

credential_terms <- c("certificate", "certification", "licen[cs]e")
credential_pattern <- regex(
  str_c("\\b(", str_c(credential_terms, collapse = "|"), ")\\b"),
  ignore_case = TRUE
)

rule_results <- sentences |>
  mutate(
    rule_match = str_detect(text, credential_pattern),
    matched_word = str_extract(text, credential_pattern),
    hand_required_credential = sentence_id %in% c("s005", "s017")
  )

matched_rows <- rule_results |>
  filter(rule_match | hand_required_credential) |>
  select(sentence_id, text, matched_word, rule_match, hand_required_credential)
Rule matches and hand markings for required credentials
Sentence ID Text Matched word Rule matched Hand-marked required credential
s005 A high school diploma is required. NA FALSE TRUE
s006 The employer pays for certification training. certification TRUE FALSE
s013 A medical records certificate is preferred. certificate TRUE FALSE
s017 A valid driver’s license is required. license TRUE TRUE
s024 DATA SUPPORT CERTIFICATE CERTIFICATE TRUE FALSE

The rule searches for certificate, certification, and either spelling of license. The author hand-marked each text unit under this definition:

Mark a unit only when it states an employer requirement for a formal qualification. The qualification must come from a school, licensing body, or certifying organisation. Do not mark a unit that merely mentions a qualification.

Under that rule, s005 and s017 qualify. One reviewer applying one rule to 28 units provides a worked example, not a reference standard.

Score the rule

A false positive is a text unit that the rule puts a mark on but the hand marking rejects. A false negative is a unit that the hand marking accepts but the rule misses. The pattern searches for credential words. The hand marking instead marks required credentials. The pattern has no requirement condition, so it can match preferred or a heading.

score_counts <- rule_results |>
  summarise(
    true_positive = sum(rule_match & hand_required_credential),
    false_positive = sum(rule_match & !hand_required_credential),
    false_negative = sum(!rule_match & hand_required_credential),
    .groups = "drop"
  ) |>
  pivot_longer(
    cols = everything(),
    names_to = "result",
    values_to = "sentences"
  )

false_positive_ids <- rule_results |>
  filter(rule_match & !hand_required_credential) |>
  pull(sentence_id)
false_negative_ids <- rule_results |>
  filter(!rule_match & hand_required_credential) |>
  pull(sentence_id)

problem_examples <- tibble(
  result = c("false positive", "false negative"),
  sentence_id = c("s006", "s005"),
  explanation = c(
    "The line says the employer pays for certification training, not that certification is required.",
    "The line says a diploma is required, but diploma was not in the rule."
  )
)
Text-unit counts for the first credential rule
Result Text units
true_positive 1
false_positive 3
false_negative 1
One false positive and one false negative
Result Sentence ID Explanation
false positive s006 The line says the employer pays for certification training, not that certification is required.
false negative s005 The line says a diploma is required, but diploma was not in the rule.

For required credentials, 3 of the 4 matches are wrong. The rule also misses 1 of the 2 hand-marked positives. Two positives cannot establish general performance. These counts illustrate failures. Both certificate and certification can appear without a requirement, while the pattern omits diploma.

Examine matches in context

A concordance lists every occurrence of a word in text. The usual display is keyword in context, or KWIC. It places the matched word between a few words of preceding and following context.

riverton_corpus <- corpus(setNames(sentences$text, sentences$sentence_id))
riverton_tokens <- tokens(riverton_corpus, remove_punct = TRUE)

credential_kwic <- kwic(
  riverton_tokens,
  pattern = credential_terms,
  window = 4,
  valuetype = "regex",
  case_insensitive = TRUE
) |>
  as.data.frame() |>
  as_tibble() |>
  select(docname, pre, keyword, post)
Concordance lines for the credential rule
Sentence ID Before Keyword After
s006 The employer pays for certification training
s013 A medical records certificate is preferred
s017 A valid driver’s license is required
s024 DATA SUPPORT CERTIFICATE

The KWIC display shows why the pattern needs revision. Its token-level regex uses the same credential terms as the scoring pattern. It finds certification training, a preferred certificate, a required license, and a flyer heading.

What to remember

  • A readable rule can still answer the wrong question.
  • This first credential rule has 1 true positive, 3 false positives, and 1 false negative.
  • Two hand-marked positives are enough for a worked example, not for a performance estimate.
  • KWIC lines show reviewers why a pattern matched.

Use this pattern only to select text for further checks. In the next revision, evaluate requirement language before adding credential terms.

Sources