Recording reading decisions as labels

Create a manual annotation codebook for workforce text

training data
annotation
workforce research
Learn how people label workforce text units without pretending that judgment is automatic or neutral.

The fictional Riverton Workforce Lab has 28 text units from the archived job page and reference flyer transcript. The team must classify training offers, application requirements, schedules, named skills, and other text. These distinctions affect the information that workers find when comparing notices. The team must state each label’s meaning before assigning labels.

A computer cannot learn these categories until people state their meanings. Manual annotation assigns labels to items according to written instructions. Annotators must decide what each label means and apply that meaning consistently. The task records what a text unit explicitly says. It does not establish worker eligibility or program suitability.

Note

The workforce text, labels, and research team are fictional. The author created the examples to illustrate annotation choices without worker or applicant records.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • describe an annotation unit and label set.
  • write rules for what each label includes and excludes.
  • create a versioned codebook with foundryR.
  • use pilot disagreements to revise rules.
  • examine ambiguous cases rather than hiding them.
  • explain why a human label is a documented judgment, not automatic truth.

Decide what one label describes

The annotation unit is one source-text line. It can be a job-detail list item or a flyer line. Each decision therefore concerns a short text that a reviewer can examine. The document_id identifies the source posting or flyer.

Not all source lines are sentences. Two are capitalized headings, three are noun phrases without verbs, and one is an imperative clause without terminal punctuation. A rule for sentences must also explain how reviewers should label a heading that makes no claim.

The next chunk loads readr for files and dplyr and tibble for tables. It also loads purrr for repeated checks and stringr for text checks. It then reads the CSV into a tibble.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)

workforce_sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

allowed_labels <- c(
  "training",
  "requirement",
  "schedule",
  "skill",
  "other"
)

first_units <- workforce_sentences |>
  select(
    sentence_id,
    document_id,
    text,
    reference_label,
    uncertainty
  ) |>
  slice_head(n = 8)
The first eight manually labeled text units
Unit ID Document ID Text Reference label Uncertainty
s001 J001 Paid 12-week training is provided. training certain
s002 J001 No prior data experience is required. requirement certain
s003 J001 Evening schedules are available. schedule certain
s004 J001 Applicants need basic spreadsheet skills. skill certain
s005 J002 A high school diploma is required. requirement certain
s006 J002 The employer pays for certification training. training certain
s007 J002 Rotating night shifts are part of the job. schedule certain
s008 J002 Workers must be able to lift 50 pounds. requirement certain

The file calls the reviewed result a reference label, not a ground-truth label. The label records the reviewer’s application of this codebook. Another defensible codebook could divide the concepts differently.

Count what the units actually are

Punctuation alone cannot establish whether a text unit is a sentence. Apply by October 15 is an imperative clause despite its missing full stop. The code therefore assigns explicit grammatical categories to the six flyer lines instead of inferring them from final characters.

unit_shapes <- workforce_sentences |>
  mutate(
    shape = case_when(
      sentence_id %in% c("s023", "s024") ~ "heading in capitals",
      sentence_id %in% c("s025", "s026", "s027") ~ "phrase without a verb",
      sentence_id == "s028" ~ "imperative clause",
      TRUE ~ "complete sentence"
    )
  )

shape_counts <- unit_shapes |>
  count(shape, name = "units")
Twenty-two sentences, two headings, three phrases, one imperative clause
Shape of the unit Units
complete sentence 22
heading in capitals 2
imperative clause 1
phrase without a verb 3

Five of the 28 units are not sentences. A sixth, Apply by October 15, is an imperative clause without terminal punctuation. All six come from a notice designed for passing readers. Paid training stipend has no verb. RIVERTON SKILLS OPEN HOUSE announces an event.

Both still receive a label, and the codebook has to say how. That is the reason the rules below talk about the unit’s main explicit claim rather than the sentence’s subject.

The six flyer rows match the six-line reference transcript. The annotation file neither expands nor interprets the source wording.

flyer_rows <- workforce_sentences |>
  filter(document_id == "F001")
flyer_transcript <- read_lines(
  "data/workforce/training-flyer-ground-truth.txt"
)
Row-level lineage from the flyer transcript
Unit ID Flyer line Verbatim text Reference label
s023 1 RIVERTON SKILLS OPEN HOUSE other
s024 2 DATA SUPPORT CERTIFICATE other
s025 3 Paid training stipend training
s026 4 Evening classes schedule
s027 5 No prior experience required requirement
s028 6 Apply by October 15 other

Write the codebook before expanding annotation

A codebook records the label definitions, instructions, examples, and version. The complete instrument lives in R/workforce-codebook.R. Its rules are:

  • training: the text unit offers or describes learning, mentoring, certification training, or an apprenticeship.
  • requirement: the text unit states or prefers experience, education, credentials, a license, a portfolio, or a physical condition.
  • schedule: the text unit describes when work or classes occur.
  • skill: the text unit names an ability used in the work.
  • other: none of the four definitions applies. Headings, deadlines, location, work setting, and travel belong here unless the text unit explicitly states another category.

The text unit’s main explicit claim receives one label. Reviewers do not infer a training offer from a job title or assume that a preferred credential is mandatory. A required shift receives schedule because timing is the main claim. Reviewers mark needs_review when more than one label remains defensible.

foundryR can store the codebook as a versioned object with a strict label schema and a content hash.

source("R/workforce-codebook.R")
annotation_codebook <- workforce_codebook()
recorded_codebook <- workforce_codebook("1.0.0")

annotation_codebook
foundry codebook: workforce-information
version: 1.1.0
hash: 7f4213ccadd8
variables:
  - label: string [training, requirement, schedule, skill, other] (Primary workforce-information label)
  - rationale: string (Short explanation using only the text unit)
  - uncertainty: string [certain, needs_review] (Whether the item needs adjudication)
examples: 7

The printed hash changes when the instructions, schema, examples, or version changes. Version 1.1.0 corrects the unit name. The archived labels retain their original 1.0.0 fingerprint because that version produced those records. A new annotation run should use 1.1.0.

The code does not call an AI service. foundryR creates the measurement instrument and calculates its fingerprint.

Count labels without interpreting the count

A label distribution can expose a missing category or a data-entry problem. It cannot establish how common these ideas are across the real labor market.

label_counts <- workforce_sentences |>
  mutate(
    reference_label = factor(
      reference_label,
      levels = allowed_labels
    )
  ) |>
  count(reference_label, name = "units", .drop = FALSE) |>
  transmute(
    label = as.character(reference_label),
    units
  )
Label counts in the teaching dataset
Reference label Text units
training 7
requirement 9
schedule 5
skill 2
other 5

The counts describe this designed example only. The examples are not a sample of jobs in Riverton or any real population.

Keep ambiguity visible

Unit s013 mentions a certificate, a word that also appears in training offers. Its main claim is that a certificate is preferred, so version 1.1.0 of the codebook labels it as a requirement.

ambiguous_case <- workforce_sentences |>
  filter(sentence_id == "s013")
A text unit that needs the codebook rule
Unit ID Document ID Text Reference label Uncertainty Rationale
s013 J004 A medical records certificate is preferred. requirement needs_review Preference is the main claim despite the training-related noun

A reviewer who chooses training may have a defensible interpretation. The difference does not necessarily indicate carelessness. Record the disagreement. Discuss the rule. Issue a new codebook version if the definition changes. Overwriting the label would remove evidence needed to improve the instrument.

Run a pilot before labeling everything

The rules came from a pilot. Two reviewers independently labeled six units without discussion. Their disagreements showed which instructions the codebook needed to clarify.

pilot_round <- tibble(
  sentence_id = c("s001", "s004", "s010", "s013", "s020", "s024"),
  reviewer_p = c(
    "training",
    "skill",
    "requirement",
    "training",
    "requirement",
    "training"
  ),
  reviewer_q = c(
    "training",
    "skill",
    "other",
    "requirement",
    "schedule",
    "other"
  )
) |>
  left_join(
    workforce_sentences |>
      select(sentence_id, text, reference_label),
    by = join_by(sentence_id)
  ) |>
  mutate(agreed = reviewer_p == reviewer_q)

pilot_agreement <- mean(pilot_round$agreed)
The six-unit pilot, before any rule was written
Unit Text Reviewer P Reviewer Q Agreed
s001 Paid 12-week training is provided. training training TRUE
s004 Applicants need basic spreadsheet skills. skill skill TRUE
s010 Six months of experience is preferred. requirement other FALSE
s013 A medical records certificate is preferred. training requirement FALSE
s020 Weekend shifts are required. requirement schedule FALSE
s024 DATA SUPPORT CERTIFICATE training other FALSE

Two of six matched in this designed pilot. Six items cannot establish a typical first-pass agreement rate. Here, the four disagreements show wording that the initial instructions did not settle.

Write one rule for each disagreement

The Lab wrote one rule per disagreement and adjudicated the four units under those rules. A third reader conducted adjudication without participating in the pilot labeling.

adjudication_log <- tibble(
  sentence_id = c("s010", "s013", "s020", "s024"),
  question = c(
    "Is a preferred condition still an entry condition?",
    "Does naming a credential make a unit a training offer?",
    "Does a required shift describe timing or an entry condition?",
    "Does a heading claim what its words name?"
  ),
  rule_added = c(
    "A preference is still an entry condition.",
    "A credential that is required or preferred is not training unless learning is offered.",
    "Timing takes precedence when a shift is required.",
    "Headings are other unless the unit explicitly states another category."
  ),
  adjudicated_label = c(
    "requirement",
    "requirement",
    "schedule",
    "other"
  )
)

instructions <- annotation_codebook$instructions
rule_clauses <- c(
  "main claim states or prefers",
  "is not training",
  "schedule takes precedence",
  "Headings, deadlines, location, work setting, and travel are other"
)

adjudication_check <- adjudication_log |>
  left_join(
    workforce_sentences |>
      select(sentence_id, reference_label, uncertainty),
    by = join_by(sentence_id)
  ) |>
  mutate(
    matches_file = adjudicated_label == reference_label,
    clause_in_codebook = map_lgl(
      rule_clauses,
      \(clause) str_detect(instructions, fixed(clause))
    )
  )
Four pilot disagreements, four rules, four recorded decisions
Unit Question raised Rule written in response Decision Uncertainty kept
s010 Is a preferred condition still an entry condition? A preference is still an entry condition. requirement certain
s013 Does naming a credential make a unit a training offer? A credential that is required or preferred is not training unless learning is offered. requirement needs_review
s020 Does a required shift describe timing or an entry condition? Timing takes precedence when a shift is required. schedule needs_review
s024 Does a heading claim what its words name? Headings are other unless the unit explicitly states another category. other needs_review

The current instructions contain every rule in the table. The check reads those rules directly from the instrument. Every adjudicated decision matches the archived label. Those rows retain version 1.0.0. Version 1.1.0 records the corrected unit wording for future work.

Three of the four disputed units retain needs_review. The file marks s010 as certain, so not every disagreement remains unresolved. Version 1.1.0 specifies one source-text line as the annotation unit. The historical sentence_id column remains a stable identifier. Its name does not mean that every row is a grammatical sentence.

This pilot shows how disagreement can produce explicit rules. Its six designed items cannot give an estimate of the effect of a pilot on a larger annotation task. They also cannot give an estimate of the time that task would require.

Protect the people doing the work

Manual annotation can involve sensitive, repetitive, or distressing material. Before assigning work:

  1. explain the data’s purpose and permitted applications.
  2. remove personal information that reviewers do not need.
  3. pay fairly for reading, training, and resolution of disagreements.
  4. give reviewers a way to skip harmful material.
  5. separate quality feedback from surveillance.
  6. record which languages and lived experience the task requires.

Expertise depends on the question. A worker, career counselor, employer, and labor researcher may interpret the same text unit differently. Include those perspectives in the study design rather than dismissing the differences.

When a language model reads the codebook too

A language model can read the codebook and propose a label for each unit. The evidence is mixed. Models match or exceed crowd-worker performance on some tasks. Careful replications find inconsistent performance between datasets. Models also follow complete codebooks more consistently than category names alone. A complete codebook contains definitions and difficult examples.

Apply the same requirements to model-label checks and human-label checks. Record the codebook version, prompt, and model revision. Compare a random sample with human decisions. Report the disagreements. The guide Using a language model in a reproducible analysis presents one example and a correction for the model’s errors.

What to remember

  • Annotation labels are measurements created from written rules.
  • Name the unit precisely. Headings and phrases are not sentences.
  • One unit per row keeps the decision small and traceable.
  • A codebook needs a version, examples, and a change record.
  • Run a pilot. Use disagreements to revise the rules.
  • Examine ambiguity instead of making an unrecorded correction.
  • Human labels require ethical working conditions and relevant expertise.
  • A language model used as an annotator needs the same codebook and checks.

The Lab stated the category meanings and preserved its reading decisions. The active-learning exercise starts with five complete job postings. It withholds two other documents’ labels from a small model during selection.

Sources