library(readr)
library(dplyr)
library(tibble)
library(tidyr)
library(purrr)
library(foundryR)
library(irr)
library(stringr)
crowd_labels <- read_csv(
"data/workforce/crowd_annotations.csv",
col_types = cols(
sentence_id = col_character(),
codebook_version = col_character(),
codebook_hash = col_character(),
reviewer_a = col_character(),
reviewer_a_uncertainty = col_character(),
reviewer_a_rationale = col_character(),
reviewer_b = col_character(),
reviewer_b_uncertainty = col_character(),
reviewer_b_rationale = col_character(),
reviewer_c = col_character(),
reviewer_c_uncertainty = col_character(),
reviewer_c_rationale = col_character()
)
)
sentence_text <- read_csv(
"data/workforce/workforce_sentences.csv",
col_types = cols(
sentence_id = col_character(),
document_id = col_character(),
source_line = col_character(),
text = col_character(),
reference_label = col_character(),
uncertainty = col_character(),
annotator_id = col_character(),
rationale = col_character(),
codebook_version = col_character(),
codebook_hash = col_character(),
derived = col_character(),
transformation = col_character()
)
)
reviewer_columns <- c(
"reviewer_a",
"reviewer_b",
"reviewer_c"
)
uncertainty_columns <- paste0(
reviewer_columns,
"_uncertainty"
)
rationale_columns <- paste0(
reviewer_columns,
"_rationale"
)
allowed_labels <- c(
"training",
"requirement",
"schedule",
"skill",
"other"
)
source("R/workforce-codebook.R")
crowd_codebook <- workforce_codebook("1.0.0")Working with several annotators
Measure disagreement before combining labels
training data
crowdsourcing
agreement
Learn how to compare workforce labels from several reviewers without treating majority vote as truth.
The provider checks kept unlike records separate. The Riverton Workforce Lab now examines whether the people assigning labels make comparable decisions. Three fictional paid reviewers apply the same codebook independently to 11 local sentences. Nine rows match. Two do not.
Crowdsourcing distributes tasks to a group of people, often through an online marketplace. Access to more reviewers can increase coverage, but a marketplace does not create expertise, fair working conditions, or valid labels. The team must design those conditions.
Note
The reviewers, annotations, and research team are fictional. The author designed the disagreements for this lesson, not to evaluate real workers.
TipWhat you will learn
By the end of this lesson, you will be able to:
- keep reviewer labels separate.
- calculate pairwise agreement with foundryR.
- distinguish observed agreement from chance-corrected measures.
- measure the effect of one item on an agreement statistic.
- find items that need adjudication.
- record adjudication decisions.
- explain why majority vote is not automatic truth.
Preserve each person’s answer
The annotation file stores one column per reviewer. The next chunk loads file, table, reshaping, repetition, agreement, and reliability helpers. It then reads the CSV files with declared column types. The code neither combines nor overwrites votes.
| Sentence ID | Reviewer A | Reviewer B | Reviewer C |
|---|---|---|---|
| s001 | training | training | training |
| s002 | requirement | requirement | requirement |
| s003 | schedule | schedule | schedule |
| s004 | skill | skill | skill |
| s005 | requirement | requirement | requirement |
| s006 | training | training | training |
| s010 | requirement | other | requirement |
| s011 | other | other | other |
| s013 | requirement | training | other |
| s016 | training | training | training |
| s017 | requirement | requirement | requirement |
Nine rows have complete agreement. Sentence s010 has one different answer. Sentence s013 has three different answers. Keeping the columns separate makes those disagreements visible, and the sample includes all five labels.
Measure all reviewer comparisons
Simple agreement is the share of answers that match. Cohen’s kappa adjusts for the agreement expected from the reviewers’ label frequencies. Krippendorff’s alpha is another chance-corrected measure. Its calculation can use data from more coders, even with missing labels, under suitable settings.
foundryR calculates several measures together. This example retains observed agreement and Cohen’s kappa for every reviewer pair. The comparison treats no reviewer as the truth. It therefore excludes directional classification measures such as precision and recall.
reviewer_pairs <- tibble(
pair = c("A and B", "A and C", "B and C"),
first = c(
"reviewer_a",
"reviewer_a",
"reviewer_b"
),
second = c(
"reviewer_b",
"reviewer_c",
"reviewer_c"
)
)
pairwise_agreement <- pmap(
reviewer_pairs,
\(pair, first, second) {
measures <- foundryR::foundry_agreement(
crowd_labels,
estimate = second,
truth = first
)
tibble(
pair = pair,
observed_agreement = filter(measures, metric == "accuracy")$value,
cohen_kappa = filter(measures, metric == "cohen_kappa")$value
)
}
) |>
list_rbind()| Reviewer pair | Observed agreement | Cohen’s kappa |
|---|---|---|
| A and B | 0.82 | 0.76 |
| A and C | 0.91 | 0.88 |
| B and C | 0.82 | 0.76 |
The B-and-C comparison is weaker than A and C. Reporting only one pair would hide that difference.
The table shows two decimals. With 11 items, one changed answer can alter observed agreement by about nine points. A third decimal would imply unsupported measurement precision. Reporting 0.818 could encourage a meaningless comparison with 0.812 from another study.
Krippendorff’s alpha summarizes all three reviewers at once. pivot_longer() converts the reviewer columns into one label column. Then pivot_wider() creates the matrix that irr needs. The calculation converts label names to category numbers without an ordering.
numeric_ratings <- crowd_labels |>
select(sentence_id, all_of(reviewer_columns)) |>
pivot_longer(
cols = all_of(reviewer_columns),
names_to = "reviewer",
values_to = "label"
) |>
mutate(category_number = match(label, allowed_labels)) |>
select(sentence_id, reviewer, category_number) |>
pivot_wider(
names_from = reviewer,
values_from = category_number
) |>
arrange(sentence_id)
rating_matrix <- numeric_ratings |>
select(all_of(reviewer_columns)) |>
as.matrix()
all_rater_alpha <- irr::kripp.alpha(
t(rating_matrix),
method = "nominal"
)$value
unanimity_by_sentence <- crowd_labels |>
select(sentence_id, all_of(reviewer_columns)) |>
pivot_longer(
cols = all_of(reviewer_columns),
names_to = "reviewer",
values_to = "label"
) |>
summarise(
unanimous = n_distinct(label) == 1L,
.by = sentence_id
) |>
arrange(sentence_id)
unanimity_share <- unanimity_by_sentence |>
summarise(value = mean(unanimous))
unanimity_share <- unanimity_share$value
tibble(
measure = c(
"Krippendorff's alpha",
"Unanimous share"
),
value = round(
c(all_rater_alpha, unanimity_share),
2
)
)# A tibble: 2 × 2
measure value
<chr> <dbl>
1 Krippendorff's alpha 0.8
2 Unanimous share 0.82The 11-item example is too small for a precise population estimate. These statistics describe the annotation exercise and help find problems. They do not certify the codebook or any reviewer.
Measure sensitivity to one item
Removing one item at a time shows the estimate’s sensitivity to individual items. With 11 items, this procedure requires 11 recalculations and no new data.
alpha_without_item <- map_dbl(
seq_len(nrow(rating_matrix)),
\(dropped) {
irr::kripp.alpha(
t(rating_matrix[-dropped, , drop = FALSE]),
method = "nominal"
)$value
}
)
sensitivity <- tibble(
sentence_id = numeric_ratings$sentence_id,
alpha_without_this_item = alpha_without_item
) |>
arrange(desc(alpha_without_this_item))
alpha_range <- range(alpha_without_item)| Item removed | Alpha for the other ten |
|---|---|
| s013 | 0.91 |
| s010 | 0.87 |
| s002 | 0.78 |
| s005 | 0.78 |
| s017 | 0.78 |
| s001 | 0.78 |
| s006 | 0.78 |
| s016 | 0.78 |
| s011 | 0.77 |
| s003 | 0.76 |
| s004 | 0.76 |
Alpha is 0.80 on all 11 items. Removing one item changes alpha to a value between 0.76 and 0.91. Removing s013 produces the highest value. All three reviewers interpreted that item differently.
This range is not a confidence interval. Do not report it as one. This single-item sensitivity leaves comparisons between codebooks, tasks, or reviewer pools uncertain. Report the range alongside the estimate. Reporting 0.797 alone would conceal that sensitivity.
A larger sample can reduce single-item sensitivity, but sample size alone does not explain the change in alpha. Label prevalence, document clustering, the removed item, and the disagreement pattern also affect the result.
For a larger study, resample whole source documents rather than individual rows. Report the resulting bootstrap distribution. These 11 items cannot predict the distribution’s width for 500 items.
Find the disagreements
An item needs a check when the three labels are not identical. left_join() adds the sentence text by matching the shared sentence identifier.
disagreements <- crowd_labels |>
left_join(
unanimity_by_sentence,
by = join_by(sentence_id)
) |>
filter(!unanimous) |>
left_join(
sentence_text |>
select(sentence_id, text),
by = join_by(sentence_id)
) |>
arrange(sentence_id)| Sentence ID | Text | Reviewer A | A uncertainty | Reviewer B | B uncertainty | Reviewer C | C uncertainty |
|---|---|---|---|---|---|---|---|
| s010 | Six months of experience is preferred. | requirement | certain | other | needs_review | requirement | certain |
| s013 | A medical records certificate is preferred. | requirement | needs_review | training | needs_review | other | needs_review |
The first sentence says experience is preferred. One reviewer chose other. The second mentions a certificate and receives three different labels. These choices classify the same notices differently for readers who compare options. The codebook must explain the category distinction instead of replacing that explanation with a vote.
Calculate a majority label carefully
A majority label can organize the next check. It should not erase the individual answers or replace adjudication.
label_counts <- crowd_labels |>
select(sentence_id, all_of(reviewer_columns)) |>
pivot_longer(
cols = all_of(reviewer_columns),
names_to = "reviewer",
values_to = "label"
) |>
count(sentence_id, label, name = "votes")
top_vote_count <- label_counts |>
summarise(top_votes = max(votes), .by = sentence_id)
top_vote_table <- label_counts |>
left_join(top_vote_count, by = join_by(sentence_id)) |>
filter(votes == top_votes) |>
summarise(
tied_top_labels = n(),
top_label = paste(sort(label), collapse = ", "),
.by = c(sentence_id, top_votes)
)
majority_label <- top_vote_table |>
mutate(
majority_label = case_when(
top_votes > length(reviewer_columns) / 2 &
tied_top_labels == 1L ~ top_label,
TRUE ~ "adjudicate"
)
) |>
select(sentence_id, majority_label)
combined_labels <- crowd_labels |>
select(sentence_id) |>
left_join(
majority_label,
by = join_by(sentence_id)
) |>
left_join(
unanimity_by_sentence,
by = join_by(sentence_id)
)| Sentence ID | Top votes | Labels tied for top | Top label or labels |
|---|---|---|---|
| s001 | 3 | 1 | training |
| s002 | 3 | 1 | requirement |
| s003 | 3 | 1 | schedule |
| s004 | 3 | 1 | skill |
| s005 | 3 | 1 | requirement |
| s006 | 3 | 1 | training |
| s010 | 2 | 1 | requirement |
| s011 | 3 | 1 | other |
| s013 | 1 | 3 | other, requirement, training |
| s016 | 3 | 1 | training |
| s017 | 3 | 1 | requirement |
| Sentence ID | Majority label | Unanimous |
|---|---|---|
| s001 | training | TRUE |
| s002 | requirement | TRUE |
| s003 | schedule | TRUE |
| s004 | skill | TRUE |
| s005 | requirement | TRUE |
| s006 | training | TRUE |
| s010 | requirement | FALSE |
| s011 | other | TRUE |
| s013 | adjudicate | FALSE |
| s016 | training | TRUE |
| s017 | requirement | TRUE |
The unanimous column distinguishes majority decisions from complete agreement. The tie check remains necessary if teams change the reviewer count. Any tie among the top labels returns adjudicate. Sentence s013 returns adjudicate because three different answers do not form a majority. The team can examine the sentence, codebook, and source labels before choosing a reference label.
Finish the adjudication
A queue of disputed items alone does not resolve disagreements. A fourth reader adjudicated both items in the queue using the codebook rather than the vote counts. That reader did not participate in the initial labeling.
adjudication <- tibble(
sentence_id = c("s010", "s013"),
question = c(
"Is a preferred condition an entry condition or something else?",
"Does naming a credential make the unit a training offer?"
),
rule_applied = c(
"requirement: the main claim states or prefers an entry condition",
"training: a credential required or preferred is not training"
),
decision = c("requirement", "requirement"),
decided_by = "fourth reader, not one of the three reviewers"
)
adjudication_outcome <- adjudication |>
left_join(
sentence_text |>
select(sentence_id, text, reference_label),
by = join_by(sentence_id)
) |>
left_join(
crowd_labels |>
select(sentence_id, all_of(reviewer_columns)),
by = join_by(sentence_id)
) |>
mutate(
matches_reference_file = decision == reference_label,
reviewers_still_disagree = reviewer_a != reviewer_b |
reviewer_b != reviewer_c
)| Sentence ID | Text | Reviewer A | Reviewer B | Reviewer C | Adjudicated label |
|---|---|---|---|---|---|
| s010 | Six months of experience is preferred. | requirement | other | requirement | requirement |
| s013 | A medical records certificate is preferred. | requirement | training | other | requirement |
Both items resolve to requirement. Each decision cites a codebook clause. The table retains the original reviewer columns and disagreements. The check confirms that adjudication did not overwrite the different reviewer answers.
Adjudication did not make s013 easy to interpret. Three trained readers reached three answers. The sensitivity analysis shows that this item accounts for most of the difference between alpha values of 0.80 and 0.91. The decision resolves the label but not the interpretive difficulty. Include the item as a worked example in the next codebook revision.
Agreement does not prove validity
Reviewers can agree because:
- the codebook is clear.
- the item is easy.
- they share the same misunderstanding.
- the task excludes a relevant perspective.
- the platform trained everyone toward the same shortcut.
Validity requires evidence that the labels measure the intended idea. Compare annotations with the research question. Consult people with relevant knowledge. Examine how conclusions change under reasonable label choices.
Treat crowd work as work
A responsible plan states:
- payment for training, annotation, and adjudication time.
- expected task length and rejection rules.
- privacy and data-handling requirements.
- access to support and the ability to skip harmful material.
- required expertise and language knowledge.
- an appeal path for disputed quality decisions.
Hidden attention checks and automatic rejection can shift measurement errors onto workers. Quality control should diagnose the task and instructions as well as individual answers.
What to remember
- Keep individual reviewer labels before combining them.
- Agreement measures summarize consistency, not truth.
- Report agreement without implying more precision than the sample size provides.
- Show how far a statistic moves when one item is removed.
- Examine disagreement items. Revise the codebook when needed.
- Complete adjudication. Record the rule for each decision.
- Preserve whether a majority label was unanimous.
- Fair pay, privacy, expertise, and appeal processes are part of data quality.
The adjudicator resolved both disputed cases under named rules. The page retains the original reviewer answers. The following augmentation exercise preserves that record and creates fictional variants. Each variant retains a synthetic label. A reviewer compares each variant with the same codebook before its inclusion in training data.