library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(tokenizers)
library(stringr)
library(udpipe)
library(digest)
sentences <- read_csv(
"data/workforce/workforce_sentences.csv",
na = character(),
col_types = cols(
sentence_id = col_character(),
document_id = col_character(),
source_line = col_character(),
text = col_character(),
reference_label = col_character(),
uncertainty = col_character(),
annotator_id = col_character(),
rationale = col_character(),
codebook_version = col_character(),
codebook_hash = col_character(),
derived = col_character(),
transformation = col_character()
)
)
learning_curve <- read_csv(
"data/treebank/tagger-learning-curve.csv",
na = character(),
col_types = cols(
training_sentences = col_integer(),
training_tokens = col_integer(),
held_out_tokens = col_integer(),
correct_tokens = col_integer(),
accuracy = col_double()
)
)
metadata <- read_csv(
"data/treebank/treebank-metadata.csv",
na = character(),
col_types = cols(
artifact = col_character(),
description = col_character(),
source = col_character(),
source_url = col_character(),
retrieved_on = col_character(),
license = col_character(),
fingerprint = col_character()
)
)
builder_script_path <- "data-raw/build-treebank-tagger.R"
builder_seed_match <- read_lines(builder_script_path) |>
str_subset("^set.seed\\(") |>
str_match("^set.seed\\((\\d+)\\)")
builder_seed <- as.integer(builder_seed_match[1, 2])
rng_settings <- RNGkind()
learning_curve_path <- "data/treebank/tagger-learning-curve.csv"
learning_curve_fingerprint <- digest(
paste(read_lines(learning_curve_path), collapse = "\n"),
algo = "sha256",
serialize = FALSE
)
expected_curve_fingerprint <- metadata |>
filter(artifact == "tagger-learning-curve.csv") |>
pull(fingerprint)
metadata_display <- metadata |>
filter(
artifact %in% c(
"en_ewt-train-excerpt.conllu",
"en_ewt-held-out.conllu",
"en_ewt-500-tagger.udpipe",
"tagger-learning-curve.csv"
)
) |>
mutate(fingerprint_start = str_sub(fingerprint, 1, 12)) |>
select(artifact, source, retrieved_on, license, fingerprint_start)
reproducibility_table <- tibble(
item = c(
"Builder seed from script",
"Current R RNG kind",
"Current normal generator",
"Current sample generator",
"Current R version"
),
value = c(
as.character(builder_seed),
rng_settings[[1]],
rng_settings[[2]],
rng_settings[[3]],
R.version.string
)
)Tagging words by grammatical function
Read part-of-speech tags from a small trained model
word parsing
part-of-speech tagging
workforce research
Learn how a part-of-speech tagger labels workforce text and why held-out accuracy has limits.
After sentence labeling, the team examines single words in a job posting. It asks which words describe actions.
This changes the level of analysis. The team no longer classifies whole sentences about training or schedules. It examines each word’s grammatical function.
A part-of-speech tagger assigns a grammatical tag to each token. Tags can identify things, actions, descriptions, helper verbs, numbers, or punctuation. This lesson uses a UDPipe tagger with 500 hand-labeled training sentences from the Universal Dependencies English Web Treebank. A treebank contains sentences with human-assigned word-level labels.
Note
The author created the fictional Riverton Workforce Lab, its job board, and its training flyer for teaching. The project includes local UDPipe model and treebank files. The lesson downloads nothing during execution.
TipWhat you will learn
By the end of this lesson, you will be able to:
- describe common Universal POS tags in plain language.
- run a local UDPipe part-of-speech tagger on fixed tokens.
- examine a tagging mistake in a job-posting sentence.
- read a held-out accuracy score.
- explain why weblog-post accuracy does not establish accuracy on job postings.
Load the local files
In this read-only pass, readr opens files, and dplyr and tibble handle tables. The purrr package processes token lists, tokenizers splits words, and udpipe assigns tags. The digest package supplies file fingerprints for comparison. The code reads the trained model from disk and records settings for the bundled run. It does not train a model.
| Artifact | Source | Retrieved | License | SHA-256 start |
|---|---|---|---|---|
| en_ewt-500-tagger.udpipe | UD English EWT v2.18, released 2026-05-15 | 2026-08-28 | CC BY-SA 4.0 | 9c3907d37e9b |
| en_ewt-held-out.conllu | UD English EWT v2.18, released 2026-05-15 | 2026-08-28 | CC BY-SA 4.0 | 468cbf678026 |
| en_ewt-train-excerpt.conllu | UD English EWT v2.18, released 2026-05-15 | 2026-08-28 | CC BY-SA 4.0 | 8eb319b01a7d |
| tagger-learning-curve.csv | UD English EWT v2.18, released 2026-05-15 | 2026-08-28 | CC BY-SA 4.0 | b1c614e3cdd1 |
| Reproducibility item | Value |
|---|---|
| Builder seed from script | 20260828 |
| Current R RNG kind | Mersenne-Twister |
| Current normal generator | Inversion |
| Current sample generator | Rejection |
| Current R version | R version 4.6.1 (2026-06-24) |
The table prints the first 12 SHA-256 characters for readability at high zoom. The full values are in data/treebank/treebank-metadata.csv. A matching hash indicates that the file remains unchanged from the recorded version. It does not establish that the numbers are correct.
The second table combines one setting from the builder script with current environment details. The committed model lacks build-time RNG kind, R version, and package-version metadata. Record those details on the next rebuild. Only the 500-sentence tagger is in this repository. The committed files cannot recreate the other four learning-curve rows.
Tag fixed tokens
The model’s training covers token tagging, not raw-text sentence splitting. The code first tokenizes each Riverton text unit. It writes one token per line and declares the input as vertical text to UDPipe.
The output contains a lemma, a word’s dictionary form. For example, is has the lemma be. The shorthand \(tokens) defines an R function that handles one sentence’s tokens at a time.
tagger <- udpipe_load_model(
"data/treebank/en_ewt-500-tagger.udpipe"
)
vertical <- sentences$text |>
tokenize_words(lowercase = FALSE, strip_punct = FALSE) |>
map_chr(\(tokens) paste(tokens, collapse = "\n"))
tagged <- udpipe_annotate(
tagger,
x = vertical,
doc_id = sentences$sentence_id,
tokenizer = "vertical",
tagger = "default",
parser = "none"
) |>
as.data.frame() |>
as_tibble()
paid_sentence_tags <- tagged |>
filter(doc_id == "s001") |>
select(token, upos, lemma)| Token | POS tag | Lemma |
|---|---|---|
| Paid | PROPN | Paid |
| 12 | NUM | 12 |
| - | PUNCT | - |
| week | NOUN | week |
| training | NOUN | training |
| is | AUX | be |
| provided | VERB | provid |
| . | PUNCT | . |
The sentence has eight tokens after keeping punctuation. Two outputs are wrong in a way a reader can judge. Paid is not a proper noun. It describes the training, and the tagger likely followed the capital letter at the start of the sentence. The lemma provid for provided is not a word. The model guessed a stem from a suffix rule.
Read the accuracy number
This score is for 13 held-out political weblog posts from the same treebank, not for job postings. Held-out means text kept back from training so it can be reserved for scoring. The posts in both treebank excerpts are weblog documents from 2003 to 2006, which strengthens the domain-transfer warning.
train_path <- "data/treebank/en_ewt-train-excerpt.conllu"
held_out_path <- "data/treebank/en_ewt-held-out.conllu"
read_gold_tokens <- function(path) {
udpipe_read_conllu(path) |>
as_tibble() |>
filter(!str_detect(token_id, "[^0-9]"), !is.na(upos))
}
train_newdoc_ids <- read_lines(train_path) |>
str_subset(fixed("# newdoc id = ")) |>
str_remove(fixed("# newdoc id = "))
held_newdoc_ids <- read_lines(held_out_path) |>
str_subset(fixed("# newdoc id = ")) |>
str_remove(fixed("# newdoc id = "))
treebank_years <- str_match(
c(train_newdoc_ids, held_newdoc_ids),
"_ENG_(\\d{4})"
)[, 2] |>
as.integer()
train_gold <- read_gold_tokens(train_path) |>
transmute(token, gold_upos = upos)
held <- read_gold_tokens(held_out_path) |>
transmute(sentence_id, token, gold_upos = upos)
lookup <- train_gold |>
count(token, gold_upos, name = "n") |>
arrange(token, desc(n), gold_upos) |>
group_by(token) |>
slice(1) |>
ungroup() |>
select(token, baseline_upos = gold_upos)
held_vertical <- held |>
group_by(sentence_id) |>
summarise(text = paste(token, collapse = "\n"), .groups = "drop")
held_predictions <- udpipe_annotate(
tagger,
x = held_vertical$text,
doc_id = held_vertical$sentence_id,
tokenizer = "vertical",
tagger = "default",
parser = "none"
) |>
as.data.frame() |>
as_tibble() |>
group_by(doc_id) |>
mutate(token_row = row_number()) |>
ungroup() |>
transmute(
sentence_id = doc_id,
token_row,
tagger_upos = upos
)
scored <- held |>
left_join(lookup, by = "token") |>
mutate(
baseline_upos = coalesce(baseline_upos, "NOUN"),
seen_in_training = token %in% train_gold$token
) |>
group_by(sentence_id) |>
mutate(token_row = row_number()) |>
ungroup() |>
left_join(held_predictions, by = c("sentence_id", "token_row")) |>
mutate(
baseline_correct = baseline_upos == gold_upos,
tagger_correct = tagger_upos == gold_upos,
easy_gold_tag = gold_upos %in% c("PUNCT", "DET")
)
scored_500 <- learning_curve |>
filter(training_sentences == 500L) |>
mutate(
calculated_accuracy = correct_tokens / held_out_tokens,
accuracy_percent = round(calculated_accuracy * 100, 2)
)
learning_curve_display <- learning_curve |>
mutate(
accuracy_percent = round(accuracy * 100, 2)
)
gain_100_to_500 <- learning_curve |>
filter(training_sentences %in% c(100L, 500L)) |>
summarise(
points = diff(accuracy) * 100,
.groups = "drop"
) |>
pull(points)
gain_500_to_2000 <- learning_curve |>
filter(training_sentences %in% c(500L, 2000L)) |>
summarise(
points = diff(accuracy) * 100,
.groups = "drop"
) |>
pull(points)
training_ratio_100_to_500 <- 500L / 100L
training_ratio_500_to_2000 <- 2000L / 500L
accuracy_summary <- scored |>
summarise(
held_out_tokens = n(),
lookup_correct = sum(baseline_correct),
lookup_accuracy = lookup_correct / held_out_tokens,
tagger_correct = sum(tagger_correct),
tagger_accuracy = tagger_correct / held_out_tokens,
gain_points = (tagger_accuracy - lookup_accuracy) * 100,
seen_tokens = sum(seen_in_training),
seen_percent = seen_tokens / held_out_tokens * 100,
easy_tokens = sum(easy_gold_tag),
easy_percent = easy_tokens / held_out_tokens * 100,
.groups = "drop"
)
content_summary <- scored |>
filter(!easy_gold_tag) |>
summarise(
held_out_tokens = n(),
lookup_correct = sum(baseline_correct),
lookup_accuracy = lookup_correct / held_out_tokens,
tagger_correct = sum(tagger_correct),
tagger_accuracy = tagger_correct / held_out_tokens,
.groups = "drop"
)
accuracy_comparison <- tibble(
scope = c(
"All held-out tokens",
"All held-out tokens",
"Excluding PUNCT and DET",
"Excluding PUNCT and DET"
),
system = c(
"Lookup baseline",
"500-sentence tagger",
"Lookup baseline",
"500-sentence tagger"
),
correct = c(
accuracy_summary$lookup_correct,
accuracy_summary$tagger_correct,
content_summary$lookup_correct,
content_summary$tagger_correct
),
tokens = c(
accuracy_summary$held_out_tokens,
accuracy_summary$held_out_tokens,
content_summary$held_out_tokens,
content_summary$held_out_tokens
),
accuracy = round(correct / tokens, 4)
)| Training sentences | Training tokens | Held-out tokens | Correct tokens | Accuracy | Accuracy percent |
|---|---|---|---|---|---|
| 100 | 2311 | 4007 | 3301 | 0.8238 | 82.38 |
| 250 | 5539 | 4007 | 3496 | 0.8725 | 87.25 |
| 500 | 11595 | 4007 | 3641 | 0.9087 | 90.87 |
| 1000 | 21857 | 4007 | 3720 | 0.9284 | 92.84 |
| 2000 | 39802 | 4007 | 3778 | 0.9429 | 94.29 |
| Tokens included | System | Correct | Tokens | Accuracy |
|---|---|---|---|---|
| All held-out tokens | Lookup baseline | 3178 | 4007 | 0.7931 |
| All held-out tokens | 500-sentence tagger | 3641 | 4007 | 0.9087 |
| Excluding PUNCT and DET | Lookup baseline | 2307 | 3123 | 0.7387 |
| Excluding PUNCT and DET | 500-sentence tagger | 2766 | 3123 | 0.8857 |
The lookup baseline assigns each word its most frequent tag from the 500 training sentences. It assigns NOUN to unseen words. It correctly tags 3,178 of 4,007 tokens, or 0.7931. The trained tagger correctly tags 3,641 of 4,007, or about 91 percent. This is about 11.6 points above the lookup baseline.
Of the 4,007 held-out tokens, 3,140 also appear in the training sentences. Those tokens account for 78.4 percent of the held-out set.
The easier tags PUNCT and DET account for 884 of 4,007 held-out tokens, or 22.1 percent. Without them, the trained tagger scores 2,766 of 3,123, or 0.8857. The lookup baseline scores 2,307 of 3,123, or 0.7387. The easier tokens increase both overall scores.
On this learning curve, accuracy increases quickly at first and then more slowly. Increasing training from 100 to 500 sentences adds about 8.5 points. Increasing from 500 to 2,000 adds about 3.4 points. Each step adds blog posts and sentences, so the curve cannot separate training quantity from topic coverage. The intervals also differ proportionally. The first is a five-fold increase, and the second is a four-fold increase.
Keep the limits attached
The score comes from one run over thirteen blog posts. Tokens within a post are not independent because they share vocabulary, topic, and author. Actual variation exceeds what the token count alone suggests. The builder script sets set.seed(20260828).
The RNG and R-version rows describe the current environment, not the committed model’s build-time environment. This page reports only the bundled run. It provides no confidence interval because it neither retrains nor rescores on new samples or domains.
The Riverton text contains job postings and flyer lines, a different type of writing. This lesson does not claim the same accuracy there. It shows individual Riverton mistakes that readers can evaluate.
Universal POS tags provide grammatical categories for this task. Other projects can choose a different tagset.
What to remember
- A part-of-speech tagger assigns a grammatical tag to each token.
- Universal POS tags are compact labels such as
NOUN,VERB, andADJ. - The local tagger labeled 178 Riverton tokens.
- Its held-out weblog-post accuracy is 3,641 of 4,007 tokens, or about 91 percent.
- Examine job-posting errors directly.
- Weblog-post accuracy does not establish job-posting accuracy.
The team can use the tagger to make a first action-word list, then examine the mistakes before treating the labels as evidence.