library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(digest)
library(tokenizers)
library(udpipe)
library(spacyr)
source("R/use-spacy.R")
sentences <- read_csv(
"data/workforce/workforce_sentences.csv",
na = character(),
col_types = cols(
sentence_id = col_character(),
document_id = col_character(),
source_line = col_character(),
text = col_character(),
reference_label = col_character(),
uncertainty = col_character(),
annotator_id = col_character(),
rationale = col_character(),
codebook_version = col_character(),
codebook_hash = col_character(),
derived = col_character(),
transformation = col_character()
)
)
parser_score_path <- "data/treebank/parser-held-out-score.csv"
parser_model_path <- "data/treebank/en_ewt-500-parser.udpipe"
held_out_path <- "data/treebank/en_ewt-held-out.conllu"
parser_score <- read_csv(
parser_score_path,
na = character(),
col_types = cols(
measure = col_character(),
value = col_double()
)
)
metadata <- read_csv(
"data/treebank/treebank-metadata.csv",
na = character(),
col_types = cols(
artifact = col_character(),
description = col_character(),
source = col_character(),
source_url = col_character(),
retrieved_on = col_character(),
license = col_character(),
fingerprint = col_character()
)
)
parser_score_fingerprint <- digest(
paste(read_lines(parser_score_path), collapse = "\n"),
algo = "sha256",
serialize = FALSE
)
parser_model_fingerprint <- digest(
parser_model_path,
algo = "sha256",
file = TRUE
)
expected_score_fingerprint <- metadata |>
filter(artifact == "parser-held-out-score.csv") |>
pull(fingerprint)
expected_model_fingerprint <- metadata |>
filter(artifact == "en_ewt-500-parser.udpipe") |>
pull(fingerprint)
held_out_lines <- read_lines(held_out_path)
held_out_tokens <- tibble(
line = held_out_lines,
sentence_index = cumsum(grepl("^# sent_id", held_out_lines))
) |>
filter(!grepl("^#", line), nzchar(line)) |>
mutate(fields = strsplit(line, "\t", fixed = TRUE)) |>
filter(map_int(fields, length) >= 8L) |>
transmute(
sentence_index,
token_id = map_chr(fields, 1),
head_token_id = map_chr(fields, 7),
dep_rel = map_chr(fields, 8)
) |>
filter(
!grepl("-", token_id, fixed = TRUE),
!grepl(".", token_id, fixed = TRUE)
)
stored_held_out_tokens <- as.integer(parser_score$value[parser_score$measure == "held_out_tokens"])
stored_correct_head <- as.integer(parser_score$value[parser_score$measure == "correct_head"])
stored_correct_head_and_relation <- as.integer(
parser_score$value[parser_score$measure == "correct_head_and_relation"]
)
computed_unlabelled <- round(stored_correct_head / stored_held_out_tokens, 4)
computed_labelled <- round(stored_correct_head_and_relation / stored_held_out_tokens, 4)
pipeline <- use_project_spacy()
spacy_info <- spacy_pipeline_version()
spacy_info_table <- tibble(
item = names(spacy_info),
value = unname(unlist(spacy_info))
)Drawing grammar links between words
Read dependency parses from a small trained model
word parsing
dependency parsing
workforce research
Learn how a dependency parser links words in workforce text and why attachment scores need limits.
Word labels are ready for the 28 Riverton text units. The team now asks what a sentence describes as the thing provided. It also asks whether the sentence names a provider.
Labels alone do not describe relationships between words. A sentence with experience and required can contradict a simple keyword interpretation. A passive sentence can also omit the provider. The team needs word relationships and direct inspection of those relationships.
A dependency parse records each token’s attachment and relation label. The attachment connects a token to its head, the word on which it depends. The local UDPipe model uses the same 500-sentence English Web Treebank training excerpt as Lesson 17. This model also performs parsing.
Note
The author created the fictional Riverton Workforce Lab, its job board, and its training flyer for teaching. Local files contain the parser and scores. The lesson downloads no model during execution.
TipWhat you will learn
By the end of this lesson, you will be able to:
- describe a dependency parse in plain language.
- read root, compound, numeric modifier, passive subject, and punctuation links.
- compare two parses of the same sentence.
- explain why a parser score from weblog posts needs a baseline and local checks.
Load the local parser inputs
The parser inputs come from local CSV files. The dplyr, tibble, and purrr packages handle tables. The tokenizers and udpipe packages parse fixed tokens, while digest records file fingerprints. The spacyr package produces a comparison parse. The helper use_project_spacy() starts the pinned spaCy pipeline.
| Measure | Value |
|---|---|
| held_out_tokens | 4007 |
| correct_head | 2899 |
| unlabelled_attachment | 0.7235 |
| correct_head_and_relation | 2629 |
| labelled_attachment | 0.6561 |
| Pipeline field | Value |
|---|---|
| name | core_web_sm |
| version | 3.8.0 |
| lang | en |
| license | MIT |
| spacy | 3.8.7 |
The chunk compares the held-out token count with the recorded value. It recomputes both ratios from stored numerators. It also compares parser and score fingerprints with the metadata. The recorded parser result has 2,899 correct heads among 4,007 held-out tokens, about 0.72. Both head and relation are correct for 2,629 tokens, about 0.66.
Parse fixed tokens
The model has no tokenizer, so the parser receives one token per line. This vertical format supplies each text unit as fixed tokens.
parser <- udpipe_load_model(parser_model_path)
vertical <- sentences$text |>
tokenize_words(lowercase = FALSE, strip_punct = FALSE) |>
map_chr(\(tokens) paste(tokens, collapse = "\n"))
parsed <- udpipe_annotate(
parser,
x = vertical,
doc_id = sentences$sentence_id,
tokenizer = "vertical",
tagger = "default",
parser = "default"
) |>
as.data.frame() |>
as_tibble()
paid_parse <- parsed |>
filter(doc_id == "s001") |>
mutate(
head_word = if_else(
head_token_id == "0",
"root",
token[match(head_token_id, token_id)]
)
) |>
select(token, head_word, dep_rel)| Token | Head word | Relation |
|---|---|---|
| Paid | week | compound |
| 12 | Paid | flat |
| - | Paid | punct |
| week | training | compound |
| training | provided | nsubj:pass |
| is | provided | aux:pass |
| provided | root | root |
| . | provided | punct |
The parse identifies provided as the root, the sentence’s central word. It attaches training to provided as a passive subject. It attaches is as a passive helper verb. The parser misinterprets Paid 12-week by attaching Paid to week and 12 to Paid. A reader would attach Paid to training and 12 to week.
The parse identifies training as the thing provided. The sentence does not name its provider. English passive constructions can omit the agent, so no provider token exists for the parser to identify.
The tables use these relation labels:
rootmarks the central word.compoundlinks words within a larger noun phrase.flatmarks name-like sequences without internal hierarchy. It is incorrect for12here.nummodmarks a numeric modifier.nmodmarks a nominal modifier.nsubj:passmarks a passive subject.aux:passmarks a passive helper.ccompmarks a clausal complement.punctmarks punctuation.
Compare one sentence with spaCy
spaCy’s English parser uses a different label scheme from Universal Dependencies. The linked UD documentation therefore does not explain its nsubjpass label.
spacy_s001 <- spacy_parse(
c(s001 = "Paid 12-week training is provided."),
pos = TRUE,
lemma = TRUE,
entity = TRUE,
dependency = TRUE,
nounphrase = TRUE
) |>
as_tibble()
spacy_paid_parse <- spacy_s001 |>
mutate(
head_word = if_else(
head_token_id == token_id,
"ROOT",
token[match(as.character(head_token_id), as.character(token_id))]
)
) |>
select(token, head_word, dep_rel, lemma, pos)
invisible(suppressMessages(spacy_finalize()))| Token | Head word | spaCy relation | Lemma | POS tag |
|---|---|---|---|---|
| Paid | training | nmod | pay | VERB |
| 12 | week | nummod | 12 | NUM |
| - | week | punct | - | PUNCT |
| week | training | compound | week | NOUN |
| training | provided | nsubjpass | training | NOUN |
| is | provided | auxpass | be | AUX |
| provided | ROOT | ROOT | provide | VERB |
| . | provided | punct | . | PUNCT |
The tools produce different structures, not merely different label spellings. spaCy attaches Paid to training and 12 to week. UDPipe attaches Paid to week and 12 to Paid. Both outputs contain tokens, heads, and labels, but their sentence analyses differ.
Read morphology in context
The parser output also includes the feats column. These are morphological features, compact labels for grammar carried by a token in this sentence.
paid_features <- parsed |>
filter(doc_id == "s001", token %in% c("is", "provided")) |>
select(token, feats)| Token | Morphological features |
|---|---|
| is | Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin |
| provided | Tense=Past|VerbForm=Part|Voice=Pass |
The feature string for is says it is a finite present-tense helper form. The feature string for provided says it is a past participle in passive voice. That is why the relation labels include passive forms.
Compare parsing with tagging
A parser score needs a baseline. The following weak baseline attaches each token to the previous token. It attaches the first sentence token to the root.
tagger_curve <- read_csv(
"data/treebank/tagger-learning-curve.csv",
na = character(),
col_types = cols(
training_sentences = col_integer(),
training_tokens = col_integer(),
held_out_tokens = col_integer(),
correct_tokens = col_integer(),
accuracy = col_double()
)
)
tagging_score <- tagger_curve |>
filter(training_sentences == 500L) |>
transmute(
system = "500-sentence tagger",
correct = correct_tokens,
tokens = held_out_tokens,
accuracy
)
attachment_baseline <- held_out_tokens |>
group_by(sentence_index) |>
mutate(previous_head = if_else(row_number() == 1L, "0", lag(token_id))) |>
ungroup() |>
summarise(
system = "Previous-token baseline, head only",
correct = sum(head_token_id == previous_head),
tokens = n(),
attachment = round(correct / tokens, 4),
.groups = "drop"
)
attachment_scores <- bind_rows(
attachment_baseline,
tibble(
system = c(
"500-sentence parser, head only",
"500-sentence parser, head and relation"
),
correct = c(stored_correct_head, stored_correct_head_and_relation),
tokens = c(stored_held_out_tokens, stored_held_out_tokens),
attachment = c(computed_unlabelled, computed_labelled)
)
)| System | Correct tokens | Tokens scored | Accuracy |
|---|---|---|---|
| 500-sentence tagger | 3641 | 4007 | 0.9087 |
| System | Correct tokens | Tokens scored | Attachment score |
|---|---|---|---|
| Previous-token baseline, head only | 304 | 4007 | 0.0759 |
| 500-sentence parser, head only | 2899 | 4007 | 0.7235 |
| 500-sentence parser, head and relation | 2629 | 4007 | 0.6561 |
The previous-token baseline has 304 correct heads among 4,007 tokens. The trained parser has 2,899 correct heads, about 0.72. The tagger’s 0.91 describes a different task with different choices. The table therefore reports tagging separately. Correct head-and-relation assignment requires a correct head, so that measure is stricter.
These numbers come from one train-and-test split of thirteen weblog posts. Tokens within a post share vocabulary, topic, and author, so they are not independent.
Examine Riverton roots
The held-out score does not evaluate local text. The Riverton set contains only 28 text units. The lesson compares each parser root with a hand-chosen root for that unit.
parser_roots <- parsed |>
filter(head_token_id == "0") |>
transmute(sentence_id = doc_id, parser_root = token)
hand_roots <- tribble(
~sentence_id, ~hand_root,
"s001", "provided",
"s002", "required",
"s003", "available",
"s004", "need",
"s005", "required",
"s006", "pays",
"s007", "part",
"s008", "able",
"s009", "required",
"s010", "preferred",
"s011", "available",
"s012", "receives",
"s013", "preferred",
"s014", "uses",
"s015", "provided",
"s016", "apprenticeship",
"s017", "required",
"s018", "outdoors",
"s019", "required",
"s020", "required",
"s021", "included",
"s022", "skill",
"s023", "HOUSE",
"s024", "CERTIFICATE",
"s025", "stipend",
"s026", "classes",
"s027", "required",
"s028", "Apply"
)
root_audit <- hand_roots |>
left_join(parser_roots, by = "sentence_id") |>
mutate(root_matches = parser_root == hand_root)
root_summary <- root_audit |>
summarise(
right_root = sum(root_matches),
wrong_root = sum(!root_matches),
total = n(),
.groups = "drop"
)
root_mismatches <- root_audit |>
filter(!root_matches)| Sentence ID | Hand root | Parser root | Root matches |
|---|---|---|---|
| s004 | need | skills | FALSE |
| s017 | required | valid | FALSE |
| s021 | included | Product | FALSE |
| s023 | HOUSE | RIVERTON | FALSE |
| s024 | CERTIFICATE | DATA | FALSE |
| s026 | classes | Evening | FALSE |
| s028 | Apply | October | FALSE |
The parser matches 21 of the 28 hand-marked Riverton roots and misses 7. Errors concentrate in noun-heavy lines and flyer headings with little sentence structure.
included_parse <- parsed |>
filter(doc_id == "s021") |>
mutate(
head_word = if_else(
head_token_id == "0",
"root",
token[match(head_token_id, token_id)]
)
) |>
select(token, head_word, dep_rel)| Token | Head word | Relation |
|---|---|---|
| Product | root | root |
| training | included | nsubj:pass |
| is | included | aux:pass |
| included | Product | ccomp |
| . | Product | punct |
In Product training is included, the main claim is that training is included. The parser incorrectly selects Product as the sentence head. It attaches included beneath Product as a subordinate clause, reversing the intended dependency.
What to remember
- A dependency parse links each token to a head word and names the relation.
- UDPipe and spaCy disagree on the heads inside
Paid 12-week. - A previous-token baseline assigns correct heads to 304 of 4,007 weblog tokens.
- The trained parser assigns correct heads to 2,899 of 4,007 tokens, about 0.72.
- The parser misses 7 of 28 hand-marked Riverton roots.
- Parser links on job-posting text need direct inspection.
Use dependency links to select text for checks, especially passive constructions. Examine the seven disputed roots before using them in downstream counts.