library(readr)
library(dplyr)
library(tibble)
library(tidyr)
library(purrr)
library(stringr)
library(digest)
library(spacyr)
source("R/use-spacy.R")
relation_schema <- tribble(
~relation, ~subject_type, ~object_type, ~direction, ~status_values, ~what_counts,
"offers", "organisation", "credential",
"organisation -> credential", "asserted, negated, hedged",
"The sentence says the organisation offers, may offer, or does not offer the credential.",
"requires", "organisation", "credential",
"organisation -> credential", "asserted, negated, hedged",
"The sentence says the organisation requires, may require, or does not require the credential."
)Finding relations between named things
Extract directed triples without treating co-occurrence as evidence
systems
relation extraction
workforce research
Learn how a small dependency-pattern extractor turns linked Riverton mentions into relation triples with status and evidence spans.
Tomas reads short notices to find organisations that offer credentials and jobs that require credentials. A word search finds required. It cannot distinguish a required licence from a preferred or unnecessary licence.
Relation extraction finds a named, directed relationship between two entities. The output is a triple such as ORG-0002 offers CRD-0002. Each triple also includes a status and a reference to its evidence sentence.
This lesson keeps the example small. It links mentions with the Riverton alias table and examines the parser links. It compares the extractor with a co-occurrence baseline. That baseline records only whether two mentions share a sentence.
Note
The author created the fictional Riverton Workforce Lab, job board, and training flyer for teaching.
TipWhat you will learn
By the end of this lesson, you will be able to:
- define a relation schema before extracting triples
- keep entity IDs, direction, status, and offsets with each triple
- explain why two linked mentions in one sentence are not automatically related
- distinguish negated and uncertain statements from asserted statements
- compare pair detection with triple extraction without mixing their units
- say why a same-author reference is only a demonstration.
Define the relation schema
A schema specifies the permitted relation types and the entity type on each side. Direction is part of the claim. An organisation offers a credential. The credential does not offer the organisation.
| Relation | Subject type | Object type | Direction | Status values | What counts |
|---|---|---|---|---|---|
| offers | organisation | credential | organisation -> credential | asserted, negated, hedged | The sentence says the organisation offers, may offer, or does not offer the credential. |
| requires | organisation | credential | organisation -> credential | asserted, negated, hedged | The sentence says the organisation requires, may require, or does not require the credential. |
The schema is small on purpose. It does not try to extract every fact in a job notice. It specifies only two relations that later lessons can put into a knowledge base.
Read the source records and aliases
The 28 Riverton sentences are the verbatim source records used in earlier lessons. The alias table supplies IDs for organisations, places, and credentials. The extractor uses those IDs for arguments, the entities on each side of a relation. It does not use named entity recognition to choose the arguments.
sentences <- read_csv(
"data/workforce/workforce_sentences.csv",
na = character(),
col_types = cols(
sentence_id = col_character(),
document_id = col_character(),
source_line = col_character(),
text = col_character(),
reference_label = col_character(),
uncertainty = col_character(),
annotator_id = col_character(),
rationale = col_character(),
codebook_version = col_character(),
codebook_hash = col_character(),
derived = col_character(),
transformation = col_character()
)
)
entities <- read_csv(
"data/riverton/riverton-entities.csv",
na = character(),
col_types = cols(
entity_id = col_character(),
canonical_name = col_character(),
entity_type = col_character(),
description = col_character()
)
) |>
select(entity_id, canonical_name, entity_type)
aliases <- read_csv(
"data/riverton/riverton-aliases.csv",
na = character(),
col_types = cols(
entity_id = col_character(),
alias = col_character()
)
) |>
left_join(entities, by = "entity_id")
metadata <- read_csv(
"data/riverton/riverton-reference-metadata.csv",
na = character(),
col_types = cols(
artifact = col_character(),
description = col_character(),
source = col_character(),
license = col_character(),
created_on = col_character(),
fingerprint = col_character()
)
)
alias_hash <- digest(
paste(read_lines("data/riverton/riverton-aliases.csv"), collapse = "\n"),
algo = "sha256",
serialize = FALSE
)| Entity ID | Canonical name | Type | Alias |
|---|---|---|---|
| CRD-0001 | Forklift Operator Licence | credential | Forklift Operator Licence |
| CRD-0001 | Forklift Operator Licence | credential | forklift certification |
| CRD-0002 | Data Support Certificate | credential | DATA SUPPORT CERTIFICATE |
| CRD-0002 | Data Support Certificate | credential | Data Support Certificate |
| LOC-0001 | Riverton | place | Riverton |
| LOC-0001 | Riverton | place | Riverton, Calder |
| LOC-0002 | Riverton | place | Riverton |
| LOC-0002 | Riverton | place | Riverton, Tidewater |
| LOC-0003 | Bellhaven | place | Bellhaven |
| ORG-0001 | Riverton Workforce Lab | organisation | Riverton Workforce Lab |
| ORG-0001 | Riverton Workforce Lab | organisation | Workforce Lab |
| ORG-0001 | Riverton Workforce Lab | organisation | the Lab |
| ORG-0002 | Riverton Skills Centre | organisation | RSC |
| ORG-0002 | Riverton Skills Centre | organisation | Riverton Skills Centre |
| ORG-0002 | Riverton Skills Centre | organisation | Skills Centre |
| ORG-0003 | Marrow County Transit | organisation | MCT |
| ORG-0003 | Marrow County Transit | organisation | Marrow County Transit |
The table includes two rows for the alias Riverton, which is an ambiguity the lesson leaves unresolved. For relation arguments, the examples below use organisation and credential aliases that link to one ID.
Examine the 28 verbatim sentences once
The job-board and flyer lines are useful for inspection, but they do not contain two distinct linked entity mentions in one sentence. The table below is the only place this lesson prints all 28 source sentences. It shows why none of these lines can produce complete relation triples.
The matcher searches each sentence for every alias. It tries longer aliases first. Thus, it matches Riverton Skills Centre as one organisation before Riverton alone can match within that name. It records the first and last character of every match. It retains aliases shared by two IDs as ambiguous rather than guessing an identity.
Show the code that finds alias mentions
escape_regex <- function(value) {
str_replace_all(value, "([.\\\\|()\\[\\]{}^$*+?])", "\\\\\\1")
}
find_mentions <- function(document_id, sentence_id, text) {
candidates <- aliases |>
mutate(normalized_alias = str_to_lower(alias, locale = "en")) |>
group_by(normalized_alias, alias) |>
summarise(
candidate_count = n_distinct(entity_id),
entity_id = paste(sort(unique(entity_id)), collapse = "; "),
canonical_name = paste(sort(unique(canonical_name)), collapse = "; "),
entity_type = paste(sort(unique(entity_type)), collapse = "; "),
.groups = "drop"
) |>
mutate(alias_length = nchar(alias)) |>
arrange(desc(alias_length), alias)
kept <- vector("list", nrow(candidates))
used <- matrix(numeric(0), ncol = 2)
kept_index <- 0L
for (row in seq_len(nrow(candidates))) {
pattern <- regex(
paste0("(?<![[:alnum:]])", escape_regex(candidates$alias[[row]]), "(?![[:alnum:]])"),
ignore_case = TRUE
)
locations <- str_locate_all(text, pattern)[[1]]
if (nrow(locations) == 0L) {
next
}
for (hit in seq_len(nrow(locations))) {
span <- locations[hit, ]
overlaps <- nrow(used) > 0L &&
any(span[1] <= used[, 2] & span[2] >= used[, 1])
if (!overlaps) {
kept_index <- kept_index + 1L
kept[[kept_index]] <- tibble(
document_id = document_id,
sentence_id = sentence_id,
entity_id = candidates$entity_id[[row]],
canonical_name = candidates$canonical_name[[row]],
entity_type = candidates$entity_type[[row]],
link_status = if_else(candidates$candidate_count[[row]] > 1L, "ambiguous", "linked"),
mention = str_sub(text, span[1], span[2]),
start = span[1],
end = span[2]
)
used <- rbind(used, span)
}
}
}
if (kept_index == 0L) {
return(tibble(
document_id = character(),
sentence_id = character(),
entity_id = character(),
canonical_name = character(),
entity_type = character(),
link_status = character(),
mention = character(),
start = integer(),
end = integer()
))
}
kept[seq_len(kept_index)] |>
list_rbind() |>
arrange(start, end, entity_id)
}
verbatim_mentions <- pmap(
sentences |>
select(document_id, sentence_id, text),
find_mentions
) |>
list_rbind()
verbatim_inspection <- sentences |>
mutate(
linked_mentions = map_chr(
sentence_id,
\(id) {
rows <- verbatim_mentions |>
filter(sentence_id == id)
if (nrow(rows) == 0L) {
"none"
} else {
paste(
paste0(
rows$mention,
" (",
rows$entity_id,
if_else(rows$link_status == "ambiguous", ", ambiguous", ""),
")"
),
collapse = "; "
)
}
}
),
schema_cue = case_when(
str_detect(str_to_lower(text), "\\boffer|\\bprovided|\\bincluded|\\bpays") ~ "offer-like wording",
str_detect(str_to_lower(text), "\\brequired|\\bmust|\\bneed") ~ "require-like wording",
TRUE ~ "none"
),
polarity_read_by_hand = case_when(
sentence_id %in% c("s002", "s027") ~ "negated",
TRUE ~ "asserted or not applicable"
),
modality_read_by_hand = case_when(
sentence_id %in% c("s010", "s013") ~ "preferred, not asserted as required",
uncertainty == "needs_review" ~ "needs review",
TRUE ~ "plain"
)
) |>
select(
sentence_id,
document_id,
text,
linked_mentions,
schema_cue,
polarity_read_by_hand,
modality_read_by_hand
)| Sentence ID | Document ID | Text | Linked mentions | Schema cue | Polarity read by hand | Modality read by hand |
|---|---|---|---|---|---|---|
| s001 | J001 | Paid 12-week training is provided. | none | offer-like wording | asserted or not applicable | plain |
| s002 | J001 | No prior data experience is required. | none | require-like wording | negated | plain |
| s003 | J001 | Evening schedules are available. | none | none | asserted or not applicable | plain |
| s004 | J001 | Applicants need basic spreadsheet skills. | none | require-like wording | asserted or not applicable | plain |
| s005 | J002 | A high school diploma is required. | none | require-like wording | asserted or not applicable | plain |
| s006 | J002 | The employer pays for certification training. | none | offer-like wording | asserted or not applicable | plain |
| s007 | J002 | Rotating night shifts are part of the job. | none | none | asserted or not applicable | plain |
| s008 | J002 | Workers must be able to lift 50 pounds. | none | require-like wording | asserted or not applicable | plain |
| s009 | J003 | A portfolio is required. | none | require-like wording | asserted or not applicable | plain |
| s010 | J003 | Six months of experience is preferred. | none | none | asserted or not applicable | preferred, not asserted as required |
| s011 | J003 | Remote work is available two days each week. | none | none | asserted or not applicable | plain |
| s012 | J003 | Each new hire receives a mentor. | none | none | asserted or not applicable | plain |
| s013 | J004 | A medical records certificate is preferred. | none | none | asserted or not applicable | preferred, not asserted as required |
| s014 | J004 | The position uses a daytime schedule. | none | none | asserted or not applicable | plain |
| s015 | J004 | On-the-job training is provided. | none | offer-like wording | asserted or not applicable | plain |
| s016 | J005 | This is a paid apprenticeship. | none | none | asserted or not applicable | plain |
| s017 | J005 | A valid driver's license is required. | none | require-like wording | asserted or not applicable | plain |
| s018 | J005 | The work is outdoors and includes local travel. | none | none | asserted or not applicable | plain |
| s019 | J006 | Two years of customer service experience are required. | none | require-like wording | asserted or not applicable | plain |
| s020 | J006 | Weekend shifts are required. | none | require-like wording | asserted or not applicable | needs review |
| s021 | J006 | Product training is included. | none | offer-like wording | asserted or not applicable | plain |
| s022 | J006 | Clear written communication is an essential skill. | none | none | asserted or not applicable | plain |
| s023 | F001 | RIVERTON SKILLS OPEN HOUSE | RIVERTON (LOC-0001; LOC-0002, ambiguous) | none | asserted or not applicable | plain |
| s024 | F001 | DATA SUPPORT CERTIFICATE | DATA SUPPORT CERTIFICATE (CRD-0002) | none | asserted or not applicable | needs review |
| s025 | F001 | Paid training stipend | none | none | asserted or not applicable | plain |
| s026 | F001 | Evening classes | none | none | asserted or not applicable | plain |
| s027 | F001 | No prior experience required | none | require-like wording | negated | plain |
| s028 | F001 | Apply by October 15 | none | none | asserted or not applicable | plain |
These 28 lines do not supply complete entity pairs to the extractor. A page or posting could name a missing employer. That would require a page-structure assumption, not sentence evidence. This lesson makes no such assumption.
The author reads polarity and modality manually to select lines for inspection. Polarity distinguishes assertion from negation. Modality records meanings such as preference or uncertainty.
Parse and align mentions to tokens
spaCy supplies dependency links, which connect tokens by grammatical role. The code matches each alias-table mention to tokens within its character offsets. It selects the token whose head points outside the mention as the mention head.
pipeline <- use_project_spacy()
pipeline_info <- spacy_pipeline_version()
parsed <- spacy_parse(
setNames(constructed_sentences$text, constructed_sentences$sentence_id),
pos = TRUE,
lemma = TRUE,
entity = TRUE,
dependency = TRUE,
additional_attributes = "idx"
) |>
as_tibble() |>
mutate(
token_id = as.integer(token_id),
head_token_id = as.integer(head_token_id),
token_start = idx + 1L,
token_end = idx + nchar(token)
)
constructed_mentions <- pmap(
constructed_sentences |>
select(document_id, sentence_id, text),
find_mentions
) |>
list_rbind()
head_for_mention <- function(current_sentence_id, current_start, current_end) {
covered <- parsed |>
filter(
doc_id == current_sentence_id,
token_start >= current_start,
token_end <= current_end
)
external <- covered |>
filter(!head_token_id %in% covered$token_id)
if (nrow(external) == 0L) {
max(covered$token_id)
} else {
external$token_id[[1]]
}
}
mention_heads <- constructed_mentions |>
mutate(
head_token_id = pmap_int(
list(sentence_id, start, end),
head_for_mention
)
) |>
ungroup()
mention_head_table <- mention_heads |>
left_join(
parsed |>
transmute(sentence_id = doc_id, head_token_id = token_id, head_token = token, head_dep = dep_rel),
by = c("sentence_id", "head_token_id")
) |>
select(sentence_id, entity_id, mention, start, end, head_token, head_dep)| Sentence ID | Entity ID | Mention | Start | End | Mention head | Head relation |
|---|---|---|---|---|---|---|
| c01 | ORG-0002 | Riverton Skills Centre | 1 | 22 | Centre | nsubj |
| c01 | CRD-0002 | Data Support Certificate | 35 | 58 | Certificate | dobj |
| c02 | CRD-0002 | Data Support Certificate | 5 | 28 | Certificate | nsubjpass |
| c02 | ORG-0002 | Riverton Skills Centre | 44 | 65 | Centre | pobj |
| c03 | ORG-0002 | Riverton Skills Centre | 1 | 22 | Centre | nsubj |
| c03 | CRD-0001 | Forklift Operator Licence | 43 | 67 | Licence | dobj |
| c05 | ORG-0002 | Riverton Skills Centre | 1 | 22 | Centre | nsubj |
| c05 | CRD-0002 | Data Support Certificate | 35 | 58 | Certificate | dobj |
| c05 | CRD-0001 | Forklift Operator Licence | 68 | 92 | Licence | conj |
| c06 | ORG-0002 | Riverton Skills Centre | 1 | 22 | Centre | nsubj |
| c06 | CRD-0001 | Forklift Operator Licence | 38 | 62 | Licence | dobj |
| c07 | ORG-0003 | Marrow County Transit | 1 | 21 | Transit | nsubj |
| c07 | ORG-0002 | Riverton Skills Centre | 37 | 58 | Centre | pobj |
| c08 | ORG-0001 | Workforce Lab | 5 | 17 | Lab | nsubj |
| c08 | CRD-0001 | Forklift Operator Licence | 52 | 76 | Licence | dobj |
| c09 | ORG-0003 | Marrow County Transit | 1 | 21 | Transit | nsubj |
| c09 | CRD-0001 | Forklift Operator Licence | 46 | 70 | Licence | dobj |
| c11 | CRD-0001 | Forklift Operator Licence | 4 | 28 | Licence | nsubjpass |
| c11 | ORG-0003 | Marrow County Transit | 45 | 65 | Transit | pobj |
| c13 | ORG-0003 | Marrow County Transit | 1 | 21 | Transit | nsubjpass |
| c13 | CRD-0001 | Forklift Operator Licence | 50 | 74 | Licence | dobj |
The offsets are one-based character positions in the unmodified sentence text. That lets the triple point back to the exact evidence instead of to a rebuilt token string.
In the head-relation column, nsubj marks an active subject and dobj marks a direct object. nsubjpass marks a passive subject. pobj marks an object of a preposition, and conj marks a coordinated item. Lesson 18 introduces dependency parses.
Extract directed triples
The extractor searches for offer and require as relation indicators. It handles active subjects, passive agents, and coordinated objects. Negation patterns include not on a verb, No on an argument, and no longer under an adverb. It also recognizes uncertainty from may. This small pattern set misses the reported requirement.
Show the extractor code
trigger_tokens <- parsed |>
filter(lemma %in% c("offer", "require"), pos == "VERB") |>
transmute(
sentence_id = doc_id,
trigger_token_id = token_id,
trigger = token,
trigger_lemma = lemma,
trigger_dep = dep_rel,
trigger_head_token_id = head_token_id
)
argument_roles <- mention_heads |>
left_join(
parsed |>
transmute(sentence_id = doc_id, token_head_id = head_token_id, head_token_id = token_id, dep_rel),
by = c("sentence_id", "head_token_id")
) |>
select(
sentence_id, document_id, entity_id, canonical_name, entity_type,
mention, start, end, head_token_id, dep_rel, token_head_id
)
object_heads <- argument_roles |>
filter(entity_type == "credential") |>
select(
sentence_id,
object_id = entity_id,
object_name = canonical_name,
object_mention = mention,
object_start = start,
object_end = end,
object_head_id = head_token_id,
object_dep = dep_rel,
object_token_head_id = token_head_id
)
active_triples <- trigger_tokens |>
inner_join(
argument_roles |>
filter(entity_type == "organisation", dep_rel == "nsubj") |>
select(
sentence_id,
subject_id = entity_id,
subject_name = canonical_name,
subject_mention = mention,
subject_start = start,
subject_end = end,
subject_head_id = head_token_id,
subject_token_head_id = token_head_id
),
by = c("sentence_id", "trigger_token_id" = "subject_token_head_id")
) |>
inner_join(
object_heads |>
filter(object_dep %in% c("dobj", "conj")),
by = "sentence_id",
relationship = "many-to-many"
) |>
filter(
pmap_lgl(
list(sentence_id, object_dep, object_token_head_id, trigger_token_id),
\(current_sentence_id, current_object_dep, current_object_token_head_id, current_trigger_token_id) {
direct_object_heads <- object_heads$object_head_id[
object_heads$sentence_id == current_sentence_id &
object_heads$object_token_head_id == current_trigger_token_id
]
current_object_token_head_id == current_trigger_token_id ||
(current_object_dep == "conj" &&
current_object_token_head_id %in% direct_object_heads)
}
)
)
passive_agents <- parsed |>
filter(dep_rel == "agent") |>
select(sentence_id = doc_id, agent_token_id = token_id, trigger_token_id = head_token_id)
passive_triples <- trigger_tokens |>
inner_join(
object_heads |>
filter(object_dep == "nsubjpass"),
by = c("sentence_id", "trigger_token_id" = "object_token_head_id")
) |>
inner_join(passive_agents, by = c("sentence_id", "trigger_token_id")) |>
inner_join(
argument_roles |>
filter(entity_type == "organisation", dep_rel == "pobj") |>
select(
sentence_id,
subject_id = entity_id,
subject_name = canonical_name,
subject_mention = mention,
subject_start = start,
subject_end = end,
subject_head_id = head_token_id,
subject_token_head_id = token_head_id
),
by = c("sentence_id", "agent_token_id" = "subject_token_head_id")
)
raw_triples <- bind_rows(
active_triples,
passive_triples
) |>
mutate(
relation = recode(trigger_lemma, offer = "offers", require = "requires")
)
status_for_trigger <- function(current_sentence_id, trigger_token_id, subject_head_id, object_head_id) {
sentence_tokens <- parsed |>
filter(doc_id == current_sentence_id)
verb_negated <- any(
sentence_tokens$head_token_id == trigger_token_id &
sentence_tokens$dep_rel == "neg"
)
advmod_ids <- sentence_tokens$token_id[
sentence_tokens$head_token_id == trigger_token_id &
sentence_tokens$dep_rel == "advmod"
]
adverb_negated <- any(
sentence_tokens$head_token_id %in% advmod_ids &
sentence_tokens$dep_rel == "neg"
)
argument_det_no <- any(
sentence_tokens$head_token_id %in% c(subject_head_id, object_head_id) &
sentence_tokens$dep_rel == "det" &
str_to_lower(sentence_tokens$lemma) == "no"
)
hedged_aux <- any(
sentence_tokens$head_token_id == trigger_token_id &
sentence_tokens$dep_rel == "aux" &
str_to_lower(sentence_tokens$lemma) %in% c("may", "might", "could")
)
trigger_row <- sentence_tokens |>
filter(token_id == trigger_token_id)
reported_parent <- nrow(trigger_row) == 1L &&
trigger_row$dep_rel == "xcomp" &&
sentence_tokens$lemma[
match(trigger_row$head_token_id, sentence_tokens$token_id)
] %in% c("report", "say")
case_when(
verb_negated | adverb_negated | argument_det_no ~ "negated",
hedged_aux | reported_parent ~ "hedged",
TRUE ~ "asserted"
)
}
extracted_triples <- raw_triples |>
mutate(
status = pmap_chr(
list(sentence_id, trigger_token_id, subject_head_id, object_head_id),
status_for_trigger
)
) |>
left_join(
constructed_sentences |>
select(sentence_id, document_id, text),
by = "sentence_id"
) |>
mutate(
evidence_start = 1L,
evidence_end = nchar(text),
evidence_text = str_sub(text, evidence_start, evidence_end)
) |>
select(
sentence_id,
document_id,
relation,
subject_id,
object_id,
status,
subject_start,
subject_end,
object_start,
object_end,
evidence_start,
evidence_end,
evidence_text
) |>
arrange(sentence_id, relation, subject_id, object_id, status)| Sentence ID | Document ID | Relation | Subject ID | Object ID | Status | Subject start | Subject end | Object start | Object end | Evidence start | Evidence end | Evidence text |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| c01 | T001 | offers | ORG-0002 | CRD-0002 | asserted | 1 | 22 | 35 | 58 | 1 | 59 | Riverton Skills Centre offers the Data Support Certificate. |
| c02 | T002 | offers | ORG-0002 | CRD-0002 | asserted | 44 | 65 | 5 | 28 | 1 | 66 | The Data Support Certificate is offered by Riverton Skills Centre. |
| c03 | T003 | offers | ORG-0002 | CRD-0001 | negated | 1 | 22 | 43 | 67 | 1 | 68 | Riverton Skills Centre does not offer the Forklift Operator Licence. |
| c05 | T005 | offers | ORG-0002 | CRD-0001 | asserted | 1 | 22 | 68 | 92 | 1 | 93 | Riverton Skills Centre offers the Data Support Certificate and the Forklift Operator Licence. |
| c05 | T005 | offers | ORG-0002 | CRD-0002 | asserted | 1 | 22 | 35 | 58 | 1 | 93 | Riverton Skills Centre offers the Data Support Certificate and the Forklift Operator Licence. |
| c06 | T006 | offers | ORG-0002 | CRD-0001 | hedged | 1 | 22 | 38 | 62 | 1 | 73 | Riverton Skills Centre may offer the Forklift Operator Licence next year. |
| c09 | T009 | requires | ORG-0003 | CRD-0001 | negated | 1 | 21 | 46 | 70 | 1 | 71 | Marrow County Transit no longer requires the Forklift Operator Licence. |
| c11 | T011 | requires | ORG-0003 | CRD-0001 | negated | 45 | 65 | 4 | 28 | 1 | 66 | No Forklift Operator Licence is required by Marrow County Transit. |
The negation check is deliberately broader than “look for not under the verb.” It finds no longer through the adverb and No Forklift Operator Licence through the determiner on the credential argument.
Compare pairs separately from triples
The co-occurrence baseline pairs every two distinct linked mentions in a constructed sentence. It does not assign a relation type or direction, so those fields are not applicable. The lesson scores the extractor twice: once for entity-pair detection and once for labelled triples.
Show the pair and triple scoring code
cooccurrence_pairs <- constructed_mentions |>
select(sentence_id, document_id, entity_id) |>
distinct() |>
group_by(sentence_id, document_id) |>
filter(n() >= 2L) |>
summarise(
pairs = list(combn(sort(entity_id), 2, simplify = FALSE)),
.groups = "drop"
) |>
unnest_longer(pairs) |>
transmute(
sentence_id,
document_id,
first_id = map_chr(pairs, 1),
second_id = map_chr(pairs, 2),
relation = "not applicable",
direction = "not applicable"
)
reference_pairs <- reference_triples |>
transmute(
sentence_id,
first_id = pmin(subject_id, object_id),
second_id = pmax(subject_id, object_id)
) |>
distinct()
extractor_pairs <- extracted_triples |>
transmute(
sentence_id,
first_id = pmin(subject_id, object_id),
second_id = pmax(subject_id, object_id)
) |>
distinct()
score_pairs <- function(system_pairs, system_name) {
full_join(
system_pairs |> mutate(predicted = TRUE),
reference_pairs |> mutate(reference = TRUE),
by = c("sentence_id", "first_id", "second_id")
) |>
mutate(
predicted = coalesce(predicted, FALSE),
reference = coalesce(reference, FALSE),
system = system_name,
result = case_when(
predicted & reference ~ "true positive",
predicted & !reference ~ "false positive",
!predicted & reference ~ "false negative",
TRUE ~ "true negative"
)
) |>
filter(result != "true negative")
}
pair_results <- bind_rows(
score_pairs(cooccurrence_pairs |> select(sentence_id, first_id, second_id), "co-occurrence"),
score_pairs(extractor_pairs, "dependency extractor")
) |>
arrange(system, sentence_id, first_id, second_id)
triple_results <- full_join(
extracted_triples |>
select(sentence_id, relation, subject_id, object_id, status) |>
mutate(extractor = TRUE),
reference_triples |>
mutate(reference = TRUE),
by = c("sentence_id", "relation", "subject_id", "object_id", "status")
) |>
mutate(
extractor = coalesce(extractor, FALSE),
reference = coalesce(reference, FALSE),
cooccurrence_relation = "not applicable",
cooccurrence_direction = "not applicable",
result = case_when(
extractor & reference ~ "true positive",
extractor & !reference ~ "false positive",
!extractor & reference ~ "false negative",
TRUE ~ "true negative"
)
) |>
filter(result != "true negative") |>
arrange(sentence_id, relation, subject_id, object_id, status)| Sentence ID | Document ID | First ID | Second ID | Relation | Direction |
|---|---|---|---|---|---|
| c01 | T001 | CRD-0002 | ORG-0002 | not applicable | not applicable |
| c02 | T002 | CRD-0002 | ORG-0002 | not applicable | not applicable |
| c03 | T003 | CRD-0001 | ORG-0002 | not applicable | not applicable |
| c05 | T005 | CRD-0001 | CRD-0002 | not applicable | not applicable |
| c05 | T005 | CRD-0001 | ORG-0002 | not applicable | not applicable |
| c05 | T005 | CRD-0002 | ORG-0002 | not applicable | not applicable |
| c06 | T006 | CRD-0001 | ORG-0002 | not applicable | not applicable |
| c07 | T007 | ORG-0002 | ORG-0003 | not applicable | not applicable |
| c08 | T008 | CRD-0001 | ORG-0001 | not applicable | not applicable |
| c09 | T009 | CRD-0001 | ORG-0003 | not applicable | not applicable |
| c11 | T011 | CRD-0001 | ORG-0003 | not applicable | not applicable |
| c13 | T013 | CRD-0001 | ORG-0003 | not applicable | not applicable |
| Sentence ID | First ID | Second ID | Predicted | Reference | System | Result |
|---|---|---|---|---|---|---|
| c01 | CRD-0002 | ORG-0002 | TRUE | TRUE | co-occurrence | true positive |
| c02 | CRD-0002 | ORG-0002 | TRUE | TRUE | co-occurrence | true positive |
| c03 | CRD-0001 | ORG-0002 | TRUE | TRUE | co-occurrence | true positive |
| c05 | CRD-0001 | CRD-0002 | TRUE | FALSE | co-occurrence | false positive |
| c05 | CRD-0001 | ORG-0002 | TRUE | TRUE | co-occurrence | true positive |
| c05 | CRD-0002 | ORG-0002 | TRUE | TRUE | co-occurrence | true positive |
| c06 | CRD-0001 | ORG-0002 | TRUE | TRUE | co-occurrence | true positive |
| c07 | ORG-0002 | ORG-0003 | TRUE | FALSE | co-occurrence | false positive |
| c08 | CRD-0001 | ORG-0001 | TRUE | FALSE | co-occurrence | false positive |
| c09 | CRD-0001 | ORG-0003 | TRUE | TRUE | co-occurrence | true positive |
| c11 | CRD-0001 | ORG-0003 | TRUE | TRUE | co-occurrence | true positive |
| c13 | CRD-0001 | ORG-0003 | TRUE | TRUE | co-occurrence | true positive |
| c01 | CRD-0002 | ORG-0002 | TRUE | TRUE | dependency extractor | true positive |
| c02 | CRD-0002 | ORG-0002 | TRUE | TRUE | dependency extractor | true positive |
| c03 | CRD-0001 | ORG-0002 | TRUE | TRUE | dependency extractor | true positive |
| c05 | CRD-0001 | ORG-0002 | TRUE | TRUE | dependency extractor | true positive |
| c05 | CRD-0002 | ORG-0002 | TRUE | TRUE | dependency extractor | true positive |
| c06 | CRD-0001 | ORG-0002 | TRUE | TRUE | dependency extractor | true positive |
| c09 | CRD-0001 | ORG-0003 | TRUE | TRUE | dependency extractor | true positive |
| c11 | CRD-0001 | ORG-0003 | TRUE | TRUE | dependency extractor | true positive |
| c13 | CRD-0001 | ORG-0003 | FALSE | TRUE | dependency extractor | false negative |
| Sentence ID | Relation | Subject ID | Object ID | Status | Extractor | Reference | Co-occurrence relation | Co-occurrence direction | Result |
|---|---|---|---|---|---|---|---|---|---|
| c01 | offers | ORG-0002 | CRD-0002 | asserted | TRUE | TRUE | not applicable | not applicable | true positive |
| c02 | offers | ORG-0002 | CRD-0002 | asserted | TRUE | TRUE | not applicable | not applicable | true positive |
| c03 | offers | ORG-0002 | CRD-0001 | negated | TRUE | TRUE | not applicable | not applicable | true positive |
| c05 | offers | ORG-0002 | CRD-0001 | asserted | TRUE | TRUE | not applicable | not applicable | true positive |
| c05 | offers | ORG-0002 | CRD-0002 | asserted | TRUE | TRUE | not applicable | not applicable | true positive |
| c06 | offers | ORG-0002 | CRD-0001 | hedged | TRUE | TRUE | not applicable | not applicable | true positive |
| c09 | requires | ORG-0003 | CRD-0001 | negated | TRUE | TRUE | not applicable | not applicable | true positive |
| c11 | requires | ORG-0003 | CRD-0001 | negated | TRUE | TRUE | not applicable | not applicable | true positive |
| c13 | requires | ORG-0003 | CRD-0001 | hedged | FALSE | TRUE | not applicable | not applicable | false negative |
The pair table and triple table answer different questions. Co-occurrence can ask whether the right two IDs appeared together. It cannot answer which relation is correct, which way the arrow points, or whether the sentence denied the claim. The reported-verb test sentence is also visible as a false negative for this small dependency-pattern extractor.
Modern relation-extraction systems
Current relation-extraction papers often use large language models, prompts, or trained classifiers. Those systems still need the same schema, argument spans, status policy, and evidence checks. Published comparisons disagree by domain and scoring method, so this page does not make a performance claim about them.
Many current pipelines ask a language model to return each triple as JSON that must match a schema. A schema guarantee restricts format, not content. A reply can parse correctly but name the wrong entity or omit a negation.
Requesting the answer before reasoning can also lower accuracy. If a schema includes a reasoning field, place that field first. Keep this page’s evidence offsets and status column. Score output against a reference written before model execution. The guide Using a language model in a reproducible analysis covers the rest of the record-keeping.
What to remember
- Relation extraction produces typed, directed triples, not loose word matches.
- The argument IDs in this lesson come from the alias table, not from NER.
Noon an argument andno longerunder an adverb can change triple status.- Co-occurrence pairs have no relation type or direction.
- This same-author score is a demonstration, not a benchmark.
- Every triple needs sentence, document, and offset evidence.
Treat candidate pairs as items to examine. Treat schema triples as claims that require sentence evidence.
Sources
- ACE English relation annotation guidelines v5.8.3
- Jurafsky and Martin, Speech and Language Processing, chapter 21
- spacyr
spacy_parse()documentation - Zhang et al. (2017), TACRED
- Stoica, Platanios, and Poczos (2021), Re-TACRED
- Farkas et al. (2010), CoNLL-2010 shared task on uncertainty
- Wadhwa, Amir, and Wallace (2023), Revisiting relation extraction in the era of LLMs
- Jimenez Gutierrez et al. (2022), Biomedical relation extraction with large language models
- Tam et al. (2024), Let me speak freely? A study on the impact of format restrictions on LLM performance
- Geng et al. (2025), JSONSchemaBench: a benchmark of structured outputs for language models (preprint)