Finding what a sentence denies

Use parser links to evaluate a simple negation rule

word processing

negation recognition

workforce research

Learn why keyword matches can give incorrect conclusions about negation and how a small parser-based rule performs on workforce text.

The team searches for experience followed by required. The search returns No prior data experience is required.

That match can mislead readers. Both keywords are present, but the sentence says experience is not required. The search returns the opposite of the intended claim.

Negation recognition finds language that reverses or blocks an ordinary positive reading. A real system needs a scope, the part of the sentence that the negating word controls.

Note

The author created the fictional Riverton Workforce Lab, its job board, and its training flyer for teaching. The hand markings are teaching judgments for these 28 text units.

What you will learn

By the end of this lesson, you will be able to:

show why a keyword search can misread negation.
find no in this corpus.
list trigger words absent from this evaluation.
write a small scope rule using dependency links.
read a development-set score without treating it as general performance.
explain why trigger words alone are not enough.

Parse the sentences

The readr package reads the CSV, and dplyr and tibble create tables. The purrr package processes scopes, and stringr matches text. The tokenizers package fixes the word split, while udpipe applies the local model. Each text unit enters udpipe as a vertical token block.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
library(tokenizers)
library(udpipe)

sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

parser <- udpipe_load_model(
  "data/treebank/en_ewt-500-parser.udpipe"
)

vertical <- sentences$text |>
  tokenize_words(lowercase = FALSE, strip_punct = FALSE) |>
  map_chr(\(tokens) paste(tokens, collapse = "\n"))

parsed <- udpipe_annotate(
  parser,
  x = vertical,
  doc_id = sentences$sentence_id,
  tokenizer = "vertical",
  tagger = "default",
  parser = "default"
) |>
  as.data.frame() |>
  as_tibble()

The parser supplies lemmas, relation labels, and head-word IDs. These columns add grammatical information to the string match.

Start with a keyword search

The naive search below lowercases each sentence and searches for experience before required.

keyword_hits <- sentences |>
  mutate(
    lowered = str_to_lower(text),
    keyword_match = str_detect(lowered, "experience.*required")
  ) |>
  filter(keyword_match) |>
  select(sentence_id, text)

Keyword matches for experience before required
Sentence ID	Text
s002	No prior data experience is required.
s019	Two years of customer service experience are required.
s027	No prior experience required

The keyword search returns three sentences. Two of them start with No, so the plain match is not enough for the requirement question.

Mark negation by hand

Before presenting the rule, the team records a judgment for each text unit. A unit counts as negated when one of its words reverses the truth of its main claim. One author supplies these teaching labels. Other readers may dispute them.

no_sentence_count <- sum(str_detect(str_to_lower(sentences$text), "\\bno\\b"))

hand_judgments <- sentences |>
  transmute(
    sentence_id,
    text,
    hand_negated = sentence_id %in% c("s002", "s027"),
    reason = case_when(
      sentence_id == "s002" ~ "No negates prior data experience; experience is not required.",
      sentence_id == "s027" ~ "No negates prior experience in this flyer headline.",
      TRUE ~ "No negating word changes the main claim."
    )
  )

Hand judgments for negation in the 28 Riverton text units
Sentence ID	Text	Hand-marked negated	Reason
s001	Paid 12-week training is provided.	FALSE	No negating word changes the main claim.
s002	No prior data experience is required.	TRUE	No negates prior data experience; experience is not required.
s003	Evening schedules are available.	FALSE	No negating word changes the main claim.
s004	Applicants need basic spreadsheet skills.	FALSE	No negating word changes the main claim.
s005	A high school diploma is required.	FALSE	No negating word changes the main claim.
s006	The employer pays for certification training.	FALSE	No negating word changes the main claim.
s007	Rotating night shifts are part of the job.	FALSE	No negating word changes the main claim.
s008	Workers must be able to lift 50 pounds.	FALSE	No negating word changes the main claim.
s009	A portfolio is required.	FALSE	No negating word changes the main claim.
s010	Six months of experience is preferred.	FALSE	No negating word changes the main claim.
s011	Remote work is available two days each week.	FALSE	No negating word changes the main claim.
s012	Each new hire receives a mentor.	FALSE	No negating word changes the main claim.
s013	A medical records certificate is preferred.	FALSE	No negating word changes the main claim.
s014	The position uses a daytime schedule.	FALSE	No negating word changes the main claim.
s015	On-the-job training is provided.	FALSE	No negating word changes the main claim.
s016	This is a paid apprenticeship.	FALSE	No negating word changes the main claim.
s017	A valid driver’s license is required.	FALSE	No negating word changes the main claim.
s018	The work is outdoors and includes local travel.	FALSE	No negating word changes the main claim.
s019	Two years of customer service experience are required.	FALSE	No negating word changes the main claim.
s020	Weekend shifts are required.	FALSE	No negating word changes the main claim.
s021	Product training is included.	FALSE	No negating word changes the main claim.
s022	Clear written communication is an essential skill.	FALSE	No negating word changes the main claim.
s023	RIVERTON SKILLS OPEN HOUSE	FALSE	No negating word changes the main claim.
s024	DATA SUPPORT CERTIFICATE	FALSE	No negating word changes the main claim.
s025	Paid training stipend	FALSE	No negating word changes the main claim.
s026	Evening classes	FALSE	No negating word changes the main claim.
s027	No prior experience required	TRUE	No negates prior experience in this flyer headline.
s028	Apply by October 15	FALSE	No negating word changes the main claim.

The author marks 2 of the 28 text units as negated. Both use No. One is a full sentence, and the other is a flyer headline.

Write a parser-based scope rule

The rule has three steps:

Find a trigger with lemma no, not, or never.
Select the trigger’s head, its clause head, and their direct dependents as the scope.
Mark the sentence negated when the scope contains an auxiliary form of be.

A surface-form backup also examines advmod and det tokens. It adds no triggers in these 28 units. The author designed the rule after reading s002, so this score is a demonstration. This trigger-and-scope structure resembles NegEx, a 2001 system for clinical notes. The method can apply beyond that setting.

negating_words <- c("no", "not", "never")

parsed_with_triggers <- parsed |>
  mutate(
    token_lower = str_to_lower(token),
    lemma_lower = str_to_lower(lemma),
    negation_trigger = lemma_lower %in% negating_words |
      (dep_rel %in% c("advmod", "det") & token_lower %in% negating_words)
  )

surface_only_triggers <- parsed_with_triggers |>
  filter(
    !(lemma_lower %in% negating_words),
    dep_rel %in% c("advmod", "det"),
    token_lower %in% negating_words
  )

trigger_inventory <- parsed_with_triggers |>
  filter(token_lower %in% negating_words) |>
  distinct(token_lower) |>
  arrange(token_lower)

triggers <- parsed_with_triggers |>
  filter(negation_trigger) |>
  transmute(
    sentence_id = doc_id,
    trigger_id = token_id,
    trigger = token,
    trigger_head_id = head_token_id
  )

scope_for_trigger <- function(sentence_id, trigger_id, trigger, trigger_head_id) {
  current_sentence_id <- sentence_id
  current_trigger_id <- trigger_id
  current_trigger <- trigger
  current_trigger_head_id <- trigger_head_id

  sentence_tokens <- parsed_with_triggers |>
    filter(doc_id == current_sentence_id)

  clause_head_id <- sentence_tokens$head_token_id[
    match(current_trigger_head_id, sentence_tokens$token_id)
  ]

  if (is.na(clause_head_id) || identical(clause_head_id, "0")) {
    clause_head_id <- current_trigger_head_id
  }

  sentence_tokens |>
    filter(
      token_id %in% c(current_trigger_id, current_trigger_head_id, clause_head_id) |
        head_token_id %in% c(current_trigger_head_id, clause_head_id)
    ) |>
    transmute(sentence_id = doc_id, trigger = current_trigger, token, lemma, upos, dep_rel)
}

trigger_scopes <- pmap(triggers, scope_for_trigger) |>
  list_rbind()

rule_predictions <- trigger_scopes |>
  group_by(sentence_id) |>
  summarise(
    rule_negated = any(lemma == "be" & upos == "AUX"),
    scope_tokens = paste(token, collapse = " "),
    .groups = "drop"
  ) |>
  right_join(sentences |> select(sentence_id), by = "sentence_id") |>
  mutate(
    rule_negated = if_else(is.na(rule_negated), FALSE, rule_negated),
    scope_tokens = if_else(is.na(scope_tokens), "", scope_tokens)
  )

Tokens inside the simple negation scope
Sentence ID	Trigger	Scoped token	Lemma	POS tag	Relation
s002	No	No	no	INTJ	det
s002	No	prior	prior	ADJ	amod
s002	No	data	data	ADJ	amod
s002	No	experience	experience	NOUN	nsubj:pass
s002	No	is	be	AUX	aux:pass
s002	No	required	requi	VERB	root
s002	No	.	.	PUNCT	punct
s027	No	No	no	DET	det
s027	No	prior	prior	ADJ	amod
s027	No	experience	experience	NOUN	nsubj:pass
s027	No	required	requi	VERB	root

The corpus contains No, but not not or never. The rule names those other triggers explicitly, but this batch does not evaluate them. The surface-form backup adds no triggers. The rule detects the full sentence because its scope contains is. It misses the headline because the headline has no helper verb.

Show the development-set score

The scoring table compares the parser-based rule with the hand judgments on the same sentences that shaped the rule. It is a demonstration, not a held-out test.

negation_scored <- hand_judgments |>
  select(sentence_id, hand_negated) |>
  left_join(rule_predictions, by = "sentence_id") |>
  mutate(
    result = case_when(
      hand_negated & rule_negated ~ "true positive",
      !hand_negated & rule_negated ~ "false positive",
      hand_negated & !rule_negated ~ "false negative",
      TRUE ~ "true negative"
    )
  )

score_summary <- negation_scored |>
  summarise(
    true_positive = sum(result == "true positive"),
    false_positive = sum(result == "false positive"),
    false_negative = sum(result == "false negative"),
    true_negative = sum(result == "true negative"),
    correct = sum(hand_negated == rule_negated),
    total = n(),
    correct_fraction = paste0(correct, "/", total),
    .groups = "drop"
  )

mistakes <- negation_scored |>
  filter(hand_negated != rule_negated) |>
  left_join(sentences |> select(sentence_id, text), by = "sentence_id") |>
  select(sentence_id, text, hand_negated, rule_negated, result, scope_tokens)

Parser-rule score against hand negation judgments
True positives	False positives	False negatives	True negatives	Correct	Total	Correct fraction
1	0	1	26	27	28	27/28

The case with an incorrect rule result
Sentence ID	Text	Hand negated	Rule negated	Result	Scope tokens
s027	No prior experience required	TRUE	FALSE	false negative	No prior experience required

The rule agrees with hand judgments on 27 of 28 text units. This does not establish performance on new data. Twenty-six units contain no negation. A rule that never predicts negation would therefore score 26 of 28.

The author examined both negated units before rule design and specifically targeted s002. That unit is not an independent test. The recall count contains one true positive and one false negative. Zero false positives among twenty-six negatives does not exclude false alarms in a larger set.

The result also depends on the 500-sentence parser, which supplies every trigger, scope, and lemma. Preserve the observed failure. The headline No prior experience required lacks the helper verb that this rule requires.

What to remember

Negation can reverse a keyword match.
This corpus evaluates no, not not or never.
A scope says which words the trigger controls.
The simple parser rule detects s002 and misses s027.
English can express negation without these triggers. Experience is unnecessary has no trigger word and the same structure as Experience is required.
The 27/28 score is a demonstration on the development set.

Use the rule to select text for checks, not to assign automatic labels. Address the missed headline in the next rule design.

Parse the sentences

Start with a keyword search

Mark negation by hand

Write a parser-based scope rule

Show the development-set score

What to remember

Sources