Examining named entities against domain text

Examine correct and incorrect NER outputs

phrases and entities

named entity recognition

workforce research

Learn how spaCy labels named entities in Riverton text and why fictional local names can produce errors.

Names in the Riverton job-board and flyer text need labels, not just highlights. The coordinator wants to know whether a span names an organisation, a date, a time, a quantity, or something else.

A trained named-entity model can supply those labels quickly. The team examines its labels for local names absent from the training examples.

Note

The author created the fictional Riverton Workforce Lab, its job board, and its training flyer for teaching.

What you will learn

By the end of this lesson, you will be able to:

describe named entity recognition in plain language.
run spaCy’s small English pipeline through spacyr.
read spaCy entity labels such as PERSON, ORG, DATE, TIME, QUANTITY, and GPE.
examine entity spans from the 28 Riverton text units.
explain why fictional local names need checks in context.

Run the released pipeline

The setup chunk reads the sentences, runs the released spaCy pipeline, and formats entity spans for inspection. Named entity recognition, or NER, finds names and assigns each one a type.

library(readr)
library(dplyr)
library(tibble)
library(stringr)
library(spacyr)
source("R/use-spacy.R")

sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

pipeline <- use_project_spacy()
pipeline_info <- spacy_pipeline_version()
parsed <- spacy_parse(
  setNames(sentences$text, sentences$sentence_id),
  pos = TRUE,
  lemma = TRUE,
  entity = TRUE,
  dependency = TRUE,
  nounphrase = TRUE
)
entities <- entity_extract(parsed, type = "all") |>
  as_tibble() |>
  transmute(
    sentence_id = doc_id,
    entity = str_replace_all(entity, "_", " "),
    entity_type
  )

example_text <- "Riverton Skills Centre will run evening classes in October."
example_parsed <- spacy_parse(
  c(example = example_text),
  pos = TRUE,
  lemma = TRUE,
  entity = TRUE,
  dependency = TRUE,
  nounphrase = TRUE
)
example_entities <- entity_extract(example_parsed, type = "all") |>
  as_tibble() |>
  transmute(
    entity = str_replace_all(entity, "_", " "),
    predicted_type = entity_type,
    actual_type = c("ORG", "TIME", "DATE"),
    correct = predicted_type == actual_type
  )

title_case_heading <- "Riverton Skills Open House"
title_case_parsed <- spacy_parse(
  c(title_case = title_case_heading),
  pos = TRUE,
  lemma = TRUE,
  entity = TRUE,
  dependency = TRUE,
  nounphrase = TRUE
)
title_case_entities <- entity_extract(title_case_parsed, type = "all") |>
  as_tibble() |>
  transmute(
    entity = str_replace_all(entity, "_", " "),
    entity_type
  )

spacy_finalize()

pipeline_table <- tibble(
  field = names(pipeline_info),
  value = unlist(pipeline_info, use.names = FALSE)
)

spaCy pipeline used for named entity recognition
Pipeline field	Value
name	core_web_sm
version	3.8.0
lang	en
license	MIT
spacy	3.8.7

NER output for a Riverton Skills Centre check sentence
Entity	Predicted type	Actual type	Correct
Riverton Skills Centre	PERSON	ORG	FALSE
evening	TIME	TIME	TRUE
October	DATE	DATE	TRUE

spaCy output for a title-case version of the flyer heading
Entity	Entity type
Riverton Skills Open House	PERSON

spaCy incorrectly labels Riverton Skills Centre as PERSON in the check sentence. The name refers to the provider, so the hand-assigned label is ORG. A facility interpretation would instead suggest FAC. The fictional name explains the domain mismatch but does not make the output correct. The title-case heading produces one PERSON span. This shows that surface form can change model output.

Read the labels

These type names come from the OntoNotes annotation project. The categories follow that project’s conventions.

PERSON means a person.
ORG covers organisations such as companies, agencies, and institutions.
DATE covers dates and periods such as months or years.
TIME marks a time of day or similar expression.
QUANTITY marks an amount with a unit.
GPE covers geopolitical places such as cities, states, and countries.

Context affects the label. In Riverton raised the training budget, Riverton can be a GPE. In Riverton Skills Centre, it can be part of an ORG.

entity_type_counts <- entities |>
  count(entity_type, name = "entities", sort = TRUE)

Entity types returned for the 28 Riverton text units
Entity type	Entities
DATE	6
TIME	2
ORG	1
QUANTITY	1

The model finds 10 entity spans in the 28 text units. Six are DATE, two are TIME, one is ORG, and one is QUANTITY. It returns no separate entity for RIVERTON in the all-capital heading. That is defensible because RIVERTON belongs to the event name rather than referring to the town.

The error concerns HOUSE. The model selects one word from a four-word event heading and labels it as an organisation. Both the span and type are wrong.

Examine the Riverton entities

The table gives hand-assigned labels for the returned spans. These teaching judgments cover 28 text units, not a public benchmark.

entity_answers <- tibble(
  sentence_id = c("s001", "s003", "s008", "s010", "s011", "s014", "s019", "s020", "s023", "s028"),
  entity = c("12 - week", "Evening", "50 pounds", "Six months", "two days", "daytime", "Two years", "Weekend", "HOUSE", "October 15"),
  expected_span = c("12 - week", "Evening", "50 pounds", "Six months", "two days", "daytime", "Two years", "Weekend", "RIVERTON SKILLS OPEN HOUSE", "October 15"),
  expected_type = c("DATE", "TIME", "QUANTITY", "DATE", "DATE", "TIME", "DATE", "DATE", "EVENT", "DATE"),
  note = c(
    rep("The span and type fit this sentence.", 8),
    "The event heading is four words; `HOUSE` alone is the wrong span and `ORG` is the wrong type.",
    "The span and type fit this sentence."
  )
)

entity_judgments <- entities |>
  left_join(entity_answers, by = c("sentence_id", "entity")) |>
  mutate(
    correct = entity == expected_span & entity_type == expected_type
  )

entity_score <- entity_judgments |>
  summarise(
    returned_spans = n(),
    correct_spans = sum(correct),
    wrong_spans = sum(!correct),
    .groups = "drop"
  )

Hand check of all spaCy entity spans returned from the 28 text units
Sentence ID	Entity	Predicted type	Expected span	Expected type	Note	Correct
s001	12 - week	DATE	12 - week	DATE	The span and type fit this sentence.	TRUE
s003	Evening	TIME	Evening	TIME	The span and type fit this sentence.	TRUE
s008	50 pounds	QUANTITY	50 pounds	QUANTITY	The span and type fit this sentence.	TRUE
s010	Six months	DATE	Six months	DATE	The span and type fit this sentence.	TRUE
s011	two days	DATE	two days	DATE	The span and type fit this sentence.	TRUE
s014	daytime	TIME	daytime	TIME	The span and type fit this sentence.	TRUE
s019	Two years	DATE	Two years	DATE	The span and type fit this sentence.	TRUE
s020	Weekend	DATE	Weekend	DATE	The span and type fit this sentence.	TRUE
s023	HOUSE	ORG	RIVERTON SKILLS OPEN HOUSE	EVENT	The event heading is four words; `HOUSE` alone is the wrong span and `ORG` is the wrong type.	FALSE
s028	October 15	DATE	October 15	DATE	The span and type fit this sentence.	TRUE

Hand-check counts for returned entity spans
Returned spans	Correct spans	Wrong spans
10	9	1

Nine of the ten returned spans match the hand judgments. This is not a general model evaluation. Eight correct spans are dates, durations, or times of day. The ninth is a weight. The only organisation-like output is wrong.

A single score conceals those differences between types. The check also says nothing about names that the model misses.

Keep domain mismatch visible

spaCy en_core_web_sm 3.8.0 is a released production pipeline. The author invented the Riverton names for this lesson. The model cannot assume those names occur in its training data. It must generalise from surface cues, word shape, and context. This illustrates domain mismatch, not proof of generally weak performance.

What to remember

NER labels spans under an annotation scheme such as OntoNotes.
The spaCy run returned 10 entity spans from the 28 Riverton text units.
Ten returned spans are too few, and too uneven by type, to evaluate a model.
Examine fictional local names directly because domain mismatch can change outputs.

For Riverton work, use NER to collect candidates. Examine the local names. Report the model version beside the examples.

Run the released pipeline

Read the labels

Examine the Riverton entities

Keep domain mismatch visible

What to remember

Sources