Choosing dictionary forms

Compare teaching lemmas with a released pipeline

word processing
lemmatization
workforce research
Learn how lemmas differ from stems and why lemmatizers built in different ways disagree on workforce text.

Stemming shortens the vocabulary but can produce strings such as requir and provid. These strings help computers compare forms. They are unsuitable as word labels in a public report.

The team wants readable labels for is, provided, and classes. It compares model-selected dictionary forms with rule-based stems. Model-selected forms can also be incorrect.

A lemma is a word’s dictionary form. Lemmatization chooses that form in context. This lesson compares a small UDPipe teaching model with a released spaCy English pipeline.

Note

The author created the fictional Riverton Workforce Lab, its job board, and its training flyer for teaching. Both models run locally. The lesson downloads nothing during execution.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • describe a lemma in plain language.
  • compare a lemma with a Snowball stem.
  • explain how tokenization changes the comparison set.
  • describe the limits of a dictionary proxy.
  • separate raw dictionary flags from broken-string counts.

Load both lemmatizers

Both local lemmatizers use the same text file and table tools. The code loads CSV and tokenization tools, the UDPipe model, and the spaCy interface. It also loads SnowballC stemming and hunspell dictionary functions. UDPipe receives one token per line. Its trained tagger returns lemmas through model, dictionary, and guesser components. spaCy receives raw text through the helper use_project_spacy().

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
library(tidytext)
library(tokenizers)
library(udpipe)
library(spacyr)
library(SnowballC)
library(hunspell)
source("R/use-spacy.R")

sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

pipeline <- use_project_spacy()
spacy_info <- spacy_pipeline_version()
spacy_info_table <- tibble(
  item = names(spacy_info),
  value = unname(unlist(spacy_info))
)

udpipe_parser <- udpipe_load_model(
  "data/treebank/en_ewt-500-parser.udpipe"
)

vertical <- sentences$text |>
  tokenize_words(lowercase = FALSE, strip_punct = FALSE) |>
  map_chr(\(tokens) paste(tokens, collapse = "\n"))

udpipe_parsed <- udpipe_annotate(
  udpipe_parser,
  x = vertical,
  doc_id = sentences$sentence_id,
  tokenizer = "vertical",
  tagger = "default",
  parser = "default"
) |>
  as.data.frame() |>
  as_tibble()

spacy_text <- sentences$text
names(spacy_text) <- sentences$sentence_id

spacy_parsed <- spacy_parse(
  spacy_text,
  pos = TRUE,
  lemma = TRUE,
  entity = TRUE,
  dependency = TRUE,
  nounphrase = TRUE
) |>
  as_tibble()
Local spaCy pipeline used by this lesson
Pipeline field Value
name core_web_sm
version 3.8.0
lang en
license MIT
spacy 3.8.7

The local pipeline is core_web_sm version 3.8.0 under the MIT license, with spaCy 3.8.7. UDPipe returns 178 tokens for the 28 text units. spaCy returns 179 tokens. It splits driver's into driver and 's, while the vertical tokenizer retains the whole word. The comparison tables exclude driver's because spaCy has no matching whole-word lemma.

Compare lemmas with stems

The next table uses word types that appear in the Riverton sentences and that both parsers emit as tokens. A stem and a lemma can agree, but they answer different questions.

known_to_hunspell <- \(words) map_lgl(hunspell(words), \(result) length(result) == 0L)

vocabulary_words <- sentences |>
  select(sentence_id, text) |>
  unnest_tokens(word, text) |>
  distinct(word) |>
  arrange(word)

udpipe_lookup <- udpipe_parsed |>
  filter(upos != "PUNCT") |>
  transmute(
    word = str_to_lower(token),
    udpipe_lemma = str_to_lower(lemma)
  ) |>
  group_by(word) |>
  summarise(udpipe_lemma = first(udpipe_lemma), .groups = "drop")

spacy_lookup <- spacy_parsed |>
  filter(pos != "PUNCT") |>
  transmute(
    word = str_to_lower(token),
    spacy_lemma = str_to_lower(lemma)
  ) |>
  group_by(word) |>
  summarise(spacy_lemma = first(spacy_lemma), .groups = "drop")

same_word_lemmas <- vocabulary_words |>
  left_join(udpipe_lookup, by = "word") |>
  left_join(spacy_lookup, by = "word")

comparison_words <- c(
  "is",
  "provided",
  "required",
  "classes",
  "shifts",
  "applicants",
  "training",
  "schedules",
  "paid",
  "pays"
)

lemma_stem_comparison <- same_word_lemmas |>
  filter(word %in% comparison_words) |>
  mutate(
    stem = wordStem(word, language = "en"),
    udpipe_known = known_to_hunspell(udpipe_lemma),
    spacy_known = known_to_hunspell(spacy_lemma)
  ) |>
  arrange(match(word, comparison_words))
Lemmas and stems for selected Riverton word types
Word UDPipe lemma spaCy lemma Snowball stem UDPipe lemma known spaCy lemma known
is be be is TRUE TRUE
provided provid provide provid FALSE TRUE
required requi require requir FALSE TRUE
classes classe class class FALSE TRUE
shifts shift shift shift TRUE TRUE
applicants applicants applicant applic TRUE TRUE
training training training train TRUE TRUE
schedules schedule schedule schedul TRUE TRUE
paid paid pay paid TRUE TRUE
pays pays pay pay TRUE TRUE

is shows why a lemma can be better than a stem. A stemmer leaves is alone, while both parsers choose be. The sharper contrast is provided: the UDPipe teaching model returns provid, while spaCy returns provide. For classes, UDPipe returns classe, while spaCy returns class.

The simplified table omits context. A word type can receive different lemmas in different sentences. The next check finds word types with multiple lemmas from at least one tool.

lemma_variation <- bind_rows(
  udpipe_parsed |>
    filter(upos != "PUNCT") |>
    transmute(system = "UDPipe", word = str_to_lower(token), lemma = str_to_lower(lemma)),
  spacy_parsed |>
    filter(pos != "PUNCT") |>
    transmute(system = "spaCy", word = str_to_lower(token), lemma = str_to_lower(lemma))
) |>
  group_by(system, word) |>
  summarise(
    distinct_lemmas = n_distinct(lemma),
    lemmas = paste(sort(unique(lemma)), collapse = ", "),
    occurrences = n(),
    .groups = "drop"
  ) |>
  filter(distinct_lemmas > 1L) |>
  arrange(system, word)
Word types whose lemmas vary across Riverton occurrences
System Word type Distinct lemmas Lemmas seen Occurrences
UDPipe evening 2 even, evening 2
UDPipe paid 2 paid, pay 3
UDPipe skills 2 skill, skills 2
spaCy skills 2 skill, skills 2

The systems differ in several ways. The project trained the UDPipe teaching model on a 500-sentence excerpt. spaCy provides a released English pipeline for general applications. UDPipe combines its trained tagger with dictionary and guesser components. spaCy’s English lemmatizer uses lookup tables and part-of-speech rules.

The tools, tokenization, and training setup all differ. This comparison therefore cannot isolate the effect of training size.

Count dictionary flags

A simple check finds lemmas absent from hunspell’s dictionary. Dictionary absence does not prove that a string is not an English word. The check cannot distinguish a valid missing entry, such as riverton, from an error. It also misses incorrect lemmas that are dictionary words. For example, it accepts leave for left in the left column.

The following analysis separates dictionary-coverage and case effects from malformed strings. It does not measure overall lemma correctness.

compared_word_lemmas <- same_word_lemmas |>
  filter(!is.na(udpipe_lemma), !is.na(spacy_lemma)) |>
  mutate(
    udpipe_known = known_to_hunspell(udpipe_lemma),
    spacy_known = known_to_hunspell(spacy_lemma),
    udpipe_flagged = !udpipe_known,
    spacy_flagged = !spacy_known,
    coverage_or_case_artifact = word %in% c("october", "riverton"),
    udpipe_broken_string = udpipe_flagged & !coverage_or_case_artifact,
    spacy_broken_string = spacy_flagged & !coverage_or_case_artifact
  )

lemma_quality <- tibble(
  system = c("UDPipe teaching model", "spaCy released pipeline"),
  word_types_compared = c(nrow(compared_word_lemmas), nrow(compared_word_lemmas)),
  raw_dictionary_flags = c(
    sum(compared_word_lemmas$udpipe_flagged),
    sum(compared_word_lemmas$spacy_flagged)
  ),
  coverage_or_case_artifacts = c(
    sum(compared_word_lemmas$udpipe_flagged & compared_word_lemmas$coverage_or_case_artifact),
    sum(compared_word_lemmas$spacy_flagged & compared_word_lemmas$coverage_or_case_artifact)
  ),
  broken_string_flags = c(
    sum(compared_word_lemmas$udpipe_broken_string),
    sum(compared_word_lemmas$spacy_broken_string)
  )
)

flagged_comparison <- compared_word_lemmas |>
  filter(udpipe_flagged | spacy_flagged) |>
  select(
    word,
    udpipe_lemma,
    spacy_lemma,
    udpipe_flagged,
    spacy_flagged,
    coverage_or_case_artifact,
    udpipe_broken_string,
    spacy_broken_string
  )
Dictionary proxy counts for lemmas on the same Riverton word types
System Word types compared Raw dictionary flags Coverage or case artifacts Broken-string flags
UDPipe teaching model 95 8 2 6
spaCy released pipeline 95 2 2 0
Where hunspell puts a mark on either model’s lemma
Word UDPipe lemma spaCy lemma UDPipe flagged spaCy flagged Coverage or case artifact UDPipe broken string spaCy broken string
classes classe class TRUE FALSE FALSE TRUE FALSE
diploma diploman diploma TRUE FALSE FALSE TRUE FALSE
includes includ include TRUE FALSE FALSE TRUE FALSE
need ne need TRUE FALSE FALSE TRUE FALSE
october october october TRUE TRUE TRUE FALSE FALSE
provided provid provide TRUE FALSE FALSE TRUE FALSE
required requi require TRUE FALSE FALSE TRUE FALSE
riverton riverton riverton TRUE TRUE TRUE FALSE FALSE

Both parsers provide lemmas for 95 word types. Hunspell puts a mark on 8 UDPipe lemmas and 2 spaCy lemmas. Both systems have flags for riverton and october, which are not model errors. The first is a fictional place name. The second results from lowercasing before dictionary lookup. spaCy originally returned October, which hunspell recognizes.

Excluding those cases, UDPipe produced six malformed strings: classe, diploman, includ, ne, provid, and requi. spaCy produced none. Six word types are a small count from 28 text units of one type.

Decide when the cost is justified

For a public table, an examined lemma is usually safer than a Snowball stem. is and are can group under be, and provided can become provide rather than provid.

Stemming requires less computation and often suffices for approximate internal matching. Strings such as requir and schedul are unsuitable for display as words. Correct lemmas provide readable labels. The UDPipe provid example shows that lemma columns still need checks.

final_check <- tibble(
  model = c("UDPipe", "spaCy"),
  provided_lemma = c(
    lemma_stem_comparison$udpipe_lemma[lemma_stem_comparison$word == "provided"],
    lemma_stem_comparison$spacy_lemma[lemma_stem_comparison$word == "provided"]
  )
)

invisible(suppressMessages(spacy_finalize()))
A final check on the clearest contrast
Model Lemma for provided
UDPipe provid
spaCy provide

What to remember

  • A lemma is a dictionary form chosen for a word in context.
  • A stem is a rule-based comparison string.
  • Tokenizers changed the comparison: driver's has no spaCy word-type match.
  • The automatic dictionary check puts a mark on unknown strings. It does not measure lemma correctness.
  • Raw hunspell flags are 8 for UDPipe and 2 for spaCy on 95 word types.
  • Excluding coverage and case artifacts, the broken-string count is 6 for UDPipe and 0 for spaCy.

Examine malformed or unexpected lemmas before using them to group public labels. This comparison finds six small-model outputs for further checks. It does not establish a causal effect of training size.

Sources