Removing word endings with stemming rules

Use Snowball stems without mistaking them for words

word processing
stemming
workforce research
Learn how stemming groups some workforce word forms and creates non-word stems.

A short vocabulary list from the job-board and flyer text is the team’s next goal. The team sees shifts, skills, and schedules and asks whether plural endings should count as separate words.

That choice affects the summary. Keeping every surface spelling separate increases the list length. Excessive removal of endings can assign one code to words with different meanings.

Stemming removes or changes word endings using fixed rules. In linguistics, a stem is the part of a word that an ending attaches to. In this lesson, a stem is whatever string these rules return. It is a comparison key. It may or may not be a word.

Note

The author created the fictional Riverton Workforce Lab, its job board, and its training flyer for teaching. SnowballC produces the stems locally.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • describe what a stemmer does.
  • find stems that are not dictionary words.
  • find collisions, where several words share one stem.
  • name related words that a suffix rule leaves apart.

Build the Riverton vocabulary

The vocabulary build uses readr, dplyr, tibble, tidytext, SnowballC, hunspell, purrr, and stringr. A word type is one distinct lowercased form in these 28 text units. The count uses Lesson 15’s tokenizers::tokenize_words() rule. A different tokenizer would change the count.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
library(tidytext)
library(SnowballC)
library(hunspell)

sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

vocabulary <- sentences |>
  select(sentence_id, text) |>
  unnest_tokens(word, text) |>
  distinct(word) |>
  arrange(word) |>
  pull(word)

known_to_hunspell <- \(words) map_lgl(hunspell(words), \(result) length(result) == 0L)

stemmed_vocabulary <- tibble(
  word = vocabulary,
  stem = wordStem(vocabulary, language = "en")
) |>
  mutate(
    changed = word != stem,
    stem_in_dictionary = known_to_hunspell(stem)
  )
Five Riverton word types after stemming
Word type Stem Changed Stem in dictionary
provided provid TRUE FALSE
required requir TRUE FALSE
shifts shift TRUE TRUE
skills skill TRUE TRUE
training train TRUE TRUE

The stemmer changes 43 of the 96 word types. shifts becomes shift, a useful result for a count that should group singular and plural forms. required becomes requir, which is not a word a reader should see.

Watch the rule on a nearby word

SnowballC’s English stemmer, often called Porter2, uses fixed rules. The word requires is absent from the Riverton vocabulary. It illustrates the same ending removal as required.

example_stems <- tibble(
  word = c("shifts", "requires", "required"),
  stem = wordStem(word, language = "en")
)
A rule can produce useful stems and non-word stems
Word Snowball stem
shifts shift
requires requir
required requir

Treat the returned stem as a machine comparison string. Do not present it as the word’s dictionary form.

Hunspell uses a spelling dictionary and affix rules. SnowballC returns stems without consulting a dictionary. It can therefore return requir, provid, and schedul.

Count stems that are words and stems that are not

The next table separates changed stems that are dictionary words from changed stems that are unknown to hunspell.

stem_quality <- stemmed_vocabulary |>
  summarise(
    word_types = n(),
    changed_by_stemmer = sum(changed),
    unchanged_by_stemmer = sum(!changed),
    changed_to_dictionary_word = sum(changed & stem_in_dictionary),
    changed_to_non_word = sum(changed & !stem_in_dictionary),
    .groups = "drop"
  )

non_word_examples <- stemmed_vocabulary |>
  filter(changed, !stem_in_dictionary) |>
  arrange(word) |>
  slice_head(n = 8)
Stemming outcomes for the Riverton vocabulary
Word types Changed Unchanged Changed to dictionary word Changed to non-word
96 43 53 21 22
Examples of non-word stems in the Riverton vocabulary
Word type Stem Changed Stem in dictionary
able abl TRUE FALSE
applicants applic TRUE FALSE
apply appli TRUE FALSE
certificate certif TRUE FALSE
certification certif TRUE FALSE
communication communic TRUE FALSE
daytime daytim TRUE FALSE
essential essenti TRUE FALSE

Among these 96 word types, 21 transformed forms are dictionary words. Another 22 transformed forms are unknown to hunspell. A dictionary-word stem can still be an unsuitable grouping key for a workforce question.

Find collisions

A collision happens when different word types reduce to the same stem. A collision can help when the words are close enough for the question. It can also hide a useful distinction.

collision_groups <- stemmed_vocabulary |>
  group_by(stem) |>
  filter(n() > 1L) |>
  summarise(
    word_types = paste(word, collapse = ", "),
    n_words = n(),
    .groups = "drop"
  ) |>
  arrange(stem)
Snowball collisions in the Riverton vocabulary
Stem Word types that share it Word types
certif certificate, certification 2
includ included, includes 2
schedul schedule, schedules 2
skill skill, skills 2

The four collision groups contain eight word types. schedule and schedules are a helpful group. certificate and certification may or may not be close enough, depending on whether the Lab counts credentials or training topics.

Find misses

A miss is a related pair that receives different stems. The team selected three pairs with suspected differences, not a survey sample. These examples show limitations of suffix rules.

miss_pairs <- tribble(
  ~word_a, ~word_b, ~reason,
  "applicants", "apply", "Both point to applying, but their stems differ.",
  "paid", "pays", "Snowball strips suffixes. 'Paid' has no suffix to strip, so no rule can reach 'pay'.",
  "work", "workers", "'-er' names the person doing the activity. Whether to group them depends on the question."
) |>
  mutate(
    stem_a = wordStem(word_a, language = "en"),
    stem_b = wordStem(word_b, language = "en"),
    same_stem = stem_a == stem_b
  )
Related Riverton word pairs with different Snowball stems
Word A Word B Reason for comparison Stem A Stem B Same stem
applicants apply Both point to applying, but their stems differ. applic appli FALSE
paid pays Snowball strips suffixes. ‘Paid’ has no suffix to strip, so no rule can reach ‘pay’. paid pay FALSE
work workers ‘-er’ names the person doing the activity. Whether to group them depends on the question. work worker FALSE

All three selected pairs receive different stems. Suffix rules can miss irregular forms and relationships whose meaning depends on more than an ending.

What to remember

  • Stemming removes or changes word endings with fixed rules.
  • The Snowball English stemmer is a Porter2 rule system.
  • SnowballC changed 43 of the 96 Riverton word types.
  • In this vocabulary, 22 changed stems are non-words.
  • Collisions and selected misses both need human judgment.

Stems can provide internal keys for approximate grouping. A public list should show recognizable words. A person must examine disputed groups.

Sources