library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
library(tokenizers)
library(hunspell)
sentences <- read_csv(
"data/workforce/workforce_sentences.csv",
na = character(),
col_types = cols(
sentence_id = col_character(),
document_id = col_character(),
source_line = col_character(),
text = col_character(),
reference_label = col_character(),
uncertainty = col_character(),
annotator_id = col_character(),
rationale = col_character(),
codebook_version = col_character(),
codebook_hash = col_character(),
derived = col_character(),
transformation = col_character()
)
)
document_counts <- sentences |>
count(document_id, name = "sentence_count") |>
arrange(document_id)Examining word parts with a dictionary
Use hunspell analyses without mistaking them for contextual morphology
word parsing
morphology
workforce research
Learn what an English spelling dictionary can and cannot say about stems, affixes, ambiguity, and context in workforce text.
Twenty-eight reviewed text units come from a job board page and a training flyer. The team wants to know whether shifts and shift should count as the same word when it summarizes the vocabulary.
That question matters because a worker may search for training, while a job posting may say shifts, certification, or certificate. The team needs a method for word-part analysis before it decides which forms to group.
Morphology is the internal structure of words. A stem gives the main meaning, and affixes are added pieces such as plural s or past-tense ed. English marks a small amount of grammar this way, so English speakers often notice word parts only when a form looks odd.
Note
The author created the fictional Riverton Workforce Lab, its job board, and its training flyer for teaching. The local hunspell package produces the dictionary output.
TipWhat you will learn
By the end of this lesson, you will be able to:
- describe a stem and an affix in plain language.
- read a dictionary-based word analysis.
- distinguish a useful decomposition from a misleading one.
- find unanalyzed and ambiguous word types in a small vocabulary.
- explain why dictionary analysis differs from contextual morphology in sentences.
This lesson examines an English spelling dictionary’s analysis of isolated word forms. Contextual morphology instead assigns grammatical feature bundles to tokens within sentences. A tagger or morphologizer produces features such as Number=Plur or Tense=Past. This page covers dictionary morphology, not contextual morphological tagging. Lesson 18 shows token-level features in its feats column.
Read the workforce sentences
The code loads readr for files and dplyr and tibble for tables. It also loads purrr for list operations and stringr for text checks. The tokenizers package splits words, and hunspell performs dictionary analysis. The code then reads the 28 reviewed workforce text units.
| Document ID | Text units |
|---|---|
| F001 | 6 |
| J001 | 4 |
| J002 | 4 |
| J003 | 4 |
| J004 | 3 |
| J005 | 3 |
| J006 | 4 |
The flyer has six rows, and the job-detail documents have 22 rows. The sample fits in a small table, but the vocabulary is large enough for several kinds of word behavior.
Ask the dictionary for analyses
hunspell_analyze() compares each word with an English spelling dictionary and its affix rules. The output is compact. st: gives a stem, and fl: gives a flag used by the dictionary’s rule system. A flag is a code in the dictionary, not a reader-friendly grammar label.
In the code, \(value) is R shorthand for a small function. Here it means that the function receives each vector as value and joins its elements into one string.
example_words <- c(
"shifts",
"requires",
"certification",
"running",
"forklift"
)
word_analyses <- hunspell_analyze(example_words)
word_stems <- hunspell_stem(example_words)
analysis_table <- tibble(
word = example_words,
analyses = map_chr(word_analyses, \(value) paste(value, collapse = "; ")),
stems = map_chr(word_stems, \(value) paste(value, collapse = ", "))
)| Word | Dictionary analyses | Returned stems |
|---|---|---|
| shifts | st:shift fl:S | shift |
| requires | fl:A st:quire fl:S | quire |
| certification | st:certification; st:certificate fl:N | certification, certificate |
| running | st:running | running |
| forklift | st:forklift | forklift |
For shifts, the dictionary finds the stem shift and marks the plural form with flag S.
The entry for requires illustrates a misleading analysis. Hunspell can satisfy its internal rules while giving a false word analysis. Its rules permit re- before dictionary words, and the dictionary contains quire. That combination does not explain the origin of require. English adopted the whole word from Old French requerre, from Latin requirere. A valid dictionary operation is not necessarily a historical explanation.
certification has two analyses. It can stand as certification, or it can be connected to certificate through another rule. The letters alone do not choose between those analyses.
The dictionary lists running as a separate entry. The analyzer therefore returns running without further decomposition. The dictionary can store a common form directly.
Count dictionary analyses
The team applies the dictionary check to the vocabulary of the 28 text units. A word type is one distinct word form after lowercasing. If training appears several times, it counts once in this vocabulary.
tokens_by_sentence <- tokenize_words(
sentences$text,
lowercase = TRUE,
strip_punct = TRUE
)
vocabulary <- sort(
unique(unlist(tokens_by_sentence, use.names = FALSE))
)
vocabulary_analyses <- hunspell_analyze(vocabulary)
vocabulary_summary <- tibble(
word = vocabulary,
analyses = vocabulary_analyses,
analysis_count = map_int(analyses, length),
recognized = analysis_count > 0L
)
summary_counts <- vocabulary_summary |>
summarise(
word_types = n(),
analyzed_by_dictionary = sum(recognized),
not_analyzed = sum(!recognized),
more_than_one_analysis = sum(analysis_count > 1L),
.groups = "drop"
)
unanalyzed_words <- vocabulary_summary |>
filter(!recognized) |>
pull(word)
multi_analysis_words <- vocabulary_summary |>
filter(analysis_count > 1L) |>
pull(word)| Word types | Types with an analysis | Types without an analysis | More than one dictionary analysis |
|---|---|---|---|
| 96 | 91 | 5 | 8 |
The dictionary gives analyses for 91 of the 96 word types. Five receive no analysis: three numbers, the month october, and the place name riverton. Eight word types have multiple analyses under the dictionary rules. Their spelling alone does not select one analysis.
Examine the ambiguous cases
Ambiguity alone does not indicate a software error. A single spelling can have several plausible dictionary analyses. The next table shows Riverton word types with more than one hunspell analysis.
ambiguous_words <- vocabulary_summary |>
filter(analysis_count > 1L) |>
transmute(
word,
analyses = map_chr(analyses, \(value) paste(value, collapse = "; "))
)| Word | Dictionary analyses |
|---|---|
| available | st:available; st:avail fl:B |
| certification | st:certification; st:certificate fl:N |
| communication | st:communication; st:communicate fl:N |
| customer | st:customer; st:custom fl:R |
| evening | st:evening; st:even fl:G |
| outdoors | st:outdoors; st:outdoor fl:S |
| provided | st:provided; st:provide fl:D |
| training | st:training; st:train fl:G |
The table shows decisions that need checks. The dictionary supplies possible word structures, not final labels. A person must decide whether each analysis addresses the research question.
Keep dictionary analysis separate from contextual morphology
English word endings express some grammatical information. For example, shifts marks plural, and provided has a form related to tense or aspect. Word order and nearby helper words express other information. Some languages mark case, person, tense, and number within one token. A tool based on English spelling assumptions will miss much of that structure.
Hunspell is a spelling dictionary with affix rules, not a universal morphological analyzer. As the requires example shows, a result can satisfy dictionary rules yet misrepresent word structure.
This lesson examines isolated spellings, not tokens within sentences. Contextual morphology usually assigns features such as Number=Plur or Tense=Past to sentence tokens. Taggers or morphologizers can produce these features, which a treebank stores in its feats column. Lesson 18 displays them in parser output. This page instead applies a spelling dictionary to a word-type list.
What to remember
- Morphology studies the parts inside words.
- A stem gives the main meaning. Affixes modify the form.
- Dictionary rules can produce useful analyses and misleading ones.
- One word form can have several analyses under the dictionary rules.
- Hunspell applies spelling-style rules to isolated words. It does not produce contextual morphological features in sentences.
The team can group shifts with shift for a vocabulary count. Every returned stem still requires human judgment about the text. Contextual grammatical feature assignment is a separate task.