Making text comparable

Normalize carefully and keep the display text intact

word processing

normalization

workforce research

Learn how case, spacing, punctuation, and Unicode normalization change workforce text.

Related phrases across the 28 text units can differ in formatting. The flyer heading RIVERTON SKILLS OPEN HOUSE uses capitals, while job details use ordinary case. Pasted text may also contain extra spacing and accented characters. The team examines each normalization step’s effect before applying it.

Before counting matches, the team has to decide which small differences to ignore. Each edit can make comparison easier and also erase information.

Normalization applies edits before text comparison. Common operations change case, trim spaces, remove punctuation, or convert Unicode characters to a consistent form.

Note

The author created the fictional Riverton Workforce Lab, its job board, and its training flyer for teaching. The text files are local and fixed.

What you will learn

By the end of this lesson, you will be able to:

explain the purpose of normalization before comparison.
examine the effects of lowercasing and space repair.
examine punctuation removal with stringr’s ICU regex engine.
compare two Unicode forms of the same visible word.

Load text with an accent check

The readr package opens CSV files, and dplyr and tibble handle tables. The stringr package handles text operations, while stringi normalizes Unicode. The purrr package applies checks to each item. Inputs contain the Riverton text and feedback with the word café.

library(readr)
library(dplyr)
library(tibble)
library(stringr)
library(stringi)
library(purrr)

sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

feedback <- read_csv(
  "data/customer_feedback.csv",
  na = character(),
  col_types = cols(
    feedback_id = col_character(),
    submitted_at = col_character(),
    channel = col_character(),
    rating = col_character(),
    comment = col_character()
  )
)

The accent check matters because normalization can change text that looks stable on screen. The visible word may stay the same while the stored code points change.

Lowercase text and repair spaces

Case folding prepares text for case-insensitive comparison. This example uses lowercasing, which differs from full Unicode case folding. The table contains a Riverton string and a copy of s002 with tripled spaces to simulate pasted text. It also compares US with us.

padded_requirement <- str_replace_all(
  sentences$text[sentences$sentence_id == "s002"],
  " ",
  "   "
)
padded_requirement <- paste0("  ", padded_requirement, "  ")

case_space_examples <- tibble(
  example = c(
    "Flyer heading",
    "Padded requirement sentence",
    "Warning pair, capital",
    "Warning pair, lower"
  ),
  raw_text = c(
    sentences$text[sentences$sentence_id == "s023"],
    padded_requirement,
    "US",
    "us"
  ),
  lowercased = str_to_lower(raw_text),
  squished = str_squish(raw_text)
)

Lowercasing and spacing changes
Example	Raw text	Lowercased	Spacing normalized
Flyer heading	RIVERTON SKILLS OPEN HOUSE	riverton skills open house	RIVERTON SKILLS OPEN HOUSE
Padded requirement sentence	No prior data experience is required.	no prior data experience is required.	No prior data experience is required.
Warning pair, capital	US	us	US
Warning pair, lower	us	us	us

Lowercasing makes RIVERTON SKILLS OPEN HOUSE easier to compare with ordinary case. It also makes US and us the same string. str_squish() removes leading and trailing spaces and replaces repeated spaces with one space.

Remove punctuation only when it is safe

Punctuation removal can make matching easier, but it can damage units such as 12-week. The next table uses Riverton strings that contain hyphens, apostrophes, and periods.

punctuation_examples <- tibble(
  sentence_id = c("s001", "s015", "s017"),
  raw_text = sentences$text[match(sentence_id, sentences$sentence_id)],
  without_punctuation = str_remove_all(raw_text, "[[:punct:]]")
)

unicode_punctuation_check <- str_remove_all("“paid—training”", "[[:punct:]]")

Punctuation removal can remove useful distinctions
Sentence ID	Raw text	After punctuation removal
s001	Paid 12-week training is provided.	Paid 12week training is provided
s015	On-the-job training is provided.	Onthejob training is provided
s017	A valid driver’s license is required.	A valid drivers license is required

Punctuation removal deletes the hyphen in 12-week and the apostrophe in driver's. The function str_remove_all() uses stringr’s ICU engine. Its [[:punct:]] class covers Unicode punctuation, such as curly quotes and em dashes from word processors.

Normalize Unicode forms

Unicode stores text as code points. In café, the letter é can occupy one code point or combine e with a separate accent. NFC keeps the composed form. NFD separates the accent into its own code point.

cafe_word <- str_extract(feedback$comment[2], "café")
cafe_nfc <- stri_trans_nfc(cafe_word)
cafe_nfd <- stri_trans_nfd(cafe_word)

unicode_forms <- tibble(
  form = c("NFC", "NFD"),
  text = c(cafe_nfc, cafe_nfd),
  code_point_count = str_length(text),
  bytes_utf8 = nchar(text, type = "bytes"),
  code_points = map_chr(text, \(value) paste(utf8ToInt(value), collapse = ", "))
)

Two Unicode forms of the visible word ‘café’
Form	Text	Code points	Bytes as UTF-8	Code point values
NFC	café	4	5	99, 97, 102, 233
NFD	café	5	6	99, 97, 102, 101, 769

The forms look the same, but identical() detects different stored strings. Both forms show four letters. NFC uses 4 code points and 5 UTF-8 bytes. NFD uses 5 code points and 6 bytes. Its accent occupies a separate code point after e.

Keep two columns

Store at least two versions: raw text for display and normalized text for comparison. Normalize case, spaces, or Unicode form as the task requires.

display_and_match <- sentences |>
  filter(sentence_id %in% c("s001", "s023")) |>
  transmute(
    sentence_id,
    display_text = text,
    comparison_text = text |>
      stri_trans_nfc() |>
      str_squish() |>
      str_to_lower()
  )

Keeping display text separate from comparison text
Sentence ID	Display text	Comparison text
s001	Paid 12-week training is provided.	paid 12-week training is provided.
s023	RIVERTON SKILLS OPEN HOUSE	riverton skills open house

Normalize text for comparison. Keep raw text for display. Some normalization operations are irreversible. After lowercasing, us does not reveal whether the source was US or us. Choose NFC for the comparison column. This recommended web-text form gives equivalent character sequences a consistent representation across systems.

What to remember

Normalization makes text easier to compare.
Lowercasing can merge strings that mean different things.
ICU punctuation classes contain more than keyboard punctuation.
Unicode forms can look the same while storing different code points.
Keep the display text and comparison text separate.
Raw text preserves information that normalization removes.

A normalized column assists matching, not display. Show readers the original Riverton text that the Lab collected.

Load text with an accent check

Lowercase text and repair spaces

Remove punctuation only when it is safe

Normalize Unicode forms

Keep two columns

What to remember

Sources