Skip to main content

Cleaning raw text

Repair common damage without pretending nothing was lost

Learn how HTML remnants, whitespace, invisible characters, and mojibake change raw text in R.

The cleaning set contains saved job-board text, flyer lines, and one accented customer-service comment. The lesson retains the unrelated comment because it contains an accented word.

Copied text can contain HTML tags, layout spacing, and invisible characters inside words. The wrong character encoding can also corrupt accented letters.

Cleaning means changing raw text so a later step can compare, count, or read it more consistently. Each repair needs a record of what it changed and what it cannot recover.

Note

The Riverton sources are fictional. The customer-service comments come from a separate teaching fixture.

TipWhat you will learn

This lesson prepares you to:

• remove HTML while preserving readable text.
• explain what whitespace squishing keeps and discards.
• detect zero-width and control characters.
• describe why mojibake repair can fail.

Remove HTML remnants

The HTML example starts with the comment file, display tables, text checks, and an HTML parser. HTML is the markup language that names page parts such as headings, paragraphs, and lists.

library(readr)
library(dplyr)
library(tibble)
library(stringr)
library(rvest)

feedback <- read_csv(
  "data/customer_feedback.csv",
  na = character(),
  col_types = cols(
    feedback_id = col_character(),
    submitted_at = col_character(),
    channel = col_character(),
    rating = col_character(),
    comment = col_character()
  )
)

entity_html <- "<p>Training &amp; tools&nbsp;provided.</p>"
regex_tag_stripped <- str_remove_all(entity_html, "<[^>]+>")
html_text <- rvest::html_text2(rvest::read_html(entity_html))

Entity handling after two HTML-cleaning approaches

Method	Result
Remove tags with a regular expression	Training & tools provided.
Read HTML with rvest	Training & tools provided.

html_text2() converts HTML into readable text and decodes entities such as & and  . A regular expression can remove visible tags, but it does not decode those entities. Keep the raw HTML when element boundaries still matter.

Normalize whitespace

Whitespace includes spaces, tabs, and line breaks. Lesson 21 covers the comparison behavior of str_squish(). Here the function replaces consecutive whitespace with one ordinary space and removes whitespace at both ends. This operation removes layout information.

workforce_sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

flyer_lines <- workforce_sentences |>
  filter(document_id == "F001") |>
  pull(text)

flyer_block <- paste(flyer_lines, collapse = "\n")
squished_flyer <- str_squish(flyer_block)

Whitespace squishing removes the flyer line layout

Version	Text
Flyer lines	RIVERTON SKILLS OPEN HOUSE DATA SUPPORT CERTIFICATE Paid training stipend Evening classes No prior experience required Apply by October 15
After str_squish()	RIVERTON SKILLS OPEN HOUSE DATA SUPPORT CERTIFICATE Paid training stipend Evening classes No prior experience required Apply by October 15

Squishing helps when accidental spaces would make two records look different. It also discards deliberate layout. The flyer heading, certificate heading, stipend, class time, requirement, and deadline become one long line.

Remove invisible characters

Unicode groups characters into categories. The format category, Cf, contains zero-width spaces that can occur inside a string without visible marks. Cc is the control category. In a stringr pattern, \\p{Cf} and \\p{Cc} name those categories.

zero_width_space <- intToUtf8(0x200B)
clean_label <- "data support"
damaged_label <- paste0("data", zero_width_space, " support")
label_repaired <- str_remove_all(damaged_label, "[\\p{Cf}]")

control_text <- str_c("shift", intToUtf8(0x0001), "starts soon")
control_marked <- str_replace_all(control_text, "[\\p{Cc}]", " ")
control_repaired <- str_squish(control_marked)
control_squished <- str_squish(control_text)

Invisible characters change a string even when it looks familiar

Example	Text	Character count
Clean label	data support	12
With zero-width space	data​ support	13
After repair	data support	12
Control after whitespace squish	shift starts soon	17
Control after category repair	shift starts soon	17

The two labels appear identical, but R treats them as different strings. Whitespace squishing does not remove the control character in the second example. The category repair removes that character. Keep an audit field if hidden characters help explain source damage.

Repair mojibake carefully

Mojibake is text damage caused by reading bytes with the wrong character encoding. UTF-8 is a common encoding for many languages. Windows-1252 is another encoding often seen in older Western-language files.

Lesson 21 showed café stored in two legal Unicode normal forms. Both forms are readable and both preserve the word. This section is about a separate failure: bytes read with the wrong decoder, or bytes replaced after a lossy read.

accented_word <- str_extract(feedback$comment[2], fixed("café"))
mojibake_word <- iconv(
  accented_word,
  from = "Windows-1252",
  to = "UTF-8"
)
repaired_word <- iconv(
  mojibake_word,
  from = "UTF-8",
  to = "Windows-1252"
)
replacement_text <- "caf\uFFFD"
encoding_examples <- tibble(
  version = c(
    "Correct UTF-8",
    "Read as Windows-1252",
    "Repaired",
    "Typed by hand to show a lossy read"
  ),
  text = c(accented_word, mojibake_word, repaired_word, replacement_text)
) |>
  mutate(
    code_points = vapply(
      text,
      \(value) str_c(utf8ToInt(value), collapse = ", "),
      character(1)
    )
  )

The word café after encoding damage and repair

Version	Text	Unicode code points
Correct UTF-8	café	99, 97, 102, 233
Read as Windows-1252	café	99, 97, 102, 195, 169
Repaired	café	99, 97, 102, 233
Typed by hand to show a lossy read	caf�	99, 97, 102, 65533

Reading UTF-8 bytes as Windows-1252 converts café into café. Both bytes remain available, so the reverse conversion restores the word. A lossy read instead replaces the source character with �, U+FFFD. The repair cannot infer the original letter from that replacement. Mojibake and U+FFFD are separate problems.

Cleaning is a recorded choice

Cleaning makes later comparison easier. It can also remove source structure, spacing, hidden characters, and evidence of encoding damage. Record each cleaning step. Keep the raw source. Store the cleaned version in a separate field.

Production pipelines count changed and removed items at each step. Analysts examine a sample of removed content for accidental loss of meaningful data. Aggressive cleaning can disproportionately remove names or phrases from specific languages or dialects. Those languages may use characters that the rule does not expect.

Production NLP typically uses established mojibake-repair tools, such as Python’s ftfy (“fixes text for you”), rather than manual iconv() conversions.

What to remember

• HTML parsers decode entities that tag removal does not decode.
• Whitespace squishing fixes accidental spacing and erases layout.
• Invisible characters and encoding damage can make matching strings unequal.
• Mojibake can be reversible, while U+FFFD means information was lost.
• Keep the raw source.
• Record every cleaning step.

Cleaned fields belong beside raw fields. Comparisons can use the cleaned copy, while the raw text keeps the evidence needed to explain the repair.

Sources

• rvest documentation for extracting text
• stringr documentation for whitespace helpers
• The Unicode Standard
• W3C Character Model for the World Wide Web