Building a text corpus

Organize separate documents as a collection

source data loading
corpus
data quality
Learn how to collect text, attach useful labels, and examine a corpus before analysis.

A person who cannot access an account needs the right help article. The help-center editor has three separate files about deliveries, returns, and account access. The password-reset instructions are somewhere in those files, but each search result also needs to retain its topic and source. That becomes harder as loose documents accumulate.

The editor needs an organized collection called a corpus. Its plural is corpora. A corpus keeps text documents together and usually stores information describing each one. We will build a three-document corpus, examine it for empty or duplicate records, and use it to find the password-reset article.

Note

The author wrote these help articles for this example. They do not come from a real support system.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • explain what a corpus is.
  • read several text files into R.
  • decide what counts as one document.
  • state the sampling frame and exclusions.
  • keep source information with each document.
  • find near-duplicate documents and keep their families together.
  • examine a corpus for common problems.

Decide what one document means

Before collecting anything, the editor must choose the document unit, the amount of text that will count as one document. Depending on the question, it might be:

  • one help article.
  • one customer message.
  • one news story.
  • one interview.
  • one paragraph within a longer report.

The right unit depends on the question. Because the editor wants to compare and search help articles, each complete article will be one document. Treating each sentence as a separate document would detach the sentence from the rest of its article.

Before loading real documents, confirm the right to use them. Record the license or permission, collection method, expected audience, and privacy risk. Record whether the people represented could reasonably expect you to use their data this way. The example files are synthetic and contain no personal records or third-party text.

The example uses three plain-text files, files that store characters without page layout or styling. list.files() asks R to find file names ending in .txt.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
library(digest)

article_paths <- sort(
  list.files(
    path = "data/help_articles",
    pattern = "[.]txt$",
    full.names = TRUE
  )
)

expected_files <- c(
  "account-access.txt",
  "delivery.txt",
  "returns.txt"
)

basename(article_paths)
[1] "account-access.txt" "delivery.txt"       "returns.txt"

R prints all three file names in alphabetical order. We retain those names as labels so each result identifies its source file.

Read every document

Finding the file names does not yet load their contents. read_lines() reads the text inside one file. These files use UTF-8, a common way to store characters from many languages. A function is a named set of steps. The small function below preserves line breaks in the source copy.

expected_line_counts <- c(
  "account-access.txt" = 1L,
  "delivery.txt" = 1L,
  "returns.txt" = 1L
)

read_document <- function(path) {
  lines <- read_lines(path)
  file_name <- basename(path)
  stopifnot(identical(length(lines), unname(expected_line_counts[file_name])))

  str_c(lines, collapse = "\n")
}

source_documents <- article_paths |>
  set_names(basename(article_paths)) |>
  map_chr(read_document)

analysis_documents <- str_replace_all(
  source_documents,
  "[\r\n]+",
  " "
)

source_documents
                                                                                                  account-access.txt 
             "Use the password reset link on the sign-in page. The link expires after 30 minutes for your security." 
                                                                                                        delivery.txt 
      "Standard delivery takes three to five business days. Tracking appears after the parcel leaves our warehouse." 
                                                                                                         returns.txt 
"You can return an unused item within 30 days. Keep the receipt and use the prepaid label included with your order."

map_chr() runs read_document() once for each path and returns a character vector. The result contains three source copies, and the file names remain attached. A separate analysis_documents object replaces line breaks with spaces. The unchanged source copy preserves paragraph boundaries so later steps can use them.

The str_length() check confirms that no document is empty. An empty result could indicate a loading failure or a blank source. Investigate either case before analysis.

Put text and labels together

The text alone does not provide its source or context. Each document also needs metadata, information about the document rather than its main text. A source file, topic, publication date, or author can be metadata.

We will place the text and its labels in a tibble, with one row per document.

hash_document <- function(path) {
  lines <- read_lines(path)
  file_name <- basename(path)
  stopifnot(identical(length(lines), unname(expected_line_counts[file_name])))

  digest::digest(
    paste(lines, collapse = "\n"),
    algo = "md5",
    serialize = FALSE
  )
}

corpus <- tibble(
  document_id = str_remove(
    basename(article_paths),
    "[.]txt$"
  ),
  topic = c("account access", "delivery", "returns"),
  source_file = basename(article_paths),
  source_hash = map_chr(
    article_paths,
    hash_document
  ),
  source_text = unname(source_documents),
  analysis_text = unname(analysis_documents)
)
The documents and their metadata
Document ID Topic Source file
account-access account access account-access.txt
delivery delivery delivery.txt
returns returns returns.txt

The document_id gives every row a stable label. The topic identifies the article’s subject without requiring the reader to open the full text. The source file records the text’s origin. The source_hash value changes when a file changes and helps detect accidental edits. A hash does not prove authorship or trustworthiness. Keeping these fields together preserves each document’s source and context.

Say what the collection is a sample of

A corpus needs a defined collection frame, even when nobody selects a sample. Record the population, selection rule, and exclusions.

These three articles form a census within the stated frame, not a sample. The collection contains every active, non-archived article that this fictional help center published by the collection date. It is not a sample of the support questions that people actually ask.

That distinction changes what the corpus can show later. A count over a census of published articles describes what the help center wrote. It does not describe what readers needed and did not find.

collection_record <- tibble(
  field = c(
    "frame",
    "selection rule",
    "collected on",
    "excluded by rule",
    "known coverage gap",
    "language",
    "unit"
  ),
  value = c(
    "active, non-archived articles from one fictional help center",
    "all active articles on the collection date, no sampling",
    "2026-08-28",
    "drafts, archived articles, and reader messages",
    "questions people asked that no article answers",
    "English (en)",
    "one complete article"
  )
)
The collection’s limits and exclusions
Field Recorded value
frame active, non-archived articles from one fictional help center
selection rule all active articles on the collection date, no sampling
collected on 2026-08-28
excluded by rule drafts, archived articles, and reader messages
known coverage gap questions people asked that no article answers
language English (en)
unit one complete article

Write this record before analysis. Otherwise, a researcher may describe the frame later in terms that exaggerate what the results can show.

Add a simple measurement

A quick count can reveal a document that loaded as empty or unexpectedly long. The function below separates text wherever it finds whitespace, then counts the resulting pieces. Here, whitespace means spaces, tabs, line breaks, and non-breaking spaces matched by [[:space:]]+.

count_whitespace_tokens <- function(text) {
  tokens <- str_split_1(str_trim(text), "[[:space:]]+")

  sum(str_length(tokens) > 0)
}

corpus <- corpus |>
  mutate(
    whitespace_token_count = map_int(
      analysis_text,
      count_whitespace_tokens
    )
  )
Whitespace-token counts for the three documents
Document ID Topic Whitespace-token count
account-access account access 18
delivery delivery 16
returns returns 21

The three articles contain 18, 16, and 21 whitespace-separated pieces. These are often called tokens, not universal word counts. Languages without spaces between words, contractions, punctuation, and specialized tokenizers can produce different results. The counts are quick checks, not quality scores.

Examine the corpus before using it

The editor now has rows, labels, and text. Before trusting the collection, we confirm that:

  • every source file still exists.
  • every document has text.
  • document IDs are unique.
  • no two documents contain exactly the same text.
  • the topic and source file are present.
quality_checks <- tibble(
  check = c(
    "source_files_exist",
    "no_empty_text",
    "unique_document_ids",
    "no_exact_duplicate_text",
    "metadata_is_complete"
  ),
  passed = c(
    all(file.exists(article_paths)),
    all(str_length(str_trim(corpus$source_text)) > 0),
    n_distinct(corpus$document_id) == nrow(corpus),
    n_distinct(corpus$source_text) == nrow(corpus),
    !any(
      is.na(
        corpus |>
          select(topic, source_file, source_hash)
      )
    )
  )
)

quality_checks
# A tibble: 5 × 2
  check                   passed
  <chr>                   <lgl> 
1 source_files_exist      TRUE  
2 no_empty_text           TRUE  
3 unique_document_ids     TRUE  
4 no_exact_duplicate_text TRUE  
5 metadata_is_complete    TRUE

Every passed value is TRUE, so the corpus passes these structural checks. They do not establish that the articles are accurate, useful, or representative. They show only that the three expected files produced three labeled, nonempty, distinct records.

Use the corpus as a collection

The collection is ready for the editor’s original question. All three documents share a structure, so one search can examine them all. This search finds the exact phrase "password reset".

matching_documents <- corpus |>
  filter(
    str_detect(
      analysis_text,
      fixed("password reset")
    )
  ) |>
  select(document_id, topic, analysis_text)
The document that mentions a password reset
Document ID Topic Text
account-access account access Use the password reset link on the sign-in page. The link expires after 30 minutes for your security.

The account-access article is the single match. Its topic, document ID, and text remain together. The editor can examine the result and its source file. Maintaining that connection becomes harder as a corpus grows.

NoteTechnical detail: a corpus can have a special R class

An R class tells R what kind of object it receives and which operations belong with it.

Packages such as quanteda and tm provide dedicated corpus objects with tools for metadata and text processing. A tibble is enough for this lesson because its rows and columns are visible and easy to examine. The researcher can convert these records to a package-specific corpus later.

A near duplicate is not an exact duplicate

A fourth file arrives. The support team copied the account-access article into a seasonal FAQ, replaced a hyphen, and added a courtesy line. To a reader it is the same article. To an exact comparison it is a new document.

The candidate below is written into the lesson so its wording is visible.

candidate_text <- paste(
  "Use the password reset link on the sign in page.",
  "The link expires after 30 minutes for your security.",
  "Contact the help desk if the message does not arrive."
)

incoming <- tibble(
  document_id = c(corpus$document_id, "seasonal-faq-account"),
  analysis_text = c(corpus$analysis_text, candidate_text)
)

exact_duplicate_found <- n_distinct(incoming$analysis_text) < nrow(incoming)

normalize_for_matching <- function(text) {
  text |>
    str_to_lower() |>
    str_replace_all("[^a-z0-9]+", " ") |>
    str_squish()
}

normalized <- normalize_for_matching(incoming$analysis_text)
similarity <- function(left, right) {
  distance <- as.integer(adist(left, right))
  1 - distance / max(str_length(left), str_length(right))
}
containment <- function(left, right) {
  left_words <- unique(str_split_1(left, " "))
  right_words <- unique(str_split_1(right, " "))
  shorter <- if (length(left_words) <= length(right_words)) {
    left_words
  } else {
    right_words
  }
  longer <- if (length(left_words) <= length(right_words)) {
    right_words
  } else {
    left_words
  }

  mean(shorter %in% longer)
}

pair_scores <- tibble(
  left = incoming$document_id[
    c(1L, 1L, 1L, 2L, 2L, 3L)
  ],
  right = incoming$document_id[
    c(2L, 3L, 4L, 3L, 4L, 4L)
  ],
  similarity = map2_dbl(
    normalized[c(1L, 1L, 1L, 2L, 2L, 3L)],
    normalized[c(2L, 3L, 4L, 3L, 4L, 4L)],
    similarity
  ),
  containment = map2_dbl(
    normalized[c(1L, 1L, 1L, 2L, 2L, 3L)],
    normalized[c(2L, 3L, 4L, 3L, 4L, 4L)],
    containment
  )
)
All six document pairs, most similar first
Document Compared with Character similarity Word containment
account-access seasonal-faq-account 0.651 1.000
account-access delivery 0.311 0.125
delivery seasonal-faq-account 0.283 0.125
delivery returns 0.277 0.125
returns seasonal-faq-account 0.276 0.200
account-access returns 0.268 0.250

The exact-duplicate check returns FALSE because no two documents are byte-identical. Both subsequent comparisons find the same pair to be similar, but their similarity values differ.

Character similarity is 0.65 for the reprint. The added courtesy line reduces similarity because each extra character counts as a difference. Word containment measures whether one document’s words all appear in the other. It returns 1.00. Containment can find a copy with added text that reduces character similarity.

The thresholds, 0.6 and 0.9, are choices rather than discoveries. Record each threshold with its measure. The same document pair can qualify as a near duplicate under one measure but not another.

Both comparisons are quadratic in the number of documents. The computational cost is negligible for four documents but prohibitive for four million.

Keep document families together and report removals

A near duplicate creates a second problem. If one copy enters training and another enters evaluation, shared wording or memorized examples may inflate the evaluation result. This lesson trains no model and cannot show which mechanism would cause that inflation.

Group related documents into a family. Assign the whole family to the same data split.

families <- incoming |>
  mutate(
    family_id = if_else(
      document_id %in% c("account-access", "seasonal-faq-account"),
      "account-access-family",
      document_id
    ),
    kept = document_id != "seasonal-faq-account"
  )

family_split <- families |>
  distinct(family_id) |>
  mutate(
    split = if_else(
      family_id == "returns",
      "evaluation",
      "development"
    )
  )

assigned <- families |>
  left_join(family_split, by = join_by(family_id))
removed <- assigned |>
  filter(!kept)

split_is_family_consistent <- assigned |>
  summarise(splits = n_distinct(split), .by = family_id) |>
  pull(splits) |>
  max()
Four candidates, three kept, one removed with its family recorded
Document Family Split Kept
account-access account-access-family development TRUE
delivery delivery development TRUE
returns returns evaluation TRUE
seasonal-faq-account account-access-family development FALSE

The output prints the removed text in full:

removed$analysis_text
[1] "Use the password reset link on the sign in page. The link expires after 30 minutes for your security. Contact the help desk if the message does not arrive."

The process retained three of four candidates and prints the removed text in full. Read removed text to find incorrect deletions. Here, the removed copy adds a sentence about contacting the help desk. A careful editor might add that sentence to the original instead of discarding it.

The split_is_family_consistent check counts distinct splits per family. Evaluation requires one split per family.

A corpus is a set of choices

The successful search does not make the collection neutral. Someone decided which files belong, where a document ends, which metadata to keep, and which text to exclude. Every later result depends on those choices.

Record:

  1. the source of each document.
  2. the collection date.
  3. the meaning of one row.
  4. the collection frame and selection rule.
  5. the reasons for exclusions.
  6. the meaning of each metadata field.
  7. the license, permission, or consent to use the text.
  8. the privacy checks.
  9. any people or communities absent from the collection.
  10. the near-duplicate rule and threshold.
  11. the fingerprint of each unchanged source file.

What to remember

  • A corpus is an organized collection of text documents.
  • The document unit should match the question being asked.
  • Name the frame the collection came from, even when it is a census.
  • Metadata keeps each document connected to its source and context.
  • Preserve source text separately from text changed for analysis.
  • Search for empty files, duplicate IDs, and duplicate text before analysis.
  • Exact-match checks miss near duplicates, which need a stated threshold.
  • Related documents belong to one family and move between splits together.
  • A technically clean corpus can still be incomplete or unrepresentative.

The editor can find the password-reset answer together with its complete article. The report records the excluded seasonal copy and its family. The editor can examine the additional sentence.

For each document in a larger corpus, record its content, source, and reason for inclusion.

Sources