Finding paragraph boundaries

Keep layout units before text conversion removes them

sentences and paragraphs
paragraph segmentation
workforce research
Learn how blank lines and HTML block elements mark paragraphs in different file formats.

Help articles and job-board HTML store paragraph boundaries differently. The coordinator needs to preserve those boundaries before text cleanup removes them.

In a plain-text help article, a blank line can mark a new paragraph. HTML gives the team block elements for paragraph boundaries. The HTML standard does not restrict its paragraph definition to <p> elements. This lesson therefore uses a working rule rather than the specification’s full definition.

Note

The help articles and job board are local fictional fixtures.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • explain a paragraph as a layout unit.
  • split plain text on blank lines.
  • collect paragraph-like blocks from HTML with rvest.
  • compare paragraph counts across file formats.
  • explain why extraction must preserve block boundaries before tag removal.

Split plain text on blank lines

readr reads files, while purrr applies the same operations across files. stringr splits text, and tidyr expands list columns. dplyr and tibble prepare tables, and rvest reads HTML. A paragraph is a block of text grouped by layout. It may contain one sentence or several.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
library(tidyr)
library(rvest)

help_files <- list.files(
  "data/help_articles",
  pattern = "[.]txt$",
  full.names = TRUE
) |>
  sort()

split_blank_lines <- function(text) {
  str_split(str_trim(text), "\\r?\\n\\s*\\r?\\n+")[[1]] |>
    str_trim() |>
    discard(\(piece) identical(piece, ""))
}

help_articles <- tibble(
  file = basename(help_files),
  text = map_chr(help_files, read_file)
) |>
  mutate(paragraph = map(text, split_blank_lines))

help_paragraphs <- help_articles |>
  select(file, paragraph) |>
  unnest_longer(paragraph) |>
  group_by(file) |>
  mutate(paragraph_number = row_number()) |>
  ungroup()

help_counts <- help_paragraphs |>
  count(file, name = "paragraphs")

total_help_paragraphs <- sum(help_counts$paragraphs)
help_blank_line_counts <- str_count(help_articles$text, "\\r?\\n\\s*\\r?\\n+")
Paragraph counts in the plain-text help articles
Help article Paragraphs from blank-line rule
account-access.txt 1
delivery.txt 1
returns.txt 1
Plain-text paragraphs after splitting on blank lines
Help article Paragraph Paragraph number
account-access.txt Use the password reset link on the sign-in page. The link expires after 30 minutes for your security. 1
delivery.txt Standard delivery takes three to five business days. Tracking appears after the parcel leaves our warehouse. 1
returns.txt You can return an unused item within 30 days. Keep the receipt and use the prepaid label included with your order. 1

None of these help files contains a blank line. The rule therefore returns each complete file without demonstrating a split. The HTML example requires an explicit choice of boundary rule.

Take blocks from HTML

HTML records layout with elements. The code below reads the job-board page and keeps headings, paragraphs, and list items. The selector h1, h2, p, li names the block types the team wants to treat as paragraph-like units.

job_page <- read_html("data/workforce/job-board.html")
block_nodes <- html_elements(job_page, "h1, h2, p, li")

html_blocks <- tibble(
  tag = html_name(block_nodes),
  text = html_text2(block_nodes)
) |>
  mutate(block_number = row_number())

html_tag_counts <- html_blocks |>
  count(tag, name = "blocks") |>
  arrange(tag)

expected_tag_counts <- tibble(
  tag = c("h1", "h2", "li", "p"),
  blocks = c(1L, 6L, 22L, 12L)
)
Paragraph-like blocks captured from the job-board HTML
HTML tag Blocks captured
h1 1
h2 6
li 22
p 12
The first eight block elements from the job board
HTML tag Text Block number
h1 Riverton workforce opportunities 1
h2 Data analyst trainee 2
p Riverton Community Health 3
p Riverton 4
li Paid 12-week training is provided. 5
li No prior data experience is required. 6
li Evening schedules are available. 7
li Applicants need basic spreadsheet skills. 8

The HTML rule finds 41 blocks: 1 page heading, 6 job headings, 12 paragraph elements, and 22 list items. The job-detail sentences in the Riverton CSV came from those list items.

Show what tag removal loses

A common conversion removes tags and retains visible words. This operation can help word counting, but it also removes the markers needed for paragraph segmentation.

html_source <- read_file("data/workforce/job-board.html")
stripped_before_squish <- html_source |>
  str_remove_all("<[^>]+>")

paragraphs_after_tag_removal <- split_blank_lines(stripped_before_squish)
paragraphs_after_squish <- stripped_before_squish |>
  str_squish() |>
  split_blank_lines()

boundary_loss <- tibble(
  source = c(
    "HTML block elements",
    "Tags removed, whitespace preserved",
    "Tags removed, whitespace squished"
  ),
  paragraph_like_units = c(
    nrow(html_blocks),
    length(paragraphs_after_tag_removal),
    length(paragraphs_after_squish)
  )
)
Whitespace cleanup collapses the tag-stripped text
Rule Paragraph-like units
HTML block elements 41
Tags removed, whitespace preserved 14
Tags removed, whitespace squished 1

Tag removal alone leaves 14 blank-line chunks because the source file still has line breaks and indentation. Those chunks reflect source layout around titles, job cards, and lists rather than paragraphs. str_squish() removes the remaining blank-line boundaries and leaves one long text string. For this file, paragraph segmentation must preserve structure before whitespace cleanup.

What to remember

  • A paragraph is a layout block, so the right rule depends on the file format.
  • These help files have no blank lines, so the blank-line rule cannot show much.
  • The chosen HTML selector captures 41 block-like units. Those units contain headings and list items.
  • Squishing whitespace after tag removal collapses the text to one unit.

The help files and the job-board HTML need different boundary rules. Here the reported counts are layout counts, not a grammar claim about where paragraphs must begin.

Sources