Finding sentence boundaries

Compare three ways to split text into sentences

sentences and paragraphs

sentence segmentation

workforce research

Learn why sentence segmentation can fail on abbreviations, decimals, initials, titles, and headings.

The coordinator counts sentences on a constructed page with flyer, pay, and job-board text. The lesson then compares sentence boundaries in the separate 28-row Riverton file.

The page contains Dr., a.m., $18.50, and a heading without a period. Those forms complicate sentence counts. A sentence boundary marks where one sentence ends and the next begins. The team must state its boundary rule before interpreting the count.

Note

The Riverton page and CSV rows are fictional teaching data.

What you will learn

By the end of this lesson, you will be able to:

explain why sentence segmentation is harder than splitting on periods.
compare an author-marked sentence count with three splitters.
examine the fragments each splitter creates.
use spaCy sentence identifiers from spacy_parse().
explain why the Riverton CSV already contains an upstream boundary choice.

Build a passage with difficult boundaries

readr reads the Riverton CSV, while dplyr and tibble build tables. purrr and stringr execute the small splitters. tokenizers supplies a package splitter, and spacyr calls a trained parser. A splitter is a rule or model that marks text boundaries.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
library(tokenizers)
library(spacyr)

sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

passage <- str_c(
  "RIVERTON SKILLS OPEN HOUSE\n",
  "Dr. Vale said shifts start at 6 a.m.\n",
  "Pay is $18.50 per hour.\n",
  "J. Rivera can apply by Friday\n",
  "Bring a photo ID."
)

expected_sentences <- c(
  "RIVERTON SKILLS OPEN HOUSE",
  "Dr. Vale said shifts start at 6 a.m.",
  "Pay is $18.50 per hour.",
  "J. Rivera can apply by Friday",
  "Bring a photo ID."
)

The five expected sentence boundaries
Expected sentence	Author-marked text
1	RIVERTON SKILLS OPEN HOUSE
2	Dr. Vale said shifts start at 6 a.m.
3	Pay is $18.50 per hour.
4	J. Rivera can apply by Friday
5	Bring a photo ID.

The expected-answer list has five sentences. Because that list and the examples share an author, agreement with it is a lesson check rather than an accuracy estimate. The first sentence is a flyer-style heading with no punctuation. The next three contain a title, an abbreviation, a decimal, and an initial.

Compare three splitters

The naive rule below splits text at every full stop. tokenizers uses a sentence-splitting rule from an R package. spaCy uses a trained English pipeline. spacy_parse() returns one row per token, and its sentence_id column records the boundary spaCy chose.

split_on_full_stop <- function(text) {
  text |>
    str_split(fixed(".")) |>
    pluck(1) |>
    str_trim() |>
    discard(\(piece) identical(piece, ""))
}

join_spacy_tokens <- function(tokens) {
  str_c(tokens, collapse = " ") |>
    str_squish() |>
    str_replace_all(fixed(" ."), ".") |>
    str_replace_all(fixed(" ,"), ",") |>
    str_replace_all(fixed("$ "), "$")
}

naive_pieces <- split_on_full_stop(passage)
tokenizer_pieces <- tokenize_sentences(passage)[[1]]

source("R/use-spacy.R")
pipeline <- use_project_spacy()
spacy_metadata <- spacy_pipeline_version()
spacy_tokens <- spacy_parse(
  setNames(passage, "test_passage"),
  pos = TRUE,
  lemma = TRUE
)
spacy_finalize()

spacy_pieces <- spacy_tokens |>
  as_tibble() |>
  summarise(piece = join_spacy_tokens(token), .by = sentence_id) |>
  pull(piece)

split_results <- bind_rows(
  tibble(splitter = "split on full stops", piece = naive_pieces),
  tibble(splitter = "tokenizers::tokenize_sentences()", piece = tokenizer_pieces),
  tibble(splitter = "spaCy sentence_id", piece = spacy_pieces)
) |>
  mutate(
    splitter = factor(
      splitter,
      levels = c(
        "split on full stops",
        "tokenizers::tokenize_sentences()",
        "spaCy sentence_id"
      )
    )
  ) |>
  arrange(splitter) |>
  group_by(splitter) |>
  mutate(
    piece_number = row_number(),
    exact_expected_sentence = piece %in% expected_sentences
  ) |>
  ungroup()

splitter_scores <- split_results |>
  summarise(
    pieces_found = n(),
    exact_expected_pieces = sum(exact_expected_sentence),
    .by = splitter
  ) |>
  mutate(
    expected_pieces = length(expected_sentences),
    count_difference = pieces_found - expected_pieces,
    matches_expected_count = pieces_found == expected_pieces
  )

Splitter counts compared with the five expected sentences
Splitter	Pieces found	Exact pieces	Expected pieces	Difference	Count matches
split on full stops	7	0	5	2	FALSE
tokenizers::tokenize_sentences()	5	1	5	0	TRUE
spaCy sentence_id	2	0	5	-3	FALSE

Where each splitter puts the boundaries
Splitter	Piece	Piece number	Exact expected sentence
split on full stops	RIVERTON SKILLS OPEN HOUSE Dr	1	no
split on full stops	Vale said shifts start at 6 a	2	no
split on full stops	m	3	no
split on full stops	Pay is $18	4	no
split on full stops	50 per hour	5	no
split on full stops	J	6	no
split on full stops	Rivera can apply by Friday Bring a photo ID	7	no
tokenizers::tokenize_sentences()	RIVERTON SKILLS OPEN HOUSE Dr.	1	no
tokenizers::tokenize_sentences()	Vale said shifts start at 6 a.m.	2	no
tokenizers::tokenize_sentences()	Pay is $18.50 per hour.	3	yes
tokenizers::tokenize_sentences()	J.	4	no
tokenizers::tokenize_sentences()	Rivera can apply by Friday Bring a photo ID.	5	no
spaCy sentence_id	RIVERTON SKILLS OPEN HOUSE Dr. Vale said shifts start at 6 a.m. Pay is $18.50 per hour.	1	no
spaCy sentence_id	J. Rivera can apply by Friday Bring a photo ID.	2	no

The naive rule finds seven pieces and none is one of the five expected sentences. It splits Dr., a.m., $18.50, and J.. The package splitter returns the expected count, but only one piece exactly matches an expected sentence. It attaches Dr. to the heading and separates J. from the name.

spaCy returns two pieces. It preserves Dr. Vale and a.m. but combines lines into fewer sentences than the layout implies. The pipeline uses syntax instead of the line-break boundaries that the flyer requires.

The code reconstructs spaCy pieces from tokens and does not preserve the original spaces or line breaks. The exact-match column therefore compares reconstructed text rather than the copied page. Equal piece counts alone do not establish correct segmentation. A splitter can return the expected number of pieces with incorrect boundary positions.

Examine the Riverton rows

The file creator assigned one sentence or flyer line to each of the 28 Riverton rows. This upstream boundary choice occurs before the lesson reads the file.

sentence_piece_counts <- tokenize_sentences(sentences$text) |>
  map_int(length)

row_boundary_summary <- sentences |>
  mutate(sentence_pieces = sentence_piece_counts) |>
  summarise(
    rows = n(),
    tokenizer_pieces = sum(sentence_pieces),
    .by = document_id
  )

flyer_rows <- sentences |>
  filter(document_id == "F001") |>
  nrow()

job_rows <- sentences |>
  filter(document_id != "F001") |>
  nrow()

Sentence splitting agrees with the stored Riverton rows
Document ID	Rows	Sentence pieces
J001	4	4
J002	4	4
J003	4	4
J004	3	3
J005	3	3
J006	4	4
F001	6	6

For these stored rows, the package splitter returns one sentence piece for every row. That agreement does not prove the rows are the only possible boundaries. It only shows agreement between this file and this splitter on the teaching set.

What to remember

Sentence segmentation marks where one sentence ends and another begins.
A full stop can appear inside titles, initials, abbreviations, and decimals.
In the test passage, the three splitters return 7, 5, and 2 pieces.
A correct count can still hide wrong boundaries.
Rebuilding model output from tokens can change the text that the code scores.
The 28 Riverton rows already contain one upstream boundary choice.

For the copied page, five expected sentences are a human layout choice. For the CSV, 28 one-row sentence units are an upstream data choice. Both counts depend on the rule named beside them.

Build a passage with difficult boundaries

Compare three splitters

Examine the Riverton rows

What to remember

Sources