library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
library(tokenizers)
library(tokenizers.bpe)
sentences <- read_csv(
"data/workforce/workforce_sentences.csv",
na = character(),
col_types = cols(
sentence_id = col_character(),
document_id = col_character(),
source_line = col_character(),
text = col_character(),
reference_label = col_character(),
uncertainty = col_character(),
annotator_id = col_character(),
rationale = col_character(),
codebook_version = col_character(),
codebook_hash = col_character(),
derived = col_character(),
transformation = col_character()
)
)
document_counts <- sentences |>
mutate(
source_group = if_else(
document_id == "F001",
"training flyer",
"job-detail rows"
)
) |>
count(source_group, name = "rows")
first_sentence <- sentences$text[1]Deciding what counts as a word
Compare word, sentence, and subword tokenization
word parsing
tokenization
workforce research
Learn why tokenization is a rule-based decision before any word count can begin.
A planning meeting begins with the Labโs labeled sentences on the table. The team has 28 reviewed text units from a job board page and a training flyer. Six came from the flyer, and 22 came from job-detail lines.
The labels describe what each text unit says. Before counting words, the Lab must also state what counts as a word.
Note
The author created the fictional Riverton Workforce Lab, its job board, and its training flyer for teaching.
TipWhat you will learn
By the end of this lesson, you will be able to:
- explain how a rule creates tokens.
- compare whitespace and package word tokenizers.
- compare row counts before and after sentence tokenization.
- train a subword model without writing files into the project folder.
- explain which settings control the number of pieces in tokenized text.
Start with the sentence on the page
The first chunk loads readr for files and dplyr and tibble for tables. It loads purrr for successive checks and stringr for strings. The tokenizers package splits words and sentences, while tokenizers.bpe splits subwords. A tibble is a tidyverse table that prints its dimensions and each columnโs type.
| Source group | Rows |
|---|---|
| job-detail rows | 22 |
| training flyer | 6 |
The first sentence fits on one line: Paid 12-week training is provided. A token is one piece of text produced by a splitting rule. The same visible sentence can produce different tokens because the rule makes different choices.
Split on spaces
One simple rule is to split wherever there is whitespace. Here stringr uses the ICU text library to find spaces, tabs, and similar separators. That rule gives five pieces. The period stays attached to provided., and 12-week stays in one piece.
whitespace_tokens <- str_split(
first_sentence,
"\\s+",
simplify = FALSE
)[[1]]| Token number | Whitespace token |
|---|---|
| 1 | Paid |
| 2 | 12-week |
| 3 | training |
| 4 | is |
| 5 | provided. |
This rule is easy to explain, but it keeps punctuation and hyphenated material attached to nearby letters. That may or may not fit the count the Lab wants.
Use a word tokenizer
tokenizers::tokenize_words() applies a different rule. It lowercases text, drops punctuation, and splits the hyphenated phrase. The same sentence becomes six tokens: paid, 12, week, training, is, and provided.
default_word_tokens <- tokenize_words(first_sentence)[[1]]| Token number | Default word token |
|---|---|
| 1 | paid |
| 2 | 12 |
| 3 | week |
| 4 | training |
| 5 | is |
| 6 | provided |
That rule is useful when Paid and paid should count together. It is a poor fit if the Lab needs to keep punctuation as its own evidence.
Keep case and punctuation visible
The function can retain case and punctuation. With those settings, Paid stays capitalized. The hyphen and full stop each become separate tokens. The sentence becomes eight tokens.
case_punctuation_tokens <- tokenize_words(
first_sentence,
lowercase = FALSE,
strip_punct = FALSE
)[[1]]
token_count_summary <- tibble(
rule = c(
"split on whitespace",
"default word tokenizer",
"keep case and punctuation"
),
tokens = c(
length(whitespace_tokens),
length(default_word_tokens),
length(case_punctuation_tokens)
)
)| Rule | Tokens from the first sentence |
|---|---|
| split on whitespace | 5 |
| default word tokenizer | 6 |
| keep case and punctuation | 8 |
Five, six, and eight are all correct counts under their respective rules. The count alone does not show the rule.
Compare sentence splits before word counts
The file stores one text unit per row. The Lab compares sentence-tokenization output with those 28 pieces. Other files may contain several sentences per row or divide one sentence across rows.
sentence_pieces <- tokenize_sentences(sentences$text)
sentence_piece_counts <- map_int(sentence_pieces, length)
source_sentence_counts <- sentences |>
mutate(sentence_pieces = sentence_piece_counts) |>
summarise(
rows = n(),
pieces = sum(sentence_pieces),
.by = document_id
)| Document ID | Rows | Sentence pieces |
|---|---|---|
| J001 | 4 | 4 |
| J002 | 4 | 4 |
| J003 | 4 | 4 |
| J004 | 3 | 3 |
| J005 | 3 | 3 |
| J006 | 4 | 4 |
| F001 | 6 | 6 |
For these rows, tokenizers::tokenize_sentences() returns one piece per row. The file and tokenizer therefore agree on unit boundaries in this teaching dataset. This agreement does not make every unit a grammatical sentence.
Train a subword tokenizer
Modern language models often split text into subwords. These pieces can be smaller than words but larger than single letters for patterns learned during training. Byte pair encoding (BPE) starts from single characters. At each step, it merges the most frequent pair. The following model trains only on the 28 Riverton text units.
The vocab_size = 250L argument controls how many pieces the trainer can keep. The coverage = 1 argument retains every character in the training text. Characters outside that set have no piece. The marker <UNK>, short for โunknownโ, records an input without a vocabulary piece. The setting threads = 1L removes variation from work division across processors.
Note
The explicit model_path sends the model to a temporary file during rendering.
corpus_file <- tempfile(fileext = ".txt")
write_lines(sentences$text, corpus_file)
bpe_model <- bpe(
corpus_file,
model_path = tempfile(fileext = ".bpe"),
vocab_size = 250L,
coverage = 1,
threads = 1L
)
first_vocabulary_rows <- bpe_model$vocabulary |>
slice_head(n = 10)| ID | Subword |
|---|---|
| 0 | <PAD> |
| 1 | <UNK> |
| 2 | <BOS> |
| 3 | <EOS> |
| 4 | โ |
| 5 | e |
| 6 | i |
| 7 | r |
| 8 | s |
| 9 | a |
The vocabulary has 250 entries. Four special markers precede the boundary marker โ and single characters. The marker โ indicates the start of a word-boundary subword.
Compare familiar and unfamiliar text
When the BPE model encodes the first sentence, common pieces such as training, is, and provided. stay together. The rarer 12-week material fragments.
first_sentence_subwords <- bpe_encode(
bpe_model,
x = "Paid 12-week training is provided.",
type = "subwords"
)[[1]]
unseen_phrase_subwords <- bpe_encode(
bpe_model,
x = "unpredictability biotechnology",
type = "subwords"
)[[1]]
subword_summary <- tibble(
text = c(
"Paid 12-week training is provided.",
"unpredictability biotechnology"
),
pieces = c(
length(first_sentence_subwords),
length(unseen_phrase_subwords)
),
subwords = c(
str_c(first_sentence_subwords, collapse = " | "),
str_c(unseen_phrase_subwords, collapse = " | ")
)
)| Text | Subword pieces | Subwords |
|---|---|---|
| Paid 12-week training is provided. | 9 | โPaid | โ1 | 2 | - | w | eek | โtraining | โis | โprovided. |
| unpredictability biotechnology | 21 | โ | un | pre | d | ic | ta | b | il | i | t | y | โb | i | o | te | ch | n | ol | o | g | y |
The first sentence produces nine subword pieces. The unseen phrase unpredictability biotechnology produces 21 pieces in this teaching model. Larger released tokenizers use different training data, vocabularies, normalizers, and unknown policies. Measure their piece counts instead of inferring counts from this result.
This example does not establish that unfamiliar material always produces more pieces. The next comparison demonstrates the limit.
What actually decides the number of pieces
Training used 28 English lines in the Latin script. The setting coverage = 1 retained every character in that text. A character outside that set has no piece, so the model uses <UNK>.
The comparison adds a Chinese phrase meaning โtraining courseโ and a Greek word. Neither text appears in training. The code also uses type = "ids" instead of type = "subwords". IDs report the vocabulary entry for each piece.
unknown_id <- bpe_model$vocabulary$id[
bpe_model$vocabulary$subword == "<UNK>"
]
texts_to_encode <- c(
familiar_sentence = "Paid 12-week training is provided.",
unseen_english = "unpredictability biotechnology",
unseen_chinese = "\u57F9\u8BAD\u8BFE\u7A0B",
unseen_greek = "\u0393\u03BB\u03C9\u03C3\u03C3\u03B1"
)
piece_report <- tibble(
example = names(texts_to_encode),
text = unname(texts_to_encode),
ids = map(
unname(texts_to_encode),
\(one_text) bpe_encode(bpe_model, x = one_text, type = "ids")[[1]]
),
subwords = map(
unname(texts_to_encode),
\(one_text) bpe_encode(bpe_model, x = one_text, type = "subwords")[[1]]
)
) |>
mutate(
characters = str_length(text),
pieces = map_int(ids, length),
unknown_pieces = map_int(
ids,
\(id_vector) sum(id_vector == unknown_id)
),
shown_as_subwords = map_chr(
subwords,
\(piece_vector) str_c(piece_vector, collapse = " | ")
),
display_text = case_when(
example == "unseen_chinese" ~
"<span lang=\"zh\">ๅน่ฎญ่ฏพ็จ</span>",
example == "unseen_greek" ~
"<span lang=\"el\">ฮฮปฯฯฯฮฑ</span>",
TRUE ~ text
),
display_subwords = case_when(
example == "unseen_chinese" ~ str_c(
"<span lang=\"zh\">",
shown_as_subwords,
"</span>"
),
example == "unseen_greek" ~ str_c(
"<span lang=\"el\">",
shown_as_subwords,
"</span>"
),
TRUE ~ shown_as_subwords
)
)| Text | Characters | Pieces | Unknown pieces | What the subword view shows |
|---|---|---|---|---|
| Paid 12-week training is provided. | 34 | 9 | 0 | โPaid | โ1 | 2 | - | w | eek | โtraining | โis | โprovided. |
| unpredictability biotechnology | 30 | 21 | 0 | โ | un | pre | d | ic | ta | b | il | i | t | y | โb | i | o | te | ch | n | ol | o | g | y |
| ๅน่ฎญ่ฏพ็จ | 4 | 2 | 1 | โ | ๅน่ฎญ่ฏพ็จ |
| ฮฮปฯฯฯฮฑ | 6 | 2 | 1 | โ | ฮฮปฯฯฯฮฑ |
The unfamiliar Chinese phrase produces two pieces. The familiar English sentence produces nine. The model learned nothing about Chinese. One of the Chinese phraseโs pieces is <UNK>, which records the missing representation.
Piece counts depend on the tokenizer, vocabulary, script, language, and normalization. The policy for unknown characters is especially important. This model can represent unfamiliar text with fewer pieces than familiar text. Replacing unknown text with <UNK> reduces piece counts but loses information.
The column โWhat the subword view showsโ displays Chinese characters without corresponding vocabulary entries. The subword view reconstructs surface text instead of reporting vocabulary entries. Count unknowns from the IDs, not the printed pieces.
Compare decoded text with the input
If a tokenizer cannot reconstruct its input, it loses information. Decoding the IDs permits a direct reconstruction check.
round_trip <- piece_report |>
mutate(
decoded = map_chr(
ids,
\(id_vector) bpe_decode(bpe_model, id_vector)[[1]]
),
recovers_input = decoded == text
)| Text | Pieces | Unknown pieces | Decoded from ids | Recovers the input |
|---|---|---|---|---|
| Paid 12-week training is provided. | 9 | 0 | Paid 12-week training is provided. | TRUE |
| unpredictability biotechnology | 21 | 0 | unpredictability biotechnology | TRUE |
| ๅน่ฎญ่ฏพ็จ | 2 | 1 | <UNK> | FALSE |
| ฮฮปฯฯฯฮฑ | 2 | 1 | <UNK> | FALSE |
Decoding recovers both Latin-script texts exactly, with their punctuation and digits. It replaces both unseen-script texts with <UNK>. Only texts with unknown pieces fail this round-trip check.
Round-trip recovery provides a check of reversible text identity, not source alignment. Some tokenizers return original-input offsets even when an unknown ID cannot decode to its text. Tasks that mark spans or highlight matches need separate checks of those offset mappings.
A word count must identify its tokenization rule. A piece count must identify its vocabulary, script coverage, and treatment of other characters.
What to remember
- Tokenization produces text pieces according to a chosen rule.
- The first Riverton sentence becomes 5, 6, or 8 word tokens under three rules.
- Sentence tokenization preserves the 28 rows in this file.
- This teaching BPE used larger pieces for some training text. That pattern is not universal.
- Piece counts depend on the vocabulary, the script, the normalization the model applies, and its policy for unknown characters.
- An unknown-piece policy can reduce piece counts for unfamiliar text while losing all its content.
- Count unknown pieces from the IDs.
- Compare the decoded text with the input.
The Lab must record its token rule before counting. The rule gives the counting unit before any summary calculation.