Building a vocabulary from tokens

Choose which words to keep and measure excluded tokens

word parsing

vocabulary

workforce research

Learn how frequency cutoffs and out-of-vocabulary checks affect a small workforce text vocabulary.

After tokenization, the coordinator prints the Riverton Workforce Lab’s word tokens. The list contains frequent words and words that occur only once. A large project needs a rule for vocabulary selection.

The team measures the rule’s effects on its small teaching dataset. It builds a word list from the job-detail rows. Then it measures the flyer tokens absent from that list.

Note

The author created the fictional Riverton Workforce Lab, its job board, and its training flyer for teaching.

What you will learn

By the end of this lesson, you will be able to:

distinguish tokens from types.
count a word vocabulary and its one-time terms.
apply minimum-frequency cutoffs.
measure out-of-vocabulary words on held-out text.
compare a word vocabulary with a subword vocabulary.

Count tokens and types

The first chunk loads readr for files and dplyr, tidyr, and tibble for tables. It also loads purrr for successive operations, tokenizers for word tokens, and tokenizers.bpe for subword tokens. A tibble is a table that prints its size and column types.

library(readr)
library(dplyr)
library(tidyr)
library(tibble)
library(purrr)
library(tokenizers)
library(tokenizers.bpe)

sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

word_token_rows <- sentences |>
  transmute(
    sentence_id,
    token = tokenize_words(text)
  ) |>
  unnest_longer(token)

word_counts <- word_token_rows |>
  count(token, name = "frequency", sort = TRUE)
top_word_counts <- word_counts |>
  slice_head(n = 8)

The eight most frequent word types in all 28 rows
Token type	Frequency
is	13
a	7
required	7
the	5
training	5
are	4
experience	4
of	3

A token is one occurrence. A type is one distinct token value. These counts use the default tokenizers::tokenize_words() rule: lowercase text, drop punctuation, and split hyphenated material. Across the 28 rows, that rule creates 153 tokens and 96 types. Of those types, 72 appear exactly once.

The most frequent types are function words and common notice terms. The word is appears 13 times. Both a and required appear 7 times each. Both the and training appear 5 times each. Both are and experience appear 4 times each. The word of appears 3 times.

Split the data before vocabulary evaluation

Evaluation on new text requires rows separate from vocabulary construction. The Lab builds its word list from the 22 job-detail rows. It uses the 6 flyer rows to find unseen tokens.

job_rows <- sentences |>
  filter(document_id != "F001")
flyer_rows <- sentences |>
  filter(document_id == "F001")

job_token_rows <- job_rows |>
  transmute(
    sentence_id,
    token = tokenize_words(text)
  ) |>
  unnest_longer(token)
flyer_token_rows <- flyer_rows |>
  transmute(
    sentence_id,
    token = tokenize_words(text)
  ) |>
  unnest_longer(token)

job_word_counts <- job_token_rows |>
  count(token, name = "frequency", sort = TRUE)

split_summary <- tibble(
  split = c("job-detail rows", "training flyer"),
  rows = c(nrow(job_rows), nrow(flyer_rows)),
  tokens = c(nrow(job_token_rows), nrow(flyer_token_rows)),
  types = c(nrow(job_word_counts), n_distinct(flyer_token_rows$token))
)

Word-token counts after splitting by document source
Split	Rows	Tokens	Types
job-detail rows	22	133	86
training flyer	6	20	20

The job-detail rows contain 133 tokens and 86 types. The flyer rows contain 20 tokens. The small split keeps evaluation separate from the text that created the vocabulary.

Keep words by minimum frequency

A minimum-frequency cutoff keeps only types that appear at least a stated number of times. The code below asks how many job-detail types remain and how many job-detail tokens those remaining types cover.

The shorthand \(cutoff) means “make a small function whose input is cutoff.”

frequency_cutoffs <- tibble(minimum_frequency = 1:3) |>
  mutate(
    kept_types = map_int(
      minimum_frequency,
      \(cutoff) sum(job_word_counts$frequency >= cutoff)
    ),
    covered_tokens = map_int(
      minimum_frequency,
      \(cutoff) {
        kept <- job_word_counts |>
          filter(frequency >= cutoff) |>
          pull(token)
        sum(job_token_rows$token %in% kept)
      }
    ),
    coverage = covered_tokens / nrow(job_token_rows),
    coverage_label = sprintf(
      "%s/%s (%s%%)",
      covered_tokens,
      nrow(job_token_rows),
      round(100 * coverage)
    )
  )

Vocabulary size and coverage under three cutoffs
Minimum frequency	Types kept	Share of job tokens covered
1	86	133/133 (100%)
2	18	65/133 (49%)
3	8	45/133 (34%)

The minimum-frequency-1 row is a reference point: it keeps every job-detail type, so coverage is full by construction. The stricter rows show how many job-detail tokens lose their own word type under each cutoff.

Measure out-of-vocabulary words

An out-of-vocabulary token is absent from the vocabulary. Here the vocabulary comes from the 22 job-detail rows. The comparison finds flyer tokens absent from that list.

A retrieval system can keep a different kind of vocabulary. A search index stores whole terms and counts for retrieval. A tokenizer or subword model stores the pieces it will use to encode new text.

job_vocabulary <- job_word_counts$token
unknown_flyer_rows <- flyer_token_rows |>
  filter(!token %in% job_vocabulary)
unknown_flyer_types <- sort(unique(unknown_flyer_rows$token))
unknown_summary <- tibble(
  measure = c(
    "flyer tokens",
    "unknown flyer tokens",
    "unknown share"
  ),
  value = c(
    as.character(nrow(flyer_token_rows)),
    as.character(nrow(unknown_flyer_rows)),
    sprintf(
      "%s/%s (about half)",
      nrow(unknown_flyer_rows),
      nrow(flyer_token_rows)
    )
  )
)

Out-of-vocabulary rate on the held-out flyer rows
Measure	Value
flyer tokens	20
unknown flyer tokens	10
unknown share	10/20 (about half)

The unknown flyer types are 15, apply, by, classes, house, october, open, riverton, stipend, and support.

For those 10 token occurrences, the job-built word list has no entry.

Try the same split with subwords

Byte pair encoding (BPE) starts from single characters. At each step, it merges the most frequent pair. The Lab trains a BPE model on job-detail rows only. It then encodes the flyer rows for the same held-out evaluation as the word vocabulary.

The word tokenizer lowercases both sources, so the BPE comparison does the same. This keeps the case policy constant while comparing tokenization methods. The vocab_size = 200L argument controls how many pieces the trainer can keep. The coverage = 1 argument retains every character in the lowercased job-detail text. The marker <UNK>, short for “unknown”, records an input without a vocabulary piece.

When two pairs occur equally often, the trainer can merge them in different orders, even with threads = 1L. Pieces and counts can therefore differ between fits. The code computes counts from the fitted model. It requires agreement between pieces and IDs and exact decoding of the held-out text.

Note

The explicit model_path sends the model to a temporary file during rendering.

job_corpus_file <- tempfile(fileext = ".txt")
write_lines(tolower(job_rows$text), job_corpus_file)

job_bpe_model <- bpe(
  job_corpus_file,
  model_path = tempfile(fileext = ".bpe"),
  vocab_size = 200L,
  coverage = 1,
  threads = 1L
)

flyer_subwords <- bpe_encode(
  job_bpe_model,
  x = tolower(flyer_rows$text),
  type = "subwords"
)
flyer_ids <- bpe_encode(
  job_bpe_model,
  x = tolower(flyer_rows$text),
  type = "ids"
)
flyer_word_ids <- bpe_encode(
  job_bpe_model,
  x = flyer_token_rows$token,
  type = "ids"
)
flat_flyer_subwords <- unlist(flyer_subwords, use.names = FALSE)
flat_flyer_ids <- unlist(flyer_ids, use.names = FALSE)
unknown_id <- job_bpe_model$vocabulary$id[
  job_bpe_model$vocabulary$subword == "<UNK>"
]
unknown_flyer_piece_count <- as.integer(sum(flat_flyer_ids == unknown_id))
flyer_word_bpe <- flyer_token_rows |>
  mutate(
    bpe_ids = flyer_word_ids,
    has_unknown_id = map_lgl(
      bpe_ids,
      \(ids) any(ids == unknown_id)
    )
  )
unknown_flyer_word_count <- sum(flyer_word_bpe$has_unknown_id)
flyer_unknown_by_line <- tibble(
  sentence_id = flyer_rows$sentence_id,
  text = flyer_rows$text,
  pieces = map_int(flyer_ids, length),
  unknown_pieces = map_int(
    flyer_ids,
    \(ids) as.integer(sum(ids == unknown_id))
  )
)
subword_test_summary <- tibble(
  measure = c(
    "BPE vocabulary rows",
    "flyer word tokens",
    "word tokens with an unknown id",
    "flyer subword pieces",
    "unknown piece ids"
  ),
  value = c(
    as.character(nrow(job_bpe_model$vocabulary)),
    as.character(nrow(flyer_word_bpe)),
    as.character(unknown_flyer_word_count),
    as.character(length(flat_flyer_ids)),
    as.character(unknown_flyer_piece_count)
  )
)

BPE results on the held-out flyer rows
Measure	Value
BPE vocabulary rows	200
flyer word tokens	20
word tokens with an unknown id	0
flyer subword pieces	68
unknown piece ids	0

Unknown subword ids by flyer line
Sentence ID	Flyer line	Subword pieces
s023	RIVERTON SKILLS OPEN HOUSE	16
s024	DATA SUPPORT CERTIFICATE	10
s025	Paid training stipend	7
s026	Evening classes	10
s027	No prior experience required	11
s028	Apply by October 15	14

Both models now have results at the word-token level. The whole-word vocabulary has no entry for 10 of 20 flyer tokens. With both sources lowercased, none of the 20 word tokens contains a BPE unknown id, and none of the 68 BPE pieces is unknown. The subword model can assemble unseen words from characters and pieces learned in the job rows.

That result applies to this 200-piece model and this case policy. It does not establish that BPE always removes out-of-vocabulary material. Case-sensitive input, a different writing system, a smaller character coverage, or another tokenizer can produce unknown ids. Comparing 10 word tokens with 68 pieces would also mix denominators, so the conclusion uses the 20 word-token outcomes.

What to remember

Tokens are occurrences. Types are distinct token values.
The 28 Riverton rows produce 153 word tokens and 96 word types.
Most word types in this small file appear exactly once.
Frequency cutoffs reduce vocabulary size and token coverage.
Held-out text exposes out-of-vocabulary words.
With the same lowercase policy, this BPE model encodes all 20 held-out word tokens. That result applies to these settings, not every BPE model.

These percentages apply only to this 28-unit teaching corpus. They show how vocabulary choices behave on the Riverton example, not how often these patterns occur in workforce text more generally.

Count tokens and types

Split the data before vocabulary evaluation

Keep words by minimum frequency

Measure out-of-vocabulary words

Try the same split with subwords

What to remember

Sources