Finding abbreviations without trusting capitals

Compare all-capital tokens with parenthetical definitions

phrases and entities
abbreviations
workforce research
Learn why capital letters create false abbreviation hits and how parenthetical long-form rules behave on constructed examples.

The Riverton Workforce Lab needs links between short forms and their long forms. One simple rule finds all-capital tokens as possible abbreviations.

The flyer shows a limitation of that rule. Its capitalized headings reflect formatting, not necessarily abbreviated meanings.

Note

The author created the fictional Riverton Workforce Lab, its job board, and its training flyer for teaching.

TipWhat you will learn

In this lesson, you will practice how to:

  • find all-capital tokens with a regular expression.
  • examine capitalised heading matches.
  • measure what a dictionary-and-length guard removes.
  • find Long Form (SHORT) definitions.
  • separate real data from constructed examples.

Find all-capital tokens

The setup reads the text file, expands regex matches, and compares words with hunspell’s dictionary. The stringr package uses the ICU regex engine. The [A-Z] range nevertheless covers only ASCII capitals. It will not match a token such as ÉCOLE.

library(readr)
library(dplyr)
library(tibble)
library(tidyr)
library(purrr)
library(stringr)
library(hunspell)

sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  na = character(),
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)

all_caps_pattern <- "\\b[A-Z]{2,}\\b"
all_caps_hits <- sentences |>
  transmute(
    sentence_id,
    text,
    token = str_extract_all(text, all_caps_pattern)
  ) |>
  unnest(token)
All-capital tokens found in the Riverton text
Sentence ID Text All-capital token
s023 RIVERTON SKILLS OPEN HOUSE RIVERTON
s023 RIVERTON SKILLS OPEN HOUSE SKILLS
s023 RIVERTON SKILLS OPEN HOUSE OPEN
s023 RIVERTON SKILLS OPEN HOUSE HOUSE
s024 DATA SUPPORT CERTIFICATE DATA
s024 DATA SUPPORT CERTIFICATE SUPPORT
s024 DATA SUPPORT CERTIFICATE CERTIFICATE

An abbreviation is a shortened written form, such as Dr. or approx.. An initialism uses initial letters pronounced separately, as in NLP. An acronym uses initial letters pronounced as a word, as in NATO. This rule detects none of those categories directly. It finds sequences of two or more ASCII capitals, which sometimes correspond to abbreviations.

Here it finds 7 tokens from RIVERTON SKILLS OPEN HOUSE and DATA SUPPORT CERTIFICATE. Neither heading contains a genuine short-form abbreviation.

Score and guard the capital rule

The lesson author examined every all-capital hit in the Riverton text and found no short-form abbreviations. The guard below keeps only tokens that are five characters or shorter and absent from the dictionary.

scored_caps <- all_caps_hits |>
  mutate(
    true_abbreviation = FALSE,
    token_lower = str_to_lower(token),
    in_dictionary = hunspell_check(token_lower),
    longer_than_five = str_length(token) > 5L,
    guarded_match = !longer_than_five & !in_dictionary,
    removal_reason = case_when(
      in_dictionary & longer_than_five ~ "dictionary word; longer than five",
      in_dictionary ~ "dictionary word",
      longer_than_five ~ "longer than five",
      TRUE ~ "kept"
    )
  )

capital_score <- tibble(
  rule = c("all-capital token", "short and not in dictionary"),
  candidates = c(nrow(scored_caps), sum(scored_caps$guarded_match)),
  hand_rejected = c(
    sum(!scored_caps$true_abbreviation),
    sum(scored_caps$guarded_match & !scored_caps$true_abbreviation)
  ),
  true_abbreviations_found = c(
    sum(scored_caps$true_abbreviation),
    sum(scored_caps$guarded_match & scored_caps$true_abbreviation)
  )
)

guard_details <- scored_caps |>
  select(sentence_id, token, in_dictionary, longer_than_five, guarded_match, removal_reason)

guard_failure_examples <- tibble(token = c("UNESCO", "IT")) |>
  mutate(
    token_lower = str_to_lower(token),
    in_dictionary = hunspell_check(token_lower),
    longer_than_five = str_length(token) > 5L,
    guarded_match = !longer_than_five & !in_dictionary
  )
Human checks of the capital-letter abbreviation rule before and after a filter
Rule Candidates kept Hand-rejected hits True abbreviations found
all-capital token 7 7 0
short and not in dictionary 0 0 0
Dictionary guard details for the all-capital tokens
Sentence ID Token In dictionary Longer than five Kept by guard Removal reason
s023 RIVERTON FALSE TRUE FALSE longer than five
s023 SKILLS TRUE TRUE FALSE dictionary word; longer than five
s023 OPEN TRUE FALSE FALSE dictionary word
s023 HOUSE TRUE FALSE FALSE dictionary word
s024 DATA TRUE FALSE FALSE dictionary word
s024 SUPPORT TRUE TRUE FALSE dictionary word; longer than five
s024 CERTIFICATE TRUE TRUE FALSE dictionary word; longer than five
Two real short forms the guard would reject
Token Lowercase token In dictionary Longer than five Kept by guard
UNESCO unesco FALSE TRUE FALSE
IT it TRUE FALSE FALSE

The guard keeps zero candidates. It removes 7 hand-rejected heading words and finds 0 abbreviations. Removing every match does not establish a good filter. Six tokens are dictionary words, and four exceed five characters. Only the length limit removes RIVERTON.

The guard also rejects UNESCO because of its length. It rejects IT because the dictionary contains it. Results depend on the hunspell dictionary available at execution time.

Find parenthetical definitions

The Riverton text has no Long Form (SHORT) examples. This section uses a short author-created passage solely to demonstrate the pattern.

In the code, \(match_table, current_sentence_id) is R shorthand for a small function that handles one sentence’s match table and ID.

definition_pattern <- regex(
  "\\b((?:[A-Z][a-z]+\\s+){1,}[A-Z][a-z]+)\\s*\\(([A-Z]{2,})\\)"
)

extract_definitions <- function(text, sentence_id) {
  match_tables <- str_match_all(text, definition_pattern)
  pieces <- map2(match_tables, sentence_id, \(match_table, current_sentence_id) {
    if (nrow(match_table) == 0L) {
      return(tibble(
        sentence_id = character(),
        long_form = character(),
        short_form = character()
      ))
    }
    tibble(
      sentence_id = current_sentence_id,
      long_form = str_squish(match_table[, 2]),
      short_form = match_table[, 3]
    )
  })
  list_rbind(pieces)
}

riverton_definitions <- extract_definitions(sentences$text, sentences$sentence_id)
riverton_parenthetical_score <- tibble(
  source = "Riverton text",
  definitions_found = nrow(riverton_definitions)
)

constructed_passage <- tibble(
  sentence_id = c("c001", "c002"),
  text = c(
    "The note defines Natural Language Processing (NLP) for the workshop.",
    "The class lists a Data Support Certificate (DSC) for applicants."
  )
)

definitions <- extract_definitions(
  constructed_passage$text,
  constructed_passage$sentence_id
)

shape_only_example <- extract_definitions(
  "Riverton Skills Centre (NLP)",
  "c003"
)

parenthetical_score <- tibble(
  true_definitions_in_constructed_passage = 2L,
  definitions_found = nrow(definitions)
)
Parenthetical definition rule on the Riverton text
Source Definitions found
Riverton text 0
Constructed parenthetical abbreviation definitions
Sentence ID Long form Short form
c001 Natural Language Processing NLP
c002 Data Support Certificate DSC
Parenthetical rule on the constructed positive cases
True definitions in constructed passage Definitions found
2 2

The parenthetical pattern finds zero Riverton definitions. It finds Natural Language Processing (NLP) and Data Support Certificate (DSC) in the constructed passage. That passage contains only positive cases. The result cannot measure false matches or missed definitions.

The pattern matches capitalised words followed by parenthesized capitals. It does not compare the short-form letters with the long form. Thus Riverton Skills Centre (NLP) also matches. The Schwartz and Hearst algorithm listed under Sources examines those letter links and accepts lowercase long forms. This lesson’s rule performs fewer checks.

What to remember

  • All-capital text is only a rough short-form signal.
  • The 7 capital-letter hits in the Riverton headings contain 0 abbreviations.
  • Keeping 0 of 7 matches does not establish that the guard preserves abbreviations.
  • The parenthetical rule finds 2 constructed positive cases and 0 Riverton definitions.
  • Parenthetical shape rules need an additional check of short-form letters against the long form.

Reject the all-capital rule for these flyer headings. For future text, retain candidate short forms during rule evaluation. Evaluate both examples and counter-examples before accepting the rule.

Sources