library(readr)
library(dplyr)
library(tibble)
library(stringr)
library(cld2)
library(cld3)
library(digest)
language_path <- "data/riverton/language-samples.csv"
language_samples <- read_csv(
language_path,
na = character(),
col_types = cols(
sample_id = col_character(),
written_in = col_character(),
text = col_character()
)
)
reference_metadata <- read_csv(
"data/riverton/riverton-reference-metadata.csv",
na = character(),
col_types = cols(
artifact = col_character(),
description = col_character(),
source = col_character(),
license = col_character(),
created_on = col_character(),
fingerprint = col_character()
)
)
language_reference <- reference_metadata |>
filter(artifact == "language-samples.csv")
sample_fingerprint <- digest::digest(
paste(read_lines(language_path), collapse = "\n"),
algo = "sha256",
serialize = FALSE
)
recorded_fingerprint <- reference_metadata |>
filter(artifact == "language-samples.csv") |>
pull(fingerprint)
detector_versions <- tibble(
detector = c("cld2", "cld3"),
package_version = c(
as.character(packageVersion("cld2")),
as.character(packageVersion("cld3"))
)
)
display_samples <- language_samples |>
mutate(
display_text = str_c(
"<span lang=\"",
written_in,
"\">",
text,
"</span>"
)
)Identifying document language
Treat language guesses as fallible classifications
documents
language identification
workforce research
Compare two language detectors on short Riverton samples and learn why no label can be safer than a wrong label.
Short notes in several languages require routing before content annotation. The coordinator must assign each note to people who can read its language.
Open house. exposes the problem. A person reads the two-word notice as English, while a language detector may have too little evidence to say.
Language identification is a classification problem. A classifier assigns a label from a fixed set, such as en for English or es for Spanish. The risk is either a wrong code or no answer, which leaves the item without a route. This lesson compares those outcomes for two detectors on a small teaching file.
Note
The Riverton inbox and the language samples are fictional teaching materials.
TipWhat you will learn
Use this lesson to practice how to:
- execute two local language detectors on fixed samples.
- compare detector output with recorded labels.
- separate agreement from correctness.
- explain why short texts are hard to classify.
- describe why returning
NAcan be safer than a wrong language code.
Load the labelled samples
The setup records detector versions and measures text length. It executes the local CLD2 and CLD3 detectors and calculates a file fingerprint. The author wrote the short samples for teaching.
| Detector package | Version |
|---|---|
| cld2 | 1.2.6 |
| cld3 | 1.6.2 |
| File | SHA-256 fingerprint |
|---|---|
| data/riverton/language-samples.csv | fa84ffaf608578b8dd0e4f0dbc7425049cd9d362e9d41a1773078973afeccf67 |
| Sample | Recorded language | Text |
|---|---|---|
| L01 | en | Evening shifts require a valid forklift certification. |
| L02 | en | Apply by October 15. |
| L03 | es | Muy útil y fácil de usar. |
| L04 | fr | Les cours du soir commencent en octobre. |
| L05 | de | Der Kurs beginnt im Oktober. |
| L06 | nl | De cursus begint in oktober. |
| L07 | en | Open house. |
| L08 | es | Casa abierta. |
The written_in column records the author’s intended language rather than an independently assessed annotation. The Spanish Casa abierta. is grammatical. As an event notice, it is a calque, a direct translation, rather than an idiomatic phrase. The results describe the detector versions printed above. A different package version can return different codes for the same strings.
Run two detectors
CLD2 and CLD3 are trained language detectors. NA means the detector returned no language label. The detectors classify the supplied characters without understanding the full context of a note.
detected_samples <- language_samples |>
mutate(
characters = str_length(text),
words = str_count(text, boundary("word")),
length_group = if_else(characters <= 13L, "short", "long"),
cld2_language = cld2::detect_language(text),
cld3_language = cld3::detect_language(text),
cld2_match = !is.na(cld2_language) & cld2_language == written_in,
cld3_match = !is.na(cld3_language) & cld3_language == written_in,
display_text = str_c(
"<span lang=\"",
written_in,
"\">",
text,
"</span>"
)
)
open_house <- detected_samples |>
filter(text == "Open house.")| Sample | Recorded language | Text | Characters | Words | CLD2 | CLD3 |
|---|---|---|---|---|---|---|
| L01 | en | Evening shifts require a valid forklift certification. | 54 | 7 | en | en |
| L02 | en | Apply by October 15. | 20 | 4 | en | en |
| L03 | es | Muy útil y fácil de usar. | 25 | 6 | es | es |
| L04 | fr | Les cours du soir commencent en octobre. | 40 | 7 | fr | fr |
| L05 | de | Der Kurs beginnt im Oktober. | 28 | 5 | de | de |
| L06 | nl | De cursus begint in oktober. | 28 | 5 | NA | NA |
| L07 | en | Open house. | 11 | 2 | NA | NA |
| L08 | es | Casa abierta. | 13 | 2 | NA | NA |
Both installed detectors abstain, or return no label, on Open house.. With the NA result, the team can send the item to a person for review rather than to a queue with an incorrect code.
Count matches, abstentions, and label differences
Agreement between two detectors does not prove correctness. They can share training data patterns and make the same mistake. The recorded labels let the team count what happened on these eight examples.
detector_counts <- tibble(
detector = c("CLD2", "CLD3"),
matched_recorded_label = c(
sum(detected_samples$cld2_match),
sum(detected_samples$cld3_match)
),
no_label = c(
sum(is.na(detected_samples$cld2_language)),
sum(is.na(detected_samples$cld3_language))
)
) |>
mutate(
differed_from_recorded_label =
nrow(detected_samples) - matched_recorded_label - no_label
)| Detector | Matched recorded label | No label | Different from recorded label |
|---|---|---|---|
| CLD2 | 5 | 3 | 0 |
| CLD3 | 5 | 3 | 0 |
Each detector matches 5 of the 8 recorded labels. It returns no label for 3 of the 8 and a different label for 0 of the 8. These fractions describe only this teaching file. Eight samples cannot establish detector accuracy on a workforce archive.
Compare short and longer text
Text length affects the amount of evidence a detector receives. This file is too small to give an estimate of the length effect. There is no principled cutoff here. The 13-character line separates the two shortest strings from the other 6 examples.
length_summary <- detected_samples |>
group_by(length_group) |>
summarise(
samples = n(),
cld2_matches = sum(cld2_match),
cld2_no_label = sum(is.na(cld2_language)),
cld3_matches = sum(cld3_match),
cld3_no_label = sum(is.na(cld3_language)),
.groups = "drop"
) |>
arrange(length_group)| Length group | Samples | CLD2 matches | CLD2 no label | CLD3 matches | CLD3 no label |
|---|---|---|---|---|---|
| long | 6 | 5 | 1 | 5 | 1 |
| short | 2 | 0 | 2 | 0 | 2 |
The two shortest samples produce no matches because both detectors abstain. Among the six other samples, each detector matches 5 recorded labels and abstains on 1. The Dutch sentence is long enough to read by eye, but both detectors return NA.
A wrong code is worse than no code
No returned code differed from the recorded labels for the eight samples. An incorrect code is still possible. The author selected the next string because it produces an incorrect code. This one case demonstrates a failure. It does not give an estimate of its rate. Its length is just above the arbitrary 13-character cutoff.
short_probe <- tibble(
text = c("Open house.", "Evening class."),
recorded_language = "en"
) |>
mutate(
characters = str_length(text),
cld2_language = cld2::detect_language(text),
cld3_language = cld3::detect_language(text),
cld2_match = !is.na(cld2_language) & cld2_language == recorded_language,
cld3_match = !is.na(cld3_language) & cld3_language == recorded_language
)| Text | Recorded language | CLD2 | CLD3 |
|---|---|---|---|
| Open house. | en | NA | NA |
| Evening class. | en | NA | lb |
Evening class. is English in this teaching example. CLD2 returns NA, while CLD3 returns lb, the ISO 639-1 code for Luxembourgish. For routing, the abstention is safer because a person examines the item before selecting a queue.
These eight samples are all written in the Latin alphabet, so this page shows nothing about other writing systems. That is a limit of the file, not a reassurance.
Language identification in production
cld2 and cld3 illustrate the routing problem. Models such as Meta’s fasttext are common alternatives for fast language identification. Deployment requires these considerations:
- Language limits and BCP 47: State which languages the project handles. Record languages with standard BCP 47 tags, such as
en-US, rather than informal names. - Per-language recall: Overall accuracy can hide failures. Measure recall separately for each language because models routinely fail on languages with little training data.
- Code-switching means alternating languages within one text. The single-label
cld2::detect_language()andcld3::detect_language()calls used here return one document label. Mixed-language text needs a span-level or token-level detector, or a person to examine the text. Single-label behavior does not describe every model. - Probability vs. certainty: A confidence score such as
0.95is not necessarily a calibrated probability. Calibration means predicted probabilities agree with observed frequencies. The internal score alone does not establish that agreement.
What to remember
- Language identification may return a code or no label.
- Agreement between detectors does not prove that a language code is correct.
- In the eight-row file, both detectors returned no label for the two shortest strings rather than a guess.
- Returning
NAcan be safer than sending text to the wrong language queue.
People must examine items with NA results and codes from short text before routing them. The selected example demonstrates a possible wrong code. The eight-row file demonstrates abstention instead of an incorrect assignment.