Reviewing documents for discovery and alerts

Calculate estimates for records that search and ranked review miss

systems
e-discovery
media monitoring
Learn how keyword terms, ranked review, an elusion sample, and a small monitoring stream expose missed records.

A fictional request asks for records about the Calder Yard contract. It requests all responsive records in a fixed collection, not a few examples. In e-discovery, responsive means a record meets the legal request. Relevant records may lack the searcher’s initial keywords.

E-discovery, or electronic discovery, finds records for review or production in response to a legal request. Technology-assisted review uses a model to rank unreviewed records. People review likely responsive records first. Media monitoring processes new items as they arrive and sends alerts for relevant items.

Grossman and Cormack describe Blair and Maron’s legal-search study. Searchers believed their searches found at least 75 percent of relevant documents. Average recall was about 20 percent for the studied requests.

Ranked review differs from lesson 9’s active learning. Active learning selects uncertain examples to improve a model. Ranked review selects high-scoring unreviewed records so reviewers find responsive material sooner.

The elusion rate is the share of unreviewed records that are responsive. A low elusion rate can coexist with many missed records. Suppose a review of 600 records from a 10,000-record collection finds 70 of its 100 responsive records. The 9,400 unreviewed records contain the remaining 30 responsive records. The elusion rate is 30 / 9,400, about 0.32 percent. Recall is 70 / 100, or 70 percent.

Note

The records in this lesson are constructed teaching examples. They do not describe a real legal matter, company, person, news source, or monitoring target.

TipWhat you will learn

This lesson shows how to:

  • declare keyword terms before calculating precision and recall
  • train a keyword-seeded ranked review model without scoring its training rows
  • stop review by a fixed rule chosen before sample selection
  • calculate an estimate of missed responsive records from one finite-population sample
  • explain why the fictional answer key exceeds the information real reviewers usually have
  • process a short monitoring stream in arrival order with duplicate suppression.

Load the constructed review collection

The builder created a short review collection for one request. It also wrote a monitoring stream and metadata fingerprints, which identify file contents. Each row contains a fictional notice. The page author wrote the reference labels. They describe this constructed collection, not independently established truth.

suppressPackageStartupMessages({
  library(digest)
  library(dplyr)
  library(ggplot2)
  library(glmnet)
  library(purrr)
  library(readr)
  library(stringdist)
  library(stringr)
  library(tibble)
  library(tidymodels)
  library(textrecipes)
  library(tidyr)
})

hash_lines <- function(path) {
  digest(
    paste(read_lines(path), collapse = "\n"),
    algo = "sha256",
    serialize = FALSE
  )
}

collection_path <- "data/riverton/riverton-review-collection.csv"
stream_path <- "data/riverton/riverton-monitoring-stream.csv"
metadata_path <- "data/riverton/riverton-review-collection-metadata.csv"

review_records <- read_csv(
  collection_path,
  na = c("", "NA"),
  col_types = cols(
    doc_id = col_character(),
    record_type = col_character(),
    doc_date = col_date(),
    source_type = col_character(),
    sender_role = col_character(),
    subject = col_character(),
    review_request = col_character(),
    reference_responsive = col_logical(),
    reference_reason = col_character(),
    construction_subtype = col_character(),
    near_duplicate_of = col_character(),
    text = col_character(),
    author_note = col_character(),
    construction_id = col_character(),
    file_position = col_integer()
  )
)

monitoring_stream <- read_csv(
  stream_path,
  na = c("", "NA"),
  col_types = cols(
    stream_id = col_character(),
    record_type = col_character(),
    arrival_order = col_integer(),
    doc_date = col_date(),
    source_type = col_character(),
    text = col_character(),
    reference_alert = col_logical(),
    reference_reason = col_character(),
    near_duplicate_of = col_character(),
    author_note = col_character()
  )
)

review_metadata <- read_csv(
  metadata_path,
  na = character(),
  col_types = cols(
    artifact = col_character(),
    description = col_character(),
    source = col_character(),
    license = col_character(),
    created_on = col_character(),
    purpose = col_character(),
    rows = col_integer(),
    class_counts = col_character(),
    fingerprint = col_character(),
    rng_kind = col_character(),
    seed = col_character(),
    effective_seeds = col_character(),
    review_request = col_character(),
    shortcut_sweep = col_character()
  )
)

actual_fingerprints <- tibble(
  artifact = c(
    "riverton-review-collection.csv",
    "riverton-monitoring-stream.csv"
  ),
  fingerprint = c(hash_lines(collection_path), hash_lines(stream_path))
)

collection_summary <- review_records |>
  count(reference_responsive, name = "records") |>
  mutate(label = if_else(reference_responsive, "responsive", "not responsive"))
Constructed review collection labels for one fictional request
Reference label Records
not responsive 204
responsive 36
Fictional request that sets the responsiveness criteria
Review request
Find records about hiding safety inspection delays, changing inspection logs, or deleting inspection-related messages for the Calder Yard contract.

The constructed prevalence is 15%, much higher than in many legal-review matters. The author chose this share to provide enough responsive examples for inspection on a small teaching page.

Each record contains a short note in one of 12 text templates, called frames. Examples start with “Reminder:” or end with “Thanks.” One frame adds nothing to the note. Every frame contains the same number of responsive records and the same number of non-responsive records.

Before writing the file, the builder examined whether listed surface features could separate the two classes. It refuses to write the file if a check reaches F1 of 0.60 or balanced accuracy of 0.75. Frame checks within keyword groups use only the balanced-accuracy limit.

Success means no listed check separated the classes that well. It does not establish that surface features contain no predictive information. Both scores increase with better class separation. Balanced accuracy, from lesson 44, averages the correctly classified share of each class. F1 combines the precision and recall described below. The table contains the builder’s record of these checks.

Checks the builder executed before writing the review collection
Surface checks recorded by the builder
Refuses to write if any check reaches positive F1 >= 0.60 or balanced accuracy >= 0.75, and stops on any NA diagnostic. Checks: single values and held-out value sets (20 split-halves) of record features (length, punctuation, pronouns, digits, month, weekday, ID digit, ID order, date rank, file position, construction ID order, first three tokens, first word, first two words, sentence count, colon, source type, sender role, subject) and of core and frame features; the 20 most frequent non-topic tokens; out-of-fold glmnet ridge and lasso rankers over snowball stop-word counts and over format cues (length, punctuation, capitalization, digits), stratified 5-fold under three fold seeds, top 36; a frame-only ranker; a 15 percent cap on any frame or core opening among non-responsive records; CK-D1, label-free rarity rules (k = 1 to 4) and a held-out ranker over the pair of frame and core opening word; and CK-D2, frame rules and a held-out frame ranker within the keyword-match and no-match strata, thresholded on balanced accuracy.

The table uses the builder’s own shorthand:

  • A held-out check learns a rule from some records and evaluates that rule on other records.
  • Out-of-fold scores come from cross-validation (lesson 43). Each record receives a score from a model that did not train on it. Top 36 means the 36 highest scores, matching the responsive-record count.
  • Ridge and lasso are two kinds of penalized regression, the same family as the ranker later on this page. Snowball stop words are a standard list of common words such as “the” and “to”.
  • The core is the note inside the frame, and its opening is its first word or first two words.
  • A rarity rule puts a mark on records whose frame-and-first-word pair occurs at most k times, without using labels.
  • CK-D1 and CK-D2 name the builder’s two frame checks. CK-D1 examines each frame together with the note’s first word. CK-D2 compares frames separately among records with and without a keyword.

Search predeclared keywords

The author fixes keyword terms before scoring. These terms are plausible for the request, but they remain guesses. The fictional collection contains reference labels, so the lesson can calculate keyword recall.

keyword_terms <- c(
  "inspection",
  "inspect",
  "delete",
  "deleted",
  "delay",
  "delayed",
  "log",
  "Calder Yard"
)

keyword_pattern <- regex(
  paste0(
    "\\b(",
    paste(keyword_terms[keyword_terms != "Calder Yard"], collapse = "|"),
    ")\\b|\\bCalder Yard\\b"
  ),
  ignore_case = TRUE
)

review_scored <- review_records |>
  mutate(
    keyword_hit = str_detect(text, keyword_pattern),
    label = factor(
      if_else(reference_responsive, "responsive", "not responsive"),
      levels = c("responsive", "not responsive")
    )
  )

keyword_metrics <- review_scored |>
  summarise(
    hits = sum(keyword_hit),
    responsive_hits = sum(keyword_hit & reference_responsive),
    responsive_total = sum(reference_responsive),
    precision = responsive_hits / hits,
    recall = responsive_hits / responsive_total
  )

keyword_examples <- bind_rows(
  review_scored |>
    filter(keyword_hit, reference_responsive) |>
    arrange(doc_id) |>
    slice_head(n = 4),
  review_scored |>
    filter(keyword_hit, !reference_responsive) |>
    arrange(doc_id) |>
    slice_head(n = 4)
) |>
  transmute(
    doc_id,
    reference_label = if_else(reference_responsive, "responsive", "not responsive"),
    construction_subtype,
    text
  )
Predeclared keyword terms on the constructed collection
Keyword hits Responsive hits Responsive records Precision Recall
39 22 36 0.564 0.611
Keyword hits include both responsive records and false hits
Document Reference label Construction note Text
RYD-0008 responsive keyword responsive Per scheduling: The contract file should show the inspection as pending, not delayed.
RYD-0028 responsive keyword responsive Following up: The safety inspection delay should stay out of the weekly Calder Yard summary.
RYD-0043 responsive keyword responsive Remove the delayed inspection line from the packet before records reviews it. Call me if anything is unclear.
RYD-0045 responsive keyword responsive Forwarding from intake: The draft says the inspection was late; delete that line before the packet goes out.
RYD-0006 not responsive keyword false hit Forwarding from intake: The classroom inspection drill note belongs in the training folder, not the Calder Yard file.
RYD-0048 not responsive keyword false hit Please note: Please delete the duplicate lunch order from the event spreadsheet.
RYD-0049 not responsive keyword false hit Ticket note: The delayed chair shipment arrived at the front desk this afternoon.
RYD-0054 not responsive keyword false hit Following up: Calder Yard is listed on the volunteer cleanup map, not the contract file.

Precision is the responsive share of reviewed search results. Recall is the share of all responsive records that search finds. The terms match whole words and the phrase Calder Yard. For example, log does not match chronology.

In this fixture, the terms find 39 records. Of those, 22 are responsive under the constructed labels. Reviewers can calculate precision after reviewing the matches: 22 divided by 39.

Recall is 22 divided by 36. Reviewers normally lack that denominator. Here, fictional labels reveal the records that keyword search missed.

Rank the rest from the reviewed seed

The reviewed seed is the initial set of keyword matches. It has at least eight records of each class. That count avoids the glmnet warning for very small classes under options(warn = 2). The model uses tf-idf features and penalized logistic regression. It trains only on reviewed records and scores only unreviewed records.

The author fixes the stopping rule before model execution. Review begins with the keyword-match seed, followed by two ranked batches of ten. This workload rule has no statistical guarantee. The elusion sample provides an estimate of remaining missed records.

Show the ranking code 
fit_review_model <- function(training_rows) {
  class_counts <- table(training_rows$label)
  stopifnot(min(class_counts) >= 8L)

  review_recipe <- recipe(label ~ text, data = training_rows) |>
    step_tokenize(text) |>
    step_tokenfilter(text, max_tokens = 80) |>
    step_tfidf(text)

  review_model <- logistic_reg(penalty = 0.01, mixture = 1) |>
    set_engine("glmnet") |>
    set_mode("classification")

  workflow() |>
    add_recipe(review_recipe) |>
    add_model(review_model) |>
    fit(data = training_rows)
}

base_columns <- names(review_scored)
reviewed_records <- review_scored |>
  filter(keyword_hit)
unreviewed_records <- review_scored |>
  filter(!keyword_hit)

review_batches <- list()
ranked_batches <- 2L
batch_size <- 10L
tie_rows <- integer(ranked_batches)
pool_rows <- integer(ranked_batches)
batch_tied_rows <- integer(ranked_batches)
tie_crosses_batch_edge <- logical(ranked_batches)

for (batch_id in seq_len(ranked_batches)) {
  fitted_review <- fit_review_model(reviewed_records)
  scored_pool <- unreviewed_records |>
    bind_cols(
      predict(
        fitted_review,
        new_data = unreviewed_records,
        type = "prob"
      )
    ) |>
    mutate(score = .pred_responsive) |>
    arrange(desc(score), doc_id)

  tied_in_pool <- duplicated(scored_pool$score) |
    duplicated(scored_pool$score, fromLast = TRUE)
  tie_rows[[batch_id]] <- sum(tied_in_pool)
  pool_rows[[batch_id]] <- nrow(scored_pool)
  batch_tied_rows[[batch_id]] <- sum(tied_in_pool[seq_len(batch_size)])
  tie_crosses_batch_edge[[batch_id]] <- identical(
    scored_pool$score[[batch_size]],
    scored_pool$score[[batch_size + 1L]]
  )

  batch_rows <- scored_pool |>
    slice_head(n = batch_size) |>
    mutate(batch = batch_id)

  review_batches[[batch_id]] <- batch_rows
  reviewed_records <- bind_rows(
    reviewed_records,
    batch_rows |> select(all_of(base_columns))
  )
  unreviewed_records <- anti_join(
    unreviewed_records,
    batch_rows |> select(doc_id),
    by = join_by(doc_id)
  )
}

ranked_reviewed <- bind_rows(review_batches)

ranked_review_display <- ranked_reviewed |>
  transmute(
    batch,
    doc_id,
    score = round(score, 3),
    reference_label = if_else(reference_responsive, "responsive", "not responsive"),
    construction_subtype,
    text
  )

batch_summary <- ranked_reviewed |>
  count(batch, reference_responsive, name = "records") |>
  complete(
    batch,
    reference_responsive = c(FALSE, TRUE),
    fill = list(records = 0L)
  ) |>
  mutate(label = if_else(reference_responsive, "responsive", "not responsive")) |>
  select(batch, label, records)

review_progress <- bind_rows(
  tibble(
    stage = "keyword seed",
    reviewed_this_stage = nrow(review_scored |> filter(keyword_hit)),
    responsive_found = sum(review_scored$keyword_hit & review_scored$reference_responsive)
  ),
  ranked_reviewed |>
    group_by(batch) |>
    summarise(
      stage = paste0("ranked batch ", first(batch)),
      reviewed_this_stage = n(),
      responsive_found = sum(reference_responsive),
      .groups = "drop"
    ) |>
    select(stage, reviewed_this_stage, responsive_found)
) |>
  mutate(
    cumulative_reviewed = cumsum(reviewed_this_stage),
    cumulative_responsive_found = cumsum(responsive_found),
    fiction_only_recall = cumulative_responsive_found /
      sum(review_scored$reference_responsive)
  )

review_stop <- tibble(
  reviewed = nrow(reviewed_records),
  responsive_found = sum(reviewed_records$reference_responsive),
  unreviewed = nrow(unreviewed_records),
  fiction_known_missed = sum(unreviewed_records$reference_responsive)
)

missed_records <- unreviewed_records |>
  filter(reference_responsive) |>
  transmute(
    doc_id,
    construction_subtype,
    text
  )
Ranked review batches after the keyword seed
Ranked batch Reference label Records
1 not responsive 6
1 responsive 4
2 not responsive 9
2 responsive 1
Review progress, with recall visible only because the collection is invented
Stage Reviewed at stage Responsive found at stage Cumulative reviewed Cumulative responsive found Fiction-only recall
keyword seed 39 22 39 22 0.611
ranked batch 1 10 4 49 26 0.722
ranked batch 2 10 1 59 27 0.750
Predeclared stopping point before the elusion sample
Reviewed Responsive found Unreviewed Fiction-known missed
59 27 181 9
Ranked records reviewed after the keyword seed
Batch Document Review score Reference label Construction note Text
1 RYD-0134 1.000 not responsive not responsive For today: If catering calls back, confirm the allergy cards are ready and ask about serving spoons.
1 RYD-0227 1.000 responsive vocabulary mismatch responsive Monday update: The draft route sheet tells too much about the missed check.
1 RYD-0230 1.000 not responsive not responsive Following up: Tell catering that the afternoon seminar needs vegetarian wraps and chilled water.
1 RYD-0012 1.000 not responsive not responsive For today: If the printer stalls, ask Omar about the paper tray and switch to plain stock.
1 RYD-0076 0.997 responsive near duplicate responsive original Please note: Keep the rough yard timeline out of the shared packet today.
1 RYD-0042 0.997 not responsive not responsive Move the portable whiteboard back after the job fair and wipe off the employer notes. Thanks.
1 RYD-0157 0.996 responsive vocabulary mismatch responsive Keep the missed-check note out of the records room until after the walkthrough. Thanks.
1 RYD-0029 0.996 not responsive not responsive Monday update: Ask someone to check whether the lab tablets charged overnight before learners sign them out.
1 RYD-0089 0.994 responsive vocabulary mismatch responsive Reminder: Tell records the rough chronology is a working copy and not ready to share.
1 RYD-0095 0.993 not responsive not responsive Reminder: Please ask maintenance about the squeaky hinge in the east hallway before tonight's open clinic.
2 RYD-0078 0.976 not responsive not responsive Keep the orange table signs with outreach materials. They match the flyers.
2 RYD-0122 0.973 responsive near duplicate responsive copy Please note: Keep the rough yard timeline away from the shared packet today.
2 RYD-0194 0.910 not responsive not responsive Use my desk copy of the table layout if the printed plan disappears. Thanks.
2 RYD-0079 0.891 not responsive not responsive If Fiona confirms availability, update the mentor roster before the newsletter draft closes. Call me if anything is unclear.
2 RYD-0187 0.880 not responsive not responsive Please keep the forklift practice cones by the south wall so the morning route stays clear.
2 RYD-0101 0.856 not responsive not responsive Following up: Tell the opening team about the lobby heater if it is still running at 7:45.
2 RYD-0203 0.847 not responsive not responsive If the portal times out, save the student form again; call IT after that. Thanks.
2 RYD-0130 0.847 not responsive not responsive Chat at noon: I answered the vendor's question about where to unload snacks during the rain.
2 RYD-0056 0.836 not responsive not responsive Reminder: Use the lapel microphone if the instructor needs it during the auditorium welcome.
2 RYD-0044 0.779 not responsive not responsive The shuttle can carry the evening mentors today if they meet by the library. Thanks.
Fiction-only responsive records still unreviewed at the stop
Document Construction note Text
RYD-0016 vocabulary mismatch responsive Chat at noon: Do not put the Tuesday problem in the customer packet yet.
RYD-0110 vocabulary mismatch responsive Keep the red binder off the shared table until the visitor leaves. Call me if anything is unclear.
RYD-0123 vocabulary mismatch responsive Ticket note: The orange tab has the gap; send the white tab if anyone asks today.
RYD-0127 near duplicate responsive copy For today: Set aside the marked route sheet until after the customer visit.
RYD-0146 vocabulary mismatch responsive Forwarding from intake: Please scrub the yard note before the noon packet goes out.
RYD-0154 vocabulary mismatch responsive Following up: Use the clean timeline in the client packet and park the rough one elsewhere.
RYD-0165 near duplicate responsive original For today: Set the marked route sheet aside until the customer visit is over.
RYD-0177 vocabulary mismatch responsive The side folder has the version with the missing Tuesday stop.
RYD-0236 vocabulary mismatch responsive Per scheduling: The file export should hide the late safety check until we settle the wording.

Reviewers classify records during review, so the team can count responsive records found. Real-matter recall requires further measurement. The fiction-only recall column uses hidden labels for unreviewed records.

In the two scoring rounds, 137 of 392 scored-pool rows had exact score ties. The code orders ties by document ID for reproducibility. Lower IDs do not indicate greater relevance.

Ranked batch 2 had a tie across its boundary. The document-ID rule selected which tied records entered review.

Show the plotting code 
source("R/lesson-figures.R")

gain_points <- review_progress |>
  add_row(
    stage = "start",
    responsive_found = 0L,
    cumulative_reviewed = 0L,
    cumulative_responsive_found = 0L,
    .before = 1
  ) |>
  mutate(
    point_label = case_when(
      stage == "start" ~ "",
      stage == "keyword seed" ~ str_c("keyword seed: ", responsive_found, " found"),
      TRUE ~ str_c(stage, ": +", responsive_found)
    )
  )

ggplot(
  gain_points,
  aes(x = cumulative_reviewed, y = cumulative_responsive_found)
) +
  geom_line(colour = lesson_colours[["accent"]], linewidth = 0.9) +
  geom_point(colour = lesson_colours[["accent"]], size = 2.8) +
  geom_text(
    aes(label = point_label),
    hjust = 0,
    vjust = 1.8,
    family = lesson_font,
    size = 9.5,
    size.unit = "pt",
    colour = lesson_colours[["body"]]
  ) +
  scale_x_continuous(
    breaks = c(0, review_progress$cumulative_reviewed),
    limits = c(0, max(review_progress$cumulative_reviewed) + 12)
  ) +
  scale_y_continuous(
    breaks = seq(0, 30, by = 10),
    limits = c(0, 30)
  ) +
  coord_cartesian(clip = "off") +
  labs(
    title = "The ranked batches add 4 responsive records, then 1",
    subtitle = "Responsive records found against records reviewed, as the reviewers see it",
    x = "Records reviewed",
    y = "Responsive records found"
  ) +
  theme_lesson(grid = "both")
Line chart of responsive records found against records reviewed. Both axes start at zero. The line starts at zero and reaches 22 found at 39 reviewed, then 26 at 49 and 27 at 59.
Figure 1: Responsive records found as review proceeds through the keyword seed and two ranked batches.

The gain curve shows information available to reviewers: responsive records found versus records reviewed. It does not establish that ranked review outperforms keyword search. The same author wrote the collection and lesson. This collection therefore provides weak evidence for comparative claims.

Select one sample from unreviewed records

An elusion sample is a random sample of unreviewed documents. The author declares the seed and sample size before selection. The code selects one sample. Its results do not alter the model or stopping decision.

The interval treats the unreviewed records as a finite population with sample selection without replacement. It evaluates every possible missed-record count under the hypergeometric distribution through phyper(). It retains counts that do not place the observed sample count in either 2.5 percent tail. It then converts missed-record bounds to recall bounds.

elusion_seed <- 7401L
elusion_n <- 50L

set.seed(elusion_seed)
elusion_sample <- unreviewed_records |>
  slice_sample(n = elusion_n) |>
  arrange(doc_id)

observed_missed_in_sample <- sum(elusion_sample$reference_responsive)
unreviewed_total <- nrow(unreviewed_records)
found_total <- sum(reviewed_records$reference_responsive)

possible_missed <- 0:unreviewed_total
kept_missed <- possible_missed[
  vapply(
    possible_missed,
    \(missed) {
      lower_tail <- phyper(
        observed_missed_in_sample,
        missed,
        unreviewed_total - missed,
        elusion_n
      )
      upper_tail <- 1 - phyper(
        observed_missed_in_sample - 1L,
        missed,
        unreviewed_total - missed,
        elusion_n
      )
      lower_tail >= 0.025 && upper_tail >= 0.025
    },
    logical(1)
  )
]

missed_interval <- range(kept_missed)
recall_interval <- found_total / (found_total + rev(missed_interval))
point_missed <- unreviewed_total *
  observed_missed_in_sample / elusion_n
point_recall <- found_total / (found_total + point_missed)
known_recall <- found_total /
  (found_total + sum(unreviewed_records$reference_responsive))

elusion_result <- tibble(
  unreviewed = unreviewed_total,
  sample_size = elusion_n,
  responsive_in_sample = observed_missed_in_sample,
  point_recall = point_recall,
  recall_low = recall_interval[1],
  recall_high = recall_interval[2],
  fiction_known_recall = known_recall
)

binomial_interval <- stats::binom.test(
  observed_missed_in_sample,
  elusion_n
)$conf.int
binomial_missed <- unreviewed_total * binomial_interval
binomial_recall <- found_total / (found_total + rev(binomial_missed))
maximum_possible_recall <- found_total /
  (found_total + observed_missed_in_sample)

binomial_warning <- tibble(
  method = "binomial Clopper-Pearson on the elusion rate in the sample",
  printed_recall_low = binomial_recall[1],
  printed_recall_high = binomial_recall[2],
  maximum_possible_recall = maximum_possible_recall
)
One finite-population elusion sample after the stop
Unreviewed Sample size Responsive in sample Point recall Interval low Interval high Fiction-known recall
181 50 2 0.789 0.551 0.931 0.75
A binomial interval treats the fixed collection of unreviewed records as if it were very large
Method Printed recall low Printed recall high Maximum possible recall after the sample
binomial Clopper-Pearson on the elusion rate in the sample 0.521 0.968 0.931

The elusion seed is 7401, the builder’s base seed. A rule fixes it before sample selection. The sample found 2 responsive records in 50.

The point estimate indicates about 7.2 missed records among the unreviewed records.

The exact finite-population interval includes 2 to 22 missed records. This corresponds to recall from about 0.551 to 0.931. The fictional reference reveals the correct missed count: 9.

A binomial Clopper-Pearson interval treats the population as very large and assumes sample selection with replacement. Here it gives a wider lower recall bound than the finite-population interval.

Confidence intervals describe a repeated-sampling procedure. They do not assign a 95 percent probability to this one interval.

Selecting samples and reviewing more records until a result looks acceptable would create statistical bias, a systematic distortion of the reported estimate. Select the sample once, after review stops.

Monitor new items in arrival order

Media monitoring processes items as they arrive. Duplicate copies can appear, and names such as Riverton can refer to multiple places. This short stream monitors organizations and program names, not individual people. It stores item IDs, decisions, and duplicate links. Real review sets may contain personal or privileged material. This lesson excludes that material.

Here, monitoring means an alerting rule for incoming text. It differs from lesson 47’s deployed-model monitoring and lesson 56’s burst detection.

The author wrote the watch phrases after reading this constructed stream. The alert count demonstrates record keeping, not precision that applies to other streams. The pattern matches Calder Yard, contract records, inspection messages, route notes, and missed check.

The code suppresses a duplicate alert only when its character-trigram Jaccard distance to an earlier alert is at most 0.35. This distance compares sets of three-character sequences.

Show the monitoring code 
normalize_stream_text <- function(text) {
  text |>
    str_to_lower(locale = "en") |>
    str_replace_all("[^a-z0-9]+", " ") |>
    str_squish()
}

stream_decisions <- monitoring_stream |>
  arrange(arrival_order) |>
  mutate(
    normalized_text = normalize_stream_text(text),
    topic_hit = str_detect(
      text,
      regex("Calder Yard|contract records|inspection messages|route notes|missed check", ignore_case = TRUE)
    ),
    ambiguous_riverton = str_detect(text, regex("\\bRiverton\\b", ignore_case = TRUE)) &
      !str_detect(text, regex("Calder|Skills Centre|Workforce Lab", ignore_case = TRUE)),
    raw_alert = topic_hit & !ambiguous_riverton
  )

previous_alerts <- character()
previous_alert_ids <- character()
duplicate_of <- rep(NA_character_, nrow(stream_decisions))
send_alert <- rep(FALSE, nrow(stream_decisions))

for (row in seq_len(nrow(stream_decisions))) {
  if (!stream_decisions$raw_alert[[row]]) {
    next
  }

  if (length(previous_alerts) == 0L) {
    send_alert[[row]] <- TRUE
    previous_alerts <- c(previous_alerts, stream_decisions$normalized_text[[row]])
    previous_alert_ids <- c(previous_alert_ids, stream_decisions$stream_id[[row]])
    next
  }

  distances <- stringdist(
    stream_decisions$normalized_text[[row]],
    previous_alerts,
    method = "jaccard",
    q = 3
  )
  if (min(distances) <= 0.35) {
    duplicate_of[[row]] <- previous_alert_ids[[which.min(distances)]]
  } else {
    send_alert[[row]] <- TRUE
    previous_alerts <- c(previous_alerts, stream_decisions$normalized_text[[row]])
    previous_alert_ids <- c(previous_alert_ids, stream_decisions$stream_id[[row]])
  }
}

stream_decisions <- stream_decisions |>
  mutate(
    matched_prior_alert = duplicate_of,
    alert_sent = send_alert,
    decision = case_when(
      alert_sent ~ "alert sent",
      raw_alert & !is.na(matched_prior_alert) ~ "suppressed duplicate",
      ambiguous_riverton ~ "needs review: ambiguous Riverton",
      TRUE ~ "no alert"
    )
  )

monitoring_results <- tibble(
  alerts_sent = sum(stream_decisions$alert_sent),
  relevant_alerts = sum(stream_decisions$alert_sent & stream_decisions$reference_alert),
  false_alerts = sum(stream_decisions$alert_sent & !stream_decisions$reference_alert),
  relevant_items_missed = sum(!stream_decisions$alert_sent & stream_decisions$reference_alert & is.na(stream_decisions$matched_prior_alert)),
  suppressed_duplicates = sum(!is.na(stream_decisions$matched_prior_alert)),
  repeated_alerts = sum(stream_decisions$alert_sent & !is.na(stream_decisions$near_duplicate_of)),
  alert_precision = relevant_alerts / alerts_sent
)

stream_display <- stream_decisions |>
  select(
    arrival_order,
    stream_id,
    near_duplicate_of,
    matched_prior_alert,
    text,
    decision,
    reference_alert
  ) |>
  mutate(
    near_duplicate_of = coalesce(near_duplicate_of, "not a copy"),
    matched_prior_alert = coalesce(matched_prior_alert, "not suppressed")
  )
The monitoring rule processes items in arrival order
Arrival Item Fixture duplicate of Matched prior alert Text Rule decision Reference alert
1 MON-008 not a copy not suppressed Riverton Skills Centre posts a reminder about fall enrollment. no alert FALSE
2 MON-003 not a copy not suppressed Calder Yard contractor confirms a safety walkthrough next week. alert sent TRUE
3 MON-024 MON-003 MON-003 Syndicated brief: Calder Yard contractor confirms safety walkthrough next week. suppressed duplicate TRUE
4 MON-004 not a copy not suppressed Riverton Tidewater hosts a harbour festival unrelated to the Calder city program. no alert FALSE
5 MON-012 not a copy not suppressed A vendor post says a delayed inspection log at Calder Yard is under review. alert sent TRUE
6 MON-010 not a copy not suppressed Community calendar lists a classroom inspection drill at Riverton Skills Centre. no alert FALSE
7 MON-015 MON-012 not suppressed Wire copy: delayed inspection log at Calder Yard is under review. alert sent TRUE
8 MON-002 not a copy not suppressed City notice says bus passes remain available for workforce classes. no alert FALSE
9 MON-006 not a copy not suppressed A post mentions Riverton without enough context to pick the city or the town. needs review: ambiguous Riverton FALSE
10 MON-021 not a copy not suppressed Maintenance bulletin reports new guardrail signs at Calder Yard. alert sent FALSE
11 MON-018 MON-021 MON-021 Syndicated maintenance bulletin reports new guardrail signs at Calder Yard. suppressed duplicate FALSE
12 MON-014 not a copy not suppressed Local brief says contract records will be audited after a missed check. alert sent TRUE
13 MON-017 not a copy not suppressed Training newsletter asks volunteers to inspect donated laptops. no alert FALSE
14 MON-007 MON-014 not suppressed Vendor post repeats that contract records will be audited after a missed check. alert sent TRUE
15 MON-013 not a copy not suppressed Riverton Calder council posts a meeting agenda about yard procurement. no alert FALSE
16 MON-019 not a copy not suppressed Harbour blog writes about Riverton Tidewater seafood trucks. needs review: ambiguous Riverton FALSE
17 MON-011 not a copy not suppressed Safety clerk account says old route notes should not be shared. alert sent TRUE
18 MON-009 not a copy not suppressed Civic notice announces a parking delay near the Calder library. no alert FALSE
19 MON-023 MON-009 not suppressed Syndicated civic notice announces a parking delay near the Calder library. no alert FALSE
20 MON-001 not a copy not suppressed Workforce Lab posts a non-alert reminder about child care during evening classes. no alert FALSE
21 MON-005 not a copy not suppressed Calder Yard contractor denies deleting any inspection messages. alert sent TRUE
22 MON-016 MON-005 MON-005 Wire copy: Calder Yard contractor denies deleting any inspection messages. suppressed duplicate TRUE
23 MON-022 not a copy not suppressed Riverton school board posts a classroom inspection checklist. needs review: ambiguous Riverton FALSE
24 MON-020 not a copy not suppressed A regional item says Riverton won a rowing race, with no contract context. needs review: ambiguous Riverton FALSE
Alert precision uses alerts sent after duplicate suppression as the denominator
Alerts sent Relevant alerts False alerts Relevant items missed Suppressed duplicates Repeated alerts Alert precision
8 7 1 0 3 2 0.875

Alert precision is 7 relevant alerts out of 8 alerts sent. The denominator counts alerts after duplicate suppression. The audit record retains suppressed copies, but they produce no new alert. In a real stream, reviewers usually find missed items through a sample of non-alerts, as with the elusion sample.

The ambiguity rule is limited. It sends MON-019, the Tidewater seafood item, to review despite text that indicates a different subject from Calder Yard.

What this does not show

This page gives no legal advice or proof that a review is defensible. It calculates no estimate of real-system recall. It excludes privilege review, redaction, email threading, family coding, dense retrieval, and language-model relevance labeling. The author’s labels are perfectly consistent. Real reviewers disagree.

The review model’s scores are ranks for this workflow, not calibrated probabilities of responsiveness. The monitoring rule is a small constructed example and is not suitable for watching individual people.

What to remember

  • Keyword precision is visible after reviewing matches. Real-work recall requires further measurement.
  • Ranked review must train on reviewed records and score only unreviewed records.
  • A gain curve shows responsive records found. Recall requires a complete fictional answer key here.
  • Select an elusion sample once, after the stop.
  • A finite-population interval matches selection without replacement from a fixed collection.
  • Monitoring streams need arrival order, duplicate suppression, ambiguity handling, and a named precision denominator.

The review log records 2 responsive records in the sample and a wide interval. Review stopped, but uncertainty about the unreviewed records remains.

Sources