Collecting text from a web page

Scrape a saved job board without losing its source

source data loading
scraping
workforce research
Learn how to extract job titles and training details from structured HTML with R.

The fictional Riverton Workforce Lab begins with six saved job postings. A worker needs to find each job’s training offers, schedule, experience requirements, and named skills. Combining all lines without source identifiers would make omissions and source errors difficult to find.

Web scraping uses code to collect information from web pages. The team will scrape the saved page, keep the job and sentence identifiers, and examine every record before it becomes research data. This first stage collects what the notices say. It does not classify the sentences or generalize from them.

The lesson uses static archived HTML. The saved file contains the text within its markup. The last sections show two page structures that the initial extraction method cannot handle.

Note

The employers, postings, and Riverton Workforce Lab are fictional. The author wrote the HTML file for this lesson without personal information.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • explain how HTML gives parts of a page names and structure.
  • select repeated job cards with rvest.
  • keep source identifiers beside extracted text.
  • compare scraped output with an expected fixture.
  • detect a renamed class and a page whose text is not in the markup.
  • describe legal, ethical, and technical checks needed before live scraping.

Begin with an archived page

Live websites change. A saved source file lets another researcher examine the same page later. The team records the file path, size, and fingerprint before extracting anything.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
library(digest)
library(rvest)

job_page_path <- "data/workforce/job-board.html"
job_page_lines <- read_lines(job_page_path)
job_page_fingerprint <- digest::digest(
  paste(job_page_lines, collapse = "\n"),
  algo = "md5",
  serialize = FALSE
)
job_page_metadata <- read_csv(
  "data/workforce/job-board-metadata.csv",
  col_types = cols(
    source_file = col_character(),
    created_for_lesson = col_character(),
    retrieval_status = col_character(),
    records = col_character(),
    md5 = col_character(),
    rights_note = col_character()
  )
)
# A size read from disk counts the line endings the local copy happens to
# use, so it changes between systems. Measuring the newline-joined text, the
# same text the fingerprint covers, describes the content instead.
job_page_size <- nchar(
  paste(job_page_lines, collapse = "\n"),
  type = "bytes"
)

tibble(
  source_file = job_page_path,
  bytes = job_page_size,
  fingerprint = job_page_fingerprint
)
# A tibble: 1 × 3
  source_file                   bytes fingerprint                     
  <chr>                         <int> <chr>                           
1 data/workforce/job-board.html  3441 2df73bc39f05311b6b59d24f52cb8f7b

The code compares the fingerprint with a stored value. The fingerprint changes if the archived HTML changes and can reveal an accidental edit. It cannot prove that the postings are correct or complete.

Read the page structure

HTML uses elements such as headings, paragraphs, and lists to organize a page. The example page places every posting inside an element whose class is job-card. A CSS selector is a short pattern that names the elements to collect.

job_page <- rvest::read_html(job_page_path)
job_cards <- rvest::html_elements(
  job_page,
  ".job-card"
)

length(job_cards)
[1] 6

R found six job cards. This count is the first signal that the selector matches the archived page we expected.

Extract one field at a time

Each card contains a job ID, posting date, title, employer, and location. The small helper below extracts one text field from each card.

card_text <- function(cards, selector) {
  map_chr(
    cards,
    \(card) {
      elements <- card |>
        rvest::html_elements(selector)
      stopifnot(identical(length(elements), 1L))

      elements |>
        rvest::html_text2()
    }
  )
}

jobs <- tibble(
  job_id = rvest::html_attr(
    job_cards,
    "data-job-id"
  ),
  posted = as.Date(
    rvest::html_attr(
      job_cards,
      "data-posted"
    )
  ),
  title = card_text(job_cards, ".job-title"),
  employer = card_text(job_cards, ".employer"),
  location = card_text(job_cards, ".location")
)
Six postings extracted from the archived job board
Job ID Posted Title Employer Location
J001 2026-08-01 Data analyst trainee Riverton Community Health Riverton
J002 2026-08-03 Maintenance technician Riverton Housing Cooperative Riverton
J003 2026-08-05 Junior web developer Riverton Digital Services Hybrid
J004 2026-08-08 Medical records clerk Riverton Family Clinic Riverton
J005 2026-08-10 Solar installation apprentice Riverton Solar Works Riverton region
J006 2026-08-12 Customer support specialist Riverton Transit Riverton

The job ID remains attached to every title. Each schedule or experience requirement must also retain the identifier of its source posting.

Keep sentences connected to their postings

The details in each card are list items with the class job-detail. We extract them in page order and add stable sentence IDs.

scraped_sentences <- map2(
  job_cards,
  jobs$job_id,
  \(card, job_id) {
    details <- card |>
      rvest::html_elements(".job-detail") |>
      rvest::html_text2()

    tibble(
      document_id = job_id,
      text = details
    )
  }
) |>
  list_rbind() |>
  mutate(
    sentence_id = sprintf("s%03d", row_number()),
    .before = document_id
  )

expected_sentences <- read_csv(
  "data/workforce/workforce_sentences.csv",
  col_types = cols(
    sentence_id = col_character(),
    document_id = col_character(),
    source_line = col_character(),
    text = col_character(),
    reference_label = col_character(),
    uncertainty = col_character(),
    annotator_id = col_character(),
    rationale = col_character(),
    codebook_version = col_character(),
    codebook_hash = col_character(),
    derived = col_character(),
    transformation = col_character()
  )
)
expected_job_sentences <- expected_sentences |>
  filter(document_id != "F001") |>
  select(sentence_id, document_id, text) |>
  as_tibble()
The first eight extracted job-detail sentences
Sentence ID Job ID Text
s001 J001 Paid 12-week training is provided.
s002 J001 No prior data experience is required.
s003 J001 Evening schedules are available.
s004 J001 Applicants need basic spreadsheet skills.
s005 J002 A high school diploma is required.
s006 J002 The employer pays for certification training.
s007 J002 Rotating night shifts are part of the job.
s008 J002 Workers must be able to lift 50 pounds.

All 22 sentences match the fixture in the same order. The check would stop the build if a selector omitted, duplicated, or reordered text.

The author wrote both the HTML and expected table for this lesson. The comparison provides a parser-behavior check, not independent truth. A live collection would need a separate audit sample.

When the page structure changes

The archived file preserves one page version. People can redesign a live board. The following HTML strings illustrate two different failures of the earlier extraction method.

The first is a redesign. The employer’s new template renames job-card to posting and moves the title into a heading.

redesigned_html <- str_c(
  "<html><body>",
  "<div class='posting' data-job-id='J007'>",
  "<h3 class='posting-title'>Warehouse trainee</h3>",
  "<ul><li class='posting-detail'>Paid training is provided.</li></ul>",
  "</div>",
  "</body></html>"
)

redesigned_page <- rvest::read_html(redesigned_html)
old_selector_hits <- length(
  rvest::html_elements(redesigned_page, ".job-card")
)
new_selector_hits <- length(
  rvest::html_elements(redesigned_page, ".posting")
)

collect_cards <- function(page, selector, expected) {
  cards <- rvest::html_elements(page, selector)
  if (!identical(length(cards), expected)) {
    stop(
      str_c(
        "selector ",
        selector,
        " found ",
        length(cards),
        " cards; expected ",
        expected
      ),
      call. = FALSE
    )
  }

  list(
    selector = selector,
    found = length(cards),
    expected = expected,
    status = "counts agree"
  )
}

drift_error <- tryCatch(
  collect_cards(redesigned_page, ".job-card", 1L),
  error = conditionMessage
)
The same selector against two versions of one site
Page Selector Cards found Cards expected
archived board .job-card 6 6
redesigned page .job-card 0 1

Zero matches can indicate either an empty page or a selector failure. Without an expected count, the scraper could report a redesign as a day with no postings. A check of the expected count stops the scraper when the selector no longer matches.

The second page’s visible container has no job text. The file stores a JSON payload in a script element. Application code could read and render that data later. This fixture does not execute JavaScript. It separates visible markup from embedded data.

dynamic_html <- str_c(
  "<html><body>",
  "<div id='board'></div>",
  "<script type='application/json' id='board-data'>",
  '{"jobs": [{"id": "J008", "title": "Night stocker"}]}',
  "</script>",
  "</body></html>"
)

dynamic_page <- rvest::read_html(dynamic_html)
visible_container_text <- rvest::html_element(
  dynamic_page,
  "#board"
) |>
  rvest::html_text2()
payload_text <- rvest::html_element(
  dynamic_page,
  "#board-data"
) |>
  rvest::html_text2()

tibble(
  source = c("visible container", "script payload"),
  characters = c(
    str_length(visible_container_text),
    str_length(payload_text)
  ),
  mentions_the_job = c(
    str_detect(visible_container_text, fixed("Night stocker")),
    str_detect(payload_text, fixed("Night stocker"))
  )
)
# A tibble: 2 × 3
  source            characters mentions_the_job
  <chr>                  <int> <lgl>           
1 visible container          0 FALSE           
2 script payload            52 TRUE

The container contains no characters. The job title exists only in the embedded script payload, which rvest can read as text but does not render into the container.

This example includes the payload in the file. Many sites request data separately after loading the initial page. Their saved initial HTML may contain no text to extract. An HTML parser cannot recover a response that the archive does not contain.

Three collection methods differ in cost and guarantees. First, search for a documented API that returns the data under stated terms. The page’s underlying data request may also accept direct calls. Direct calls can be faster but depend on an interface that the provider may change.

A headless browser runs the page as a browser would. R can control one through chromote, and Python can use Playwright or Selenium. This is the general solution and the most expensive of these methods to run and maintain.

Record the collection method. Text from a rendered page and text from a saved file are different evidence. Readers need to know which method produced the reported numbers.

A saved page is still incomplete evidence

The archived HTML shows what was captured, not every posting that existed. Before scraping a live site, the team must examine:

  1. the site’s terms and robots policy.
  2. whether collection is lawful and expected in the relevant jurisdiction.
  3. whether personal or sensitive information appears.
  4. permitted request frequency.
  5. whether JavaScript loads page content later.
  6. the dates, filters, and geographic limits of the results.
  7. how the saved HTML and retrieval time will be preserved.

The Robots Exclusion Protocol communicates crawler preferences. It is not a grant of legal permission, and permission does not settle every ethical question. A public page can still contain information that should not be collected or republished.

What to remember

  • Scraping converts page structure into data. It does not establish that the data is correct or representative.
  • Archive the source. Record its collection date.
  • Keep source IDs attached to extracted text.
  • Assert expected counts and content so selector failures cannot return zero rows without an error.
  • Zero matches and an empty page look identical without an expected count.
  • Some pages load text separately from the saved markup. An HTML parser cannot extract text absent from its input.
  • Examine terms, rate limits, privacy, and research ethics before live collection.

The team can examine 22 sentences with their source identifiers from the six notices. The table excludes one source, a fictional training flyer saved as an image. That source requires OCR instead of HTML extraction. Preserve the image. Examine the extracted wording before using it.

Sources