library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
library(digest)
library(rvest)
job_page_path <- "data/workforce/job-board.html"
job_page_lines <- read_lines(job_page_path)
job_page <- rvest::read_html(job_page_path)
page_title <- job_page |>
rvest::html_element("title") |>
rvest::html_text2()
meta_nodes <- job_page |>
rvest::html_elements("meta")
meta_table <- tibble(
element = "meta",
attributes = map_chr(
rvest::html_attrs(meta_nodes),
\(attributes) str_c(names(attributes), attributes, sep = "=", collapse = "; ")
)
)Extracting document metadata
Describe a source before text analysis
documents
metadata
workforce research
Learn how to extract HTML metadata and compute provenance fields for a Riverton source file.
A sentence table does not contain every detail about its source. The coordinator also needs the document’s origin and recorded properties.
Metadata describes a document rather than its main content. People use metadata to filter collections, cite documents, and examine their records.
Lesson 6 extracted sentences from this page and recorded a file fingerprint. This lesson examines the page’s title, declared character set, headings, and absent links. It separates recorded claims from facts calculated from the stored copy.
Note
The job board page is a fictional fixture written for metadata practice.
TipWhat you will learn
This lesson shows how to:
- explain what document metadata records.
- extract a title, meta elements, headings, and link counts from HTML.
- assemble one row per document.
- separate author claims from computed provenance facts.
Read the document description
The code reads source lines and recorded fields, extracts HTML metadata, and calculates a fingerprint. It combines those values into document records. A provenance field records a document’s origin and information needed to examine the stored copy.
| HTML element | Recorded value |
|---|---|
| title | Riverton workforce opportunities |
| meta | charset=utf-8 |
The title and charset value are claims written inside the file. They are useful metadata, but they come from the document itself.
Extract headings and links
Headings summarize the structure around the job text. Link targets record destination addresses. The author included no <a> elements in this fixture, so its link count is zero. Real job boards usually contain application links. A zero count therefore requires a file check.
h1_text <- job_page |>
rvest::html_elements("h1") |>
rvest::html_text2()
h2_text <- job_page |>
rvest::html_elements("h2") |>
rvest::html_text2()
heading_table <- bind_rows(
tibble(level = "h1", text = h1_text),
tibble(level = "h2", text = h2_text)
)
heading_summary <- heading_table |>
group_by(level) |>
summarise(
headings = n(),
example = first(text),
.groups = "drop"
)
link_targets <- job_page |>
rvest::html_elements("a") |>
rvest::html_attr("href")
link_markup_count <- sum(str_detect(job_page_lines, "<a(\\s|>)"))| Level | Headings | Example |
|---|---|---|
| h1 | 1 | Riverton workforce opportunities |
| h2 | 6 | Data analyst trainee |
The seven headings describe the page shape without reprinting the scraped sentence table. The zero link count is also metadata: this fixture has no outbound targets to preserve.
Make one row per document
A collection table needs one record per document. The display can shorten long lists while the code retains the complete extraction.
document_metadata <- tibble(
document_id = "job-board-html",
title = page_title,
meta_count = nrow(meta_table),
h1_count = sum(heading_table$level == "h1"),
h2_count = sum(heading_table$level == "h2"),
link_count = length(link_targets),
link_status = if_else(
length(link_targets) == 0L,
"no anchor elements in fixture",
str_c(link_targets, collapse = " | ")
)
)| Document ID | Title | Meta elements | H1 headings | H2 headings | Links | Link status |
|---|---|---|---|---|---|---|
| job-board-html | Riverton workforce opportunities | 1 | 1 | 6 | 0 | no anchor elements in fixture |
The row gives the coordinator a document-level record for filtering and audit. It keeps the zero-link result visible rather than hiding an empty vector in code.
Add provenance fields
HTML self-description is not enough. A collection also needs fields supplied by the collector and fields computed from the stored copy. The fingerprint here is SHA-256 over the file text joined with \n line endings.
recorded_page_metadata <- read_csv(
"data/workforce/job-board-metadata.csv",
na = character(),
col_types = cols(
source_file = col_character(),
created_for_lesson = col_character(),
retrieval_status = col_character(),
records = col_character(),
md5 = col_character(),
rights_note = col_character()
)
)
# Byte size depends on how the file's line endings were stored, so a size
# read straight from disk changes between systems. Measuring the same
# newline-joined text the fingerprint uses gives a number that describes the
# content rather than the copy.
source_lines <- read_lines(job_page_path)
source_text <- paste(source_lines, collapse = "\n")
source_size <- nchar(source_text, type = "bytes")
content_fingerprint <- digest::digest(
source_text,
algo = "sha256",
serialize = FALSE
)
provenance <- document_metadata |>
mutate(
source_file = job_page_path,
recorded_creation_date = recorded_page_metadata$created_for_lesson,
retrieval_status = recorded_page_metadata$retrieval_status,
size_bytes = source_size,
sha256 = content_fingerprint,
.after = document_id
)| Document ID | Source file | Recorded creation date | Retrieval status | Bytes | SHA-256 fingerprint | Author-claimed title |
|---|---|---|---|---|---|---|
| job-board-html | data/workforce/job-board.html | 2026-08-28 | synthetic archived page | 3441 | ce1c515cfda7658ff586b4f42dd1d71c805041a6c32bc88c97b54a20f0fd5eba | Riverton workforce opportunities |
Someone recorded the title, charset, creation date, and retrieval status as assertions. The code calculates the size and SHA-256 fingerprint from the stored copy. It follows the course’s hashing rule. It reads text lines, joins them with \n, and hashes the resulting string.
tools::md5sum() instead hashes raw file bytes, so line endings affect that result.
What to remember
- Metadata describes a document rather than its main content.
- HTML can claim its own title, character set, headings, and links.
- A collection needs recorded provenance fields and computed file checks.
- Hash text files with a recorded line-ending rule and SHA-256.
- Author claims and computed fingerprints answer different audit questions.
For this page, the title and creation date are claims. The byte count, zero links, and SHA-256 are checks on the saved fixture. Those checks help audit the file. They do not establish that the fixture is complete or comes from an external source.