Loading data from an API

Ask an online service for information. Examine its reply.

source data loading
API
JSON
Learn how to create a data table from an API reply and examine the result in R.

A librarian gathering possible books about natural language processing could copy search results from a catalog into a table. Search results are candidates, not recommendations. The librarian still needs to judge whether each record is relevant, accurate, and useful.

An application programming interface, or API, defines how software can request information from another service. This lesson reconstructs a query to the Open Library Search API and runs checks on the saved reply. The displayed table uses a dated response. That saved response preserves the published output if the service becomes unavailable or changes its ranking.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • explain the request-and-response pattern used by an API.
  • add search choices to an API request.
  • examine the request’s success status.
  • convert a JSON response to a table in R.
  • replay a rate limit, a second page, and a changed schema from saved responses.
  • handle changing results, service limits, and secret API keys.

Start with a request and a reply

Every API exchange in this lesson has two parts:

  1. R sends a request to a web address.
  2. The service sends a response back.

The request records the librarian’s question. The response contains a status code, descriptive details called headers, and a body with the requested data.

Our request uses this endpoint, the web address assigned to book searches: https://openlibrary.org/search.json.

Build the request

We build the request before sending it so each choice remains visible. The httr2 package adds those choices one step at a time.

library(readr)
library(dplyr)
library(tibble)
library(purrr)
library(stringr)
library(httr2)
library(digest)
library(htmltools)

book_request <- httr2::request(
  "https://openlibrary.org/search.json"
) |>
  httr2::req_url_query(
    q = "natural language processing",
    fields = str_c(
      c("key", "title", "author_name", "first_publish_year"),
      collapse = ","
    ),
    limit = 5
  ) |>
  httr2::req_user_agent(
    str_c(
      "periodic-table-nlp-r/1.0 ",
      "(https://github.com/farach/periodic_table_nlp)"
    )
  ) |>
  httr2::req_timeout(20) |>
  httr2::req_retry(max_tries = 3)

book_request
<httr2_request>
GET https://openlibrary.org/search.json?q=natural%20language%20processing&fields=key%2Ctitle%2Cauthor_name%2Cfirst_publish_year&limit=5
Body: empty
Options:
* useragent     : "periodic-table-nlp-r/1.0 (https://github.com/farach/periodic_table_nlp)"
* timeout_ms    : 20000
* connecttimeout: 0
Policies:
* retry_max_tries        : 3
* retry_on_failure       : FALSE
* retry_failure_threshold: Inf
* retry_failure_timeout  : 30
* retry_realm            : "openlibrary.org"

The printed request begins with GET, the standard method used here to ask a service to send information. Printing it lets us examine the destination and search choices without contacting the service again.

The choices added to the address are called query parameters:

Query parameters used in the Open Library request
Parameter What it asks for
q Books related to the words “natural language processing.”
fields Only the four pieces of information used in this lesson.
limit No more than five results.

The user-agent identifies the project making the request. The 20-second timeout keeps R from waiting indefinitely. The retry setting permits at most three attempts when a temporary connection problem interrupts the exchange. It does not guarantee a response from an unavailable service.

Understand what happened when the request was sent

The original request ran once on the recorded retrieval date. Here, req_perform() receives the same request but replays saved response bytes through a local mock. The example exercises the request-to-response step without contacting Open Library again. A normal req_perform() call stops on unsuccessful HTTP statuses such as 404 or 500.

JSON, short for JavaScript Object Notation, is a text format that can contain named values, lists, and records. Many programming languages can read it, which makes it common in API responses.

Keep a dated response

One successful request cannot guarantee future service availability or an unchanged ranking. The repository keeps the JSON response and its retrieval time, request address, HTTP status, and content type. It also records the record level, licensing reference, and fingerprint.

fixture_path <- "data/openlibrary-nlp-search.json"
metadata_path <- "data/openlibrary-nlp-search-metadata.csv"

fixture_metadata <- read_csv(
  metadata_path,
  col_types = cols(
    retrieved_at_utc = col_character(),
    request_url = col_character(),
    http_status = col_character(),
    content_type = col_character(),
    record_level = col_character(),
    api_documentation = col_character(),
    licensing_reference = col_character(),
    md5 = col_character()
  )
)
fixture_lines <- read_lines(fixture_path)
fixture_md5 <- digest::digest(
  paste(fixture_lines, collapse = "\n"),
  algo = "md5",
  serialize = FALSE
)
Provenance for the saved API response
Fixture detail Recorded value
Retrieved at 2026-08-28T12:18:04Z
HTTP status 200
Content type application/json
Record level work-level catalog record
File fingerprint d65543b9f71a664cf568c25a436ad614

The MD5 fingerprint detects an accidental change to the saved file. It is not proof that the source is accurate or trustworthy. The request address and API documentation remain in the metadata file for a later audit.

Perform the request against the saved reply

A mock replaces the network exchange with a local function. httr2 passes the prepared request to req_perform() and returns an httr2_response. The mock supplies the dated status, header, and JSON body.

replay_saved_response <- function(req) {
  httr2::response(
    status_code = as.integer(fixture_metadata$http_status),
    url = req$url,
    method = "GET",
    headers = list(
      `content-type` = fixture_metadata$content_type
    ),
    body = charToRaw(paste(fixture_lines, collapse = "\n"))
  )
}

fixture_response <- httr2::req_perform(
  book_request,
  mock = replay_saved_response
)

response_summary <- tibble(
  detail = c("Response class", "HTTP status", "Content type"),
  value = c(
    class(fixture_response)[[1]],
    as.character(httr2::resp_status(fixture_response)),
    httr2::resp_header(fixture_response, "content-type")
  )
)
The prepared request returns a replayed httr2 response
Response detail Value
Response class httr2_response
HTTP status 200
Content type application/json

Convert the saved response to a table

Open Library places search results inside a JSON section named docs. resp_body_json() translates that section into rows and columns because this response has the same fields for each catalog record.

fixture_data <- httr2::resp_body_json(
  fixture_response,
  simplifyVector = TRUE
)
book_records <- as_tibble(fixture_data$docs)

format_authors <- function(author_names) {
  if (length(author_names) == 0 || all(is.na(author_names))) {
    return("Not listed")
  }

  str_c(author_names, collapse = ", ")
}

books <- book_records |>
  transmute(
    title = title,
    authors = map_chr(author_name, format_authors),
    first_published = first_publish_year,
    catalog_url = str_c("https://openlibrary.org", key)
  )

escape_html <- function(value) {
  as.character(htmltools::htmlEscape(value))
}

catalog_links <- map2_chr(
  books$catalog_url,
  books$title,
  \(url, title) {
    safe_url <- as.character(
      htmltools::htmlEscape(url, attribute = TRUE)
    )
    safe_title <- as.character(
      htmltools::htmlEscape(
        title,
        attribute = TRUE
      )
    )
    sprintf(
      '<a href="%s" aria-label="View Open Library record for %s">View record</a>',
      safe_url,
      safe_title
    )
  }
)

display_books <- books |>
  transmute(
    title = escape_html(title),
    authors = escape_html(authors),
    first_published = first_published,
    catalog_link = catalog_links
  )
Five candidate work-level catalog records retrieved 2026-08-28T12:18:04Z
Title Author or authors First published Open Library record
Natural language processing and information retrieval Tanveer Siddiqui, U. S. Tiwary 2008 View record
Natural Language Processing With Python Edward Loper, Steven Bird, Ewan Klein 2009 View record
Practical Natural Language Processing Sowmya Vajjala, Bodhisattwa Majumder, Anuj Gupta, Harshit Surana 2020 View record
Natural Language Processing with Transformers Lewis Tunstall, Leandro von Werra, Thomas Wolf 2022 View record
Natural language processing Randall Rustin 1973 View record

Open Library returns work-level catalog records by default. A row can represent many editions of the same work. The table contains candidate search results, not an examined reading list or a judgment about a book’s quality.

The code escapes author names and titles before display so API text cannot become page markup. It renders only validated Open Library addresses as links.

Live data can change

Open Library may add records, correct dates, or reorder results. This page does not contact the service during rendering. The displayed table therefore remains based on the saved response.

For a new request, examine the response structure rather than requiring five specific titles:

  • the service returned a successful status.
  • the response is JSON.
  • the response contains between one and five records.
  • every returned record has a title.

The saved fixture makes the displayed table repeatable. Its metadata records when and how retrieval occurred. Open Library states that the Internet Archive does not assert new proprietary rights over the database. Existing rights may vary by contribution and jurisdiction.

Read the current licensing statement. Do not assume that one license covers all contributions or every way you plan to use them.

Respect the service’s rate limit and stated preference for low-volume, human-facing requests.

Replay refusal, pagination, and schema changes

The next examples cover three cases: refusal, results that require several pages, and changed response fields. The lesson constructs these examples with httr2::response(), using documented status codes and body structures. No request leaves the machine.

saved_response <- function(
  status,
  body,
  url = book_request$url,
  extra_headers = list()
) {
  httr2::response(
    status_code = status,
    url = url,
    method = "GET",
    headers = c(
      list("content-type" = "application/json"),
      extra_headers
    ),
    body = charToRaw(body)
  )
}

make_docs <- function(keys, titles) {
  str_c(
    sprintf(
      '{"key": "%s", "title": "%s"}',
      keys,
      titles
    ),
    collapse = ", "
  )
}

page_one_request <- book_request |>
  httr2::req_url_query(offset = 0)
page_two_request <- book_request |>
  httr2::req_url_query(offset = 5)

rate_limited <- saved_response(
  429L,
  '{"error": "too many requests"}',
  url = page_one_request$url,
  extra_headers = list("retry-after" = "30")
)
date_limited <- saved_response(
  429L,
  '{"error": "too many requests"}',
  url = page_one_request$url,
  extra_headers = list(
    "retry-after" = "Wed, 21 Oct 2037 07:28:00 GMT"
  )
)

page_one <- saved_response(
  200L,
  str_c(
    '{"numFound": 7, "start": 0, "docs": [',
    make_docs(
      sprintf("/works/OL%d", 1:5),
      sprintf("Candidate title %d", 1:5)
    ),
    "]}"
  ),
  url = page_one_request$url
)

page_two <- saved_response(
  200L,
  str_c(
    '{"numFound": 7, "start": 5, "docs": [',
    make_docs(
      c("/works/OL5", "/works/OL6"),
      c("Candidate title 5", "Candidate title 6")
    ),
    "]}"
  ),
  url = page_two_request$url
)

field_renamed <- saved_response(
  200L,
  str_c(
    '{"numFound": 1, "start": 0, "docs": [',
    '{"key": "/works/OL7", "name": "Title moved to a new field"}',
    "]}"
  ),
  url = page_one_request$url
)

docs_missing <- saved_response(
  200L,
  '{"numFound": 1, "start": 0}',
  url = page_one_request$url
)

httr2::resp_status(rate_limited)
[1] 429

The following function must handle each saved reply.

required_fields <- c("key", "title")
required_top_fields <- c("numFound", "start", "docs")
retry_reference_time <- as.POSIXct(
  "2037-10-21 07:27:30",
  tz = "GMT"
)

parse_retry_after <- function(value, now) {
  if (length(value) == 0L || is.na(value) || identical(value, "")) {
    return(NA_integer_)
  }
  if (str_detect(value, "^[0-9]+$")) {
    return(as.integer(value))
  }

  parsed <- suppressWarnings(curl::parse_date(value))
  if (length(parsed) == 0L || is.na(parsed)) {
    stop("Retry-After is neither delay-seconds nor an HTTP-date")
  }

  max(
    0L,
    as.integer(ceiling(as.numeric(difftime(parsed, now, units = "secs"))))
  )
}

read_saved_page <- function(
  response,
  page_size,
  retry_now = retry_reference_time
) {
  status <- httr2::resp_status(response)

  if (identical(status, 429L)) {
    retry_after <- httr2::resp_header(response, "retry-after")
    return(list(
      outcome = "rate limited",
      request_url = response$url,
      wait_seconds = parse_retry_after(retry_after, retry_now),
      reported_total = NA_integer_,
      next_start = NA_integer_,
      records = tibble(
        key = character(),
        title = character()
      )
    ))
  }

  if (!identical(status, 200L)) {
    return(list(
      outcome = "http error",
      request_url = response$url,
      wait_seconds = NA_integer_,
      reported_total = NA_integer_,
      next_start = NA_integer_,
      records = tibble(
        key = character(),
        title = character()
      )
    ))
  }

  body <- httr2::resp_body_json(response)
  missing_top_fields <- setdiff(required_top_fields, names(body))
  if (length(missing_top_fields) > 0L) {
    return(list(
      outcome = str_c(
        "schema failure: missing top-level ",
        str_c(missing_top_fields, collapse = ", ")
      ),
      request_url = response$url,
      wait_seconds = NA_integer_,
      reported_total = NA_integer_,
      next_start = NA_integer_,
      records = tibble(
        key = character(),
        title = character()
      )
    ))
  }

  top_level_types_valid <-
    is.numeric(body$numFound) &&
      length(body$numFound) == 1L &&
      is.numeric(body$start) &&
      length(body$start) == 1L &&
      is.list(body$docs)
  if (!top_level_types_valid) {
    return(list(
      outcome = "schema failure: invalid top-level types",
      request_url = response$url,
      wait_seconds = NA_integer_,
      reported_total = NA_integer_,
      next_start = NA_integer_,
      records = tibble(
        key = character(),
        title = character()
      )
    ))
  }

  documents <- body$docs
  absent <- required_fields[
    !map_lgl(
      required_fields,
      \(field) all(map_lgl(documents, \(record) field %in% names(record)))
    )
  ]

  if (length(absent) > 0) {
    return(list(
      outcome = str_c(
        "schema failure: missing ",
        str_c(absent, collapse = ", ")
      ),
      request_url = response$url,
      wait_seconds = NA_integer_,
      reported_total = as.integer(body$numFound),
      next_start = NA_integer_,
      records = tibble(
        key = character(),
        title = character()
      )
    ))
  }

  start <- as.integer(body$start)
  reported_total <- as.integer(body$numFound)
  records_read <- length(documents)
  more_pages <- start + records_read < reported_total
  if (records_read == 0L && more_pages) {
    return(list(
      outcome = "schema failure: empty page before reported total",
      request_url = response$url,
      wait_seconds = NA_integer_,
      reported_total = reported_total,
      next_start = NA_integer_,
      records = tibble(
        key = character(),
        title = character()
      )
    ))
  }

  list(
    outcome = if (more_pages) "more pages" else "last page",
    request_url = response$url,
    wait_seconds = NA_integer_,
    reported_total = reported_total,
    next_start = if (more_pages) start + records_read else NA_integer_,
    records = tibble(
      key = map_chr(documents, "key"),
      title = map_chr(documents, "title")
    )
  )
}

page_size <- 5L
refused <- read_saved_page(rate_limited, page_size)
date_refused <- read_saved_page(date_limited, page_size)
first_page <- read_saved_page(page_one, page_size)
second_page <- read_saved_page(page_two, page_size)
drifted <- read_saved_page(field_renamed, page_size)
missing_docs <- read_saved_page(docs_missing, page_size)
missing_retry_header <- saved_response(
  429L,
  '{"error": "too many requests"}',
  url = page_one_request$url
)
missing_wait <- read_saved_page(missing_retry_header, page_size)

collected <- bind_rows(
  first_page$records,
  second_page$records
)
distinct_records <- collected |>
  distinct(key, .keep_all = TRUE)
shortfall <- first_page$reported_total - nrow(distinct_records)
Six saved replies and what the reading function did with each
Saved reply Outcome Records used
429 with delay-seconds rate limited 0
429 with HTTP-date rate limited 0
page 1 of results more pages 5
page 2 of results last page 2
title field renamed schema failure: missing title 0
docs field missing schema failure: missing top-level docs 0

Six saved replies exercised two retry formats, two result pages, and two schema failures. Each refusal or schema failure returned an empty records table with an explicit failure outcome.

The refusal returned no records. Its Retry-After header supplied the waiting time. Guessing that time or retrying immediately can cause a ban.

The first page reports start = 0, returns five records, and reports seven total records. The condition start + records returned < numFound therefore supplies the next offset. The second page reports start = 5. Adding its two records reaches the reported total, so the client stops. A full page alone does not prove that another page exists.

The renamed field requires a separate check. The record still parsed as JSON and contained a key. The required-field list detected the schema failure instead of accepting a table with a missing column.

Compare received and distinct record counts

The service reported seven matching records. Seven rows arrived across two pages, but only six were distinct. One work appeared on both pages. Ranking changes between requests can cause such duplication.

pagination_report <- tibble(
  measure = c(
    "reported by the service",
    "rows received",
    "distinct works kept",
    "unexplained shortfall"
  ),
  value = c(
    first_page$reported_total,
    nrow(collected),
    nrow(distinct_records),
    shortfall
  )
)
Reported, received, and kept records after paging
Measure Count
reported by the service 7
rows received 7
distinct works kept 6
unexplained shortfall 1

The shortfall of one shows a difference from the reported total. Reporting six records alone would make the collection appear complete.

Record the service’s reported total. Record the received and retained row counts. Investigate any difference between those counts.

Keep API keys private

Open Library’s search endpoint does not require a key. Many other APIs do. An API key is a secret value that identifies an account and may permit billable or private access.

Never place a real key in:

  • an R or Quarto file.
  • a public repository.
  • a screenshot.
  • printed output.

An environment variable is a named setting that R can read at run time. Store secrets outside the project, usually in environment variables. Revoke any exposed key. Create a replacement key.

When an API request fails

A failed request leaves the librarian without an examined list. An empty table for that failure would hide the difference between “no books found” and “the service did not answer.” Report the failure. Keep the reported error. Then ask:

  1. Is the device connected to the internet?
  2. Is the endpoint address correct?
  3. Does the service require a key or other permission?
  4. Did the request exceed a rate limit?
  5. Did the response format change?

A normal req_perform() call stops on unsuccessful HTTP statuses. Here, read_saved_page() instead returns a recorded failure outcome for the saved refusals and schema failures. Its empty records table does not mean that the service found no matching records.

What to remember

  • An API request asks a service for structured information.
  • Query parameters describe what to return and how much.
  • Examine the status and response type before using the body.
  • Some JSON structures can be reshaped into rows and columns in R.
  • Saved replies let you examine refusals, pagination, and renamed fields before sending requests.
  • Honor the waiting time a service sends instead of guessing one.
  • Compare the total a service reports with the rows you actually kept.
  • Live API results may change even when the code stays the same.
  • A dated fixture can preserve the output used in a published example.
  • Keys and other secrets never belong in published code.

The librarian ends with five candidate records, not five recommendations. The request can be tried again, while the dated fixture preserves the reply shown here. Human judgment still decides which books belong on the reading list.

Sources