Natural language processing in R, one idea at a time
Natural language processing (NLP) is the study of how computers process human language. These lessons explain text tasks through R examples. Select a tile to read a lesson.
The tasks
Related tasks share a column. The six stages cover source data, language structure, models, generation, systems, and visualization. Position groups related subjects. It does not rank methods or prescribe a workflow. All 81 lessons are written.
For a course, read in number order. Later examples use R concepts from earlier lessons. For a specific problem, open the relevant lesson. Each page explains its NLP concept and assumptions.
For model-assisted analysis, read Using a language model in a reproducible analysis. The guide explains model records and compares model labels with a sample examined by a person.
For whole-system evaluation, read Beyond the periodic table. Its first lesson compares two versions in R and applies a release rule.
Beyond individual tasks: Beyond the periodic table explains systems, tools, and evaluation.
Select a solid-bordered tile to open its lesson. If the whole map does not fit, use the buttons, arrow keys, or horizontal scrolling.
Source and training dataSource data loading
- 1conceptBits to character encodingRead lesson
- 2practiceManual examples and pattern matchingRead lesson
- 3practiceLoading a structured data fileRead lesson
- 4practiceGenerating a corpusRead lesson
- 5practiceLoading from an APIRead lesson
- 6practiceCollecting text from a web pageRead lesson
- 7taskExtracting text from an imageRead lesson
Source and training dataTraining data generation
- 8practiceTurning decisions into labelsRead lesson
- 9practiceChoosing what to label nextRead lesson
- 10serviceInspecting provider schemas and provenanceRead lesson
- 11serviceWorking with several annotatorsRead lesson
- 12practiceCreating additional training examplesRead lesson
- 13practiceLabeling text with written rulesRead lesson
Language structureWord parsing
Language structureWord processing
Language structurePhrases and entities
Language structureEntity enrichment
Language structureSentences and paragraphs
Language structureDocuments
Models and analysisModel development
Models and analysisClassification
Models and analysisSignals and discovery
Models and analysisSimilarity
Language generationNatural language generation
Language systemsSystems
VisualizationInformation visualization
- 75applicationInteractive app creationRead lesson
- 76applicationAnnotated text visualizationRead lesson
- 77applicationWord cloudsRead lesson
- 78applicationWord embedding visualizationRead lesson
- 79applicationEvents on a timelineRead lesson
- 80applicationLocations on a mapRead lesson
- 81applicationKnowledge graph visualizationRead lesson
This is a teaching aid, not a scientific taxonomy or a complete account of the field. A taxonomy is a system of classification. The map includes tasks, methods, objectives, resources, services, applications, and work throughout a model’s life. Each tile names its item type. Some inherited group names describe different types of work. The relevant lessons explain these limits.
Why these lessons exist
Suppose your director asks how many of last quarter’s 40,000 customer comments are complaints. Next quarter’s support budget depends on the answer. An AI assistant reports 22 percent. You still need evidence that the estimate is useful for this decision.
First, give the meaning of a complaint. Decide whether the category includes a blunt question, a partly critical compliment, or a report of a broken link. Every model run applies a definition to all 40,000 comments, even if nobody chose that definition explicitly. A more capable model cannot decide what you intended to measure.
Next, examine the uncertainty in “about 22 percent.” You can compare automatic answers with a sample of the same material that a person examined. The conclusions depend on the sampling method and the rules for people who give labels. Some methods calculate error estimates from agreement among automatic labellers. Those estimates depend on assumptions about how the errors relate. A person might need to examine examples to evaluate those assumptions.
Also examine the input text. Each result describes the version of the text that the software received. Transfers between systems can damage characters. A comma can split a row. Text extraction can read a scanned digit as a letter.
These failures do not always produce an error message. The answer can change and still look reasonable. Lessons 1, 3 and 7 deliberately produce these failures so that you can examine them.
You must still decide what to measure, which material to use, and when to reject a result. These decisions give a tool its instructions. You also use these decisions to accept or reject its output.
You can buy data with labels or hire an annotation service. A model can also give labels to many kinds of text, although it can refuse some requests. These services can report agreement scores for their labels. An agreement score does not explain every limit of those labels. Calculating scores on your own text helps you understand their limits.
You do not have to do all the work by hand. Tools can help with drafts, summaries, and unfamiliar text. The checks must match the decision. Examine the definition, the comparison, and the received text. The necessary checks depend on your text and the cost of a wrong answer. This page cannot choose them for you.
The guide Using a language model in a reproducible analysis applies these checks to a constructed collection. It records a local model as a measurement instrument. It compares model labels with a random sample examined by a person. It then corrects an estimate for the errors found in that sample.
Lessons 1 through 13 cover text collection from files, web pages, and scans, then the creation of labels. Lessons 14 through 42 cover words, grammar, phrases, entities, sentences, paragraphs, and documents. These lessons explain what each transformation helps you do and what information it can remove.
Lessons 43 through 68 cover models for classification, discovery, comparison, and text generation. Each result has a baseline or a stated limit. Lessons 69 through 74 combine tasks into small systems. Examples include relation extraction, a knowledge base, question answering, dialogue, search, and document review with a recall estimate. Lessons 75 through 81 present results through an app, annotated text, charts, maps, and graphs. They explain the conclusions each display permits.
About this site
Available means a lesson source exists and its project-recorded automated checks passed. Those passed values are project assertions, not independent certification. Manual accessibility tests and final human approval remain pending. The separate status fields are in data/lesson_reviews.csv.
Methods in this field change quickly. A lesson states which claims can change over time and gives a date. The publishing rules are in CONTRIBUTING.md and RESEARCH_STANDARDS.md.
This repository builds the source and does automated checks, but does not deploy it. The canonical published location is workforcefutures.net/learn/nlp/. A successful build produces a rendered site. It does not prove that the published site is current or that pending manual checks passed.
The task names and groupings are adapted from Rob van Zoest’s Periodic Table of NLP Tasks. That page states no licence and invites reuse. The task names are standard terms in the field. This project’s grouping, wording, interface, and lessons are separate work. This project is responsible for errors in that work. If the original author prefers different terms, this site will use them.