How computers store text

Bits, bytes, Unicode, and UTF-8 in R

source data loading
encoding
Unicode
Learn why text sometimes becomes garbled and how to protect it when working in R.

A community class attendance file contains Anaïs where AnaΓ―s should be. A person can infer the intended name. A search, badge printer, or duplicate check may treat the damaged name as a different name. The error is easy to miss until the damaged name reaches a badge or a lookup fails.

One possible cause is a wrong character encoding, the rules that connect stored byte values with written text. We will recreate that failure from known bytes and decode the bytes correctly. Then we will compare text values that look identical but fail an exact comparison.

Note

This lesson uses a constructed attendance file and incident. The encoding failure is real, but the story does not describe an actual person or class.

TipWhat you will learn

By the end of this lesson, you will be able to:

  • distinguish a visible text unit, a code point, and a byte.
  • explain the different functions of Unicode and UTF-8.
  • convert text to UTF-8 in R.
  • recognize bytes that no UTF-8 decoder can read, and a byte-order mark.
  • examine text that looks the same but behaves differently.

What happens when a computer saves a word

A check of AnaΓ―s’s name begins with a precise account of what the computer saved. People see letters, punctuation, and emoji. A computer stores numbers, so saving text takes a few steps:

  1. Unicode assigns numbers called code points to abstract text elements.
  2. UTF-8 encodes Unicode scalar values as one or more bytes.
  3. Software saves those bytes in a file or sends them across a network.
  4. Software uses the same encoding to decode the bytes as text.

What a reader sees as one character is often called a grapheme cluster. A grapheme cluster can contain one code point or several. The difference becomes visible later when we compare two ways to store Γ©.

Consider two characters:

Unicode code points and UTF-8 bytes for two characters
Character Unicode code point Bytes used by UTF-8
A U+0041 41
Γ© U+00E9 C3 A9

The table uses hexadecimal, a compact way to write numbers. There is no need to memorize them. For now, notice that A needs one UTF-8 byte while Γ© needs two.

Bits and bytes

A bit records one binary choice, 0 or 1. Eight bits make one byte, which has 256 possible combinations numbered from 0 through 255.

NoteWhy eight bits make a byte

A byte did not always have a fixed size. Early computers grouped bits according to the characters and numbers each machine needed. A byte could therefore mean different things on different systems.

Two standards helped settle the size:

  1. ASCII defined 128 text and control values, which fit in seven bits. RFC 20 described early network interchange in 1969. It placed each seven-bit ASCII value inside an eight-bit byte and set the extra high bit to zero.
  2. IBM’s System/360 family, announced in 1964, made an eight-bit byte the basic unit of its storage formats. Its compatible machines became widely used. Software and hardware for those machines used the same byte size.

Eight bits provide \(2^8 = 256\) patterns: enough for every ASCII value with one unused bit. Today, a byte almost always means eight bits. Technical standards sometimes use octet when they need to exclude the older, variable-size meaning.

We will examine A and Γ©, two characters with short byte sequences. R’s charToRaw() function shows bytes. The word β€œraw” means that R should display the stored values without interpreting them as letters. map() from purrr applies that byte check to each text value and returns a list. The list can contain byte results of different lengths.

library(tibble)
library(purrr)
library(stringi)

text <- c(
  letter_A = "A",
  e_with_accent = "\u00E9"
)

bytes <- map(enc2utf8(text), charToRaw)

bytes
$letter_A
[1] 41

$e_with_accent
[1] c3 a9

R prints raw bytes as two-digit hexadecimal values. The output shows 41 for A and c3 a9 for Γ©. Hexadecimal digits may appear in upper- or lowercase without changing the values. The bytes c3 a9 become Γ© only when software reads them using UTF-8.

Recreate the garbled name

We know that the source intended to store AnaΓ―s as UTF-8. The next example keeps one byte sequence and asks R to interpret it two ways.

expected_name <- "Ana\u00EFs"
name_bytes <- charToRaw(enc2utf8(expected_name))

misread_name <- rawToChar(name_bytes)
Encoding(misread_name) <- "latin1"
garbled_name <- enc2utf8(misread_name)

utf8_name <- rawToChar(name_bytes)
Encoding(utf8_name) <- "UTF-8"
decoded_name <- enc2utf8(utf8_name)

c(
  read_as_latin1 = garbled_name,
  read_as_utf8 = decoded_name
)
read_as_latin1   read_as_utf8 
      "Anaïs"        "AnaΓ―s"

Both results came from the same bytes. Reading the bytes as Latin-1 produced Anaïs. Reading them as UTF-8 produced AnaΓ―s. The correct choice comes from the source’s encoding declaration or other trustworthy metadata, not from guessing based on appearance.

Unicode and UTF-8 have different functions

Unicode and UTF-8 answer different questions. Unicode is a shared catalog that assigns code points to text from many writing systems. The catalog also includes punctuation, symbols, and emoji.

UTF-8 describes how to store Unicode scalar values as bytes. It uses:

  • one byte for common ASCII characters.
  • two bytes for many accented letters.
  • three bytes for many other characters.
  • four bytes for characters such as many emoji.

UTF-16 and UTF-32 encode the same Unicode code points in other ways. The reader of a file still needs to know which encoding the writer used.

NoteWhere ASCII and Latin-1 fit

Incompatible machines needed a common text format. ASCII supplied it. The standard uses seven bits for 128 values covering English letters, digits, punctuation, and control instructions such as a line break. UTF-8 deliberately keeps those same values, so an ASCII file is also valid UTF-8.

Latin-1 came later and uses all eight bits, giving it 256 values. It adds characters used by several Western European languages, but it still cannot represent most of the world’s writing systems. Its byte values above 127 do not mean the same thing in UTF-8. Confusing Latin-1 with UTF-8 is a common cause of garbled text.

Unicode assigns a common number to each text element. Encodings such as UTF-8 specify how those numbers become bytes. That separation is why one system can cover far more than any single eight-bit code page.

What R knows about a string

A string is one text value. R may label a string as Latin-1, UTF-8, or a sequence of bytes. Encoding() reports that label, so we can examine how R will interpret the text it received. We will display the answer in a table.

strings <- c(
  plain_letter = "A",
  accented_letter = "\u00E9"
)

declared_encoding <- Encoding(strings)
How R labels two character strings
Text What R reports
A unknown
Γ© UTF-8

R reports the plain letter as "unknown". The letter is still valid. ASCII text uses the same byte values in the encodings R supports, so R does not need a special label here.

An encoding label describes how R should interpret bytes. It does not convert them. Assigning a different label can leave every byte untouched. When the bytes need to change, use a conversion function.

Transcode known Latin-1 text

Some files correctly use Latin-1 but need conversion for a UTF-8 workflow. This operation is called transcoding. The next example creates "cafΓ©" from known Latin-1 bytes, then uses iconv() to convert them.

latin1_text <- rawToChar(as.raw(c(0x63, 0x61, 0x66, 0xE9)))
Encoding(latin1_text) <- "latin1"

utf8_text <- iconv(
  latin1_text,
  from = "latin1",
  to = "UTF-8",
  sub = NA_character_
)

result <- list(
  text = utf8_text,
  latin1_bytes = charToRaw(latin1_text),
  utf8_bytes = charToRaw(utf8_text)
)

result
$text
[1] "cafΓ©"

$latin1_bytes
[1] 63 61 66 e9

$utf8_bytes
[1] 63 61 66 c3 a9

The converted text still reads "cafΓ©", but its final byte changes from e9 to the two-byte UTF-8 sequence c3 a9. sub = NA_character_ tells R to return a missing value rather than replace a byte without reporting the failure. That visible missing value gives us a chance to investigate the source.

Bytes that are not well-formed UTF-8

The attendance file came from a system that no longer gave the correct encoding. Part of the file is not valid UTF-8. UTF-8 has a strict structure. A byte such as c3 requires a continuation byte. When the required byte is missing, the sequence is malformed. A UTF-8 decoder must reject or replace that sequence.

Another encoding, such as Latin-1, can assign characters to the same bytes. The bytes alone cannot show the interpretation that the author intended.

The example below builds Hi followed by c3 28. The byte 28 is an opening parenthesis, not a continuation byte, so the pair is malformed.

damaged_bytes <- as.raw(c(0x48, 0x69, 0x20, 0xC3, 0x28))
damaged_text <- rawToChar(damaged_bytes)
Encoding(damaged_text) <- "UTF-8"

prefix_is_valid <- map_lgl(
  seq_along(damaged_bytes),
  \(position) validUTF8(rawToChar(damaged_bytes[seq_len(position)]))
)
first_broken_byte <- which(!prefix_is_valid)[1]

strict_result <- iconv(
  damaged_text,
  from = "UTF-8",
  to = "UTF-8",
  sub = NA_character_
)
marked_result <- iconv(
  damaged_text,
  from = "UTF-8",
  to = "UTF-8",
  sub = "\uFFFD"
)
One malformed byte sequence read three ways
Step Result
Bytes received 48 69 20 c3 28
First byte where the prefix stops being valid 4
Strict decode with sub = NA_character_ NA
Decode with a replacement character Hi οΏ½(

The functions report three different results. validUTF8() answers yes or no. Strict conversion returns NA, which is visible and easy to count. Replacement produces U+FFFD, the replacement character. The replacement keeps the row readable but removes evidence of the original bytes.

Position four is where the prefix stops being decodable. Record that position. Report the file, the offset, and the raw bytes rather than the repaired string. A byte offset identifies the relevant source position. The repaired string Hi \uFFFD( does not.

Do not overwrite the source with repaired text. Keep the original bytes. Record the substitution rule. Count the values that needed substitution.

A byte-order mark and a file with two encodings

The same class list has two more problems. Some tools begin a UTF-8 file with the three bytes ef bb bf, a byte-order mark, usually shortened to BOM. It contains no text. If retained, the mark becomes an invisible prefix on the first field. The prefixed name no longer matches an unprefixed name.

The file also mixes encodings because two systems appended rows. Mixed encodings are common in exported data. One encoding rule cannot decode the whole file.

byte_order_mark <- as.raw(c(0xEF, 0xBB, 0xBF))
file_rows <- list(
  header = c(byte_order_mark, charToRaw("name")),
  row_from_utf8_system = charToRaw(enc2utf8("caf\u00E9")),
  row_from_latin1_system = as.raw(c(0x63, 0x61, 0x66, 0xE9))
)
declared_source_encoding <- c("UTF-8", "UTF-8", "latin1")

drop_byte_order_mark <- function(bytes) {
  if (
    length(bytes) >= 3L &&
      identical(bytes[1:3], byte_order_mark)
  ) {
    bytes[-(1:3)]
  } else {
    bytes
  }
}

decode_row <- function(bytes, declared) {
  text <- rawToChar(drop_byte_order_mark(bytes))
  Encoding(text) <- declared
  enc2utf8(text)
}

row_report <- tibble(
  row = names(file_rows),
  stored_bytes = unname(map_chr(
    file_rows,
    \(bytes) paste(as.character(bytes), collapse = " ")
  )),
  starts_with_bom = unname(map_lgl(
    file_rows,
    \(bytes) {
      length(bytes) >= 3L &&
        identical(bytes[1:3], byte_order_mark)
    }
  )),
  valid_as_utf8 = unname(map_lgl(
    file_rows,
    \(bytes) validUTF8(rawToChar(drop_byte_order_mark(bytes)))
  )),
  declared_by_source = declared_source_encoding,
  decoded = unname(map2_chr(
    file_rows,
    declared_source_encoding,
    decode_row
  ))
)
Three rows with two encodings and one byte-order mark
Row Stored bytes Starts with BOM Valid as UTF-8 Declared by source Decoded text
header ef bb bf 6e 61 6d 65 TRUE TRUE UTF-8 name
row_from_utf8_system 63 61 66 c3 a9 FALSE TRUE UTF-8 cafΓ©
row_from_latin1_system 63 61 66 e9 FALSE FALSE latin1 cafΓ©

The last two rows display the same word from different bytes. The provenance column, declared_by_source, records the source of each decoding rule. This metadata comes from the sending system rather than a guess based on appearance. Decoding the whole file as UTF-8 would lose the third row because its bytes are not valid UTF-8.

Detecting an encoding by inspection is possible and sometimes necessary, but a detector returns a ranked guess. Keep the guess, its confidence, and the byte evidence beside the decoded text. A later reader can then assess the guess.

What current R does

This site builds and runs checks with R 4.6.1. That version built this page but may not remain the newest release. CRAN’s R for Windows page lists the current installer.

Current 64-bit R builds use UTF-8 as their native encoding on Windows 10 version 1903 and later. This also applies to Windows Server 2022 and later. The native encoding matches the usual setup on macOS and Linux. This removed many older Windows limitations around file names, comments, symbols, and text passed to external libraries.

R 4.5 made package metadata files such as DESCRIPTION and PACKAGES UTF-8. It added readLines(encoding = "bytes") for examining a file without creating invalid strings. It also made iconv(from = "") prefer a string’s declared encoding.

Those improvements make modern R safer. They do not show an outside file’s encoding. Older software may still produce Windows-1252, Latin-1, or another encoding, and one file may still contain rows from several systems.

We can ask R about the current session:

encoding_info <- l10n_info()

encoding_info[c("UTF-8", "Latin-1")]
$`UTF-8`
[1] TRUE

$`Latin-1`
[1] FALSE

A value of TRUE next to UTF-8 means that the current R session uses UTF-8 as its native encoding. This describes the session. A file received from somewhere else may use different rules.

Why identical-looking text may not match

Correcting the encoding fixes the obvious damage, but an exact comparison can still fail. Unicode can represent some visible characters in more than one way. The accented letter Γ© can be:

  • one code point for the complete letter.
  • the letter e followed by a separate accent mark.

Both forms look the same on screen. A duplicate check may still treat them as different sequences.

same_appearance <- c(
  one_code_point = "\u00E9",
  letter_plus_accent = "e\u0301"
)

counts <- nchar(same_appearance, type = "chars")
Two ways to represent the same visible character
Form Displayed text Code point count
one code point Γ© 1
letter plus accent é 2

An exact comparison reports different sequences until normalization:

exact_match_before <- identical(
  unname(same_appearance[1]),
  unname(same_appearance[2])
)
exact_match_after <- identical(
  unname(stri_trans_nfc(same_appearance[1])),
  unname(stri_trans_nfc(same_appearance[2]))
)

c(
  exact_match_before_normalizing = exact_match_before,
  exact_match_after_normalizing = exact_match_after
)
exact_match_before_normalizing  exact_match_after_normalizing 
                         FALSE                           TRUE

Normalization changes equivalent Unicode sequences into a consistent form. The example uses Normalization Form C, usually shortened to NFC. Normalize a working copy when equivalent forms should compare as equal. Keep the source text when its exact representation matters.

A safe routine for text files

Source metadata that identifies UTF-8 lets us decode the attendance file’s bytes without guessing. When text comes from a file, website, or another system:

  1. Find the encoding that the source declares.
  2. Read the text using that encoding.
  3. Remove a byte-order mark before comparing or joining on the first field.
  4. Convert the working copy to UTF-8.
  5. Count the values that failed to decode. Keep the byte offsets of those values.
  6. Examine accented letters, non-Latin scripts, and emoji.
  7. Normalize only when the task requires equivalent forms to match.
  8. Keep conversion warnings visible.

Avoid useBytes = TRUE for normal text work. It tells R to work one byte at a time and can split a multi-byte character.

What to remember

  • A visible grapheme cluster can contain one Unicode code point or several.
  • UTF-8 stores Unicode scalar values as bytes.
  • Reading bytes with the wrong encoding is one common cause of garbled text, but appearance alone does not prove the cause.
  • Some byte sequences are invalid in every declared encoding. Decide whether to use NA or a replacement character. Record the choice.
  • A byte-order mark is invisible and still breaks an exact match.
  • One file can contain rows written under two encodings.
  • An encoding label does not convert text.
  • UTF-8 is a practical working format for text from many languages.
  • Text that looks identical may need Unicode normalization before comparison.

These checks preserve AnaΓ―s when software displays the file. They also let equivalent spellings match when the task requires it. The person who entered a name should receive the same name in the output.

Sources