Strings, UTF-8, and Text in Rust

Text handling trips up more newcomers to Rust than almost any other part of the language, not because the API is unusually large, but because Rust refuses to let you pretend text is simpler than it actually is. Two types (String and &str), one storage invariant (valid UTF-8, always), and one blunt refusal (no s[i] character indexing) are the whole story - and each of those three decisions exists to prevent a specific, real bug class that other languages quietly allow.

This page is the conceptual anchor for the strings-and-text section. Strings Basics shows the day-to-day API; this page explains why the API is shaped the way it is - why ownership splits String from &str the same way it splits every other Rust collection, what the UTF-8 guarantee actually buys you, and why byte-indexing panics instead of silently returning something wrong.

Summary

• String is an owned, growable, heap-allocated buffer of bytes guaranteed to be valid UTF-8; &str is a borrowed view into such a buffer, with the same guarantee.
• Insight: Text is variable-width in UTF-8 - a "character" can be 1 to 4 bytes - so any API that let you index by character position in O(1) would either be lying about performance or lying about correctness; Rust chooses to be honest instead.
• Key Concepts: UTF-8 encoding, owned vs. borrowed text, char boundary, Deref coercion (String -> &str), byte length vs. character count.
• When to Use: &str for function parameters and read-only views (maximum flexibility, no allocation); String when you need to own, build, or mutate text over time.
• Limitations/Trade-offs: No O(1) character indexing is possible while keeping the UTF-8 guarantee; correctly handling user-perceived characters (emoji, combining marks) needs a crate beyond the standard library.
• Related Topics: ownership and borrowing, slices, collections and iterators, Cow for conditional ownership.

Foundations

String and &str split the same way Vec<T> and [T] (a slice) do, and for the same reason. String owns a heap-allocated, growable buffer - you can push to it, own it, move it, and it is dropped when it goes out of scope. &str is a borrowed view into a sequence of UTF-8 bytes, which might live inside a String's buffer, inside the compiled binary (a &'static str literal), or inside any other byte source that upholds the UTF-8 guarantee. Every String can produce a &str view of itself essentially for free (through Deref), but not every &str has a String behind it - a string literal "hello" is &'static str pointing directly into the program's read-only data, with no owning String anywhere.

This split is why idiomatic function signatures take &str rather than String or &String: a &str parameter accepts a borrow of an owned String, a string literal, or a slice of either, uniformly, without forcing the caller to allocate just to call your function. String as a parameter type is a signal that the function specifically needs to own the text (to store it, extend it, or hand it off elsewhere).

The second foundational fact is the UTF-8 guarantee itself: Rust's compiler and standard library treat "this is a str" and "this is valid UTF-8" as the same statement. There is no way to construct a &str or String containing invalid UTF-8 through safe code - String::from_utf8 returns a Result specifically because the input bytes might not be valid, and constructing one from bytes you haven't validated requires an unsafe function (from_utf8_unchecked) that puts the burden of proof on you.

Mechanics & Interactions

UTF-8 encodes each Unicode scalar value ("character," loosely) as one to four bytes, with the byte count determined by the character's code point. ASCII characters take exactly one byte and are byte-for-byte identical to plain ASCII text; most non-Latin scripts and emoji take two to four bytes. This variability is the entire reason str cannot be indexed by character position the way an array can:

  "Hello"   -> 5 bytes, 5 chars  (all ASCII, 1 byte each)
  "Hello 🦀" -> 10 bytes, 7 chars (🦀 alone is 4 bytes)

An array index like arr[3] is fast specifically because element size is fixed, so the address of element n is a simple multiplication. Character n in a UTF-8 string has no such fixed offset - finding it requires walking from the start (or a known boundary) and decoding each character's byte width along the way, which is an O(n) operation, not O(1). Rather than offer a method named .char_at(n) that quietly costs O(n) and lure people into calling it in a loop (which silently becomes O(n^2)), Rust simply does not implement Index<usize> for str at all. s[3] is a compile error, not a runtime one - the honesty is enforced as early as possible.

What str does support is indexing by byte range (&s[0..3]), which is O(1) exactly the way you'd expect slicing to be, but comes with an obligation: both endpoints of the range must fall on a char boundary - the start of an encoded character, not the middle of one. Slicing into the middle of a multi-byte character panics at runtime, because the result would be a str containing a truncated, invalid UTF-8 sequence, which the type's own invariant forbids ever existing.

fn safe_prefix(s: &str, max_bytes: usize) -> &str {
    if s.len() <= max_bytes { return s; }
    let mut end = max_bytes;
    // Walk backward from the requested byte offset until we land on a
    // real character boundary - `str` slicing panics otherwise, because
    // a half-encoded UTF-8 sequence would violate the type's guarantee.
    while !s.is_char_boundary(end) { end -= 1; }
    &s[..end]
}

This is also why len() on a str returns the byte length, not a character count: byte length is the only measurement that is genuinely O(1), since it's just the buffer's size. Getting a character count requires s.chars().count(), which walks and decodes the whole string - an O(n) operation with a different cost profile that the API makes visible by requiring an explicit method call rather than hiding it behind .len().

Advanced Considerations & Applications

Even "character count" is a slippery target, because Unicode has three commonly confused notions of "how long is this text": bytes (len()), Unicode scalar values (chars().count()), and user-perceived characters, or grapheme clusters (what a person would call "one letter on screen"). A single grapheme like an emoji with a skin-tone modifier, or a Latin letter plus a combining accent mark, can be multiple chars and several more bytes, all while a human reads it as one character. The standard library stops at char (a Unicode scalar value) deliberately; grapheme segmentation is locale- and script-aware in ways that would bloat std, so it lives in the unicode-segmentation crate instead.

Truncating text safely for display is a concrete case where all of this matters at once: truncating by byte count risks slicing mid-character (a panic, or corrupted output with unsafe slicing); truncating by chars().count() can still cut a grapheme cluster in half (splitting an emoji sequence or a combining accent from its base letter); only grapheme-aware truncation matches what a user would consider a "whole character" removed cleanly. Protocol and file-format code, by contrast, almost always wants byte length - it needs to match what's actually on the wire, not what a human reader would count.

Interop with the operating system is the other place the UTF-8 guarantee visibly ends: file paths and OS-level strings are not guaranteed to be valid UTF-8 on every platform (Windows paths are UTF-16-ish, Unix paths are arbitrary bytes), which is exactly why std::ffi::OsString and std::path::Path exist as separate types from String - they intentionally do not carry the UTF-8 invariant, because it isn't a guarantee the OS actually provides.

Common Misconceptions

• "String and &str are basically the same type, one just has a &" - They differ the same way Vec<T> and [T] do: one owns a growable buffer, the other borrows a view; understanding them through that lens (not as "mutable vs. immutable string") clarifies most confusion about which to use where.
• "s.len() tells you how many characters are in the string" - It tells you the byte length of the UTF-8 encoding, which equals the character count only for pure-ASCII text; anything outside ASCII needs chars().count(), and even that isn't the user-perceived character count.
• "s[i] should just work like it does for arrays" - Array indexing is fast because every element is the same size; UTF-8 characters are 1-4 bytes each, so there is no O(1) way to find "the nth character," and Rust refuses to offer an API that pretends otherwise.
• "A String and the &str slices taken from it are separate copies" - A &str borrowed from a String points directly into that String's existing heap buffer; no copy happens unless you explicitly call .to_string() or .to_owned().
• "Any sequence of bytes can be treated as a string if you just cast it" - String/str guarantee valid UTF-8 as an invariant enforced by the type system; turning arbitrary bytes into one either validates them (String::from_utf8, which can fail) or requires unsafe code asserting a guarantee you must actually uphold.

FAQs

Related

• Strings Basics - hands-on String/&str syntax, push_str, and format!.
• UTF-8 & Indexing - safe slicing, char_indices, and is_char_boundary in practice.
• Chars, Bytes & Graphemes - the three ways to measure text length, including grapheme clusters.
• String Manipulation - splitting, trimming, replacing, and building strings.
• OsString, Path & CStr - text types that deliberately don't carry the UTF-8 guarantee.

Stack versions: This page is conceptual and not tied to a specific stack version.