Chars, Bytes & Graphemes

Iterate str as Unicode scalars (chars), raw bytes (bytes), or user-perceived grapheme clusters (crate).

Recipe

fn main() {
    let s = "e\u{0301}"; // e + combining acute
    println!("chars: {}", s.chars().count()); // may be 2
}

When to reach for this: Correct length limits, cursor movement, collation - not raw bytes.

Working Example

fn ascii_words(s: &str) -> impl Iterator<Item = &str> {
    s.split_whitespace()
}
 
fn byte_len(s: &str) -> usize {
    s.as_bytes().len()
}
 
fn main() {
    let text = "Hello 世界";
    println!("bytes={} chars={}", byte_len(text), text.chars().count());
    for w in ascii_words(text) { println!("{w}"); }
}

What this demonstrates:

• chars().count() vs len() bytes
• as_bytes() for protocol work
• Grapheme complexity needs unicode-segmentation for UI

Deep Dive

unicode-segmentation = "1"

use unicode_segmentation::UnicodeSegmentation;
for g in "👨‍👩‍👧".graphemes(true) { println!("{g}"); }

Gotchas

• chars == user characters - False for combining marks. Fix: Grapheme segmentation.
• bytes() for UTF-8 validation - Already valid in &str. Fix: from_utf8 on [u8].
• encode_utf16 for Windows - OS APIs sometimes UTF-16.
• char is 32-bit - Not UTF-8 encoded in char type.
• Performance chars nth - O(n).

Alternatives

FAQs

Related

• UTF-8 & Indexing
• Strings Basics
• String Manipulation
• OsString, Path & CStr

Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.

WTI5a1pXZDFhV1JsY3k1cGIzeGpaMmx2TWpFeWZESXdNall3Tnc9PQ==