Tokenizers

The Hugging Face tokenizers crate in Rust - fast BPE/WordPiece pipelines for LLM and classifier preprocessing.

Recipe

use tokenizers::Tokenizer;
 
fn encode_line(tok: &Tokenizer, text: &str) -> tokenizers::Result<Vec<u32>> {
    Ok(tok.encode(text, true)?.get_ids().to_vec())
}

When to reach for this:

• Every NLP inference path needs IDs before tensors
• Sharing tokenizer.json between Python training and Rust serving
• Batching prompts with padding for GPU inference
• Building RAG pipelines that chunk documents consistently

Working Example

use tokenizers::Tokenizer;
use tokenizers::PaddingParams;
use tokenizers::TruncationParams;
 
fn main() -> tokenizers::Result<()> {
    let mut tok = Tokenizer::from_file("tokenizer.json")?;
    tok.with_padding(Some(PaddingParams::default()));
    tok.with_truncation(Some(TruncationParams::default()))?;
 
    let batch = tok.encode_batch(
        vec!["Rust ML is fast", "Tokenizers matter"],
        true,
    )?;
 
    for enc in &batch {
        println!("len {} ids {:?}", enc.get_ids().len(), enc.get_ids());
    }
    Ok(())
}

What this demonstrates:

• Loading HF-compatible tokenizer JSON
• Configuring padding and truncation policies
• Batch encoding for variable-length strings

Deep Dive

How It Works

• Tokenizer merges rules (BPE) or vocab map text to IDs offline in Rust.
• encode returns Encoding with ids, tokens, attention mask, offsets.
• Padding aligns batch tensors; truncation enforces max_length from model config.
• Special tokens ([CLS], <s>, etc.) insert when add_special_tokens is true.

Key APIs

Gotchas

• Wrong tokenizer.json for weights - garbage generations. Fix: download tokenizer from the exact model revision.
• Truncation side wrong - left vs right truncate changes meaning for QA models. Fix: match HF tokenizer.model_max_length policy.
• No padding in batch - cannot stack tensors. Fix: set PaddingParams with pad_id from vocab.
• Unicode normalization drift - NFC vs NFKC changes token splits vs Python. Fix: normalize input strings in both training and serving.
• Rebuilding tokenizer per request - slow file IO. Fix: Arc<Tokenizer> loaded once at startup.

Alternatives

FAQs

Related

• LLM Inference & Serving - generation loop
• candle - transformer forward
• Embeddings & Vector Search - document chunking
• ML in Rust Basics - tokenize example

Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.