Machine Learning in Rust: The Landscape

Machine learning in Rust is not a single library the way PyTorch is for Python. It is a loose federation of crates that agree on a shared execution model: computation expressed as tensors flowing through an explicit graph of operations, backed by a pluggable device abstraction for CPU, CUDA, Metal, or WGPU. Understanding that shared model matters more than memorizing any one crate's API, since it explains why candle, burn, ort, and the tokenizer and serving crates in this section fit together the way they do.

This page is the conceptual anchor for the rest of the ml-in-rust section. Where the Basics page shows working code for ten common tasks, this page explains the execution model underneath those tasks, and why Rust's ML tooling optimizes for different goals than Python's. Python's ecosystem is built for fast iteration on a research idea. Rust's is built for embedding a trained model where a Python runtime cannot go, serving it with predictable latency, and catching shape mistakes before they reach production.

Summary

• Rust's ML ecosystem is a set of crates that share a tensor/computation graph execution model and specialize for inference, embedding, and safety rather than research iteration speed.
• Insight: Knowing the shared model lets you predict how an unfamiliar crate will behave, and choose the right one for training versus inference versus serving without re-learning everything from its docs.
• Key Concepts: tensor, computation graph, device abstraction, eager vs. graph execution, autodiff, interchange format (ONNX, safetensors, GGUF).
• When to Use: reach for Rust ML when you need to embed a model in a single binary, serve inference at low latency without a Python runtime, or enforce compile-time shape and type guarantees around a model boundary.
• Limitations/Trade-offs: the training side of the ecosystem, meaning dataset loaders, optimizer research, and published architectures, lags Python by years, so most teams still train in Python and bring weights into Rust for the parts that need speed or safety.
• Related Topics: PyTorch and JAX execution models, ONNX interchange, embedded and edge deployment, GPU compute backends.

Foundations

A tensor is an n-dimensional array with a fixed shape and dtype, the same building block every ML framework uses, from NumPy arrays to PyTorch tensors to candle's Tensor. What differs across ecosystems is how operations on tensors assemble into a computation graph, the network of nodes (operations) and edges (tensors) describing how outputs derive from inputs. Some frameworks build that graph implicitly as code runs, a style called define-by-run or eager execution. Others consume a graph already built and frozen elsewhere, closer to define-and-run.

Rust's ML crates split cleanly along that line, and that split is the first thing to internalize about the landscape. candle and burn build graphs eagerly, call by call, much like PyTorch does, and both can differentiate through that graph for training. ort does something different: it loads a graph exported as ONNX from PyTorch, scikit-learn, or TensorFlow, and only replays it forward. There is no training story in ort, by design.

A useful analogy is the difference between a full kitchen and modular power tools bolted to a workbench. Python's ecosystem is the kitchen, one coherent environment where every appliance already knows how to talk to every other, at the cost of carrying the whole kitchen with you. Rust's ML crates are the power tools: pick a tensor library, a tokenizer, and a serving layer separately, and assemble them around the job.

This is also why Rust has no dominant framework. candle exists because Hugging Face wanted to run transformer weights without a Python runtime. burn exists because researchers wanted backend-agnostic training with compile-time tensor rank safety. ort exists because production teams needed to run graphs trained elsewhere, as fast and as safely as possible. Each crate solved a narrower problem than "be PyTorch," and the landscape is the sum of those solutions.

Mechanics & Interactions

Trace a typical request through the shared model and the design becomes concrete. Raw text or bytes get tokenized into integer IDs, those IDs become an input tensor sized [batch, sequence], and that tensor enters a graph of matmuls, activations, and normalizations built at call time (candle, burn) or replayed from a frozen ONNX graph (ort). The output tensor comes out the other side and decodes back into labels, tokens, or embeddings.

Two mechanics distinguish Rust's version of this flow from Python's. First, ownership. In Python, a tensor is a reference-counted object that any code with a reference can mutate. In Rust, a tensor's underlying buffer has a single owner (or is explicitly shared via Arc), so most operations consume their input and hand back a new tensor rather than mutating in place. This is why the idiom across candle, burn, and ndarray-based code is "call this op, get a new tensor back," a direct consequence of the borrow checker rather than a style choice.

Second, the device abstraction. Every framework here represents CPU, CUDA, Metal, and WGPU behind one Device enum or trait, and tensor operations dispatch to whichever backend the device points at. The graph itself is backend-agnostic; only the execution of each node is backend-specific. This is what lets burn train on NdArray in CI and deploy on CUDA in production without touching model code, and it is the same reason candle runs an identical forward function on a laptop CPU and a datacenter GPU.

use candle_core::{Device, Tensor};
 
// `x` owns its buffer. This function does not mutate `x` in place;
// it returns a new tensor node, so calling it twice composes two
// graph nodes rather than clobbering shared state - the same
// discipline every crate in this landscape follows.
fn scale(x: &Tensor, factor: f64) -> candle_core::Result<Tensor> {
    x.affine(factor, 0.0)
}

Autodiff is where the eager-graph crates and the replay-only crate diverge sharply. candle and burn retain (or can reconstruct) the graph backward to accumulate gradients during training, which is why they expose optimizers and training loops at all. ort never needs this, because a frozen ONNX graph has no gradients to compute; it only forward-executes. A common reasoning pitfall follows directly from this: because candle's API reads like PyTorch, it is easy to assume it carries PyTorch's training ecosystem and maturity with it. In reality the API vocabulary is familiar, but the surrounding tooling, meaning schedulers, published training recipes, and community-debugged edge cases, is far thinner.

Advanced Considerations & Applications

Shape handling is where the model-in-practice gets messier than the model-in-theory. ONNX graphs frequently bake static shapes in at export time, so a service receiving variable-length input has to pad or truncate to a fixed shape before the graph accepts it, or re-export with dynamic axes. candle and burn are more forgiving at call time since the graph rebuilds per call, but that flexibility costs a small amount of per-request graph-construction overhead a frozen ONNX graph does not pay.

Serving concurrency is a second area where Rust's story differs from Python's. A typical pattern pairs Axum with tokio, but inference itself is CPU- or GPU-bound work, not I/O, so it has to run off the async runtime's reactor threads (via spawn_blocking or a dedicated pool) to avoid starving other requests. Model weights load once at startup and share read-only across handlers through Arc, so concurrent inference calls never contend on a lock, only on the compute device itself.

Rust's memory safety is real but bounded. It covers the Rust code that builds and threads tensors through the graph, but it does not extend past the FFI boundary into a CUDA driver, a GPU kernel, or a native ONNX Runtime shared library. Crashes and undefined behavior can still originate there, and pulling in prebuilt native binaries (ort's download-binaries feature, bundled CUDA libraries) is a different supply-chain trust boundary than a pure-Rust crate graph.

The clearest sign the whole landscape is organized around "a graph gets executed" rather than "one canonical way in" is how much of the tooling is interchange formats. safetensors serializes tensor weights, ONNX serializes an entire computation graph, and GGUF serializes a quantized checkpoint for local LLM inference. Each format exists to hand a graph or its weights from wherever it was trained to wherever it needs to run, almost always Python to Rust in production settings today.

Common Misconceptions

• "Rust ML means training big models in Rust." Most production Rust ML is inference-only over weights trained in Python; native training is common for small models and research, not frontier-scale ones.
• "candle looks like PyTorch, so it has PyTorch's ecosystem." The API vocabulary is deliberately familiar, but the surrounding tooling and community-tested edge cases behind PyTorch took a decade to accumulate and are not matched yet.
• "Any model that trains in Python will export cleanly to ONNX." Op coverage gaps exist, and exotic layers can fail import, often requiring a simplified export graph or a fallback path.
• "Rust's memory safety makes the whole inference stack safe." Safety guarantees stop at the FFI boundary; GPU drivers and native runtime libraries are still a source of crashes.
• "Choosing a Rust ML crate is just a syntax preference versus Python." It changes the operational shape of the system, deployment footprint, latency profile, and whether training is even possible, not just which language you write in.
• "You need a GPU to get value from Rust ML." A large share of real Rust ML usage is CPU-bound inference and edge deployment, an area where Python's per-request overhead is comparatively weak.

FAQs

Related

• ML in Rust Basics - hands-on walkthrough that puts this model into practice across ten examples.
• candle - the eager, HF-centric tensor and transformer framework.
• burn - the backend-agnostic training framework with compile-time tensor ranks.
• ONNX Runtime (ort) - the replay-only path for graphs trained elsewhere.
• LLM Inference & Serving - how the execution model shows up in a serving architecture.

Stack versions: This page is conceptual and not tied to a specific stack version.