candle

Q: How do I load HF weights?

Use candle-transformers model builders plus safetensors files from the model repo - match config.json architecture names.

Q: CPU vs CUDA?

Enable cuda feature on candle-core and use Device::new_cuda(0)? when hardware exists.

Q: Thread safety?

Share read-only weights with Arc; per-request state (KV cache) should not race across threads without synchronization.

Hugging Face's minimalist deep-learning framework in Rust - tensors, nn modules, and transformer inference without Python.

Recipe

use candle_core::{Device, Tensor};
use candle_nn::{linear, Linear, Module, VarBuilder};
 
fn mlp(vb: VarBuilder) -> candle_core::Result<Linear> {
    linear(768, 128, vb.pp("proj"))
}
 
fn forward(x: &Tensor, layer: &Linear) -> candle_core::Result<Tensor> {
    layer.forward(x)
}

When to reach for this:

Running HF-compatible transformer weights in pure Rust
Prototyping small models with familiar tensor APIs
Shipping inference where Python/GIL overhead is unacceptable
Teaching DL mechanics with readable Rust code

Working Example

use candle_core::{Device, Tensor};
use candle_nn::{linear, Linear, Module, VarBuilder, VarMap};
 
fn main() -> candle_core::Result<()> {
    let device = Device::Cpu;
    let varmap = VarMap::new();
    let vb = VarBuilder::from_varmap(&varmap, candle_core::DType::F32, &device);
    let layer = linear(4, 2, vb.pp("fc"))?;
    let x = Tensor::new(&[[1.0f32, 0.5, -0.2, 3.0]], &device)?;
    let y = layer.forward(&x)?;
    println!("output shape: {:?}", y.shape());
    Ok(())
}

What this demonstrates:

VarMap + VarBuilder for learnable parameters
A linear layer forward pass on CPU
Shape inspection before downstream softmax or loss

Deep Dive

How It Works

candle-core provides Tensor, Device, and autodiff (feature-gated).
candle-nn bundles layers, optimizers, and loss helpers.
candle-transformers hosts model definitions matching HF configs.
Weights load from safetensors or numpy-compatible dumps.

Crate Map

Crate	Role
`candle-core`	Tensor math, devices
`candle-nn`	Layers, training utilities
`candle-transformers`	BERT, Llama, etc.

Gotchas

API churn across minor versions - pin git tags or versions in production. Fix: lock Cargo.lock and CI against HF examples.
Wrong dtype on GPU - f64 kernels may be missing. Fix: standardize on F32 unless model requires BF16.
Shape [batch, seq, hidden] confusion - attention fails silently with broadcast bugs. Fix: assert ranks at module boundaries.
Loading full LLM on CPU RAM - multi-GB weights OOM laptops. Fix: quantized GGUF paths or GPU with enough VRAM.
Missing CUDA driver in container - runtime falls back or errors late. Fix: probe Device::cuda_if_available() at startup and log choice.

Alternatives

Alternative	Use When	Don't Use When
burn	Backend-agnostic training research	You want HF-maintained transformer ports
ort	Exported ONNX from any framework	You need training in Rust
tch (PyTorch bindings)	Must run arbitrary PyTorch graphs	You want pure Rust deps
Python HF	Largest model zoo and trainers	Single-binary edge deploy

FAQs

Can candle train large models?

Possible for small/medium models; large LLM training still usually happens in Python, then weights convert for candle inference.

How do I load HF weights?

Use candle-transformers model builders plus safetensors files from the model repo - match config.json architecture names.

CPU vs CUDA?

Enable cuda feature on candle-core and use Device::new_cuda(0)? when hardware exists.

Does candle support quantization?

GGUF and quantized checkpoints are common for LLM inference paths - follow HF candle examples for each model family.

How do I debug NaNs?

Enable gradient clipping, lower learning rate, and inspect tensor stats after each layer in training loops.

Can I export to ONNX?

Training in candle may still export via separate tooling; many teams train elsewhere and infer in candle from published weights.

How does this compare to burn?

candle is HF-centric for transformers; burn is general-purpose with pluggable backends - see burn.

Is autodiff production-ready?

Fine for research-scale training; verify numerics against PyTorch on a golden batch before trusting loss curves.

How do I batch prompts?

Pad token sequences, build attention masks, run one forward - pair with Tokenizers.

Thread safety?

Share read-only weights with Arc; per-request state (KV cache) should not race across threads without synchronization.

ML in Rust Basics - landscape
burn - alternative framework
LLM Inference & Serving - serving patterns
GPU & Acceleration - backends

Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.

Recipe

use candle_core::{Device, Tensor};
use candle_nn::{linear, Linear, Module, VarBuilder};
 
fn mlp(vb: VarBuilder) -> candle_core::Result<Linear> {
    linear(768, 128, vb.pp("proj"))
}
 
fn forward(x: &Tensor, layer: &Linear) -> candle_core::Result<Tensor> {
    layer.forward(x)
}

When to reach for this:

Running HF-compatible transformer weights in pure Rust
Prototyping small models with familiar tensor APIs
Shipping inference where Python/GIL overhead is unacceptable
Teaching DL mechanics with readable Rust code

Working Example

use candle_core::{Device, Tensor};
use candle_nn::{linear, Linear, Module, VarBuilder, VarMap};
 
fn main() -> candle_core::Result<()> {
    let device = Device::Cpu;
    let varmap = VarMap::new();
    let vb = VarBuilder::from_varmap(&varmap, candle_core::DType::F32, &device);
    let layer = linear(4, 2, vb.pp("fc"))?;
    let x = Tensor::new(&[[1.0f32, 0.5, -0.2, 3.0]], &device)?;
    let y = layer.forward(&x)?;
    println!("output shape: {:?}", y.shape());
    Ok(())
}

What this demonstrates:

VarMap + VarBuilder for learnable parameters
A linear layer forward pass on CPU
Shape inspection before downstream softmax or loss

Deep Dive

How It Works

candle-core provides Tensor, Device, and autodiff (feature-gated).
candle-nn bundles layers, optimizers, and loss helpers.
candle-transformers hosts model definitions matching HF configs.
Weights load from safetensors or numpy-compatible dumps.

Crate Map

Crate	Role
`candle-core`	Tensor math, devices
`candle-nn`	Layers, training utilities
`candle-transformers`	BERT, Llama, etc.

Gotchas

API churn across minor versions - pin git tags or versions in production. Fix: lock Cargo.lock and CI against HF examples.
Wrong dtype on GPU - f64 kernels may be missing. Fix: standardize on F32 unless model requires BF16.
Shape [batch, seq, hidden] confusion - attention fails silently with broadcast bugs. Fix: assert ranks at module boundaries.
Loading full LLM on CPU RAM - multi-GB weights OOM laptops. Fix: quantized GGUF paths or GPU with enough VRAM.
Missing CUDA driver in container - runtime falls back or errors late. Fix: probe Device::cuda_if_available() at startup and log choice.

Alternatives

Alternative	Use When	Don't Use When
burn	Backend-agnostic training research	You want HF-maintained transformer ports
ort	Exported ONNX from any framework	You need training in Rust
tch (PyTorch bindings)	Must run arbitrary PyTorch graphs	You want pure Rust deps
Python HF	Largest model zoo and trainers	Single-binary edge deploy

FAQs

Can candle train large models?

Possible for small/medium models; large LLM training still usually happens in Python, then weights convert for candle inference.

How do I load HF weights?

Use candle-transformers model builders plus safetensors files from the model repo - match config.json architecture names.

CPU vs CUDA?

Enable cuda feature on candle-core and use Device::new_cuda(0)? when hardware exists.

Does candle support quantization?

GGUF and quantized checkpoints are common for LLM inference paths - follow HF candle examples for each model family.

How do I debug NaNs?

Enable gradient clipping, lower learning rate, and inspect tensor stats after each layer in training loops.

Can I export to ONNX?

Training in candle may still export via separate tooling; many teams train elsewhere and infer in candle from published weights.

How does this compare to burn?

candle is HF-centric for transformers; burn is general-purpose with pluggable backends - see burn.

Is autodiff production-ready?

Fine for research-scale training; verify numerics against PyTorch on a golden batch before trusting loss curves.

How do I batch prompts?

Pad token sequences, build attention masks, run one forward - pair with Tokenizers.

Thread safety?

Share read-only weights with Arc; per-request state (KV cache) should not race across threads without synchronization.

ML in Rust Basics - landscape
burn - alternative framework
LLM Inference & Serving - serving patterns
GPU & Acceleration - backends

Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.

candle

Search Documentation

candle