ONNX Runtime (ort)

Q: How do I enable CUDA?

Build ort with CUDA features and register CUDA execution provider in Session::builder() - driver must exist in container.

Q: Can I run multiple models?

Yes - one Session per model, share across threads with Arc if ORT thread safety docs allow for your EP.

Q: Model opset version?

Match ort supported opset; re-export from training framework if import fails.

Run exported ONNX models in Rust with ONNX Runtime - the standard path from Python training to native inference.

Recipe

use ndarray::Array;
use ort::{inputs, session::Session, value::Tensor};
 
fn predict(session: &Session, flat: Vec<f32>) -> ort::Result<f32> {
    let input = Array::from_shape_vec((1, flat.len()), flat)?;
    let outputs = session.run(inputs![Tensor::from_array(input)?])?;
    let view = outputs[0].try_extract_tensor::<f32>()?;
    Ok(view[0])
}

When to reach for this:

Deploying sklearn/PyTorch/TensorFlow models without Python
Standardizing inference across languages in one ONNX file
Needing CUDA/TensorRT execution providers in production
Teams already exporting ONNX in training pipelines

Working Example

use ndarray::Array;
use ort::{inputs, session::Session, value::Tensor};
 
fn main() -> ort::Result<()> {
    let session = Session::builder()?.commit_from_file("model.onnx")?;
    let features = vec![0.1f32, 0.2, 0.3, 0.4];
    let input = Array::from_shape_vec((1, 4), features)?;
    let outputs = session.run(inputs![Tensor::from_array(input)?])?;
    let tensor = outputs[0].try_extract_tensor::<f32>()?;
    println!("score: {}", tensor[0]);
    Ok(())
}

What this demonstrates:

Building a session from disk once at startup
Creating a batch-1 input tensor with ndarray shapes
Extracting f32 outputs by index

Deep Dive

How It Works

ONNX graph is frozen operators with typed inputs/outputs.
Session::run maps Rust tensors to ORT memory, executes providers, returns outputs.
Execution providers (CPU, CUDA, TensorRT) plug in via session options.
Dynamic batch or sequence axes require explicit dimensions each run.

Export Checklist

Step	Action
Train	PyTorch/sklearn/etc.
Export	`torch.onnx.export` or framework tool
Validate	`onnx.checker` + sample inference in Rust

Gotchas

Input name mismatch - graph expects input_ids not input. Fix: print session.inputs and match strings.
NCHW vs NHWC - vision models flip channel order. Fix: document layout in model card; transpose in preprocess.
Preprocessing drift - different normalization than training. Fix: golden vector test from Python export notebook.
FP16 on CPU EP - unsupported or slow paths. Fix: keep FP32 for CPU-only deploys.
Huge models loaded per request - multi-second latency. Fix: Arc<Session> singleton at process start.

Alternatives

Alternative	Use When	Don't Use When
candle	Native Rust transformer without ONNX	Model only exists as ONNX today
burn import	Stay in burn ecosystem	Graph uses unsupported ONNX ops
Python microservice	Rapid iteration	Latency/ops cost of sidecar
TensorRT directly	NVIDIA-only max perf	Need portable CPU fallback

FAQs

How do I enable CUDA?

Build ort with CUDA features and register CUDA execution provider in Session::builder() - driver must exist in container.

Can I run multiple models?

Yes - one Session per model, share across threads with Arc if ORT thread safety docs allow for your EP.

Dynamic batch size?

Set batch dimension in input tensor shape at runtime if export used dynamic axis annotations.

String inputs?

Most ONNX classifiers expect numeric tensors - tokenize text before ORT, not inside generic ORT APIs.

How do I debug slow inference?

ORT profiling flags, smaller graph via onnx-simplifier, and GPU EP when batch size warrants it.

Model opset version?

Match ort supported opset; re-export from training framework if import fails.

Batching requests?

Stack rows in batch dimension when graph supports dynamic batch; pad to max length for sequences.

Security of model files?

Treat ONNX like code - verify checksums and sign artifacts in object storage.

How to test?

Check in tiny ONNX fixture and assert output tensor within epsilon of Python reference.

Serve with Axum?

See Serving Models with Axum for HTTP wiring.

ML in Rust Basics - ONNX intro
Tokenizers - text preprocessing
Serving Models with Axum - HTTP layer
GPU & Acceleration - CUDA EP

Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.

Recipe

use ndarray::Array;
use ort::{inputs, session::Session, value::Tensor};
 
fn predict(session: &Session, flat: Vec<f32>) -> ort::Result<f32> {
    let input = Array::from_shape_vec((1, flat.len()), flat)?;
    let outputs = session.run(inputs![Tensor::from_array(input)?])?;
    let view = outputs[0].try_extract_tensor::<f32>()?;
    Ok(view[0])
}

When to reach for this:

Deploying sklearn/PyTorch/TensorFlow models without Python
Standardizing inference across languages in one ONNX file
Needing CUDA/TensorRT execution providers in production
Teams already exporting ONNX in training pipelines

Working Example

use ndarray::Array;
use ort::{inputs, session::Session, value::Tensor};
 
fn main() -> ort::Result<()> {
    let session = Session::builder()?.commit_from_file("model.onnx")?;
    let features = vec![0.1f32, 0.2, 0.3, 0.4];
    let input = Array::from_shape_vec((1, 4), features)?;
    let outputs = session.run(inputs![Tensor::from_array(input)?])?;
    let tensor = outputs[0].try_extract_tensor::<f32>()?;
    println!("score: {}", tensor[0]);
    Ok(())
}

What this demonstrates:

Building a session from disk once at startup
Creating a batch-1 input tensor with ndarray shapes
Extracting f32 outputs by index

Deep Dive

How It Works

ONNX graph is frozen operators with typed inputs/outputs.
Session::run maps Rust tensors to ORT memory, executes providers, returns outputs.
Execution providers (CPU, CUDA, TensorRT) plug in via session options.
Dynamic batch or sequence axes require explicit dimensions each run.

Export Checklist

Step	Action
Train	PyTorch/sklearn/etc.
Export	`torch.onnx.export` or framework tool
Validate	`onnx.checker` + sample inference in Rust

Gotchas

Input name mismatch - graph expects input_ids not input. Fix: print session.inputs and match strings.
NCHW vs NHWC - vision models flip channel order. Fix: document layout in model card; transpose in preprocess.
Preprocessing drift - different normalization than training. Fix: golden vector test from Python export notebook.
FP16 on CPU EP - unsupported or slow paths. Fix: keep FP32 for CPU-only deploys.
Huge models loaded per request - multi-second latency. Fix: Arc<Session> singleton at process start.

Alternatives

Alternative	Use When	Don't Use When
candle	Native Rust transformer without ONNX	Model only exists as ONNX today
burn import	Stay in burn ecosystem	Graph uses unsupported ONNX ops
Python microservice	Rapid iteration	Latency/ops cost of sidecar
TensorRT directly	NVIDIA-only max perf	Need portable CPU fallback

FAQs

How do I enable CUDA?

Build ort with CUDA features and register CUDA execution provider in Session::builder() - driver must exist in container.

Can I run multiple models?

Yes - one Session per model, share across threads with Arc if ORT thread safety docs allow for your EP.

Dynamic batch size?

Set batch dimension in input tensor shape at runtime if export used dynamic axis annotations.

String inputs?

Most ONNX classifiers expect numeric tensors - tokenize text before ORT, not inside generic ORT APIs.

How do I debug slow inference?

ORT profiling flags, smaller graph via onnx-simplifier, and GPU EP when batch size warrants it.

Model opset version?

Match ort supported opset; re-export from training framework if import fails.

Batching requests?

Stack rows in batch dimension when graph supports dynamic batch; pad to max length for sequences.

Security of model files?

Treat ONNX like code - verify checksums and sign artifacts in object storage.

How to test?

Check in tiny ONNX fixture and assert output tensor within epsilon of Python reference.

Serve with Axum?

See Serving Models with Axum for HTTP wiring.

ML in Rust Basics - ONNX intro
Tokenizers - text preprocessing
Serving Models with Axum - HTTP layer
GPU & Acceleration - CUDA EP

Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.

ONNX Runtime (ort)

Search Documentation

ONNX Runtime (ort)