Data Basics

9 examples to get you started with high-performance data in Rust - 7 basic and 2 intermediate.

Prerequisites

cargo new data-playground && cd data-playground
cargo add polars --features lazy,csv,parquet
cargo add arrow
cargo add clap --features derive

Tooling: Examples target Rust 1.97.0 (edition 2024) and Polars 0.46+.

Basic Examples

1. Read a CSV with Polars (eager)

Load a small file and inspect schema before any transforms.

use polars::prelude::*;
 
fn main() -> PolarsResult<()> {
    let df = CsvReadOptions::default()
        .with_has_header(true)
        .try_into_reader_with_file_path(Some("sales.csv".into()))?
        .finish()?;
    println!("{}", df.head(Some(3)));
    Ok(())
}

CsvReadOptions centralizes delimiter, dtypes, and null handling.
Eager mode materializes the full frame - fine for files that fit in RAM.
Set explicit dtypes early to avoid silent string inference on numeric columns.

Related: CSV/Parquet I/O - format-specific tuning

2. Lazy scan and filter before collect

Push predicates down so Polars reads less data from disk.

use polars::prelude::*;
 
fn main() -> PolarsResult<()> {
    let q = LazyCsvReader::new("events.csv".into())
        .with_has_header(true)
        .finish()?
        .filter(col("amount").gt(lit(100)))
        .select([col("user_id"), col("amount")]);
    let df = q.collect()?;
    println!("rows: {}", df.height());
    Ok(())
}

Lazy plans optimize before execution - filters and column picks fuse together.
collect() is the boundary where memory is allocated for the result.
Prefer lazy pipelines when the source is larger than available RAM.

Related: Polars - lazy vs eager trade-offs

3. Group-by aggregation in one expression

Summarize metrics without manual hash maps.

use polars::prelude::*;
 
fn revenue_by_region(df: LazyFrame) -> PolarsResult<LazyFrame> {
    Ok(df
        .group_by([col("region")])
        .agg([col("revenue").sum().alias("total_revenue")])
        .sort(["total_revenue"], SortMultipleOptions::default()))
}

group_by + agg maps cleanly to SQL GROUP BY.
Sorting in the lazy plan keeps downstream joins cheaper.
Name aliases explicitly - unnamed expressions are harder to debug in plans.

4. Join two frames on a key

Relational-style merges without leaving Rust.

use polars::prelude::*;
 
fn join_users(orders: LazyFrame, users: LazyFrame) -> LazyFrame {
    orders.join(
        users,
        [col("user_id")],
        [col("id")],
        JoinArgs::new(JoinType::Inner),
    )
}

Inner joins drop non-matching keys - validate row counts after join.
Cast key columns to the same dtype before joining to avoid silent mismatches.
For large tables, filter each side before joining to shrink working sets.

5. Write Parquet for downstream analytics

Columnar output compresses well and preserves types.

use polars::prelude::*;
 
fn write_parquet(df: &mut DataFrame, path: &str) -> PolarsResult<()> {
    let file = std::fs::File::create(path)?;
    ParquetWriter::new(file).finish(df)?;
    Ok(())
}

Parquet stores column statistics - query engines can skip row groups.
ZSTD compression is a good default for analytic workloads.
Partition large outputs by date or region for faster selective reads.

Related: CSV/Parquet I/O - compression and schema

6. Stream CSV in chunks

Bound memory when the file exceeds RAM.

use polars::prelude::*;
use std::fs::File;
 
fn process_chunks(path: &str, chunk_rows: usize) -> PolarsResult<()> {
    let file = File::open(path)?;
    let reader = CsvReader::new(file).with_chunk_size(chunk_rows);
    for chunk in reader.into_iter() {
        let df = chunk?;
        println!("chunk rows: {}", df.height());
    }
    Ok(())
}

Chunked readers never load the full file at once.
Aggregate per-chunk partials, then reduce - do not keep every chunk in memory.
Pick chunk size from available RAM and column width.

Related: Streaming & Large Data - backpressure patterns

7. Build a data CLI with clap

Ship ETL as a single binary with typed flags.

use clap::Parser;
use polars::prelude::*;
 
#[derive(Parser)]
struct Cli {
    #[arg(long)]
    input: String,
    #[arg(long, default_value = "out.parquet")]
    output: String,
}
 
fn main() -> PolarsResult<()> {
    let cli = Cli::parse();
    let mut df = CsvReadOptions::default()
        .try_into_reader_with_file_path(Some(cli.input.into()))?
        .finish()?;
    ParquetWriter::new(std::fs::File::create(&cli.output)?).finish(&mut df)?;
    Ok(())
}

clap derive gives help text and validation for free.
Return PolarsResult from main for readable error chains.
Add --filter or --columns flags as your pipeline grows.

Related: Building Data CLIs & Services - production CLI layout

Intermediate Examples

8. Apache Arrow RecordBatch between processes

Share columnar memory without serializing row-by-row JSON.

use arrow::array::{Int32Array, StringArray};
use arrow::record_batch::RecordBatch;
use std::sync::Arc;
 
fn sample_batch() -> RecordBatch {
    let ids = Int32Array::from(vec![1, 2, 3]);
    let names = StringArray::from(vec!["alpha", "beta", "gamma"]);
    RecordBatch::try_from_iter(vec![
        ("id", Arc::new(ids) as _),
        ("name", Arc::new(names) as _),
    ])
    .expect("valid batch")
}

RecordBatches are the interchange unit for Arrow, Polars, DataFusion, and Python.
Column arrays are immutable and reference-counted - cheap to clone handles.
Schema is enforced at batch creation - mismatched lengths fail fast.

Related: Apache Arrow - zero-copy semantics

9. Embed SQL with DataFusion

Query in-memory Arrow tables without a separate database server.

use datafusion::prelude::*;
use datafusion::error::Result;
 
#[tokio::main]
async fn main() -> Result<()> {
    let ctx = SessionContext::new();
    ctx.register_csv("orders", "orders.csv", CsvReadOptions::default()).await?;
    let df = ctx.sql("SELECT region, SUM(amount) AS total FROM orders GROUP BY region").await?;
    df.show().await?;
    Ok(())
}

DataFusion parses SQL, optimizes logical plans, and executes over Arrow.
Register tables from CSV, Parquet, or in-memory batches.
Use for embedded analytics, custom query engines, or SQL-over-files services.

Related: DataFusion - custom UDFs and optimizers

Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.

Data Basics

9 examples to get you started with high-performance data in Rust - 7 basic and 2 intermediate.

Prerequisites

cargo new data-playground && cd data-playground
cargo add polars --features lazy,csv,parquet
cargo add arrow
cargo add clap --features derive

Tooling: Examples target Rust 1.97.0 (edition 2024) and Polars 0.46+.

Basic Examples

1. Read a CSV with Polars (eager)

Load a small file and inspect schema before any transforms.

use polars::prelude::*;
 
fn main() -> PolarsResult<()> {
    let df = CsvReadOptions::default()
        .with_has_header(true)
        .try_into_reader_with_file_path(Some("sales.csv".into()))?
        .finish()?;
    println!("{}", df.head(Some(3)));
    Ok(())
}

CsvReadOptions centralizes delimiter, dtypes, and null handling.
Eager mode materializes the full frame - fine for files that fit in RAM.
Set explicit dtypes early to avoid silent string inference on numeric columns.

Related: CSV/Parquet I/O - format-specific tuning

2. Lazy scan and filter before collect

Push predicates down so Polars reads less data from disk.

use polars::prelude::*;
 
fn main() -> PolarsResult<()> {
    let q = LazyCsvReader::new("events.csv".into())
        .with_has_header(true)
        .finish()?
        .filter(col("amount").gt(lit(100)))
        .select([col("user_id"), col("amount")]);
    let df = q.collect()?;
    println!("rows: {}", df.height());
    Ok(())
}

Lazy plans optimize before execution - filters and column picks fuse together.
collect() is the boundary where memory is allocated for the result.
Prefer lazy pipelines when the source is larger than available RAM.

Related: Polars - lazy vs eager trade-offs

3. Group-by aggregation in one expression

Summarize metrics without manual hash maps.

use polars::prelude::*;
 
fn revenue_by_region(df: LazyFrame) -> PolarsResult<LazyFrame> {
    Ok(df
        .group_by([col("region")])
        .agg([col("revenue").sum().alias("total_revenue")])
        .sort(["total_revenue"], SortMultipleOptions::default()))
}

group_by + agg maps cleanly to SQL GROUP BY.
Sorting in the lazy plan keeps downstream joins cheaper.
Name aliases explicitly - unnamed expressions are harder to debug in plans.

4. Join two frames on a key

Relational-style merges without leaving Rust.

use polars::prelude::*;
 
fn join_users(orders: LazyFrame, users: LazyFrame) -> LazyFrame {
    orders.join(
        users,
        [col("user_id")],
        [col("id")],
        JoinArgs::new(JoinType::Inner),
    )
}

Inner joins drop non-matching keys - validate row counts after join.
Cast key columns to the same dtype before joining to avoid silent mismatches.
For large tables, filter each side before joining to shrink working sets.

5. Write Parquet for downstream analytics

Columnar output compresses well and preserves types.

use polars::prelude::*;
 
fn write_parquet(df: &mut DataFrame, path: &str) -> PolarsResult<()> {
    let file = std::fs::File::create(path)?;
    ParquetWriter::new(file).finish(df)?;
    Ok(())
}

Parquet stores column statistics - query engines can skip row groups.
ZSTD compression is a good default for analytic workloads.
Partition large outputs by date or region for faster selective reads.

Related: CSV/Parquet I/O - compression and schema

6. Stream CSV in chunks

Bound memory when the file exceeds RAM.

use polars::prelude::*;
use std::fs::File;
 
fn process_chunks(path: &str, chunk_rows: usize) -> PolarsResult<()> {
    let file = File::open(path)?;
    let reader = CsvReader::new(file).with_chunk_size(chunk_rows);
    for chunk in reader.into_iter() {
        let df = chunk?;
        println!("chunk rows: {}", df.height());
    }
    Ok(())
}

Chunked readers never load the full file at once.
Aggregate per-chunk partials, then reduce - do not keep every chunk in memory.
Pick chunk size from available RAM and column width.

Related: Streaming & Large Data - backpressure patterns

7. Build a data CLI with clap

Ship ETL as a single binary with typed flags.

use clap::Parser;
use polars::prelude::*;
 
#[derive(Parser)]
struct Cli {
    #[arg(long)]
    input: String,
    #[arg(long, default_value = "out.parquet")]
    output: String,
}
 
fn main() -> PolarsResult<()> {
    let cli = Cli::parse();
    let mut df = CsvReadOptions::default()
        .try_into_reader_with_file_path(Some(cli.input.into()))?
        .finish()?;
    ParquetWriter::new(std::fs::File::create(&cli.output)?).finish(&mut df)?;
    Ok(())
}

clap derive gives help text and validation for free.
Return PolarsResult from main for readable error chains.
Add --filter or --columns flags as your pipeline grows.

Related: Building Data CLIs & Services - production CLI layout

Intermediate Examples

8. Apache Arrow RecordBatch between processes

Share columnar memory without serializing row-by-row JSON.

use arrow::array::{Int32Array, StringArray};
use arrow::record_batch::RecordBatch;
use std::sync::Arc;
 
fn sample_batch() -> RecordBatch {
    let ids = Int32Array::from(vec![1, 2, 3]);
    let names = StringArray::from(vec!["alpha", "beta", "gamma"]);
    RecordBatch::try_from_iter(vec![
        ("id", Arc::new(ids) as _),
        ("name", Arc::new(names) as _),
    ])
    .expect("valid batch")
}

RecordBatches are the interchange unit for Arrow, Polars, DataFusion, and Python.
Column arrays are immutable and reference-counted - cheap to clone handles.
Schema is enforced at batch creation - mismatched lengths fail fast.

Related: Apache Arrow - zero-copy semantics

9. Embed SQL with DataFusion

Query in-memory Arrow tables without a separate database server.

use datafusion::prelude::*;
use datafusion::error::Result;
 
#[tokio::main]
async fn main() -> Result<()> {
    let ctx = SessionContext::new();
    ctx.register_csv("orders", "orders.csv", CsvReadOptions::default()).await?;
    let df = ctx.sql("SELECT region, SUM(amount) AS total FROM orders GROUP BY region").await?;
    df.show().await?;
    Ok(())
}

DataFusion parses SQL, optimizes logical plans, and executes over Arrow.
Register tables from CSV, Parquet, or in-memory batches.
Use for embedded analytics, custom query engines, or SQL-over-files services.

Related: DataFusion - custom UDFs and optimizers

Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.

Data Basics

Buscar en la documentación

Data Basics