CSV/Parquet I/O

Read and write CSV and Parquet in Rust with predictable schemas, compression, and scan performance.

Recipe

Quick-reference recipe card - copy-paste ready.

use polars::prelude::*;
 
fn csv_to_parquet(csv_path: &str, parquet_path: &str) -> PolarsResult<()> {
    let mut df = CsvReadOptions::default()
        .with_has_header(true)
        .try_into_reader_with_file_path(Some(csv_path.into()))?
        .finish()?;
    ParquetWriter::new(std::fs::File::create(parquet_path)?)
        .with_compression(ParquetCompression::Zstd(None))
        .finish(&mut df)?;
    Ok(())
}

When to reach for this:

• Landing zone ETL from vendor CSV exports
• Publishing analytic datasets for DuckDB, Spark, or Polars consumers
• Shrinking storage and egress with columnar compression
• Validating schemas at ingest before downstream SQL

Working Example

use polars::prelude::*;
 
fn main() -> PolarsResult<()> {
    let schema = Schema::from_iter([
        Field::new("order_id".into(), DataType::Int64),
        Field::new("region".into(), DataType::String),
        Field::new("amount".into(), DataType::Float64),
    ]);
 
    let df = CsvReadOptions::default()
        .with_has_header(true)
        .with_schema(Some(Arc::new(schema)))
        .try_into_reader_with_file_path(Some("orders.csv".into()))?
        .finish()?;
 
    let filtered = df
        .lazy()
        .filter(col("amount").gt(lit(0.0)))
        .collect()?;
 
    let mut out = filtered.clone();
    ParquetWriter::new(std::fs::File::create("orders_clean.parquet")?)
        .with_row_group_size(Some(128_000))
        .finish(&mut out)?;
 
    Ok(())
}

What this demonstrates:

• Supplying an explicit CSV schema instead of inferring types
• Filtering invalid rows before writing Parquet
• Setting row group size for later predicate pushdown

Deep Dive

How It Works

• CSV is row text - parsers must infer or accept dtypes per column.
• Parquet stores typed column chunks with per-row-group statistics (min/max).
• Readers skip row groups when predicates contradict statistics.
• Writers choose compression (Snappy, ZSTD, Gzip) per column encoding.

Format Comparison

Rust Notes

// Lazy scan avoids loading full CSV:
LazyCsvReader::new("big.csv".into()).with_has_header(true).finish()?;
 
// Read subset of Parquet columns:
LazyFrame::scan_parquet("big.parquet", ScanArgsParquet::default())?
    .select([col("user_id"), col("event_time")]);

Gotchas

• Locale-specific CSV decimals - 1,5 vs 1.5 breaks inference. Fix: normalize files or set decimal separator in options.
• UTF-8 BOM in headers - first column name includes invisible BOM. Fix: strip BOM or use utf8-lossy preprocessing.
• Huge row groups - poor pushdown and memory spikes on read. Fix: target 128K-1M rows per group depending on width.
• Writing Utf8 for everything - Parquet still stores strings but loses numeric stats. Fix: cast before write.
• Appending Parquet by concatenating files - duplicate row groups and schema conflicts. Fix: write partitioned dataset directories with consistent schema.

Alternatives

FAQs

Related

• Polars - primary I/O API in examples
• Streaming & Large Data - chunked CSV
• Apache Arrow - in-memory target format
• Building Data CLIs & Services - CLI wrappers

Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.