High-Performance Data Best Practices

Q: How do I document breaking schema changes?

Bump a schema_version field in table metadata and maintain a one-release dual-read window when possible.

Vectorized, memory-aware pipelines in Rust - rules for Polars, Arrow, and production ETL.

How to Use This List

Apply A-C when designing a new pipeline before writing code.
Walk D-F before shipping to production or cron.
Revisit after incidents involving OOM, schema drift, or slow scans.
Check boxes in PR descriptions for data-touching changes.

A - Schema and Ingest

Declare CSV dtypes explicitly. Inference turns numeric IDs into strings and breaks joins downstream.
Version your output Parquet schema. Consumers fail fast instead of reading garbage types.
Normalize timestamps to UTC at ingest. Mixed offsets cause silent join skew across files.
Reject or quarantine bad rows with counted metrics. Threshold-based job failure beats silent drops.
Prefer Parquet over CSV for anything queried twice. Compression and stats pay back immediately.

B - Execution and Memory

Default to lazy Polars for file-backed work. Push filters and column picks before collect().
Collect once at the end of a lazy chain. Intermediate collects multiply memory and CPU.
Stream or chunk when input exceeds ~40% of RAM. HashMap partials merge only keys you need.
Project columns early in Parquet scans. Wide tables punish select * on object stores.
Run CPU-bound transforms on spawn_blocking inside Axum. Protect the Tokio I/O pool.

C - Interop and Formats

Use Arrow IPC or Parquet for Python handoff. JSON row arrays do not scale past megabytes.
Pin Polars and PyArrow versions in CI. Extension type mismatches surface as decode errors.
Document nullable vs non-nullable per column. Arrow and Parquet enforce this at boundaries.
Dictionary-encode low-cardinality string dimensions. Shrinks memory and speeds group-by.
Align row group size (128K-1M rows) with query patterns. Tiny groups add metadata overhead; huge groups hurt pushdown.

D - Operations and Reliability

Write outputs atomically (temp file + rename). Crash mid-write must not publish corrupt Parquet.
Emit structured logs per stage with row counts. tracing spans make nightly jobs debuggable.
Return distinct CLI exit codes for usage vs data vs system errors. Orchestrators need correct retries.
Benchmark chunk size on representative hardware. Laptop defaults do not transfer to 512GB servers.
Cap HTTP payload and query cost in data services. Unbounded summaries are DoS vectors.

E - Performance Culture

Measure with POLARS_MAX_THREADS tuned per deployment. Colocated services may need fewer threads.
Compare against baseline pandas only once. Rust wins should be tracked in CI benchmarks, not anecdotes.
Explain lazy plans during development. .explain(true) catches missing pushdown before production.
Prefer vectorized expressions over row loops. Manual for over rows forfeits SIMD and parallelism.
Spill or push heavy joins to a database when key cardinality explodes. In-process hash joins are not infinite.

FAQs

What is the single highest-impact rule?

Lazy execution with early filters on large inputs - it reduces bytes read and memory allocated before any business logic runs.

When is eager Polars acceptable?

Unit tests, prototypes on tiny frames, and in-memory steps after a single final collect() on bounded data.

How do I enforce this list in CI?

Add integration tests on fixture CSVs, schema assertions on output Parquet, and clippy/lint on forbidden unwrap in ETL paths.

Should every pipeline use DataFusion?

Only when SQL is the right interface. Polars expressions in Rust stay simpler for typed application code.

How do I handle PII in logs?

Log paths and row counts, not column values. Redact sample rows in debug modes behind feature flags.

What row group size is default-safe?

Start at 128_000 rows for wide analytic tables; increase if scans are always full-table and memory allows.

When should I partition Parquet directories?

When queries almost always filter on date or region - partition keys must match real filter predicates.

How do I document breaking schema changes?

Bump a schema_version field in table metadata and maintain a one-release dual-read window when possible.

Is Rust always faster than Python?

For tight numeric loops and I/O heavy ETL, usually yes. IO-bound network copies and tiny scripts may not justify Rust ops cost.

Where do I learn chunked patterns?

See Streaming & Large Data for partial aggregation examples.

Data Basics - section orientation
Polars - lazy execution details
Streaming & Large Data - bounded memory
Building Data CLIs & Services - operational shape

Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.

High-Performance Data Best Practices

Vectorized, memory-aware pipelines in Rust - rules for Polars, Arrow, and production ETL.

How to Use This List

Apply A-C when designing a new pipeline before writing code.
Walk D-F before shipping to production or cron.
Revisit after incidents involving OOM, schema drift, or slow scans.
Check boxes in PR descriptions for data-touching changes.

A - Schema and Ingest

Declare CSV dtypes explicitly. Inference turns numeric IDs into strings and breaks joins downstream.
Version your output Parquet schema. Consumers fail fast instead of reading garbage types.
Normalize timestamps to UTC at ingest. Mixed offsets cause silent join skew across files.
Reject or quarantine bad rows with counted metrics. Threshold-based job failure beats silent drops.
Prefer Parquet over CSV for anything queried twice. Compression and stats pay back immediately.

B - Execution and Memory

Default to lazy Polars for file-backed work. Push filters and column picks before collect().
Collect once at the end of a lazy chain. Intermediate collects multiply memory and CPU.
Stream or chunk when input exceeds ~40% of RAM. HashMap partials merge only keys you need.
Project columns early in Parquet scans. Wide tables punish select * on object stores.
Run CPU-bound transforms on spawn_blocking inside Axum. Protect the Tokio I/O pool.

C - Interop and Formats

Use Arrow IPC or Parquet for Python handoff. JSON row arrays do not scale past megabytes.
Pin Polars and PyArrow versions in CI. Extension type mismatches surface as decode errors.
Document nullable vs non-nullable per column. Arrow and Parquet enforce this at boundaries.
Dictionary-encode low-cardinality string dimensions. Shrinks memory and speeds group-by.
Align row group size (128K-1M rows) with query patterns. Tiny groups add metadata overhead; huge groups hurt pushdown.

D - Operations and Reliability

Write outputs atomically (temp file + rename). Crash mid-write must not publish corrupt Parquet.
Emit structured logs per stage with row counts. tracing spans make nightly jobs debuggable.
Return distinct CLI exit codes for usage vs data vs system errors. Orchestrators need correct retries.
Benchmark chunk size on representative hardware. Laptop defaults do not transfer to 512GB servers.
Cap HTTP payload and query cost in data services. Unbounded summaries are DoS vectors.

E - Performance Culture

Measure with POLARS_MAX_THREADS tuned per deployment. Colocated services may need fewer threads.
Compare against baseline pandas only once. Rust wins should be tracked in CI benchmarks, not anecdotes.
Explain lazy plans during development. .explain(true) catches missing pushdown before production.
Prefer vectorized expressions over row loops. Manual for over rows forfeits SIMD and parallelism.
Spill or push heavy joins to a database when key cardinality explodes. In-process hash joins are not infinite.

FAQs

What is the single highest-impact rule?

Lazy execution with early filters on large inputs - it reduces bytes read and memory allocated before any business logic runs.

When is eager Polars acceptable?

Unit tests, prototypes on tiny frames, and in-memory steps after a single final collect() on bounded data.

How do I enforce this list in CI?

Add integration tests on fixture CSVs, schema assertions on output Parquet, and clippy/lint on forbidden unwrap in ETL paths.

Should every pipeline use DataFusion?

Only when SQL is the right interface. Polars expressions in Rust stay simpler for typed application code.

How do I handle PII in logs?

Log paths and row counts, not column values. Redact sample rows in debug modes behind feature flags.

What row group size is default-safe?

Start at 128_000 rows for wide analytic tables; increase if scans are always full-table and memory allows.

When should I partition Parquet directories?

When queries almost always filter on date or region - partition keys must match real filter predicates.

How do I document breaking schema changes?

Bump a schema_version field in table metadata and maintain a one-release dual-read window when possible.

Is Rust always faster than Python?

For tight numeric loops and I/O heavy ETL, usually yes. IO-bound network copies and tiny scripts may not justify Rust ops cost.

Where do I learn chunked patterns?

See Streaming & Large Data for partial aggregation examples.

Data Basics - section orientation
Polars - lazy execution details
Streaming & Large Data - bounded memory
Building Data CLIs & Services - operational shape

Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.

High-Performance Data Best Practices

Buscar en la documentación

High-Performance Data Best Practices