Rust's High-Performance Data Model
Every dataframe library or query engine has to decide how it lays data out in memory before it decides how to compute over that data. In Rust's data ecosystem this decision was made once, by the Apache Arrow project, and nearly everything downstream inherits it. Polars and DataFusion look like they compete on API surface, expressions versus SQL, but underneath they share the same columnar substrate, and understanding that substrate explains why both are fast in the same ways and slow in the same ways.
This page is the conceptual anchor for the rest of this section. It does not walk through installing crates or writing a query; it explains the memory model that makes the rest of the section's examples work the way they do.
Summary
- Arrow defines a columnar, typed, contiguous in-memory layout, and Polars and DataFusion both build directly on that layout instead of inventing their own row-based structures.
- Insight: Row-oriented structures force a scan to touch every field of every record even when only one column is needed, which defeats CPU vectorization and wastes cache bandwidth on data the query never uses.
- Key Concepts: columnar layout, RecordBatch, zero-copy, vectorized execution, predicate pushdown, validity bitmap.
- When to Use: Deciding between Polars and DataFusion for a project, designing a Rust service that hands data to Python or the JVM, evaluating whether to build a custom in-memory format, or diagnosing why an operation you assumed was fast is not.
- Limitations/Trade-offs: Columnar layouts make single-row lookups and frequent small in-place mutations more expensive than a row store built for exactly that access pattern.
- Related Topics: Apache Arrow, Polars, DataFusion, Parquet.
Foundations
A row-oriented structure stores one full record contiguously, then the next record right after it. Think of a spreadsheet read left to right, one entire row at a time, before moving to the next row. That layout is intuitive and it matches how most application code accesses data: fetch one record, read its fields.
A columnar layout inverts this. It stores every value from one column together, in one contiguous buffer, before storing the next column's values in a separate buffer. The spreadsheet is now read top to bottom, one column at a time. This sounds like a small rearrangement, but it changes almost everything about how fast a query engine can run.
Apache Arrow standardized this columnar layout as a cross-language specification rather than a library-specific detail. Before Arrow, every analytics engine (pandas, Spark, Drill, dplyr) held its own private notion of "a table in memory," and moving data between them meant serializing to some intermediate row format and paying that conversion cost every time. Arrow's contribution was not the idea of columnar storage itself, database engines had used it for years, but the decision to make the byte layout identical across languages and tools so that data could move between them without conversion.
A useful analogy is a shipping container standard. Any port, ship, or truck built to the container spec can move any container without repacking its contents. Arrow is that standard for in-memory tabular data: any Arrow-aware tool, in Rust, Python, or the JVM, can read the same buffers without repacking.
Polars and DataFusion are two different front doors into that same container yard. Polars gives you a DataFrame expression API; DataFusion gives you SQL. Both compile down to operations over Arrow-formatted columns.
Mechanics & Interactions
Internally, an Arrow column is a typed, contiguous buffer plus a validity bitmap that marks which slots are null. Because every value in a buffer shares one type and sits next to its neighbors in memory, the CPU can load several values in a single instruction. This is what enables SIMD (single-instruction, multiple-data) vectorization: a kernel summing an Int64 column processes four or eight values per cycle instead of one, because it never has to jump past unrelated fields the way a row scan does.
Cache locality compounds this effect. When a query only needs the amount column out of a fifty-column table, a columnar engine reads only the amount buffer. A row-oriented scan of the same table would pull every column of every row into cache just to discard forty-nine of them, wasting memory bandwidth that is usually the real bottleneck in analytic workloads, not raw CPU cycles.
Polars' DataFrame and LazyFrame are, structurally, collections of Arrow-backed Series, chunked into one or more buffers per column. DataFusion goes a step further and makes the RecordBatch, a horizontal slice across several Arrow columns sharing one length, the actual unit that flows between physical execution operators. Instead of a classic row-at-a-time "volcano" model where each operator pulls one row from the next, DataFusion's operators pull one batch of thousands of rows at a time. This vectorized batch-at-a-time execution is only possible because the data underneath is already columnar; a row-oriented volcano model would gain nothing from batching, since operators still work one field at a time.
Row-oriented (one record at a time):
[id1, name1, amt1] [id2, name2, amt2] [id3, name3, amt3] ...
Columnar (one field at a time, across all records):
id: [id1, id2, id3, ...]
name: [name1, name2, name3, ...]
amt: [amt1, amt2, amt3, ...]Sharing this layout is also what makes cross-tool zero-copy handoffs possible. When Polars produces a result and hands it to DataFusion, or when a Rust service returns an Arrow RecordBatch to a Python caller through PyArrow, no row-by-row conversion happens. The receiving process reads the same buffers, wrapped in Arc<dyn Array> inside Rust so cloning a handle never deep-copies megabytes of data. Zero-copy is a property of matching layouts, not a special trick either library performs.
Lazy query planning in both engines exists to decide, ahead of execution, which columns and which row ranges actually need to be materialized into buffers at all. Predicate pushdown and column projection happen at the plan level, before any Arrow array is built, so the columnar model pays off twice: once by skipping unread columns, and again by skipping unread rows in formats like Parquet that store per-row-group statistics.
Advanced Considerations & Applications
The most consequential architectural decision in this ecosystem was not "should we use columns," it was "should every tool agree on the exact same byte layout." Before that agreement, interop between two "fast" analytics tools still meant a serialization tax at every boundary. Arrow Flight (a gRPC-based transport for Arrow data) and the Arrow C Data Interface (a stable ABI for sharing Arrow buffers across FFI boundaries without copying) exist specifically to extend zero-copy semantics across process and language boundaries, not just within one Rust binary.
This is also why hand-rolling a custom struct-of-arrays format in a Rust service rarely pays off today. The raw layout is not hard to reproduce, but the ecosystem around Arrow, PyArrow interop, Flight transport, DataFusion's optimizer, Polars' expression engine, is the actual value, and a bespoke format gets none of it.
Vectorized execution interacts with async runtimes in a specific way worth naming: a RecordBatch is a synchronous, CPU-bound unit of work, while the pipeline moving batches between stages (especially in DataFusion services) is often async. A batch that is too large blocks a Tokio worker thread for too long; a batch that is too small reintroduces per-batch overhead and starts eroding the cache-locality benefit that justified going columnar in the first place. Batch sizing (commonly tens of thousands of rows) is a real tuning knob, not an implementation detail to ignore.
| Approach | Strength | Weakness | Best Fit |
|---|---|---|---|
Row-oriented struct (Vec<Record>) | Fast single-record reads and mutation | Poor scan throughput, no cross-tool interop | Small in-process state, OLTP-style access |
| Custom struct-of-arrays | Good scan throughput within one codebase | No shared spec, every boundary needs a converter | Isolated numeric kernels with no interop needs |
| Arrow-backed (Polars, DataFusion) | Vectorized scans, zero-copy interop, shared tooling | Costlier point lookups and small mutations | Analytics, ETL, and any pipeline crossing language or process boundaries |
As of Polars 0.46+, the expression API (col, lit, when) sits directly on top of this Arrow-backed layout and is the documented, stable surface to target; the underlying columnar guarantees described here have not changed across recent Polars releases, only the ergonomics on top of them have.
Common Misconceptions
- "Columnar storage automatically makes my code vectorized." Layout alone does not vectorize anything; a naive per-row loop written over Arrow arrays in application code still runs one value at a time. Vectorization comes from kernels (in Polars, DataFusion, or
arrow's compute module) written to exploit contiguous same-type buffers, not from the layout by itself. - "Arrow is a file format, like Parquet." Arrow describes an in-memory layout; Parquet describes an on-disk, compressed layout. Engines routinely decode Parquet files into Arrow batches for computation, but the two specs solve different problems.
- "Zero-copy works across any two processes automatically." It requires matching buffer alignment and, for cross-machine cases, a shared-memory or mmap arrangement; a network hop between two services still copies bytes at least once on the receiving side.
- "Polars and DataFusion have fundamentally different internals because their APIs look so different." The expression DSL and the SQL parser are just two front ends; both compile to operations over the same Arrow-shaped batches, and results from one can be registered as tables in the other.
- "Columnar is strictly faster than row-oriented, full stop." It wins for scans and aggregations over few columns of many rows. It loses for point lookups, frequent single-record mutation, and any access pattern that legitimately needs a whole record at once.
- "I could get the same benefits by writing my own struct-of-arrays type." You would get the scan-speed benefit inside your own code, but not the interoperability that made the ecosystem approach worth adopting in the first place: shared Flight transport, PyArrow handoff, and every downstream tool that already speaks Arrow.
FAQs
What actually makes a layout "columnar" instead of "row-oriented"?
It is about which values sit next to each other in memory.
- Row-oriented: all fields of one record are contiguous, then the next record follows.
- Columnar: all values of one field are contiguous across every record, then the next field's buffer follows.
- The same data, rearranged, produces very different scan and cache behavior.
Why didn't Polars and DataFusion just build their own memory formats?
They could have, and earlier-generation tools often did.
Standardizing on Arrow means both projects inherit zero-copy interop with Python, the JVM, and each other for free, instead of each maintaining a private format and a converter for every other tool they want to talk to.
How does columnar layout actually produce SIMD speedups?
Because every value in a column buffer shares one type and sits contiguously in memory, the CPU can load and operate on several values with a single vector instruction instead of one value at a time. A row scan interrupts that pattern every time it crosses into an unrelated field.
What is a RecordBatch, in plain terms?
A horizontal slice across a fixed set of Arrow columns that all share the same length. It is the unit that flows between operators in DataFusion's execution engine and the unit exchanged over Arrow IPC or Flight between processes.
Does using Arrow mean my Rust code is automatically fast?
No. The layout enables vectorized kernels to be fast; code you write that loops over array values one at a time does not automatically benefit just because the underlying buffer is columnar.
How are null values represented in a columnar array?
Each array carries a separate validity bitmap alongside its value buffer, one bit per slot, marking which positions are null. The value buffer still reserves space at null positions; the bitmap is what tells readers to ignore it.
Is Arrow the same thing as Parquet?
No. Arrow is an in-memory layout; Parquet is a compressed on-disk layout. Query engines commonly decode Parquet row groups directly into Arrow batches for computation, which is why the two are often mentioned together.
When is the columnar model the wrong choice?
For workloads dominated by single-record reads or writes, think fetching one user profile by ID, or updating one row at a time in a hot loop, a row-oriented store or a traditional OLTP database will outperform a columnar engine, which pays overhead reassembling or rewriting a full record from scattered column buffers.
How does zero-copy interop with Python actually work?
Both sides agree on the same byte layout, so a Rust process can hand a pointer and schema to a Python process (typically through PyArrow) and Python reads the same buffers without any row-by-row deserialization step, as long as alignment and endianness match.
Why do Polars and DataFusion feel so different if they share the same memory model?
The difference is entirely at the API layer. Polars exposes a Rust-native expression DSL (col, lit, method chaining); DataFusion exposes SQL text parsed into a logical plan. Both plans ultimately execute as operations over Arrow-shaped batches, which is why moving a result from one into the other as a registered table is cheap.
Does a lazy query touch Arrow at every step, or only at the end?
Only at the end, by design. The lazy plan is a description of work (filters, projections, joins) that the optimizer rewrites before any Arrow buffer is allocated; collect() (Polars) or query execution (DataFusion) is the point where buffers actually get built.
Do I need to understand Arrow to use Polars effectively?
Not to write basic queries, but it explains behavior you will otherwise find surprising: why collect() is a memory and CPU spike boundary, why wide select * on a huge Parquet file is expensive, and why exporting to Python or DataFusion is nearly free compared to a JSON round-trip.
Related
- Data Basics - hands-on walkthrough that builds on this mental model
- Apache Arrow - the columnar format itself, with RecordBatch construction and IPC
- Polars - the expression-based DataFrame API built on this layout
- DataFusion - the SQL engine built on the same layout
Stack versions: This page was written for Rust 1.97.0 (edition 2024), Tokio 1.x, Axum 0.8, serde 1.0, sqlx 0.8, clap 4, and Polars 0.46+.