Polars vs Pandas — A Benchmark That Changed How I Process Data

I was skeptical about Polars. Another DataFrame library? We have pandas, Dask, Vaex, cuDF — did we need another? Then I ran the benchmarks. I’ve been migrating pipelines to Polars ever since.

This post presents reproducible benchmarks across real operations — not cherry-picked toy examples — and explains why Polars is faster. More importantly, it covers the cases where pandas still wins and the migration path.

Table of contents

The Setup

All benchmarks run on a single machine (Ubuntu 22.04, AMD Ryzen 9 5900X 12-core, 64GB RAM, NVMe SSD). No GPU acceleration. No distributed computing. Just single-node performance.

Datasets: financial transaction records with realistic structure:

import pandas as pd
import polars as pl
import numpy as np
import time

def generate_dataset(n: int) -> dict:
    np.random.seed(42)
    return {
        'transaction_id': np.arange(n, dtype=np.int64),
        'account_id': np.random.randint(0, n // 100, n),  # ~100 tx per account
        'merchant_id': np.random.randint(0, 10_000, n),
        'amount': np.random.exponential(100, n).astype(np.float32),
        'currency': np.random.choice(['EUR', 'USD', 'GBP', 'CHF'], n),
        'category': np.random.choice(
            ['retail', 'food', 'transport', 'entertainment', 'utilities', 'other'], n
        ),
        'is_fraud': np.random.choice([True, False], n, p=[0.01, 0.99]),
        'timestamp': pd.date_range('2020-01-01', periods=n, freq='1s'),
    }

sizes = [1_000_000, 10_000_000, 50_000_000, 100_000_000]

Benchmark 1: GroupBy + Aggregation

The most common data operation: group by account, compute stats.

# Pandas
def pandas_groupby(df):
    return df.groupby('account_id').agg(
        total_amount=('amount', 'sum'),
        tx_count=('transaction_id', 'count'),
        avg_amount=('amount', 'mean'),
        fraud_rate=('is_fraud', 'mean'),
    ).reset_index()

# Polars
def polars_groupby(df):
    return df.group_by('account_id').agg([
        pl.col('amount').sum().alias('total_amount'),
        pl.len().alias('tx_count'),
        pl.col('amount').mean().alias('avg_amount'),
        pl.col('is_fraud').mean().alias('fraud_rate'),
    ])

Results (seconds):

The speedup grows with dataset size because pandas is single-threaded for groupby, while Polars uses all available cores automatically.

Benchmark 2: Filter + Sort + Window Function

More complex: find accounts with above-average fraud rates, compute rolling average.

# Pandas
def pandas_window(df):
    # Step 1: Account-level fraud rate
    fraud_by_account = (
        df.groupby('account_id')['is_fraud']
        .mean()
        .reset_index(name='account_fraud_rate')
    )
    # Step 2: Join back
    df = df.merge(fraud_by_account, on='account_id')
    # Step 3: Filter high-risk accounts
    high_risk = df[df['account_fraud_rate'] > 0.02]
    # Step 4: Sort and rolling average per account
    high_risk = high_risk.sort_values(['account_id', 'timestamp'])
    high_risk['rolling_amount'] = (
        high_risk.groupby('account_id')['amount']
        .transform(lambda x: x.rolling(7, min_periods=1).mean())
    )
    return high_risk

# Polars
def polars_window(df):
    return (
        df
        .with_columns(
            pl.col('is_fraud').mean().over('account_id').alias('account_fraud_rate')
        )
        .filter(pl.col('account_fraud_rate') > 0.02)
        .sort(['account_id', 'timestamp'])
        .with_columns(
            pl.col('amount').rolling_mean(7, min_periods=1).over('account_id')
            .alias('rolling_amount')
        )
    )

Results (seconds):

The over() function in Polars is the key — it applies window functions without an explicit groupby-merge cycle. The API is more expressive and the implementation is multithreaded.

Benchmark 3: Join Performance

Joining a 10M row transaction table with a 100K row merchant reference table:

# Pandas
def pandas_join(transactions, merchants):
    return transactions.merge(merchants, on='merchant_id', how='left')

# Polars
def polars_join(transactions, merchants):
    return transactions.join(merchants, on='merchant_id', how='left')

Results:

Polars uses a hash join with parallel partition building. pandas also uses hash-based joins internally, but executes them single-threaded — the parallelism gap is the dominant factor here.

Benchmark 4: Reading Parquet

Both use Arrow/Parquet under the hood, but Polars has better predicate pushdown:

# Reading 100M rows, filtering to ~1M
# Pandas
t0 = time.time()
df = pd.read_parquet('transactions.parquet')
filtered = df[df['is_fraud'] == True]
print(f"pandas: {time.time()-t0:.1f}s")

# Polars lazy (predicate pushdown)
t0 = time.time()
filtered = (
    pl.scan_parquet('transactions.parquet')
    .filter(pl.col('is_fraud') == True)
    .collect()
)
print(f"polars lazy: {time.time()-t0:.1f}s")

Results:

The Polars lazy reader pushes the filter into the Parquet scan — it only reads rows where is_fraud == True from disk, without materializing the full dataset. This is the killer feature for large files.

Why Polars Is Faster: The Technical Reasons

1. Written in Rust, compiled to native code: No Python interpreter overhead in the hot path. The groupby, join, and filter code runs as optimized machine code.

2. Multithreaded by default: Polars uses Rayon (Rust’s parallel iterator library) to split operations across all CPU cores. pandas core operations are single-threaded (though some NumPy operations release the GIL).

3. SIMD vectorization: Polars uses SIMD (AVX2/AVX-512) instructions for arithmetic operations — processing 8-16 float32 values per CPU instruction. Combined with cache-friendly column-oriented storage, this is dramatically faster.

4. Apache Arrow memory format: Polars uses Arrow internally by design. pandas 2.x supports Arrow-backed columns via dtype_backend='pyarrow' opt-in, but still defaults to NumPy arrays. Polars’ native Arrow format means no copy operations between internal and export formats.

5. Query optimizer: The lazy API compiles your operations into a query plan and optimizes it (predicate pushdown, projection pushdown, common subexpression elimination). pandas executes each operation eagerly.

Where Pandas Still Wins

Polars is not a drop-in replacement. Cases where you should stick with pandas:

1. Integration with the Python ML ecosystem: scikit-learn, matplotlib, seaborn all expect pandas DataFrames. Polars has .to_pandas() but adding that call negates performance gains.

2. Datetime index and resample: pandas resample() and DatetimeIndex are more mature. Time series resampling in Polars requires more verbose code.

3. String operations on small datasets: For <100K rows with complex string manipulation, the overhead of Polars setup and Arrow conversion can exceed pandas.

4. Excel I/O: Polars now has pl.read_excel() and DataFrame.write_excel() (since v0.18+), so this is no longer a gap. Pandas still has broader options support (xlsxwriter, openpyxl engines).

5. Existing codebase: If your team is pandas-fluent and you have 100k lines of pandas code, the migration cost is high. Optimize your pandas code first (see my previous post on vectorization).

The Polars API: Key Differences

Method chaining is idiomatic:

# Polars style — everything chains
result = (
    df
    .filter(pl.col('amount') > 100)
    .with_columns(
        (pl.col('amount') * 1.05).alias('amount_with_fee'),
        pl.col('category').str.to_uppercase().alias('category_upper'),
    )
    .group_by('account_id')
    .agg(pl.col('amount_with_fee').sum())
    .sort('amount_with_fee', descending=True)
    .head(100)
)

No inplace operations: Polars DataFrames are immutable. Every operation returns a new DataFrame. This prevents the common pandas gotcha of accidentally modifying the original.

Explicit null handling: Polars distinguishes None (Python null) from NaN (float IEEE not-a-number). pandas conflates them, causing subtle bugs with numeric nulls.

over() instead of transform(): The Polars equivalent of groupby().transform():

# pandas
df['account_avg'] = df.groupby('account_id')['amount'].transform('mean')

# polars
df = df.with_columns(
    pl.col('amount').mean().over('account_id').alias('account_avg')
)

Lazy vs Eager Evaluation

This is Polars’ most important architectural decision:

# Eager (default): executes immediately, like pandas
df = pl.read_parquet('file.parquet')
result = df.filter(pl.col('amount') > 100).select(['account_id', 'amount'])

# Lazy: builds a query plan, optimizes, then executes
result = (
    pl.scan_parquet('file.parquet')    # No data loaded yet
    .filter(pl.col('amount') > 100)    # Added to query plan
    .select(['account_id', 'amount'])  # Added to query plan
    .collect()                          # Execute optimized plan
)

For files larger than RAM, lazy mode is the only viable approach. The query optimizer can push filters and column selection into the scan, avoiding materializing data that will be immediately discarded.

Migration Strategy

Start with new pipelines, not existing code:

1. New ETL jobs → Polars from the start
2. Bottleneck identification → profile your pandas code (see [Clinic.js equivalent for Python: line_profiler])
3. Hot paths only → replace the slowest pandas operations with Polars equivalents
4. Convert at boundaries → keep Polars for computation, convert to pandas for ML model training

The benchmark numbers are real. If you’re processing more than a few million rows regularly, Polars will meaningfully reduce your infrastructure costs and pipeline runtime. The learning curve is shallow for anyone already familiar with pandas — the concepts transfer, the syntax is cleaner, and the performance is not close.