Modern Data Science and Machine Learning Systems
Data Manipulation with Polars
Beyond Pandas
If you've worked with data in Python, you know Pandas. It's the reliable tool we all learned first. But as datasets grow and performance becomes critical, you start to feel its limits. This is where Polars comes in. It's not just a faster version of Pandas; it's a completely rethought approach to data manipulation.
Built from the ground up in , Polars is designed for the modern multi-core processor. It parallelizes your work automatically, meaning it uses all of your computer's power without you having to write complex code. This, combined with its use of the memory format, leads to dramatic speedups for common data tasks like joins, aggregations, and transformations.
Think Lazy, Act Fast
The biggest shift when moving to Polars is the concept of "lazy evaluation." In Pandas, when you write a line of code, it runs immediately. This is called eager execution. It's straightforward but can be inefficient.
Imagine ordering a coffee. Eager execution is like telling the barista to get the cup, then telling them to add the espresso, then telling them to add the milk. Each command is a separate step.
Polars encourages a lazy approach. You first build a complete plan of all the steps you want to perform. Then, you tell Polars to execute the plan. This allows its powerful query optimizer to analyze the entire workflow, find shortcuts, and run everything in the most efficient way possible. It's like giving the barista your entire order at once so they can prepare it logically.
import polars as pl
# Eager execution: Each step runs immediately
df_eager = pl.read_csv('data.csv')
df_filtered = df_eager.filter(pl.col('sales') > 1000)
df_final = df_filtered.select(['name', 'sales'])
# Lazy execution: We build a plan first
lazy_frame = pl.scan_csv('data.csv') # scan_csv creates a LazyFrame
query_plan = (
lazy_frame
.filter(pl.col('sales') > 1000)
.select(['name', 'sales'])
)
# Now, we execute the plan
df_lazy_final = query_plan.collect()
Notice the scan_csv and collect methods. scan_csv doesn't actually read the file; it just scans the schema and creates a LazyFrame. The work only happens when you call collect().
The Power of Expressions
At the heart of Polars are expressions. Instead of chaining methods that immediately return a new DataFrame, you define transformations on columns. These expressions are the core components of your lazy query plan.
The main contexts where you use expressions are:
select(): To choose or create new columns.filter(): To select rows based on a condition.with_columns(): To add or overwrite columns.
# Let's build a more complex query plan
q = (
pl.scan_csv('sales_data.csv')
.filter(pl.col('region') == 'North')
.with_columns([
# Create a new column for sales tax
(pl.col('price') * 0.07).alias('tax'),
# Standardize product names to lowercase
pl.col('product_name').str.to_lowercase().alias('product_lower')
])
.group_by('product_lower')
.agg([
# Aggregate data: sum of price and count of records
pl.sum('price').alias('total_revenue'),
pl.count().alias('units_sold')
])
.sort('total_revenue', descending=True)
)
# Execute the plan to get the result
top_products = q.collect()
This code is highly readable. Each line declares a logical step in the transformation. Polars figures out the best way to execute this chain of expressions in parallel when collect() is called.
Handling Larger-Than-Memory Data
The combination of lazy evaluation and an efficient memory format allows Polars to do something incredible: process datasets that are too big to fit in your computer's RAM. This is known as streaming.
When you enable streaming, Polars processes the data in smaller chunks that do fit in memory. It performs all the planned transformations on one chunk, then moves to the next, combining the results as it goes. You enable this with a simple argument to the collect method.
# Imagine 'huge_dataset.csv' is 50 GB and you only have 16 GB of RAM
q_streaming = (
pl.scan_csv('huge_dataset.csv')
.filter(pl.col('value') > 0)
.group_by('category')
.agg(pl.mean('value'))
)
# Polars will process this file in chunks without crashing
result = q_streaming.collect(streaming=True)
This simple streaming=True flag unlocks the ability to work on massive datasets on a regular laptop, a task that would have previously required specialized hardware or a distributed computing cluster. It's a powerful demonstration of how a modern, well-designed tool can change what's possible in data analysis.
What is the primary reason Polars often outperforms Pandas on modern hardware?
In Polars, the concept of building a query plan first and executing it later is known as: