Practical Machine Learning and Algorithm Selection
Data Engineering Pipelines
Automating Your Workflow
In machine learning, your raw data is rarely ready for a model. You need to handle missing values, scale numerical features, and encode categorical ones. Doing these steps manually for your training, validation, and test sets is tedious and, more importantly, prone to error. A common mistake is data leakage, where information from the test set accidentally influences the training process, leading to overly optimistic performance metrics.
This is where pipelines come in. A pipeline chains together multiple processing steps, from data cleaning to model training, into a single, reusable object. By encapsulating your entire workflow, you ensure that the same transformations are applied consistently to every piece of data that passes through it. This is the key to creating reproducible and robust models.
Pipelines in Scikit-learn puts chain preprocessing steps and models into a single workflow, ensuring consistency during training and testing.
The core component for this in scikit-learn is the Pipeline class. You provide it with a list of steps, each a tuple containing a name for the step and the transformer or estimator object itself.
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
# Define the pipeline steps
steps = [
('imputer', SimpleImputer(strategy='mean')), # Step 1: Handle missing values
('scaler', StandardScaler()), # Step 2: Scale features
('model', LogisticRegression()) # Step 3: The model itself
]
# Create the pipeline
pipe = Pipeline(steps)
# Now you can treat 'pipe' like a regular model
# pipe.fit(X_train, y_train)
# pipe.predict(X_test)
This is great for datasets with uniform data types, but what about real-world data, which often contains a mix of numerical and categorical columns? Applying StandardScaler to a categorical feature makes no sense. For this, we need a more surgical tool.
Handling Mixed Data Types
The ColumnTransformer is designed for exactly this scenario. It allows you to apply different transformers to different columns of your data. You specify which transformers to use and which columns to apply them to, and it handles the rest, neatly combining the processed columns back into a single array.
Let's break down the individual transformers you'd use within a ColumnTransformer.
Preprocessing Transformers
Handling Missing Values
Datasets often have missing entries. The SimpleImputer is a straightforward way to handle them. You can replace missing values (NaN) with the mean, median, or most frequent value of the column, or with a constant.
Using the 'median' strategy is often a good choice for numerical data because it's robust to outliers.
Scaling Numerical Features
Many models, especially linear ones and those using distance calculations (like SVMs or K-Nearest Neighbors), perform better when numerical features are on a similar scale. Here are three common scalers:
| Scaler | How it Works | Best For |
|---|---|---|
StandardScaler | Rescales data to have a mean of 0 and a standard deviation of 1. | Most linear models. Assumes data is normally distributed. |
MinMaxScaler | Scales features to a given range, typically [0, 1]. | Algorithms that don't assume a specific data distribution. Good for image data. |
RobustScaler | Uses statistics that are robust to outliers (median and interquartile range). | Datasets with significant outliers that you don't want to remove. |
Encoding Categorical Features
Models need numbers, not text. We use encoding to convert categorical features into a numerical format.
OneHotEncoder: Creates a new binary column for each category. For a 'color' column with values 'red', 'green', and 'blue', it would create three columns: 'color_red', 'color_green', and 'color_blue'. This is the most common approach as it doesn't imply any ordering between categories.OrdinalEncoder: Converts categories into integer values (e.g., 'low' -> 0, 'medium' -> 1, 'high' -> 2). This should only be used when the categories have a clear, intrinsic order.
Building the Full Pipeline
Now, let's combine everything into a single, powerful preprocessing pipeline using ColumnTransformer. This object will handle imputing and scaling for numerical columns, and imputing and one-hot encoding for categorical columns. Finally, we'll place this preprocessor inside a main Pipeline that feeds the processed data into a model.
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.ensemble import RandomForestClassifier
# Assume 'X' is your DataFrame and 'y' is your target series
# Separate target from predictors
# X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Identify column types
numerical_cols = X.select_dtypes(include=['int64', 'float64']).columns
categorical_cols = X.select_dtypes(include=['object', 'category']).columns
# Create the preprocessing pipelines for numerical and categorical data
numerical_transformer = Pipeline(steps=[
('imputer', SimpleImputer(strategy='median')),
('scaler', StandardScaler())
])
categorical_transformer = Pipeline(steps=[
('imputer', SimpleImputer(strategy='most_frequent')),
('onehot', OneHotEncoder(handle_unknown='ignore'))
])
# Combine preprocessing steps with ColumnTransformer
preprocessor = ColumnTransformer(
transformers=[
('num', numerical_transformer, numerical_cols),
('cat', categorical_transformer, categorical_cols)
])
# Create the final pipeline with the model
model_pipeline = Pipeline(steps=[
('preprocessor', preprocessor),
('classifier', RandomForestClassifier())
])
# Fit the model
# model_pipeline.fit(X_train, y_train)
# Make predictions
# predictions = model_pipeline.predict(X_test)
By fitting this model_pipeline on X_train, we ensure that the imputer's median and the scaler's mean/standard deviation are calculated only from the training data. When we call .predict(X_test), the pipeline uses these stored values to transform the test data before feeding it to the classifier. This elegant workflow completely prevents data leakage and makes deploying your model far simpler.
Time to check your understanding of these concepts.
What is data leakage in the context of machine learning?
What is the primary benefit of using a scikit-learn Pipeline?