Practical Machine Learning and Model Optimization
Advanced Data Preprocessing
Smarter Ways to Fill in the Gaps
You already know how to fill missing values with the mean or median. It's a quick fix, but it can distort the natural spread of your data. When you replace every blank with the same number, you reduce the feature's variance, which can weaken your model's ability to see important patterns.
Let's explore more sophisticated methods that preserve the underlying structure of your dataset.
One such method is the K-Nearest Neighbors (KNN) imputer. Instead of using a single global value, it looks at the local neighborhood of the missing point. For a missing value, it finds the 'k' most similar rows based on the other features. It then fills the gap with the average (or mode) of those neighbors. This approach is powerful because the imputed value is tailored to the specific context of that data point.
from sklearn.impute import KNNImputer
import numpy as np
# Sample data with a missing value
X = [[1, 2, np.nan], [3, 4, 3], [np.nan, 6, 5], [8, 8, 7]]
# Initialize the imputer to look at 2 nearest neighbors
imputer = KNNImputer(n_neighbors=2)
# Fill the missing values
X_filled = imputer.fit_transform(X)
print(X_filled)
# Output might be:
# [[1. 2. 4. ]
# [3. 4. 3. ]
# [5.5 6. 5. ]
# [8. 8. 7. ]]
Another powerful technique is the Iterative Imputer. This method takes a more model-based approach. It treats each feature with missing values as a target variable, y, and all other features as predictors, X. It then trains a regression model to predict the missing values in y based on X.
This process is done iteratively. The imputer cycles through all features with missing values, predicting and filling them in. It repeats this cycle several times, with each iteration using the newly imputed values to improve the predictions in the next round. This continues until the imputed values stabilize, giving you a more robust and internally consistent dataset.
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer
import numpy as np
# Sample data with missing values
X = [[1, 2, np.nan], [3, 4, 3], [np.nan, 6, 5], [8, 8, 7]]
# Initialize the iterative imputer
imputer = IterativeImputer(max_iter=10, random_state=0)
# Fill the missing values
X_filled = imputer.fit_transform(X)
print(X_filled)
# Output might be:
# [[ 1. 2. 3.99...]
# [ 3. 4. 3. ]
# [ 5.5... 6. 5. ]
# [ 8. 8. 7. ]]
Balancing the Scales
Real-world datasets are often imbalanced. Think of fraud detection, where fraudulent transactions are rare, or medical diagnoses for an uncommon disease. If 99% of your data belongs to one class, a lazy model can achieve 99% accuracy by simply always predicting that majority class. This makes the model useless for its intended purpose.
Simply duplicating the minority class samples (over-sampling) can lead to overfitting. We need a way to generate new, believable minority samples.
SMOTE
noun
Synthetic Minority Over-sampling Technique. An algorithm used to address class imbalance by generating new, synthetic data points for the minority class.
SMOTE works by creating synthetic samples instead of just copying existing ones. For each minority class sample, it finds its k-nearest minority neighbors. It then creates a new synthetic sample somewhere along the line segment connecting the original sample and one of its randomly chosen neighbors. This populates the decision boundary with new, plausible examples, helping the model learn the distinction between classes.
An even smarter variant is ADASYN (Adaptive Synthetic Sampling). ADASYN is similar to SMOTE, but it puts more focus on the minority samples that are harder to learn. It identifies minority points that have a higher ratio of majority-class neighbors. These are the points most likely to be misclassified. ADASYN then generates more synthetic data around these difficult points, adaptively shifting the decision boundary where it's needed most.
# Assumes you have X_train, y_train from your dataset
# And that the 'imblearn' library is installed (pip install imbalanced-learn)
from imblearn.over_sampling import SMOTE, ADASYN
# Using SMOTE
smote = SMOTE(random_state=42)
X_smote, y_smote = smote.fit_resample(X_train, y_train)
# Using ADASYN
adasyn = ADASYN(random_state=42)
X_adasyn, y_adasyn = adasyn.fit_resample(X_train, y_train)
# Now, train your model on the resampled data (e.g., X_smote, y_smote)
Encoding with Meaning
What do you do with categorical features that have thousands of unique values, like zip codes, product IDs, or user names? This is known as the high-cardinality problem. One-hot encoding would create thousands of new columns, making your model slow and unwieldy.
Target encoding offers a clever solution. It replaces each category with the average of the target variable for that category. For example, if you're predicting customer churn, the 'San Francisco' category in a 'City' column would be replaced by the average churn rate of all customers from San Francisco. This captures a direct relationship between the category and what you're trying to predict, all within a single feature.
A word of caution: target encoding can lead to overfitting. If a category has very few samples, its encoded value will be heavily influenced by noise. To prevent this, it's common to use a smoothing technique, which blends the category's average with the overall global average of the target.
Taming Skewed Data
Many machine learning models, especially linear models and neural networks, perform best when the numerical features have a normal (Gaussian) distribution. Real-world data, however, is often skewed. For instance, data on income or house prices typically has a long tail of very high values.
Power transforms are a family of functions that can make your data more Gaussian-like. They stabilize variance and minimize skewness.
The Box-Cox transform is a popular choice, but it has one major limitation: it only works on strictly positive data. The Yeo-Johnson transform is a more flexible alternative that works with both positive and negative values, making it a more general-purpose tool.
from sklearn.preprocessing import PowerTransformer
import numpy as np
# Create some skewed data (e.g., log-normal distribution)
skewed_data = np.random.lognormal(size=(100, 1))
# Apply the Yeo-Johnson transform
pt = PowerTransformer(method='yeo-johnson', standardize=True)
transformed_data = pt.fit_transform(skewed_data)
# The transformed_data will now have a distribution
# that is much closer to a standard normal distribution.
By applying these advanced preprocessing steps, you move beyond just cleaning your data. You are actively engineering your features to give your model the best possible chance to learn and generalize from complex, messy, real-world information.
How does the Iterative Imputer handle missing values in a dataset?
In the context of an imbalanced dataset for fraud detection, why can a model with 99% accuracy still be considered useless?