Exploratory Data Analysis

Beyond Cleaning Basics

You've handled the basics of data cleaning. Now, we move into Exploratory Data Analysis (EDA), where we diagnose and treat more subtle issues. This is less about fixing typos and more about understanding the structure and integrity of your dataset before you build a model.

Let's start by combining datasets. It's rare to have all the information you need in a single file. More often, you'll need to merge data from different sources. Pandas provides powerful tools for this, far beyond a simple copy-paste. The merge() function is your primary tool, acting like a SQL join to combine DataFrames based on common columns or indices.

import pandas as pd

# DataFrame of customer demographics
df_customers = pd.DataFrame({
    'customer_id': [101, 102, 103],
    'age': [34, 45, 28]
})

# DataFrame of purchase history
df_purchases = pd.DataFrame({
    'customer_id': [101, 101, 103],
    'item': ['Laptop', 'Mouse', 'Keyboard'],
    'price': [1200, 25, 75]
})

# Merge the two DataFrames on the common 'customer_id' column
df_merged = pd.merge(df_customers, df_purchases, on='customer_id', how='inner')

print(df_merged)

The how parameter is key. An 'inner' join keeps only the rows where the key (customer_id) exists in both tables. A 'left' join keeps all rows from the left DataFrame and matching ones from the right. An 'outer' join keeps all rows from both, filling non-matches with NaN (Not a Number). Choosing the right join type is critical for preserving the data you need without introducing errors.

Handling Missing Data

Simply dropping rows with missing values is often too destructive. You might discard valuable information, or worse, introduce bias if the missing data isn't random. A more sophisticated approach is imputation, or filling in missing values using a statistical strategy.

Common strategies include filling with the mean, median, or mode of a column. The choice depends on the data's distribution. For normally distributed data, the mean is often fine. But for skewed data, the median is more robust to outliers. The mode is best for categorical data.

import numpy as np

# Create a DataFrame with missing values
data = {'Score': [88, 92, np.nan, 74, 85, 95, 69, np.nan]}
df = pd.DataFrame(data)

# Impute with the mean
mean_score = df['Score'].mean()
df['Score_mean_imputed'] = df['Score'].fillna(mean_score)

# Impute with the median
median_score = df['Score'].median()
df['Score_median_imputed'] = df['Score'].fillna(median_score)

print(df)

More advanced methods, like K-Nearest Neighbors (KNN) imputation, can be even more accurate. This technique finds the 'k' most similar data points (the neighbors) to the one with the missing value and uses their values to estimate the missing one. This preserves relationships between features, but is computationally more expensive.

Detecting and Managing Outliers

Outliers are data points that differ significantly from other observations. They can be legitimate data points or errors, but they can skew statistical measures and degrade model performance. Two common statistical methods for identifying them are the Z-score and the Interquartile Range ().

The tells us how many standard deviations away a data point is from the mean. A common rule of thumb is to classify any point with a Z-score greater than 3 or less than -3 as an outlier.

Z = \frac{X - \mu}{\sigma}

The IQR method defines outliers as any points that fall below $Q1 - 1.5 \times IQR$ or above $Q3 + 1.5 \times IQR$ , where Q1 is the 25th percentile and Q3 is the 75th. This method is generally preferred for skewed distributions.

Once identified, you can remove outliers, transform them (e.g., using a log transformation to pull them closer to the mean), or impute a new value. The right choice depends on the domain and the cause of the outlier.

Visualizing Relationships

The final step of EDA is to visualize the data to uncover patterns. A correlation matrix is a great starting point for numerical data. It shows the correlation coefficient between every pair of features. A value close to 1 means a strong positive correlation, -1 means a strong negative correlation, and 0 means no linear correlation. Plotting this as a heatmap makes it easy to spot strong relationships at a glance.

import seaborn as sns
import matplotlib.pyplot as plt

# Assuming df_final is our cleaned DataFrame
# df_final = ...

# Calculate the correlation matrix
corr_matrix = df_final.corr()

# Plot the heatmap
plt.figure(figsize=(10, 8))
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm')
plt.title('Correlation Matrix')
plt.show()

Beyond correlations, you need to understand the distribution of individual features. A histogram or a kernel density plot can show if a feature is normally distributed, skewed, or bimodal. This is crucial for selecting the right kind of machine learning model. Many models assume that your data follows a specific distribution, and violating that assumption can lead to poor results.

Using libraries like Matplotlib and Seaborn, you can create these plots with just a few lines of code, turning raw numbers into actionable insights. This visual inspection is where a data scientist's intuition really shines, bridging the gap between clean data and a powerful predictive model.

Quiz Questions 1/6

You are merging a customers DataFrame (left) with an orders DataFrame (right). You want a final table that includes every customer, with their order details populated if they exist and NaN otherwise. Which merge strategy should you use?

Quiz Questions 2/6

When cleaning a dataset of house prices, you find that the distribution is right-skewed (a few mansions are pulling the average up). Which imputation method is generally the most robust for filling in missing price values in this scenario?

Data Science and Machine Learning Foundations