DataFrame Operations

Building Your Data Table

You already know that a DataFrame is the workhorse of pandas, essentially a table for holding your data. But how do you actually create one from scratch? One of the most common ways is to start with a Python dictionary.

Imagine a dictionary where each key is a column header and each value is a list of the data for that column. Pandas can transform this directly into a DataFrame.

import pandas as pd

# Data in a dictionary of lists
data = {
    'Player': ['Maria', 'Leo', 'Chen', 'Amir'],
    'Points': [24, 31, 19, 28],
    'Team': ['Dragons', 'Wizards', 'Dragons', 'Knights']
}

# Create the DataFrame
df = pd.DataFrame(data)

print(df)

The keys (Player, Points, Team) become the column headers. The lists become the data in those columns. Notice the numbers on the far left (0, 1, 2, 3). This is the DataFrame's index, an automatic label pandas assigns to each row.

Another method is using a list of dictionaries, where each dictionary represents a single row.

# Data in a list of dictionaries
data_list = [
    {'Player': 'Maria', 'Points': 24, 'Team': 'Dragons'},
    {'Player': 'Leo', 'Points': 31, 'Team': 'Wizards'},
    {'Player': 'Chen', 'Points': 19, 'Team': 'Dragons'},
    {'Player': 'Amir', 'Points': 28, 'Team': 'Knights'}
]

# Create the DataFrame
df_from_list = pd.DataFrame(data_list)

print(df_from_list)

The result is identical. This approach is often useful when your data comes from sources like JSON files, which are structured similarly.

Selecting and Slicing

Once you have a DataFrame, you'll need to grab specific pieces of it. This is called indexing or selecting. The simplest way is to select a column by its name, using square brackets.

# Select the 'Player' column
players = df['Player']

print(players)

This returns a pandas Series, which is like a single column of the DataFrame. To select multiple columns, you pass a list of column names inside the brackets.

# Select the 'Player' and 'Team' columns
player_teams = df[['Player', 'Team']]

print(player_teams)

Notice the double square brackets [[]]. The outer brackets are for selecting, and the inner brackets create the list of columns you want. This operation returns a new, smaller DataFrame.

For more precise selections of rows and columns, pandas provides two powerful tools: .loc and .iloc.

.loc selects data by its label or index name.
.iloc selects data by its integer position.

Let's use .loc to grab the first row. Since our index labels are currently 0, 1, 2, and 3, we'll use the label 0.

# Select the first row by its label
first_row = df.loc[0]

print(first_row)

To select a specific cell, you provide both the row and column labels. Here's how to get the points for the player at index 2.

# Get the 'Points' for the row with label 2
chen_points = df.loc[2, 'Points']

print(chen_points)

Now, let's use .iloc. It works the same way but always uses integer positions, starting from 0. To get the third row (position 2):

# Select the third row by its integer position
third_row = df.iloc[2]

print(third_row)

You can also slice data, grabbing a range of rows or columns. This works just like slicing a Python list. To get the first two rows:

# Select the first two rows using a slice
first_two = df.iloc[0:2]

print(first_two)

Remember, the slice 0:2 includes the element at position 0 but stops before the element at position 2.

Filtering with Conditions

Perhaps the most powerful selection method is filtering based on a condition. This lets you pull out all the rows that meet certain criteria. The process involves creating a series of True and False values and then using it to filter the DataFrame.

First, let's create the condition. For example, which players scored more than 25 points?

# Create a boolean Series
# True for rows where 'Points' > 25, False otherwise
condition = df['Points'] > 25

print(condition)

This gives us a Series of booleans. Now, we can pass this series back into our DataFrame to select only the rows where the value is True.

# Use the boolean Series to filter the DataFrame
high_scorers = df[condition]

print(high_scorers)

Pandas returns a new DataFrame containing only the rows for Leo and Amir, who both scored more than 25 points. This is a fundamental technique for exploring and cleaning data.

You can combine multiple conditions using & for "and" and | for "or". When you do this, each condition must be wrapped in parentheses. Let's find players on the 'Dragons' team who scored more than 20 points.

# Combine two conditions
dragons_high_scorers = df[(df['Team'] == 'Dragons') & (df['Points'] > 20)]

print(dragons_high_scorers)

This isolates just the single row for Maria, who meets both criteria.

Modifying Data

Changing values in a DataFrame uses the same selection techniques. You just select the data you want to change and assign a new value. To change Chen's point total to 21, we can use .loc to find that specific cell.

# Change a single value
df.loc[2, 'Points'] = 21

print(df)

You can also add new columns. Let's add a 'Jersey' number for each player. We can do this by assigning a list or Series to a new column name.

# Add a new 'Jersey' column
df['Jersey'] = [10, 7, 8, 11]

print(df)

If you assign a single value to a new column, pandas will apply that value to every single row, which is useful for setting default values.

# Add a 'Status' column with a default value
df['Status'] = 'Active'

print(df)

These building blocks, creating, selecting, filtering, and modifying, form the foundation of almost all data analysis work you'll do with pandas.

Quiz Questions 1/5

Given a dictionary data = {'name': ['Alice', 'Bob'], 'score': [95, 88]}, which line of code correctly creates a pandas DataFrame?

Quiz Questions 2/5

What is the fundamental difference between using .loc and .iloc for selecting data from a DataFrame?

Pandas Essentials