Advanced Data Structures

Beyond the Basics

You already know how to store data in Python's basic collections. Now, it's time to use them more effectively. The goal isn't just to store data, but to do it in a way that makes your code faster, cleaner, and more readable, especially when dealing with the large datasets common in AI and data processing.

Let's start with a common task: creating a new list based on an existing one. The straightforward way involves a for loop, but there's a more concise and often faster method.

List Comprehensions

List comprehensions are a compact way to create lists. Their syntax can feel like a condensed for loop written inside square brackets. They are not just shorter; they are often more performant because the looping happens at a lower level in Python's C implementation.

# Get the squares of all even numbers from 0 to 9

# The traditional way
squares = []
for i in range(10):
    if i % 2 == 0:
        squares.append(i * i)

# The comprehension way
squares_comp = [i * i for i in range(10) if i % 2 == 0]

# squares and squares_comp are now the same: [0, 4, 16, 36, 64]

The structure is [expression for item in iterable if condition]. The if condition part is optional but powerful for filtering.

This technique extends to nested lists, which are common when working with matrices or structured data like an image's pixel grid. You can use nested comprehensions to transform this data efficiently.

# Flatten a matrix (a list of lists)
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]

flat_list = [num for row in matrix for num in row]

# flat_list is now [1, 2, 3, 4, 5, 6, 7, 8, 9]

Read nested comprehensions from left to right, just like you would read nested for loops. The leftmost for is the outer loop.

Smarter Dictionaries

Dictionaries are indispensable for mapping keys to values, like user IDs to user data. A common problem arises when you try to access a key that doesn't exist, which raises a KeyError.

You can handle this with .get() or a try/except block, but for tasks like counting items or grouping elements into lists, there's a better tool: defaultdict.

from collections import defaultdict

# Group words by their first letter
words = ['apple', 'ant', 'ball', 'bat', 'cat']

# Using a regular dict
by_letter = {}
for word in words:
    letter = word[0]
    if letter not in by_letter:
        by_letter[letter] = []
    by_letter[letter].append(word)

# Using defaultdict
by_letter_dd = defaultdict(list)
for word in words:
    by_letter_dd[word[0]].append(word)

# Both produce the same result, but defaultdict is cleaner:
# {'a': ['apple', 'ant'], 'b': ['ball', 'bat'], 'c': ['cat']}

When you create a defaultdict, you provide it with a factory function (like list, int, or set). If you try to access a key that doesn't exist, the defaultdict automatically calls this function to create a default value for that key, adds it to the dictionary, and returns it. This eliminates the need for manual checks.

Sets and Tuples

Sets and tuples are often underutilized but offer significant performance and clarity benefits when used correctly.

Use a set when you need to store a collection of unique items and perform membership testing (checking if an item is in the collection) very quickly. Checking for an item in a set has an average time complexity of $O(1)$ , compared to $O(n)$ for a list.

This makes sets ideal for deduplication and filtering. For example, if you have a list of user IDs from an API response and want to find the unique ones:

user_ids = [101, 102, 105, 102, 108, 101, 109]

unique_ids = set(user_ids)

# unique_ids is now {101, 102, 105, 108, 109}

Sets also support powerful mathematical operations like union, intersection, and difference, which are perfect for comparing two collections of data.

admins = {101, 105, 204}
moderators = {102, 105, 301}

# Users who are admins OR moderators (union)
print(admins | moderators)  # {101, 102, 105, 204, 301}

# Users who are both admins AND moderators (intersection)
print(admins & moderators)  # {105}

# Users who are admins but NOT moderators (difference)
print(admins - moderators)  # {101, 204}

Tuples, on the other hand, are immutable lists. This means they cannot be changed after creation. While that might seem restrictive, it makes them perfect for data that shouldn't change, like coordinates or RGB color values.

One of their best features is unpacking, which allows you to assign elements of a tuple to multiple variables at once.

point = (10, 25)
x, y = point  # Tuple unpacking

print(f"The x coordinate is {x}") # The x coordinate is 10

For more complex tuples, accessing elements by index (point[0]) can make code hard to read. The collections module offers namedtuple to solve this. It gives you tuples where you can access values by name, combining the readability of an object with the efficiency of a tuple.

from collections import namedtuple

# Define the structure of our namedtuple
Point = namedtuple('Point', ['x', 'y'])

# Create an instance
p = Point(x=10, y=25)

# Access by name or index
print(p.x)       # 10
print(p[1])      # 25

Performance Trade-offs

Choosing the right data structure is a balance between memory usage, speed, and code readability. There's no single "best" option; the right choice depends on your specific problem.

Data Structure	Key Strength	Common Use Case
List	Ordered, mutable collection.	Storing a sequence of items.
Tuple	Ordered, immutable collection.	Protecting data that shouldn't change.
Set	Unordered, unique items.	Membership testing, removing duplicates.
Dictionary	Key-value pairs.	Fast lookups, mapping data.

For example, checking if an item exists in a large list is slow. If you find yourself writing if item in my_list: where my_list is very long, consider converting it to a set first to speed up the check significantly. This simple change can make a huge difference in performance-critical data pipelines.

Time to test your knowledge on these advanced data structures.

Quiz Questions 1/5

What is the primary advantage of using a list comprehension like [x*x for x in numbers] over a traditional for loop to create a new list?

Quiz Questions 2/5

You are processing a large list of words to count the frequency of each word. You try to use a dictionary, but your code crashes with a KeyError when you first encounter a new word. Which of the following is the most elegant solution to this problem?

By mastering these tools, you move from simply writing code that works to writing code that is efficient, elegant, and professional.

Intermediate Python Mastery and Application