Practical Python Engineering
Pythonic Data Structures
Beyond the Basics
You've learned how to store data in lists and dictionaries. Now, let's explore how to work with them more effectively. Writing 'Pythonic' code means using the language's features to create solutions that are not just correct, but also clear, concise, and efficient. It's about choosing the right tool for the job.
Pythonic code leverages the language's unique syntax and paradigms to write readable and efficient programs.
Comprehensions and Generators
One of the hallmarks of Pythonic code is the use of comprehensions. They provide a compact way to create a list, set, or dictionary from an existing iterable. Instead of a multi-line for loop with .append() calls, you can often build a new list in a single, readable line.
# Create a list of squares for even numbers from 0 to 9
squares_of_evens = [x**2 for x in range(10) if x % 2 == 0]
print(squares_of_evens)
# Output: [0, 4, 16, 36, 64]
This same concise syntax works for sets and dictionaries, too. Dictionary comprehensions are particularly useful for transforming or filtering existing dictionaries.
words = ['apple', 'banana', 'cherry']
# Create a dictionary with words as keys and their lengths as values
word_lengths = {word: len(word) for word in words}
print(word_lengths)
# Output: {'apple': 5, 'banana': 6, 'cherry': 6}
List comprehensions are great, but they create the entire list in memory at once. What if you're working with a massive dataset? A generator expression looks almost identical but uses parentheses instead of square brackets. It creates a generator object that produces items one by one, on demand. This approach is far more memory-efficient for large sequences because it doesn't store the whole collection.
import sys
# A list comprehension (uses more memory)
list_comp = [i for i in range(10000)]
print(f"List Comp Size: {sys.getsizeof(list_comp)} bytes")
# A generator expression (uses very little memory)
gen_exp = (i for i in range(10000))
print(f"Generator Exp Size: {sys.getsizeof(gen_exp)} bytes")
# The generator yields values as you iterate over it
# total = sum(gen_exp)
The Collections Module
Python's standard library includes the collections module, a powerhouse of specialized container datatypes that act as high-performance alternatives to general-purpose containers like dict and list. Let's look at a few of the most useful ones.
namedtuple
noun
A factory function for creating tuple subclasses with named fields. It improves code readability by allowing access to elements by name instead of index.
Tuples are fast and memory-efficient, but accessing elements by index (e.g., point[0]) can make code hard to read. A namedtuple solves this by assigning names to each position.
from collections import namedtuple
# Define the structure of our namedtuple
Point = namedtuple('Point', ['x', 'y'])
# Create an instance
p1 = Point(10, 20)
# Access data by name
print(f"The x-coordinate is {p1.x}") # Output: The x-coordinate is 10
# Still works like a regular tuple
print(f"The y-coordinate is {p1[1]}") # Output: The y-coordinate is 20
Another common task is counting items. You might write a for loop and use an if statement to check if a key is already in a dictionary. The defaultdict simplifies this. It's a dictionary subclass that calls a factory function to supply missing values. If you try to access a key that doesn't exist, it automatically creates it with a default value.
from collections import defaultdict
# Create a defaultdict where the default value for a new key is an integer (0)
word_counts = defaultdict(int)
sentence = "the quick brown fox jumps over the lazy dog"
for word in sentence.split():
word_counts[word] += 1 # No need to check if the key exists!
print(word_counts['the']) # Output: 2
print(word_counts['cat']) # Output: 0 (it was created on access)
For an even more direct way to count hashable objects, there's Counter. It's a dictionary subclass where elements are stored as keys and their counts are stored as values. It's purpose-built for tallying.
from collections import Counter
inventory = ['apple', 'orange', 'apple', 'banana', 'orange', 'apple']
# Create a Counter object directly from an iterable
fruit_count = Counter(inventory)
print(fruit_count)
# Output: Counter({'apple': 3, 'orange': 2, 'banana': 1})
# It has useful methods, like finding the most common items
print(fruit_count.most_common(1))
# Output: [('apple', 3)]
Efficiency Matters
Why do we have so many different data structures? Because the right choice can have a massive impact on your code's performance. This is often discussed in terms of 'time complexity' or Big O notation. For example, checking if an item is in a list requires, on average, checking half the items. For a set or dictionary, this operation is incredibly fast and takes roughly the same amount of time, regardless of how many items are in it.
| Operation | List | Set/Dictionary |
|---|---|---|
item in container | ||
container[key] | ||
del container[key] | ||
container.add(item) |
This difference is crucial. If your code frequently checks for the existence of an item in a large collection, using a set instead of a list can make it dramatically faster. Choosing the right data structure isn't just a matter of style; it's fundamental to writing high-performance code.
Let's test your understanding of these Pythonic data structures.
Which of the following code snippets is the most 'Pythonic' way to create a list of the squares of the numbers from 0 to 9?
What is the primary advantage of using a generator expression (i for i in my_list) over a list comprehension [i for i in my_list]?
By mastering these tools, you move from simply storing data to shaping it. You start to think in terms of efficiency, readability, and choosing the most elegant tool for the task at hand.