Intermediate Python and Practical Design
Pythonic Data Structures
Beyond the Basics
You're already familiar with Python's workhorses: lists and dictionaries. They're great for storing data, but as your programs grow, how you manage that data becomes crucial for both performance and readability. Writing 'Pythonic' code means using the language's features as they were intended, resulting in code that's not just functional, but also clean and efficient.
One of the most powerful Pythonic features is the comprehension. It's a compact way to create a list or dictionary from an existing iterable. Instead of writing a full for loop, you can do it all in one line.
Let's say we want a list of the squares of the first ten numbers. The traditional way involves initializing an empty list and appending to it in a loop.
squares = []
for x in range(10):
squares.append(x**2)
# squares is now [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
A list comprehension achieves the same result in a single, expressive line. The structure is [expression for item in iterable].
squares = [x**2 for x in range(10)]
# same result, much cleaner
You can even add a conditional to filter items. To get the squares of only the even numbers, just add an if clause at the end.
even_squares = [x**2 for x in range(10) if x % 2 == 0]
# even_squares is [0, 4, 16, 36, 64]
Dictionary comprehensions work similarly, using curly braces {} and specifying a key-value pair.
square_dict = {x: x**2 for x in range(5)}
# square_dict is {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}
Sets and Specialized Containers
While lists are ordered and allow duplicates, sets are unordered collections of unique items. This property makes them incredibly fast for checking if an item is present in a collection. For a list, Python has to check each element one by one. For a set, it can do it in a single step, regardless of size.
This efficiency is described using [{}] as (constant time) for sets, versus (linear time) for lists. When working with thousands of items, this difference is enormous.
You can use sets to quickly find unique items or perform mathematical set operations like union, intersection, and difference.
numbers = [1, 2, 2, 3, 4, 4, 4, 5]
unique_numbers = set(numbers)
# unique_numbers is {1, 2, 3, 4, 5}
set_a = {1, 2, 3}
set_b = {3, 4, 5}
# Union: items in either set
print(set_a | set_b) # {1, 2, 3, 4, 5}
# Intersection: items in both sets
print(set_a & set_b) # {3}
Sometimes, the standard list, dictionary, and set aren't quite right. Python's built-in collections module offers more specialized data structures. Two of the most useful are namedtuple and deque.
namedtuple
noun
A factory function for creating tuple subclasses with named fields. It allows you to access items by name instead of just by index, making your code more readable.
from collections import namedtuple
# Define the structure
Point = namedtuple('Point', ['x', 'y'])
# Create an instance
p1 = Point(10, 20)
print(p1.x) # Access by name: 10
print(p1[1]) # Access by index: 20
A deque (pronounced 'deck') stands for double-ended queue. It's like a list, but it's optimized for adding and removing items from both the beginning and the end. With a regular list, removing an item from the beginning is slow because all other elements have to be shifted over.
from collections import deque
line = deque(['Alice', 'Bob', 'Charlie'])
# A new person arrives at the end
line.append('David')
# line is now deque(['Alice', 'Bob', 'Charlie', 'David'])
# The first person is served
line.popleft()
# line is now deque(['Bob', 'Charlie', 'David'])
Memory and Performance
A close relative of the list comprehension is the generator expression. It looks almost identical but uses parentheses instead of square brackets.
# List comprehension (builds the whole list in memory)
list_comp = [x**2 for x in range(10)]
# Generator expression (yields items one by one)
gen_exp = (x**2 for x in range(10))
The difference is crucial. A list comprehension creates the entire list in memory at once. A generator expression creates a 'generator object' that produces the values on demand, one at a time. It doesn't store the whole list.
| Feature | List Comprehension | Generator Expression |
|---|---|---|
| Syntax | [x for x in items] | (x for x in items) |
| Memory | Stores the entire list | Stores one item at a time |
| Use Case | Need the full list immediately | Iterating over a very large sequence |
If you were processing a file with millions of lines, using a list comprehension could exhaust your computer's memory. A generator expression would handle it easily, because it only ever holds one line in memory at a time.
Finally, let's touch on unpacking and slicing. You can assign elements of a list or tuple to multiple variables at once. The * operator can be used to grab remaining items.
numbers = [1, 2, 3, 4, 5]
first, second, *rest = numbers
print(first) # 1
print(second) # 2
print(rest) # [3, 4, 5]
Slicing, which you've likely used to get parts of a list, also has an optional third argument: step. This lets you skip items. A step of -1 is a common trick for reversing a sequence.
letters = ['a', 'b', 'c', 'd', 'e', 'f']
# Get every second letter
print(letters[::2]) # ['a', 'c', 'e']
# Reverse the list
print(letters[::-1]) # ['f', 'e', 'd', 'c', 'b', 'a']
Let's test your knowledge of these Pythonic tools.
Which of these is the most memory-efficient way to process a very large file, line by line?
What is the output of the following code snippet? numbers = [1, 2, 3, 4, 5]; print(numbers[::-2])
Mastering these techniques will make your code more efficient, readable, and professional.