Intermediate Python and Practical Application
Advanced Data Structures
Beyond Lists and Dictionaries
You already know that lists and dictionaries are the workhorses of Python. They're flexible and powerful, but for more specialized tasks, using them can lead to code that's either slow or hard to read. Professional Python code often needs more specialized tools.
Enter Python's collections module. It's like a professional toolkit full of high-performance data structures that solve common problems more elegantly and efficiently than the basics. Think of it as upgrading from a standard screwdriver to a full power-tool set.
Readable Records with namedtuple
Imagine you're processing data about cars. You could use a plain tuple: ('Toyota', 'Camry', 2021, 'Blue'). This is memory-efficient, but what does car[2] mean? It's not immediately obvious. You have to remember the order of the fields.
A dictionary like {'make': 'Toyota', ...} is more readable, but it uses more memory. For thousands or millions of these objects, that adds up.
This is where namedtuple comes in. It gives you the best of both worlds: the memory efficiency of a tuple with the readability of object-style attribute access.
from collections import namedtuple
# Define the structure of our 'Car' record
Car = namedtuple('Car', ['make', 'model', 'year', 'color'])
# Create an instance
my_car = Car(make='Toyota', model='Camry', year=2021, color='Blue')
# Access data by name - much clearer!
print(f"My car is a {my_car.year} {my_car.make} {my_car.model}.")
# >> My car is a 2021 Toyota Camry.
# You can still access by index, like a regular tuple
print(f"The model is: {my_car[1]}")
# >> The model is: Camry
By defining a simple Car structure, your code becomes self-documenting. Anyone reading my_car.year instantly knows what data they're working with. It’s a simple change that makes your code cleaner and more professional without the overhead of defining a full class.
High-Speed Queues
Queues are a fundamental concept in computer science, following a "first-in, first-out" (FIFO) pattern. A common mistake is to use a Python list for this. While list.append() is fast, removing an item from the front with list.pop(0) is slow. To remove the first element, Python has to shift every other element one position to the left. For a long list, this is a significant performance bottleneck.
The collections module provides deque (pronounced "deck"), which stands for double-ended queue. It's specifically designed for fast appends and pops from both ends. Both append() and popleft() operations on a deque have an average time complexity of O(1), making them constant time regardless of the deque's size.
from collections import deque
# Create a deque
printer_queue = deque(['job1.pdf', 'job2.png', 'job3.docx'])
# Add a new job to the end of the queue (fast)
printer_queue.append('job4.jpg')
# Process the first job in the queue (also fast)
current_job = printer_queue.popleft()
print(f"Processing: {current_job}")
print(f"Jobs remaining: {list(printer_queue)}")
# >> Processing: job1.pdf
# >> Jobs remaining: ['job2.png', 'job3.docx', 'job4.jpg']
What if order is based on priority, not just arrival time? For this, we need a priority queue. The heapq module helps us build one. It uses a standard Python list but cleverly maintains a "heap" property, which ensures the smallest item is always at the first position (heap[0]).
import heapq
# Items are tuples of (priority, task_name)
# Lower numbers have higher priority.
tasks = []
heapq.heappush(tasks, (5, 'Answer non-urgent emails'))
heapq.heappush(tasks, (1, 'Fix critical server bug'))
heapq.heappush(tasks, (3, 'Write weekly report'))
# The list itself may not look sorted
# print(tasks) gives: [(1, 'Fix critical server bug'), (5, 'Answer non-urgent emails'), (3, 'Write weekly report')]
# But heappop will always return the smallest item
next_task = heapq.heappop(tasks)
print(f"Highest priority task: {next_task[1]}")
# >> Highest priority task: Fix critical server bug
Using heapq is perfect for situations like scheduling tasks, finding the top 'N' items in a collection without fully sorting it, or implementing certain graph algorithms. Pushing and popping from the heap are both efficient O(log n) operations.
Smarter Counting
How often have you written code to count item frequencies? It usually looks something like this: create an empty dictionary, loop through your data, and use an if/else block to check if a key exists before incrementing its count.
It works, but it's verbose. collections.Counter makes this task trivial. It's a dictionary subclass designed specifically for counting hashable objects.
from collections import Counter
votes = ['red', 'blue', 'red', 'green', 'blue', 'red']
vote_counts = Counter(votes)
print(vote_counts)
# >> Counter({'red': 3, 'blue': 2, 'green': 1})
# It has useful methods, like finding the most common items
print(vote_counts.most_common(1))
# >> [('red', 3)]
# You can also access counts for items that weren't there
print(f"Votes for yellow: {vote_counts['yellow']}")
# >> Votes for yellow: 0
Notice that asking for a non-existent key doesn't raise a KeyError; it just returns 0. This convenient behavior is also available in another tool, defaultdict. A defaultdict is like a regular dictionary, but if you try to access a key that doesn't exist, it automatically creates a default value for it instead of erroring out. You specify the type of this default value when you create it.
from collections import defaultdict
# Create a defaultdict where the default value is an empty list
student_grades = defaultdict(list)
student_grades['Alice'].append(95)
student_grades['Bob'].append(88)
student_grades['Alice'].append(92)
print(student_grades)
# >> defaultdict(<class 'list'>, {'Alice': [95, 92], 'Bob': [88]})
# No need to check if 'Charlie' is already a key
print(student_grades['Charlie'])
# >> []
Using defaultdict(int) is a clean way to write a frequency counter yourself, while defaultdict(list) is perfect for grouping items into lists. It simplifies your code by removing the need for boilerplate key-checking logic.
You are building an application that processes thousands of car records. Each record has a make, model, year, and colour. For memory efficiency and code readability (e.g., accessing my_car.year), which data structure from the collections module is the most suitable choice?
A developer uses a standard Python list to implement a queue where items are added to the end and removed from the beginning. Why is this approach inefficient for a very long list?