No history yet

Advanced Data Structures

Wrangling Complex Data

When you work with automation tools or pull data from an API, you rarely get a simple, flat list. Instead, you get a tangle of nested dictionaries and lists. Think of it like a set of Russian dolls, where each doll you open might contain another doll, or maybe a few smaller ones. This is the world of JSON-like structures, and mastering them is key to building powerful workflows.

Let's say we're processing a list of users from a web service. Each user is a dictionary, and that dictionary might contain other dictionaries or lists.

users = [
  {
    "id": 101,
    "name": "Alice",
    "roles": ["admin", "editor"],
    "profile": {
      "email": "alice@example.com",
      "last_login": "2023-10-26"
    }
  },
  {
    "id": 102,
    "name": "Bob",
    "roles": ["viewer"],
    "profile": {
      "email": "bob@example.com",
      "last_login": "2023-10-25"
    }
  }
]

To get Alice's email, you have to dig through the layers. You'd access the first user in the list, then the profile dictionary, and finally the email key.

# Accessing a nested value
alice_email = users[0]["profile"]["email"]
print(alice_email)  # Output: alice@example.com

This pattern of chaining keys and indices is fundamental. It's how you navigate the complex data structures you'll receive from virtually any modern web service.

Efficient Filtering and Transformation

Now, what if you need a list of just the emails for all users? A standard for loop works, but Python offers a more concise and often faster way: list comprehensions and dictionary comprehensions. They let you build a new list or dictionary from an existing one in a single, readable line.

# Using a list comprehension to get all emails
emails = [user["profile"]["email"] for user in users]
# emails is now ['alice@example.com', 'bob@example.com']

# Using a dictionary comprehension to map names to roles
user_roles = {user["name"]: user["roles"] for user in users}
# user_roles is {'Alice': ['admin', 'editor'], 'Bob': ['viewer']}

Comprehensions can also include filtering logic. Suppose we only want the names of users who are admins. We can add an if condition right inside the comprehension.

# Get names of all admins
admins = [user["name"] for user in users if "admin" in user["roles"]]
# admins is now ['Alice']

This is incredibly useful in automation. You can take a large payload from an API and, in one line, extract exactly the pieces of information you need for the next step in your workflow.

Advanced Sorting and Grouping

Sorting a simple list of numbers is easy. But how do you sort our users list by their last login date? The sort() method or sorted() function can take a key argument. This key is a function that tells Python what to sort by. For simple, one-off functions, a is perfect.

# Sort users by the 'last_login' date (string comparison works here)
# The lambda function takes a user dictionary and returns the value to sort by
sorted_users = sorted(users, key=lambda user: user["profile"]["last_login"])

# To sort in reverse order:
sorted_users_desc = sorted(users, key=lambda user: user["profile"]["last_login"], reverse=True)

For more advanced data wrangling, Python's built-in collections module is a treasure trove. Two particularly useful tools for automation are defaultdict and Counter.

collections

noun

A built-in Python module that implements specialized container datatypes, providing alternatives to Python's general-purpose built-in containers like dict, list, set, and tuple.

A defaultdict is like a regular dictionary, but if you try to access a key that doesn't exist, it automatically creates a default value for it instead of raising an error. This is perfect for grouping items. For instance, if we wanted to group a list of tasks by their status ('pending', 'complete'), we could do it without checking if the key exists first.

from collections import defaultdict

tasks = [
    {'name': 'Task 1', 'status': 'complete'},
    {'name': 'Task 2', 'status': 'pending'},
    {'name': 'Task 3', 'status': 'complete'}
]

grouped_tasks = defaultdict(list)

for task in tasks:
    grouped_tasks[task['status']].append(task['name'])

# grouped_tasks is now:
# defaultdict(<class 'list'>, {'complete': ['Task 1', 'Task 3'], 'pending': ['Task 2']})

A Counter is a dictionary subclass for counting hashable objects. It's an incredibly simple way to tally items in a list. Imagine you have a long list of user roles from hundreds of users and want to see the distribution.

from collections import Counter

all_roles = ['admin', 'editor', 'viewer', 'editor', 'viewer', 'viewer']

role_counts = Counter(all_roles)

# role_counts is now:
# Counter({'viewer': 3, 'editor': 2, 'admin': 1})

These tools allow you to take messy, complex data from an API and quickly summarize, group, and prepare it for whatever your automation needs to do next.

Quiz Questions 1/6

Given the following Python data structure:

data = {
    "id": "001",
    "results": [
        {
            "user": "Alice",
            "details": {
                "email": "alice@example.com",
                "status": "active"
            }
        }
    ]
}

Which expression correctly retrieves Alice's email address?

Quiz Questions 2/6

You have a list of user dictionaries and want to create a new list containing only the names of users with the 'admin' role. Which list comprehension correctly accomplishes this?

users = [
    {"name": "Bob", "role": "editor"},
    {"name": "Charlie", "role": "admin"},
    {"name": "Dana", "role": "admin"}
]