Python Web Scraping with Selenium
Python for Scraping
Organizing Your Scraped Data
Web scraping is all about gathering data. But once you have it, where do you put it? Shoving everything into a single variable won't work. You need structured containers to hold the information you pull from websites. The two workhorses for this in Python are lists and dictionaries.
Lists are perfect for storing a sequence of similar items. Imagine you're scraping the titles of the top five articles from a news site. A list is the natural way to store them.
article_titles = [
"Tech Giant Unveils New Gadget",
"Global Markets React to Economic News",
"Breakthrough in Medical Research",
"Local Sports Team Wins Championship",
"Guide to Weekend Travel Destinations"
]
Dictionaries, on the other hand, are for storing related pieces of information about a single item. Instead of just the title, what if you wanted to scrape the title, author, and publication date for one article? A dictionary uses key-value pairs to keep this data organized and meaningful.
article_details = {
"title": "Tech Giant Unveils New Gadget",
"author": "Jane Doe",
"publication_date": "2023-10-27"
}
Often, you'll combine these. A common pattern is a list of dictionaries, where each dictionary represents a complex object you've scraped, like a product on an e-commerce site.
products = [
{
"name": "Smart Watch",
"price": 299.99,
"in_stock": True
},
{
"name": "Wireless Headphones",
"price": 149.50,
"in_stock": False
}
]
Processing Data with Loops and Logic
Scraping isn't a one-and-done action. You typically need to perform the same extraction task on many elements on a page. This is where loops come in. The for loop is essential for iterating over a collection of web elements (which you'll soon see are often treated like lists) and pulling data from each one.
Let's say you have a list of raw product data. You can loop through it to process each item.
# Assume 'products' is the list of dictionaries from the previous example
for product in products:
print(f"Processing: {product['name']}")
# In a real scraper, you'd do more here
But what if the data isn't uniform? Some products might be on sale, while others are out of stock. You need conditional logic to handle these differences. An if/else statement lets your scraper make decisions. For instance, you could decide to only process products that are currently in stock.
for product in products:
if product['in_stock'] == True:
print(f"{product['name']} is available for 💲{product['price']}.")
else:
print(f"{product['name']} is currently out of stock.")
Cleaning and Storing Your Findings
Data extracted from websites is rarely clean. You'll often find unwanted whitespace, currency symbols, or other text that needs to be removed before the data is useful. Python's built-in string manipulation methods are your tools for this job.
Let's say you scrape a price that comes out as a string like " \$59.99 ". You can't perform math on it. You need to clean it up first.
| Method | Description | Example |
|---|---|---|
.strip() | Removes leading/trailing whitespace. | " hello ".strip() → "hello" |
.replace(old, new) | Replaces a substring with another. | "\$59.99".replace("\$", "") → "59.99" |
.lower() | Converts the string to lowercase. | "Product".lower() → "product" |
.split(delimiter) | Splits a string into a list. | "a,b,c".split(",") → ['a', 'b', 'c'] |
Once your data is structured and clean, you need to save it. Writing to a file is a fundamental skill for any scraper. The simplest way is to write to a plain text file. For more structured data, a [{
Python's with open(...) syntax is the standard for [{<file I/O>}] because it automatically handles closing the file for you, even if errors occur.
# Using the same 'products' list
import csv
# Define the headers for our CSV file
headers = ['name', 'price', 'in_stock']
# 'w' means write mode, newline='' prevents extra blank rows
with open('products.csv', 'w', newline='') as file:
writer = csv.DictWriter(file, fieldnames=headers)
writer.writeheader() # Write the column titles
writer.writerows(products) # Write all the product data
print("Data saved to products.csv")
This code creates a file named products.csv with the data from your list of dictionaries, neatly organized into columns. You've now successfully collected, structured, cleaned, and saved data—the core workflow of any web scraping project.
Now that you've refreshed these core Python skills, let's test your understanding.
You are scraping a news website and want to store the title, author, and publication date for each of the top 10 articles. Which combination of data structures is most suitable for this task?
When scraping many items from a webpage, such as all the product links from a search results page, which Python construct is essential for processing each item one by one?
With this foundation, you're ready to start interacting with actual websites.