Amazon MSK Security Essentials for Consultants
Introduction to Streaming Data
Data in Motion
Imagine you're having a text message conversation with a friend. You send a message, they reply. You send another, they reply again. The conversation flows back and forth in a continuous stream of small messages. This is the basic idea behind streaming data.
Streaming Data
noun
Data that is generated continuously by thousands of data sources, which typically send the data records in simultaneously.
Streaming data isn't a file you download or a report you open. It's an endless river of information, sent in tiny pieces, one after another, in real-time. Think of it as data that is always on and always flowing.
Streaming vs. Batch
The opposite of streaming is batch processing. If streaming data is a live conversation, batch processing is like getting a letter in the mail. Instead of a real-time flow, you receive a large chunk of information all at once, after a delay.
Let's use another analogy. Watching a live soccer match is like processing streaming data. You see every goal, every penalty, and every substitution the moment it happens. Reading the final score in the newspaper the next day is like batch processing. You get all the key information, but it's collected and delivered much later.
Most traditional data systems were built for batch processing. For example, a company might run a report every night to calculate the total sales for the day. All the day's transaction data is gathered up and processed in one big batch.
Streaming changes this. Instead of waiting, systems can react instantly as new data arrives.
| Feature | Streaming Processing | Batch Processing |
|---|---|---|
| Data Flow | Continuous stream | Discrete chunks (batches) |
| Data Size | Small packets (kilobytes) | Large volumes (gigs or terabytes) |
| Timing | Real-time (milliseconds) | Delayed (minutes, hours, or days) |
| Analysis | Analyze data as it arrives | Analyze data after it's stored |
Streaming in Action
You interact with streaming data applications every day, often without realizing it. They're built for situations where immediate information is critical.
Ride-sharing apps: The dot representing your driver on the map doesn't just jump from block to block. It moves smoothly because the app is constantly receiving a stream of location data from the driver's phone.
Other examples are all around us:
Financial services: Stock prices on a trading platform update in real-time. Banks also use streaming data to analyze credit card transactions the moment they happen, allowing them to detect and block fraudulent activity instantly.
Social media feeds: When you scroll through your feed, new posts, likes, and comments appear without you needing to refresh the page. That's a live stream of activity from the people you follow.
Online gaming: In a multiplayer game, the actions of every player—moving, jumping, interacting with objects—are streamed to all other players to create a shared, seamless experience.
These applications wouldn't work with batch processing. Imagine if your ride-sharing app only updated the driver's location every ten minutes, or if a bank only checked for fraud at the end of the day. The value comes from processing information the instant it's created.
Instead of waiting for data to accumulate before running transformations, streaming pipelines ingest and process each piece of data as it arrives.
Now that you understand the what and why of streaming data, you're ready to learn about the tools that make it possible. Let's check your understanding first.
Which of the following scenarios is the best analogy for streaming data?
What is the primary difference between streaming data and batch processing?
Understanding the difference between streaming and batch data is the first step in learning about modern data architecture.