Understanding Data

Data and Information

At its core, data is a collection of raw facts, figures, and observations. Think of a list of numbers: 15, 22, 18, 25. By itself, this list doesn't mean much. It's just data.

But what if we add context? What if these are the daily high temperatures in Celsius for a week in Auckland? Suddenly, the numbers become useful. We've transformed raw data into information. Information is data that has been processed, organised, and given context, making it meaningful.

This distinction is crucial. In any field, the goal is to move from collecting raw data to generating actionable information that can guide decisions.

Structured vs Unstructured Data

Data comes in two main flavours: structured and unstructured. Understanding the difference helps determine how we can work with it.

Structured data is highly organised and follows a predefined model. Think of a spreadsheet or a database table. Each column has a name, and each row contains a specific type of data, like a name, date, or number. It’s neat, tidy, and easy for computers to search and analyse.

Examples of structured data include customer records in a CRM system, sales transactions, or stock inventory levels.

Unstructured data, on the other hand, has no predefined format. It’s the messy, complex stuff that makes up most of the digital world. This includes emails, social media posts, videos, audio files, and documents. While it's rich with insights, it's much harder to process and analyse.

There's also a middle ground called semi-structured data. It isn't stored in a rigid table, but it contains tags or other markers to separate elements and create a hierarchy. Common examples are JSON and XML files, which are often used in web applications.

{
  "customer": {
    "firstName": "Tāne",
    "lastName": "Williams",
    "email": "tane.williams@example.co.nz",
    "orders": [
      { "orderId": "1122", "amount": 99.50 },
      { "orderId": "1125", "amount": 45.00 }
    ]
  }
}

The Data Lifecycle

Data isn't static; it moves through a lifecycle, from its creation to its eventual retirement. While different models exist, most follow a similar path.

Data Collection: This is where data is born. It's gathered from sources like user inputs on a website, sensors in a smart device, business transactions, or public records.
Data Storage: Once collected, data needs a place to live. This could be a database for structured data, a data lake for unstructured data, or a data warehouse for large-scale analytics.
Data Processing: Raw data is often messy. This stage involves cleaning, organising, and transforming it into a usable format for analysis.
Data Use: Here, the data is analysed to find patterns, generate insights, and inform decisions. This is where data creates value, whether it's through business intelligence reports or machine learning models.
Data Archival/Destruction: Not all data is needed forever. At the end of its life, data may be archived for long-term storage (if it needs to be kept for compliance) or securely destroyed.

Why Data Matters for Decisions

The ultimate purpose of collecting and analysing data is to make better, more informed decisions. Relying on intuition or guesswork can be risky, but decisions backed by data are more likely to be successful. This applies everywhere.

A retail company analyses purchasing data to optimise its stock and marketing campaigns. A city council uses traffic data to plan new roads and public transport routes. In healthcare, patient data helps doctors diagnose diseases more accurately and develop effective treatment plans. Even sports teams use player performance data to shape their strategy, a practice popularised by the story of the Oakland Athletics baseball team, famously known as "Moneyball".

Effectively, data provides the evidence needed to move from "I think" to "I know".

By understanding the types of data available and how it moves through its lifecycle, we can begin to harness its power to answer important questions and solve real-world problems.

Ready to check your understanding? Let's test what you've learned.

Quiz Questions 1/5

A list of recent property sale prices in Wellington is considered data. What is required to turn this data into information?

Quiz Questions 2/5

Which of the following is the best example of unstructured data?

Now that you have a solid grasp of what data is, its different forms, and its lifecycle, you're ready to explore how we start working with it.

Data Analysis Fundamentals