Raw data is rarely perfect. It arrives with missing pieces, strange entries, and inconsistencies. Attempting to analyse it in this state is like building a house on a faulty foundation; the final structure will be unreliable, no matter how sophisticated your building techniques are. This is where data cleaning comes in. It's the essential process of tidying up your dataset to ensure your analysis is accurate and your conclusions are sound.

It’s often the most time-consuming part of a data project, but skipping it can invalidate your entire effort. A clean, reliable dataset is the prerequisite for any meaningful insight.

One of the most common issues you'll face is missing data. Values can be absent for many reasons, from data entry errors to a user intentionally skipping an optional field in a form. How you handle these gaps depends on the context and the amount of missing data.

An alternative to deletion is imputation, which means filling in the missing values. The method you choose can have a big impact.

• Mean/Median Imputation: For numerical data, you can replace missing values with the mean or median of the entire column. The median is generally preferred because it's less sensitive to outliers.
• Mode Imputation: For categorical data, the best strategy is often to use the mode, which is the most frequently occurring value in the column.

More advanced techniques, like regression imputation or using algorithms like K-Nearest Neighbours (KNN), can predict the missing values based on other data points, but these add complexity.

Outliers are data points that are unusually far from the other values in a dataset. They might be genuine, extreme values (like the salary of a CEO in a company-wide dataset) or they could be errors from measurement or data entry. Because they can skew statistical measures like the mean and standard deviation, it's crucial to identify and address them.

Visualisation, especially with a box plot, is a great first step for spotting outliers. Statistically, the Interquartile Range (IQR) method is robust. Any point that falls below $Q1 - 1.5 \times IQR$ or above $Q3 + 1.5 \times IQR$ is considered an outlier.

Once identified, you can remove outliers, but do so with caution. Removing them might mean throwing away valuable information. Alternatives include transforming the data with a function like a logarithm to pull in extreme values, or capping (also called winsorizing), where you replace outliers with the nearest value that isn't an outlier (e.g., the 95th percentile value).

Data inconsistencies are often the trickiest to solve because they require domain knowledge. These errors can hide in plain sight.

Imagine a dataset of customer information. You might find a single customer, Jane Smith, entered in multiple ways:

To an algorithm, these are four different people. Standardisation is the key to fixing this. This involves enforcing consistent formats, such as converting all text to lowercase, trimming leading and trailing white space, and establishing a uniform date format (e.g., YYYY-MM-DD).

Similarly, check for contradictions. A customer listed as being 15 years old shouldn't have a record of purchasing alcohol. A dataset about cities should have consistent country names—not 'USA', 'U.S.', and 'United States'. Creating a data dictionary that defines what each field means and what its valid entries are can prevent many of these issues.

Ultimately, data cleaning is an iterative process. You might clean the data, begin your analysis, discover more issues, and loop back to cleaning again. It's a foundational skill that directly impacts the quality and reliability of any data-driven conclusion.