An AI model doesn't understand raw data any more than a car engine understands crude oil. Before a model can learn from text, images, or spreadsheets, that data must be refined into a clean, consistent, and numerical format. This transformation happens in a data pipeline, a series of automated steps that acts like an assembly line, preparing raw material for the complex machinery of a neural network.

This pipeline begins with ingestion, the process of pulling data from its various sources, whether that's a live feed from an API, a massive database, or a folder of user-uploaded images. Each source has its own quirks and formats, so the first challenge is simply to get all the data into one place in a standardised way.

Real-world data is inherently messy. Text contains typos and inconsistent capitalisation. Images come in different sizes and resolutions. Tabular data often has missing values. The cleaning stage addresses these issues. It involves tasks like removing duplicate entries, correcting obvious errors, and handling missing data points through deletion or imputation—the process of filling in missing values based on other available data.

Normalisation follows, forcing the data into a standard format. For images, this might mean resizing every picture to 256x256 pixels. For text, it usually involves converting everything to lowercase. For numerical data, it means scaling all values to a specific range, like 0 to 1, which prevents features with large values from disproportionately influencing the model's training process.

Once text is clean and normalised, it must be converted into numbers. This process starts with tokenisation: breaking down sentences into smaller units called tokens. The simplest approach is to split by spaces, so "The quick brown fox" becomes ["The", "quick", "brown", "fox"]. But this method struggles with punctuation and compound words.

A more robust solution is subword tokenisation. Instead of full words, the text is broken into common pieces. For example, "tokenisation" might become ["token", "##isation"]. This approach is incredibly powerful. It allows the model to understand new words by recognising their components (like prefixes and suffixes) and it keeps the vocabulary size manageable.

Two popular subword algorithms are Byte-Pair Encoding (BPE) and WordPiece. They operate on similar principles but with a subtle difference in how they decide which subwords to create.

After tokenisation, each unique token is assigned an integer index. This is called vocabulary mapping. The sentence "the cat sat" might become [5, 12, 88]. This list of numbers is the final, model-ready format.

But what happens if the model encounters a word during use that wasn't in its training vocabulary? This is the out-of-vocabulary (OOV) problem. Subword tokenisation helps immensely, as a rare word like "hydrogeologist" can be broken down into known parts: ["hydro", "##geo", "##logist"]. For words that truly can't be represented, a special <UNK> (unknown) token is used. This signals to the model that it has encountered something entirely new.