Artificial intelligence models, especially those that work with language or images, face a fundamental problem: they operate on mathematics, not meaning. A computer doesn't understand the concept of a "cat" from a photo or the sentiment in a sentence. It only understands numbers. The first, critical step in any AI workflow is translating our messy, unstructured world into the clean, numerical language of a machine. This process is called vectorisation.

Think of it as creating a vast, multidimensional map of concepts. On this map, every word, sentence, or even image is assigned a specific coordinate, or vector. This isn't random. The coordinates are carefully calculated so that items with similar meanings are placed close together. For example, the vectors for "king" and "queen" would be neighbours in this space, as would "walking" and "running." This conceptual map, known as an embedding space, is the foundation upon which models learn to identify patterns and relationships that would be impossible to grasp from raw text or pixels.

Before text can be converted into vectors, it must be broken down into smaller, manageable pieces. This process is called tokenisation. A "token" can be a word, a character, or something in between. The strategy you choose has significant consequences.

Word-based tokenisation is simple: "the quick brown fox" becomes four tokens: ["the", "quick", "brown", "fox"]. But this approach struggles with new words, misspellings, and languages that don't use spaces. Character-based tokenisation solves the new-word problem by splitting everything into individual letters, but it loses the inherent meaning of words and creates very long sequences.

The modern solution for Large Language Models (LLMs) is subword tokenisation. It breaks words into common sub-units. For example, "tokenisation" might become ["token", "isation"]. This handles new words gracefully (a new word can be built from known subwords) while preserving more meaning than character-level tokens. A popular algorithm for this is Byte-Pair Encoding (BPE).

Once we have our numerical vectors, there's one last step before feeding them to a model: normalisation. The different features within our data often exist on vastly different scales. One feature might range from 0 to 1, while another ranges from 10,000 to 500,000. Many machine learning algorithms are sensitive to this and can give undue weight to features with larger values, not because they are more important, but simply because their numbers are bigger.

Feature scaling and normalisation are techniques used to put all features on a common scale. Two common methods are Min-Max Scaling, which rescales the data to a fixed range (usually 0 to 1), and Standardisation, or Z-score normalisation, which rescales data to have a mean of 0 and a standard deviation of 1. This ensures that the model learns from the actual contribution of each feature, not its arbitrary scale.

Choosing the right dimensionality for your vectors is a balancing act. Too few dimensions, and you can't capture the complex relationships between your data points—this is called underfitting. Too many dimensions, and you increase computational cost and run the risk of the model memorising the training data instead of learning general patterns, a problem known as overfitting. The art of data ingestion lies in finding the sweet spot that preserves meaning without unnecessary complexity.

Let's check your understanding of these initial steps in the AI data workflow.

This transformation from raw data to clean, normalised vectors is the unseen but vital foundation for everything that follows in an AI system.