The first practical step in any natural language processing (NLP) pipeline is preparing the raw text. Human language is messy, filled with capital letters, punctuation, and common filler words. To turn this text into a numerical format like a Bag of Words vector, we first need to break it down into individual units, or tokens. This process is called tokenisation.

But it's not as simple as splitting a sentence by its spaces. The choices you make during tokenisation and the subsequent cleaning steps, known as normalisation, directly impact the size of your vocabulary and the performance of your machine learning model. A well-designed preprocessing pipeline creates a clean, consistent set of tokens that captures the essential meaning of the text without unnecessary noise.

Normalisation aims to standardise your tokens so that superficial differences don't create unique entries in your vocabulary. The most common technique is case folding, which involves converting all text to a single case, usually lowercase. This ensures that “Apple” (the company) and “apple” (the fruit) are treated as the same token initially. While this can sometimes lose context, it drastically reduces the number of unique words the model needs to handle.

Punctuation is another source of noise. A simple approach is to remove all punctuation marks. The word “dog!” becomes “dog”, which is usually what we want. However, this can be problematic. A hyphen in “state-of-the-art” is meaningful, and removing it creates three separate, less useful tokens. Similarly, an apostrophe in “it’s” is important. Deciding how to handle punctuation is a balancing act between simplifying the data and preserving its meaning.

Another key step is filtering out stop words—extremely common words that appear frequently in a language but often carry little semantic weight. Words like “the”, “a”, “is”, and “in” are classic examples. Removing them helps the model focus on the words that are more likely to be predictive of the text's topic or sentiment.

However, removing stop words isn't always the right move. In sentiment analysis, for example, the word “not” is a stop word, but it completely reverses the meaning of a sentence. The decision to filter them out depends heavily on the specific NLP task you're tackling.

Sometimes, treating every word as an independent token causes you to lose crucial context. The phrase “machine learning” means something very specific, but if you tokenise it into “machine” and “learning”, that combined meaning is lost. This is where n-grams come in.

N-grams are contiguous sequences of n tokens from a given text. When n=1, we have unigrams (the single tokens we've been discussing). When n=2, we have bigrams, and for n=3, we have trigrams. By including bigrams like “machine learning” or “not good” in our vocabulary, we can capture more contextual information.

A more advanced approach, especially for languages with complex word structures and for handling out-of-vocabulary words, is subword tokenisation. Instead of splitting text by spaces, this method breaks words into smaller, meaningful parts. For instance, the word “tokenisation” might be split into “token” and “##isation”. This allows the model to understand that words like “tokenise” and “tokenisation” are related. It also helps the model handle new or rare words by breaking them down into familiar subword units.

The preprocessing strategy you choose is the foundation of your entire NLP model. A thoughtful approach that considers the trade-offs between vocabulary size, context preservation, and computational cost will lead to a more robust and accurate system. There's no single best way; the right choices depend entirely on your data and the problem you are trying to solve.