A large language model doesn't see words, sentences, or paragraphs. It sees numbers. Before an LLM can predict the next word in a sentence or summarize a document, it must first translate human language into a numerical format it can process. This crucial first step is called tokenization.

At its core, tokenization breaks a piece of text into smaller units called tokens. These tokens can be words, parts of words (subwords), or even individual characters. The goal is to create a fixed vocabulary of these tokens that the model can understand. But how do you decide what goes into this vocabulary? If you only use whole words, you'll need a massive vocabulary to cover every word in a language, including rare ones, misspellings, and new slang. This is inefficient.

To solve this, modern LLMs use subword tokenization algorithms. These methods create a vocabulary that balances coverage and size by breaking words into more common, smaller pieces.

The most common algorithms build their vocabularies by analyzing a massive corpus of text and identifying the most efficient ways to split words. Three popular methods are Byte Pair Encoding, WordPiece, and SentencePiece.

Byte Pair Encoding (BPE) starts by treating every individual character as a token. It then iteratively scans the text and merges the most frequently occurring adjacent pair of tokens into a new, single token. This process repeats for a set number of merges. For example, 'e' and 'r' might merge into 'er', and then 'er' and ' ' (space) might merge into 'er '. Eventually, common words become single tokens, while rare words are represented as a sequence of subword tokens. This allows the model to understand 'running' and 'runner' by recognizing the shared root 'run'.

WordPiece, used by models like BERT, is similar to BPE but uses a different criterion for merging. Instead of choosing the most frequent pair, WordPiece merges pairs that maximize the likelihood of the training data. In simple terms, it asks, "Which merge will make the words in our text most probable, given our current vocabulary?" This often results in a vocabulary that keeps common words whole and breaks down rare words into intuitive subword units.

SentencePiece takes a more generalized approach. It treats the text, including spaces, as a raw sequence of Unicode characters. This is especially useful for languages without clear word delimiters like spaces. It can operate in either BPE or a unigram language model mode, giving it flexibility. The key advantage is that it provides a fully self-contained system for tokenization and detokenization without relying on language-specific rules.

These algorithms produce a vocabulary and a set of merge rules. When new text is fed into the model, it's converted into a sequence of token IDs based on this vocabulary. Words not in the vocabulary, known as out-of-vocabulary (OOV) terms, are handled gracefully by being broken down into known subword tokens. This is a huge advantage over word-level tokenizers, which would have to assign a generic "unknown" token to any word they hadn't seen before.

After tokenization, we have a sequence of integer IDs, like [50256, 318, 262, 1110, 345, 257, 4394, 290, 13]. This is still not something a neural network can work with directly. The next step is the embedding layer, which converts each token ID into a dense vector of continuous numbers. This process is essentially a lookup table.

Each row in this embedding matrix corresponds to a token in the vocabulary. The matrix is initialized with random values, and as the model trains, these vectors are adjusted. The goal is for tokens with similar meanings to have similar vectors. In this high-dimensional space, the directions and distances between vectors capture semantic relationships. For example, the vector for 'king' minus 'man' plus 'woman' might be very close to the vector for 'queen'.

This is represented mathematically as:

Embedding('king') - Embedding('man') + Embedding('woman') ≈ Embedding('queen')

The dimensionality of these vectors (e.g., 768 for BERT-base, 4096 for Llama 2) determines the richness of the semantic space the model can learn. More dimensions allow for more nuanced relationships but also increase computational cost.

The choice of tokenizer directly affects the model's efficiency and context window. A tokenizer that creates many subword tokens for common words will result in longer sequences, consuming more of the context window and requiring more computation. For instance, if a model's context window is 2048 tokens, a more efficient tokenizer might fit an entire document, while a less efficient one might only fit the first few paragraphs.

This is why building a good tokenizer is a critical, often overlooked, part of creating a powerful LLM. The balance between vocabulary size, tokenization strategy, and embedding dimensions sets the stage for everything that follows in the Transformer architecture's pipeline.

Let's test your understanding of how text is prepared for a large language model.