Splitting text by spaces is a simple first step, but it quickly runs into trouble. What happens when the model encounters a word it has never seen during training? This is the out-of-vocabulary, or OOV, problem. The model has no representation for the new word, which leads to a loss of information. Character-level tokenization solves this by breaking everything down to individual letters, but it creates extremely long sequences and loses the semantic meaning of whole words.

Subword tokenization offers a practical middle ground. It keeps common words as single units while breaking down rare or unknown words into smaller, meaningful pieces. This allows a model to maintain a fixed, manageable vocabulary size while still being able to process any word it encounters. Modern large language models rely on these clever algorithms to understand and generate text.

Originally a data compression algorithm, Byte Pair Encoding was adapted for NLP and is used in models like the GPT family. The process starts by building a vocabulary of all individual characters present in a training corpus. Then, it iteratively counts the most frequent pair of adjacent tokens and merges them into a single, new token.

This process repeats for a predetermined number of merges. This number defines the final vocabulary size. Let's see how it works with a small corpus containing the words: low, lower, newest, widest.

After several merges, the vocabulary contains common subwords like low, er, and est. Now, if the model sees the OOV word "lowest", it can represent it as two known tokens: low and est. It has successfully understood a new word without ever seeing it before.

While BPE is effective, other methods have emerged. WordPiece, used by models like BERT, is very similar to BPE but uses a different criterion for merging. Instead of merging the most frequent pair, it merges the pair that maximizes the likelihood of the training data once added to the vocabulary. Essentially, it asks, "Which merge makes the resulting data the most 'predictable'?" This subtle change helps it create a slightly different and often more efficient vocabulary.

Then there's SentencePiece, which takes a more radical approach. It's used in models like T5 and LLaMA. Its key innovation is treating the input text as a raw character stream, including spaces. It doesn't assume words are separated by spaces, which makes it truly language-independent. This is a huge advantage for languages like Chinese, Japanese, or Thai, where words aren't clearly delineated by spaces.

Because it handles spaces internally, SentencePiece can also perfectly convert tokens back into the original text, a feature called lossless tokenization. This avoids ambiguity where, for example, `[