The most straightforward way to tokenize text is to split it by words. This usually means breaking a sentence apart wherever there's a space or punctuation. For a simple sentence like "Cats are nice," this method works perfectly. You get three tokens: [Cats, are, nice].

But language is messy. What about a word like "don't"? Splitting by the apostrophe gives you [do, n't]. This is often better than keeping "don't" as one unit, because the model can learn the meaning of negation (n't) separately. However, this approach quickly runs into problems.

First, there's the issue of vocabulary size. A word-based tokenizer needs to know every single word it might encounter. If it sees a new word, like "supercalifragilisticexpialidocious," it has no idea what to do. This is called the "out-of-vocabulary" or OOV problem. The model has to assign a generic "unknown" token, losing all the information in the new word.

Second, it struggles with related words. "Run," "running," and "ran" are treated as three completely separate tokens, even though they share a core meaning. The model has to learn their relationship from scratch.

To solve the problems of word-level tokenization, researchers developed subword methods. The core idea is to break words into smaller, meaningful pieces. Common words like "and" or "the" can remain as single tokens, but rare or complex words are split.

For example, the word "tokenization" might be broken into [token, ##ization]. The ## indicates that "ization" is the end of a word. Now, the model can learn that the suffix "-ization" often turns a noun into a process. If it later sees the new word "globalization," it can guess its meaning by combining its knowledge of [global, ##ization].

This approach keeps the vocabulary manageable while avoiding the dreaded unknown token. Two popular algorithms for creating these subword vocabularies are Byte Pair Encoding (BPE) and WordPiece.

Byte Pair Encoding (BPE) starts by breaking all words in the training text into individual characters. Then, it iteratively finds the most frequent pair of adjacent tokens (or characters, at first) and merges them into a new, single token. It repeats this process for a set number of merges, resulting in a final vocabulary of characters, common subwords, and whole words.

WordPiece is very similar to BPE. It also starts with a vocabulary of individual characters. However, instead of merging the most frequent pair, it merges the pair that increases the likelihood of the training data the most. This subtle difference means WordPiece prioritizes merges that are more "valuable" for reconstructing the original text, not just more common. WordPiece is used by popular models like BERT.

Character-level tokenization is the simplest of all. It just splits text into a sequence of characters. The sentence "Hello world" becomes [H, e, l, l, o, , w, o, r, l, d]. The vocabulary is tiny (just the set of all possible characters), and there are no unknown tokens. However, this method creates very long sequences, which increases the computational cost for models. It also forces the model to learn the concept of words entirely from scratch, which can be inefficient.

Some languages present unique challenges. Languages like Chinese, Japanese, and Thai don't use spaces to separate words. A sentence like 我爱北京 (wǒ ài Běijīng, "I love Beijing") is a continuous string of characters. Word tokenization is impossible without a specialized, language-specific algorithm to first segment the text into words. Subword tokenization is especially powerful for these languages, as it can operate on the characters without needing to know where the word boundaries are.

Modern transformer architectures, like those that power today's large language models, almost universally use subword tokenization. It provides the perfect balance. The fixed, moderately-sized vocabulary is efficient, while the ability to handle any word prevents information loss from unknown tokens. The meaningful subwords also give the model a head start in understanding word structure and meaning.

Time to check your understanding of these different techniques.

Choosing the right tokenization strategy is a crucial first step in any NLP pipeline, directly impacting a model's performance, efficiency, and ability to understand language.