Computers are powerful, but they have a fundamental limitation: they don't understand language the way we do. Words, sentences, and nuance are lost on them. At their core, computers only understand numbers. So, how do we bridge this gap? How does a large language model (LLM) process a sentence you type into a chat window?

The very first step is a process called tokenization. It’s the essential translation layer between our human language and the computer's numerical world.

Think of tokens as Lego bricks. You can build almost anything with a basic set of Lego shapes. You don't need a unique brick for every possible object in the world—a car, a house, a tree. Instead, you combine smaller, reusable pieces to create them.

Tokenization applies this same logic to language. Instead of treating every single word as a unique item, the model breaks text down into a vocabulary of common pieces. Sometimes a token is a whole word, like "apple". Other times, it might be a prefix like "un-", a suffix like "-ing", or just a piece of a word, like "ize".

The simplest way to tokenize is to split a sentence by its words. This is called word tokenization. You just separate the text wherever you see a space. It's a straightforward approach that works well for many common words.

But what happens when the model encounters a word it has never seen before? Or different forms of the same word, like "run," "running," and "ran"? If the model's vocabulary only contains "run," it won't understand the others. This is where a smarter approach comes in handy.

Sub-word tokenization breaks words into smaller, more meaningful parts. Using our Lego analogy, instead of having a single, unique brick for the word "unhappiness," the model might have three smaller bricks: un, happi, and ness. By combining these, it can understand and even construct new words it hasn't explicitly seen during training. This makes the model much more flexible and efficient.

Notice how this method handles punctuation like ! and . as separate tokens. It also preserves information about spacing. The tokens is and amazing start with a space to show that they weren't attached to the previous word. This is a clever way to ensure that when the model generates text back, it knows where to put spaces and can reconstruct the sentence perfectly.

Chopping text into tokens is only half the job. Remember, computers need numbers. The final step of tokenization is to convert these text-based tokens into numerical IDs.

Every language model has a pre-defined dictionary, called a vocabulary index. This index is a giant list that maps every possible token to a unique integer. For example, The might be ID 464, quick might be 3259, and so on. After the sentence is broken into tokens, the model looks up each token in its vocabulary and swaps it for the corresponding ID.

The end result is a sequence of numbers. This sequence is what the LLM actually works with. It has no concept of the original words, only these numerical IDs and the patterns it has learned between them from analyzing billions of examples.

This simple but brilliant process is the foundation for everything an LLM does, from writing an email to explaining a complex scientific theory.