When you type a question into a large language model (LLM) and get an answer, you're witnessing a process called inference. Inference is simply the term for using a trained model to make predictions on new data. The model isn't learning anymore; it's putting its knowledge to work.

Think of a chef who has spent years learning recipes and techniques. That's the training phase. When you hand them a basket of specific ingredients and ask for a dish, the process of them cooking that meal is inference. The LLM does something similar, but with words. This happens in three main steps: tokenization, execution, and generation.

Computers don't understand words and sentences directly. They work with numbers. The first step in inference is to translate your text prompt into a format the model can use. This translation process is called tokenization.

Tokenization breaks the input text into smaller chunks called tokens. A token can be a whole word, a part of a word (like ing or est), or even a single punctuation mark. Each unique token in the model's vocabulary is mapped to a specific integer.

This method allows the model to handle a vast vocabulary efficiently. It can recognize parts of words, which helps it understand and generate new or rare words it hasn't seen before.

Once the prompt is a sequence of numbers, it's fed into the LLM. The model itself is a massive neural network with a specific architecture, often called a Transformer. This network is made of many layers that process the input sequence.

At its core, the model is a sophisticated prediction engine. For each token it receives, it performs a huge number of mathematical calculations to predict the most probable next token. It considers the context provided by all the previous tokens to make this prediction. This process flows through the network's layers, which are trained to weigh different parts of the input to understand grammar, facts, and context.

The final step is to generate a response. LLMs do this one token at a time in a process called autoregressive generation. The model predicts the single most likely token to come next, adds it to the sequence, and then feeds the entire new sequence back into itself to predict the following token. This loop continues until the response is complete.

But how does the model choose the next token when there are many possibilities? Instead of always picking the single most probable token (a method called greedy search), models often use sampling strategies to produce more creative and natural-sounding text.

The generation process stops when the model produces a special "end-of-sequence" token or reaches a predefined length limit. The resulting sequence of tokens is then converted back into human-readable text, and you see the final response.

Time to check your understanding of the core concepts of LLM inference.

Together, these three steps—tokenization, execution, and generation—form the inference process that allows a large language model to turn a simple prompt into a coherent and useful answer.