When you give a Large Language Model (LLM) a prompt, it doesn't read the words like a person does. Instead, it breaks the text down into smaller pieces called tokens. These tokens can be words, parts of words, or even just punctuation. This process is called tokenization.

For example, the sentence "LLMs are powerful" might become three tokens: ["LLMs", "are", "powerful"]. A more complex word like "unbelievably" could be broken into ["un", "believe", "ably"]. This allows the model to handle a vast vocabulary and even words it hasn't seen before.

Once your prompt is tokenized, the LLM begins the process of generating a response. This process, known as inference, happens in two main stages: the prefill phase and the decode phase.

The prefill phase is all about understanding your initial request. In this stage, the model takes all the tokens from your prompt and processes them simultaneously. Think of it like reading an entire question on a test before you start to formulate an answer. The model is absorbing the full context of what you've asked.

The main goal of the prefill phase is to generate a probability distribution over its entire vocabulary for what the very next token should be. It essentially makes a highly educated guess for the first piece of its response based on the entirety of your prompt. This initial, parallel processing step sets the stage for the actual text generation.

After the prefill phase predicts the first token of the response, the decode phase begins. This is where the text is generated, one token at a time, in a loop. This step-by-step process is called autoregressive generation.

Let's walk through a simple example. Suppose your prompt is "The sun is very".

1. Prefill: The model processes ["The", "sun", "is", "very"] all at once and predicts the next most likely token is "bright".
2. Decode Step 1: The model appends its prediction. The new sequence is now ["The", "sun", "is", "very", "bright"]. It processes this new sequence to predict the next token. Let's say it predicts "." (a period).
3. Decode Step 2: The model appends the period. The sequence is now ["The", "sun", "is", "very", "bright", "."]. It processes this sequence and might predict a special "end-of-sequence" token, telling it the response is complete.

This cycle repeats, with each newly generated token extending the context, until the model determines the response is finished.

So, while it feels like the model generates a full response instantly, it's actually building it piece by piece in a rapid, iterative loop. The prefill phase handles the initial prompt, and the decode phase constructs the answer one token at a time.

This two-phase process of prefill and decode is fundamental to how today's large language models generate text, turning a simple prompt into a coherent and contextually relevant response.