In a standard Transformer model, every token passes through the same feed-forward network (FFN) in each layer. A Mixture of Experts (MoE) model replaces this single FFN with a collection of smaller, specialized FFNs called 'experts.' But how does the model decide which expert should handle a given token?

This decision is made by a small but critical component: the gating network, or router. Think of it as a smart traffic controller. For every token that arrives at the MoE layer, the gating network examines its representation (its embedding vector) and determines which of the available experts is best suited for the job.

This process happens for every single token independently. A token representing the word "capital" in "capital of France" might get routed to an expert specialized in geography, while a token for "capital" in "capital gains" might go to an expert focused on finance. This specialization is the core idea behind MoE's efficiency and power.

The gating network is itself a simple neural network, typically just a single linear layer. It takes the token's embedding vector as input and produces a vector of scores, called logits. There is one logit for each expert in the layer. A higher logit suggests a better fit between the token and that expert.

To turn these logits into a usable routing decision, we first apply the softmax function. This converts the raw scores into a probability distribution, where each value is between 0 and 1, and all values sum to 1. These values represent the router's confidence in sending the token to each expert.

However, sending a little bit of the token's information to every expert would defeat the purpose of sparsity. Instead, MoE models use Top-K routing. For each token, we simply select the K experts with the highest softmax scores. In many popular models, like Mixtral 8x7B, K is set to 2. This means only the top two most relevant experts are activated to process the token. All other experts remain dormant, saving a tremendous amount of computation.

This brings us to a crucial distinction: the total number of parameters versus the number of active parameters.

A model like Mixtral 8x7B has a total of around 47 billion parameters. A dense model of this size would be incredibly slow and expensive to run, as it would require all 47 billion parameters for every single token. However, because Mixtral uses Top-2 routing within its MoE layers, only a fraction of those parameters are used in any given forward pass.

This is the essence of conditional computation. The computation performed is conditional on the input token itself. Instead of running the token through a massive, monolithic network, the router dynamically creates a smaller, specialized path for it through the model. This allows MoE models to have a vast repository of knowledge (total parameters) while maintaining the computational footprint (active parameters, or FLOPs) of a much smaller dense model.

While standard softmax gating is common during inference, some training methods introduce 'noisy gating.' This involves adding random noise to the logits before the Top-K selection. This encourages the model to explore different experts during training, preventing it from over-relying on just a few and helping to ensure all experts become specialized.

By activating only the necessary experts for a given task, MoE transforms a dense computation into a sparse and efficient one.