Given a fixed compute budget for training, a central optimization problem is how to allocate resources between model size (number of parameters, $N$) and dataset size (number of tokens, $D$). The relationship is not arbitrary; it's governed by predictable scaling laws. The loss $L$ of a transformer model can be accurately predicted by a power-law function:

The training compute budget, measured in FLOPs, is approximately proportional to the product of model size and dataset size: $C \approx 6ND$. This approximation arises because each forward and backward pass for a single token requires about $2N$ FLOPs, and the model sees each of the $D$ tokens roughly three times over the course of training (hence the factor of 6). Our goal is to minimize $L(N, D)$ subject to the constraint $C = 6ND$.

This leads to the Chinchilla scaling laws, which dictate that the optimal number of parameters $N_{opt}$ and tokens $D_{opt}$ should scale as functions of the compute budget $C$:

The original Chinchilla paper found empirically that $\alpha \approx \beta$, which simplifies the relationship significantly. It implies that for every doubling of model size, the number of training tokens should also be doubled. This finding corrected a prior trend of training very large models on comparatively small datasets, demonstrating that many large models of the pre-Chinchilla era were severely undertrained.

The $N$ and $D$ parameters are not the only variables in the equation. More recent research has extended scaling laws to other architectural choices, such as vocabulary size and the use of sparse expert models.

For instance, vocabulary size ($V$) shouldn't be static. The optimal $V$ scales sub-linearly with the number of model parameters $N$. Intuitively, a larger model has the capacity to learn a more granular representation of language, making a larger vocabulary beneficial. However, the gains diminish, and an excessively large vocabulary can waste parameters on rare tokens. A common heuristic is $V \propto N^k$ where $k$ is roughly in the range of 0.3 to 0.5.

Another critical development is the Mixture-of-Experts (MoE) architecture. In a dense transformer, every parameter is used for every token during a forward pass. In an MoE model, the feed-forward network layers are replaced by a set of parallel 'expert' networks. For each token, a router network selects a small subset of these experts (often just one or two) to process the token.

This changes the compute constraint. The training cost is no longer proportional to the total parameters $N_{total}$, but to the number of active parameters per token, $N_{active}$. The scaling laws still apply, but they apply to $N_{active}$, not $N_{total}$. This allows for the creation of models with trillions of total parameters while maintaining a computationally feasible number of active parameters, effectively decoupling total model capacity from inference cost.

The effectiveness of these scaling laws hinges on a fundamental assumption about the data itself: the manifold hypothesis. This hypothesis posits that high-dimensional data, like the token embeddings used in transformers, actually lies on or near a low-dimensional manifold embedded within the higher-dimensional space.

The intrinsic dimensionality ($d$) of the data manifold is a measure of its complexity. A model doesn't need to learn the structure of the entire ambient space, only the structure of this much simpler manifold where the data resides. This is why generalization is possible. The model learns a compressed representation of the world as described by the training data.

The exponents $\alpha$ and $\beta$ in the scaling laws are not universal. They are deeply connected to the intrinsic dimensionality of the training data. Data with a lower intrinsic dimensionality is 'easier' to learn, resulting in larger values for $\alpha$ and $\beta$, which means the loss decreases more rapidly as model size and data grow. This underscores the importance of data quality and curation. Pre-training on a high-quality, information-dense corpus can lead to more efficient scaling than simply using more, lower-quality data.

Understanding these mathematical underpinnings is crucial for designing the next generation of efficient and powerful models. It's a continuous process of refining our understanding of the interplay between architecture, data, and the fundamental limits of learning.