The core hypothesis behind LoRA is that the change in weights, $\Delta W$, required to adapt a pre-trained model to a new task has a low "intrinsic rank." This means the information needed for adaptation can be compressed into a much smaller dimensional space than the full weight matrix $W$. The low-rank update matrix, $BA$, is an explicit attempt to model this low-dimensional subspace. The rank $r$ is the hyperparameter that defines the dimensionality of this projection.

Choosing $r$ is a balancing act. A very small $r$ may not provide enough expressive capacity to capture the nuances of the downstream task, leading to underfitting. The model simply lacks the parameters to learn the target function effectively. Conversely, an excessively large $r$ increases the number of trainable parameters, approaching the complexity of full fine-tuning. This not only diminishes the memory and computational benefits of LoRA but also elevates the risk of overfitting the training data or inducing catastrophic forgetting of the model's pre-trained knowledge.

The choice of $r$ can be theoretically grounded in the Singular Value Decomposition (SVD) of the weight update matrix $\Delta W$. If we were to perform full fine-tuning and then analyze the resulting $\Delta W$, SVD would decompose it into three matrices: $U$, $\Sigma$, and $V^T$.

The LoRA hypothesis implies that the singular value spectrum of $\Delta W$ decays rapidly. A few large singular values capture most of the variance, while the rest are negligible. The optimal rank $r$ for the LoRA adapter corresponds to the 'knee' or 'elbow' in the distribution of these singular values, the point after which the values plummet. By selecting an $r$ that captures, say, 90% or 95% of the total energy (sum of squared singular values), we can create a low-rank approximation that is nearly as effective as the full update.

In practice, we don't compute the SVD of $\Delta W$. We instead search for an optimal $r$ empirically. This search reveals a characteristic rank-efficiency curve. As $r$ increases from a small value (like 1, 2, or 4), model performance on the validation set typically improves dramatically. Each additional rank adds significant expressive power, allowing the model to better capture the task-specific subspace.

This plateau effect confirms the low intrinsic rank hypothesis. Once $r$ is large enough to span the task's essential subspace, further increasing its dimensionality adds redundancy rather than useful capacity. The additional parameters may start fitting noise in the training data instead of the underlying signal.

Interestingly, the optimal rank is not necessarily correlated with the rank of the original pre-trained weight matrix $W$. A high-rank matrix $W$ does not imply that a high-rank adaptation $\Delta W$ is needed. The adaptation's complexity is a function of the downstream task, not the base model's architecture. Simple classification tasks might require a smaller $r$, while complex reasoning or code generation tasks might benefit from a slightly larger one, but typically not beyond a certain threshold like 64 or 128 for most common architectures.