In non-linear least squares, the objective is to find a parameter vector $x$ that minimizes the sum of squares of a set of non-linear residual functions. This can be expressed as minimizing $S(x) = \frac{1}{2} \|r(x)\|^2$, where $r(x): \mathbb{R}^n \rightarrow \mathbb{R}^m$ is a vector of $m$ residuals depending on $n$ parameters. Unlike linear problems, a closed-form solution is generally unavailable. Instead, we rely on iterative methods that build a local model of the objective function at each step.

The foundation of methods like Gauss-Newton and Levenberg-Marquardt is the linearization of the residual vector $r(x)$ around the current estimate $x_k$. By applying a first-order Taylor expansion, we can approximate the residual vector for a small step $\delta$ in the parameter space.

This linear approximation transforms the complex, non-linear optimization problem into a sequence of simpler, linear least squares problems. At each iteration, we seek the step $\delta$ that minimizes the norm of the linearized residual, $\|r(x_k) + J(x_k)\delta\|^2$. Solving this subproblem yields the normal equations for the Gauss-Newton method.

The Jacobian matrix is more than just a collection of partial derivatives; it provides a profound geometric insight into the optimization landscape. The function $r(x)$ defines an $n$-dimensional surface embedded within the $m$-dimensional residual space. This surface is known as the residual manifold of the problem. Each point on this manifold corresponds to the residual vector for a specific choice of parameters $x$.

The Jacobian $J(x_k)$ acts as a linear map from the parameter space's tangent space at $x_k$ (which is simply $\mathbb{R}^n$) to the residual space's tangent space at $r(x_k)$. The columns of the Jacobian form a basis for the tangent plane to the residual manifold at the point $r(x_k)$. Therefore, the expression $J\delta$ represents a vector lying in this tangent plane—it is our best linear approximation of how the residual vector changes in response to a parameter step $\delta$.

The stability and efficiency of the optimization process hinge on the properties of the Jacobian. The normal equations require solving a system involving the matrix $J^T J$. This system is solvable with a unique solution for $\delta$ if and only if $J^T J$ is invertible. This, in turn, is true if and only if the Jacobian $J$ has full column rank, meaning its columns are linearly independent.

Geometrically, rank deficiency means the mapping from the parameter space to the residual manifold is degenerate near $x_k$. The tangent plane is "squashed" and has a dimension less than $n$. This creates flat valleys or ridges in the objective function's landscape, where many different steps $\delta$ result in the same minimal improvement. An optimizer can get stuck in these regions, unable to determine a unique descent direction. This is a primary source of ill-conditioning in non-linear least squares, and it's why techniques like the Levenberg-Marquardt algorithm were developed. They regularize the problem by adding a term $\lambda I$ to $J^T J$, ensuring the system remains invertible even when the Jacobian is rank-deficient.

Analyzing the singular values of the Jacobian is a powerful technique for diagnosing such issues. Small singular values correspond to directions in the parameter space that have a weak effect on the residuals, signaling potential rank deficiency and a difficult, ill-conditioned optimization landscape.