Every matrix represents a linear transformation. When a matrix acts on a vector, it transforms it into another vector. But what's really happening during this transformation? It turns out any linear map, no matter how complex it seems, can be broken down into three fundamental geometric steps: a rotation, a scaling, and another rotation.

Imagine a set of perpendicular vectors, like the standard axes in a coordinate system. A matrix transformation first rotates this set of axes. Then, it stretches or shrinks the space along these new, rotated axes. Finally, it performs one last rotation to get to the final orientation. This three-step process is the essence of Singular Value Decomposition (SVD).

This geometric insight is what makes SVD so powerful. It lets us express any matrix $A$ as the product of three other matrices, each corresponding to one of these fundamental actions.

So, how do we find these rotation and scaling matrices? The answer lies in a clever connection to eigenvalue decomposition. While SVD works for any matrix, eigenvalue decomposition is more restrictive; it only works for certain square matrices. SVD cleverly gets around this by focusing on two related, and always square, matrices: $A^T A$ and $A A^T$.

These matrices are special. They are symmetric and positive semi-definite, which guarantees their eigenvalues are real and non-negative, and their eigenvectors are orthogonal. This is exactly what we need.

Let's see how they connect to our SVD equation, $A = U \Sigma V^T$. We can start by multiplying both sides by $A^T$.

Because $U$ is an orthogonal matrix, $U^T U$ is the identity matrix, $I$. So the expression simplifies nicely.

This last equation, $A^T A = V (\Sigma^T \Sigma) V^T$, is exactly the eigenvalue decomposition of the matrix $A^T A$. This tells us two critical things:

1. The columns of $V$ are the eigenvectors of $A^T A$.
2. The diagonal entries of $\Sigma^T \Sigma$ are the eigenvalues of $A^T A$. The singular values in $\Sigma$ are therefore the square roots of these eigenvalues.

A similar process for $A A^T$ shows that the columns of $U$ are the eigenvectors of $A A^T$.

This derivation also proves that an SVD exists for any rectangular matrix $A$. Because $A^T A$ is always a square, symmetric matrix, it is guaranteed to have an eigenvalue decomposition. From that decomposition, we can always construct the matrices $V$ and $\Sigma$. Once we have those, we can solve for $U$ from the original equation $A = U \Sigma V^T$.

But is the decomposition unique? Mostly, but not entirely.

The singular values (the diagonal entries of $\Sigma$) are always unique. By convention, they are ordered from largest to smallest.

However, the matrices $U$ and $V$ are not always unique. If a matrix has repeated singular values, the corresponding eigenvectors (the columns of $U$ and $V$) span a subspace, and any set of orthonormal vectors that also spans that subspace will work. Think of it like a circle: you can pick any two perpendicular axes through its center, and they are equally valid. A similar ambiguity arises if any singular values are zero.

Understanding these foundations allows us to see SVD not just as a computational tool, but as a fundamental statement about the nature of linear transformations.