Principal Component Analysis and Correspondence Analysis Demystified
Eigenvalues and Eigenvectors
The Unchanging Direction
Linear transformations stretch, compress, and rotate vector spaces. Think of a transformation as a gust of wind blowing across a field of arrows. Most arrows will be knocked off their original line, pointing in new directions. But some special arrows, the eigenvectors, will only be stretched or shrunk. They keep pointing along their original line.
The term "eigen" is German for "own" or "characteristic." So, an eigenvector is a characteristic vector of a transformation. It represents an axis that remains unchanged in direction. The amount by which this eigenvector is scaled is its corresponding eigenvalue, a scalar often denoted by the Greek letter lambda ().
eigenvector
noun
A non-zero vector that changes by only a scalar factor when a linear transformation is applied to it. Its direction remains unchanged.
eigenvalue
noun
The scalar factor by which an eigenvector is stretched or compressed after a linear transformation is applied.
Visualizing the Effect
Let's visualize this. Imagine a matrix that stretches everything horizontally by a factor of 3 and does nothing vertically. Any horizontal vector is an eigenvector because it stays on the x-axis, just gets longer. Its eigenvalue is 3. Any vertical vector is also an eigenvector; it stays on the y-axis and isn't scaled, so its eigenvalue is 1. A vector pointing diagonally, however, will be tilted and point in a new direction. It's not an eigenvector.
The eigenvalue tells you the story of the scaling. An eigenvalue greater than 1 means the eigenvector is stretched. If it's between 0 and 1, the vector is compressed. A negative eigenvalue means the vector is flipped to point in the opposite direction.
Eigenvectors provide the stable axes of a transformation, while eigenvalues tell us the magnitude of the action along those axes.
How to Find Them
Finding these special vectors and scalars is a core task in linear algebra. It involves a bit of algebraic manipulation. We start with the defining equation and rearrange it.
For the equation to have a non-zero solution for (remember, eigenvectors can't be zero vectors), the matrix must be singular. This means its determinant must be zero. This gives us a way to find the eigenvalues first.
Step 1: Find the Eigenvalues (λ) Solve the characteristic equation: . This will be a polynomial equation in . The roots of this polynomial are the eigenvalues.
Step 2: Find the Eigenvectors (v) For each eigenvalue you found, plug it back into the equation and solve for the vector . The set of all solutions for a given forms a subspace called the eigenspace.
Let's walk through an example. Consider the matrix .
First, we set up the characteristic equation.
Factoring the polynomial, we get . So, our eigenvalues are and .
Now, we find the eigenvector for each eigenvalue.
For :
Any vector where the x and y components are equal, like or , is an eigenvector for . A common choice is the simplest non-zero vector, so we'll pick .
For :
An eigenvector for could be . So, .
Eigenvalues and eigenvectors form the foundation for many advanced techniques. They reveal the deep, underlying structure of a linear transformation.