Introduction to Eigenvalues and Eigenvectors
Start your free 7-days trial now!
Eigenvalues and eigenvectors
Eigenvectors are special vectors that can be transformed by some square matrix to become a scalar multiple of themselves, that is:
Where:
is the given square matrix. is a non-zero vector referred to as an eigenvector of . is a scalar referred to as an eigenvalue of .
Given a square matrix and a vector, we can easily verify whether or not the vector is an eigenvector of the matrix. Let's now go through some examples.
Verifying that a vector is not an eigenvector of a matrix
Suppose we have the following matrix and vector:
Show that
Solution. By definition, for
Where
Clearly, the left vector is not a scalar multiple of the right vector, which means that a
Verifying that a vector is an eigenvector of a matrix
Suppose we have the following matrix and vector:
Show that
Solution. Again, we check whether
We see that if
Interestingly,
Let's check whether
This equation holds if
As we have just demonstrated, verifying whether or not the given vector is an eigenvector of a matrix is easy. We will later introduce a procedure by which we can derive the eigenvalues and eigenvectors of a given matrix, but let's first go through their geometric intuition.
Geometric intuition behind eigenvalues and eigenvectors
Recall from this sectionlink that multiplying vectors by a scalar

The matrix-vector product
We will now derive a method to compute the eigenvalues and eigenvectors of given a matrix.
Characteristic equation and polynomial
Proof. By definition, for a given square matrix
Remember, eigenvectors are defined to be non-zero, that is, not all elements in the vector are zero.
Let's rearrange
We are now going to assume that the matrix
This means that the eigenvector
This equation is called the characteristic equation of
Computing eigenvalues
Observe how there is only one unknown value in
Computing the eigenvalues of a 2x2 matrix
Compute the eigenvalues of the following matrix:
Solution. The characteristic equation of
By theoremlink, we can compute the determinant of the
This means that the eigenvalues of
Computing eigenvectors
Now that we have found the eigenvalues of a given matrix
Here,
Here, we've added a subscript to
Since
Because
Let's focus on the first row of
The only unknown here is the components of the eigenvectors
For any value
The right-hand side is fully defined and is equal to some scalar value, say
Note that we found this particular eigenvector by setting
Now that we have found the eigenvectors
substituting
into .expressing
in terms of or vice versa.setting some value for
to find .
Don't worry if you find this step confusing - we will now go through a concrete example of computing the eigenvectors of a
Computing eigenvectors of a 2x2 matrix
Find the eigenvectors of the following matrix:
Note that this is the same matrix as the one in the previous examplelink.
Solution. Recall that the eigenvalues of
Since each eigenvalue has corresponding eigenvectors, we need to compute a set of eigenvectors for
When
Notice how dividing the top row by
Let's now focus on the top row:
This tells us that components of eigenvector
Note that because eigenvectors are defined to be non-zero vectors, we cannot set
Now that we've computed an eigenvector for
Taking the top row gives:
To avoid fractions, let's pick
To summarise our results, an eigenvector corresponding to the eigenvalue
An eigenvector corresponding to the eigenvalue
Again, keep in mind that there is an infinite number of eigenvectors and that these are just one of them.
We have so far covered examples of computing eigenvalues and eigenvectors of
Computing eigenvalues and eigenvectors of a 3x3 matrix
Compute the eigenvalues and eigenvectors of the following matrix:
Solution. The flow is to find the eigenvalues first and then find a corresponding eigenvector for each eigenvalue.
Computing eigenvalues
The first step is to find the eigenvalues
In matrix form, this translates to:
From our guide on determinants, we know how to compute the determinant of a
We now compute the eigenvectors for each of these eigenvalues.
Computing 1st eigenvector
Recall that to compute the eigenvectors of an eigenvalue, we use the following equation:
When
Let's perform Gaussian elimination to solve the system:
Because the last row is all zeros, we have that
Finally, the first row of
Therefore, the eigenvector corresponding to eigenvalue
Just as we did for the
We've managed to find an eigenvector corresponding to the first eigenvalue! To find the eigenvectors corresponding to the other eigenvalues, we just repeat the same process.
Computing 2nd eigenvector
When
Performing Gaussian elimination gives:
From the second row, we have that
Therefore, the eigenvector corresponding to the second eigenvalue is:
For simplicity, let's set
Now on to the eigenvectors of the 3rd eigenvalue!
Once again, we first obtain the system of linear equations:
Performing Gaussian elimination gives:
Again, because the third row contains all zeros,
The first row gives us:
Therefore, the general form of the eigenvector is:
Let's set
Summary of results
The eigenvalues of our matrix
The corresponding eigenvectors are:
Again, be reminded there are infinitely many eigenvectors that correspond to a single eigenvalue - we just chose simple eigenvectors here.
What is the importance of eigenvalues and eigenvectors in machine learning?
Eigenvalues and eigenvectors crop up in several machine learning topics such as:
principal component analysis (PCA) - a popular technique for dimensionality reduction. It turns out that projecting data points onto the eigenvectors of the variance-covariance matrix of the features preserves the most amount of information.
spectral clustering - a clustering technique that can handle data points with a non-convex layout. Similar to PCA, eigenvalues and eigenvectors are computed to perform dimensionality reduction before the actual clustering takes place.