Orthogonal Complements and Decompositions

Orthogonal Complements

We will now consider the set of vectors that are orthogonal to every vector in a given subspace. As a quick example, consider the \(xy\)-plane in \(\RR ^3\). Clearly, every scalar multiple of the standard unit vector \(\vec {k}\) in \(\RR ^3\) is orthogonal to every vector in the \(xy\)-plane. We say that the set \(\{c\vec {k} \mid c \in \RR \}\) is an orthogonal complement of \(\{a\vec {i}+b\vec {j} \mid a, b \in \RR \}\).

[Picture]

The following theorem collects some useful properties of the orthogonal complement; the proof of 1 and 2 is left as Practice Problem .

We must show that \(W^\perp = \{\vec {x} \mid \vec {x} \dotp \vec {x}_{i} = 0 \mbox { for each } i\}\). To show that two sets are equal, we must show that all elements of one set are included in the other set, and then we must show the reverse inclusion.

If \(\vec {x}\) is in \(W^\perp \) then \(\vec {x} \dotp \vec {x}_{i} = 0\) for all \(i\) because each \(\vec {x}_{i}\) is in \(W\). This shows \(W^\perp \subseteq \{\vec {x} \mid \vec {x} \dotp \vec {x}_{i} = 0 \mbox { for each } i\}\). For the reverse inclusion, suppose that \(\vec {x} \dotp \vec {x}_{i} = 0\) for all \(i\); we need to show that \(\vec {x}\) is in \(W^\perp \). We need to show \(\vec {x} \dotp \vec {y} = 0\) for each \(\vec {y}\) in \(W\). We can write \(\vec {y} = c_{1}\vec {x}_{1} + c_{2}\vec {x}_{2} + \dots + c_{k}\vec {x}_{k}\), where each \(c_{i}\) is in \(\RR \). Then

\begin{equation*} \vec {x} \dotp \vec {y} = c_{1}(\vec {x} \dotp \vec {x}_{1}) + c_{2}(\vec {x} \dotp \vec {x}_{2})+ \dots +c_{k}(\vec {x} \dotp \vec {x}_{k}) = c_{1}0 + c_{2}0 + \dots + c_{k}0 = 0 \end{equation*}
as required, and the proof of equality is complete.

Some of the important subspaces we studied earlier are orthogonal complements of each other. Recall the following definitions associated with an \(m \times n\) matrix \(A\).

1.
The null space of \(A\), \(\mbox {null}(A) = \{\vec {x}\in \RR ^n \mid A\vec {x} = \vec {0}\}\), is a subspace of \(\RR ^n\).
2.
The row space of \(A\), \(\mbox {row}(A) = \mbox {span} ( \mbox {the rows of } A)\), is a subspace of \(\RR ^n\).
3.
The column space of \(A\), \(\mbox {col}(A) = \mbox {span} ( \mbox {the columns of } A)\), is a subspace of \(\RR ^m\).
In the following GeoGebra interactive, you can change the coordinates of the vectors \(\vec {v}\) and \(\vec {w}\) using the sliders. (At this stage make sure that \(\vec {v}\) and \(\vec {w}\) are not collinear.) Let \(A=\begin{bmatrix}-&\vec {v}&-\\-&\vec {w}&-\end{bmatrix}\). Then \(\mbox {row}(A) = \mbox {span}(\vec {v},\vec {w})\). RIGHT-CLICK and DRAG to rotate the coordinate system for a better view.

1.
Follow the prompts in the interactive to visualize \(\mbox {row}(A)\) and \(\mbox {null}(A)\). What relationships do you observe between \(\mbox {row}(A)\) and \(\mbox {null}(A)\)?
2.
It is possible to “break" this interactive (for certain choices of \(\vec {v}\) and \(\vec {w}\)). If \(\vec {v}\) and \(\vec {w}\) are scalar multiples of each other, then \(\mbox {row}(A)\) is a Point, Line,Plane, and the dimension of \(\mbox {null}(A)\) is 1, 2,3. The interactive does not accommodate this situation. To see what happens when \(\vec {v}\) and \(\vec {w}\) are scalar multiples of each other, see Practice Problem .

Before proving this theorem, let’s examine what it says about a couple of our examples. In Example 4, we solved for the unknown vectors \(\vec {x} = \begin{bmatrix} x \\ y \\ z \\ w \end{bmatrix}\). Notice that this is equivalent to creating a \(2 \times 4\) matrix \(A\) whose rows are \(\vec {v}_1\) and \(\vec {v}_2\), and then finding the null space of that matrix \(A\). You can check that a basis for \(\mbox {null}\left (\begin{bmatrix} 1 & -1 & 2 & 0 \\ 1 & 0 & -2 & 3 \end{bmatrix}\right )\) is given by \(\left \{\begin{bmatrix} 2 \\ 4 \\ 1 \\ 0 \end{bmatrix}, \begin{bmatrix} 3 \\ 3 \\ 0 \\ -1 \end{bmatrix}\right \}\).

We now return to the proof of Theorem 8.

Let \(\vec {x}\in \RR ^n\). \(\vec {x}\in \left (\mbox {row}(A)\right )^\perp \) if and only if x is orthogonal to every row of \(A\). But this is true if and only if \(A\vec {x}=\vec {0}\), which is equivalent to saying \(\vec {x}\in \mbox {null}(A)\), which proves 1. To prove 2, we simply replace \(A\) with \(A^T\), and we may apply 1 since \(\mbox {col}(A) = \mbox {row}(A^T)\).

Orthogonal Decomposition Theorem

Now that we have defined the orthogonal complement of a subspace, we are ready to state the main theorem of this section. If you have studied physics or multi-variable calculus, you are familiar with the idea of expressing a vector in as the sum of its tangential and normal components. (If you haven’t yet taken those courses, this section will help to prepare you for them!) The following theorem is a generalization of this idea.

This is an example of an “existence and uniqueness” theorem, so there are two things to prove. If we have an orthogonal basis \(\{\vec {f}_{1}, \vec {f}_{2}, \dots , \vec {f}_{m}\}\) for \(W\), then it is easy to show that our orthogonal decomposition exists for \(\vec {x}\). We let \(\vec {w}=\mbox {proj}_W(\vec {x})\), which is clearly in \(W\), and we let \(\vec {w}^\perp = \vec {x} - \vec {w}\), and we have \(\vec {w} + \vec {w}^\perp = \vec {w} + (\vec {x} - \vec {w}) = \vec {x}\), so we need to see that \(\vec {w}^\perp \in W^\perp \).

By Theorem 2 3, it suffices to show that \(\vec {w}^\perp \) is orthogonal to each of the basis vectors \(\vec {f}_i, i=1,\ldots ,m\). We compute for \(i=1,\ldots ,m\)

\begin{align*} \vec {f}_i \dotp \vec {w}^\perp &= \vec {f}_i \dotp (\vec {x} - \vec {w}) \\ &= \vec {f}_i \dotp \vec {x} - \vec {f}_i \dotp \left (\frac {\vec {x} \dotp \vec {f}_{1}}{\norm {\vec {f}_{1}}^2}\vec {f}_{1} + \frac {\vec {x} \dotp \vec {f}_{2}}{\norm {\vec {f}_{2}}^2}\vec {f}_{2}+ \dots +\frac {\vec {x} \dotp \vec {f}_{m}}{\norm {\vec {f}_{m}}^2}\vec {f}_{m}\right ) \\ &= \vec {f}_i \dotp \vec {x} - \left (\frac {\vec {x} \dotp \vec {f}_{1}}{\norm {\vec {f}_{1}}^2}\vec {f}_i \dotp \vec {f}_{i} \right ) = \vec {f}_i \dotp \vec {x} - (\vec {x} \dotp \vec {f}_i) = 0. \end{align*}

This proves that \(\vec {w}^\perp \in W^\perp \).

The reason we need to prove this decomposition is unique is because we started with the orthogonal basis \(\{\vec {f}_{1}, \vec {f}_{2}, \dots , \vec {f}_{m}\}\) for \(W\), but what would happen if we chose a different orthogonal basis?

Suppose that \(\{\vec {f}_1^\prime , \vec {f}_2^\prime , \dots , \vec {f}_m^\prime \}\) is another orthogonal basis of \(W\), and let

\begin{equation*} \vec {w}^{\prime } = \left (\frac {\vec {x} \dotp \vec {f}^{\prime }_{1}}{\norm {\vec {f}^{\prime }_{1}}^2}\right )\vec {f}^{\prime }_{1} + \left (\frac {\vec {x} \dotp \vec {f}^{\prime }_{2}}{\norm {\vec {f}^{\prime }_{2}}^2}\right )\vec {f}^{\prime }_{2} + \dots +\left (\frac {\vec {x} \dotp \vec {f}^{\prime }_{m}}{\norm {\vec {f}^{\prime }_{m}}^2}\right )\vec {f}^{\prime }_{m} \end{equation*}
As before, \(\vec {w}^{\prime } \in W\) and \(\vec {x} - \vec {w}^{\prime } \in W^\perp \), and we must show that \(\vec {w}^{\prime } = \vec {w}\). To see this, write the vector \(\vec {w} - \vec {w}^\prime \) as follows:
\begin{equation*} \vec {w} - \vec {w}^{\prime } = (\vec {x} - \vec {w}^{\prime }) - (\vec {x} - \vec {w}) \end{equation*}
This vector is in \(W\) (because \(\vec {w}\) and \(\vec {w}^\prime \) are in \(W\)) and it is in \(W^\perp \) (because \(\vec {x} - \vec {w}^\prime \) and \(\vec {x} - \vec {w}\) are in \(W^\perp \)), and so it must be the zero vector (it is orthogonal to itself!). This means \(\vec {w}^\prime = \vec {w}\) as desired.

The final theorem of this section shows that projection onto a subspace of \(\RR ^n\) is actually a linear transformation from \(\RR ^n\) to \(\RR ^n\).

If \(W = \{\vec {0}\}\), then \(W^\perp = \RR ^n\), and so \(T(\vec {x}) = \vec {0}\) for all \(\vec {x}\). Thus \(T = 0\) is the zero (linear) operator, so 1, 2, and 3 hold. Hence assume that \(W \neq \{\vec {0}\}\).

1.
If \(\{\vec {q}_{1}, \vec {q}_{2}, \dots , \vec {q}_{m}\}\) is an orthonormal basis of \(W\), then
\begin{equation}\label {orthonormalUeq} T(\vec {x}) = (\vec {x} \dotp \vec {q}_{1})\vec {q}_{1} + (\vec {x} \dotp \vec {q}_{2})\vec {q}_{2} + \dots + (\vec {x} \dotp \vec {q}_{m})\vec {q}_{m} \quad \mbox { for all }\vec {x}\in \RR ^n \end{equation}
by the definition of the projection. Thus \(T\) is a linear transformation because
\begin{equation*} (\vec {x} + \vec {y}) \dotp \vec {q}_{i} = \vec {x} \dotp \vec {q}_{i} + \vec {y} \dotp \vec {q}_{i} \quad \mbox { and } \quad (r\vec {x}) \dotp \vec {q}_{i} = r(\vec {x} \dotp \vec {q}_{i}) \quad \mbox { for each } i. \end{equation*}
2.
We have \(\mbox {im}(T)\) is a subset of \(W\) by (1) because each \(\vec {q}_{i}\) is in \(W\). But if \(\vec {x}\) is in \(W\), then \(\vec {x} = T(\vec {x})\) by (1) and Theorem ?? applied to the space \(W\). This shows that \(W\) is a subset of \(\mbox {im}(T)\), so \(\mbox {im}(T)\) is \(W\).

Now suppose that \(\vec {x}\) is in \(W^\perp \). Then \(\vec {x} \dotp \vec {q}_{i} = 0\) for each \(i\) (again because each \(\vec {q}_{i}\) is in \(W\)) so \(\vec {x}\) is in \(\mbox {ker}(T)\) by (2). Hence \(W^\perp \) is in \(\mbox {ker}(T)\). On the other hand, Theorem 2 shows that \(\vec {x} - T(\vec {x})\) is in \(W^\perp \) for all \(\vec {x}\) in \(\RR ^n\), and it follows that \(\mbox {ker}(T)\) is in \(W^\perp \). Hence \(\mbox {ker}(T)\) is \(W^\perp \), proving 2.

3.
This follows from 1, 2, and the Rank-Nullity theorem (Theorem ??).

Practice Problems

Solve the linear system in Example 4 and use your result to find a basis for \(W^\perp \) if \(W = \mbox {span}\left (\begin{bmatrix}1\\ -1\\ 2\\ 0\end{bmatrix}, \begin{bmatrix}1\\ 0\\ -2\\ 3\end{bmatrix}\right )\) in \(\RR ^4\).
In this problem we return to the GeoGebra interactive in Exploration , and we consider the case where the matrix \(A\) has rank 1 (which Exploration could not handle). This time, the sliders define row 1 of matrix \(A\), and row 2 will be 2 times row 1. Follow the prompts in the interactive to visualize \(\mbox {row}(A)\) and \(\mbox {null}(A)\). What relationships do you observe between \(\mbox {row}(A)\) and \(\mbox {null}(A)\)?

In this problem you are asked to finish Example 9. More specifically, for the matrix
\[A=\begin{bmatrix}2&-1&1&-4&1\\1&0&3&3&0\\-2&1&-1&5&2\\4&-1&7&2&1\end{bmatrix},\]
show that \(\mbox {null}(A^T) = (\mbox {col}(A))^\perp \). It may be helpful to consult Subspaces of \(\RR ^n\) Associated with Matrices, where we found a basis for the column space of this matrix.

Problems -

In each case, write \(\vec {x}\) as \(\vec {x} = \vec {w} + \vec {w}^\perp \), where \(\vec {w}=\mbox {proj}_W(\vec {x})\) and \(\vec {w}^\perp = \vec {x} - \vec {w}\).

\(\vec {x} = \begin{bmatrix}2\\ 1\\ 6\end{bmatrix}\), \(W = \mbox {span}\left (\begin{bmatrix}3\\ -1\\ 2\end{bmatrix}, \begin{bmatrix}2\\ 0\\ -3\end{bmatrix}\right )\)
\(\vec {x} = \begin{bmatrix}2\\ 0\\ 1\\ 6\end{bmatrix}\), \(W = \mbox {span}\left (\begin{bmatrix}1\\ 1\\ 1\\ 1\end{bmatrix}, \begin{bmatrix}1\\ 1\\ -1\\ -1\end{bmatrix}, \begin{bmatrix}1\\ -1\\ 1\\ -1\end{bmatrix}\right )\)
\(\vec {x} = \begin{bmatrix}a\\ b\\ c\\ d\end{bmatrix}\), \(W = \mbox {span}\left (\begin{bmatrix}1\\ -1\\ 2\\ 0\end{bmatrix}, \begin{bmatrix}-1\\ 1\\ 1\\ 1\end{bmatrix}\right )\)
Let \(W = \mbox {span}\left (\vec {w}_{1}, \vec {w}_{2}, \dots , \vec {w}_{k}\right )\), \(\vec {w}_{i}\in \RR ^n\), and let \(A\) be the \(k \times n\) matrix with the \(\vec {w}_{i}\) as rows.
1.
Show that \(W^\perp = \{\vec {x} \mid \vec {x}\in \RR ^n, A\vec {x}^{T} = \vec {0}\}\).
2.
Use part (a) to find \(W^\perp \) if
\[W = \mbox {span}\left (\begin{bmatrix}1\\ -1\\ 2\\ 1\end{bmatrix}, \begin{bmatrix}1\\ 0\\ -1\\ 1\end{bmatrix}\right ).\]
1.
Prove part 1 of Theorem 2.
2.
Prove part 2 of Theorem 2.
Let \(W\) be a subspace of \(\RR ^n\). If \(\vec {x}\) in \(\RR ^n\) can be written in any way at all as \(\vec {x} = \vec {p} + \vec {q}\) with \(\vec {p}\) in \(W\) and \(\vec {q}\) in \(W^\perp \), show that necessarily \(\vec {p} = \mbox {proj}_W(\vec {x})\).
Let \(W\) be a subspace of \(\RR ^n\) and let \(\vec {x}\) be a vector in \(\RR ^n\). Using Practice Problem , or otherwise, show that \(\vec {x}\) is in \(W\) if and only if \(\vec {x} = \mbox {proj}_W(\vec {x})\).
Write \(\vec {w} = \mbox {proj}_W(\vec {x})\). Then \(\vec {w}\) is in \(W\) by definition. If \(\vec {x}\) is \(W\), then \(\vec {x} - \vec {w}\) is in \(W\). But \(\vec {x} - \vec {w}\) is also in \(W^\perp \), so \(\vec {x} - \vec {w}\) is in \(W \cap U^\perp = \{\vec {0}\}\). Thus \(\vec {x} = \vec {w}\).
If \(W\) is a subspace of \(\RR ^n\), show that \(\mbox {proj}_W(\vec {x}) = \vec {x}\) for all \(\vec {x}\) in \(W\).
Let \(\{\vec {q}_{1}, \vec {q}_{2}, \dots , \vec {q}_{m}\}\) be an orthonormal basis of \(W\). If \(\vec {x}\) is in \(W\) the expansion theorem gives \(\vec {x} = (\vec {x} \dotp \vec {q}_{1})\vec {q}_{1} + (\vec {x} \dotp \vec {q}_{2})\vec {q}_{2} + \dots + (\vec {x} \dotp \vec {q}_{m})\vec {q}_{m} = \mbox {proj}_W(\vec {x})\).
If \(W\) is a subspace of \(\RR ^n\), show that \(\vec {x} = \mbox {proj}_W(\vec {x}) + \mbox {proj}_{W^\perp }(\vec {x})\) for all \(\vec {x}\) in \(\RR ^n\).
If \(\{\vec {v}_{1}, \dots , \vec {v}_{n}\}\) is an orthogonal basis of \(\RR ^n\) and \(W = \mbox {span}\left (\vec {v}_{1}, \dots , \vec {v}_{m}\right )\), \(m<n\), show that \(W^\perp = \mbox {span}\left (\vec {v}_{m + 1}, \dots , \vec {v}_{n}\right )\).

Text Source

This section was adapted from the second part of Section 8.1 of Keith Nicholson’s Linear Algebra with Applications. (CC-BY-NC-SA)

W. Keith Nicholson, Linear Algebra with Applications, Lyryx 2018, Open Edition, p. 415–423.

Example 14 was adapted from Example 4.148 of Ken Kuttler’s A First Course in Linear Algebra. (CC-BY)

Ken Kuttler, A First Course in Linear Algebra, Lyryx 2017, Open Edition, p. 249.