Kuzee
Linear Algebra

Inner Products and Orthogonalization

Inner Product Spaces

Inner Product Space

Let VV be a real vector space. If for every a⃗,b⃗∈V\vec{a}, \vec{b} \in V a real number (a⃗,b⃗)(\vec{a}, \vec{b}) is defined, called the inner product of a⃗\vec{a} and b⃗\vec{b}, and it satisfies:

  1. Symmetry: (a⃗,b⃗)=(b⃗,a⃗)(\vec{a}, \vec{b}) = (\vec{b}, \vec{a})
  2. Linearity: (ka⃗+b⃗,c⃗)=k(a⃗,c⃗)+(b⃗,c⃗)(k\vec{a} + \vec{b}, \vec{c}) = k(\vec{a}, \vec{c}) + (\vec{b}, \vec{c})
  3. Positive definiteness: (a⃗,a⃗)≥0(\vec{a}, \vec{a}) \geq 0, with equality if and only if a⃗=0⃗\vec{a} = \vec{0}

then VV is called an inner product space.

Euclidean Spaces

The Euclidean Inner Product

In Rn\mathbb{R}^n the standard inner product is (a⃗,b⃗)=a1b1+a2b2+⋯+anbn=a⃗⋅b⃗(\vec{a}, \vec{b}) = a_1b_1 + a_2b_2 + \dots + a_nb_n = \vec{a} \cdot \vec{b}

The Length and Angle of Vectors

The Length (Norm) of a Vector

The Length of a Vector

∥a⃗∥=(a⃗,a⃗)=a⃗⋅a⃗\|\vec{a}\| = \sqrt{(\vec{a}, \vec{a})} = \sqrt{\vec{a} \cdot \vec{a}}

The Angle Between Vectors

The Angle Between Vectors

cos⁡θ=(a⃗,b⃗)∥a⃗∥∥b⃗∥\cos\theta = \frac{(\vec{a}, \vec{b})}{\|\vec{a}\| \|\vec{b}\|}

Orthogonal Vectors

Orthogonal Vectors

If (a⃗,b⃗)=0(\vec{a}, \vec{b}) = 0, then the vectors a⃗\vec{a} and b⃗\vec{b} are called orthogonal, written a⃗⊥b⃗\vec{a} \perp \vec{b}.

Orthogonal Sets

Orthogonal Set

If every two distinct vectors of a set are orthogonal, the set is called an orthogonal set.

Orthonormal Sets

Orthonormal Set

If a set is an orthogonal set and every vector in it has length 1, the set is called an orthonormal set (or orthonormal basis when it is also a basis).

The Gram-Schmidt Orthogonalization Process

Theorem1

Any linearly independent set of vectors can be turned into an orthonormal set by the Gram-Schmidt orthogonalization process.

The Gram-Schmidt Procedure

Let a⃗1,a⃗2,…,a⃗n\vec{a}_1, \vec{a}_2, \dots, \vec{a}_n be a linearly independent set of vectors. Construct an orthogonal set b⃗1,b⃗2,…,b⃗n\vec{b}_1, \vec{b}_2, \dots, \vec{b}_n as follows:

Gram-Schmidt Orthogonalization
b⃗1=a⃗1b⃗2=a⃗2−(a⃗2,b⃗1)(b⃗1,b⃗1)b⃗1b⃗3=a⃗3−(a⃗3,b⃗1)(b⃗1,b⃗1)b⃗1−(a⃗3,b⃗2)(b⃗2,b⃗2)b⃗2⋮b⃗k=a⃗k−∑j=1k−1(a⃗k,b⃗j)(b⃗j,b⃗j)b⃗j\begin{aligned} \vec{b}_1 &= \vec{a}_1 \\ \vec{b}_2 &= \vec{a}_2 - \frac{(\vec{a}_2, \vec{b}_1)}{(\vec{b}_1, \vec{b}_1)} \vec{b}_1 \\ \vec{b}_3 &= \vec{a}_3 - \frac{(\vec{a}_3, \vec{b}_1)}{(\vec{b}_1, \vec{b}_1)} \vec{b}_1 - \frac{(\vec{a}_3, \vec{b}_2)}{(\vec{b}_2, \vec{b}_2)} \vec{b}_2 \\ &\vdots \\ \vec{b}_k &= \vec{a}_k - \sum_{j=1}^{k-1} \frac{(\vec{a}_k, \vec{b}_j)}{(\vec{b}_j, \vec{b}_j)} \vec{b}_j \end{aligned}

Orthonormalisation

Scale the orthogonal set to unit length:

Orthonormalisation

e⃗i=b⃗i∥b⃗i∥\vec{e}_i = \frac{\vec{b}_i}{\|\vec{b}_i\|}

Orthogonal Matrices

Orthogonal Matrix

If a real matrix QQ satisfies QTQ=IQ^T Q = I (equivalently QT=Q−1Q^T = Q^{-1}), then QQ is called an orthogonal matrix.

Properties of Orthogonal Matrices

Theorem2

An orthogonal matrix has the following properties:

  1. QT=Q−1Q^T = Q^{-1}
  2. ∣Q∣=±1|Q| = \pm 1
  3. The column (row) vectors of QQ form an orthonormal set
  4. An orthogonal transformation preserves the lengths and the angles of vectors

Exercises

Exercise 1

Use the Gram-Schmidt orthogonalization process to orthogonalize the set {(1,1,0),(1,0,1)}\{(1,1,0), (1,0,1)\}.

Gram-Schmidt processorthogonal vectorsinner product

Approach: Apply the Gram-Schmidt formula step by step: the first vector stays as it is, and the second one has its projection onto the first vector subtracted. The result is checked by recomputing the inner product of the two vectors, which has to come out as zero.

Steps:

  1. b⃗1=a⃗1=(1,1,0)\vec{b}_1 = \vec{a}_1 = (1,1,0)

  2. Compute b⃗2\vec{b}_2: b⃗2=a⃗2−(a⃗2,b⃗1)(b⃗1,b⃗1)b⃗1\vec{b}_2 = \vec{a}_2 - \frac{(\vec{a}_2, \vec{b}_1)}{(\vec{b}_1, \vec{b}_1)} \vec{b}_1 (a⃗2,b⃗1)=(1,0,1)⋅(1,1,0)=1+0+0=1(\vec{a}_2, \vec{b}_1) = (1,0,1) \cdot (1,1,0) = 1 + 0 + 0 = 1 (b⃗1,b⃗1)=(1,1,0)⋅(1,1,0)=1+1+0=2(\vec{b}_1, \vec{b}_1) = (1,1,0) \cdot (1,1,0) = 1 + 1 + 0 = 2 b⃗2=(1,0,1)−12(1,1,0)=(1,0,1)−(0.5,0.5,0)=(0.5,−0.5,1)\vec{b}_2 = (1,0,1) - \frac{1}{2}(1,1,0) = (1,0,1) - (0.5, 0.5, 0) = (0.5, -0.5, 1)

  3. Orthogonal set: {(1,1,0),(0.5,−0.5,1)}\{(1,1,0), (0.5,-0.5,1)\}

Answer:The orthogonal set is {(1,1,0),(0.5,−0.5,1)}\{(1,1,0), (0.5,-0.5,1)\}

Exercise 2

Decide whether the matrix Q=(121212−12)Q = \begin{pmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} \end{pmatrix} is an orthogonal matrix.

orthogonal matrixmatrix propertiesdeterminant

Approach: Verify QTQ=IQ^T Q = I by computing the product; the determinant then gives a consistency check.

Steps:

  1. Compute QTQ^T: QT=(121212−12)Q^T = \begin{pmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} \end{pmatrix}

  2. Compute QTQQ^T Q: QTQ=(121212−12)(121212−12)Q^T Q = \begin{pmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} \end{pmatrix} \begin{pmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} \end{pmatrix}

    The result is: (12+12,12−12)=(1,0)( \frac{1}{2} + \frac{1}{2}, \frac{1}{2} - \frac{1}{2} ) = (1, 0) (12−12,12+12)=(0,1)( \frac{1}{2} - \frac{1}{2}, \frac{1}{2} + \frac{1}{2} ) = (0, 1)

    So QTQ=IQ^T Q = I

  3. Determinant: det⁡Q=12(−12)−12⋅12=−12−12=−1\det Q = \frac{1}{\sqrt{2}}(-\frac{1}{\sqrt{2}}) - \frac{1}{\sqrt{2}}\cdot\frac{1}{\sqrt{2}} = -\frac{1}{2} - \frac{1}{2} = -1

Answer:QQ is an orthogonal matrix

Summary

Symbols Used in This Article

SymbolTypePronunciation / NoteMeaning in This Article
(a⃗,b⃗)(\vec{a}, \vec{b})Inner productinner productThe inner product of the vectors a and b
∥a⃗∥\|\vec{a}\|Vector lengthvector normThe length of a vector
a⃗⊥b⃗\vec{a} \perp \vec{b}OrthogonalityorthogonalThe orthogonality relation between two vectors
QQOrthogonal matrixorthogonal matrixA matrix with the orthogonality property
Previous
Vector Spaces, Bases, and Dimension
Next
Comprehensive Exercises on Vectors

Found an issue with this content?

Tell us what could be improved.