Covariance and the Correlation Coefficient

Definition of the Covariance

Mathematical Definition of the Covariance

Computation Formula for the Covariance

Discrete random variables: $Cov(X,Y) = \sum_{i,j} (x_i - E(X))(y_j - E(Y))P(X = x_i, Y = y_j)$

Continuous random variables: $Cov(X,Y) = \int_{-\infty}^{+\infty} \int_{-\infty}^{+\infty} (x - E(X))(y - E(Y))f(x,y) dx dy$

Simplified Computation Formula for the Covariance

Proof: $Cov(X,Y) = E[(X - E(X))(Y - E(Y))] = E(XY - XE(Y) - YE(X) + E(X)E(Y))$ $= E(XY) - E(X)E(Y) - E(Y)E(X) + E(X)E(Y) = E(XY) - E(X)E(Y)$

Intuitive Understanding of the Covariance

Understanding: the covariance reflects the degree of linear correlation between two random variables. A positive value indicates a positive correlation, a negative value indicates a negative correlation, and a zero value indicates no correlation.

Examples of the Covariance

Example 1: suppose the joint distribution law of $X$ and $Y$ is:

Find $Cov(X,Y)$.

Solution:

1. $E(X) = 0 \times 0.5 + 1 \times 0.5 = 0.5$
2. $E(Y) = 0 \times 0.5 + 1 \times 0.5 = 0.5$
3. $E(XY) = 0 \times 0 \times 0.3 + 0 \times 1 \times 0.2 + 1 \times 0 \times 0.2 + 1 \times 1 \times 0.3 = 0.3$
4. $Cov(X,Y) = E(XY) - E(X)E(Y) = 0.3 - 0.5 \times 0.5 = 0.05$

Properties of the Covariance

Symmetry

Property 1: $Cov(X,Y) = Cov(Y,X)$

Proof: $Cov(X,Y) = E[(X - E(X))(Y - E(Y))] = E[(Y - E(Y))(X - E(X))] = Cov(Y,X)$

Linearity

Property 2: $Cov(aX + b, cY + d) = ac \cdot Cov(X,Y)$, where $a, b, c, d$ are constants.

Proof: $Cov(aX + b, cY + d) = E[(aX + b - E(aX + b))(cY + d - E(cY + d))]$ $= E[(aX - aE(X))(cY - cE(Y))] = ac \cdot E[(X - E(X))(Y - E(Y))] = ac \cdot Cov(X,Y)$

Additivity

Property 3: $Cov(X_1 + X_2, Y) = Cov(X_1, Y) + Cov(X_2, Y)$

Proof: $Cov(X_1 + X_2, Y) = E[(X_1 + X_2 - E(X_1 + X_2))(Y - E(Y))]$ $= E[(X_1 - E(X_1) + X_2 - E(X_2))(Y - E(Y))]$ $= E[(X_1 - E(X_1))(Y - E(Y))] + E[(X_2 - E(X_2))(Y - E(Y))] = Cov(X_1, Y) + Cov(X_2, Y)$

Relation with the Variance

Property 4: $Cov(X, X) = D(X)$

Proof: $Cov(X, X) = E[(X - E(X))^2] = D(X)$

Covariance of Independent Random Variables

Property 5: if $X$ and $Y$ are independent, then $Cov(X, Y) = 0$

Proof: if $X$ and $Y$ are independent, then $E(XY) = E(X)E(Y)$, so $Cov(X, Y) = E(XY) - E(X)E(Y) = 0$

Note: the converse does not hold, that is, $Cov(X, Y) = 0$ does not necessarily mean that $X$ and $Y$ are independent.

Definition of the Correlation Coefficient

Mathematical Definition of the Correlation Coefficient

Properties of the Correlation Coefficient

Property 1: $-1 \leq \rho_{XY} \leq 1$

Proof: use the Cauchy-Schwarz inequality: $|E[(X - E(X))(Y - E(Y))]| \leq \sqrt{E[(X - E(X))^2]E[(Y - E(Y))^2]}$ Hence $|Cov(X,Y)| \leq \sigma_X \sigma_Y$, that is, $|\rho_{XY}| \leq 1$.

Property 2: $\rho_{XY} = 1$ if and only if $Y = aX + b$, where $a > 0$

Property 3: $\rho_{XY} = -1$ if and only if $Y = aX + b$, where $a < 0$

Property 4: $\rho_{XY} = 0$ if and only if $Cov(X,Y) = 0$

Intuitive Understanding of the Correlation Coefficient

Understanding:

• $\rho_{XY} = 1$: perfect positive correlation
• $\rho_{XY} = -1$: perfect negative correlation
• $\rho_{XY} = 0$: no correlation
• $0 < \rho_{XY} < 1$: positive correlation
• $-1 < \rho_{XY} < 0$: negative correlation

Examples of the Correlation Coefficient

Example 2: continuing Example 1, find $\rho_{XY}$.

Solution:

1. $D(X) = E(X^2) - [E(X)]^2 = 0.5 - 0.25 = 0.25$
2. $D(Y) = E(Y^2) - [E(Y)]^2 = 0.5 - 0.25 = 0.25$
3. $\sigma_X = \sigma_Y = \sqrt{0.25} = 0.5$
4. $\rho_{XY} = \frac{Cov(X,Y)}{\sigma_X \sigma_Y} = \frac{0.05}{0.5 \times 0.5} = 0.2$

Covariance Matrix

Covariance Matrix of a Two-Dimensional Random Variable

Properties of the Covariance Matrix

Property 1: the covariance matrix is a symmetric matrix

Property 2: the covariance matrix is a positive semidefinite matrix

Property 3: if $X$ and $Y$ are independent, then the covariance matrix is a diagonal matrix

Covariance of Multidimensional Random Variables

Covariance Matrix of a Multidimensional Random Variable

Properties for Multidimensional Random Variables

Property 1: the covariance matrix is a symmetric matrix

Property 2: the covariance matrix is a positive semidefinite matrix

Property 3: if $X_1, X_2, \dots, X_n$ are pairwise independent, then the covariance matrix is a diagonal matrix

Applications of the Covariance and the Correlation Coefficient

Applications in Statistics

Application 1: linear regression The correlation coefficient is an important indicator for measuring the strength of a linear relationship.

Application 2: principal component analysis The covariance matrix is the foundation of principal component analysis.

Application 3: multivariate statistical analysis The covariance matrix is the core of multivariate statistical analysis.

Applications in Finance

Application 4: portfolio theory The covariance matrix is used to compute the risk of a portfolio.

Application 5: risk management The correlation coefficient is used to evaluate the effect of risk diversification among different assets.

Exercises

Exercise 1

Given $E(X)=1, E(Y)=2, Cov(X,Y)=3, \sigma_X=2, \sigma_Y=1$, find $\rho_{XY}$.

Exercise 2

Given that the joint distribution law of the random variables $X$ and $Y$ is:

Find $Cov(X,Y)$.

Exercise 3

Given that $X$ and $Y$ are independent, $D(X) = 4$, and $D(Y) = 9$, find $D(X + Y)$.

Exercise 4

Given that the correlation coefficient of the random variables $X$ and $Y$ is $0.8$, $D(X) = 4$, and $D(Y) = 9$, find $Cov(X,Y)$.

Exercise 5

Prove that if $X$ and $Y$ are independent, then $\rho_{XY} = 0$.

---

Summary

Symbols Used in This Article

---