Kuzee
Probability and Mathematical Statistics

Central Limit Theorem

Overview of the Central Limit Theorem

Intuitive Understanding of the Central Limit Theorem

Understanding: the central limit theorem describes the fact that as the number of independent random variables tends to infinity, the distribution of their sum tends to a normal distribution. It is one of the most important theorems in probability theory, and it explains why the normal distribution is so important in statistics.

Importance of the Central Limit Theorem

Importance:

  1. It provides a theoretical foundation for statistics
  2. It explains the ubiquity of the normal distribution
  3. It provides theoretical support for parameter estimation and hypothesis testing

De Moivre-Laplace Theorem

Statement of the De Moivre-Laplace Theorem

De Moivre-Laplace Theorem

Let Xn∼B(n,p)X_n \sim B(n,p). Then, when nn is large and pp is not close to 00 or 11: Xn−npnp(1−p)→dN(0,1)\frac{X_n - np}{\sqrt{np(1-p)}} \xrightarrow{d} N(0,1)

Proof
  1. Xn=∑i=1nYiX_n = \sum_{i=1}^n Y_i, where Yi∼B(1,p)Y_i \sim B(1,p)
  2. E(Yi)=pE(Y_i) = p, D(Yi)=p(1−p)D(Y_i) = p(1-p)
  3. By the Lindeberg-Lévy theorem, Xn−npnp(1−p)→dN(0,1)\frac{X_n - np}{\sqrt{np(1-p)}} \xrightarrow{d} N(0,1)

Significance of the De Moivre-Laplace Theorem

Significance: the binomial distribution can be approximated by the normal distribution, that is: B(n,p)≈N(np,np(1−p))B(n,p) \approx N(np, np(1-p))

Applications of the De Moivre-Laplace Theorem

Application 1: approximate computation of binomial probabilities When nn is large, the normal distribution can be used to approximately compute binomial probabilities.

Application 2: construction of confidence intervals Used to construct confidence intervals for the parameters of a binomial distribution.

Application 3: hypothesis testing Used for hypothesis testing of binomial distributions.

Lindeberg-Lévy Theorem

Statement of the Lindeberg-Lévy Theorem

Lindeberg-Lévy Theorem

Let X1,X2,…,Xn,…X_1, X_2, \dots, X_n, \dots be a sequence of independent and identically distributed random variables with E(Xi)=μE(X_i) = \mu and D(Xi)=σ2D(X_i) = \sigma^2. Then: ∑i=1nXi−nμσn→dN(0,1)\frac{\sum_{i=1}^n X_i - n\mu}{\sigma\sqrt{n}} \xrightarrow{d} N(0,1)

Proof
  1. Let Sn=∑i=1nXiS_n = \sum_{i=1}^n X_i
  2. E(Sn)=nμE(S_n) = n\mu, D(Sn)=nσ2D(S_n) = n\sigma^2
  3. Standardization: Sn−nμσn\frac{S_n - n\mu}{\sigma\sqrt{n}}
  4. Use the characteristic function or the moment generating function to show that the limiting distribution is the normal distribution

Significance of the Lindeberg-Lévy Theorem

Significance: the sum of independent and identically distributed random variables tends to the standard normal distribution after standardization.

Applications of the Lindeberg-Lévy Theorem

Application 1: distribution of the sample mean The standardized form of the sample mean tends to a normal distribution.

Application 2: approximate computation of sums When there are many terms, the normal distribution can be used to approximately compute the probability of a sum.

Application 3: statistical inference It provides a theoretical foundation for statistical inference.

Central Limit Theorem for the Sample Mean

Statement of the Central Limit Theorem for the Sample Mean

Central Limit Theorem for the Sample Mean

Let X1,X2,…,XnX_1, X_2, \dots, X_n be independent and identically distributed random variables with E(Xi)=μE(X_i) = \mu and D(Xi)=σ2D(X_i) = \sigma^2. Then: X‾n−μσ/n→dN(0,1)\frac{\overline{X}_n - \mu}{\sigma/\sqrt{n}} \xrightarrow{d} N(0,1)

Proof
  1. X‾n=1n∑i=1nXi\overline{X}_n = \frac{1}{n}\sum_{i=1}^n X_i
  2. E(X‾n)=μE(\overline{X}_n) = \mu, D(X‾n)=σ2nD(\overline{X}_n) = \frac{\sigma^2}{n}
  3. Standardization: X‾n−μσ/n\frac{\overline{X}_n - \mu}{\sigma/\sqrt{n}}
  4. By the Lindeberg-Lévy theorem, it tends to N(0,1)N(0,1)

Significance of the Central Limit Theorem for the Sample Mean

Significance: the standardized form of the sample mean tends to the standard normal distribution, that is: X‾n→dN(μ,σ2n)\overline{X}_n \xrightarrow{d} N(\mu, \frac{\sigma^2}{n})

Applications of the Central Limit Theorem for the Sample Mean

Application 1: confidence intervals Used to construct confidence intervals for the population mean.

Application 2: hypothesis testing Used for hypothesis testing of the population mean.

Application 3: determination of the sample size Used to determine the sample size required.

Generalizations of the Central Limit Theorem

Central Limit Theorem for Independent Non-identically Distributed Variables

Lindeberg Central Limit Theorem

Let X1,X2,…,XnX_1, X_2, \dots, X_n be independent random variables with E(Xi)=μiE(X_i) = \mu_i and D(Xi)=σi2D(X_i) = \sigma_i^2. If the Lindeberg condition is satisfied, then: ∑i=1nXi−∑i=1nμi∑i=1nσi2→dN(0,1)\frac{\sum_{i=1}^n X_i - \sum_{i=1}^n \mu_i}{\sqrt{\sum_{i=1}^n \sigma_i^2}} \xrightarrow{d} N(0,1)

Multidimensional Central Limit Theorem

Multidimensional Central Limit Theorem

Let X1,X2,…,XnX_1, X_2, \dots, X_n be independent and identically distributed kk-dimensional random vectors with E(Xi)=μE(X_i) = \mu and covariance matrix Σ\Sigma. Then: ∑i=1nXi−nμn→dN(0,Σ)\frac{\sum_{i=1}^n X_i - n\mu}{\sqrt{n}} \xrightarrow{d} N(0, \Sigma)

Applications of the Central Limit Theorem

Applications in Statistics

Application 1: parameter estimation The central limit theorem provides the theoretical foundation for parameter estimation.

Application 2: hypothesis testing The central limit theorem provides theoretical support for hypothesis testing.

Application 3: confidence intervals The central limit theorem provides theoretical guidance for the construction of confidence intervals.

Applications in Finance

Application 4: risk assessment The central limit theorem is used for risk assessment and portfolio analysis.

Application 5: option pricing The central limit theorem is used in option pricing models.

Applications in Engineering

Application 6: quality control The central limit theorem is used for sampling inspection in quality control.

Application 7: reliability analysis The central limit theorem is used for system reliability analysis.

Limitations of the Central Limit Theorem

Rate of Convergence

Limitation 1: the central limit theorem only states convergence; it does not state the rate of convergence.

Distribution Requirements

Limitation 2: the central limit theorem places certain requirements on the distribution of the random variables.

Practical Applications

Limitation 3: in practical applications the sample size is always finite.

Relation Between the Central Limit Theorem and the Law of Large Numbers

Connections

Connection 1: the central limit theorem deepens and extends the law of large numbers.

Connection 2: the law of large numbers states convergence, whereas the central limit theorem states the shape of the distribution.

Connection 3: the central limit theorem provides more precise information.

Differences

Difference 1: the law of large numbers concerns convergence, whereas the central limit theorem concerns the shape of the distribution.

Difference 2: the law of large numbers applies to any distribution, whereas the central limit theorem imposes requirements on the distribution.

Difference 3: the central limit theorem provides a more precise approximation.

Exercises

Exercise 1

For the binomial distribution B(n,p)B(n,p) with nn large and pp not close to 0 or 1, write the approximating normal distribution.

central limit theoremDe Moivre-Laplace theoremnormal approximation

Approach: Use the De Moivre-Laplace theorem.

Steps:

  1. The expectation of the binomial distribution B(n,p)B(n,p): E(X)=npE(X) = np
  2. The variance of the binomial distribution B(n,p)B(n,p): D(X)=np(1−p)D(X) = np(1-p)
  3. By the De Moivre-Laplace theorem, B(n,p)≈N(np,np(1−p))B(n,p) \approx N(np, np(1-p))

Answer: B(n,p)≈N(np,np(1−p))B(n,p) \approx N(np, np(1-p))

Exercise 2

Let X1,X2,…,XnX_1, X_2, \dots, X_n be independent and identically distributed random variables with E(Xi)=μE(X_i) = \mu and D(Xi)=σ2D(X_i) = \sigma^2. Find the asymptotic distribution of X‾n\overline{X}_n.

central limit theoremsample meanapplication of the theorem

Approach: Use the central limit theorem for the sample mean.

Steps:

  1. Sample mean: X‾n=1n∑i=1nXi\overline{X}_n = \frac{1}{n}\sum_{i=1}^n X_i
  2. By the central limit theorem, X‾n→dN(μ,σ2n)\overline{X}_n \xrightarrow{d} N(\mu, \frac{\sigma^2}{n})
  3. That is, the asymptotic distribution of X‾n\overline{X}_n is N(μ,σ2n)N(\mu, \frac{\sigma^2}{n})

Answer: N(μ,σ2n)N(\mu, \frac{\sigma^2}{n})

Exercise 3

Let X∼B(100,0.3)X \sim B(100, 0.3). Use the normal distribution to approximately compute P(25≤X≤35)P(25 \leq X \leq 35).

central limit theoremDe Moivre-Laplace theorembinomial distribution

Approach: Use the De Moivre-Laplace theorem for the approximation.

Steps:

  1. E(X)=100×0.3=30E(X) = 100 \times 0.3 = 30
  2. D(X)=100×0.3×0.7=21D(X) = 100 \times 0.3 \times 0.7 = 21
  3. σ=21≈4.58\sigma = \sqrt{21} \approx 4.58
  4. P(25≤X≤35)≈P(25−304.58≤Z≤35−304.58)P(25 \leq X \leq 35) \approx P(\frac{25-30}{4.58} \leq Z \leq \frac{35-30}{4.58})
  5. =P(−1.09≤Z≤1.09)=2Φ(1.09)−1≈0.724= P(-1.09 \leq Z \leq 1.09) = 2\Phi(1.09) - 1 \approx 0.724

Answer: approximately 0.7240.724

Exercise 4

Let X1,X2,…,XnX_1, X_2, \dots, X_n be independent and identically distributed random variables with E(Xi)=μE(X_i) = \mu and D(Xi)=σ2D(X_i) = \sigma^2. Prove that: ∑i=1nXi−nμσn→dN(0,1)\frac{\sum_{i=1}^n X_i - n\mu}{\sigma\sqrt{n}} \xrightarrow{d} N(0,1)

central limit theoremLindeberg-Lévy theoremindependent and identically distributed

Approach: Use the Lindeberg-Lévy theorem.

Steps:

  1. Let Sn=∑i=1nXiS_n = \sum_{i=1}^n X_i
  2. E(Sn)=nμE(S_n) = n\mu, D(Sn)=nσ2D(S_n) = n\sigma^2
  3. Standardization: Sn−nμσn\frac{S_n - n\mu}{\sigma\sqrt{n}}
  4. By the Lindeberg-Lévy theorem, it tends to N(0,1)N(0,1)

Answer: the proof is complete

Exercise 5

Explain the difference between the law of large numbers and the central limit theorem.

law of large numberscentral limit theoremdifference between the two

Approach: Compare the differences between the law of large numbers and the central limit theorem.

Steps:

  1. Law of large numbers: focuses on the sample mean converging to the expectation, and only states convergence
  2. Central limit theorem: focuses on the distribution of the sample mean tending to a normal distribution, and describes the shape of the distribution
  3. Law of large numbers: is the foundation of the central limit theorem
  4. Central limit theorem: provides more precise information than the law of large numbers

Answer: the law of large numbers focuses on the sample mean converging to the expectation, whereas the central limit theorem focuses on the distribution of the sum tending to a normal distribution

Summary

Symbols Used in This Article

Greek LetterPronunciationDescription
σ\sigmaSigma (sigma)denotes the standard deviation; in the normal distribution it represents the degree of dispersion of the distribution
μ\muMu (mu)denotes the expectation or mean; in the normal distribution it represents the center of the distribution
Previous
Law of Large Numbers
Next
Comprehensive Exercises on the Law of Large Numbers and the Central Limit Theorem

Found an issue with this content?

Tell us what could be improved.