Kuzee
Probability and Mathematical Statistics

Law of Large Numbers

Overview of the Law of Large Numbers

Intuitive Understanding of the Law of Large Numbers

Understanding: the law of large numbers describes the regularity that random phenomena exhibit as the number of trials tends to infinity. It tells us that over a large number of repeated trials, the frequency of a random event tends to its probability, and the average of a random variable tends to its expectation.

Importance of the Law of Large Numbers

Importance:

  1. It provides a theoretical foundation for probability theory
  2. It provides theoretical support for statistics
  3. It provides theoretical guidance for practical applications

Chebyshev’s Law of Large Numbers

Statement of Chebyshev’s Law of Large Numbers

Chebyshev's Law of Large Numbers

Let X1,X2,…,Xn,…X_1, X_2, \dots, X_n, \dots be a sequence of mutually independent random variables. If there exists a constant CC such that D(Xi)≤CD(X_i) \leq C for i=1,2,…i = 1, 2, \dots, then for any ε>0\varepsilon > 0: lim⁡n→∞P(∣1n∑i=1nXi−1n∑i=1nE(Xi)∣≥ε)=0\lim_{n \to \infty} P\left(\left|\frac{1}{n}\sum_{i=1}^n X_i - \frac{1}{n}\sum_{i=1}^n E(X_i)\right| \geq \varepsilon\right) = 0

Proof
  1. Let X‾n=1n∑i=1nXi\overline{X}_n = \frac{1}{n}\sum_{i=1}^n X_i and μn=1n∑i=1nE(Xi)\mu_n = \frac{1}{n}\sum_{i=1}^n E(X_i)
  2. E(X‾n)=μnE(\overline{X}_n) = \mu_n
  3. D(X‾n)=1n2∑i=1nD(Xi)≤nCn2=CnD(\overline{X}_n) = \frac{1}{n^2}\sum_{i=1}^n D(X_i) \leq \frac{nC}{n^2} = \frac{C}{n}
  4. By Chebyshev’s inequality: P(∣X‾n−μn∣≥ε)≤D(X‾n)ε2≤Cnε2P(|\overline{X}_n - \mu_n| \geq \varepsilon) \leq \frac{D(\overline{X}_n)}{\varepsilon^2} \leq \frac{C}{n\varepsilon^2}
  5. As n→∞n \to \infty, Cnε2→0\frac{C}{n\varepsilon^2} \to 0

Significance of Chebyshev’s Law of Large Numbers

Significance: the sample mean converges in probability to the mean of the expectations, that is: X‾n→P1n∑i=1nE(Xi)\overline{X}_n \xrightarrow{P} \frac{1}{n}\sum_{i=1}^n E(X_i)

Bernoulli’s Law of Large Numbers

Statement of Bernoulli’s Law of Large Numbers

Bernoulli's Law of Large Numbers

Let nAn_A be the number of occurrences of event AA in nn independent repeated trials, and let p=P(A)p = P(A). Then for any ε>0\varepsilon > 0: lim⁡n→∞P(∣nAn−p∣≥ε)=0\lim_{n \to \infty} P\left(\left|\frac{n_A}{n} - p\right| \geq \varepsilon\right) = 0

Proof
  1. Let Xi={1,the i-th trial, event A occurs0,the i-th trial, event A does not occurX_i = \begin{cases} 1, & \text{the } i \text{-th trial, event } A \text{ occurs} \\ 0, & \text{the } i \text{-th trial, event } A \text{ does not occur} \end{cases}
  2. X1,X2,…,XnX_1, X_2, \dots, X_n are independent and identically distributed random variables
  3. E(Xi)=pE(X_i) = p, D(Xi)=p(1−p)≤14D(X_i) = p(1-p) \leq \frac{1}{4}
  4. nAn=1n∑i=1nXi\frac{n_A}{n} = \frac{1}{n}\sum_{i=1}^n X_i
  5. By Chebyshev’s law of large numbers, nAn→Pp\frac{n_A}{n} \xrightarrow{P} p

Significance of Bernoulli’s Law of Large Numbers

Significance: over a large number of independent repeated trials, the frequency of occurrence of an event converges in probability to its probability.

Applications of Bernoulli’s Law of Large Numbers

Application 1: the Monte Carlo method Estimating probabilities through a large number of random trials.

Application 2: quality control Estimating product quality through extensive sampling.

Application 3: opinion polls Estimating public opinion through extensive surveys.

Khinchin’s Law of Large Numbers

Statement of Khinchin’s Law of Large Numbers

Khinchin's Law of Large Numbers

Let X1,X2,…,Xn,…X_1, X_2, \dots, X_n, \dots be a sequence of independent and identically distributed random variables with E(Xi)=μE(X_i) = \mu. Then for any ε>0\varepsilon > 0: lim⁡n→∞P(∣1n∑i=1nXi−μ∣≥ε)=0\lim_{n \to \infty} P\left(\left|\frac{1}{n}\sum_{i=1}^n X_i - \mu\right| \geq \varepsilon\right) = 0

Proof
  1. Since X1,X2,…,Xn,…X_1, X_2, \dots, X_n, \dots are independent and identically distributed, D(Xi)=σ2D(X_i) = \sigma^2 exists
  2. By Chebyshev’s law of large numbers, 1n∑i=1nXi→Pμ\frac{1}{n}\sum_{i=1}^n X_i \xrightarrow{P} \mu

Significance of Khinchin’s Law of Large Numbers

Significance: the sample mean of independent and identically distributed random variables converges in probability to its expectation.

Applications of Khinchin’s Law of Large Numbers

Application 1: parameter estimation The sample mean is a consistent estimator of the population expectation.

Application 2: Monte Carlo integration Estimating the value of an integral through random sampling.

Application 3: simulation Estimating expected values through a large number of simulations.

Strong Law of Large Numbers

Statement of the Strong Law of Large Numbers

Strong Law of Large Numbers

Let X1,X2,…,Xn,…X_1, X_2, \dots, X_n, \dots be a sequence of independent and identically distributed random variables with E(Xi)=μE(X_i) = \mu. Then: P(lim⁡n→∞1n∑i=1nXi=μ)=1P\left(\lim_{n \to \infty} \frac{1}{n}\sum_{i=1}^n X_i = \mu\right) = 1

Difference Between the Strong and the Weak Law of Large Numbers

Difference:

  • Weak law of large numbers: convergence in probability, P(∣X‾n−μ∣≥ε)→0P(|\overline{X}_n - \mu| \geq \varepsilon) \to 0
  • Strong law of large numbers: almost sure convergence, P(X‾n→μ)=1P(\overline{X}_n \to \mu) = 1

Significance of the Strong Law of Large Numbers

Significance: the strong law of large numbers is stronger than the weak law; it guarantees that the sample mean converges almost surely to the expectation.

Applications of the Law of Large Numbers

Applications in Statistics

Application 1: parameter estimation The law of large numbers provides the theoretical foundation for parameter estimation.

Application 2: hypothesis testing The law of large numbers provides theoretical support for hypothesis testing.

Application 3: regression analysis The law of large numbers provides theoretical guidance for regression analysis.

Applications in Finance

Application 4: risk assessment Estimating risk from a large amount of historical data.

Application 5: portfolio optimization Optimizing portfolios through a large number of simulations.

Applications in Engineering

Application 6: quality control Controlling product quality through extensive sampling.

Application 7: reliability analysis Analyzing system reliability through a large number of trials.

Limitations of the Law of Large Numbers

Rate of Convergence

Limitation 1: the law of large numbers only states convergence; it does not state the rate of convergence.

Distribution Requirements

Limitation 2: different laws of large numbers place different requirements on the distribution of the random variables.

Practical Applications

Limitation 3: in practical applications the number of trials is always finite.

Exercises

Exercise 1

Explain the practical significance of Bernoulli’s law of large numbers.

law of large numbersBernoulli's law of large numbersintuitive meaning

Approach: Explain the intuitive meaning of Bernoulli’s law of large numbers.

Steps:

  1. Bernoulli’s law of large numbers shows that over a large number of independent repeated trials, the frequency of occurrence of an event converges in probability to its probability
  2. This means that when the number of trials is sufficiently large, the frequency of occurrence of the event is very close to its true probability
  3. This provides the theoretical foundation for the statistical definition of probability

Answer: over a large number of independent repeated trials, the frequency of occurrence of an event tends to the probability

Exercise 2

Let XiX_i be independent and identically distributed with E(Xi)=μ,D(Xi)=σ2E(X_i)=\mu, D(X_i)=\sigma^2. Write the central limit theorem for the sample mean.

central limit theoremsample meanstatement of the theorem

Approach: Write the central limit theorem for the sample mean.

Steps:

  1. Sample mean: X‾n=1n∑i=1nXi\overline{X}_n = \frac{1}{n}\sum_{i=1}^n X_i
  2. Standardization: X‾n−μσ/n\frac{\overline{X}_n - \mu}{\sigma/\sqrt{n}}
  3. Central limit theorem: X‾n−μσ/n→dN(0,1)\frac{\overline{X}_n - \mu}{\sigma/\sqrt{n}} \xrightarrow{d} N(0,1)

Answer: X‾n−μσ/n→dN(0,1)\frac{\overline{X}_n - \mu}{\sigma/\sqrt{n}} \xrightarrow{d} N(0,1)

Exercise 3

Let X1,X2,…,XnX_1, X_2, \dots, X_n be independent and identically distributed random variables with E(Xi)=μE(X_i) = \mu and D(Xi)=σ2D(X_i) = \sigma^2. Prove that: X‾n→Pμ\overline{X}_n \xrightarrow{P} \mu

Chebyshev's inequalityproofinequality for concentration

Approach: Prove it using Chebyshev’s inequality.

Steps:

  1. E(X‾n)=μE(\overline{X}_n) = \mu
  2. D(X‾n)=σ2nD(\overline{X}_n) = \frac{\sigma^2}{n}
  3. By Chebyshev’s inequality: P(∣X‾n−μ∣≥ε)≤σ2nε2P(|\overline{X}_n - \mu| \geq \varepsilon) \leq \frac{\sigma^2}{n\varepsilon^2}
  4. As n→∞n \to \infty, σ2nε2→0\frac{\sigma^2}{n\varepsilon^2} \to 0
  5. Hence X‾n→Pμ\overline{X}_n \xrightarrow{P} \mu

Answer: the proof is complete

Exercise 4

Let X1,X2,…,XnX_1, X_2, \dots, X_n be independent and identically distributed random variables with E(Xi)=μE(X_i) = \mu and D(Xi)=σ2D(X_i) = \sigma^2. Find the expectation and the variance of X‾n\overline{X}_n.

mathematical expectationvarianceapplication of properties

Approach: Use the properties of expectation and variance.

Steps:

  1. E(X‾n)=E(1n∑i=1nXi)=1n∑i=1nE(Xi)=μE(\overline{X}_n) = E(\frac{1}{n}\sum_{i=1}^n X_i) = \frac{1}{n}\sum_{i=1}^n E(X_i) = \mu
  2. D(X‾n)=D(1n∑i=1nXi)=1n2∑i=1nD(Xi)=σ2nD(\overline{X}_n) = D(\frac{1}{n}\sum_{i=1}^n X_i) = \frac{1}{n^2}\sum_{i=1}^n D(X_i) = \frac{\sigma^2}{n}

Answer: E(X‾n)=μE(\overline{X}_n) = \mu, D(X‾n)=σ2nD(\overline{X}_n) = \frac{\sigma^2}{n}

Exercise 5

Explain the difference between the law of large numbers and the central limit theorem.

law of large numberscentral limit theoremdifference between the two

Approach: Compare the differences between the law of large numbers and the central limit theorem.

Steps:

  1. Law of large numbers: focuses on the sample mean converging to the expectation, and only states convergence
  2. Central limit theorem: focuses on the distribution of the sample mean tending to a normal distribution, and describes the shape of the distribution
  3. Law of large numbers: is the foundation of the central limit theorem
  4. Central limit theorem: provides more precise information than the law of large numbers

Answer: the law of large numbers focuses on the sample mean converging to the expectation, whereas the central limit theorem focuses on the distribution of the sum tending to a normal distribution

Summary

Symbols Used in This Article

Greek LetterPronunciationDescription
ε\varepsilonEpsilon (epsilon)denotes an arbitrary positive number; in the law of large numbers it represents the degree of deviation
μ\muMu (mu)denotes the population mean or the expectation
σ\sigmaSigma (sigma)denotes the population standard deviation
Previous
Chebyshev's Inequality
Next
Central Limit Theorem

Found an issue with this content?

Tell us what could be improved.