Method of Moments and Maximum Likelihood Estimation

Method of Moments

Basic Idea of the Method of Moments

Idea: the method of moments estimates parameters on the principle that the sample moments equal the population moments. Its core idea is to use the sample moments to estimate the population moments, and then to estimate the parameters through the relation between the population moments and the parameters.

Theoretical Basis of the Method of Moments

Theoretical basis: by the law of large numbers, when the sample size is sufficiently large the sample moments converge in probability to the population moments, that is: $\frac{1}{n}\sum_{i=1}^n X_i^k \xrightarrow{P} E(X^k)$

Steps of the Method of Moments

Steps:

1. Determine the number of parameters: suppose the population distribution has $k$ unknown parameters $\theta_1, \theta_2, \dots, \theta_k$
2. Compute the population moments: compute the first through the $k$-th population moments, expressed in terms of the parameters
3. Compute the sample moments: compute the first through the $k$-th sample moments
4. Establish the equations: equate the sample moments to the population moments to establish $k$ equations
5. Solve the equations: solve these $k$ equations to obtain the estimators of the parameters

Advantages and Disadvantages of the Method of Moments

Advantages:

• Simple to compute and easy to understand
• Does not require knowing the specific form of the population distribution
• Applicable to all kinds of distributions

Disadvantages:

• May not yield the optimal estimator
• For some distributions the moment estimator may not exist
• The computation of high-order moments may be unstable

Example of the Method of Moments

Example 1: Parameter Estimation for the Normal Distribution

Population: $X \sim N(\mu, \sigma^2)$

Steps:

1. First population moment: $E(X) = \mu$
2. Second population moment: $E(X^2) = \mu^2 + \sigma^2$
3. First sample moment: $\frac{1}{n}\sum_{i=1}^n X_i = \overline{X}$
4. Second sample moment: $\frac{1}{n}\sum_{i=1}^n X_i^2$
5. Establish the equations:
   • $\overline{X} = \mu$
   • $\frac{1}{n}\sum_{i=1}^n X_i^2 = \mu^2 + \sigma^2$
6. Solving gives:
   • $\hat{\mu} = \overline{X}$
   • $\hat{\sigma}^2 = \frac{1}{n}\sum_{i=1}^n X_i^2 - \overline{X}^2 = \frac{1}{n}\sum_{i=1}^n (X_i - \overline{X})^2$

Example 2: Parameter Estimation for the Poisson Distribution

Population: $X \sim Poisson(\lambda)$

Steps:

1. First population moment: $E(X) = \lambda$
2. First sample moment: $\frac{1}{n}\sum_{i=1}^n X_i = \overline{X}$
3. Establish the equation: $\overline{X} = \lambda$
4. Solving gives: $\hat{\lambda} = \overline{X}$

Example 3: Parameter Estimation for the Exponential Distribution

Population: $X \sim Exp(\lambda)$

Steps:

1. First population moment: $E(X) = \frac{1}{\lambda}$
2. First sample moment: $\frac{1}{n}\sum_{i=1}^n X_i = \overline{X}$
3. Establish the equation: $\overline{X} = \frac{1}{\lambda}$
4. Solving gives: $\hat{\lambda} = \frac{1}{\overline{X}}$

Example 4: Parameter Estimation for the Uniform Distribution

Population: $X \sim U(a,b)$

Steps:

1. First population moment: $E(X) = \frac{a+b}{2}$
2. Second population moment: $E(X^2) = \frac{a^2 + ab + b^2}{3}$
3. First sample moment: $\frac{1}{n}\sum_{i=1}^n X_i = \overline{X}$
4. Second sample moment: $\frac{1}{n}\sum_{i=1}^n X_i^2$
5. Establish the equations:
   • $\overline{X} = \frac{a+b}{2}$
   • $\frac{1}{n}\sum_{i=1}^n X_i^2 = \frac{a^2 + ab + b^2}{3}$
6. Solving gives:
   • $\hat{a} = \overline{X} - \sqrt{3(\frac{1}{n}\sum_{i=1}^n X_i^2 - \overline{X}^2)}$
   • $\hat{b} = \overline{X} + \sqrt{3(\frac{1}{n}\sum_{i=1}^n X_i^2 - \overline{X}^2)}$

Maximum Likelihood Estimation

Basic Idea of Maximum Likelihood Estimation

Idea: maximum likelihood estimation is based on the thought that “the parameter value most likely to have produced the observed sample is the true value”. It chooses as the estimator the parameter value that makes the observed sample values most likely to occur.

Likelihood Function

Steps of Maximum Likelihood Estimation

Steps:

1. Construct the likelihood function: $L(\theta) = \prod_{i=1}^n f(X_i;\theta)$
2. Take the logarithm: $\ln L(\theta) = \sum_{i=1}^n \ln f(X_i;\theta)$
3. Differentiate: $\frac{\partial \ln L(\theta)}{\partial \theta} = 0$
4. Solve the equation: solve the equation above to obtain the maximum likelihood estimator

Properties of Maximum Likelihood Estimation

Property 1: invariance If $\hat{\theta}$ is the maximum likelihood estimator of $\theta$ and $g(\theta)$ is a function of $\theta$, then $g(\hat{\theta})$ is the maximum likelihood estimator of $g(\theta)$.

Property 2: asymptotic properties Under regularity conditions, the maximum likelihood estimator has:

• Consistency: $\hat{\theta}_n \xrightarrow{P} \theta$
• Asymptotic normality: $\sqrt{n}(\hat{\theta}_n - \theta) \xrightarrow{d} N(0, I^{-1}(\theta))$
• Asymptotic efficiency: its variance attains the Cramér-Rao lower bound

Examples of Maximum Likelihood Estimation

Example 1: Parameter Estimation for the Normal Distribution

Population: $X \sim N(\mu, \sigma^2)$

Steps:

1. Likelihood function: $L(\mu, \sigma^2) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi}\sigma} e^{-\frac{(X_i-\mu)^2}{2\sigma^2}}$
2. Log-likelihood function: $\ln L(\mu, \sigma^2) = -\frac{n}{2}\ln(2\pi) - \frac{n}{2}\ln(\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^n (X_i-\mu)^2$
3. Differentiate with respect to $\mu$: $\frac{\partial \ln L}{\partial \mu} = \frac{1}{\sigma^2}\sum_{i=1}^n (X_i-\mu) = 0$
4. Solving gives: $\hat{\mu} = \overline{X}$
5. Differentiate with respect to $\sigma^2$: $\frac{\partial \ln L}{\partial \sigma^2} = -\frac{n}{2\sigma^2} + \frac{1}{2\sigma^4}\sum_{i=1}^n (X_i-\mu)^2 = 0$
6. Solving gives: $\hat{\sigma}^2 = \frac{1}{n}\sum_{i=1}^n (X_i-\overline{X})^2$

Example 2: Parameter Estimation for the Poisson Distribution

Population: $X \sim Poisson(\lambda)$

Steps:

1. Likelihood function: $L(\lambda) = \prod_{i=1}^n \frac{\lambda^{X_i} e^{-\lambda}}{X_i!}$
2. Log-likelihood function: $\ln L(\lambda) = \sum_{i=1}^n (X_i \ln \lambda - \lambda - \ln X_i!) = \ln \lambda \sum_{i=1}^n X_i - n\lambda - \sum_{i=1}^n \ln X_i!$
3. Differentiate: $\frac{\partial \ln L}{\partial \lambda} = \frac{1}{\lambda}\sum_{i=1}^n X_i - n = 0$
4. Solving gives: $\hat{\lambda} = \overline{X}$

Example 3: Parameter Estimation for the Exponential Distribution

Population: $X \sim Exp(\lambda)$

Steps:

1. Likelihood function: $L(\lambda) = \prod_{i=1}^n \lambda e^{-\lambda X_i} = \lambda^n e^{-\lambda \sum_{i=1}^n X_i}$
2. Log-likelihood function: $\ln L(\lambda) = n \ln \lambda - \lambda \sum_{i=1}^n X_i$
3. Differentiate: $\frac{\partial \ln L}{\partial \lambda} = \frac{n}{\lambda} - \sum_{i=1}^n X_i = 0$
4. Solving gives: $\hat{\lambda} = \frac{n}{\sum_{i=1}^n X_i} = \frac{1}{\overline{X}}$

Example 4: Parameter Estimation for the Bernoulli Distribution

Population: $X \sim B(1,p)$

Steps:

1. Likelihood function: $L(p) = \prod_{i=1}^n p^{X_i}(1-p)^{1-X_i} = p^{\sum_{i=1}^n X_i}(1-p)^{n-\sum_{i=1}^n X_i}$
2. Log-likelihood function: $\ln L(p) = \sum_{i=1}^n X_i \ln p + (n-\sum_{i=1}^n X_i) \ln(1-p)$
3. Differentiate: $\frac{\partial \ln L}{\partial p} = \frac{\sum_{i=1}^n X_i}{p} - \frac{n-\sum_{i=1}^n X_i}{1-p} = 0$
4. Solving gives: $\hat{p} = \frac{1}{n}\sum_{i=1}^n X_i = \overline{X}$

Comparison of the Two Methods

Computational Complexity

Method of moments: simple to compute, only the sample moments need to be computed Maximum likelihood estimation: relatively more complex to compute, requiring differentiation and solving equations

Estimation Performance

Method of moments: may not yield the optimal estimator Maximum likelihood estimation: yields the optimal estimator under regularity conditions

Applicability

Method of moments: applicable to all kinds of distributions, without needing to know the specific form Maximum likelihood estimation: requires knowing the specific form of the distribution

Asymptotic Properties

Method of moments: has consistency, but does not necessarily have asymptotic efficiency Maximum likelihood estimation: has consistency, asymptotic normality, and asymptotic efficiency

Exercises

Exercise 1

Estimate the parameter by the method of moments: given that the sample $x_1, x_2, \dots, x_n$ comes from $X \sim Poisson(\lambda)$.

Exercise 2

Estimate the parameters by maximum likelihood estimation: given that the sample $x_1, x_2, \dots, x_n$ comes from $X \sim N(\mu, \sigma^2)$.

Exercise 3

Let $X_1, X_2, \dots, X_n$ be a sample from the population $X \sim U(0, \theta)$. Estimate the parameter $\theta$ by the method of moments.

Exercise 4

Let $X_1, X_2, \dots, X_n$ be a sample from the population $X \sim Exp(\lambda)$. Estimate the parameter $\lambda$ by maximum likelihood estimation.

Exercise 5

Compare the advantages and disadvantages of the method of moments and maximum likelihood estimation.

---

Summary

Symbols Used in This Article

---