Point Estimation and Estimators

Basic Concepts of Point Estimation

Definition of Point Estimation

Difference Between an Estimator and an Estimate

Estimator: $\hat{\theta} = \hat{\theta}(X_1, X_2, \dots, X_n)$ is a random variable

Estimate: $\hat{\theta} = \hat{\theta}(x_1, x_2, \dots, x_n)$ is a concrete numerical value

Intuitive Understanding of Point Estimation

Understanding: point estimation uses sample information to estimate the concrete numerical value of a population parameter. Since the sample is random, the estimator is random as well, so we need criteria for evaluating the quality of an estimator.

Criteria for Evaluating Estimators

Unbiasedness

Significance: unbiasedness means that the expected value of the estimator equals the true value, so in the long run the estimate does not systematically deviate from the true value.

Example 1: the sample mean $\overline{X} = \frac{1}{n}\sum_{i=1}^n X_i$ is an unbiased estimator of the population mean $\mu$.

Proof: $E(\overline{X}) = E(\frac{1}{n}\sum_{i=1}^n X_i) = \frac{1}{n}\sum_{i=1}^n E(X_i) = \frac{1}{n} \cdot n\mu = \mu$

Efficiency

Significance: efficiency reflects the precision of an estimator; the smaller the variance, the more accurate the estimator.

Example 2: for a normal population $N(\mu, \sigma^2)$, the sample mean $\overline{X}$ is more efficient than a single observation $X_1$.

Proof: $Var(\overline{X}) = \frac{\sigma^2}{n} < \sigma^2 = Var(X_1)$

Consistency

Significance: consistency means that as the sample size increases, the estimator gets closer and closer to the true value.

Example 3: the sample mean $\overline{X}_n$ is a consistent estimator of the population mean $\mu$.

Proof: by the law of large numbers, $\overline{X}_n \xrightarrow{P} \mu$

Asymptotic Normality

Significance: asymptotic normality provides the theoretical foundation for constructing confidence intervals and conducting hypothesis tests.

Estimators for the Parameters of Common Distributions

Parameter Estimation for the Normal Distribution

Population: $X \sim N(\mu, \sigma^2)$

Mean estimation:

• Unbiased estimator: $\hat{\mu} = \overline{X} = \frac{1}{n}\sum_{i=1}^n X_i$
• Properties: unbiased, efficient, consistent

Variance estimation:

• Unbiased estimator: $S^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \overline{X})^2$
• Maximum likelihood estimator: $\hat{\sigma}^2 = \frac{1}{n}\sum_{i=1}^n (X_i - \overline{X})^2$

Parameter Estimation for the Poisson Distribution

Population: $X \sim Poisson(\lambda)$

Parameter estimation:

• Unbiased estimator: $\hat{\lambda} = \overline{X} = \frac{1}{n}\sum_{i=1}^n X_i$
• Properties: unbiased, efficient, consistent

Parameter Estimation for the Exponential Distribution

Population: $X \sim Exp(\lambda)$

Parameter estimation:

• Unbiased estimator: $\hat{\lambda} = \frac{1}{\overline{X}}$
• Maximum likelihood estimator: $\hat{\lambda} = \frac{1}{\overline{X}}$

Parameter Estimation for the Uniform Distribution

Population: $X \sim U(a,b)$

Parameter estimation:

• Estimator of $a$: $\hat{a} = \min(X_1, X_2, \dots, X_n)$
• Estimator of $b$: $\hat{b} = \max(X_1, X_2, \dots, X_n)$

Methods for Constructing Estimators

Method of Moments

Idea: equate the sample moments to the population moments and solve the resulting equation to obtain the estimator of the parameter.

Steps:

1. Compute the population moments (expressed in terms of the parameters)
2. Compute the sample moments
3. Equate the sample moments to the population moments
4. Solve the equation to obtain the estimator of the parameter

Example 4: let $X \sim Poisson(\lambda)$; estimate $\lambda$ by the method of moments.

Solution:

1. First population moment: $E(X) = \lambda$
2. First sample moment: $\frac{1}{n}\sum_{i=1}^n X_i = \overline{X}$
3. Set $\overline{X} = \lambda$
4. We obtain $\hat{\lambda} = \overline{X}$

Maximum Likelihood Estimation

Idea: choose as the estimator the parameter value that makes the observed sample values most likely to occur.

Steps:

1. Construct the likelihood function $L(\theta) = \prod_{i=1}^n f(x_i;\theta)$
2. Take the logarithm to obtain the log-likelihood function $\ln L(\theta)$
3. Differentiate with respect to the parameter and set the derivative to zero
4. Solve the equation to obtain the maximum likelihood estimator

Example 5: let $X \sim N(\mu, \sigma^2)$; estimate $\mu$ and $\sigma^2$ by maximum likelihood estimation.

Solution:

1. Likelihood function: $L(\mu, \sigma^2) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi}\sigma} e^{-\frac{(x_i-\mu)^2}{2\sigma^2}}$
2. Log-likelihood function: $\ln L(\mu, \sigma^2) = -\frac{n}{2}\ln(2\pi) - \frac{n}{2}\ln(\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^n (x_i-\mu)^2$
3. Differentiate with respect to $\mu$: $\frac{\partial \ln L}{\partial \mu} = \frac{1}{\sigma^2}\sum_{i=1}^n (x_i-\mu) = 0$
4. Solving gives: $\hat{\mu} = \overline{X}$
5. Differentiate with respect to $\sigma^2$: $\frac{\partial \ln L}{\partial \sigma^2} = -\frac{n}{2\sigma^2} + \frac{1}{2\sigma^4}\sum_{i=1}^n (x_i-\mu)^2 = 0$
6. Solving gives: $\hat{\sigma}^2 = \frac{1}{n}\sum_{i=1}^n (x_i-\overline{X})^2$

Comparison of the Properties of Estimators

Unbiasedness and Efficiency

Relation: unbiasedness and efficiency are two different criteria; an estimator may be unbiased but not efficient, and may be efficient but biased.

Example 6: for a normal population $N(\mu, \sigma^2)$:

• $\overline{X}$ is an unbiased estimator of $\mu$
• $\frac{1}{n}\sum_{i=1}^n X_i^2$ is an unbiased estimator of $\sigma^2 + \mu^2$, but not an unbiased estimator of $\sigma^2$

Consistency and Efficiency

Relation: consistency is an asymptotic property, while efficiency is a finite-sample property. An estimator may be consistent but not efficient, and may be efficient but not consistent.

Asymptotically Efficient Estimator

Exercises

Exercise 1

Let the population be $X \sim N(\mu, \sigma^2)$, with sample mean $\overline{X}$ and sample variance $S^2$. Write the unbiased estimators of $\mu$ and $\sigma^2$.

Exercise 2

Let $X_1, X_2, \dots, X_n$ be a sample from the population $X \sim Poisson(\lambda)$. Prove that the sample mean $\overline{X}$ is an unbiased estimator of $\lambda$.

Exercise 3

Let $X_1, X_2, \dots, X_n$ be a sample from the population $X \sim N(\mu, \sigma^2)$. Compare $\frac{1}{n}\sum_{i=1}^n (X_i - \overline{X})^2$ and $\frac{1}{n-1}\sum_{i=1}^n (X_i - \overline{X})^2$, and determine which one is the unbiased estimator of $\sigma^2$.

Exercise 4

Let $X_1, X_2, \dots, X_n$ be a sample from the population $X \sim U(0, \theta)$. Estimate the parameter $\theta$ by the method of moments.

Exercise 5

Consider the estimators $\hat{\theta}_1, \hat{\theta}_2$. If $Var(\hat{\theta}_1)<Var(\hat{\theta}_2)$, which one is more efficient?

---

Summary

Symbols Used in This Article

---