Estimation

How do we estimate an unknown parameter well, and how well can we possibly do?

1. Likelihood and the maximum likelihood estimator (Def. 1.1), with standard examples (Example 1.2) and invariance (Prop. 1.3).
2. Bias, variance and mean squared error (Def. 1.4, Prop. 1.5).
3. Sufficient statistics (Def. 1.6) and the factorisation theorem (Thm 1.7).
4. Fisher information (Def. 1.8) and the Cramér–Rao lower bound (Thm 1.9).

For an i.i.d. sample $\mathbf{x} = (x_1, \dots, x_n)$ from $f(x;\theta)$, the likelihood is $L(\theta) = \prod_if(x_i;\theta)$ and the log-likelihood $\ell(\theta) = \sum\log f(x_i;\theta)$. A maximum likelihood estimate $\hat\theta$ maximises $L$ over $\Theta$; often it solves the score equation $\ell'(\theta) = 0$ (check it is a maximum and not on the boundary).

For i.i.d. $N(\mu, \sigma^2)$: $\ell(\mu, \sigma^2) = -\frac n2\log(2\pi\sigma^2) - \frac1{2\sigma^2}\sum(x_i - \mu)^2$. Setting partial derivatives to zero gives $\hat\mu = \bar x$ and $\hat\sigma^2 = \frac1n\sum(x_i - \bar x)^2$. Note $\hat\sigma^2$ divides by $n$, so it is biased; the unbiased version divides by $n - 1$.

If $\hat\theta$ is the MLE of $\theta$ and $g$ is a bijection, then $g(\hat\theta)$ is the MLE of $g(\theta)$. (With the usual convention, the same holds for any function $g$.)

For an estimator $T$ of $\theta$: the bias is $\operatorname{bias}(T) = \mathbb{E}_\theta[T] - \theta$; $T$ is unbiased if the bias is $0$ for all $\theta$; the mean squared error is $\operatorname{MSE}(T) = \mathbb{E}_\theta[(T - \theta)^2]$. $T$ is consistent if $T\to\theta$ in probability as $n\to\infty$.

$\operatorname{MSE}(T) = \operatorname{Var}(T) + \operatorname{bias}(T)^2$. Consequently a biased estimator can have smaller MSE than an unbiased one.

A statistic $T = T(\mathbf{X})$ is sufficient for $\theta$ if the conditional distribution of $\mathbf{X}$ given $T$ does not depend on $\theta$.

$T$ is sufficient for $\theta$ iff the joint density/mass function factorises as $f(\mathbf{x};\theta) = g(T(\mathbf{x}), \theta)h(\mathbf{x})$.

For one observation from a regular model, the score is $U(\theta) = \frac{\partial}{\partial\theta}\log f(X;\theta)$ and the Fisher information is $I(\theta) = \operatorname{Var}_\theta(U) = \mathbb{E}_\theta[U^2]$. Under regularity (differentiation under the integral), $\mathbb{E}_\theta[U] = 0$ and $I(\theta) = -\mathbb{E}_\theta\big[\frac{\partial^2}{\partial\theta^2}\log f(X;\theta)\big]$. A sample of size $n$ has information $nI(\theta)$.

Under regularity conditions, every unbiased estimator $T$ of $\theta$ based on $n$ i.i.d. observations satisfies $\operatorname{Var}_\theta(T)\ge\frac1{nI(\theta)}$. An unbiased estimator attaining the bound is efficient.

• Write $\ell(\theta)$; solve the score equation; check the second derivative and the boundary (support depending on $\theta$!).
• Bias: compute $\mathbb{E}T$; MSE = Var + bias².
• Sufficiency: factorise the likelihood.
• Efficiency: compare $\operatorname{Var}T$ with $\frac1{nI(\theta)}$.