Point Estimation & Bias-Variance

A statistic is any measurable function $T = g(X_1, \ldots, X_n)$ of the observed sample. Equivalently, $T$ is a random variable whose value is determined by the data, and whose distribution is induced by the joint distribution of the sample.

A point estimator of a parameter $\theta$ is a statistic $\hat{\theta}_n = g(X_1, \ldots, X_n)$ whose value is used as a guess for $\theta$. The estimator is a random variable; its value $\hat{\theta}_n(\omega)$ for a particular sample realization is a point estimate.

Suppose $X_1, \ldots, X_n$ are iid with unknown mean $\mu$ and finite variance. Three candidate estimators of $\mu$:

1. Sample mean: $\hat{\mu}_1 = \bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i$.
2. Sample median: $\hat{\mu}_2 = \operatorname{median}(X_1, \ldots, X_n)$, the middle order statistic — whose asymptotics are developed in Topic 29 §29.6.
3. Trimmed mean: $\hat{\mu}_3$ = mean of the sample after discarding the top $10\%$ and bottom $10\%$ of observations.

All three are statistics (they depend only on the data), all three target the same $\mu$, and all three are reasonable candidates. They produce different values on the same sample, and their sampling distributions are different shapes. Which one is “best” depends on the underlying distribution and on what we mean by “best” — the rest of this topic builds the vocabulary to answer that question precisely.

In an introductory statistics class, you compute the sample mean of a dataset and report it as “the” estimate. In estimation theory, we take one step back and ask: if we could repeatedly draw fresh samples of size $n$ from the population and compute the sample mean each time, what distribution of values would we see? That distribution — not any single computed value — is the object we analyze. Its center is where our estimates cluster on average (bias tells us whether that center is $\mu$). Its spread tells us how precise any single estimate is (variance). Its shape tells us how the estimator behaves in the tails, and for large $n$, what confidence intervals look like (asymptotic normality). Every theorem in this topic is a statement about that distribution.

This is also why we write $\hat{\theta}_n$ with a hat: the hat signals “random variable, function of the data,” distinguishing it from the unknown-but-fixed true parameter $\theta$. Reading a formula, always ask: which symbols are random? which are fixed? The answer changes everything.

Let $\hat{\theta}_n$ be an estimator of $\theta$. Its bias is the difference between its expected value and the true parameter:

The expectation is taken over the sampling distribution of $\hat{\theta}_n$, treating $\theta$ as fixed.

The estimator $\hat{\theta}_n$ is unbiased for $\theta$ if $\operatorname{Bias}(\hat{\theta}_n) = 0$, equivalently $\mathbb{E}[\hat{\theta}_n] = \theta$, for every value of $\theta$ in the parameter space.

If $X_1, \ldots, X_n$ are iid with $\mathbb{E}[X_i] = \mu$ (assumed finite), then the sample mean $\bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i$ is unbiased for $\mu$:

The iid assumption in Theorem 1 is stronger than needed: we only used that $\mathbb{E}[X_i] = \mu$ for every $i$. So the sample mean is unbiased whenever the summands share a common mean, even if their variances differ or their distributions are different. For example, if $X_1 \sim \operatorname{Exponential}(1)$ and $X_2 \sim \operatorname{Gamma}(2, 2)$ both have mean $1$, then $\bar{X}_2 = (X_1 + X_2)/2$ is still unbiased for $1$. Unbiasedness of linear statistics is astonishingly robust.

Contrast this with the sample median, which is not generally unbiased for $\mu$ unless the distribution is symmetric around $\mu$. For Exponential$(1)$, the population mean is $1$ but the population median is $\log 2 \approx 0.693$; the sample median is unbiased for the median, not the mean.

The natural candidate for estimating the population variance $\sigma^2$ is the average squared deviation from the sample mean:

This is the maximum-likelihood estimate of $\sigma^2$ when the underlying distribution is Normal, and it is the “natural” variance of the empirical distribution. But it is biased. A direct calculation — expanding the squared deviation, using $\operatorname{Var}(\bar{X}_n) = \sigma^2/n$ (Theorem 2 below), and collecting terms — shows

So $S_n^2$ underestimates $\sigma^2$ by a factor of $(n-1)/n$. The fix is to divide by $n-1$ instead of $n$:

This is the unbiased sample variance, and the $(n-1)$ denominator is called Bessel’s correction. The correction is intuitive: in computing $\bar{X}_n$ from the data we have “used up” one degree of freedom, so only $n - 1$ independent residuals remain. The unbiasedness of $S^2$ follows immediately: $\mathbb{E}[S^2] = \frac{n}{n-1}\cdot \mathbb{E}[S_n^2] = \sigma^2$.

Which divisor should you use in practice? That turns out to be a more subtle question than it first appears — see Example 5 in §13.4.

Unbiasedness feels like a bedrock virtue, but consider the estimator $\tilde{\mu} = X_1$ — the first observation, ignoring everything else. It is unbiased for $\mu$: $\mathbb{E}[X_1] = \mu$. It is also a terrible estimator: its variance is $\sigma^2$, compared with $\sigma^2/n$ for the sample mean, so after any real amount of data it is $n$ times worse. Unbiasedness is satisfied; precision is not.

More dramatically, some parameters admit no unbiased estimator at all (e.g., the reciprocal of a Poisson mean), while others admit only pathological ones. And, as we will see, biased estimators can systematically outperform unbiased ones in mean squared error — shrinkage estimators, ridge regression, and James–Stein all exploit this. Unbiasedness is one input into evaluation, not the whole story. The right criterion combines bias and variance, which is what MSE does.

The standard error of an estimator $\hat{\theta}_n$ is the standard deviation of its sampling distribution:

When the estimator’s variance depends on unknown population parameters, the estimated standard error $\widehat{\operatorname{SE}}(\hat{\theta}_n)$ plugs in sample-based substitutes.

If $X_1, \ldots, X_n$ are iid with $\operatorname{Var}(X_i) = \sigma^2$ (assumed finite), then the sample mean has variance

and therefore standard error $\operatorname{SE}(\bar{X}_n) = \sigma/\sqrt{n}$.

If $X_1, \ldots, X_n$ are iid $\mathcal{N}(\mu, \sigma^2)$, the sample mean satisfies $\bar{X}_n \sim \mathcal{N}(\mu, \sigma^2/n)$ exactly — no approximation, no CLT needed. Its standard error is $\sigma/\sqrt{n}$. For $\sigma = 2$ and $n = 25$, that is $2/5 = 0.4$: a single sample mean will typically land within $\pm 0.4$ of $\mu$ (about one SE), and almost always within $\pm 0.8$ (two SEs).

When $\sigma$ is unknown — the usual case — we plug in the sample standard deviation $S = \sqrt{S^2}$ (from Example 3) to get the estimated standard error $\widehat{\operatorname{SE}}(\bar{X}_n) = S/\sqrt{n}$. The distinction matters: confidence intervals built from $S$ use the Student’s $t$ distribution, not the standard Normal, because $S$ introduces additional uncertainty. That $t$-correction is Topic 11’s Example 7, and the machinery of confidence intervals will develop it systematically.

It is easy to conflate “standard deviation” and “standard error.” The difference is crucial:

• The standard deviation $\sigma$ describes the spread of the population. It does not shrink as you collect more data.
• The standard error $\sigma/\sqrt{n}$ describes the spread of the estimator. It shrinks like $1/\sqrt{n}$.

When you report a sample mean as ”$\bar{X}_n = 4.72 \pm 0.40$,” the $\pm 0.40$ is the standard error — how uncertain you are about the mean. When you report that your data has standard deviation $2.0$, that is the standard deviation — how spread out the individual observations are. SE says how precisely you know the average; SD says how dispersed the raw data are. SE is precision about precision — a derived uncertainty.

The mean squared error of an estimator $\hat{\theta}_n$ of $\theta$ is the expected squared deviation from the truth:

The expectation is taken over the sampling distribution of $\hat{\theta}_n$, treating $\theta$ as fixed.

For any estimator $\hat{\theta}_n$ of $\theta$ with finite second moment,

Equivalently, $\mathbb{E}[(\hat{\theta}_n - \theta)^2]$ splits cleanly into a mean-offset term and a dispersion term.

From Example 3, the two candidates for estimating $\sigma^2$ are:

• $S^2 = \frac{1}{n-1}\sum(X_i - \bar{X}_n)^2$, unbiased.
• $S_n^2 = \frac{1}{n}\sum(X_i - \bar{X}_n)^2 = \frac{n-1}{n}S^2$, biased with $\mathbb{E}[S_n^2] = \frac{n-1}{n}\sigma^2$.

For iid Normal data, it can be shown that $\operatorname{Var}(S^2) = \frac{2\sigma^4}{n-1}$ and $\operatorname{Var}(S_n^2) = \frac{2(n-1)\sigma^4}{n^2}$. Plugging into the MSE decomposition:

For $n = 10$: $\operatorname{MSE}(S^2) = 2\sigma^4/9 \approx 0.222\sigma^4$ vs. $\operatorname{MSE}(S_n^2) = 19\sigma^4/100 = 0.190\sigma^4$. The biased estimator has lower MSE. This is not a trick: Bessel’s correction removes bias at the cost of increased variance, and the tradeoff does not always favor unbiasedness. It turns out that $\hat{\sigma}^2_{\text{opt}} = \frac{1}{n+1}\sum(X_i - \bar{X}_n)^2$ — shrinking even more aggressively than the MLE — has smaller MSE still for Normal data. Minimum-MSE and unbiasedness are genuinely different objectives.

Consider estimating $\mu \in \mathbb{R}$ from iid $\mathcal{N}(\mu, \sigma^2)$ data with the family of shrinkage estimators

At $c = 1$ we recover the sample mean (unbiased, MSE $= \sigma^2/n$). At $c = 0$ we get the trivial estimator that always returns zero (biased by $-\mu$, variance zero, MSE $= \mu^2$). In between, bias and variance trade off:

Differentiating with respect to $c$ and setting to zero gives the MSE-optimal shrinkage

Note that $c^* < 1$ strictly whenever $\sigma^2/n > 0$ — which it always is. So the sample mean is never MSE-optimal in this family; there always exists a shrunk version that strictly dominates it. The catch: $c^*$ depends on $\mu$ itself, so we cannot use it as a standalone estimator. But this example is the seed of every shrinkage, regularization, and empirical-Bayes method: allowing bias opens a strictly better option in terms of MSE.

Theorem 3 makes the bias-variance tradeoff precise: MSE has two components, and reducing one often increases the other. This is not a metaphor — it is an algebraic identity. Every strategy for minimizing MSE must navigate this tension:

• More data ($n \uparrow$): variance shrinks like $\sigma^2/n$, bias typically unaffected. So MSE is dominated by bias at large $n$ — unbiased consistent estimators eventually win.
• Simpler models / more shrinkage: bias increases, variance decreases. Good for small $n$ when variance dominates.
• More flexible models / less shrinkage: bias decreases, variance increases. Good for large $n$ when bias dominates.

In an ML context, this is the polynomial-fitting story: a degree-1 line is highly biased but low-variance; a degree-15 polynomial is nearly unbiased but enormously variable. Regularization, cross-validation, and model selection all exist to find the sweet spot. Section 13.9 develops the ML connection in detail.

An estimator $\hat{\theta}_n$ is consistent for $\theta$ if, as $n \to \infty$, it converges to $\theta$ in probability: for every $\varepsilon > 0$,

Equivalently, $\hat{\theta}_n \overset{P}{\to} \theta$.

An estimator $\hat{\theta}_n$ is strongly consistent for $\theta$ if $\hat{\theta}_n \overset{a.s.}{\to} \theta$: the set of sample sequences on which the running estimates fail to converge to $\theta$ has probability zero.

If $\operatorname{MSE}(\hat{\theta}_n) \to 0$ as $n \to \infty$, then $\hat{\theta}_n$ is consistent for $\theta$.

If $X_1, X_2, \ldots$ are iid with $\mathbb{E}[|X_1|] < \infty$ and $\mu = \mathbb{E}[X_1]$, then the sample mean is strongly consistent for $\mu$:

If only $\mathbb{E}[X_1^2] < \infty$ is known, the weaker statement $\bar{X}_n \overset{P}{\to} \mu$ follows from Chebyshev (Theorem 4 above) or from the weak law of large numbers.

Let $X_1, \ldots, X_n$ be iid with $\mathbb{E}[X_1^4] < \infty$ (a fourth-moment condition, stronger than what Theorem 5 required). The unbiased sample variance $S^2 = \frac{1}{n-1}\sum(X_i - \bar{X}_n)^2$ is consistent for $\sigma^2 = \operatorname{Var}(X_1)$.

Sketch. Expand

By the LLN applied to $\{X_i^2\}$, the first average converges almost surely to $\mathbb{E}[X_1^2]$. By the LLN applied to $\{X_i\}$ and continuous-mapping, $\bar{X}_n^2 \overset{a.s.}{\to} \mu^2$. The factor $n/(n-1) \to 1$. Combining:

Consistency of the biased version $S_n^2 = \frac{n-1}{n}S^2$ follows immediately by the same argument (multiplying by $(n-1)/n \to 1$). Both divisors yield consistent estimators of $\sigma^2$; they differ only in finite-sample MSE.

Consider the estimator $\tilde{\mu} = X_1$ of the population mean $\mu$. It is unbiased ($\mathbb{E}[X_1] = \mu$), but it ignores all but the first observation — no matter how many data points you collect, the estimate never updates after $n = 1$.

Formally: $\tilde{\mu}$ does not depend on $n$ at all, so $P(|\tilde{\mu} - \mu| > \varepsilon) = P(|X_1 - \mu| > \varepsilon)$ — a fixed, positive number for any $\varepsilon < \sigma$ and any non-degenerate distribution. The probability does not shrink with $n$. Therefore $\tilde{\mu}$ is inconsistent, despite being unbiased.

This is the cleanest demonstration that unbiasedness and consistency are independent properties. $\tilde{\mu}$ has $\operatorname{MSE} = \operatorname{Var}(X_1) = \sigma^2$ for every $n$, which does not shrink, so Theorem 4 does not kick in.

Examples 7 and 8 make the point: consistency and unbiasedness are logically independent. Exhaustive possibilities:

• Both: sample mean, unbiased sample variance. The ideal case.
• Unbiased, inconsistent: $\tilde{\mu} = X_1$. Unbiased but fails to improve with more data.
• Biased, consistent: biased sample variance $S_n^2$. Its bias shrinks as $n \to \infty$ ($\frac{n-1}{n}\sigma^2 \to \sigma^2$), so it is asymptotically unbiased; with vanishing variance, it is consistent.
• Neither: the constant estimator $\tilde{\theta} = 7$ (or any fixed value unrelated to $\theta$). Ignores the data, biased except by coincidence, not consistent.

For practical purposes, consistency is the more important of the two: we typically want an estimator that eventually gets the right answer, and we tolerate small-sample bias as long as it washes out. Unbiasedness is sometimes useful as a finite-sample property (e.g. for building exact confidence intervals under Normal assumptions), but asymptotic unbiasedness — $\operatorname{Bias}(\hat{\theta}_n) \to 0$ — is what modern statistics actually requires most of the time.

The estimator $\hat{\theta}_n$ is asymptotically normal with asymptotic variance $v(\theta)$ if

where $\overset{d}{\to}$ denotes convergence in distribution. Informally, for large $n$, $\hat{\theta}_n \approx \mathcal{N}(\theta, v(\theta)/n)$.

If $X_1, X_2, \ldots$ are iid with $\operatorname{Var}(X_1) = \sigma^2 \in (0, \infty)$, then the sample mean is asymptotically normal with asymptotic variance $\sigma^2$:

Let $\hat{\theta}_n$ be asymptotically normal: $\sqrt{n}(\hat{\theta}_n - \theta) \overset{d}{\to} \mathcal{N}(0, v(\theta))$. Let $g : \mathbb{R} \to \mathbb{R}$ be differentiable at $\theta$ with $g'(\theta) \neq 0$. Then $g(\hat{\theta}_n)$ is asymptotically normal for $g(\theta)$:

We want the limit distribution of $S_n^2 = \frac{1}{n}\sum(X_i - \bar{X}_n)^2$ for iid data with finite fourth moment. Write the sample variance as a function of two sample means: $S_n^2 = \overline{X^2}_n - \bar{X}_n^2$ where $\overline{X^2}_n = \frac{1}{n}\sum X_i^2$.

By the multivariate CLT applied to the random vector $(X_i, X_i^2)$, the vector $\big(\bar{X}_n, \overline{X^2}_n\big)$ is jointly asymptotically Normal around $(\mu, \mathbb{E}[X_1^2])$ with a covariance matrix determined by the first four moments of $X_1$. Applying the delta method to $g(a, b) = b - a^2$:

where $\mu_4 = \mathbb{E}[(X_1 - \mu)^4]$ is the central fourth moment. For Normal data, $\mu_4 = 3\sigma^4$, so the asymptotic variance simplifies to $2\sigma^4$ — which matches the exact (small-sample) variance formula from Example 5 after the $1/n$ rescaling. For non-Normal data, the formula picks up the excess kurtosis: heavier tails mean the sample variance has larger asymptotic variance, because extreme observations dominate the squared residuals.

Asymptotic normality gives us an apples-to-apples way to compare estimators: look at their asymptotic variances. For two asymptotically normal estimators $\hat{\theta}_{1,n}$ and $\hat{\theta}_{2,n}$ of the same parameter, the asymptotic relative efficiency is

the ratio of their asymptotic variances (in the order that makes values above $1$ mean ”$\hat{\theta}_1$ wins”). For Normal data, the sample median has ARE $2/\pi \approx 0.637$ relative to the sample mean — meaning 1,000 observations for the median give about the same precision as $\pi/2 \times 1{,}000 \approx 1{,}570$ for the mean. For Cauchy data (where the sample mean has no well-defined asymptotic variance), the ARE comparison flips: the median is the efficient choice and the mean is catastrophic. Asymptotic efficiency is distribution-dependent; efficient-for-Normal and efficient-for-Cauchy are different estimators.

The next section, on the Cramér–Rao bound, quantifies the ceiling on achievable asymptotic variance — and therefore the ceiling on asymptotic efficiency.

Let $\{f(x; \theta) : \theta \in \Theta\}$ be a parametric family of densities (or PMFs) with $\Theta \subseteq \mathbb{R}$. The score function is the partial derivative of the log-likelihood with respect to the parameter:

For an iid sample, the total score is the sum of per-observation scores: $S(\theta) = \sum_{i=1}^n s(\theta; X_i)$.

The Fisher information at $\theta$ is the variance of the score:

Under regularity conditions, an equivalent form is the negative expected second derivative of the log-likelihood:

Under regularity conditions that allow interchange of differentiation and integration,

An unbiased estimator $\hat{\theta}_n$ whose variance attains the Cramér–Rao lower bound (Theorem 9) for every $\theta$ is called efficient. An asymptotically normal estimator whose asymptotic variance attains the asymptotic Cramér–Rao bound $1/I(\theta)$ is called asymptotically efficient.

Let $\hat{\theta}_n$ be an unbiased estimator of $\theta$ based on an iid sample of size $n$ from a parametric family satisfying the regularity conditions. Then

Three canonical computations using $I(\theta) = -\mathbb{E}[\partial^2 \log f / \partial \theta^2]$:

Normal $\mathcal{N}(\mu, \sigma^2)$ with respect to $\mu$ (known $\sigma^2$): $\log f(x; \mu) = -\tfrac{1}{2}\log(2\pi\sigma^2) - (x-\mu)^2/(2\sigma^2)$. Two derivatives: $\partial^2 \log f / \partial \mu^2 = -1/\sigma^2$. Taking the negative expectation (which is constant):

The Fisher information does not depend on $\mu$, only on $\sigma^2$: narrow distributions carry more information per sample about their location.

Bernoulli$(p)$: $\log f(x; p) = x \log p + (1-x) \log(1-p)$. Second derivative $\partial^2 \log f / \partial p^2 = -x/p^2 - (1-x)/(1-p)^2$. Expectation: $-p/p^2 - (1-p)/(1-p)^2 = -1/p - 1/(1-p) = -1/(p(1-p))$. Negating:

Fisher information is maximized at $p = 1/2$ (where variance is highest and $x$ is least predictable) and diverges as $p \to 0$ or $p \to 1$ — near the boundary, each observation is enormously informative about whether $p$ is exactly $0$ or exactly $1$.

Exponential$(\lambda)$: $\log f(x; \lambda) = \log \lambda - \lambda x$ for $x > 0$. Second derivative $-1/\lambda^2$. Negating:

Larger rate $\lambda$ means faster decay, which means observations cluster near zero and carry more information about $\lambda$.

For iid $\mathcal{N}(\mu, \sigma^2)$ with known $\sigma^2$, the sample mean is unbiased and has variance $\sigma^2/n$ (Theorem 2). From Example 10, $I(\mu) = 1/\sigma^2$. The Cramér–Rao lower bound is

Equality. The sample mean is the efficient estimator of the Normal mean — no unbiased estimator based on $n$ iid Normal observations can do better. This is the canonical result that motivates the CRLB in the first place, and the reason the sample mean is the point estimator you reach for whenever Normality is plausible. For non-Normal data the sample mean is still unbiased and consistent, but it may no longer be efficient — the median beats it for Cauchy, and trimmed means dominate for heavy-tailed distributions.

The Cramér–Rao proof requires regularity conditions that allow interchange of differentiation and integration. Informally: the support of $f(x; \theta)$ must not depend on $\theta$, and $\partial \log f / \partial \theta$ must be well-behaved enough to differentiate under the integral. These conditions hold for essentially every standard family — Normal, Poisson, Bernoulli, Exponential, Gamma, Beta — but fail in a few important cases:

• Uniform$(0, \theta)$: the support is $[0, \theta]$, which depends on $\theta$. The MLE $\hat{\theta} = \max_i X_i$ has variance $\sim \theta^2/n^2$, faster than the $1/n$ rate the CRLB would predict. The CRLB does not apply.
• Shifted exponentials: $f(x; \theta) = e^{-(x-\theta)}$ for $x \geq \theta$. Same issue — boundary of the support moves with $\theta$.
• Heavy-tailed distributions with infinite Fisher information (Cauchy w.r.t. scale, for instance): the bound is $1/\infty = 0$, which is trivially satisfied but uninformative.

When regularity holds, the CRLB is the ceiling: no unbiased estimator can beat $1/(nI(\theta))$. When regularity fails, different tools — the $L_1$ theory, order-statistics asymptotics (Topic 29 §29.6), the Hájek-Le Cam framework — take over. For standard statistical modeling, regularity holds and the CRLB is the right benchmark.

An estimator $\hat{\theta}_1$ is dominated by $\hat{\theta}_2$ if $\operatorname{MSE}(\hat{\theta}_2; \theta) \leq \operatorname{MSE}(\hat{\theta}_1; \theta)$ for all $\theta \in \Theta$, with strict inequality for at least one $\theta$. An estimator that is not dominated by any other is admissible.

An estimator $\hat{\theta}$ is minimax if it minimizes the maximum risk:

Minimax estimators are conservative: they trade off finite-sample performance at typical parameter values for worst-case robustness.

Let $X \sim \mathcal{N}(\theta, \sigma^2 I_d)$ with unknown mean vector $\theta \in \mathbb{R}^d$ and known scale $\sigma^2$. For $d \geq 3$, the James–Stein estimator

has strictly smaller total MSE than the sample mean $X$ for every value of $\theta$:

Therefore the sample mean is inadmissible in dimension $d \geq 3$.