Likelihood-Ratio Tests & Neyman-Pearson

A level-$\alpha$ test is valid if its Type I error is at most $\alpha$. That is not a hard standard to meet. The trivial “reject with probability $\alpha$ regardless of data” test is valid — and has power exactly $\alpha$ at every alternative. Every test we wrote down in Topic 17 (z-test, t-test, $\chi^2$ variance test, binomial exact) is valid for its parametric family. But validity does not single out which level-$\alpha$ test to use.

Optimality asks the harder question: among all valid level-$\alpha$ tests, which maximizes power? The answer depends on whether the hypotheses are simple or composite and on the structural geometry of the likelihood family. Where the answer exists — NP lemma, Karlin-Rubin — it is a uniqueness result, not merely an existence one. Where it does not — two-sided composite alternatives in most families — the LRT is the best general-purpose substitute, and we pay a price for that generality in the form of asymptotic rather than finite-sample optimality.

Topic 18 is built around four results. The first two hold at every sample size; the last two are asymptotic.

1. Neyman-Pearson lemma (§18.2). Simple-vs-simple optimality via a three-step indicator-function argument. Provable from integration alone — no convergence theory required.

2. Karlin-Rubin theorem (§18.3). One-sided composite optimality via monotone likelihood ratio (MLR), using the NP lemma as an internal engine. The MLR families are exactly where “one test is most powerful at every alternative simultaneously” — i.e., UMP — becomes possible.

3. Wilks’ theorem (§18.6). The composite LRT’s null distribution converges: $-2\log\Lambda_n \xrightarrow{d} \chi^2_k$ where $k$ is the number of parameter restrictions. Proved in 8 steps via Taylor expansion around $\hat\theta_n$, remainder control, observed-information-to-Fisher-information, and MLE asymptotic normality (Topic 14 Thm 14.3).

4. Three-tests equivalence (§18.7). Wald, Score, and LRT differ by $O_P(n^{-1/2})$ under $H_0$. All three collapse to the same quadratic form $n I(\theta_0)(\hat\theta_n - \theta_0)^2$ at leading order. §18.8 quantifies the finite-sample divergence the $o_P(1)$ hides.

The residual sections then extend the theory in specific directions. §18.4 catalogues UMP tests in action; §18.5 formalizes the composite LRT; §18.8 proves reparameterization invariance; §18.9 characterizes local power via non-central $\chi^2$; §18.10 points to the scope boundaries.

Every proof in Topic 18 is written for scalar $\theta \in \mathbb{R}$. Vector-$\theta$ extensions — $k$-df Wilks, UMP unbiased, Hunt-Stein invariance, Chernoff 1954 boundary cases — are stated with pointers to Lehmann-Romano (LEH2005) and nothing more. This is not pedagogical convenience; it is a deliberate scope decision. The scalar case captures every main idea (the pointwise LR comparison of Proof 1, the MLR-yields-UMP argument of Proof 2, the quadratic-approximation of Proof 3). The vector generalizations are mostly bookkeeping — matrix inverses in place of scalar divisions, quadratic forms $\mathbf{q}^\top \mathbf{I}(\theta_0) \mathbf{q}$ in place of $q^2 I(\theta_0)$, and a $\chi^2_k$ limit in place of $\chi^2_1$. Readers who need the vector results in production should consult LEH2005 Ch. 3 (vector UMP), Ch. 6 (invariance), and Ch. 12 (large-sample theory). Topic 19 and Topic 20 extend Track 5 with CI duality and multiple testing; vector-$\theta$ extensions to Wilks reappear organically in GLM inference on formalml.com/generalized-linear-models.

A test $\varphi: \mathcal{X} \to [0, 1]$ (where $\varphi(x)$ is the probability of rejecting $H_0$ at observation $x$) is most powerful (MP) at level $\alpha$ against $H_1: \theta = \theta_1$ if

That is, $\varphi^*$ maximizes the power at $\theta_1$ among all tests of size at most $\alpha$. The maximum need not be unique on the boundary of the constraint; uniqueness holds in the interior.

For simple $H_0: \theta = \theta_0$ vs simple $H_1: \theta = \theta_1$, the likelihood ratio at observation $x$ is

For iid data $X = (X_1, \ldots, X_n)$ with density $\prod_i f(x_i; \theta)$, the LR factorizes: $\Lambda(x) = \prod_i f(x_i; \theta_1)/f(x_i; \theta_0)$, and we typically work with $\log \Lambda(x) = \sum_i [\log f(x_i; \theta_1) - \log f(x_i; \theta_0)]$ for numerical stability.

The $\Lambda(x)$ above is distinct from the composite generalized likelihood ratio $\Lambda_n$ defined in Topic 17 §17.9 Def 15. The simple-vs-simple LR compares two specific parameter values; the composite $\Lambda_n = \sup_{\theta \in \Theta_0} L(\theta) / \sup_{\theta \in \Theta} L(\theta)$ compares the best null fit to the best full-model fit. They coincide when both hypotheses are singletons. §18.5 reintroduces the composite $\Lambda_n$, and §18.6 proves its $\chi^2$ asymptotic limit.

Let $X$ have density $f(x; \theta)$ with respect to a common $\sigma$-finite measure $\mu$. For testing simple $H_0: \theta = \theta_0$ vs simple $H_1: \theta = \theta_1$, fix $\alpha \in (0, 1)$ and let $k \ge 0$ be a threshold. The Neyman-Pearson test

with randomization on the boundary $\{f_1 = k f_0\}$ chosen to make $E_{\theta_0}[\varphi^*] = \alpha$, is most powerful level-$\alpha$ among all tests $\varphi$ with $E_{\theta_0}[\varphi] \le \alpha$.

Let $X_1, \ldots, X_n$ be iid Bernoulli$(p)$. Test $H_0: p = p_0$ vs $H_1: p = p_1$ with $p_1 > p_0$. The likelihood ratio at $x$ is

which is monotone increasing in $T(x) = \sum x_i$ (since $p_1 > p_0$ implies $p_1/p_0 > 1$ and $(1-p_1)/(1-p_0) < 1$). The NP test rejects iff $T(X) > c$ for some threshold $c$; randomization at $T = c$ achieves exact size $\alpha$.

Concretely: for $n = 20$, $p_0 = 0.5$, $p_1 = 0.7$, $\alpha = 0.05$, the NP test rejects at $T > 15$ with exact size $\approx 0.021$ (the conservative size of §18.4). Power at $p_1 = 0.7$: $P_{0.7}(T > 15) \approx 0.416$ (verified in binomialExactPower of Topic 17’s testing.ts).

Let $X_1, \ldots, X_n$ be iid $\mathcal{N}(\mu, \sigma^2)$ with $\sigma$ known. Test $H_0: \mu = \mu_0$ vs $H_1: \mu = \mu_1$ with $\mu_1 > \mu_0$. The log-likelihood ratio is

monotone increasing in $\bar x$. The NP test rejects iff $\bar x > c$ for $c = \mu_0 + z_{1-\alpha}\, \sigma / \sqrt n$, where $z_{1-\alpha}$ is the standard-Normal $(1-\alpha)$-quantile. The threshold depends on $\alpha$ and $n$ and $\sigma$ — but not on $\mu_1$. The same test is NP against every $\mu_1 > \mu_0$. That is the geometric seed of the Karlin-Rubin theorem in §18.3.

Numerical check: $n = 25$, $\mu_0 = 0$, $\mu_1 = 1$, $\sigma = 1$, $\alpha = 0.05$. Then $c = 0 + 1.645 \cdot 1/\sqrt{25} \approx 0.329$, matching npCriticalValue('normal-mean-known-sigma', 0, 1, 25, 0.05, 1) in testing.ts.

Let $X_1, \ldots, X_n$ be iid Exponential$(\lambda)$ with density $f(x; \lambda) = \lambda e^{-\lambda x}$ on $x \ge 0$. Test $H_0: \lambda = \lambda_0$ vs $H_1: \lambda = \lambda_1$. The log-likelihood ratio is

If $\lambda_1 > \lambda_0$ (faster rate, shorter expected waits), the LR is decreasing in $T(x) = \sum x_i$: short totals favor $H_1$. The NP test rejects iff $T < c_{\text{lo}}$. Under $H_0$, $T \sim \Gamma(n, \lambda_0)$; using $(2\lambda_0 T) \sim \chi^2_{2n}$ we get the exact size at $c_{\text{lo}} = \chi^2_{2n, \alpha} / (2\lambda_0)$.

The direction reversal — shorter totals trigger rejection of “the slower rate is true” — is a common pitfall in exponential-reliability testing. npCriticalValue('exponential', ...) in Topic 17’s testing.ts handles the direction via an explicit Tform label noting the side.

The NP test’s “randomize on the boundary” clause looks strange but is needed only when $P_{\theta_0}(\Lambda = k) > 0$ — i.e., when the test statistic is discrete and $k$ falls on an atom. For continuous tests (Normal, Exponential), the boundary has measure zero and no randomization is needed; the size is exactly $\alpha$ by choice of $k$. For discrete tests (Bernoulli, Poisson), the exact size without randomization is usually strictly less than $\alpha$ — the conservative size of §17.3 Thm 1 and §17.6 Ex 11. We treat randomization as a theoretical device: in practice we rarely implement it, accepting the conservative size in exchange for interpretability.

Theorem 1 is a template — given the two parameter values and a chosen $\alpha$, it tells you what the MP test looks like. It does not by itself produce a usable test of a practical hypothesis like “is this drug better than placebo?” because practical hypotheses are rarely simple-vs-simple. The real $H_1$ is usually composite (“some positive effect of unspecified size”), and the MP test depends on which specific alternative you face. §18.3 asks: when is the NP rejection region the same for every alternative in a composite $H_1$? That is the MLR / Karlin-Rubin story.

A family $\{f(\cdot; \theta) : \theta \in \Theta \subseteq \mathbb{R}\}$ has monotone likelihood ratio in $T(x)$ if there exists a statistic $T(x)$ such that for every pair $\theta_1 < \theta_2$ in $\Theta$, the ratio

Equivalently, $\log[f(x; \theta_2)/f(x; \theta_1)]$ is a non-decreasing function of $T(x)$.

A level-$\alpha$ test $\varphi^*$ is uniformly most powerful (UMP) against a composite alternative $H_1: \theta \in \Theta_1$ if it is MP level-$\alpha$ at every $\theta_1 \in \Theta_1$ simultaneously. In symbols: for every $\theta_1 \in \Theta_1$ and every level-$\alpha$ test $\varphi$,

UMP existence is a strong statement — the same rejection region beats every competitor at every alternative.

Let $\{f(\cdot; \theta) : \theta \in \Theta \subseteq \mathbb{R}\}$ have MLR in $T(x)$. For testing

there exists a threshold $c$ (possibly with randomization at $T = c$ for discrete $T$) such that the test

has size exactly $\alpha$ and is UMP level-$\alpha$.

Let $f(x; \eta) = h(x) \exp\bigl(\eta T(x) - A(\eta)\bigr)$ be a one-parameter exponential family with natural parameter $\eta$ and sufficient statistic $T(x)$. For any $\eta_1 < \eta_2$,

This is a monotone increasing function of $T(x)$ because $\eta_2 - \eta_1 > 0$. So every one-parameter exponential family has MLR in its natural sufficient statistic. Karlin-Rubin then hands us a UMP test for every one-sided composite hypothesis on $\eta$ — Bernoulli, Normal mean (with $\sigma$ known), Poisson, Exponential, Gamma (with shape known), Geometric, and so on.

This is the pedagogical payoff of Topic 16’s factorization + completeness machinery: once you know the sufficient statistic, the UMP test writes itself.

Let $X_1, \ldots, X_n$ be iid Uniform$(0, \theta)$, $\theta > 0$. The density is $f(x; \theta) = \theta^{-1} \mathbf{1}\{0 \le x \le \theta\}$, and the joint is $\theta^{-n} \mathbf{1}\{\max_i x_i \le \theta\}$. For $\theta_1 < \theta_2$,

The ratio is non-decreasing in $\max_i x_i$ (jumps from a constant to $+\infty$ at $\max = \theta_1$). Uniform$(0, \theta)$ has MLR in $T = \max X_i$, even though it is not an exponential family. By Karlin-Rubin, the one-sided UMP test for $H_0: \theta \le \theta_0$ vs $H_1: \theta > \theta_0$ rejects iff $\max X_i > c$ where $c = \theta_0 \cdot \alpha^{1/n}$ gives size exactly $\alpha$ (the left-tail quantile of $\max/\theta_0$, which is distributed as $U^{1/n}_{\max}$).

Karlin-Rubin gives UMP for one-sided composite alternatives. For two-sided alternatives $H_1: \theta \ne \theta_0$, UMP tests typically do not exist, because the best rejection region for $\theta_1 > \theta_0$ points right while the best for $\theta_1 < \theta_0$ points left, and no single region handles both. The standard workaround is to restrict to unbiased tests (power $\ge \alpha$ at every alternative) and prove UMP within that class — the UMP unbiased (UMPU) construction of Lehmann-Romano (LEH2005 Ch. 4). That theory is beyond Topic 18’s scope; we note only that for the Normal variance test (one-sided in $\sigma^2$) the UMPU and the equal-tail $\chi^2$ test coincide, while for the two-sided Normal mean the z-test is UMPU but not UMP in the full class of level-$\alpha$ tests. §18.4 Remark 8 returns to this.

The structural property was isolated by Karlin and Rubin (KAR1956) in their 1956 paper on MLR decision procedures, but the term “monotone likelihood ratio” itself was coined the next year in Karlin’s solo paper on Pólya-type distributions (KAR1957). The MLR families are exactly the totally positive of order 2 ($\mathrm{TP}_2$) families in Karlin’s terminology — a classification that generalizes to kernels and sequences in a way that powers modern results in statistical order (Shaked & Shanthikumar 2007) and in asymptotic efficiency bounds for monotone regression (Groeneboom & Jongbloed 2014).

Let $X_1, \ldots, X_n$ be iid Bernoulli$(p)$. Test $H_0: p \le p_0$ vs $H_1: p > p_0$. The Bernoulli family is exponential with natural parameter $\eta = \log[p/(1-p)]$ (log-odds) and sufficient statistic $T = \sum X_i \sim \text{Binomial}(n, p)$. By Example 4, MLR holds in $T$. By Karlin-Rubin, the test

is UMP level-$\alpha$. This is exactly the binomial exact test of Topic 17 §17.6 Example 11 — now equipped with an optimality certificate. At $n = 20$, $p_0 = 0.5$, $\alpha = 0.05$, the boundary is $c = 15$ with exact size $0.0207$ (binomialExactRejectionBoundary in testing.ts).

The payoff: the binomial exact test is not just valid — it is the best level-$\alpha$ test for every alternative $p_1 > 0.5$ simultaneously, with no competitor having higher power at any alternative.

Let $X_1, \ldots, X_n$ be iid $\mathcal{N}(\mu, \sigma^2)$ with $\sigma$ known. For $H_0: \mu \le \mu_0$ vs $H_1: \mu > \mu_0$, MLR in $T = \bar X$ follows from Example 4 (Normal with $\sigma$ known is exp-family with $\eta = \mu/\sigma^2$, $T = \sum x_i$). Karlin-Rubin gives UMP

with exact size $\alpha$. No level-$\alpha$ test of this one-sided hypothesis has higher power at any $\mu > \mu_0$ than the one-sided z-test. This result justifies the A/B-testing convention of reporting one-sided p-values when the practical question is directional.

Let $X_1, \ldots, X_n$ be iid $\mathcal{N}(\mu, \sigma^2)$ with $\sigma^2$ unknown. For $H_0: \mu \le \mu_0$ vs $H_1: \mu > \mu_0$, MLR fails: the likelihood depends on both $(\mu, \sigma^2)$ and no scalar $T$ captures the likelihood-ratio monotonicity uniformly. UMP does not exist in the full class of level-$\alpha$ tests. The one-sample t-test is UMP within the restricted class of tests that are invariant under scale transformations $X_i \mapsto c X_i$ — a Hunt-Stein invariance argument (LEH2005 Ch. 6). The t-test retains an optimality certificate, but only conditionally.

The $\sigma$-unknown case is the historical reason Wilks pursued a different asymptotic framework (§18.6): when Karlin-Rubin’s MLR argument fails, the composite LRT provides a general-purpose procedure that does not require MLR but gives up finite-sample optimality in exchange.

The two-sided Normal mean test $H_0: \mu = \mu_0$ vs $H_1: \mu \ne \mu_0$ has no UMP test (Remark 7). The equal-tail z-test is UMPU (LEH2005 Thm 4.4.1): MP within the class of unbiased tests, i.e., those with power $\ge \alpha$ at every alternative. The practical consequence is minor — in concrete applications the UMPU test and the only serious competitors differ by at most a few percent in power. But conceptually it is important: UMP is the exception, UMPU / invariance / LRT are the rule. The composite LRT of §18.5 is the price we pay for general-purpose applicability, and Wilks’ theorem is the justification for believing that price is low.

Let $\Theta$ be the full parameter space and $\Theta_0 \subset \Theta$ the null subspace. The generalized likelihood ratio for a sample $X_1, \ldots, X_n$ is

where $\tilde\theta_n$ is the restricted MLE (maximizer under $H_0$) and $\hat\theta_n$ is the unrestricted MLE. Both are functions of the data. Since $\Theta_0 \subseteq \Theta$, always $0 \le \Lambda_n \le 1$.

The likelihood-ratio test (LRT) at level $\alpha$ rejects $H_0$ when

where $c_\alpha$ is chosen to achieve size $\alpha$ under $H_0$. The conventional asymptotic choice $c_\alpha = \chi^2_{k, 1-\alpha}$, where $k$ is the number of parameter restrictions, is justified by Wilks’ theorem (§18.6 Thm 4). At finite $n$, $c_\alpha$ may need to be calibrated by Monte Carlo for accurate size.

Let $\eta = g(\theta)$ be a smooth one-to-one reparameterization with $H_0^\eta = g(H_0^\theta)$. Then the composite LRT $\Lambda_n$ is invariant:

i.e., computing the LRT in either parameterization yields the same test statistic and the same rejection decision. The Wald and Score tests lack this invariance; see §18.8 Theorem 6 and the concrete logit-vs-raw Bernoulli example in Example 13.

Let $X_1, \ldots, X_n$ be iid $\mathcal{N}(\mu, \sigma^2)$ with both unknown. Test $H_0: \mu = \mu_0$ vs $H_1: \mu \ne \mu_0$.

Under $H_0$: $\sigma^2$ is the only free parameter; its MLE is $\tilde\sigma^2_0 = n^{-1} \sum (X_i - \mu_0)^2$. Under $H_1 \cup H_0$: the MLEs are $\hat\mu = \bar X$, $\hat\sigma^2 = n^{-1} \sum (X_i - \bar X)^2$. A short calculation (expand the likelihoods) gives

where $T_n = \sqrt n (\bar X - \mu_0)/S$ is the t-statistic with $S^2 = (n-1)^{-1} \sum (X_i - \bar X)^2$. So $-2 \log \Lambda_n = n \log(1 + T_n^2/(n-1))$ is a strictly increasing function of $|T_n|$ — the LRT rejects iff $|T_n|$ exceeds a threshold, which is exactly the two-sided t-test rejection rule. The t-test is the LRT.

At large $n$, $-2 \log \Lambda_n \approx T_n^2$ and the asymptotic $\chi^2_1$ reference of §18.6 Thm 4 reduces to the squared-t approximation. At finite $n$, the exact t-test reference (Topic 17 Thm 5 via Basu) is preferred over the asymptotic $\chi^2$.

Let $X_1, \ldots, X_n$ be iid $\mathcal{N}(\mu, \sigma^2)$ with $\mu$ unknown. Test $H_0: \sigma^2 = \sigma_0^2$ vs $H_1: \sigma^2 \ne \sigma_0^2$.

Under $H_0$: $\hat\mu = \bar X$ is free; plug-in gives likelihood $L(\bar X, \sigma_0^2)$. Under $H_1$: $\hat\sigma^2 = n^{-1} \sum (X_i - \bar X)^2$ is the unrestricted MLE. The LRT statistic simplifies to

a function of $W = n \hat\sigma^2 / \sigma_0^2$. The equal-tail $\chi^2$ variance test of Topic 17 §17.8 rejects for extreme $W$; the LRT agrees asymptotically via Wilks’ Thm 4, and the exact $(n-1)S^2/\sigma_0^2 \sim \chi^2_{n-1}$ distribution gives the exact size.

When there are nuisance parameters, the restricted MLE $\tilde\theta_n$ in $\Lambda_n$ is obtained by maximizing over the nuisance parameters at fixed $\theta_0$. This operation — “eliminate the nuisance by profiling” — yields the profile likelihood

a marginal likelihood in $\theta$ alone. Profile likelihood is a CI-construction tool first (Topic 19) and a testing artefact second; §18.6 Example 11 treats the Bernoulli case (no nuisance) directly, and §18.9’s local-power argument likewise handles the scalar-$\theta$ case. For a full treatment of profile, integrated, and conditional likelihoods, see Topic 19 and the Reid-Fraser approach in the nonparametric Bayesian literature.

Wilks’ generalized LRT plays the role of the Swiss-Army-knife testing procedure: applicable to every composite-vs-composite setting (subject to regularity), with a known asymptotic null distribution. It gives up finite-sample optimality (no Karlin-Rubin-style UMP guarantee) but retains asymptotic efficiency in a precise sense. The pragmatic algorithm is the following. If your family has MLR (exponential families, Uniform-scale family), prefer Karlin-Rubin UMP. If not, default to the LRT. If the LRT is analytically intractable, approximate with Wald or Score — noting that the three differ in finite samples (§18.7–§18.8) and that LRT is usually the safest choice when the three disagree. FER1967 develops this decision-theoretic calculus in full; LEH2005 Ch. 12 gives the modern view.

Let $X_1, \ldots, X_n$ be iid from $\{f(\cdot; \theta) : \theta \in \Theta\}$ with $\Theta \subseteq \mathbb{R}$ open. Under $H_0: \theta = \theta_0$ with $\theta_0$ in the interior of $\Theta$, and under Wilks’ regularity — the MLE $\hat\theta_n$ is consistent and asymptotically normal (Topic 14 Thm 14.3); the Fisher information $I(\theta)$ is continuous and positive at $\theta_0$; the third log-density derivative $\partial^3 \log f / \partial\theta^3$ is uniformly bounded in a neighborhood of $\theta_0$ by a function $M(x)$ with $E_{\theta_0}[M(X)] < \infty$ — the log-likelihood-ratio statistic converges in distribution:

where $\Lambda_n = L(\theta_0) / L(\hat\theta_n)$.

Let $X_1, \ldots, X_n$ be iid Bernoulli$(p)$, test $H_0: p = p_0$ vs $H_1: p \ne p_0$. Write $\hat p = \bar X$ for the MLE and recall the three statistics from Topic 17 §17.9:

All three converge to $\chi^2_1$ under $H_0$ (Wilks for the LRT; Wald-and-Score from Topic 17 Thm 7). They differ only by the variance plug-in: Wald uses $\hat p$, Score uses $p_0$, LRT uses an interpolation via the logarithm.

Numerical check: at $p_0 = 0.5$, $n = 100$, $M = 2000$ MC replications with seed 42 (wilksSimulate('bernoulli', 0.5, 100, 2000, undefined, 42) in testing.ts): empirical mean $\approx 1.000$, 95th percentile $\approx 4.03$. The $\chi^2_1$ theoretical targets are mean $1$, 95% quantile $3.84$. The 95th percentile’s slight overshoot at $n = 100$ is the finite-sample overdispersion Wilks cures asymptotically.

The classical regularity used in Proof 3 — existence and continuity of the first three log-density derivatives, uniform boundedness of the third, consistency and asymptotic normality of the MLE — traces back to Wilks’ 1938 paper and Cramér’s 1946 textbook. A rigorous modern treatment via Local Asymptotic Normality (LAN) and contiguity removes the third-derivative hypothesis and replaces it with differentiability in quadratic mean; see [VAN1998] van der Vaart §16 for the empirical-process formulation. For the practical user, the classical conditions suffice; the LAN treatment matters when the underlying family is non-smooth or non-identifiable at specific points (e.g., mixture models, non-regular tails).

The vector-$\theta$ extension replaces the scalar quadratic $n I(\theta_0) (\hat\theta_n - \theta_0)^2$ with the quadratic form $n (\hat\theta_n - \theta_0)^\top \mathbf{I}(\theta_0) (\hat\theta_n - \theta_0)$, where $\mathbf{I}(\theta_0)$ is the Fisher information matrix. By the multivariate CLT and continuous mapping, this quadratic form converges to $\chi^2_k$ where $k = \dim(\Theta) - \dim(\Theta_0)$. The 8-step scalar proof above extends line-for-line; no new ideas are needed, only more bookkeeping. See LEH2005 Thm 12.4.2 for the rigorous statement and proof.