Hypothesis Testing Framework

Fisher’s significance testing (1922) framed the p-value as a continuous measure of evidence against a single null hypothesis: small p-values are surprising, large p-values are not, and the analyst uses judgment. Neyman and Pearson’s decision-theoretic formulation (1933) recast testing as a choice between two hypotheses with two error types (Type I and Type II) and two controlled probabilities ($\alpha$ and $1 - \beta$). The modern textbook framework merges the two: a test statistic produces a p-value, and the p-value is compared to a pre-specified $\alpha$. Different communities weight the Fisherian and Neyman-Pearsonian views differently, and much of the replication-crisis discourse (Remark 6) stems from confusing the two. Topic 17 develops both threads: §17.5 treats the p-value on Fisher’s terms, while §17.3 and §17.4 set up size and power in the Neyman-Pearson frame.

The running example throughout Topic 17 is the two-variant A/B test. Variant A has conversion rate $p_A$; variant B has conversion rate $p_B$. The question is: does deploying variant B produce a higher conversion rate than the incumbent A? The null $H_0: p_B \le p_A$ is the skeptical baseline (no improvement); the alternative $H_1: p_B > p_A$ is the claim we might act on. Every experimentation platform — Optimizely, Statsig, internal platforms at Google / Meta / Netflix — runs some flavor of this test billions of times per year. Sample-size calculations (§17.4 Example 6), two-sample proportion z-tests (§17.6 Example 10), and the multiple-testing correction for many simultaneous experiments (§17.11 Remark 16) are the theoretical foundation of that infrastructure. This connects Topic 17 to formalML: A/B testing platforms .

In estimation, we minimize a single risk (mean squared error, say). In testing, we manage two competing risks simultaneously: Type I error (rejecting a true $H_0$) and Type II error (failing to reject a false $H_0$). These pull in opposite directions — shrinking the rejection region reduces Type I error but increases Type II, and vice versa. No test can drive both to zero at fixed $n$; the entire Neyman-Pearson framework is built around this trade-off. The pragmatic convention is to fix $\alpha$ (Type I rate) at a conventional level (0.05, 0.01, or 0.001) and then maximize power (1 − Type II rate) subject to that constraint. Topic 18 formalizes “maximize power” via the uniformly-most-powerful (UMP) property; here we set up the framework.

Suppose $X_1, \ldots, X_n$ is a sample from a family $\{P_\theta : \theta \in \Theta\}$. A partition $\Theta = \Theta_0 \sqcup \Theta_1$ with $\Theta_0 \ne \emptyset$ and $\Theta_1 \ne \emptyset$ defines two hypotheses:

• The null hypothesis $H_0: \theta \in \Theta_0$ — the claim we wish to disprove.
• The alternative hypothesis $H_1: \theta \in \Theta_1$ — the claim we wish to act on.

A test is a decision rule that accepts one of $H_0$ or $H_1$ on the basis of the observed data. Note the asymmetry: $H_0$ is the default, retained unless the data provide strong evidence against it. “Failing to reject $H_0$” is not the same as “accepting $H_0$” — we remain agnostic, having merely failed to accumulate enough contrary evidence.

A hypothesis $H_i: \theta \in \Theta_i$ is simple if $\Theta_i$ is a singleton (one parameter value) and composite otherwise. A test with simple $H_0$ and simple $H_1$ is the cleanest case — Topic 18’s Neyman-Pearson lemma gives a uniformly most powerful test for it. Most real-world tests are composite on at least one side: $H_0: \mu = 0$ (simple) vs $H_1: \mu \ne 0$ (composite two-sided) is typical.

For a scalar $\theta \in \mathbb{R}$, the standard alternatives are:

• One-sided right: $H_1: \theta > \theta_0$ (used when the analyst has a directional hypothesis — e.g., “the new drug helps”).
• One-sided left: $H_1: \theta < \theta_0$ (the mirror image).
• Two-sided: $H_1: \theta \ne \theta_0$ (the agnostic alternative — “something is different”).

One-sided tests are more powerful against alternatives in the specified direction, but they commit the analyst to that direction before seeing the data. Choosing the side after looking at the data is a form of p-hacking (Remark 6).

Variants A and B have conversion rates $p_A$ and $p_B$. The product team is considering deploying B only if it strictly improves on A. The hypotheses are

$H_0: p_B \le p_A \qquad \text{vs} \qquad H_1: p_B > p_A.$

Both are composite (each covers an interval in $p_B$ at fixed $p_A$). The relevant test statistic (§17.6 Example 10) is the pooled-variance two-sample proportion z-statistic. The one-sided framing captures the business constraint that the team is only interested in improvements, not in detecting arbitrary changes — and, as a bonus, gives more power at fixed $\alpha$ than a two-sided test against the same effect size.

If the product team cares whether A and B differ at all — perhaps because a decrease would also prompt action (roll back B; investigate B’s unintended effects) — the two-sided framing is appropriate:

$H_0: p_B = p_A \qquad \text{vs} \qquad H_1: p_B \ne p_A.$

Here $H_0$ is simple (one parameter value: $p_A$, with $p_B$ forced to equal it) and $H_1$ is composite. The rejection region is the union of two tails, and at the same nominal $\alpha$ it has roughly half the power of Example 1 against a given directional effect — the price paid for detecting the opposite direction as well.

A clinical trial compares a new treatment (mean response $\mu_T$) to standard care ($\mu_S$). Three framings:

• Simple: $H_0: \mu_T = \mu_S$, $H_1: \mu_T = \mu_S + \delta$ for a pre-specified clinically meaningful effect $\delta > 0$. Requires knowing $\delta$ up front; rare in practice but crucial for power calculations.
• One-sided composite: $H_0: \mu_T \le \mu_S$, $H_1: \mu_T > \mu_S$. Matches the regulatory framing: we only approve the treatment if it improves on standard care.
• Two-sided composite: $H_0: \mu_T = \mu_S$, $H_1: \mu_T \ne \mu_S$. Appropriate when an inferior treatment should also prompt action (withdraw, study further).

Choice among the three is a substantive decision driven by the clinical, regulatory, and ethical context — not a statistical one.

A test statistic is a measurable function $T : \mathcal{X}^n \to \mathbb{R}$ that maps the data $X = (X_1, \ldots, X_n)$ to a single real number. Large $|T|$ (or large $T$ for one-sided tests) is interpreted as evidence against $H_0$; the reduction is the one we met in Topic 16 under the sufficiency banner, and indeed most canonical test statistics are functions of a sufficient statistic (e.g., the z-statistic is a function of $\bar X$, which is sufficient for $\mu$ in the Normal family with $\sigma^2$ known).

The rejection region $R \subseteq \mathcal{X}^n$ is the set of data values for which the test rejects $H_0$. When $R$ is specified via the test statistic as $R = \{X : T(X) > c\}$ (right-tailed), $R = \{X : T(X) < c\}$ (left-tailed), or $R = \{X : |T(X)| > c\}$ (two-tailed), the threshold $c$ is called the critical value and denoted $c_\alpha$ when calibrated to level $\alpha$.

The size of a test with rejection region $R$ is

$\alpha = \sup_{\theta \in \Theta_0} P_\theta(X \in R).$

That is, the size is the worst-case Type I error rate over the null set. A test has level $\alpha$ if its size is at most $\alpha$ — every test of size $\alpha$ has level $\alpha$, but a test of level $\alpha$ may have strictly smaller size. The distinction matters when the null distribution is discrete (Thm 1, Remark 4, Example 4).

For continuous null distributions, any $\alpha \in (0, 1)$ is achievable exactly: the critical value $c_\alpha$ can be chosen so that $P_{H_0}(T > c_\alpha) = \alpha$ on the nose. For discrete null distributions (e.g., Binomial, Poisson), only a countable set of $\alpha$ values is achievable; other levels force a conservative test, with actual size strictly less than the designed level.

When the null distribution is discrete, the exact achievable sizes are the tail probabilities of that distribution — a discrete set. For Binomial$(20, 0.5)$, the right-tail probabilities $P(X \ge x)$ at integer $x$ give exact sizes $\{1, 0.999, 0.994, \ldots, 0.021, 0.006, 0.001, 0.00009, 10^{-6}\}$ — the sizes jump at each integer. A test designed to have level $\alpha = 0.05$ must pick a boundary whose exact size is $\le 0.05$; for $H_0: p = 0.5$ at $n = 20$, the best achievable right-tailed size is $0.021$ (at boundary $x = 15$), strictly less than $0.05$. This is not a flaw; it is the price of exactness. Conservative tests preserve the Type I guarantee at the cost of some power, and they are preferred over alternatives (randomization, mid-p) in most practical settings.

Test $H_0: p = 0.5$ vs $H_1: p > 0.5$ using $T = \sum_{i=1}^{20} X_i$ with rejection region $\{T \ge 15\}$. Under $H_0$, $T \sim \mathrm{Binomial}(20, 0.5)$, so

$P_{H_0}(T \ge 15) = \sum_{k = 15}^{20} \binom{20}{k} (0.5)^{20} = \frac{15504 + 4845 + 1140 + 190 + 20 + 1}{2^{20}} = \frac{21700}{1048576} \approx 0.0207.$

The exact size is $0.0207 < 0.05$, so the test is conservative at the nominal $\alpha = 0.05$. Moving the boundary to $T \ge 14$ would give size $\approx 0.058 > 0.05$ — unacceptable. The next smaller achievable size is $0.0207$; the one after that is $0.0059$. No boundary achieves $\alpha = 0.05$ exactly, and the analyst must choose between conservative ($0.0207$) and anti-conservative ($0.0577$) rounding. This is the discrete-test pedagogical example that §17.6 Example 11 revisits in the binomial-exact treatment.

• Type I error: rejecting $H_0$ when $H_0$ is true. Its probability is the size $\alpha$ (§17.3 Def 6).
• Type II error: failing to reject $H_0$ when $H_1$ is true. Its probability at a specific $\theta \in \Theta_1$ is conventionally written $1 - \beta(\theta)$ in the Neyman-Pearson convention where $\beta$ denotes power; some texts reverse this convention and write $\beta(\theta)$ for the Type II rate. Topic 17 follows Lehmann-Romano: $\beta(\theta)$ is power (probability of rejection at $\theta$), so Type II rate is $1 - \beta(\theta)$.

The power function of a test with rejection region $R$ is

$\beta(\theta) = P_\theta(X \in R), \qquad \theta \in \Theta.$

Evaluated at $\theta \in \Theta_0$, $\beta(\theta)$ is the Type I error rate at that null parameter (and $\sup_{\theta \in \Theta_0}\beta(\theta) = \alpha$, the size). Evaluated at $\theta \in \Theta_1$, $\beta(\theta)$ is the (actual) probability of correctly rejecting $H_0$ at the specific alternative $\theta$. A good test has $\beta$ close to $\alpha$ on $\Theta_0$ and close to 1 on $\Theta_1$; how close depends on the sample size and the distance between $\theta$ and $\Theta_0$.

For a one-parameter exponential family with monotone likelihood ratio (MLR) in the sufficient statistic $T$, the one-sided right-tail test $\{T > c\}$ has power $\beta(\theta)$ that is non-decreasing in $\theta$. In particular, for any $\theta_1 > \theta_0$, $\beta(\theta_1) \ge \beta(\theta_0) = \alpha$.

For iid $\mathcal{N}(\mu, \sigma^2)$ data with $\sigma^2$ known, consider the right-tailed z-test of $H_0: \mu = \mu_0$ vs $H_1: \mu > \mu_0$ at level $\alpha$. The test rejects when $Z = \sqrt n(\bar X - \mu_0)/\sigma > z_\alpha$, where $z_\alpha = \Phi^{-1}(1 - \alpha)$ is the $(1 - \alpha)$-quantile of the standard Normal. Under $\mu$, $Z \sim \mathcal{N}(\sqrt n(\mu - \mu_0)/\sigma,\, 1)$, so

$\beta(\mu) = P_\mu(Z > z_\alpha) = 1 - \Phi\!\left(z_\alpha - \frac{\sqrt n\,(\mu - \mu_0)}{\sigma}\right).$

This closed-form expression is the analytic engine of the sample-size calculator in Example 6 and the PowerCurveExplorer component. Key observations: $\beta$ increases monotonically in $\mu$ (Thm 2); $\beta(\mu_0) = \alpha$; $\beta(\mu) \to 1$ as $\mu \to \infty$ at rate $\sqrt n / \sigma$ (the test is consistent).

Suppose we want power $1 - \beta = 0.8$ against the alternative $\mu = \mu_0 + 0.5\sigma$ (a “half-$\sigma$” effect) at level $\alpha = 0.05$, one-sided. Setting $\beta(\mu) = 0.8$ in Example 5 and solving:

$0.8 = 1 - \Phi\!\left(z_{0.05} - \frac{\sqrt n\,(0.5\sigma)}{\sigma}\right) = 1 - \Phi(1.645 - 0.5\sqrt n).$

Inverting, $\Phi(1.645 - 0.5\sqrt n) = 0.2$, so $1.645 - 0.5\sqrt n = \Phi^{-1}(0.2) = -0.8416$, giving $\sqrt n = (1.645 + 0.8416)/0.5 \approx 4.97$, hence $n \ge 24.7$. Rounding up, $n = 25$. More generally, for effect size $\delta$ measured in $\sigma$-units,

$n = \frac{(z_\alpha + z_\beta)^2 \sigma^2}{\delta^2},$

the textbook sample-size formula. The PowerCurveExplorer (§17.4 component) implements this closed form for the Normal scenarios and a numerical inversion for the binomial exact test.

In a regular parametric family, the Cramér-Rao lower bound (Thm 13.9) says any unbiased estimator $\hat\theta$ satisfies $\mathrm{Var}(\hat\theta) \ge 1/[n I(\theta)]$, where $I(\theta)$ is the Fisher information at $\theta$. This translates directly into an upper bound on achievable power for any asymptotically-Normal-based test: the smaller the estimator variance, the more concentrated its sampling distribution, and the larger the separation between $H_0$ and $H_1$ sampling distributions — hence higher power. Topic 18 formalizes this as the CRLB giving a “power envelope” that UMP tests achieve. At intermediate level, the intuition is: power is to the estimator’s variance what accuracy is to its bias, and both are bounded below by Fisher information.

For a test with test statistic $T(X)$ and a right-tailed rejection rule $\{T > c\}$, the p-value of an observed data point $x$ is

$p(x) = P_{H_0}(T(X) \ge T(x)).$

That is, the p-value is the tail probability, under $H_0$, of the test statistic being at least as extreme as the observed value. For left-tailed tests replace $\ge$ with $\le$; for two-sided tests use $2 \min(P_{H_0}(T \ge T(x)),\,P_{H_0}(T \le T(x)))$ for symmetric null distributions (a convention; a few alternatives exist for non-symmetric nulls, discussed in §17.6 Example 11 for the binomial case).

The rejection rule “reject when $p \le \alpha$” is equivalent to “reject when $T > c_\alpha$” — the two formulations give identical decisions. The advantage of the p-value is that it makes the level at which rejection occurs transparent, rather than forcing an all-or-nothing decision at a fixed $\alpha$.

Let $T$ be a test statistic with continuous CDF $F$ under $H_0$. For a right-tailed test, the p-value $p(X) = 1 - F(T(X))$. Under $H_0$, $p(X) \sim \mathrm{Uniform}(0, 1)$.

Simulate 10,000 samples of size $n = 30$ from $\mathcal{N}(0, 1)$ (the null distribution). For each, compute the z-test statistic $Z = \sqrt n \bar X$ and its right-tailed p-value $p = 1 - \Phi(Z)$. The histogram of the 10,000 p-values is visibly flat on $[0, 1]$ — the empirical verification of Thm 3. Under an alternative (say $\mu = 0.5$), the same MC run gives a histogram concentrated near zero: power visualized as a distribution on the p-value scale. The PValueDemonstrator component (§17.5) makes this interactive.

A string of papers from the 2005–2016 period argued that naive p-value interpretation is a major driver of non-replicable research. Ioannidis (2005), Why Most Published Research Findings Are False, used a prior-probability argument to show that a statistically significant finding ($p < 0.05$) is often more likely to be a false positive than a true positive, depending on the prior probability of the null — p-values are emphatically not $P(H_0 \mid \text{data})$. Gelman & Loken (2013), The Garden of Forking Paths, argued that even well-intentioned researchers inflate Type I error through researcher degrees of freedom (p-hacking, HARKing) without formal multiple testing. The ASA Statement on p-Values (Wasserstein & Lazar, 2016) synthesized the critique into six principles, emphasizing that a p-value quantifies inconsistency with a null model under specific assumptions, not the probability of a hypothesis, and not the importance of an effect.

These are genuine warnings; Topic 20 addresses the multiple-testing piece formally (Bonferroni, Holm, Benjamini-Hochberg FDR, Šidák — see especially §20.7 for the featured BH proof and §20.8 for the Ioannidis / Gelman-Loken framing). For Topic 17, the takeaway is: a p-value is a single number reporting a tail probability under a stated null; it is not a posterior probability, not an effect size, and not a replacement for thinking about the experimental design, power, and prior plausibility.

One-sample z-test. For iid data $X_1, \ldots, X_n$ from a distribution with mean $\mu$ and known variance $\sigma^2$, the z-statistic for testing $H_0: \mu = \mu_0$ is

$Z = \frac{\sqrt n\,(\bar X - \mu_0)}{\sigma}.$

Under $H_0$, and under Normality of the data, $Z \sim \mathcal{N}(0, 1)$ exactly. For non-Normal data, $Z \xrightarrow{d} \mathcal{N}(0, 1)$ under $H_0$ by the CLT — an asymptotic result with finite-sample accuracy governed by Berry-Esseen (Topic 11 §11.4).

Two-sample z-test. For two independent samples $X_1, \ldots, X_{n_1}$ and $Y_1, \ldots, Y_{n_2}$ with known variances $\sigma_1^2$ and $\sigma_2^2$, the two-sample z-statistic for testing $H_0: \mu_1 = \mu_2$ is

$Z = \frac{\bar X - \bar Y}{\sqrt{\sigma_1^2/n_1 + \sigma_2^2/n_2}}.$

Under $H_0$ (and Normality or asymptotically), $Z \sim \mathcal{N}(0, 1)$.

Let $X_1, \ldots, X_n$ be iid $\mathcal{N}(\mu_0, \sigma^2)$ with $\sigma^2$ known. For the two-sided test that rejects $H_0: \mu = \mu_0$ when $|Z| > z_{\alpha/2}$ (where $Z = \sqrt n(\bar X - \mu_0)/\sigma$ and $z_{\alpha/2} = \Phi^{-1}(1 - \alpha/2)$), the size is exactly $\alpha$.

A random sample of $n = 50$ adults has mean IQ $\bar x = 103.2$ and we wish to test whether the population mean differs from the standardized value $\mu_0 = 100$, assuming $\sigma = 15$ (the calibration target of the instrument). The z-statistic is

$z = \frac{\sqrt{50}\,(103.2 - 100)}{15} = \frac{7.071 \cdot 3.2}{15} \approx 1.508.$

Two-sided p-value: $p = 2\,(1 - \Phi(1.508)) \approx 0.131$. At $\alpha = 0.05$ we fail to reject. The point estimate is higher than $\mu_0$, but the evidence is not strong — a half-$\sigma$ effect would need $n \gtrsim 32$ to reach power 0.8 (Example 6), and we observed about $0.21 \sigma$.

A trial enrolls $n_1 = 100$ patients in the treatment arm (blood-pressure reduction $\bar X = 12.4$ mmHg, assumed $\sigma_1 = 8$) and $n_2 = 100$ in the placebo arm ($\bar Y = 8.1$, $\sigma_2 = 8$). Testing $H_0: \mu_T = \mu_P$:

$Z = \frac{12.4 - 8.1}{\sqrt{64/100 + 64/100}} = \frac{4.3}{\sqrt{1.28}} \approx 3.80.$

Two-sided p-value $\approx 0.00015$ — strongly significant. The observed treatment effect of $4.3$ mmHg at these standard errors is well outside the rejection region at any conventional $\alpha$.

An experiment ran $n_A = n_B = 10000$ impressions per variant. Variant A produced $k_A = 1200$ conversions ($\hat p_A = 0.12$); variant B produced $k_B = 1260$ ($\hat p_B = 0.126$). Testing $H_0: p_A = p_B$ with the pooled-variance z-statistic:

$\hat p = \frac{k_A + k_B}{n_A + n_B} = \frac{2460}{20000} = 0.123, \qquad \mathrm{SE} = \sqrt{\hat p (1 - \hat p)\left(\frac{1}{n_A} + \frac{1}{n_B}\right)} = \sqrt{0.123 \cdot 0.877 \cdot 0.0002} \approx 0.00465.$

$Z = \frac{0.126 - 0.12}{0.00465} \approx 1.29.$

Two-sided p-value $\approx 0.197$; we fail to reject at $\alpha = 0.05$. The detected lift (0.6 percentage points, 5% relative) is plausible — but not strong enough at this sample size. A typical platform would either run the experiment longer or accept the null (“no detectable effect”) given the sample-size budget. This is the canonical output of every A/B-testing platform on the planet; the interactive version of this calculation is in the NullAlternativeSimulator and PowerCurveExplorer components.

Return to $H_0: p = 0.5$ vs $H_1: p > 0.5$ at $n = 20$ with observed $x_{\mathrm{obs}} = 15$. Two approaches:

Exact test. The null distribution of $X = \sum X_i$ is $\mathrm{Binomial}(20, 0.5)$. The right-tailed p-value is the exact tail probability:

$p_{\mathrm{exact}} = P_{H_0}(X \ge 15) = \sum_{k=15}^{20} \binom{20}{k} (0.5)^{20} \approx 0.0207.$

At $\alpha = 0.05$, we reject. The rejection region $\{X \ge 15\}$ has exact size $0.0207$ — conservative (Remark 4).

Normal approximation. Under $H_0$, $X \approx \mathcal{N}(10, 5)$; the standardized observed value is $z = (15 - 10)/\sqrt 5 \approx 2.236$, giving p-value $\approx 0.0127$. With a continuity correction (using $14.5$ instead of $15$), the p-value becomes $\approx 0.0207$ — close to the exact. Without continuity correction, the Normal approximation overstates significance.

Why this matters. At $n = 20$, the Normal approximation p-value can be 30-40% off the exact p-value at moderate thresholds. At $n = 200$ it is typically within a few percent. The notebook figure (right panel) shows the size comparison across $n \in [10, 200]$: exact size stays below $\alpha$ by a discrete jump; Normal-approximation size oscillates above and below $\alpha$, converging as $n \to \infty$. The binomial exact test has the additional property — covered in Topic 18 — of being UMP one-sided for the Bernoulli family.

For small-$n$ discrete tests, the Normal approximation can miss the actual size by several percentage points — not an academic concern but a practical one. An experimenter who believes their nominal $\alpha$ is $0.05$ but whose actual Type I rate is $0.08$ is reporting misleading guarantees. For A/B tests at $n$ in the hundreds or more, the Normal approximation is typically accurate to within a percentage point; for medical or behavioral studies at $n \le 50$, exact tests are the responsible choice. The PowerCurveExplorer component includes a Binomial exact scenario that inverts the exact size via root-find (not the Normal approximation), letting the reader see this effect directly.

One-sample t-test. For iid Normal data $X_1, \ldots, X_n \sim \mathcal{N}(\mu, \sigma^2)$ with both $\mu$ and $\sigma^2$ unknown, the t-statistic for testing $H_0: \mu = \mu_0$ is

$T = \frac{\sqrt n\,(\bar X - \mu_0)}{S}, \qquad S^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar X)^2.$

Under $H_0$, $T \sim t_{n-1}$ (Student’s t distribution with $n - 1$ degrees of freedom) exactly — Thm 5 below, via Basu.

Two-sample pooled t-test. For two independent iid Normal samples with a common unknown variance, the pooled t-statistic for testing $H_0: \mu_1 = \mu_2$ is

$T = \frac{\bar X - \bar Y}{S_p\sqrt{1/n_1 + 1/n_2}}, \qquad S_p^2 = \frac{(n_1 - 1)S_1^2 + (n_2 - 1)S_2^2}{n_1 + n_2 - 2}.$

Under $H_0$ and the common-variance assumption, $T \sim t_{n_1 + n_2 - 2}$ exactly. When variances are unequal, Welch’s modification (welchTStatistic in testing.ts) gives an approximate $t_{\nu}$ distribution with Satterthwaite degrees of freedom — asymptotic exactness only.

Let $X_1, \ldots, X_n$ be iid $\mathcal{N}(\mu_0, \sigma^2)$ with both parameters unknown. Under $H_0: \mu = \mu_0$, the one-sample t-statistic

$T_n = \frac{\sqrt n\,(\bar X - \mu_0)}{S}$

has distribution $t_{n-1}$ exactly.

A small clinical study measures fasting glucose in $n = 10$ patients on a new regimen: $\bar x = 105$ mg/dL, $s = 12$ mg/dL. Testing $H_0: \mu = 100$ (the standard reference value) two-sided:

$T = \frac{\sqrt{10}\,(105 - 100)}{12} = \frac{5\sqrt{10}}{12} \approx 1.318.$

Using t: Two-sided p-value from the $t_9$ distribution: $p_t = 2\,(1 - F_{t_9}(1.318)) \approx 0.220$.

Using z (wrong!): Two-sided z-approximation p-value: $p_z = 2\,(1 - \Phi(1.318)) \approx 0.187$.

The t p-value is $\sim 18\%$ larger than the z approximation — the $t_9$ density has heavier tails than $\mathcal{N}(0, 1)$, so extreme values are less surprising under $t_9$ and the resulting p-value is larger. For small $n$, ignoring this difference inflates the Type I rate. At $n = 30$ the discrepancy shrinks to $\sim 3\%$; at $n = 100$ it is negligible.

A lab tests two catalysts for a reaction yield: $n_1 = 8$ trials with catalyst A ($\bar X = 72$, $S_1 = 4.5$), $n_2 = 8$ trials with catalyst B ($\bar Y = 68$, $S_2 = 5.1$). Testing $H_0: \mu_A = \mu_B$ with equal-variance pooling:

$S_p^2 = \frac{7 \cdot 4.5^2 + 7 \cdot 5.1^2}{14} = \frac{141.75 + 182.07}{14} \approx 23.13, \qquad S_p \approx 4.81.$

$T = \frac{72 - 68}{4.81\,\sqrt{1/8 + 1/8}} = \frac{4}{4.81 \cdot 0.5} = \frac{4}{2.405} \approx 1.663.$

Two-sided p-value from $t_{14}$: $\approx 0.119$. We fail to reject at $\alpha = 0.05$; the observed difference in yields is consistent with sampling noise at this $n$. For a detectable effect at $\alpha = 0.05$ with $1 - \beta = 0.8$, a similar magnitude would need roughly $n \ge 23$ per arm — a typical power-calculation output.

Proof 3 Step 4 is the load-bearing step. Without Basu’s $\bar X \perp\!\!\!\perp S^2$, the ratio $T_n = Z/\sqrt{V/(n-1)}$ has a distribution that depends in a complicated way on the joint distribution of $(\bar X, S^2)$ — and $t_{n-1}$ is no longer the answer. Every finite-sample inference procedure for the one-sample Normal mean with unknown variance — the t-test, the t-confidence-interval, and their two-sample siblings — rests on this independence. Topic 16 gave Basu with a full proof and flagged the forward payoff; Topic 17 §17.7 is that payoff. For the featured visualization of this, see the TTestBasuFoundation component below, which shows the decorrelated scatter of $(\bar X, S^2)$ across MC replications, the resulting t-statistic histogram matching $t_{n-1}$, and a direct link back to Topic 16 §16.9.

Variance test. For iid Normal data $X_1, \ldots, X_n \sim \mathcal{N}(\mu, \sigma^2)$ with unknown $\mu$, the test statistic for $H_0: \sigma^2 = \sigma_0^2$ is

$W = \frac{(n-1)\,S^2}{\sigma_0^2}.$

Under $H_0$, $W \sim \chi^2_{n-1}$ exactly (Thm 6 below).

Pearson goodness-of-fit. For a categorical distribution with $k$ cells, observed counts $O_1, \ldots, O_k$ (summing to $n$), and expected counts $E_1, \ldots, E_k$ under $H_0$, the Pearson statistic is

$X^2 = \sum_{k=1}^K \frac{(O_k - E_k)^2}{E_k}.$

Under $H_0$, and assuming all $E_k$ are not too small (rule of thumb: $E_k \ge 5$), $X^2 \xrightarrow{d} \chi^2_{k - 1 - r}$ where $r$ is the number of parameters estimated from the data (zero if the null fixes all cell probabilities). The derivation is asymptotic, based on the multinomial CLT and continuous mapping.

For iid $\mathcal{N}(\mu, \sigma_0^2)$ data under $H_0: \sigma^2 = \sigma_0^2$, the test statistic $W = (n-1)\,S^2/\sigma_0^2$ has distribution $\chi^2_{n-1}$ exactly.

A manufacturing process specifies $\sigma_0^2 = 4$ (mm²) for a critical dimension. A sample of $n = 15$ units has $s^2 = 6.8$. Testing $H_0: \sigma^2 = 4$ vs $H_1: \sigma^2 > 4$ (right-tailed, since excess variance is the quality concern):

$W = \frac{14 \cdot 6.8}{4} = 23.8.$

Right-tailed p-value from $\chi^2_{14}$: $p = 1 - F_{\chi^2_{14}}(23.8) \approx 0.048$. We reject at $\alpha = 0.05$ (narrowly) — evidence that the process variance exceeds the specification. Contrast with the two-sided version: for a two-sided variance test at $\alpha = 0.05$, we use the equal-tailed p-value $2\,\min(F, 1 - F) = 2 \cdot 0.048 = 0.096$ — we would not reject. The asymmetry of the $\chi^2$ density makes the two-sided variance test less powerful than the one-sided version against one-sided alternatives.

A die is rolled $n = 120$ times; the six face counts are $O = (18, 22, 17, 25, 19, 19)$. Under $H_0$: the die is fair ($E_k = 20$ for all $k$):

$X^2 = \sum_{k=1}^{6} \frac{(O_k - 20)^2}{20} = \frac{4 + 4 + 9 + 25 + 1 + 1}{20} = \frac{44}{20} = 2.2.$

Right-tailed p-value from $\chi^2_5$ (df $= k - 1 - 0 = 5$, no parameters estimated): $p \approx 0.82$. The observed counts are easily consistent with a fair die. Note that the $\chi^2$ asymptotic here rests on the multinomial CLT, and degrades when some $E_k$ are small ($< 5$ is the classical rule of thumb); Lehmann-Romano Ch. 14 covers small-$E$ corrections.