Multiple Testing & False Discovery

A product team runs twenty simultaneous A/B tests on unrelated features, each at level $\alpha = 0.05$. None of the twenty has a true effect — the global null holds. The probability that at least one test falsely flags its feature as significant is $1 - 0.95^{20} \approx 0.64$. A team that deploys whichever features cross the $p < 0.05$ threshold will, in expectation, ship at least one null feature in this batch. Doing this every quarter across a company compounds the effect: after four quarters, the probability of never having shipped a null feature is $0.36^4 \approx 0.017$.

A GWAS tests $m \approx 10^6$ common variants against a trait. The union-bound control at $\alpha = 0.05$ gives a per-test threshold of $\alpha/m = 5 \times 10^{-8}$ — the number that has become the de facto field standard for “genome-wide significance.” It is exactly the Bonferroni correction of §20.4 applied to the GWAS scale. The FDR-controlled threshold for the same data is typically two or three orders of magnitude less stringent (§20.7 Ex 8), which is why BH-based analyses routinely identify hits that genome-wide-significance analyses miss.

A multiple-testing problem consists of $m$ hypotheses $H_1, \ldots, H_m$ with null distributions $P_1^{(0)}, \ldots, P_m^{(0)}$ and test statistics $T_1, \ldots, T_m$ producing p-values $p_1, \ldots, p_m$. Let $H_0 \subseteq \{1, \ldots, m\}$ denote the subset of indices where $H_i$ is true (the true nulls), with $m_0 = |H_0|$ and $m_1 = m - m_0 = |H_0^c|$ the count of true alternatives. Write $\pi_0 = m_0 / m$ for the true null fraction.

A multiple-testing procedure is a function $(p_1, \ldots, p_m) \mapsto \mathcal{R} \subseteq \{1, \ldots, m\}$ returning the set of rejected hypotheses. Counts of interest: $R = |\mathcal{R}|$ (total rejections), $V = |\mathcal{R} \cap H_0|$ (false rejections), $S = |\mathcal{R} \cap H_0^c|$ (true rejections); by construction $R = V + S$. Throughout the topic, $\mathcal{R}$ denotes the rejection set and $R$ its cardinality.

Bonferroni’s correction — apply each test at level $\alpha/m$ — is the canonical fix and will be formalized in §20.4. It works, but it is uniformly conservative: the FWER it actually achieves under the global null is at most $m_0 \alpha / m \le \alpha$, often much less. The rest of this topic is about more powerful procedures that retain the FWER guarantee (Holm, Hochberg, closed testing) or relax it to the weaker-but-often-preferred FDR guarantee (BH, BY, Storey). The pedagogical arc: $\alpha/m$ is the floor, not the ceiling.

The exact formula $1 - (1 - \alpha)^m$ uses independence; the union bound $m \alpha$ does not. Under positive dependence — the typical regime for correlated tests in an A/B platform or for linkage-disequilibrium-correlated SNPs in a GWAS — the exact FWER is smaller than the independent-case formula (the false rejections tend to cluster). Under negative dependence, it can be larger, but negative dependence is rare in practice. The upshot: $m \alpha$ is a safe universal upper bound; $1 - (1 - \alpha)^m$ is tight under independence; the truth-under-positive-dependence is somewhere in between.

Let $\phi_I$ be a level-$\alpha$ test of $H_I = \bigcap_{i \in I} H_i$ for every non-empty $I \subseteq \{1, \ldots, m\}$. Define the closed procedure to reject $H_i$ iff $\phi_J$ rejects for every $J \ni i$. Then for any true-null configuration $H_0 \subseteq \{1, \ldots, m\}$:

$\Pr_{H_0}\bigl(\text{at least one } H_i \text{ with } i \in H_0 \text{ is rejected}\bigr) \;\le\; \alpha.$

No dependence assumption on the individual test statistics is required.

The theorem above says “pick level-$\alpha$ tests of every intersection and combine them correctly.” It does not tell you which tests to pick. Bonferroni-based intersection tests (reject $H_I$ iff $\min_{i \in I} p_i \le \alpha / |I|$) give the Holm procedure. Fisher-combination intersection tests (reject $H_I$ iff $-2 \sum_{i \in I} \log p_i$ exceeds a $\chi^2_{2|I|}$ critical value) give a different, often more powerful closed procedure. Simes-combination tests give Hochberg. The space of reasonable closed procedures is large; practitioners gravitate to Holm because it is parameter-free and always-valid.

The full proof machinery — proving strong control and showing Holm is the closed procedure with Bonferroni intersection tests — takes roughly 15 additional lines of careful indexing. It adds no conceptual insight beyond what the one-line sketch above already conveys, and it distracts from the BH proof in §20.7, which does deserve a full derivation. Closed testing is the scaffold; the specific procedures we prove correct (Bonferroni, Holm, BH) are the load-bearing walls.

For a multiple-testing procedure producing rejection set $\mathcal{R}$ on data with true-null configuration $H_0$, the family-wise error rate at $H_0$ is

$\text{FWER}(H_0) \;=\; \Pr_{H_0}(V \ge 1) \;=\; \Pr_{H_0}(\mathcal{R} \cap H_0 \ne \emptyset).$

A procedure controls FWER at level $\alpha$ if $\text{FWER}(H_0) \le \alpha$ for every configuration $H_0$.

A procedure controls FWER in the weak sense if $\text{FWER}(H_0) \le \alpha$ only when $H_0 = \{1, \ldots, m\}$ (the global null — every $H_i$ true). It controls FWER in the strong sense if $\text{FWER}(H_0) \le \alpha$ for every $H_0 \subseteq \{1, \ldots, m\}$, including the partial-null configurations where some $H_i$ are true and some are false.

Strong control is what practitioners want: the guarantee should hold whether or not the alternatives are real. Weak control is an academic fallback. Every procedure in this topic controls FWER or FDR in the strong sense unless explicitly noted.

At $m = 1$ FWER collapses to the per-test Type I error $\alpha$ of formalML: . FWER is the multi-hypothesis generalization: it replaces “probability this particular test falsely rejects” with “probability any of the $m$ tests falsely rejects.” The two notions agree at $m = 1$ and diverge sharply for large $m$, as §20.1 Fig 1 shows.

Holding each test’s nominal level fixed, controlling FWER at level $\alpha$ forces each individual test to use a smaller effective $\alpha_i \le \alpha$. Smaller per-test level means smaller per-test power. The trade-off is quantitative: under Bonferroni at global $\alpha = 0.05$, a test of a single alternative with effect size $\delta$ that had power $0.80$ at $\alpha = 0.05$ with $m = 1$ retains only power $\approx 0.60$ at $m = 20$ and $\approx 0.25$ at $m = 200$ — a dramatic loss. The FDR framework of §20.6 sacrifices some of FWER’s protection to claw back this power, and is the right tool whenever false discoveries (rather than any false rejection whatsoever) are the cost metric.

The Bonferroni procedure rejects $H_i$ iff $p_i \le \alpha/m$. It controls FWER at level $\alpha$ in the strong sense under any joint dependence structure among the test statistics.

Suppose the test statistics $T_1, \ldots, T_m$ are independent under $H_0$. Then the Šidák procedure — reject $H_i$ iff $p_i \le 1 - (1 - \alpha)^{1/m}$ — controls FWER in the strong sense at exactly $\alpha$ when $m_0 = m$ and at most $\alpha$ otherwise.

Per-test threshold $\alpha/m = 0.0025$. A test with a $p$-value of $0.04$ — significant at the $\alpha = 0.05$ level for a single test — is not significant under Bonferroni. The union-bound-guaranteed FWER is at most $20 \times 0.0025 = 0.05$; under independence the exact FWER is $1 - (1 - 0.0025)^{20} = 0.0488$, slightly below the bound.

For essentially every routine application, the two give numerically indistinguishable thresholds. The deciding factor is the dependence story: Bonferroni is correct under any dependence, Šidák is exact under independence and conservative-then-liberal under positive-then-negative dependence. Textbook tables default to Bonferroni; the software outputs often default to Šidák (e.g., R’s p.adjust(method = 'bonferroni') vs the closely-related Šidák implementations). The practical answer is almost always the same hypothesis-level decisions.

The Bonferroni intersection test — “reject $H_I$ iff $\min_{i \in I} p_i \le \alpha / |I|$” — is itself a valid level-$\alpha$ test of the intersection hypothesis $\bigcap_{i \in I} H_i$. Applying closed testing with Bonferroni intersection tests produces exactly the Holm step-down procedure of §20.5 (LEH2005 §9.1). This is the cleanest way to see why Holm strictly dominates Bonferroni: the closed procedure exploits the fact that once a hypothesis is rejected, the remaining intersections shrink, and subsequent Bonferroni thresholds become less stringent.

Order the $p$-values ascending: $p_{(1)} \le p_{(2)} \le \cdots \le p_{(m)}$. Let $k^\dagger$ be the smallest rank $k$ such that $p_{(k)} > \alpha / (m - k + 1)$ (set $k^\dagger = m + 1$ if no such $k$ exists). The Holm procedure rejects the hypotheses corresponding to $p_{(1)}, \ldots, p_{(k^\dagger - 1)}$. This controls FWER at level $\alpha$ in the strong sense under any joint dependence structure.

Order the $p$-values ascending. Let $k^*$ be the largest rank $k$ such that $p_{(k)} \le \alpha / (m - k + 1)$ (set $k^* = 0$ if no such $k$ exists). The Hochberg procedure rejects $p_{(1)}, \ldots, p_{(k^*)}$. This controls FWER at level $\alpha$ under independence or positive regression dependence (PRDS).

Suppose 3 of the 20 $p$-values are below $0.0025$: specifically $p_{(1)} = 0.0003, p_{(2)} = 0.0012, p_{(3)} = 0.0021$, and $p_{(4)} = 0.004$. Bonferroni rejects the first three. Holm: $p_{(1)} = 0.0003 \le 0.05/20 = 0.0025$ (pass), $p_{(2)} = 0.0012 \le 0.05/19 \approx 0.00263$ (pass), $p_{(3)} = 0.0021 \le 0.05/18 \approx 0.00278$ (pass), $p_{(4)} = 0.004 \le 0.05/17 \approx 0.00294$? No — $0.004 > 0.00294$, so Holm stops and rejects $p_{(1)}, p_{(2)}, p_{(3)}$. Same rejections as Bonferroni in this case, but Holm’s threshold at rank 4 is less stringent than Bonferroni’s constant $0.0025$, so if $p_{(4)}$ had been, say, $0.0027$, Holm would additionally reject it while Bonferroni would not.

Take $m = 5$, $\alpha = 0.05$, ordered $p$-values $0.006, 0.012, 0.018, 0.030, 0.045$. Holm: $p_{(1)} = 0.006 \le 0.01$ (pass), $p_{(2)} = 0.012 \le 0.0125$ (pass), $p_{(3)} = 0.018 \le 0.0167$? No — $0.018 > 0.0167$, stop. Rejects 2. Hochberg: start at $p_{(5)} = 0.045 \le 0.05$? Yes — reject all 5. The same inputs give very different answers: Hochberg’s step-up detects the global pattern (“all five look moderately small”) while Holm’s step-down commits early. Hochberg is valid under independence; Holm makes no such assumption.

Open the explorer below; set $m = 200$, $\alpha = 0.05$; sweep $\pi_0$ from $0.5$ (dense signal) to $0.99$ (sparse). At $\pi_0 = 0.5$ Holm and BH both recover most of the 100 alternatives; Bonferroni recovers perhaps $40$. At $\pi_0 = 0.99$ Holm and Bonferroni converge (both at $\approx 2$ rejections) while BH retains nearly all of the 2 expected true alternatives. This is the core FDR-vs-FWER story and it is best seen interactively.

The step-down structure guarantees that Holm’s rejection set contains Bonferroni’s: if $p_{(k)} \le \alpha/m$, then a fortiori $p_{(k)} \le \alpha/(m-k+1)$. So every Bonferroni rejection is a Holm rejection, and Holm potentially adds more as the rank grows. There is no regime where using Bonferroni instead of Holm is better; use Holm by default.

Hochberg’s step-up is strictly more powerful than Holm’s step-down under independence, but not under arbitrary dependence. In GWAS (LD-correlated tests) and most A/B test multiplicity, positive dependence holds and Hochberg is safe and preferred. In high-dimensional regression where the design matrix is ill-conditioned, dependence structure can be complex and Holm is the conservative default.

Applying the closed-testing scaffold (§20.2 Thm 0) with Bonferroni intersection tests — “reject $H_I$ iff $\min_{i \in I} p_i \le \alpha / |I|$” — produces exactly the Holm step-down procedure. This is formalML: §9.1 Thm 9.1.2. The derivation explains why Holm’s rank-dependent threshold is $\alpha/(m - k + 1)$: at step $k$, the remaining candidate intersection has size $m - k + 1$, and its Bonferroni threshold is $\alpha/(m - k + 1)$.

No. Holm is admissible among step-down procedures that control FWER under arbitrary dependence (you can’t uniformly improve it), but it is not UMP. The closed-testing framework allows other intersection-test choices (Fisher combination, Simes, …) that can dominate Holm on specific alternative configurations while still controlling FWER in the strong sense. Procedure choice depends on the expected dependence and signal structure. See LEH2005 §9.1 for the admissibility theorems.

For a multiple-testing procedure with rejection set $\mathcal{R}$ of cardinality $R = |\mathcal{R}|$ and false-rejection count $V$, the false discovery proportion is

$\mathrm{FDP} \;=\; \begin{cases} V/R & \text{if } R \ge 1, \\ 0 & \text{if } R = 0. \end{cases}$

The false discovery rate is its expectation:

$\mathrm{FDR} \;=\; \mathbb{E}_{H_0}[\mathrm{FDP}].$

A procedure controls FDR at level $\alpha$ if $\mathrm{FDR} \le \alpha$ for every true-null configuration $H_0$.

FDP is a random variable (it depends on which tests happen to reject in this specific sample); FDR is its expectation over the data distribution. A procedure can have FDR $= 0.05$ while producing individual samples with FDP $= 0$ (zero false discoveries) or FDP $= 0.3$ (many false discoveries) — on average the false-discovery proportion is $5\%$. Bounding FDR is a weaker guarantee than bounding $\Pr(\mathrm{FDP} > \alpha)$, which is the “FDP-exceedance” control studied in more advanced literature (Lehmann–Romano 2005, Genovese–Wasserman 2006).

For any procedure, $\mathrm{FDR} \le \mathrm{FWER}$. Proof: $\mathrm{FDP} \le \mathbf{1}\{V \ge 1\}$ almost surely (whenever $V \ge 1$, $\mathrm{FDP} \le 1$; whenever $V = 0$, $\mathrm{FDP} = 0$). Taking expectations: $\mathrm{FDR} = \mathbb{E}[\mathrm{FDP}] \le \Pr(V \ge 1) = \mathrm{FWER}$. So any FWER-controlling procedure automatically controls FDR — but FDR-controlling procedures (like BH) can achieve much more power by not controlling FWER, letting the $\Pr(V \ge 1)$ float up while keeping the expected $V/R$ in check.

FDR is appropriate when the cost of a false discovery is roughly linear in the number of false discoveries — e.g., each false hit triggers a follow-up experiment that wastes resources, and the total cost is proportional to the count. FWER is appropriate when any false discovery is catastrophic — e.g., a drug-approval decision, where one false positive leads to a product recall regardless of how many true positives also emerged. Regulatory biostatistics leans FWER; genomics and high-dimensional discovery lean FDR. Most A/B-test platforms split the difference: FWER for final launch decisions, FDR for exploratory dashboards.

The comparator below runs multiTestingMonteCarlo for a selected procedure across a grid of signal strengths $\delta \in [0, 4]$ at $m = 100$, $\pi_0 = 0.8$. For BH, the FDR curve stays pinned near $\alpha \cdot \pi_0 = 0.04$ (below the $\alpha = 0.05$ line) at all $\delta$, while the FWER curve climbs toward 1 as $\delta$ increases — BH does not control FWER. For Bonferroni / Holm, FWER stays near $\alpha$ and FDR slopes down. The power curve in the right panel shows BH’s advantage: at moderate $\delta$, BH’s power is more than double Bonferroni’s.

Order the $p$-values ascending: $p_{(1)} \le \cdots \le p_{(m)}$. Let $k^*$ be the largest rank $k$ such that $p_{(k)} \le k \alpha / m$ (set $k^* = 0$ if no such $k$ exists). The Benjamini–Hochberg procedure rejects $H_{(1)}, \ldots, H_{(k^*)}$ — i.e., the $k^*$ hypotheses with the smallest $p$-values.

If the $p$-values are independent (or positively regression-dependent on the subset, PRDS), then

$\mathrm{FDR} \;\le\; \pi_0 \cdot \alpha \;\le\; \alpha.$

$p = (0.0005, 0.003, 0.011, 0.021, 0.043, 0.051, 0.087, 0.21, 0.33, 0.55)$, $\alpha = 0.05$, $m = 10$. The BH threshold at rank $k$ is $k \cdot 0.005$. Walk from $k = 10$ down: $p_{(10)} = 0.55 > 0.05$, $p_{(9)} = 0.33 > 0.045$, …, $p_{(4)} = 0.021 > 0.020$ (fail!), $p_{(3)} = 0.011 \le 0.015$ (pass — stop). $k^* = 3$; reject ranks 1, 2, 3. The critical point: $p_{(4)} = 0.021$ fails its threshold by a hair ($0.021$ vs $0.020$); had it been below, BH would have rejected 4 or more. The regression test T1 in testing.test.ts pins this behavior.

Using multiTestingMonteCarlo('bh', m = 200, π₀ = 0.8, δ = 3.0, α = 0.05, 3000, seed = 123), empirical FDR lands in $[0.030, 0.050]$. The theoretical bound is $\pi_0 \cdot \alpha = 0.040$. Empirical values below $0.040$ are normal — BH is conservative when the signal is strong, because strong alternatives drive $R$ up while $V$ stays fixed. The test harness T3 in testing.test.ts locks this range. The same simulation shows empirical FWER well above $0.50$: BH deliberately does not control FWER, and large samples with moderate signal produce many false rejections on their way to producing even more true rejections.

The visualizer below runs the step-up walk one rank at a time. Load the canonical T1 preset, press Step. The active rank descends from $k = 10$, lighting up red (fail) at ranks 10, 9, …, 4, then green (pass) at $k = 3$. The rejection region shades in, containing ranks 1 through 3.

The BH threshold function $k \mapsto k\alpha/m$ has two endpoints: $\alpha/m$ at $k = 1$ (exactly Bonferroni) and $\alpha$ at $k = m$ (no adjustment). The linear interpolation between them is what makes the independence cancellation in Proof 5 Step 3 work: the per-rank threshold increment $\alpha/m$ is also the per-rank denominator increment $1/R$, and the two cancel. Any nonlinear threshold function breaks this. Storey’s adaptive variant (§20.8) rescales to $k\alpha/(m \cdot \hat{\pi}_0)$ — still linear, but with $\hat{\pi}_0$ shrinking the effective $m$.

The BEN2001 Lemma 2.1 independence argument extends to the PRDS (positive regression dependence on the subset) regime — where the conditional distribution of the non-null $p$-values given the null $p$-values is non-decreasing in the null $p$-values. This covers most “positively correlated” scenarios in practice: LD-correlated SNPs in a GWAS, correlated coefficients in a regression with positively-correlated predictors. Under PRDS, BH still controls FDR at $\pi_0 \alpha \le \alpha$. Under negative dependence, BH can fail; §20.8 Thm 6 (BY) handles this case at an $H_m = \sum_{k=1}^m 1/k$ power cost.

Unlike Holm, BH is not obtained from closed testing with Bonferroni intersection tests; those would produce Holm. BH corresponds to a different closed-testing construction (Simes intersection tests) that is valid for FWER control only under independence, and the resulting closed procedure controls FDR as well. The cleaner modern view: BH is its own step-up procedure derived directly from the FDR-control calculation of Proof 5, independent of the closed-testing scaffold. Closed testing explains why FWER procedures are nested (Holm dominates Bonferroni); FDR procedures live in a separate algebraic universe.

Let $c_m = \sum_{k=1}^m 1/k$ be the harmonic number. Apply BH with thresholds $k \alpha / (m \cdot c_m)$ at rank $k$. Under any joint dependence structure of the $p$-values (positive, negative, or arbitrary), this BY procedure controls FDR at $\pi_0 \alpha \le \alpha$.

Let $\hat{\pi}_0(\lambda) = |\{i : p_i > \lambda\}| / (m (1 - \lambda))$ at a tuning parameter $\lambda \in (0, 1)$ (canonically $\lambda = 0.5$). The Storey q-value at rank $k$ is

$\hat{q}_{(k)} \;=\; \min_{j \ge k} \frac{\hat{\pi}_0 \cdot m \cdot p_{(j)}}{j}.$

The Storey procedure rejects hypotheses with $\hat{q}_{(k)} \le \alpha$. Asymptotically (as $m \to \infty$), Storey controls FDR at exactly $\alpha$, recovering the $\pi_0$-slack that BH leaves on the table.

Simulated $m = 20{,}000$ hypotheses, 200 true alternatives at effect size $\delta = 3.5$, $\alpha = 0.05$. Bonferroni’s threshold is $0.05 / 20{,}000 = 2.5 \times 10^{-6}$, so alternatives need $z$-scores above $\approx 4.7$ — the tails where only the strongest signals live. BH rejects all $p$-values below $k \cdot 0.05 / 20{,}000$ at the realized $k^*$, which for moderate signal lands in the hundreds. The genomics figure below (which will render once the supplementary cell in 20_multiple_testing.ipynb is executed) shows BH recovering roughly $3\times$ more true positives than Bonferroni at the same (empirical) FWER cost.

John Ioannidis’s 2005 paper asked: given a statistically significant finding ($p < 0.05$), what is the probability the corresponding alternative is true? The answer depends on the prior probability of the alternative. For a screening study with $\pi_0 = 0.99$ and moderate power $0.5$: expected true positives $= 0.01 \cdot m \cdot 0.5 = 0.005 m$; expected false positives $= 0.99 \cdot m \cdot 0.05 = 0.0495 m$. The positive predictive value $\mathrm{PPV} = 0.005 / (0.005 + 0.0495) \approx 0.092$. A "$p < 0.05$" finding in this regime is a false positive $\approx 91\%$ of the time. Controlling FDR at $\alpha$ aligns the reported significance claim with $\mathrm{PPV} \ge 1 - \alpha$ for the rejected set, which is the practical guarantee that FWER alone does not deliver.

Even researchers who never consciously run multiple tests inflate $\alpha$ via unstated analytical flexibility: outcome choice, covariate selection, data cleaning rules, subgroup definitions, alternative models. Each branching analytic decision, if it depends on the data, effectively increases $m$. GEL2014 estimate that a “single” analysis often corresponds to $m$ between $10$ and $100$ latent decisions, so the naive $p < 0.05$ threshold corresponds to an unadjusted FWER of roughly $0.4$ to $0.99$ under the global null. Pre-registration is the technical remedy; multiple-testing correction is the retroactive one.

A/B testing with fixed pre-specified metrics on a common user population: positive regression dependence (PRDS) — BH is appropriate. GWAS within a single chromosome: positive dependence via LD — BH is appropriate. Testing a single hypothesis across many unrelated datasets (pooled meta-analysis with unknown dependence): BY is the safer choice. Post-hoc subgroup analysis with unstated inclusion rules: no procedure saves you — this is the Gelman–Loken regime, and the fix is pre-registration, not a different threshold.

The canonical $\lambda = 0.5$ is a sensible default but not optimal for every distribution. STO2002 proposes a bootstrap-based tuning: compute $\hat{\pi}_0(\lambda)$ over a grid and pick the $\lambda^*$ minimizing estimated MSE. For large $m$ the choice barely matters; for $m < 500$, the bootstrap can stabilize estimates. The Bioconductor qvalue package runs the bootstrap by default.

If $\pi_0$ is genuinely close to 1 (GWAS: $\pi_0 \ge 0.99$), Storey and BH are numerically indistinguishable. The Storey adjustment matters most for moderately dense signals ($0.5 \le \pi_0 \le 0.9$) common in transcriptomics, proteomics, and moderate-dimensional regression coefficient testing. Bioinformatics pipelines default to Storey (via qvalue); statisticians default to plain BH unless they have explicit evidence $\pi_0 \ll 1$.

The 2005–2016 replication-crisis literature (Ioannidis, Gelman–Loken, Open Science Collaboration 2015) pointed out that naive $p < 0.05$ in the face of unacknowledged multiplicity produces irreproducible findings at scale. The technical remedy is BH or Storey with appropriately calibrated $\alpha$; the sociological remedy is pre-registration, adversarial collaboration, and transparency about analytical forking. §17.12 Rem 16 and §17.12 Rem 19-bullet defer to this discussion; both are now fulfilled.

Let $C_i^{(\alpha')}$ be a $(1 - \alpha')$ CI for $\theta_i$ obtained by inverting a level-$\alpha'$ test (Topic 19 Thm 1).

(a) Bonferroni simultaneous CIs. Set $\alpha' = \alpha/m$. Then

$\Pr\bigl(\theta_i \in C_i^{(\alpha/m)} \text{ for every } i\bigr) \;\ge\; 1 - \alpha.$

(b) Šidák simultaneous CIs. Assume $(C_1, \ldots, C_m)$ are based on independent test statistics. Set $\alpha' = 1 - (1 - \alpha)^{1/m}$. Then

$\Pr\bigl(\theta_i \in C_i^{(\alpha')} \text{ for every } i\bigr) \;\ge\; 1 - \alpha,$

with equality when every $\theta_i$ equals the value against which the test was constructed.

Both follow immediately from Topic 19 Thm 1 combined with the union bound (Bonferroni) or the independence product (Šidák). Stated without separate proof.

Ten independent sample means $\bar{X}_i$ with SE $= 1/\sqrt{n}$, $n = 25$, $\alpha = 0.05$. The per-interval critical value is $z_{1 - \alpha/(2m)} = z_{0.9975} = 2.807$, so each Bonferroni CI is $\bar{X}_i \pm 2.807 / 5 = \bar{X}_i \pm 0.561$. The unadjusted critical value is $z_{0.975} = 1.96$, giving width $\pm 0.392$. The Bonferroni width is $2.807/1.96 \approx 1.43\times$ the unadjusted. Over 500 simulations with all true means $= 0$: empirical joint coverage for unadjusted is $\approx 0.60$; for Bonferroni $\approx 0.95$.

Same setup as Ex 13. Šidák’s per-interval $\alpha' = 1 - 0.95^{0.1} = 0.00512$, so the critical value is $z_{1 - 0.00512/2} = z_{0.99744} = 2.800$ — only $0.007$ smaller than Bonferroni’s. CI width is $\approx 0.560$ (vs Bonferroni’s $0.561$), a gain of $0.2\%$. The Šidák construction achieves exact joint coverage $0.95$ under independence, not just $\ge 0.95$, so the marginal gain is worth knowing exists, though the practical difference is negligible.

We have covered the general case where no additional structure relates the $m$ parameters. Two prominent special cases give tighter bounds:

• Scheffé’s method for linear contrasts in ANOVA: if the parameters of interest are all linear combinations $\sum c_i \mu_i$ of group means in a one-way ANOVA, Scheffé’s simultaneous CIs use the $F$-distribution critical value, which is tighter than Bonferroni for large $m$. See formalML: or CAS2002 §11.2.4.
• Tukey’s HSD for all pairwise mean differences: exact simultaneous CIs via the studentized range distribution, tighter than Bonferroni when all $\binom{m}{2}$ pairwise contrasts are of interest.
• Working–Hotelling for regression coefficient trajectories: tighter simultaneous bands via the $F$-distribution over a contiguous parameter space.

These are Topic 20’s immediate neighbors and are covered in depth in Track 6’s regression material — §21.8 Rem 16 delivers both Bonferroni-adjusted CIs and Working–Hotelling bands with full F-distribution-based derivations.

An FDR procedure like BH controls the expected false-discovery proportion but does not produce a simultaneous CI — there is no ”$(1 - \alpha)$ joint coverage” interpretation of BH rejections. When the practitioner reports “10 features rejected at BH $\alpha = 0.05$,” the claim is that $\le 5\%$ of those 10 are expected to be false rejections, not that the 10 parameters are jointly confined to specific intervals. For joint confidence statements on multiple parameters, Bonferroni / Šidák CIs are the correct tool; for discovery-rate-controlled flag lists, BH is correct.

Topic 19 explicitly deferred simultaneous CIs to Topic 20. The construction above — Bonferroni-adjusted inversion of FWER-controlled tests — is the formal treatment Topic 19 promised. Every claim about test–CI duality in Topic 19 carries over unchanged; the adjustment is to the level (not the construction).

The setup of §20.3 assumes the $m$ hypotheses and stopping rule are fixed in advance. Always-valid confidence sequences (Howard, Ramdas, McAuliffe, Sekhon 2021) and e-processes (Vovk 2019, Grünwald, de Heide, Koolen 2024) dispense with this: the inferential guarantee holds at every stopping time, including data-dependent ones. The mathematical machinery is martingale-based and lives in Track 8 / formalml.com. Every A/B-testing platform that allows “peeking” without Type I inflation uses these tools.

What if hypotheses arrive sequentially and we must decide to reject immediately without seeing future $p$-values? Offline BH no longer applies. The Online-BH procedure (Javanmard–Montanari 2018) and SAFFRON / LORD / Adaptive-LORD family (Ramdas et al. 2018) control FDR in this streaming regime by spending an $\alpha$-wealth budget across time. formalML: lives in Track 8.

Barber–Candès (2015) construct synthetic “knockoff” features $\tilde{X}_1, \ldots, \tilde{X}_m$ exchangeable with the real features $X_1, \ldots, X_m$ under the null, then apply a BH-like step-up on feature-statistic signs. The FDR control at $\alpha$ is the §20.7 Thm 5 result extended to a test-statistic-free setting — knockoffs work without requiring the per-feature $p$-values that BH demands. This is the workhorse of modern high-dimensional variable selection.

Westfall–Young (1993) permutation-based multiplicity adjustment computes FWER directly from the joint distribution of the $m$ test statistics via resampling, avoiding any dependence assumption. It is the gold standard for FWER control when strict validity is required (e.g., FDA-regulated settings) and computation is affordable. formalML: has the full treatment.

In a fully Bayesian framework, multiplicity is handled by the prior: placing a sparse prior on the $m$-dimensional parameter vector (e.g., spike-and-slab, horseshoe) automatically shrinks most estimates toward zero, achieving a form of implicit multiple-testing correction. The posterior probability that the $i$-th alternative is true is the Bayesian counterpart to the frequentist q-value. Topic 25 §25.10 names this pointer; the full local-FDR framework is developed in Topic 27 §27.9.

We stated Thm 0 (closed testing) without proof in §20.2. The full Marcus–Peritz–Gabriel argument is 15–20 lines and lives in formalML: . The key step is the observation — already stated in §20.2 — that a false rejection requires the true-null intersection’s level-$\alpha$ test to reject, which happens with probability $\le \alpha$.

Hommel (1988) and Rom (1990) are two additional step-up procedures that fall between Hochberg and the full Simes-based closed procedure in the FWER-control hierarchy. Both require independence; both are slightly more powerful than Hochberg on specific $p$-value configurations and slightly less powerful on others. They are named here for completeness; practitioners overwhelmingly default to Hochberg or BH.