The Bootstrap

Let $\theta(F)$ be a functional of the unknown distribution $F$ — for example, $\theta(F) = \mathrm{Var}_F(T_n)$, the variance of a statistic under repeated sampling from $F$. The plug-in estimator of $\theta$ is $\theta(F_n)$: the same functional evaluated at the empirical distribution. When $\theta$ is sufficiently smooth as a functional of its CDF argument, $\theta(F_n) \to \theta(F)$ — a functional Glivenko–Cantelli. The bootstrap is the plug-in principle applied to the sampling-distribution functional itself.

Suppose $T_n = \bar X_n$ is the sample mean. Classical theory tells us $\mathrm{Var}_F(\bar X_n) = \sigma^2 / n$, where $\sigma^2 = \mathrm{Var}_F(X_1)$. The plug-in answer is $\mathrm{Var}_{F_n}(\bar X_n) = \hat\sigma^2_n / n$, where $\hat\sigma^2_n = n^{-1}\sum_i (X_i - \bar X_n)^2$ is the sample variance (non-Bessel-corrected — it’s the variance of $F_n$, which places mass $1/n$ at each sample point). Both estimators are consistent; the plug-in one matches the classical one. Now replace $\bar X_n$ by the sample median. Classical theory says $\mathrm{Var}_F(\mathrm{median}) \approx (4 n f(\xi_{0.5})^2)^{-1}$, requiring the population density $f$ at the median — an object Topic 29 §29.6 struggled with. The plug-in answer is $\mathrm{Var}_{F_n}(\mathrm{median})$, which the bootstrap computes by Monte Carlo resampling. No density estimate needed; the resampled medians handle everything.

The bootstrap introduces two distinct approximations: (i) using $F_n$ instead of $F$ — this is the asymptotic error that vanishes as $n \to \infty$, and Theorem 3 in §31.3 controls it; and (ii) using a finite number $B$ of Monte Carlo resamples instead of the exact-plug-in answer $\mathrm{Var}_{F_n}(\cdot)$ — this is the Monte-Carlo error that vanishes as $B \to \infty$, independently of $n$. In practice we fix $B$ large (say $B = 2000$) and treat the MC error as negligible; the asymptotic error is the object of theoretical study.

Track 4 (Topics 17–20) built hypothesis tests and CIs on parametric models — assume $X_i \sim F_\theta$ for some family $\{F_\theta : \theta \in \Theta\}$, derive the sampling distribution from the model, use likelihood ratios or pivots for inference. The bootstrap drops the family assumption entirely. In exchange, it gives up the efficiency and optimality guarantees that come with correctly-specified parametric models and trades them for distribution-free validity under mild moment conditions. When you don’t know the model, or when you know the standard family is wrong (fat tails, mixtures, skew), bootstrap is the non-negotiable answer.

Given an iid sample $X_1, \dots, X_n \sim F$, the nonparametric bootstrap draws a resample $X^\ast_1, \dots, X^\ast_n$ iid from the empirical distribution $F_n$:

Equivalently, each $X^\ast_i$ selects an index $J_i \sim \mathrm{Uniform}\{1, \dots, n\}$ independently and sets $X^\ast_i = X_{J_i}$. We draw $B$ independent resamples $X^{\ast(1)}, \dots, X^{\ast(B)}$, compute the statistic $T^{\ast(b)} = T(X^{\ast(b)})$ on each, and take the empirical distribution of $\{T^{\ast(b)}\}_{b=1}^B$ as the bootstrap estimate of the sampling distribution of $T_n$. Write $P^\ast, E^\ast$ for probability and expectation conditional on the observed data $X_1, \dots, X_n$ — the bootstrap world.

Under finite-second-moment regularity, the bootstrap estimator of $\mathrm{Var}(T_n)$,

satisfies $\widehat{\mathrm{Var}}^\ast(T_n) \to \mathrm{Var}(T_n)$ in probability as $n, B \to \infty$. The rate is controlled by $n^{-1/2}$ for the asymptotic component and $B^{-1/2}$ for the Monte-Carlo component.

Under the same regularity, the bootstrap quantile $\hat q^\ast_p = \inf\{t : \hat F^\ast_B(t) \ge p\}$ of the bootstrap empirical CDF $\hat F^\ast_B(t) = B^{-1}\sum_b \mathbf{1}\{T^{\ast(b)} \le t\}$ satisfies $\hat q^\ast_p \to q_p$ in probability, where $q_p$ is the $p$-quantile of the true sampling distribution of $T_n$.

On a Normal$(0, 1)$ sample of size $n = 100$ and $B = 10{,}000$, the bootstrap SE of the sample median is approximately $0.125$. The asymptotic formula $(4 n f(\xi_{0.5})^2)^{-1/2} \approx (4 \cdot 100 \cdot \varphi(0)^2)^{-1/2} = (4 \cdot 100 \cdot (2\pi)^{-1})^{-1/2} \approx 0.125$ matches to two digits. The bootstrap recovered the asymptotic answer without requiring $f(0)$ — it did the density estimation implicitly through resampling.

Two samples $(X_i, Y_i)_{i=1}^n$ iid from Exponential$(1) \times$ Exponential$(1)$, $n = 50$. The statistic $T_n = \bar X_n / \bar Y_n$ has no closed-form sampling distribution — it’s a ratio of independent Gammas divided by themselves, near-Cauchy in the tails. Classical delta-method intervals rely on a Taylor expansion around $E[X]/E[Y] = 1$ that becomes unstable for small $\bar Y_n$. Bootstrap: generate $B = 2000$ resamples, compute $T^{\ast(b)}$ on each, use the empirical quantiles for a CI. Topic 19’s Wald CI gives $(0.74, 1.38)$; the bootstrap percentile CI (coming in §31.4) gives $(0.79, 1.35)$. Both close; the bootstrap’s advantage is that it doesn’t depend on the delta-method expansion.

Cross-validation estimates out-of-sample risk by holding out folds, but the CV estimate itself has variance that depends on how the folds partition the data. Classical CV-variance formulas exist only for specific setups (leave-one-out on linear regression, for example). Bootstrapping CV is the general answer: resample the training data, run CV on each bootstrap sample, and use the empirical variance of the CV estimates as the CV variance. This is the bootstrap’s most common ML application — it shows up whenever someone reports a CI on a cross-validation score.

Pick a single $n$ and let $B \to \infty$: the bootstrap’s answer converges to $\mathrm{Var}_{F_n}(T_n)$ — the plug-in exact answer, which still differs from $\mathrm{Var}_F(T_n)$ by the $O(n^{-1/2})$ asymptotic gap. No amount of MC refinement can close that gap; it’s a property of using $F_n$ instead of $F$. The practical consequence: $B$ should be large enough to make MC error negligible relative to asymptotic error, but beyond that, increasing $B$ buys nothing. Topics 29 §29.5’s DKW band gives a coarse lower bound on the asymptotic error that can guide $B$-selection.

Every ML practitioner who has stared at a cross-validation score and wondered “how much should I trust this number?” is asking a bootstrap question. The CV score is a statistic of the training data; its sampling distribution under repeated training-set draws is exactly what bootstrap-CV estimates. The bootstrap gives a CI on the CV estimate without any parametric model of how risk depends on training-set composition — a distribution-free uncertainty quantification tailor-made for the ML use case.

Let $G_n, G$ be CDFs with $G$ continuous. If $G_n(x) \to G(x)$ pointwise for every $x$, then $\sup_x |G_n(x) - G(x)| \to 0$.

Let $X_1, \dots, X_n$ be iid with CDF $F$ satisfying $E[X_1^2] < \infty$. Write $\mu = E[X_1]$, $\sigma^2 = \mathrm{Var}(X_1) > 0$. Let $\bar X_n$ be the sample mean and $\bar X^\ast_n$ the bootstrap-sample mean — conditional on the data, this is the mean of $n$ iid draws from $F_n$. Define

Then $\sup_x |H^\ast_n(x) - H_n(x)| \to 0$ almost surely as $n \to \infty$.

When $X_i \sim \mathcal{N}(0, 1)$ and $T_n = \bar X_n$, the sampling distribution $H_n$ is analytic: $\sqrt{n}\bar X_n \sim \mathcal{N}(0, 1)$ exactly. So the “true” reference curve in the featured component’s first panel is the standard Normal density at sample size $n$, no Monte Carlo needed. The KS-distance panel shows $D_{KS}(H^\ast_n, \Phi) \to 0$ at rate $n^{-1/2}$, exactly the $1/\sqrt{n}$ envelope one would expect from the CLT remainder.

The three-step structure of the proof — (i) write the bootstrap statistic as an empirical average of iid-conditional-on-data terms, (ii) apply a triangular-array CLT to the linear part, (iii) upgrade pointwise convergence to uniform via Polya — is exactly the same template as Topic 29’s Bahadur representation of the sample quantile and Topic 30’s AMISE derivation for KDE. Every major Track 8 result reduces to this linearization pattern. Topic 29 §29.6 Rem 13 called this out as the unifying thread; Theorem 3’s proof makes it literally visible. Topic 32’s empirical-process generalization lifts the same structure one level: the “empirical average” becomes a stochastic integral against a sample-path-continuous limit process, the triangular-array CLT becomes Donsker’s theorem, and Polya’s upgrade becomes the uniform continuity of the limit Gaussian process.

The natural ML descendant of Theorem 3: treat a model’s predictions as a statistic, bag multiple bootstrap replicates of the training set, refit on each, and use the distribution of test-time predictions as a proxy for posterior uncertainty. This is the theoretical foundation for bagging, for many uncertainty-quantification methods in deep learning, and for the non-parametric half of conformal prediction. Theorem 3 guarantees that as $n$ grows, the bagged-prediction distribution matches the sampling distribution of the model’s prediction under re-draw of the training set — which is what honest uncertainty quantification actually asks for.

The percentile CI at level $1 - \alpha$ is the pair of $\alpha / 2$ and $1 - \alpha/2$ quantiles of the bootstrap replicates:

This is the intuitive construction — the one that inverts the bootstrap empirical CDF without further adjustment. It’s exact under symmetry about $\theta$ but under-covers asymmetric sampling distributions (§31.5 makes this precise).

The basic CI reflects the percentile endpoints around the observed $T_n$:

The motivation: treat $T^\ast_n - T_n$ as a pivot whose distribution mimics that of $T_n - \theta$; invert the pivot. Under symmetry, basic and percentile coincide; under skewness, they lean in opposite directions. Hall 1992 §3.3 explains why basic is often the better default.

The bias-corrected and accelerated CI uses two plug-in constants to adjust the percentile endpoints. Let $\hat z_0 = \Phi^{-1}\bigl(B^{-1}\sum_b \mathbf{1}\{T^\ast_{(b)} < T_n\}\bigr)$ — the bias correction. Let $\hat a$ be the jackknife acceleration,

where $T_{(i)}$ is the leave-one-out estimate and $T_{(\cdot)} = n^{-1}\sum_i T_{(i)}$. The adjusted quantile level for $p \in \{\alpha/2,\ 1-\alpha/2\}$ is

The BCa CI is $[\hat q^\ast_{\tilde{p}_\mathrm{lo}},\ \hat q^\ast_{\tilde{p}_\mathrm{hi}}]$, with the adjusted quantile levels substituted for $\alpha/2$ and $1-\alpha/2$.

The studentized CI builds a pivot from the studentized statistic. For each outer bootstrap replicate, draw an inner bootstrap to estimate $\mathrm{SE}^\ast_b$, then form

Let $\hat t^\ast_p$ be the $p$-quantile of $\{T^\ast_b\}_{b=1}^B$. The CI is

where $\widehat{\mathrm{SE}}(T_n)$ is the analytic standard-error estimate on the observed sample. The inversion mimics Topic 19’s $t$-CI construction — hence the name “bootstrap-$t$.”

Under the regularity of Theorem 3, each of the four CI constructions above achieves asymptotic coverage $1 - \alpha$ as $n \to \infty$: for each CI type $\mathrm{CI}^{(\bullet)}_{1-\alpha}$,

$X_1, \dots, X_{100} \sim \mathrm{Exp}(1)$; target $\theta = \log 2$, the population median. Four bootstrap CIs at $\alpha = 0.05$ with $B = 2000$: percentile $[0.52, 0.88]$, basic $[0.54, 0.90]$, BCa $[0.54, 0.91]$, studentized $[0.53, 0.92]$. All four cover $\log 2 \approx 0.693$; length differences under 5 %. At $n = 30$ the gap between percentile and BCa widens — percentile falls below nominal, BCa holds — which is the small-sample signature of the skewness correction.

Same ratio-of-means statistic as Example 3, but at $n = 30$. Wald CI $(0.43, 1.61)$ with 92 % coverage over 10 000 simulated datasets (under-coverage from delta-method bias). Bootstrap percentile $(0.51, 1.48)$, 94 %. BCa $(0.56, 1.52)$, 95 %. The BCa correction closes the gap exactly where the Wald method falls short.

Classical Fisher $z$-transform CI for the sample correlation works only under bivariate Normal — fails wildly outside that family. Bootstrap percentile on $\hat \rho_n$ directly (no transform, no Normal assumption) tracks nominal coverage for almost any joint distribution. Bootstrap won’t rescue us from tiny $n$ or from near-perfect-correlation degeneracy, but it covers the “my data isn’t bivariate Normal” case that kills the $z$-transform.

The bootstrap’s workhorse role in ML. Given a trained model $\hat m$ and test point $x_0$, the prediction $\hat y_0 = \hat m(x_0)$ has two sources of uncertainty: model-estimation uncertainty (how would $\hat m$ differ under a re-drawn training set?) and irreducible noise (how would $y_0$ differ given $\hat m$ fixed?). Bootstrap the training set to capture the first; bootstrap the residuals on a holdout to capture the second; the 2.5 % and 97.5 % percentiles of $\hat m^\ast(x_0) + \epsilon^\ast$ are a valid 95 % prediction interval under mild regularity. Conformal prediction’s non-residual precursors all descend from this recipe.

When the statistic is roughly symmetric and the sample size is moderate, percentile is fine — it’s cheapest and the interpretation is the most direct. When the statistic is skewed or moderately small-$n$, BCa is the right default. Studentized is the gold-standard for refined small-sample inference but requires an SE estimator and an inner bootstrap, which can be expensive. Basic is a theoretical bridge more than a practical choice. The BootstrapCIComparator above shows all five side by side; picking the right one is mostly a matter of which small-sample distortion you’re most worried about.

Topic 19 §19.7 established the CI↔test duality: inverting a family of level-$\alpha$ tests produces a $1-\alpha$ CI, and a $1-\alpha$ CI corresponds to a test that rejects $H_0: \theta = \theta_0$ if $\theta_0 \notin \mathrm{CI}$. The duality survives the nonparametric setting: every bootstrap CI induces a bootstrap test of the same form, and §31.6 develops the test-first framing explicitly. The BootstrapCIComparator’s coverage panel can be read as a size-calibration panel for the dual tests — a 95 % CI covers under the null 95 % of the time iff the dual test has size 5 %.

Under smooth-functional regularity (two-term Edgeworth expansion valid for $T_n$ and its studentized version), the percentile and basic bootstrap CIs have coverage error $O(n^{-1/2})$, while the BCa and studentized CIs achieve coverage error $O(n^{-1})$.

To test $H_0: \theta(F) = \theta_0$ against $H_1: \theta(F) \neq \theta_0$, construct a transformation $F^{(0)}_n$ of the empirical distribution under which $\theta(F^{(0)}_n) = \theta_0$ exactly — typically by shifting, re-centring, or re-scaling $F_n$. Draw $B$ resamples from $F^{(0)}_n$, compute the test statistic $T^{\ast(b)}$ on each, and reject $H_0$ at level $\alpha$ if the observed $T_n$ exceeds the $1 - \alpha$ quantile of the null-resample distribution. Alternative-sided tests adapt the rejection region accordingly; the $p$-value is $B^{-1}\sum_b \mathbf{1}\{T^{\ast(b)} \ge T_n\}$ plus continuity corrections.

Under the regularity of Theorem 3 plus uniform-in-$\theta_0$ Edgeworth validity, the bootstrap test defined above has size $\alpha + O(n^{-1/2})$ (first-order accurate) or $\alpha + O(n^{-1})$ (second-order, when paired with studentized or BCa-adjusted test statistics). Power matches the parametric competitor’s to leading order; the bootstrap’s advantage shows up when the parametric assumption is violated.

Two-sample test of $H_0: \mu_X = \mu_Y$ against $H_1: \mu_X \neq \mu_Y$. Under the null, the two samples are exchangeable with common mean $\hat\mu = (\bar X_n + \bar Y_m) / 2$ (equal sample sizes; the $\mathrm{mean}$-weighted pooled version for unequal $n, m$). Centre both samples to this common mean: $\tilde X_i = X_i - \bar X_n + \hat\mu$, $\tilde Y_j = Y_j - \bar Y_m + \hat\mu$. Draw bootstrap resamples from $\tilde X$ and $\tilde Y$ separately; compute the difference of resample means; compare to the observed $\bar X_n - \bar Y_m$. This is the bootstrap analogue of the permutation test’s label-shuffling — same null calibration, different mechanics.

Conversion rates $X \in \{0, 1\}$ with $n_A = n_B = 5000$, observed rates $0.043$ vs. $0.051$. Parametric $z$-test $p$-value: $0.041$. Bootstrap-$t$ null-resample $p$-value: $0.038$ ($B = 10000$). The two agree to within MC error because $n$ is large and the Normal approximation is excellent. Now imagine $X$ is a heavy-tailed revenue-per-user metric — the $z$-test’s Normal approximation is wrong and the parametric $p$-value drifts; the bootstrap-$t$ tracks the truth because it doesn’t assume Normality.

Permutation tests are the classical distribution-free alternative. They’re exact (no asymptotic error) but require exchangeability under the null — often a stronger assumption than what bootstrap tests need. Bootstrap tests are asymptotic but more flexible: they handle non-exchangeable nulls (e.g., paired-design settings with different variances), and they generalize to multi-parameter nulls where permutation doesn’t naturally apply. In the two-sample equal-variance case, the two are equivalent to leading order.

Revenue-per-user, time-on-site, and other heavy-tailed ML metrics routinely violate the Normal-tail assumption that the standard $z$- or $t$-test relies on. Bootstrap tests are the production-grade answer: they give valid $p$-values under the actual metric distribution without requiring the analyst to specify what that distribution is. Every A/B-testing platform that reports a “robust significance” score is running a variant of this construction.

Fit a parametric family $\{F_\theta : \theta \in \Theta\}$ to the observed sample via MLE or another estimator, obtaining $\hat\theta_n$. Draw bootstrap resamples $X^\ast_1, \dots, X^\ast_n$ iid from $F_{\hat\theta_n}$ — not from $F_n$. Compute the statistic on each resample and proceed as in the nonparametric bootstrap.

Under the regularity of Topic 14 (parametric MLE consistency) plus Topic 11’s Edgeworth validity, the parametric bootstrap distribution $H^{\ast,\mathrm{par}}_n$ converges in Kolmogorov distance to the true sampling distribution $H_n$ at rate $O(n^{-1/2})$, matching the nonparametric bootstrap in first-order accuracy. When the parametric model is correctly specified, the parametric bootstrap achieves the same second-order accuracy as the studentized or BCa versions without needing the studentization step.

Data: $n = 100$ iid from Student-$t_3$ (heavy tails). Parametric bootstrap under a Normal model: fit $\hat\mu_n = \bar X_n$, $\hat\sigma^2_n = n^{-1}\sum_i (X_i - \bar X_n)^2$; resample from $\mathcal{N}(\hat\mu_n, \hat\sigma^2_n)$. The CI on $\mu$ is $(\bar X_n \pm 1.96 \hat\sigma_n / \sqrt{n})$, i.e. the Wald interval — and it under-covers because the Normal tails are wrong. Nonparametric bootstrap on the same data gives an interval that correctly includes the $t_3$ tail contribution; its coverage matches nominal within MC error. Misspecification matters for the parametric bootstrap; the nonparametric bootstrap is immune.

When a neural network or probabilistic model supplies an explicit likelihood $p(y \mid x; \theta)$, parametric bootstrap is the natural uncertainty-quantification tool: sample training data from the fitted model, refit, observe variability. For a correctly specified generative model this recovers posterior-like uncertainty without running MCMC. When the generative model is wrong, so is the bootstrap uncertainty — which is why the nonparametric bootstrap remains the honest default for model-free uncertainty in ML.

Residual bootstrap for regression is the canonical hybrid: parametric for the mean function (linear regression’s fitted line), nonparametric for the error distribution (resample residuals with replacement). Wild bootstrap extends this to heteroscedastic errors. These variants are deferred to §31.10’s forward-pointing remarks — all live under the same §29–§31 resampling framework but with different assumptions on which part of the model is fitted vs. empirical.

Let $\hat f_h$ be the Gaussian-kernel KDE from Topic 30 with bandwidth $h$ (typically $h$ = Silverman’s rule from Topic 30 §30.9). The smooth bootstrap resamples from $\hat f_h$ rather than from $F_n$:

The $J_i$ pick a sample point and the Gaussian $h Z_i$ adds a small kernel-shaped jitter. The resulting sample is iid from $\hat f_h$ by construction.

Under the regularity of Topic 30 Thm 6 (pointwise KDE consistency) plus Theorem 3’s moment condition, the smooth-bootstrap sampling distribution converges to the true sampling distribution in Kolmogorov distance, at the same $O(n^{-1/2})$ rate as the nonparametric bootstrap. For density-functional statistics (median, quantiles, integrals of $f$), smooth bootstrap strictly dominates: its MC variance is smaller at every $B$, and the $O(n^{-1/2})$ rate constant is smaller too.

Topic 30 §30.5 Rem 15 promised the smooth bootstrap as a simultaneous-envelope tool. Given $\hat f_h$, draw $B$ smooth-bootstrap samples; on each, compute $\hat f^\ast_h$; form the pointwise $2.5$ % and $97.5$ % envelopes across the $B$ replicates. Under mild regularity this recovers a valid $95$ % uniform confidence band for $f$ — the KDE analogue of the DKW band from Topic 29 §29.5. For the median fixture above, the envelope construction on a moderate-$n$ sample gives a smoothly varying band that the naïve bootstrap can’t produce because naïve bootstrap places mass only at sample points.

Smooth bootstrap introduces the bandwidth $h$ as a new free parameter. Topic 30’s data-driven selectors — Silverman (§30.9), Scott (§30.9), Sheather–Jones — all carry over. Silverman’s rule is the practical default; it’s nearly bandwidth-invariant in the $0.5 h_\mathrm{Silverman}$ to $2 h_\mathrm{Silverman}$ range (the right panel of SmoothBootstrapDemo shows this). Under-smoothing ($h \to 0$) recovers the naïve bootstrap; over-smoothing ($h$ large) over-regularizes and biases the smooth-bootstrap SE upward.

Nadaraya–Watson and local-polynomial regressors (forward to formalML) are kernel-weighted averages of $Y_i$. Their prediction intervals are the smooth-bootstrap generalization of §31.4’s prediction intervals: smooth-bootstrap the covariates $X_i$, refit the regressor on each bootstrap sample, and take the pointwise percentile envelopes of the predictions. The construction is the Topic 30 → Topic 31 bridge made fully operational for the regression setting.

The bootstrap bias estimator for $T_n$ as an estimator of $\theta$ is

The bias-corrected estimator is

In words: reflect the bootstrap mean around the observed statistic. The motivation is the Taylor linearization $T_n - \theta \approx (T_n - E[T_n]) + \mathrm{bias}(T_n)$; subtracting the bootstrap-estimated bias gives a reduced-bias estimator at the cost of slightly inflated variance.

Under the regularity of Theorem 3, the bias-corrected estimator satisfies $\mathrm{bias}(\tilde T_n) = O(n^{-2})$ — one order better than the original $T_n$‘s $O(n^{-1})$ bias. The variance inflation is $\mathrm{Var}(\tilde T_n) = \mathrm{Var}(T_n) (1 + O(n^{-1}))$, negligible at moderate $n$. MSE-wise, bias correction dominates when the leading-order bias is large relative to the SE — typically at small $n$ or for highly nonlinear statistics.

The sample variance $\hat\sigma^2_n = n^{-1}\sum_i (X_i - \bar X_n)^2$ has bias $-\sigma^2/n$ relative to the population variance $\sigma^2$. The Bessel-corrected $s^2_n = \hat\sigma^2_n \cdot n/(n-1)$ fixes this analytically — a well-known first-order correction. The bootstrap bias-correction recovers the same fix without the analytic insight: $\widehat{\mathrm{bias}}^\ast(\hat\sigma^2_n) = -\hat\sigma^2_n / n + O_P(n^{-3/2})$, matching the analytic bias. Subtracting gives $\tilde\sigma^2_n = \hat\sigma^2_n (1 + 1/n) + O_P(n^{-1/2})$, first-order equivalent to $s^2_n$. The bootstrap recovered Bessel’s correction by Monte Carlo.

Bias correction inflates variance. At small $n$ the bias dominates and correction helps; at large $n$ the bias is negligible and the variance inflation hurts. A rule of thumb: bias-correct when $|\widehat{\mathrm{bias}}^\ast(T_n)| > \widehat{\mathrm{SE}}^\ast(T_n) / 4$. Below that threshold, leave $T_n$ alone.

Cross-validation is known to underestimate test risk by the optimism: the gap between the training-set risk and the test-set risk. Bootstrap bias-correction of the CV risk is the standard debiasing technique — resample the training set, recompute CV on each resample, use the $\bar{\mathrm{CV}}^\ast - \mathrm{CV}_{\mathrm{obs}}$ difference as the optimism estimate. The .632+ bootstrap (Efron–Tibshirani 1997) refines this with a weighted combination of resubstitution and out-of-bag risk; it’s a direct descendant of the bias-correction construction above.

Künsch 1989 extended the bootstrap to stationary time-series data by resampling blocks of consecutive observations instead of individual points. The block length $\ell$ is a new tuning parameter — too small, autocorrelation structure is lost; too large, the number of blocks is too small for MC convergence. Variants: overlapping blocks (Künsch), moving blocks, circular blocks, stationary bootstrap (Politis–Romano). The entire family lives in the dependent-observations regime that Topic 31 excluded; formalML’s time-series inference chapters will treat it in full.

Politis–Romano 1994 showed that subsampling — resampling without replacement, at a size $m < n$ — achieves asymptotic validity under milder conditions than the bootstrap. Where bootstrap requires $E[X_1^2] < \infty$ for the sample mean, subsampling gets by with much weaker tail conditions. The trade-off is a smaller effective sample size $m$ and a tuning choice for $m$. Subsampling is the right tool for genuinely heavy-tailed distributions where the bootstrap can fail; Topic 31 assumes the moment conditions hold, so subsampling is a §31.10 footnote rather than a §31.x section.

Rubin 1981 replaced the bootstrap’s multinomial resample weights $\{1/n\}$ with Dirichlet-distributed random weights, producing a Bayesian-flavoured construction that behaves asymptotically like the nonparametric bootstrap but has a posterior-like interpretation. The full treatment belongs with Track 7’s Dirichlet-process machinery — the Bayesian bootstrap is the Dirichlet-process posterior for the special case of a vague Dirichlet prior.

Residual bootstrap is the regression-specific variant: fit the regression, compute residuals, resample residuals with replacement, and add back to the fitted mean to get new response values. Wild bootstrap generalizes to heteroscedastic errors by rescaling each residual by an independent mean-zero multiplier. Both are indispensable for valid inference on regression coefficients in misspecified settings — and both are deferred to formalML’s regression-inference chapters because they require the regression machinery Topic 31 deliberately didn’t build.

Topic 29 built the empirical CDF machinery. Topic 30 smoothed it into densities. Topic 31 — this topic — resampled from it. Topic 32 closes the track by embedding all three into the empirical-process framework: $F_n$ becomes a sample-path-continuous limit process (the Brownian bridge), $\hat f_h$ becomes a smoothed version of that process, and the bootstrap becomes a resampling operation inside a function space. Donsker’s theorem is the functional CLT that unifies the three; the bootstrap-consistency proof of §31.3 is a finite-dimensional shadow of Donsker. The empirical-process chapters of Topic 32 are the on-ramp to the uniform convergence and stochastic equicontinuity that underwrite modern high-dimensional statistics.