Model Selection & Information Criteria

For an estimator $\hat\theta$ trained on $\mathbf{y} = (Y_1, \ldots, Y_n)$ and a fresh observation $Y_{\text{new}} \sim f_0$ from the same data-generating distribution, the prediction risk under loss $L$ is

$R(\hat\theta) \;:=\; \mathbb{E}_{Y_{\text{new}}}\!\left[L(\hat\theta;\, Y_{\text{new}})\right].$

For log-likelihood loss $L(\hat\theta; y) = -\log f(y; \hat\theta)$, the risk is the negative expected log-predictive

$R(\hat\theta) \;=\; -\mathbb{E}_{Y_{\text{new}}}\!\left[\log f(Y_{\text{new}}; \hat\theta)\right] \;=\; -\mathrm{EL}(\hat\theta).$

The training empirical risk is $\hat R(\hat\theta) = (1/n)\sum_{i=1}^n L(\hat\theta; Y_i)$. The optimism gap measures how much in-sample loss underestimates out-of-sample loss in expectation:

$\mathrm{opt} \;:=\; \mathbb{E}\!\left[R(\hat\theta)\right] - \mathbb{E}\!\left[\hat R(\hat\theta)\right].$

For correctly-specified parametric models satisfying Topic 14’s regularity conditions, $\mathrm{opt} = k/n + o(1/n)$ where $k = \dim(\theta)$ — the optimism is exactly the parameter count divided by sample size, to leading order.

Generate $n=80$ observations from $y = \sin(2\pi x) + \mathcal{N}(0, 0.25^2)$ with $x \sim \mathcal{U}(0, 1)$. Fit polynomials of degree $d = 0, 1, \ldots, 12$ by ordinary least squares. The training $\mathrm{RSS}$ is monotone-decreasing in $d$ (every extra coefficient can only reduce the in-sample residual sum). The out-of-sample prediction risk, however, follows a U-shape: it falls as $d$ rises from $0$ to $\sim 6$ (model approximates the sine well), then climbs as $d$ exceeds $\sim 8$ (the polynomial wiggles to fit noise). Figure 1 shows the gap.

Model selection is a decision problem over the candidate set $\{\mathcal{M}_1, \ldots, \mathcal{M}_M\}$ — distinct from parameter estimation, which decides over $\Theta_k$ for a fixed $\mathcal{M}_k$. AIC, BIC, and cross-validation correspond to three different loss functions on the model space, each with its own asymptotic guarantee.

Three asymptotic targets organize the literature: selection consistency ($\mathbb{P}(\hat{\mathcal{M}} = \mathcal{M}_*) \to 1$ when the truth is in the candidate set — BIC’s target), prediction efficiency (minimax-rate optimal $L^2$ risk in the misspecified-nonparametric regime — AIC and CV’s target), and sparsity recovery (correctly identifying $\mathrm{supp}(\boldsymbol\beta_*)$ — Topic 23’s lasso target). §24.6 Thm 5 shows the first two are formally incompatible.

For an OLS fit of a candidate model with $k$ free parameters (intercept + slopes + $\sigma^2$) to $n$ observations under the Gaussian linear model $\mathbf{y} = \mathbf{X}\boldsymbol\beta + \mathcal{N}(\mathbf{0}, \sigma^2\mathbf{I})$, with residual sum of squares $\mathrm{RSS}_k$, Mallows’ $C_p$ is

$C_p \;:=\; \frac{\mathrm{RSS}_k}{\hat\sigma^2_{\text{ref}}} + 2k - n,$

where $\hat\sigma^2_{\text{ref}}$ is the MLE error variance from the largest candidate model (the conventional reference).

Under the Gaussian linear model with $\hat\sigma^2_{\text{ref}}$ known (or treated as known via the reference convention), the expected scaled in-sample residual sum is

$\mathbb{E}\!\left[\mathrm{RSS}_k / \sigma^2\right] \;=\; n - k.$

Adding $2k - n$ to both sides:

$\mathbb{E}[C_p] \;=\; (n - k) + 2k - n \;=\; k.$

A separate calculation (Mallows 1973 §3) gives the expected scaled prediction risk on a fresh test set of size $n$:

$\mathbb{E}\!\left[\mathrm{R}_{\text{scaled}}\right] \;=\; k + (\text{model bias term}).$

Under correct specification (model bias $= 0$), $\mathbb{E}[C_p] = \mathbb{E}[\mathrm{R}_{\text{scaled}}] = k$. Otherwise $C_p$ overestimates the parameter count by exactly the model bias — making $C_p$ minus its parameter count a calibrated estimator of model bias, the original use case Mallows 1973 §3 emphasized.

Under the Gaussian linear model, $\mathrm{AIC} = n\log(\mathrm{RSS}/n) + 2k$ (dropping additive constants), and a Taylor expansion of $\log(\mathrm{RSS}/n)$ around $\sigma^2_{\text{ref}}$ recovers $C_p$ up to higher-order terms. The two criteria rank candidates identically under Gaussian-homoscedastic errors; AIC is the strict generalization to other exponential families (§24.3 Thm 1).

The convention $\hat\sigma^2_{\text{ref}} = \mathrm{RSS}_{\text{full}}/(n - k_{\text{full}})$ — using the unbiased estimator from the largest candidate — has the cleanest theory: $\hat\sigma^2_{\text{ref}}$ is unbiased for $\sigma^2$ if the largest model is correctly specified, and $C_p$‘s argmin then targets the bias-variance-optimal submodel. T10.8 in regression.test.ts pins $\hat\sigma^2_{\text{ref}} = 0.0497586381$ for the canonical POLY_DGP at $n=80$, giving $\arg\min_d C_p = 6$.

On the canonical POLY_DGP ($n=80$, $\sigma=0.25$, seed $42$), with $\hat\sigma^2_{\text{ref}} = 0.0497586381$ from the $d = 12$ fit, the Mallows $C_p$ values for $d = 0, 1, \ldots, 12$ have argmin at $d = 6$ with $C_p = 21.4569$ (T10.8). The argmin coincides with AIC’s argmin (T10.3), illustrating Rem 3’s $C_p \equiv \mathrm{AIC}$ equivalence on the Gaussian linear model. Figure 2 plots the $C_p$ curve alongside the in-sample $\mathrm{RSS}/\hat\sigma^2_{\text{ref}}$ to make the $+2k - n$ correction visually concrete.

For two densities $f_0$ (truth) and $f(\cdot;\theta)$ (model), the Kullback–Leibler divergence is

$D(f_0 \,\|\, f_\theta) \;:=\; \mathbb{E}_{Y \sim f_0}\!\left[\log\frac{f_0(Y)}{f(Y;\theta)}\right] \;=\; \mathbb{E}_{f_0}[\log f_0(Y)] - \mathrm{EL}(\theta),$

where the expected log-likelihood under truth is

$\mathrm{EL}(\theta) \;:=\; \mathbb{E}_{Y \sim f_0}[\log f(Y;\theta)].$

Since $\mathbb{E}_{f_0}[\log f_0(Y)]$ does not depend on $\theta$, maximizing $\mathrm{EL}(\theta)$ is equivalent to minimizing $D(f_0 \,\|\, f_\theta)$.

Let $Y_1, \ldots, Y_n$ be iid from unknown density $f_0$, and let $\{f(\cdot;\theta) : \theta \in \Theta \subset \mathbb{R}^k\}$ be a parametric model satisfying Topic 14’s regularity conditions: smooth log-likelihood, positive-definite Fisher information, MLE asymptotic normality. Let $\hat\theta$ be the MLE, $\hat\ell = \ell(\hat\theta;\mathbf{y})$, and define $\mathrm{AIC} := -2\hat\ell + 2k$. Then under correct specification ($f_0 \in \{f(\cdot;\theta)\}$),

$\mathbb{E}\!\left[\mathrm{AIC}\right] \;=\; -2n\cdot\mathbb{E}\!\left[\mathrm{EL}(\hat\theta)\right] + o(1).$

That is, $\mathrm{AIC}$ is asymptotically unbiased for $-2n$ times the expected out-of-sample log-predictive evaluated at the MLE.

On POLY_DGP ($n=80$, $\sigma=0.25$, seed $42$), $\mathrm{AIC}(d)$ values for $d = 0, 1, \ldots, 12$ have argmin at $d = 6$ with $\mathrm{AIC} = 8.2633$ (T10.3). The $\mathrm{AIC}$ curve is U-shaped: $\mathrm{AIC}(d=0) = 179.0128$ (extreme underfit, T10.1); $\mathrm{AIC}(d=3) = 13.5077$ (undershooting; T10.2); $\mathrm{AIC}(d=6) = 8.2633$ (argmin, recovers the prediction-risk argmin from §24.1 Ex 1); $\mathrm{AIC}(d=12) = 14.9845$ (overfit, T10.4). Figure 3 overlays AIC and the bias-correction term $+2k$ to separate the empirical log-likelihood from the optimism penalty.

AICc corrects AIC’s small-sample bias when $n/k$ is not large (Hurvich & Tsai 1989; Sugiura 1978):

$\mathrm{AICc} \;=\; \mathrm{AIC} + \frac{2k(k+1)}{n - k - 1}.$

Exact for Gaussian-linear models; asymptotic correction otherwise. The penalty $2k(k+1)/(n-k-1) \to 0$ as $n \to \infty$, so AICc and AIC agree asymptotically. Burnham & Anderson 2002 §2.4 recommend AICc as the default whenever $n/k < 40$.

Takeuchi (1976) generalized AIC to misspecified parametric models:

$\mathrm{TIC} \;=\; -2\hat\ell + 2\operatorname{tr}(\hat{\mathcal{K}}^{-1}\hat{\mathcal{J}}),$

with $\hat{\mathcal{J}}$ the empirical score outer product and $\hat{\mathcal{K}}$ the observed Hessian of $-\ell$. Under correct specification $\hat{\mathcal{K}}^{-1}\hat{\mathcal{J}} \to \mathbf{I}_k$ and TIC $\to$ AIC. TIC sees little practical use because $\hat{\mathcal{J}}$ and $\hat{\mathcal{K}}$ are hard to estimate stably at moderate $n$; the broader information-geometric framing lives at formalml.

For a model $\mathcal{M}$ with parameter $\theta \in \Theta$, prior $\pi(\theta)$, and likelihood $L(\theta;\mathbf{y})$, the marginal likelihood of the data under $\mathcal{M}$ is

$m(\mathbf{y}) \;:=\; \int_\Theta \pi(\theta)\, L(\theta;\mathbf{y})\,\mathrm d\theta.$

The Bayes factor comparing two models $\mathcal{M}_1, \mathcal{M}_2$ is the ratio $m_1(\mathbf{y})/m_2(\mathbf{y})$. Combined with prior model odds $\pi(\mathcal{M}_1)/\pi(\mathcal{M}_2)$, it gives posterior model odds.

Let $f(\mathbf{y};\theta)$ be a smooth parametric model with $\Theta \subset \mathbb{R}^k$ and prior $\pi(\theta)$ continuous and strictly positive at the MLE $\hat\theta$. Let $\hat\ell = \ell(\hat\theta;\mathbf{y})$. Define $\mathrm{BIC} := -2\hat\ell + k\log n$. Then under standard regularity conditions,

$-2\log m(\mathbf{y}) \;=\; \mathrm{BIC} + O_p(1).$

The leading-order discrepancy between $-2\log m(\mathbf{y})$ and $\mathrm{BIC}$ is constant in $n$ (depending on the prior and Fisher information), so model rankings by BIC and by $-2\log m(\mathbf{y})$ agree asymptotically.

Suppose the true model $\mathcal{M}_*$ is in the candidate set $\{\mathcal{M}_1, \ldots, \mathcal{M}_M\}$, and standard regularity conditions hold (smooth log-likelihood, identifiable parameter, positive-definite Fisher information at $\theta_0$). Let $\hat{\mathcal{M}}_{\text{BIC}} = \arg\min_k \mathrm{BIC}(\mathcal{M}_k)$ be the BIC-selected model. Then

$\mathbb{P}\!\left(\hat{\mathcal{M}}_{\text{BIC}} = \mathcal{M}_*\right) \;\longrightarrow\; 1 \quad \text{as } n \to \infty.$

BIC is selection-consistent: it identifies the true model with probability $1$ in the large-$n$ limit, when the true model is in the candidate set. Proof outline in CLA2008 §3.2; uses BIC’s $O_p(1)$ gap to log marginal likelihood (Thm 2) plus the asymptotic comparison of nested-model marginal likelihoods.

On the canonical POLY_DGP, BIC values for $d = 0, 1, \ldots, 12$ have argmin at $d = 3$ with $\mathrm{BIC} = 25.4179$ (T10.6) — strictly below AIC’s argmin at $d = 6$. The shift reflects BIC’s $\log n$ vs AIC’s $2$ per-parameter penalty: at $n = 80$, $\log 80 \approx 4.38$, more than double AIC’s penalty. BIC favors a sparser model. This is a core part of the AIC/BIC tension §24.6 Thm 5 will formalize. Figure 4 visualizes the Laplace approximation underlying BIC.

$\exp(-\mathrm{BIC}/2)$ is proportional (asymptotically, by Thm 2) to the unnormalized posterior model probability; normalizing across the candidate set gives BIC weights $w_k^{\text{BIC}} = \exp(-\Delta\mathrm{BIC}_k/2) / \sum_j \exp(-\Delta\mathrm{BIC}_j/2)$. Burnham & Anderson 2002 §2.6 popularized the AIC analog $w_k^{\text{AIC}} = \exp(-\Delta\mathrm{AIC}_k/2) / \sum_j \exp(-\Delta\mathrm{AIC}_j/2)$ as Akaike weights. Both are the discrete-model special case of Bayesian model averaging (BMA), where predictions are weighted by posterior model mass instead of conditioning on a single $\hat{\mathcal{M}}$. Full forward-pointer to BMA at §24.10 Rem 23 + Track 7.

Thm 2’s BIC derivation implicitly assumes a uniform prior on the candidate set $\{\mathcal{M}_1, \ldots, \mathcal{M}_M\}$. Bayesian model comparison admits richer priors — reference priors (Berger–Pericchi), spike-and-slab priors (George–McCulloch 1993), intrinsic priors (Berger–Pericchi 1996) — each tilting the ranking toward sparser or denser models. Full development at Track 7 (Bayesian Foundations).

The per-parameter AIC penalty is $2$; the per-parameter BIC penalty is $\log n$. They equalize at $n = e^2 \approx 7.39$. For all practical sample sizes ($n \geq 10$), BIC penalizes complexity more aggressively than AIC; for $n = 80$ (the POLY_DGP), the BIC per-parameter penalty $\log 80 \approx 4.38$ is more than double AIC’s, explaining T10.6’s argmin shift from $d = 6$ (AIC) to $d = 3$ (BIC).

The Laplace approximation underlying Thm 2 has $O_p(1)$ error — fine for ranking ($\arg\min$ stable up to monotone transforms) but not for reporting a numerical posterior probability. Exact marginal-likelihood computation requires nested sampling (Skilling 2006), thermodynamic integration, or bridge sampling (Meng & Wong 1996); each is a Track-7 topic on its own. BIC’s appeal is that the asymptotic approximation is prior-free and computationally trivial — only $\hat\ell$ and $k$ are needed.

For data $(\mathbf{X}, \mathbf{y})$ with $n$ rows, let $\hat\theta^{(-i)}$ be the estimator fit on the $n - 1$ rows excluding observation $i$, and let $\hat y_i^{(-i)}$ be its prediction at $\mathbf{x}_i$. The leave-one-out cross-validation estimate of mean squared prediction error is

$\mathrm{LOO\text{-}CV} \;:=\; \frac{1}{n}\sum_{i=1}^n \left(y_i - \hat y_i^{(-i)}\right)^2.$

For Gaussian OLS with hat-matrix diagonal $h_{ii}$, the hat-matrix shortcut (PRESS statistic; Allen 1974) avoids the $n$ refits:

$y_i - \hat y_i^{(-i)} \;=\; \frac{y_i - \hat y_i}{1 - h_{ii}}.$

The shortcut requires $h_{ii} < 1$ for all $i$; looCV in regression.ts throws when $\max h_{ii} \geq 0.999$.

Consider the Gaussian linear model $\mathbf{y} = \mathbf{X}\boldsymbol\beta + \boldsymbol\varepsilon$ with $\boldsymbol\varepsilon \sim \mathcal{N}_n(\mathbf{0}, \sigma^2\mathbf{I})$ and a fixed full-rank design $\mathbf{X}$. Under regularity (balanced design, fixed $p$, $\max h_{ii} = O(\log n / n)$),

$n\log\mathrm{LOO\text{-}CV} \;=\; \mathrm{AIC}^* + O_p(n^{-1}),$

where $\mathrm{AIC}^* := n\log(\mathrm{RSS}/n) + 2k$ is AIC up to model-invariant additive constants. Consequently,

$\arg\min_{\mathcal{M}} \mathrm{LOO\text{-}CV}(\mathcal{M}) \;=\; \arg\min_{\mathcal{M}} \mathrm{AIC}(\mathcal{M}) + o_p(1).$

LOO-CV and AIC asymptotically select the same model from any nested family on the Gaussian-linear model.

On POLY_DGP, both $\arg\min_d \mathrm{LOO\text{-}CV} = 6$ and $\arg\min_d \mathrm{AIC} = 6$ over $d \in [0, 11]$ (T10.26 — d=12 is excluded because the monomial Vandermonde is too ill-conditioned for the hat-matrix shortcut; the argmin lies safely inside the range). At the joint argmin $d = 6$, $\mathrm{LOO\text{-}CV} = 0.063835$ (T10.10) and $\mathrm{AIC}^* = -218.6963$ (computed from the full Gaussian AIC by stripping the $n(\log 2\pi + 1)$ constants); the gap $|n\log\mathrm{LOO\text{-}CV} - \mathrm{AIC}^*| < 2.5$ (T10.27) is the order-1 constant Proof 3 Step 4 predicts. Figure 5 overlays the LOO-CV curve, the 5-fold and 10-fold CV curves, and the AIC curve over $d$.

The single-loop CV Topic 23 §23.8 used to select $\lambda$ should not also serve as the test-error estimator: reporting $\min_\lambda \mathrm{CV}(\lambda)$ as test error leaks tuning information into the test estimate (selection bias). Nested cross-validation fixes this: an outer CV loop holds out folds for honest test-error estimation, and within each outer fold, an inner CV loop selects $\lambda$. Bates–Hastie–Tibshirani (2024) give the recent rigorous treatment. The result is an asymptotically unbiased generalization-error estimate at the cost of $K_\text{outer} \cdot K_\text{inner}$ refits.

$k$-fold CV is the finite-sample analog of LOO: each fold holds out $n/k$ observations. As $k \to n$, the procedure converges to LOO. Smaller $k$ has lower computational cost but higher bias (more held-out fold per refit means more train-set shrinkage). Hastie et al. 2009 §7.10 and Claeskens & Hjort 2008 §4.3 recommend $k = 10$ as the modern default — slightly biased upward vs LOO but with lower Monte Carlo variance.

Thm 4 identifies LOO-CV with AIC; no analogous result exists for BIC at fixed $k$. The mismatch is asymptotic: BIC’s $k\log n$ penalty grows in $n$, while CV’s effective penalty grows like $2k$. To recover a BIC-like criterion from CV, one would have to use a vanishing-fraction held-out set ($n_\text{val} = o(n)$), not standard $k$-fold (Shao 1993).

Let $\hat{\mathcal{M}}$ be a model-selection procedure operating on a candidate family $\{\mathcal{M}_1, \ldots, \mathcal{M}_M\}$. Suppose $\hat{\mathcal{M}}$ is selection-consistent in the well-specified regime: when $\mathcal{M}_*$ is in the candidate set, $\mathbb{P}(\hat{\mathcal{M}} = \mathcal{M}_*) \to 1$. Then $\hat{\mathcal{M}}$ is not minimax-rate-optimal in the misspecified-nonparametric regime: there exists a family of underlying truths and a sample-size sequence on which the prediction risk of $\hat{\mathcal{M}}$ achieves a strictly worse rate than the minimax-optimal procedure (e.g., AIC or CV).

Proof: Yang 2005 §2–3, via a minimax lower-bound argument outside the scope of Topic 24.

The Yang race compares procedures across two DGP regimes:

• Tab A — well-specified. Truth is a degree-3 polynomial; candidate set $d \in [0, 8]$ contains the truth. As $n$ grows, BIC’s selection frequency at $d = 3$ tends to $1$ (consistency); AIC’s selection frequency at $d \geq 4$ persists at $\sim 25\%$ (asymptotic over-fit). Prediction risks are similar at any finite $n$.
• Tab B — misspecified. Truth is $\sin(2\pi x)$; candidate set is polynomials $d \in [0, 12]$ (truth not in the candidate set). AIC’s minimax-optimal selection adapts $d$ to $n$; BIC’s $\log n$ penalty over-shrinks toward sparsity, giving worse prediction risk on the polynomial scale of the truth’s smoothness. The risks diverge in $n$.

Figure 6 plots both regimes side by side.

Thm 5 forces a choice the practitioner cannot duck: either prioritize identifying the right model (BIC) or prioritize accurate predictions (AIC, CV). The choice is not an empirical question to be settled by simulation but a value judgment about the use case — interpretive parsimony vs predictive accuracy. Shao 1997 and Burnham & Anderson 2002 §2.10 give complementary discussions of how to pick a side in a given application context.

Shao (1993) showed that leaving out $n_v = n - n^{1/2}$ observations per fold (held-out fraction shrinking to zero asymptotically) gives a CV variant that is selection-consistent on Gaussian linear models with bounded $p$. The result is a limited positive: Shao’s CV variant is not minimax-rate-optimal for prediction in the nonparametric regime, so it lives on the BIC side of Yang’s split rather than circumventing the incompatibility.

Standard practice: report both AIC and BIC. Agreement is reassuring; disagreement is informative — it tells the reader which side of the consistency-vs-efficiency tradeoff matters for the application. For prediction-focused use cases, prefer the AIC argmin and report 10-fold CV as a sanity check; for inference-focused applications where parsimony matters (e.g. model identification in epidemiology), prefer the BIC argmin.

Let $\mathcal{M}_{\text{full}}$ and $\mathcal{M}_{\text{red}}$ be nested with $k_{\text{full}} = k_{\text{red}} + q$ free parameters ($q \geq 1$). Let $\hat\ell_{\text{full}}, \hat\ell_{\text{red}}$ be their MLE log-likelihoods on the same data. Then

$\Delta\mathrm{AIC} \;:=\; \mathrm{AIC}(\mathcal{M}_{\text{full}}) - \mathrm{AIC}(\mathcal{M}_{\text{red}}) \;=\; (-2\hat\ell_{\text{full}} + 2k_{\text{full}}) - (-2\hat\ell_{\text{red}} + 2k_{\text{red}}) \;=\; 2q - 2\Delta\hat\ell,$

with $\Delta\hat\ell = \hat\ell_{\text{full}} - \hat\ell_{\text{red}}$. AIC prefers the full model iff $\Delta\mathrm{AIC} < 0$ iff $2\Delta\hat\ell > 2q$ — exactly the LRT rejection rule with threshold $2q$ instead of the chi-square critical value $\chi^2_{q,1-\alpha}$. BIC uses the analogous threshold $q\log n$ in place of $2q$: BIC prefers the full model iff $2\Delta\hat\ell > q\log n$.

On the nested-Poisson DGP ($n = 200$, $\eta_{\text{true}} = 1.0 + 0.8 x_1 - 0.5 x_2$, $x_3$ has no true effect, default_rng(123)), fit the reduced model with $\{1, x_1, x_2\}$ and the full model with $\{1, x_1, x_2, x_3\}$. The pinned values: $\hat\beta_{\text{red}} \approx (0.9815, 0.8216, -0.5769)$ (T10.12–T10.14); $\hat\beta_{\text{full}}[3] \approx -0.0535$ (T10.15, near-zero as expected). The likelihood ratio is

$\mathrm{LR} \;=\; 2\Delta\hat\ell \;\approx\; 0.5547 \quad (\text{T10.16}).$

Since $q = 1$ and the LRT critical value at $\alpha = 0.05$ is $\chi^2_{1, 0.95} \approx 3.84$, the LRT does NOT reject (p-value $\approx 0.46$).

Now apply Thm 6:

$\Delta\mathrm{AIC} \;=\; 2(1) - \mathrm{LR} \;=\; 2 - 0.5547 \;\approx\; 1.4453 > 0 \quad (\text{T10.17}).$

$\Delta\mathrm{BIC} \;=\; (1)\log(200) - \mathrm{LR} \;\approx\; 5.298 - 0.5547 \;\approx\; 4.7436 > 0 \quad (\text{T10.18}).$

Both AIC and BIC prefer the reduced model. T10.19 verifies the algebraic identity $\Delta\mathrm{AIC} = 2 - \mathrm{LR}$ to within $10^{-10}$. Figure 7 plots the chi-square null density with the observed LR and both AIC/BIC thresholds for visual comparison.

Thm 6’s identification of AIC with a fixed-threshold LRT means AIC has an effective $\alpha$ that depends on $q$. For $q = 1$, $\mathbb{P}(\chi^2_1 \geq 2) \approx 0.157$; for $q = 5$, $\mathbb{P}(\chi^2_5 \geq 10) \approx 0.075$; for $q = 10$, $\mathbb{P}(\chi^2_{10} \geq 20) \approx 0.029$. AIC is more liberal (admits more parameters) at small $q$, more conservative at large $q$ — the opposite of the LRT-with-fixed-$\alpha$ rule, which has fixed type-I error regardless of $q$.

Thm 6 applies only to nested comparisons. For non-nested candidates (e.g. degree-5 polynomial vs degree-3 spline with the same effective dimension), the likelihood-ratio statistic is undefined and the LRT framework breaks down. The IC framework still applies: compute $\mathrm{AIC}$ or $\mathrm{BIC}$ for each candidate and rank by the smaller value, no chi-square reference needed.

For a fitting procedure $\hat{\mathbf{f}}$ that maps data $\mathbf{y}$ to fitted values $\hat{\mathbf{f}}(\mathbf{y})$ on the same $n$ rows, the effective degrees of freedom is

$\mathrm{df}(\hat{\mathbf{f}}) \;:=\; \frac{1}{\sigma^2}\sum_{i=1}^n \operatorname{Cov}\!\left(\hat f_i, y_i\right).$

For OLS, $\mathrm{df} = p + 1$ (intercept + $p$ slopes — exactly the parameter count). For penalized estimators, $\mathrm{df}$ can be non-integer and depends continuously on the regularization parameter.

For ridge regression on a centered design $\tilde{\mathbf{X}}$ with SVD $\tilde{\mathbf{X}} = \mathbf{U}\mathbf{D}\mathbf{V}^\top$ (singular values $d_1, \ldots, d_p > 0$), the smoother matrix is

$\mathbf{H}_\lambda \;=\; \tilde{\mathbf{X}}(\tilde{\mathbf{X}}^\top\tilde{\mathbf{X}} + \lambda\mathbf{I})^{-1}\tilde{\mathbf{X}}^\top.$

Substituting the SVD and using $\tilde{\mathbf{X}}^\top\tilde{\mathbf{X}} = \mathbf{V}\mathbf{D}^2\mathbf{V}^\top$ plus the rotational invariance of the trace:

$\mathrm{df}(\lambda) \;=\; \operatorname{tr}(\mathbf{H}_\lambda) \;=\; \sum_{j=1}^p \frac{d_j^2}{d_j^2 + \lambda}.$

The DOF is monotone-decreasing in $\lambda$: $\mathrm{df}(0) = p$ (unpenalized OLS), $\mathrm{df}(\lambda) \to 0$ as $\lambda \to \infty$. Equivalently and computationally cheaper, $\mathrm{df}(\lambda) = p - \lambda \cdot \operatorname{tr}((\tilde{\mathbf{X}}^\top\tilde{\mathbf{X}} + \lambda\mathbf{I})^{-1})$ via Cholesky inversion (the form regression.ts’s hatMatrixTrace uses).

For lasso regression on a centered, orthonormal design (or under the more general restricted-eigenvalue conditions of WAI2019 Ch. 7), the active set $\mathcal{A}(\hat{\boldsymbol\beta})$ — the indices $j$ with $\hat\beta_j \neq 0$ — satisfies

$\mathbb{E}\!\left[\mathrm{df}(\hat{\boldsymbol\beta}_\lambda)\right] \;=\; \mathbb{E}\!\left[|\mathcal{A}(\hat{\boldsymbol\beta}_\lambda)|\right].$

The active-set size is an unbiased estimator of expected effective DOF. Caveats: the result fails at “knots” of the lasso path where the active set changes discontinuously, and ties in the optimization can make $|\mathcal{A}|$ data-dependent in a non-smooth way. Tibshirani & Taylor 2012 give the precise regularity statement.

For the poly $d = 10$ design on $x \sim \mathcal{U}(0,1)$ at $n = 80$ (the canonical POLY_DGP $x$-values), the ridge effective DOF as $\lambda$ varies:

At $\lambda = 0$, $\mathrm{df} = 11$ exactly (intercept + 10 polynomial coefficients). Even modest regularization ($\lambda = 0.01$) drops the effective DOF by more than half — the high-order polynomial columns have small singular values and shrink rapidly under the ridge penalty.

Refit the prostate-cancer lasso path of Topic 23 §23.9 Ex 14 ($n = 97$, $p = 8$, response lpsa) on a 100-point log-grid of $\lambda$. For each $\lambda$, compute $\mathrm{df}(\lambda) = |\mathcal{A}(\hat{\boldsymbol\beta}_\lambda)|$ (Thm 8) and apply