Bayesian Foundations & Prior Selection

A frequentist answer to “what is $\theta$?” is a single number (the MLE) plus an interval around it justified by long-run coverage. A Bayesian answer is a density: the full posterior $p(\theta \mid \mathbf{y})$, which assigns probabilities to every neighborhood of parameter space. Point estimates (posterior mean, posterior median, MAP) and interval estimates (credible intervals, HPD intervals) are summaries of the posterior, not substitutes for it.

The philosophical debate between subjective Bayes (LIN2014 — priors as genuine personal beliefs) and objective Bayes (JEF1961 — priors as reference or non-informative distributions that minimize subjective input) is out of scope for Topic 25. We treat priors as substantive modeling choices whose sensitivity we examine empirically (§25.7), and we note where the two schools diverge as we go.

Topic 25 covers: Bayes’ theorem for $\theta$ (§25.2), Track 7 notation (§25.3), exponential-family conjugacy with full proof (§25.4), five canonical conjugate pairs with Beta-Binomial in full (§25.5), credible intervals and posterior predictive (§25.6), prior selection including Jeffreys (§25.7), Bernstein–von Mises with a sketch proof (§25.8), multivariate Normal-Inverse-Wishart as a pointer (§25.9), and the forward-map to Track 7 (§25.10). It does not cover: MCMC algorithms (Topic 26), Bayes factors or BMA beyond naming (Topic 27), hierarchical and empirical Bayes (Topic 28), variational inference or Bayesian nonparametrics (formalml). Each deferral gets a §25.10 pointer.

Let $\theta \in \Theta$ be an unknown parameter and $\mathbf{y} = (y_1, \ldots, y_n)$ an observed sample. The Bayesian framework attaches five densities to the inference problem:

• Prior $\pi(\theta)$: distribution on $\Theta$ encoding information about $\theta$ before observing data.
• Likelihood $L(\theta; \mathbf{y}) = p(\mathbf{y} \mid \theta)$: the sampling density, viewed as a function of $\theta$ with $\mathbf{y}$ fixed.
• Posterior $p(\theta \mid \mathbf{y})$: distribution on $\Theta$ conditional on the observed data.
• Marginal likelihood $m(\mathbf{y}) = \int_\Theta L(\theta; \mathbf{y}) \pi(\theta) \, d\theta$: the prior-predictive density of $\mathbf{y}$, also called the evidence.
• Posterior predictive $p(\tilde y \mid \mathbf{y}) = \int_\Theta p(\tilde y \mid \theta) p(\theta \mid \mathbf{y}) \, d\theta$: distribution of a future observation $\tilde y$ integrating out posterior uncertainty.

Under the measure-theoretic conditions of Topic 4 Thm 4 (absolute continuity of the joint distribution with respect to a product measure on $\Theta \times \mathcal{Y}^n$),

$p(\theta \mid \mathbf{y}) \;=\; \frac{L(\theta; \mathbf{y}) \, \pi(\theta)}{m(\mathbf{y})},$

where $m(\mathbf{y}) = \int_\Theta L(\theta; \mathbf{y}) \pi(\theta) \, d\theta$ is the marginal likelihood. Equivalently, in unnormalized form,

$p(\theta \mid \mathbf{y}) \;\propto\; L(\theta; \mathbf{y}) \, \pi(\theta).$

Proof is a direct application of Topic 4 Thm 4 to the joint density $p(\theta, \mathbf{y}) = \pi(\theta) L(\theta; \mathbf{y})$; the computation is exactly the event version with $P$ replaced by density integrals.

Let $Y \sim \mathrm{Binomial}(n=4, \theta)$ and place a uniform prior $\theta \sim \mathrm{Beta}(1, 1)$ (flat on $(0, 1)$). Observe $y = 3$. Then:

$L(\theta; y = 3) \;=\; \binom{4}{3} \theta^3 (1-\theta)^1 \;\propto\; \theta^3 (1-\theta),$

and the unnormalized posterior is

$p(\theta \mid y = 3) \;\propto\; \theta^3 (1 - \theta) \cdot 1 \;=\; \theta^3 (1-\theta),$

which is the kernel of $\mathrm{Beta}(4, 2)$. Normalizing, $p(\theta \mid y = 3) = \mathrm{Beta}(\theta; 4, 2)$, with posterior mean $4/6 = 2/3$. The data shifted the posterior mean from $1/2$ (prior) to $2/3$ — and shrunk the posterior SD from $\sqrt{1/12} \approx 0.29$ to $\sqrt{4 \cdot 2 / (6^2 \cdot 7)} \approx 0.178$.

In practice one rarely computes the marginal likelihood $m(\mathbf{y})$ directly — the proportionality $p(\theta \mid \mathbf{y}) \propto L(\theta) \pi(\theta)$ together with the constraint $\int p(\theta \mid \mathbf{y}) \, d\theta = 1$ determines the posterior uniquely. For conjugate pairs (§25.5) the identification of the posterior kernel with a known density family sidesteps the integral entirely. For non-conjugate cases, MCMC (Topic 26) sidesteps it via ratio-based acceptance.

A family of distributions $\{\pi(\theta \mid \chi, \nu) : \chi \in \mathbb{R}, \nu > 0\}$ is conjugate to a likelihood $\{f(y; \theta) : \theta \in \Theta\}$ if, for every prior $\pi(\theta \mid \chi_0, \nu_0)$ in the family and every sample $\mathbf{y}$, the posterior $p(\theta \mid \mathbf{y})$ is also in the family — only the hyperparameters update.

Let $\{f(y; \theta) : \theta \in H\}$ be a one-parameter exponential family in canonical form,

$f(y; \eta) \;=\; h(y) \exp\!\big(\eta T(y) - A(\eta)\big),$

with natural parameter $\eta \in H \subseteq \mathbb{R}$, sufficient statistic $T(y)$, base measure $h(y)$, and log-partition $A(\eta)$. Define the conjugate-prior family indexed by $(\chi_0, \nu_0) \in \mathbb{R} \times (0, \infty)$ by

$\pi(\eta \mid \chi_0, \nu_0) \;=\; K(\chi_0, \nu_0) \exp\!\big(\chi_0 \eta - \nu_0 A(\eta)\big),$

where $K(\chi_0, \nu_0)$ is the normalizing constant (finite whenever the integral on $H$ converges). For $n$ iid observations $\mathbf{y} = (y_1, \ldots, y_n)$ with sufficient-statistic total $S = \sum_{i=1}^n T(y_i)$, the posterior is

$p(\eta \mid \mathbf{y}) \;=\; \pi(\eta \mid \chi_0 + S, \; \nu_0 + n).$

The hyperparameter $\nu_0$ is the pseudo-sample-size: the prior is worth $\nu_0$ equivalent observations before any data arrive. $\chi_0$ is the pseudo-sufficient-statistic total.

Topic 7 §7.7 Thm 3 constructed the conjugate-prior family for an exponential likelihood — the algebra of Proof 1 Step 4 but stripped of inferential framing. Thm 2 adds the inferential layer: the updated hyperparameters are now the posterior, and every downstream Bayesian quantity (credible interval, posterior mean, posterior predictive) follows from this one identification. Where Topic 7 asked “what family of priors is closed under multiplication by this likelihood?” Topic 25 answers “what is the posterior given this prior and these data?” — but they are literally the same calculation.

The hyperparameter pair $(\chi_0, \nu_0)$ is interpretable as a pseudo-dataset of $\nu_0$ observations whose sufficient-statistic total is $\chi_0$. Under this reading, the prior encodes prior information by committing to the equivalent of $\nu_0$ imaginary observations, and Bayes’ theorem literally concatenates them with the real data. Informative priors have large $\nu_0$; weakly informative priors have $\nu_0 \sim 1$; non-informative improper priors correspond to the formal limit $\nu_0 \to 0$ (§25.7). This pseudo-observation framing is the clearest way to calibrate how much prior information to commit.

Let $Y \sim \mathrm{Binomial}(n, \theta)$ and place a $\mathrm{Beta}(\alpha_0, \beta_0)$ prior on $\theta \in (0, 1)$. After observing $k$ successes in $n$ trials,

$\theta \mid Y = k \;\sim\; \mathrm{Beta}(\alpha_0 + k, \; \beta_0 + n - k).$

The posterior mean is

$\mathbb{E}[\theta \mid k] \;=\; \frac{\alpha_0 + k}{\alpha_0 + \beta_0 + n} \;=\; w \cdot \frac{\alpha_0}{\alpha_0 + \beta_0} \;+\; (1-w) \cdot \frac{k}{n},$

a convex combination of the prior mean and the sample proportion, with weight $w = (\alpha_0 + \beta_0) / (\alpha_0 + \beta_0 + n)$ proportional to the pseudo-sample-size.

Let $Y_i \mid \mu \overset{\text{iid}}{\sim} \mathcal{N}(\mu, \sigma^2)$ with $\sigma^2$ known, and prior $\mu \sim \mathcal{N}(\mu_0, \sigma_0^2)$. The posterior on $\mu$ given the sample mean $\bar y$ and sample size $n$ is $\mathcal{N}(\mu_n, \sigma_n^2)$ with

$\frac{1}{\sigma_n^2} \;=\; \frac{1}{\sigma_0^2} + \frac{n}{\sigma^2}, \qquad \mu_n \;=\; \sigma_n^2 \left(\frac{\mu_0}{\sigma_0^2} + \frac{n \bar y}{\sigma^2}\right).$

The posterior precision is the sum of prior precision and data precision — the canonical precision-weighted averaging formula. The posterior mean is the precision-weighted average of the prior mean and the sample mean. Derivation mirrors Proof 2 but with Gaussian kernels; see Topic 7 §7.7 for the exp-family algebra in the location-family case.

When $\sigma^2$ is unknown, place a joint Normal-Inverse-Gamma prior

$\mu \mid \sigma^2 \sim \mathcal{N}(\mu_0, \sigma^2/\kappa_0), \qquad \sigma^2 \sim \mathrm{InvGamma}(\alpha_0, \beta_0).$

After observing $n$ iid data with sample mean $\bar y$ and sample variance $s^2$, the posterior is Normal-Inverse-Gamma with updated hyperparameters (GEL2013 §3.3 Eqs 3.3–3.5):

$\kappa_n = \kappa_0 + n, \quad \mu_n = \frac{\kappa_0 \mu_0 + n \bar y}{\kappa_n}, \quad \alpha_n = \alpha_0 + \tfrac{n}{2},$

$\beta_n = \beta_0 + \tfrac{n s^2}{2} + \tfrac{\kappa_0 n (\bar y - \mu_0)^2}{2 \kappa_n}.$

The marginal posterior on $\mu$ is a non-standardized Student-t with $2\alpha_n$ degrees of freedom, location $\mu_n$, and scale $\sqrt{\beta_n / (\alpha_n \kappa_n)}$ — heavier tails than Normal because $\sigma^2$‘s uncertainty widens the marginal on $\mu$.

Let $Y_i \mid \lambda \overset{\text{iid}}{\sim} \mathrm{Poisson}(\lambda)$ and prior $\lambda \sim \mathrm{Gamma}(\alpha_0, \beta_0)$ (shape-rate). Observing $S = \sum_{i=1}^n y_i$, the posterior is

$\lambda \mid \mathbf{y} \;\sim\; \mathrm{Gamma}(\alpha_0 + S, \; \beta_0 + n),$

with posterior mean $(\alpha_0 + S)/(\beta_0 + n)$. The algebra reduces to the exp-family update rule of Thm 2 with $T(y) = y$, $A(\eta) = e^\eta$ (Poisson canonical form). See Topic 7 §7.7 Ex 3 for the matching derivation.

Let $(Y_1, \ldots, Y_K) \sim \mathrm{Multinomial}(n, \mathbf{p})$ with $\sum_k p_k = 1$, and prior $\mathbf{p} \sim \mathrm{Dirichlet}(\boldsymbol\alpha_0)$ for $\boldsymbol\alpha_0 = (\alpha_{0,1}, \ldots, \alpha_{0,K})$. Observing counts $\mathbf{x} = (x_1, \ldots, x_K)$ with $\sum_k x_k = n$, the posterior is

$\mathbf{p} \mid \mathbf{x} \;\sim\; \mathrm{Dirichlet}(\boldsymbol\alpha_0 + \mathbf{x}),$

the natural generalization of Beta-Binomial to $K > 2$ categories. Topic 8 §8 Thm 6 works this as pure algebra; Topic 25 adds the inferential layer (credible sets on the simplex, marginal posteriors on individual $p_k$).

Every pair in §25.5 is a specialization of the exponential-family conjugacy theorem. Beta-Binomial: $T(y) = y$, $A(\eta) = \log(1 + e^\eta)$ (logit-parameterized Bernoulli). Normal-Normal known $\sigma^2$: $T(y) = y$, $A(\eta) = \eta^2 / 2$ (location-only Gaussian). Gamma-Poisson: $T(y) = y$, $A(\eta) = e^\eta$. Dirichlet-Multinomial: the multi-parameter extension where $T(\mathbf{y}) = \mathbf{y}$ is a count vector. Normal-Normal-IG is the two-parameter case with $(\mu, \sigma^2)$ jointly — the algebra is more involved but the conjugacy logic is identical.

Conjugate priors exist only for exponential-family likelihoods. Non-exponential-family models — mixtures, hierarchical models beyond Topic 28, neural networks — have no closed-form conjugate priors. In these cases, the posterior is known only up to the intractable normalizing constant $m(\mathbf{y})$, and one must resort to approximation: MCMC (Topic 26) for exact asymptotic sampling, variational inference (formalml) for fast approximate posteriors, or Laplace approximation (§25.8 Rem 16) for local Gaussian approximations at the MAP.

For a posterior $p(\theta \mid \mathbf{y})$ and level $1 - \alpha$, a set $C \subseteq \Theta$ is a $(1 - \alpha)$ credible set if $\int_C p(\theta \mid \mathbf{y}) \, d\theta = 1 - \alpha$. Two canonical choices:

• Equal-tailed credible interval: $C = [q_{\alpha/2}, q_{1 - \alpha/2}]$ where $q_p$ is the posterior $p$-quantile.
• Highest-posterior-density (HPD) interval: $C^{\text{HPD}} = \{\theta : p(\theta \mid \mathbf{y}) \ge c_\alpha\}$ for the largest $c_\alpha$ such that $C^{\text{HPD}}$ has posterior mass $1 - \alpha$. The HPD is the shortest $(1-\alpha)$ credible set.

For symmetric unimodal posteriors the two coincide. For skewed posteriors the HPD is narrower and shifted toward the mode.

The three standard Bayesian point estimators of $\theta$ are:

• Posterior mean: $\hat\theta_{\text{PM}} = \mathbb{E}[\theta \mid \mathbf{y}] = \int \theta \, p(\theta \mid \mathbf{y}) \, d\theta$.
• Posterior median: $\hat\theta_{\text{med}}$, the $0.5$-quantile of $p(\theta \mid \mathbf{y})$.
• MAP estimate: $\hat\theta_{\text{MAP}} = \arg\max_\theta p(\theta \mid \mathbf{y})$.

Each is Bayes-optimal under a different loss: posterior mean under squared-error loss, posterior median under absolute-error loss, MAP under 0-1 (indicator) loss — as $\epsilon \to 0$ in $L(\hat\theta, \theta) = \mathbf{1}\\{|\hat\theta - \theta| > \epsilon\\}$.

Under the Beta-Binomial setup of Thm 3 with posterior $\theta \mid \mathbf{y} \sim \mathrm{Beta}(\alpha^\star, \beta^\star)$ where $\alpha^\star = \alpha_0 + k$, $\beta^\star = \beta_0 + n - k$, the posterior-predictive distribution of a future count $\tilde y \in \{0, 1, \ldots, m\}$ from $m$ new independent Bernoulli($\theta$) trials is the Beta-Binomial compound:

$p(\tilde y \mid \mathbf{y}) \;=\; \binom{m}{\tilde y} \frac{B(\tilde y + \alpha^\star, \; m - \tilde y + \beta^\star)}{B(\alpha^\star, \beta^\star)}.$

The variance of $\tilde y$ under the posterior predictive strictly exceeds the variance under the plug-in Binomial$(m, \hat\theta)$ — the Bayesian predictive distribution inflates uncertainty to reflect posterior uncertainty in $\theta$.

Under the standard data scenario used in §25.5 (the posterior hyperparameters from each conjugate-pair example), equal-tailed 95% credible intervals are:

• Beta-Binomial with $\alpha^\star = 12, \beta^\star = 42$: $[0.123, 0.341]$ (via the Beta-quantile function).
• Normal-Normal with $\mu_n = 0.988, \sigma_n^2 = 0.049$: $\mu_n \pm 1.96\sigma_n = [0.554, 1.422]$.
• Gamma-Poisson with $\alpha^\star = 17, \beta^\star = 6$: approximately $[1.65, 4.47]$.
• Normal-Normal-IG with $2\alpha_n$ df Student-t marginal on $\mu$: non-symmetric if the df is small.
• Dirichlet-Multinomial on individual $p_k$: marginal is Beta with hyperparameters $(\alpha_k, \sum_{j \ne k} \alpha_j)$.

The HPD interval is slightly narrower and shifted toward the posterior mode in every skewed case (Beta-Binomial for $\alpha^\star \ne \beta^\star$, Gamma-Poisson, asymmetric Dirichlet marginals).

Under squared-error loss $L(\hat\theta, \theta) = (\hat\theta - \theta)^2$, the Bayes estimator — the one minimizing posterior expected loss $\mathbb{E}[L \mid \mathbf{y}]$ — is exactly the posterior mean. This is the Bayesian reading of the bias-variance decomposition: the posterior-mean point estimator is optimal in the $L^2$ sense against the posterior, the same way the sample mean is $L^2$-optimal against the empirical distribution. See ROB2007 §2 for the decision-theoretic development.

For a symmetric unimodal posterior the HPD and equal-tailed intervals coincide. For skewed posteriors (e.g., Beta(2, 20) with pronounced right skew), the equal-tailed interval allocates equal tail mass $\alpha/2$ to each side, including a long right tail of low-density values; the HPD instead follows the density and is strictly narrower. The tradeoff: equal-tailed is invariant under monotone reparameterization; HPD is not.

For Normal-Normal known $\sigma^2$ the posterior predictive for a future $\tilde y$ is $\mathcal{N}(\mu_n, \sigma_n^2 + \sigma^2)$ — the posterior variance plus the likelihood variance, again wider than the plug-in $\mathcal{N}(\hat\mu, \sigma^2)$. For Normal-Normal-Inverse-Gamma (unknown $\sigma^2$), the predictive is a non-standardized Student-t with $2\alpha_n$ df — heavier tails still, because $\sigma^2$‘s uncertainty further inflates predictive variance. See Topic 6 §6.7 for the Student-t distribution.

Priors divide roughly into three informational tiers:

• Informative: encode substantive prior knowledge. Example: a pharmacologist’s prior on a drug’s dose-response parameter based on ten years of related trials. Hyperparameters reflect that knowledge (small posterior variance when knowledge is strong).
• Weakly informative: encode loose, regularization-style constraints. The canonical example is $\mathrm{Cauchy}(0, 2.5)$ on logistic-regression coefficients (Gelman et al. 2008) — excludes implausible values like $|\beta| > 100$ without committing to any specific scale.
• Non-informative: minimize prior influence. Jeffreys (§25.7 Thm 5), reference priors (Bernardo 1979 — deferred), flat priors (often improper).

The pragmatic recommendation (GEL2013 Ch. 2): use weakly informative priors by default; use informative priors when genuine prior knowledge exists; use non-informative priors for benchmarking.

For a one-parameter regular family with Fisher information $\mathcal{I}(\theta)$, the Jeffreys prior is

$\pi_J(\theta) \;\propto\; \sqrt{\mathcal{I}(\theta)}.$

The Jeffreys prior is reparameterization-invariant: under a smooth monotone reparameterization $\phi = g(\theta)$ with Jacobian $|d\theta/d\phi|$, $\pi_J(\phi) = \pi_J(\theta) |d\theta/d\phi|$ — the transformed Jeffreys prior equals the Jeffreys prior computed directly in the $\phi$-parameterization. This invariance is the principled argument for the Jeffreys construction as a reference prior: any other “non-informative” choice (e.g., $\pi(\theta) =$ const) depends on the parameterization.

Derivation (Bernoulli). For $Y \sim \mathrm{Bernoulli}(\theta)$, $\mathcal{I}(\theta) = 1/(\theta(1-\theta))$, so $\pi_J(\theta) \propto \theta^{-1/2} (1-\theta)^{-1/2}$, which is the $\mathrm{Beta}(1/2, 1/2)$ density up to normalization.

Derivation (Normal-scale). For $Y \sim \mathcal{N}(\mu_0, \sigma^2)$ with $\mu_0$ known and $\sigma > 0$ unknown, $\mathcal{I}(\sigma) = 2/\sigma^2$, so $\pi_J(\sigma) \propto 1/\sigma$ — the classical log-flat scale prior. (Improper; see Def 7.)

Invariance property proof uses the Jacobian identity for Fisher information: $\mathcal{I}(\phi) = \mathcal{I}(\theta) (d\theta/d\phi)^2$, so $\sqrt{\mathcal{I}(\phi)} = \sqrt{\mathcal{I}(\theta)} |d\theta/d\phi|$. See JEF1961 Ch. III for the full invariance argument.

Observe 10 successes in 50 Bernoulli trials. Compare three posteriors:

• Informative $\mathrm{Beta}(2, 8)$ peaked at $0.2$: posterior $\mathrm{Beta}(12, 48)$, mean $0.2$.
• Weakly informative $\mathrm{Beta}(2, 2)$ centered at $0.5$: posterior $\mathrm{Beta}(12, 42)$, mean $\approx 0.222$.
• Non-informative Jeffreys $\mathrm{Beta}(1/2, 1/2)$: posterior $\mathrm{Beta}(10.5, 40.5)$, mean $\approx 0.206$.

At $n = 50$ all three posterior means lie in $[0.20, 0.23]$ — the data dominates the informative prior’s pull toward $0.2$. At $n = 10$, the three posterior means are far apart ($\sim 0.21$, $\sim 0.23$, $\sim 0.25$); at $n = 1000$ they are numerically identical. The interpolation is exactly what Bernstein–von Mises (§25.8) predicts.

A prior $\pi(\theta)$ is improper if $\int_\Theta \pi(\theta) \, d\theta = \infty$ — i.e., it is not a valid probability density. Examples: $\pi(\mu) = \text{const}$ on $\mathbb{R}$; $\pi(\sigma) \propto 1/\sigma$ on $(0, \infty)$ (the Normal-scale Jeffreys prior).

Bayesian inference with an improper prior is valid if and only if the posterior is proper:

$\int_\Theta L(\theta) \pi(\theta) \, d\theta \;<\; \infty.$

When this integrable-posterior condition holds, the posterior $p(\theta \mid \mathbf{y}) \propto L(\theta) \pi(\theta)$ is a well-defined probability density and all downstream quantities (credible intervals, posterior moments, posterior predictive) are well-defined. When it fails, no Bayesian inference is possible.

Improper priors can produce paradoxes when naively combined. Stone (1970) and later Dawid, Stone, and Zidek (1973) showed that two proper-posterior improper priors on nested parameter spaces can yield conditionally inconsistent posteriors — the marginal posterior derived one way disagrees with the marginal posterior derived another way on the same parameter. The resolution requires careful measure-theoretic handling of the improper prior limit. Topic 25 names the paradox once; full treatment deferred to Topic 27 or formalml.

Bernardo (1979) proposed reference priors as a generalization of Jeffreys that maximizes asymptotic expected Kullback–Leibler divergence between prior and posterior — i.e., the prior is chosen to be as “non-informative” as possible in an information-theoretic sense. For one-parameter regular families, reference priors coincide with Jeffreys priors; for multi-parameter problems they differ. Full development is deferred to Topic 27 or formalml.

In applied work, priors are rarely chosen from theoretical principles alone. Prior elicitation — the systematic conversion of expert knowledge into prior hyperparameters — is a substantive modeling activity (ROB2007 §3; GEL2013 §2.9). Practical approaches: ask experts for quantile estimates and fit a prior that matches; use historical data from related problems; elicit pseudo-sample-size directly. The sensitivity analysis of Ex 7 is the diagnostic: if the inference is sensitive to the prior, the prior deserves more attention; if not, the data has spoken.

Let $Y_1, \ldots, Y_n \overset{\text{iid}}{\sim} f(\cdot; \theta_0)$ with $\theta_0$ in the interior of $\Theta \subseteq \mathbb{R}$, a regular parametric family with Fisher information $\mathcal{I}(\theta_0) \in (0, \infty)$. Let $\hat\theta_n$ be the MLE. For any proper prior $\pi$ continuous and positive at $\theta_0$,

$\sup_{B \in \mathcal{B}(\mathbb{R})} \left| \Pr_{\theta \sim p(\cdot \mid \mathbf{y})}\!\left(\sqrt{n}(\theta - \hat\theta_n) \in B\right) - \Pr_{Z \sim \mathcal{N}(0, \mathcal{I}(\theta_0)^{-1})}(Z \in B) \right| \;\xrightarrow{P}\; 0.$

That is, the rescaled posterior converges in total variation to $\mathcal{N}(0, \mathcal{I}(\theta_0)^{-1})$ in probability under the true distribution $\Pr_{\theta_0}$.

The MAP estimate maximizes the log-posterior:

$\hat\theta_{\text{MAP}} \;=\; \arg\max_\theta \big[\ell(\theta) + \log\pi(\theta)\big].$

When $\pi(\theta) = \mathcal{N}(0, \tau^2)$ is Gaussian, $\log\pi(\theta) = -\theta^2/(2\tau^2) + \text{const}$, so $\hat\theta_{\text{MAP}}$ maximizes $\ell(\theta) - \theta^2/(2\tau^2)$ — exactly ridge regression with $\lambda = \sigma^2/\tau^2$. When $\pi(\theta) = \mathrm{Laplace}(0, b)$, $\log\pi(\theta) = -|\theta|/b + \text{const}$, so $\hat\theta_{\text{MAP}}$ maximizes $\ell(\theta) - |\theta|/b$ — exactly lasso with $\lambda = \sigma^2/b$. This discharges Topic 14 §14.11 Rem 9 (the MAP/regularized-MLE correspondence preview) and Topic 23 §23.7 Thm 5 (the MAP-penalization formal correspondence). The arc closes here.

Under BvM, $\hat\theta_{\text{MAP}}$ and $\hat\theta_{\text{MLE}}$ converge at rate $1/\sqrt n$ in TV — the regularization vanishes asymptotically, confirming that ridge and lasso are prior-informed procedures whose regularization is a commitment that data eventually overrides.

The same machinery that produces BvM yields a Gaussian approximation to the posterior at any finite $n$ — the Laplace approximation. Taylor-expanding $\log p(\theta, \mathbf{y})$ to second order at $\hat\theta_{\text{MAP}}$ and exponentiating gives $p(\theta \mid \mathbf{y}) \approx \mathcal{N}(\hat\theta_{\text{MAP}}, H^{-1})$ where $H = -\nabla^2 \log p(\theta, \mathbf{y})|_{\theta = \hat\theta_{\text{MAP}}}$. The same expansion approximates the marginal likelihood:

$\log m(\mathbf{y}) \;\approx\; \log p(\mathbf{y} \mid \hat\theta_{\text{MAP}}) + \log\pi(\hat\theta_{\text{MAP}}) + \tfrac{k}{2}\log(2\pi) - \tfrac{1}{2}\log|H|.$

This is the generalization of Topic 24 §24.4 Proof 2: BIC drops the prior term and approximates $\log |H| \approx k \log n$, yielding BIC = $-2\log p(\mathbf{y} \mid \hat\theta_{\text{MAP}}) + k \log n + O(1)$. The Laplace approximation is the full version — Topic 27 returns to it for Bayes-factor computation.