Regularization & Penalized Estimation

The prostate-cancer dataset (HTF Table 3.3, used as the running benchmark throughout this topic) has 97 patients and 8 predictors. Two of those predictors — lcavol (log cancer volume) and lcp (log capsular penetration) — are tightly correlated ($\rho \approx 0.68$). If we artificially amplify that correlation by replacing lcp with $\rho \cdot \text{lcavol} + \sqrt{1 - \rho^2} \cdot \text{lcp}$ for $\rho \to 1$, the OLS coefficient vector $\hat{\boldsymbol\beta}$ does not just become unstable — it diverges. The figure below tracks $\|\hat{\boldsymbol\beta}\|_2$ as a function of the artificial correlation strength: by $\rho = 0.999$ the L²-norm has grown by three orders of magnitude. Individual coefficients on lcavol and lcp swing in opposite directions, fitting noise.

This is the §21.5 multicollinearity warning made concrete. The variance inflation factor (VIF) flags it; ridge regression fixes it. The cure works by adding $\lambda \|\boldsymbol\beta\|_2^2$ to the objective: the closed form $(\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{y}$ is well-defined as long as $\lambda > 0$, no matter how singular $\mathbf{X}^\top\mathbf{X}$ becomes.

Topic 22 §22.4 Rem 8 flagged separation as the second canonical pathology: when a hyperplane in predictor space perfectly classifies the binary response, the logistic log-likelihood is monotone-increasing along the direction $\boldsymbol\beta$, and IRLS iterates run away to $\pm\infty$. The brief promised the fix would arrive in Topic 23. Here it is.

The EXAMPLE_10_GLM_DATA preset (n = 120, p = 2) is a near-separation problem: the two classes are almost linearly separable, with only a handful of overlap points. Bare glmFit from Topic 22 reports converged = false after the maximum iteration count — by iteration 50, $\|\boldsymbol\beta^{(t)}\|$ has crossed 100 and is still growing. Adding a ridge penalty with $\lambda = 0.1$ changes the picture immediately: convergence in around 10 outer iterations to a finite, well-determined estimate. The §23.6 Ex 8 worked example develops this in detail, but the punchline lands here in §23.1: regularization is not just a remedy for the squared-error case — it restores well-posedness for IRLS too.

Penalized estimation is older than computer-era statistics. Tikhonov 1943 introduced $(\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}$ for integral-equation regularization in physics — 27 years before its statistical rediscovery. Hoerl & Kennard 1970 put ridge on the statistical map with the MSE-improvement theorem (§23.3 Thm 1) and the ridge-trace diagnostic. Tibshirani 1996 introduced the lasso, replacing $\|\cdot\|_2^2$ with $\|\cdot\|_1$ to obtain variable selection — a paper that has accumulated more than 40,000 citations. Firth 1993 had earlier introduced a Jeffreys-prior penalty as a separation cure for logistic regression; it is one of two practical answers to the §23.1 Ex 2 pathology, with ridge being the other (§23.6 Rem 11). The unifying framework — penalty as additive term on the negative log-likelihood — is what we develop next.

The ridge regression estimator at penalty parameter $\lambda \ge 0$ is

The closed form, derivable by setting the gradient $-\mathbf{X}^\top(\mathbf{y} - \mathbf{X}\boldsymbol\beta) + \lambda\boldsymbol\beta$ to zero, is

The matrix $\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I}$ is positive-definite for any $\lambda > 0$, so the estimator exists and is unique even when $\mathbf{X}$ is rank-deficient — the regression.ts ridgeFit implements exactly this closed form via Cholesky on the augmented Gram matrix. The $\tfrac{\lambda}{2}$ scaling on the penalty matches the lasso convention below ($\tfrac12$ on residuals throughout) and lines up with sklearn.linear_model.Ridge(alpha=λ) directly: alpha = λ gives identical fits.

The lasso estimator at penalty parameter $\lambda \ge 0$ is

The regression.ts lassoFit adopts this convention (no $1/n$ on residuals); to get the same fit out of sklearn.linear_model.Lasso(alpha) — whose objective is $(2n)^{-1}\|\mathbf{y} - \mathbf{X}\mathbf{w}\|_2^2 + \alpha\|\mathbf{w}\|_1$ — call it with alpha = λ/n. There is no closed form in general; coordinate descent (§23.4) is the standard solver. Unlike ridge, the lasso produces exact zeros in $\hat{\boldsymbol\beta}$ — the variable-selection property, established formally in Thm 2 below.

For a general log-likelihood $\ell(\boldsymbol\beta)$ and a convex penalty $P : \mathbb{R}^{p+1} \to \mathbb{R}_{\ge 0}$, the penalized maximum-likelihood estimator is

This is the unifying framework: ridge is the squared-error special case with $P = \|\cdot\|_2^2$; lasso is $P = \|\cdot\|_1$; penalized GLMs (§23.6) replace $-\ell$ with the GLM negative log-likelihood. The Bayesian reading via §23.7 Thm 5 is that $\lambda P(\boldsymbol\beta)$ is the negative log-prior, making penalized MLE identical to MAP estimation.

Setting $\nabla_{\boldsymbol\beta}[\|\mathbf{y} - \mathbf{X}\boldsymbol\beta\|_2^2 + \lambda\|\boldsymbol\beta\|_2^2] = \mathbf{0}$:

Rearranging,

The Hessian $\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I}$ is positive-definite for $\lambda > 0$, so the stationary point is the unique global minimum. Compare to OLS (Topic 21 §21.4 Thm 3): the only difference is the $\lambda\mathbf{I}$ term added to the diagonal. Geometrically, ridge replaces an ill-conditioned matrix with a well-conditioned one by inflating its smallest eigenvalues from near-zero up to at least $\lambda$.

Both ridge and lasso are not scale-equivariant: rescaling a predictor changes its penalty contribution. If $X_j$ is measured in centimeters, its coefficient is 100× the value it would be in meters; the lasso would shrink the centimeter-scale coefficient harder. Standard practice is to standardize predictors before fitting — center each $X_j$ at zero and scale to unit variance. The intercept is then absorbed by centering $y$, and the recovered coefficients are converted back to original units after the fit. The regression.ts implementation does this by default (standardize: true).

This standardization preserves the geometric clarity of the penalty (every coefficient sits on the same scale) at the cost of breaking equivariance under reparameterization — the price of a penalty that ignores the model’s sufficient statistic structure (Topic 16 §16.4). For genuine equivariance, one would need a scale-invariant penalty like $\sum_j |\beta_j| \cdot \text{sd}(X_j)$, which is rarely used in practice.

Standard convention — adopted by glmnet, sklearn, and regression.ts — is to leave the intercept unpenalized. The intuition: shifting all predictors by a constant should not change the model, but it would change the intercept; penalizing the intercept would couple the model to the location of the predictors. Mechanically, after centering $y$ and standardizing $\mathbf{X}$, the intercept becomes $\bar{y}$ and is recovered post-fit as $\hat\beta_0 = \bar{y} - \sum_j \hat\beta_j \cdot \text{mean}(X_j)$. The penalty applies only to the slope coefficients $j = 1, \dots, p$.

Write $\mathbf{X} = \mathbf{U}\mathbf{D}\mathbf{V}^\top$ with $\mathbf{D} = \text{diag}(d_1, \dots, d_{p+1})$ the singular values. The ridge fit can be written

OLS is the $\lambda = 0$ case: $\hat{\boldsymbol\beta}_{\text{OLS}} = \mathbf{V} \cdot \text{diag}(1/d_j) \cdot \mathbf{U}^\top\mathbf{y}$. Ridge replaces $1/d_j$ with $d_j/(d_j^2 + \lambda)$ — a smooth shrinkage that hits hardest where $d_j$ is small (the directions of low data variance). Lasso is non-uniform: its shrinkage acts in the original predictor coordinates, not the SVD coordinates, and produces exact zeros rather than smooth shrinkage. The two penalties differ both in what they shrink and how.

Let $\boldsymbol\beta \neq \mathbf{0}$ be the true coefficient vector under the fixed-design Normal linear model $\mathbf{y} = \mathbf{X}\boldsymbol\beta + \boldsymbol\varepsilon$ with $\boldsymbol\varepsilon \sim \mathcal{N}_n(\mathbf{0}, \sigma^2\mathbf{I})$ and $\mathbf{X}$ of full column rank. Then there exists $\lambda^\star > 0$ such that for every coordinate $j$,

The total MSE $\sum_j \text{MSE}(\hat\beta_j)$ is strictly smaller at $\lambda^\star$ than at $\lambda = 0$.

Let $\hat{\boldsymbol\beta} = \arg\min_{\boldsymbol\beta} \tfrac12\|\mathbf{y} - \mathbf{X}\boldsymbol\beta\|_2^2 + \lambda\|\boldsymbol\beta\|_1$. Define the partial score $s_j(\boldsymbol\beta) = \mathbf{x}_j^\top(\mathbf{y} - \mathbf{X}\boldsymbol\beta)$. The KKT optimality conditions are

For orthonormal designs $\mathbf{X}^\top\mathbf{X} = \mathbf{I}$, the solution has the closed form

where $S_\lambda(x) = \text{sign}(x)\,\max(|x| - \lambda, 0)$ is the soft-thresholding operator.

Plugging the prostate-cancer design into the SVD-coordinate MSE formula from Proof 1, we can compute the bias² and variance contributions to total MSE as a function of $\lambda$. At $\lambda = 0$ (OLS), bias² is zero and variance carries all the weight. As $\lambda$ grows, bias² rises quadratically while variance falls roughly proportionally to $\lambda \sum_j 1/d_j^4$. The total MSE traces a U-shape with minimum somewhere in the middle. For the prostate dataset, that minimum lies near $\lambda \approx 1$ — large enough to substantially reduce variance on the small singular values, small enough to keep bias modest.

Take $\mathbf{X} = \mathbf{I}_4$ (so $\mathbf{X}^\top\mathbf{X}/n = \mathbf{I}/4$, scaling $\lambda$ by $1/4$ for cleanness — for the canonical orthogonal-design corollary). Suppose $\mathbf{y} = (3, -2, 0.5, 0.1)^\top$. Then $\frac{1}{n}\mathbf{X}^\top\mathbf{y} = (0.75, -0.5, 0.125, 0.025)^\top$. Apply soft-thresholding at $\lambda = 0.2$:

The third and fourth coordinates land in the dead zone and are zeroed out — this is variable selection in action. As $\lambda$ grows, more coordinates fall into the dead zone; at $\lambda > 0.75$ the entire estimate is the zero vector. This is the geometric content of Thm 2’s orthogonal-design corollary made concrete.

Statistics courses introduce the bias-variance tradeoff with the equation $\text{MSE} = \text{bias}^2 + \text{variance}$ and an exhortation to “find the right balance.” Thm 1 sharpens this to a precise mathematical statement: for any design and any nonzero true $\boldsymbol\beta$, some positive $\lambda$ strictly beats OLS in total MSE — and component-wise, too. The tradeoff has a positive side: there is always free MSE on the table to be picked up by a small ridge penalty. The remaining question is how to find the right $\lambda$, which §23.8 answers.

The Hoerl–Kennard ridge MSE-improvement theorem has a 14-year-older sibling: Stein 1956’s inadmissibility result. For the multivariate Normal mean problem $\mathbf{X} \sim \mathcal{N}_p(\boldsymbol\mu, \mathbf{I})$ with $p \ge 3$, the natural estimator $\bar{\mathbf{X}}$ is inadmissible under squared-error loss — there exist estimators (the James–Stein shrinkage family) with strictly smaller risk for every $\boldsymbol\mu$. Ridge is a covariate-version of the same idea; the full Stein-identity proof is Topic 28 §28.5 Thm 2. The Lehmann–Romano admissibility framework (LEH2005 Ch. 5) shows that in the regression setting, ridge dominates OLS for specific $\lambda$ ranges depending on $\sigma^2$ and the design. The §23.7 Rem 17 returns to the Hodges-superefficiency tie-in flagged at Topic 14 §14.6 Rem 6.

Both ridge and lasso can be re-cast as constrained optimization problems via Lagrangian duality:

• Ridge: $\min \|\mathbf{y} - \mathbf{X}\boldsymbol\beta\|^2$ subject to $\|\boldsymbol\beta\|_2 \le t$ — minimize a quadratic on a Euclidean ball.
• Lasso: $\min \|\mathbf{y} - \mathbf{X}\boldsymbol\beta\|^2$ subject to $\|\boldsymbol\beta\|_1 \le t$ — minimize a quadratic on an L¹ ball (a “diamond” in 2D, a cross-polytope in higher dimensions).

The OLS solution is the center of the elliptical contours of the quadratic loss. Ridge’s solution is the closest point on the Euclidean ball — a smooth shrinkage. Lasso’s solution is the closest point on the diamond — and because the diamond has corners precisely where one or more coordinates are zero, the closest point typically lands on a corner. Sparsity is the geometric consequence of the L¹ ball’s vertices. This is the picture the LevelSetExplorer makes interactive.

The active set of a coefficient vector $\boldsymbol\beta$ is

For a lasso fit $\hat{\boldsymbol\beta}_{\text{lasso}}(\lambda)$, $\mathcal{A}(\hat{\boldsymbol\beta})$ is the selected variable set — the predictors the procedure judges worth keeping. As $\lambda$ shrinks, $|\mathcal{A}|$ grows; at $\lambda = 0$, $\mathcal{A} = \{1, \dots, p\}$ recovers OLS.

Let $f(\boldsymbol\beta) = g(\boldsymbol\beta) + \sum_{j=1}^{p+1} h_j(\beta_j)$ where $g : \mathbb{R}^{p+1} \to \mathbb{R}$ is convex and continuously differentiable and each $h_j$ is convex (possibly non-differentiable). Assume $f$ is bounded below and coercive. Starting from $\boldsymbol\beta^{(0)}$ and updating coordinate-by-coordinate as

the sequence $\{\boldsymbol\beta^{(t)}\}$ converges to a global minimizer of $f$. The lasso objective satisfies the hypothesis with $g(\boldsymbol\beta) = \frac{1}{2n}\|\mathbf{y} - \mathbf{X}\boldsymbol\beta\|_2^2$ and $h_j(\beta_j) = \lambda|\beta_j|$.

A two-coefficient lasso problem ($p = 2$) lets us visualize the iterate path directly. With $\mathbf{X}$ a small synthetic design and $\boldsymbol\beta^\star = (2, 1)$, the cold-start CD iterate trajectory makes a zigzag through coefficient space: each axis-aligned step minimizes one coordinate while holding the other fixed. The objective decreases monotonically (the right panel of the visualizer); the path converges to the global minimizer in around 47 coordinate updates (about 24 sweeps of 2 coordinates) at tolerance $10^{-6}$. With warm-start from the neighbouring $\lambda$, the same problem solves in 5–10 coordinate updates.

Before glmnet, the standard lasso solver was LARS (Least Angle Regression; Efron, Hastie, Johnstone & Tibshirani 2004). LARS computes the entire piecewise-linear regularization path exactly by tracking when each coordinate enters or leaves the active set as $\lambda$ decreases. It is elegant — the path is piecewise linear — but slower in practice than coordinate descent for large $p$, because each LARS step requires solving a linear system on the current active set. glmnet won out on engineering: cyclic CD is embarrassingly simple to vectorize, scales to $p$ in the millions, and supports any convex penalty (LARS is lasso-only). Topic 23 treats LARS as a historical pointer; for the algorithm itself, see HAS2009 §3.4.4.

A defining feature of glmnet is path computation with warm starts: instead of solving for one $\lambda$ at a time from scratch, compute the path over a 100-point log-grid in one pass, using the solution at $\lambda_k$ as the starting point for $\lambda_{k+1}$. Because consecutive $\lambda$ values give nearby solutions, each warm-started fit needs only a few sweeps to converge — the per-$\lambda$ cost drops by roughly an order of magnitude. The grid runs from $\lambda_{\max} = \max_j |\mathbf{x}_j^\top\mathbf{y}/n|$ (the smallest $\lambda$ for which $\hat{\boldsymbol\beta} = \mathbf{0}$) down to $\lambda_{\min} = \lambda_{\max} \cdot 10^{-4}$. The regularizationPath function in regression.ts implements this scheme.

For very large $p$ (gene-expression studies with $p = 10^5$ predictors), even cyclic CD becomes a bottleneck. Two acceleration techniques are widely deployed: strong screening rules (Tibshirani et al. 2012) eliminate predictors that the KKT conditions guarantee will be inactive at the next $\lambda$, before any coord-descent updates — a $10\times$–$100\times$ speedup. Proximal-gradient methods (ISTA, FISTA) take small gradient steps on the smooth part and apply soft-thresholding as the proximal operator; they generalize beyond separable non-smooth penalties (where coord descent fails) at the cost of slightly worse constants. regression.ts ships only the basic cyclic CD; production deployments should reach for glmnet or sklearn.linear_model.{Lasso, LassoCV} (also a CD implementation) with screening rules enabled.

For $\alpha \in [0, 1]$ and $\lambda \ge 0$, the elastic-net penalty is

and the elastic-net objective is

$\alpha = 1$ recovers lasso ($\tfrac12\|\cdot\|^2 + \lambda\|\boldsymbol\beta\|_1$); $\alpha = 0$ recovers ridge with penalty $(\lambda/2)\|\boldsymbol\beta\|^2$ — matching Def 1’s convention exactly. For $\alpha \in (0, 1)$, the penalty is strictly convex even when $\mathbf{X}^\top\mathbf{X}$ is rank-deficient — a property ridge has but lasso does not.

For any $\alpha \in [0, 1)$ and $\lambda > 0$, the elastic-net objective is strictly convex in $\boldsymbol\beta$, and the minimizer $\hat{\boldsymbol\beta}$ is unique — regardless of the rank of $\mathbf{X}$.

Brief argument: the Hessian of the smooth part is $\frac{1}{n}\mathbf{X}^\top\mathbf{X} + \lambda(1-\alpha)\mathbf{I}$, which is positive-definite whenever $\lambda(1-\alpha) > 0$. The L¹ term is convex (subdifferentiably), so the full objective is strictly convex. ZOU2005 Thm 1.

Take a synthetic problem with three pairs of perfectly correlated predictors plus four independent ones (the CORRELATED_FEATURES_DATA preset). The true coefficient is nonzero only at $X_1$ and $X_3$ — both members of correlated pairs. Lasso, run at the CV-optimal $\lambda$, picks one predictor from each correlated pair and zeros the other — and which one it picks varies arbitrarily across bootstrap samples. Elastic net with $\alpha = 0.5$ at the same effective penalty strength assigns nonzero weight to all four members of the correlated pairs, distributing weight across the group. The grouping effect: correlated predictors get similar coefficients, and the active set is more stable.

Zou & Hastie 2005 prove that for any two predictors $X_j$ and $X_k$ with $\mathbf{X}^\top\mathbf{x}_j = \mathbf{X}^\top\mathbf{x}_k$ (a strong correlation condition), the elastic-net coefficients satisfy

where $\rho$ is the correlation. As correlation $\rho \to 1$, the bound forces $\hat\beta_j \to \hat\beta_k$ — the coefficients are pulled together. For pure lasso ($\alpha = 1$), the bound is vacuous (denominator zero), which is why lasso lacks the grouping effect.

Elastic net introduces a second hyperparameter, $\alpha$, on top of $\lambda$. Most production workflows fix a small grid of $\alpha$ values ($\alpha \in \{0.1, 0.5, 0.9\}$, say), run cross-validation to pick $\lambda$ for each, and choose the combination with smallest CV error. The 2D grid is more expensive than the 1D lasso/ridge search but rarely prohibitively so. The figure shows the $(\alpha, \lambda)$ parameter plane on the prostate-cancer data, with CV-error contours and the optimal point marked.

For a GLM with log-likelihood $\ell(\boldsymbol\beta)$ (Topic 22 Def 3) and a convex penalty $P$, the penalized-GLM estimator is

For ridge, $P = \|\boldsymbol\beta\|_2^2$; for lasso, $P = \|\boldsymbol\beta\|_1$; for elastic net, $P_\alpha$ as in Def 5. The unpenalized intercept convention applies as usual.

Under a canonical link and a convex sub-differentiable penalty $P$, the penalized-IRLS update at iteration $t$ solves

where $\mathbf{W}^{(t)}$ and $\mathbf{z}^{(t)} = \boldsymbol\eta^{(t)} + \mathbf{W}^{(t)\,-1}(\mathbf{y} - \boldsymbol\mu^{(t)})$ are Topic 22 Thm 2’s IRLS weights and adjusted response. For $P = \|\cdot\|_2^2$ this is a ridge-WLS problem with closed form $(\mathbf{X}^\top\mathbf{W}\mathbf{X} + 2\lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{W}\mathbf{z}$. For $P = \|\cdot\|_1$ it is a weighted-lasso coord-descent inner pass.

Brief derivation: a second-order Taylor expansion of $-\ell$ around $\boldsymbol\beta^{(t)}$ produces the WLS quadratic surrogate exactly as in Topic 22 Proof 2; adding $2\lambda P(\boldsymbol\beta)$ gives the penalized surrogate. Minimizing the surrogate is the next IRLS step. Convergence proceeds as in Tseng 2001 with the surrogate playing the role of $f$.

The EXAMPLE_10_GLM_DATA preset (n = 120, p = 2) is the near-separation problem from §23.1 Ex 2. Topic 22’s bare glmFit reports converged = false after 50 iterations: $\|\boldsymbol\beta^{(t)}\|$ has crossed 100 and is still growing. Calling penalizedGLMFit with family: 'binomial', link: 'logit', penalty: 'ridge', lambda: 0.1 instead converges in around 10 outer IRLS iterations to a finite estimate $\hat{\boldsymbol\beta}$ with $\|\hat{\boldsymbol\beta}\|_\infty \approx 1$ — a small fraction of what the unpenalized fit would have produced. The ridge penalty makes the otherwise non-coercive objective coercive; the IRLS surrogate is then guaranteed to have a unique bounded minimizer at every step.

The Topic 22 EXAMPLE_6_GLM_DATA (credit-default logistic, n = 200, p = 2) becomes a high-dimensional problem when its predictors are expanded to degree 3 polynomials. The expanded design has p = 34 features (the CREDIT_DEFAULT_EXPANDED_DATA preset). Fitting a lasso-logistic at the CV-selected $\lambda$ produces a sparse $\hat{\boldsymbol\beta}$ with around 6–10 active coefficients, automatically selecting the polynomial terms that contribute predictive signal. Pure logistic regression on the expanded design overfits badly; lasso-logistic regularizes both selection and shrinkage in one pass.

Ridge is one cure for logistic-regression separation; Firth’s penalized likelihood (FIR1993) is the other. Firth replaces the log-likelihood with $\ell(\boldsymbol\beta) + \tfrac12 \log|\mathbf{I}(\boldsymbol\beta)|$ where $\mathbf{I}$ is the Fisher information — equivalent to a Jeffreys prior on $\boldsymbol\beta$. This penalty bounds the estimator without requiring a tuning parameter. In practice, Firth is the default in many epidemiology workflows where a single hyperparameter-free fix is preferred; ridge wins when cross-validation is feasible and the user wants explicit control over the bias-variance tradeoff. Topic 23 develops only the ridge cure; logistf in R implements Firth.

Topic 22 §22.9 Rem 19 flagged influential observations and outliers as a robustness concern, and pointed forward to two cures: M-estimation (defer to a future topic) and shrinkage. Ridge is the shrinkage answer: by penalizing $\|\boldsymbol\beta\|$, it limits the leverage that any single observation can exert on individual coefficients. The §22.9 Rem 19 promise is discharged here. Lasso gives a different kind of robustness: by zeroing weak signals, it makes the surviving coefficients depend on stronger structural patterns rather than noise-driven fluctuations.

A standard mistake: read off Wald confidence intervals from a penalized estimator using $(\mathbf{X}^\top\mathbf{W}\mathbf{X})^{-1}$ as if the penalty did not exist. This is wrong. The penalty introduces bias $\mathbb{E}[\hat{\boldsymbol\beta}] - \boldsymbol\beta \neq 0$, breaking the central condition for Wald inference. Profile-likelihood CIs on penalized estimators have similar issues. The correct procedure for inference under regularization is post-selection inference — selecting variables with lasso, then refitting unpenalized OLS on the selected variables (with caveats), or using the debiased lasso (Javanmard–Montanari 2014). Topic 23 stops at the warning; the methodology lives at formalml’s high-dimensional regression topic.

For a likelihood $L(\boldsymbol\beta)$ and a prior $\pi(\boldsymbol\beta)$, the maximum a-posteriori (MAP) estimate is the mode of the posterior:

The proportionality constant $p(\mathbf{y})$ drops out of the argmax. The MAP coincides with the posterior mean only for symmetric posteriors (e.g. Gaussian); otherwise the two diverge.

Let the prior density be $\pi(\boldsymbol\beta) = C \exp(-\lambda P(\boldsymbol\beta))$ for normalizing constant $C$ and convex $P$. Then

Embedded derivation. The posterior is $p(\boldsymbol\beta \mid \mathbf{y}) \propto L(\boldsymbol\beta) \pi(\boldsymbol\beta)$. Taking logs, $\log p(\boldsymbol\beta \mid \mathbf{y}) = \ell(\boldsymbol\beta) - \lambda P(\boldsymbol\beta) + \text{const}$. The MAP maximizes this expression — equivalently, minimizes $-\ell + \lambda P$, which is the penalized-MLE objective. ∎

Specializations. The Gaussian prior $\pi(\boldsymbol\beta) = \mathcal{N}_{p+1}(\mathbf{0}, \tau^2\mathbf{I})$ gives $\log\pi(\boldsymbol\beta) = -\|\boldsymbol\beta\|_2^2/(2\tau^2) + \text{const}$ — that is, ridge with $\lambda = 1/(2\tau^2)$. The Laplace prior $\pi(\boldsymbol\beta) = \prod_j \frac{1}{2b}\exp(-|\beta_j|/b)$ gives $\log\pi(\boldsymbol\beta) = -\|\boldsymbol\beta\|_1/b + \text{const}$ — that is, lasso with $\lambda = 1/b$. Discharges Topic 14 §14.11 Rem 9.