Kernel Density Estimation

Figure 1 takes a single Normal$(0, 1)$ sample of size $n = 200$ and plots three histograms (bin widths $b \in \{0.1, 0.4, 1.0\}$) alongside three Gaussian-kernel KDEs (bandwidths $h \in \{0.08, 0.3, 0.8\}$). At small $b$ or small $h$, both estimators are spiky and variance-dominated; at large $b$ or large $h$, both over-smooth and bias-dominate. The KDEs remove one obvious source of arbitrariness — bin edges — but the bandwidth problem remains, just in a smoother form.

Histograms have two genuine advantages KDEs lack: they are exactly piecewise-constant (so integrals over bin intervals are trivial) and they are visually honest about the underlying empirical counts. KDEs are smoother and easier to differentiate further (for mode-finding or density-gradient methods), but the smoothness is imposed, not inherent. A reader who wants to know “how many data points fell in $[a, b]$?” should look at a histogram; a reader who wants a differentiable density estimate should look at a KDE.

Topic 7 built density families (Normal, Gamma, Beta) and fit parametric $\theta$ from data. That program is efficient when the family is right and biased when it’s wrong. KDE drops the family assumption and estimates $f$ directly from the sample — a distribution-free procedure in the sense of Topic 29 §29.1 Rem 1. The price is a slower convergence rate: we’ll see in §30.6 that AMISE decays at $n^{-4/5}$, slower than the parametric $n^{-1}$, and that this is the price of distribution-freeness.

A kernel is a function $K : \mathbb{R} \to \mathbb{R}_{\ge 0}$ satisfying

Nonnegativity and unit mass make $K$ a density on $\mathbb{R}$; symmetry makes it a density centered at $0$. We additionally require finite variance $\mu_2(K) := \int u^2 K(u)\,du < \infty$ and finite roughness $R(K) := \int K(u)^2\,du < \infty$, both used throughout §30.4–§30.6.

For bandwidth $h > 0$, the scaled kernel is

The scaling preserves the density property — $\int K_h(u)\,du = 1$ for any $h > 0$ — and turns $h$ into a width parameter: small $h$ makes $K_h$ tall and narrow (close to a Dirac delta); large $h$ makes it short and wide.

Given an iid sample $X_1, \dots, X_n$ from an unknown density $f$, the kernel density estimator with kernel $K$ and bandwidth $h$ is

Both forms are in active use. The first makes it visually obvious that $\hat f_h$ is an average of $n$ kernel-shaped bumps centered at the data. The second groups the bandwidth normalization into a single $nh$ factor, which will be convenient when we compute bias and variance in §30.4.

Under the definitions above, $\hat f_h$ is a bona-fide probability density:

Stated; the proof is elementary. Nonnegativity is immediate from $K \ge 0$. Integration reduces to $\frac{1}{n}\sum_i \int K_h(x - X_i)\,dx = \frac{1}{n}\sum_i 1 = 1$ by the scaled-kernel normalization.

Take $n = 2$ observations $X_1 = 0$, $X_2 = 1$ and Gaussian kernel $K(u) = \varphi(u) = (2\pi)^{-1/2} e^{-u^2/2}$. At bandwidth $h = 0.5$:

where we used $\varphi(1) = \varphi(-1) \approx 0.2420$. The KDE at the midpoint $x = 0.5$ receives equal contribution from both observations and is therefore twice $\varphi(1)/h$. This is the T30.8 test pin — a two-point sanity check on the kdeEvaluate implementation.

Writing $\mathbb{P}_n = n^{-1} \sum_i \delta_{X_i}$ for the empirical distribution (a probability measure that places mass $1/n$ at each sample point), we have $\hat f_h = K_h \ast \mathbb{P}_n$. KDE is the convolution of a smooth kernel with the empirical distribution — equivalently, it is the empirical density against a smoothed observation model. This formulation is convenient in formalml where the convolution structure generalizes to multivariate bandwidths and non-iid samples.

Symmetry $K(-u) = K(u)$ makes the first-moment condition $\int u K(u)\,du = 0$ automatic, which simplifies the bias calculation in §30.4 (the $h f'(x)$ Taylor term drops out). Nonnegativity guarantees $\hat f_h \ge 0$, so the estimator is always a valid density. A fourth-order or higher-order kernel with $\mu_2(K) = 0$ can achieve faster bias rates (down to $O(h^4)$) but requires $K$ to take negative values, and then $\hat f_h$ may be negative in places — a practical complication we’ll mention in §30.6 Rem 13 and defer to Serfling 1980 §1.11.

The five standard symmetric kernels, with their second moments and roughness:

Stated. All values are elementary integrals; Example 3 verifies Epanechnikov explicitly, and the kernelProperties export in nonparametric.ts §8 encodes all five (T30.5–T30.6).

For $K_E(u) = \frac{3}{4}(1 - u^2)\mathbf{1}\{|u| \le 1\}$:

The values $\mu_2(K_E) = 1/5$ and $R(K_E) = 3/5$ are load-bearing for Theorem 5’s Epanechnikov optimality proof in §30.6.

The efficiencies in Theorem 2 all lie in $[0.93, 1.00]$. Substituting into the AMISE-optimal formula from §30.6, switching from Epanechnikov to Gaussian changes the optimal AMISE value by less than 5%, and the optimal bandwidths differ only by a multiplicative constant. The practical consequence: once we’ve committed to using a kernel with finite variance and roughness, the bandwidth choice does almost all the work. §30.8 is the real bandwidth chapter; §30.3 is a tour of an almost-equivalence class.

Let $X_1, \dots, X_n$ be iid with density $f$ having bounded, continuous second derivative $f''$ in a neighborhood of $x$ with $f(x) > 0$. Let $K$ be a symmetric kernel satisfying the moment conditions $\int K = 1$, $\int u K = 0$, $\mu_2(K) < \infty$, $R(K) < \infty$. As $n \to \infty$, $h \to 0$, and $nh \to \infty$:

Stated here; the derivation is Step (i) and Step (ii) of the §30.6 Theorem 4 proof. Both calculations are a second-order Taylor expansion of $f$ around $x$ combined with the kernel moment conditions.

For $f = \varphi$ (standard Normal), $x = 0$, Gaussian kernel ($\mu_2 = 1$, $R = (2\sqrt\pi)^{-1}$), $h = 0.5$, $n = 200$:

The bias is roughly 13% of $f(0)$; the standard deviation is roughly 8%. The leading-order formulas predict this at large $n$; at $n = 200$ there is finite-$n$ slippage, but the orders of magnitude are right.

The Taylor expansion of $f(x - hu)$ around $x$ has a linear $-hu f'(x)$ term. When integrated against the symmetric kernel $K$, the linear term is $-h f'(x) \int u K(u)\,du = 0$. Symmetry of $K$ is doing work here — without it we’d pick up an $O(h)$ bias term, and the AMISE-optimal bandwidth in §30.6 would have a very different rate. This is also why asymmetric kernels (one-sided boundary kernels in §30.5) need separate treatment.

Theorem 3 says bias is $O(h^2)$ and variance is $O((nh)^{-1})$ under standard regularity — the two rates are intrinsic to the kernel-smoothing program and cannot be both improved without changing the estimator class. A fourth-order kernel with $\mu_2(K) = 0$ can achieve $O(h^4)$ bias but potentially negative $\hat f_h$ (§30.2 Rem 4); local polynomial regression (Fan–Gijbels 1996; §30.5 Rem 9) trades one estimator structure for another. In the plain-KDE world with a symmetric nonnegative kernel, Rosenblatt 1956 is the ceiling.

For $f = \varphi$, Gaussian kernel, $n = 200$: the closed-form AMISE-optimal bandwidth (from §30.6 Thm 4) is $h^\ast \approx 0.3705$. At that $h$, AMISE $\approx 0.01025$. The exact MISE computed by numerical integration over the same grid is $\approx 0.01058$ — about 3% higher than AMISE. The gap is the $o(h^2)$ and $o(1/(nh))$ terms we dropped; at $n = 200$ they are nonzero but small, and they vanish as $n \to \infty$.

Beta$(2, 5)$ has density $f(x) = 30 x (1 - x)^4$ on $[0, 1]$ and $f(x) = 0$ outside. A naïve Gaussian KDE on a Beta$(2, 5)$ sample places nonzero density at negative $x$ (where the true density is zero) because the Gaussian kernel has infinite support. The leakage is not a small correction: at $n = 500$ with Silverman bandwidth $h \approx 0.09$, the mass below $x = 0$ is approximately $0.02$, or 2% of the total probability mass. Figure 4 shows the naïve and reflection-corrected KDEs side by side; the reflection trick (augment the sample with $\{-X_i\}$ and renormalize) recovers nearly all the leaked mass.

For a density with bounded support $[a, b]$ and a kernel with support $[-r, r]$ (or all of $\mathbb{R}$ with rapid decay), the KDE near $x = a$ averages over $X_i$ values in a window that extends below $a$. But no $X_i$ values are below $a$ — the sample is entirely on $[a, b]$. The KDE therefore under-estimates $f(a)$ and smooths probability mass into the forbidden region $(-\infty, a)$. At $x$ exactly at the boundary, the bias is $O(h)$ — worse than the interior $O(h^2)$ — and the AMISE analysis of §30.6 does not apply. Any production density estimator for bounded-support data needs a boundary correction.

Three standard boundary corrections: Jones 1993 reflection (simplest; used by BoundaryBiasDemo); Müller 1991 boundary kernels (asymmetric kernels within one bandwidth of the boundary); Fan–Gijbels 1996 local polynomial regression, which automatically adapts at the boundary as a byproduct of fitting a local polynomial rather than a local constant. Local polynomial is the modern standard and generalizes cleanly to the regression setting $E[Y \mid X]$; the full treatment is deferred to formalml local-regression. Topic 30 keeps things simple with reflection.

Let $X_1, \dots, X_n$ be iid with density $f$ having bounded, continuous second derivative $f''$. Let $K$ be a symmetric kernel with $\int K = 1$, $\int u K = 0$, $\mu_2(K) \in (0, \infty)$, $R(K) < \infty$. As $n \to \infty$, $h \to 0$, $nh \to \infty$,

where $R(f'') := \int f''(x)^2\,dx$. The AMISE-optimal bandwidth is

with matching minimum AMISE

Consider the class of symmetric kernels $K : \mathbb{R} \to \mathbb{R}_{\ge 0}$ satisfying $\int K = 1$ and $\mu_2(K) = \sigma_0^2$ for a fixed positive constant $\sigma_0^2$. The kernel minimizing $R(K) = \int K^2$ subject to these constraints is the Epanechnikov kernel

(By rescaling $h$, the conclusion holds for any fixed $\sigma_0^2 > 0$; the statement with $\sigma_0^2 = 1/5$ fixes the conventional support $[-1, 1]$.)

With $f = \varphi$ and Gaussian kernel, the constants are $R(K) = 1/(2\sqrt\pi) \approx 0.2821$, $\mu_2(K) = 1$, and $R(\varphi'') = 3/(8\sqrt\pi) \approx 0.2116$. Substituting into Theorem 4:

This matches the T30.12 test pin — the oracle-bandwidth computation at $n = 100$ for $f = N(0, 1)$. At $n = 1000$ the bandwidth shrinks by the factor $10^{-1/5} \approx 0.631$ (T30.13), and the optimal AMISE shrinks by $10^{-4/5} \approx 0.158$ (T30.14).

A parametric MLE of a Normal mean has MSE $O(n^{-1})$ — the square of the standard $\sqrt n$ convergence. The KDE’s AMISE rate $O(n^{-4/5})$ is slower because we’re estimating an entire function, not a finite-dimensional parameter. The rate is a direct consequence of Theorem 4: the bias–variance trade-off forces $h \to 0$ at rate $n^{-1/5}$, not the $n^{-1/2}$ that parametric estimators enjoy. In high dimensions the rate gets worse still — Scott 2015 Ch. 6 derives the multivariate rate $O(n^{-4/(4+d)})$, which we’ll pointer-forward in §30.10.

The AMISE-optimal $h^\ast$ is the first-order condition on a quadratic-in-$h^2$ loss function. The structural parallel to the Bahadur representation of the sample quantile (Topic 29 §29.6) is that both reduce nonparametric inference to a balance between a variance term that shrinks with more data and a bias term controlled by an asymptotic parameter: bandwidth $h$ in the KDE case, remainder-term order in the Bahadur case. Every Track 8 result is a version of this trade-off — bootstrap, empirical processes, and density-ratio estimation each have their own “h”: a resample count, a bracketing radius, a regularization strength.

Theorem 5 minimizes $R(K)$ subject to $\mu_2(K) = \sigma_0^2$. Substituting into $\mathrm{AMISE}^\ast$, the kernel enters through $\{R(K)^4 \mu_2(K)^2\}^{1/5}$. For the five standard kernels, this combined factor ranges from $0.98^{1/5}$ (uniform) to $1.00$ (Epanechnikov) — a spread of under 2%. In log-space, switching kernels moves AMISE$^\ast$ by less than 0.01. Bandwidth choice, by contrast, can move AMISE by an order of magnitude. The practical advice: pick any standard kernel (Gaussian for simplicity, Epanechnikov for compact support), and spend your energy on bandwidth selection (§30.8).

A fourth-order kernel is a kernel with $\mu_2(K) = 0$; the bias in Theorem 3 then vanishes to leading order, and the next term is $O(h^4)$ involving $f^{(4)}$. The AMISE rate improves from $n^{-4/5}$ to $n^{-8/9}$ — a meaningful speedup at large $n$. The catch: a fourth-order kernel must take negative values (to integrate $u^2 K = 0$ with $\int K = 1$), so $\hat f_h$ can be negative. In practice this is usually patched by truncation, but Topic 30 follows the standard monograph convention of restricting to second-order kernels. Serfling 1980 §1.11 and Wand–Jones 1995 §2.8 cover higher-order kernels at length; formalml may return to them in the context of density-ratio estimation.

Under the hypotheses of Theorem 4 (symmetric kernel satisfying $\int K = 1$, $\int u K = 0$, $\mu_2(K) < \infty$, $R(K) < \infty$; $f$ continuous at $x$ with $f(x) > 0$), and under the bandwidth regime $h \to 0$, $nh \to \infty$,