Exponential Families

A family of distributions indexed by a scalar parameter $\theta \in \Theta$ is a one-parameter exponential family if the PMF or PDF can be written as

where:

• $h(x) \geq 0$ is a base measure that depends only on the data, not on $\theta$
• $\eta(\theta)$ is the natural parameter (also called the canonical parameter), a function of $\theta$ alone
• $T(x)$ is the sufficient statistic, a function of the data alone
• $A(\theta) = \log \int h(x) \exp\!\bigl(\eta(\theta) \cdot T(x)\bigr) \, d\mu(x)$ is the log-partition function (also called the cumulant function), which ensures the density integrates to 1

The key structural feature is that $\theta$ and $x$ interact only through the product $\eta(\theta) \cdot T(x)$. Everything else separates cleanly: $h(x)$ absorbs the data-only terms, and $A(\theta)$ absorbs the parameter-only normalization.

A family of distributions indexed by a parameter vector $\boldsymbol{\theta} \in \Theta \subseteq \mathbb{R}^k$ is a $k$-parameter exponential family if the PMF or PDF can be written as

where $\boldsymbol{\eta}(\boldsymbol{\theta}) = (\eta_1(\boldsymbol{\theta}), \ldots, \eta_k(\boldsymbol{\theta}))$ is a vector of natural parameters, $\mathbf{T}(x) = (T_1(x), \ldots, T_k(x))$ is a vector of sufficient statistics, and the dot product $\boldsymbol{\eta} \cdot \mathbf{T} = \sum_{j=1}^k \eta_j T_j$ replaces the scalar product. The log-partition function $A(\boldsymbol{\theta})$ normalizes the distribution.

The dimension of the exponential family is $k$ — the number of natural parameters. The Bernoulli, Geometric, Poisson, and Exponential are one-dimensional ($k = 1$). The Normal, Gamma, and Beta are two-dimensional ($k = 2$). The Binomial and Negative Binomial are one-dimensional when $n$ (resp. $r$) is fixed.

A critical requirement — often stated implicitly — is that the support of $f(x \mid \theta)$ must be independent of $\theta$. The set of $x$ values where $f(x \mid \theta) > 0$ must be the same for all $\theta \in \Theta$.

Why? Because $h(x)$ is the only factor that can restrict the support (by being zero), and $h(x)$ does not depend on $\theta$. If the support depends on $\theta$, we would need an indicator function $\mathbf{1}_{\{x \in S(\theta)\}}$ that cannot be absorbed into the $\eta \cdot T$ structure.

This is exactly why the Uniform$(a, b)$ distribution fails to be an exponential family member, as we will see in Section 7.4.

The Bernoulli$(p)$ PMF is $f(x \mid p) = p^x (1-p)^{1-x}$ for $x \in \{0, 1\}$. Taking the logarithm:

Reading off the components:

• $h(x) = 1$
• $\eta(p) = \log\!\frac{p}{1-p}$ (the log-odds, also called the logit)
• $T(x) = x$
• $A(\eta) = \log(1 + e^\eta)$ (the softplus function)

The natural parameter $\eta$ ranges over all of $\mathbb{R}$: as $p \to 0$, $\eta \to -\infty$; as $p \to 1$, $\eta \to +\infty$. The inverse map is $p = e^\eta / (1 + e^\eta) = \sigma(\eta)$, the logistic function — and this is exactly why logistic regression uses the logit link.

The Binomial$(n, p)$ PMF is $f(k \mid p) = \binom{n}{k} p^k (1-p)^{n-k}$ for $k \in \{0, 1, \ldots, n\}$, with $n$ fixed. Taking the logarithm:

Reading off:

• $h(k) = \binom{n}{k}$
• $\eta(p) = \log\!\frac{p}{1-p}$ (same logit as Bernoulli)
• $T(k) = k$
• $A(\eta) = n \log(1 + e^\eta)$

The Binomial with fixed $n$ is a one-parameter exponential family in $p$. When both $n$ and $p$ are unknown, the family is no longer exponential in the standard sense because $n$ enters through $\binom{n}{k}$, which couples data and parameters outside the $\eta \cdot T$ structure.

The Geometric$(p)$ PMF is $f(k \mid p) = p(1-p)^{k-1}$ for $k \in \{1, 2, 3, \ldots\}$ (counting the trial on which the first success occurs). Taking the logarithm:

Reading off:

• $h(k) = 1$
• $\eta(p) = \log(1 - p)$
• $T(k) = k$
• $A(\eta) = \eta - \log(1 - e^\eta)$

Since $p \in (0, 1)$, we have $1 - p \in (0, 1)$, so $\eta = \log(1-p) < 0$. The natural parameter space is $\eta \in (-\infty, 0)$.

The Negative Binomial$(r, p)$ PMF is $f(k \mid p) = \binom{k-1}{r-1} p^r (1-p)^{k-r}$ for $k \in \{r, r+1, r+2, \ldots\}$, with $r$ fixed. Taking the logarithm:

Reading off:

• $h(k) = \binom{k-1}{r-1}$
• $\eta(p) = \log(1 - p)$
• $T(k) = k$
• $A(\eta) = r\eta - r\log(1 - e^\eta)$

The natural parameter space is $\eta < 0$, identical to the Geometric. This makes sense: the Geometric is the Negative Binomial with $r = 1$, and setting $r = 1$ in $A(\eta) = r\eta - r\log(1 - e^\eta)$ recovers the Geometric log-partition function exactly when $T(k) = k$.

The Poisson$(\lambda)$ PMF is $f(k \mid \lambda) = \frac{\lambda^k e^{-\lambda}}{k!}$ for $k \in \{0, 1, 2, \ldots\}$. Taking the logarithm:

Reading off:

• $h(k) = \frac{1}{k!}$
• $\eta(\lambda) = \log \lambda$
• $T(k) = k$
• $A(\eta) = e^\eta = \lambda$

The natural parameter $\eta = \log \lambda$ ranges over all of $\mathbb{R}$ (since $\lambda > 0$). The log-partition function $A(\eta) = e^\eta$ is the exponential function itself — making the Poisson the cleanest example for studying $A$ and its derivatives.

The Normal$(\mu, \sigma^2)$ PDF is $f(x \mid \mu, \sigma^2) = \frac{1}{\sigma\sqrt{2\pi}} \exp\!\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)$ for $x \in \mathbb{R}$. This is a two-parameter family. Expanding the exponent:

So the full log-density is:

Reading off:

• $h(x) = \frac{1}{\sqrt{2\pi}}$
• $\boldsymbol{\eta} = \left(\frac{\mu}{\sigma^2}, \; -\frac{1}{2\sigma^2}\right)$, so $\eta_1 = \mu/\sigma^2$ and $\eta_2 = -1/(2\sigma^2)$
• $\mathbf{T}(x) = (x, \; x^2)$
• $A(\boldsymbol{\eta}) = \frac{\mu^2}{2\sigma^2} + \log \sigma = -\frac{\eta_1^2}{4\eta_2} - \frac{1}{2}\log(-2\eta_2)$

The natural parameter space is $\eta_1 \in \mathbb{R}$, $\eta_2 < 0$. When $\sigma^2$ is known, the Normal reduces to a one-parameter exponential family with natural parameter $\eta = \mu/\sigma^2$, sufficient statistic $T(x) = x$, and log-partition function $A(\eta) = \sigma^2 \eta^2 / 2$.

The Exponential$(\lambda)$ PDF is $f(x \mid \lambda) = \lambda e^{-\lambda x}$ for $x \geq 0$. Taking the logarithm:

Reading off:

• $h(x) = 1$ for $x \geq 0$
• $\eta(\lambda) = -\lambda$
• $T(x) = x$
• $A(\eta) = -\log(-\eta)$

The natural parameter $\eta = -\lambda < 0$ (since $\lambda > 0$). Note the sign: the natural parameter is the negative of the rate. This is a one-parameter exponential family with natural parameter space $\eta \in (-\infty, 0)$.

The Gamma$(\alpha, \beta)$ PDF is $f(x \mid \alpha, \beta) = \frac{\beta^\alpha}{\Gamma(\alpha)} x^{\alpha - 1} e^{-\beta x}$ for $x > 0$. Taking the logarithm:

This is a two-parameter family. Reading off:

• $h(x) = 1$ for $x > 0$
• $\boldsymbol{\eta} = (\alpha - 1, \; -\beta)$, so $\eta_1 = \alpha - 1$ and $\eta_2 = -\beta$
• $\mathbf{T}(x) = (\log x, \; x)$
• $A(\boldsymbol{\eta}) = \log \Gamma(\eta_1 + 1) - (\eta_1 + 1) \log(-\eta_2)$

The natural parameter space is $\eta_1 > -1$, $\eta_2 < 0$. Setting $\alpha = 1$ (i.e., $\eta_1 = 0$) and keeping only the second component recovers the Exponential canonical form: $\eta = -\beta = -\lambda$, $T(x) = x$, $A(\eta) = -\log(-\eta)$. The Gamma genuinely subsumes the Exponential as a special case, and the canonical forms are consistent.

The Beta$(\alpha, \beta)$ PDF is $f(x \mid \alpha, \beta) = \frac{1}{B(\alpha, \beta)} x^{\alpha - 1} (1 - x)^{\beta - 1}$ for $x \in (0, 1)$. Taking the logarithm:

Reading off:

• $h(x) = 1$ for $x \in (0, 1)$
• $\boldsymbol{\eta} = (\alpha - 1, \; \beta - 1)$, so $\eta_1 = \alpha - 1$ and $\eta_2 = \beta - 1$
• $\mathbf{T}(x) = (\log x, \; \log(1 - x))$
• $A(\boldsymbol{\eta}) = \log B(\eta_1 + 1, \eta_2 + 1) = \log \Gamma(\eta_1 + 1) + \log \Gamma(\eta_2 + 1) - \log \Gamma(\eta_1 + \eta_2 + 2)$

The natural parameter space is $\eta_1 > -1$, $\eta_2 > -1$. The sufficient statistics $T_1(x) = \log x$ and $T_2(x) = \log(1 - x)$ are the log-data and the log-complement — both of which appear naturally in the Beta-Bernoulli conjugate updating that we will derive in Section 7.7.

The Student’s $t(\nu)$ PDF is proportional to $(1 + t^2/\nu)^{-(\nu+1)/2}$, and the $F(d_1, d_2)$ PDF involves a similar power-of-a-polynomial structure. In both cases, the parameter ($\nu$, or $d_1, d_2$) appears as an exponent on a function of $x$. There is no way to factor the log-density into a sum $\eta(\theta) \cdot T(x)$ plus separate functions of $x$ and $\theta$ alone.

Concretely, for the $t(\nu)$ distribution:

The term $-\frac{\nu + 1}{2} \log(1 + t^2/\nu)$ cannot be written as $\eta(\nu) \cdot T(t)$ because $\nu$ appears both as a multiplicative factor ($(\nu+1)/2$) and inside the argument of the logarithm ($t^2/\nu$). This double entanglement of data and parameter prevents the factorization.

The same issue affects the $F$ distribution. Both are derived from the Normal and Chi-squared via ratios and transformations that destroy the exponential family structure.

The Continuous Uniform$(a, b)$ has PDF

where $\mathbf{1}_{[a, b]}(x)$ is the indicator function that equals 1 when $a \leq x \leq b$ and 0 otherwise. The support of $f$ is the interval $[a, b]$, which depends on the parameters $a$ and $b$.

We can try to force this into exponential family form by taking the log:

The indicator $\log \mathbf{1}_{[a, b]}(x)$ equals 0 when $x \in [a, b]$ and $-\infty$ otherwise. This indicator depends on both $x$ and $\theta = (a, b)$, but it cannot be written as $\eta(\theta) \cdot T(x)$ — it is a hard constraint, not a smooth inner product.

The Discrete Uniform fails for exactly the same reason: its support $\{a, a+1, \ldots, b\}$ depends on the parameters. The Hypergeometric fails similarly: its support $\{\max(0, n - N + K), \ldots, \min(n, K)\}$ depends on $(N, K, n)$.

This is the content of Remark 7.1: the support of an exponential family density must be independent of the parameter. Parameter-dependent support is the most common reason a distribution fails the exponential family test.

Let $f(x \mid \eta) = h(x) \exp(\eta \cdot T(x) - A(\eta))$ be a one-parameter exponential family in natural parameterization. Then:

1. $A'(\eta) = E_\eta[T(X)]$
2. $A''(\eta) = \text{Var}_\eta(T(X))$

In words: the first derivative of the log-partition function gives the expected value of the sufficient statistic, and the second derivative gives its variance.

Since $A''(\eta) = \text{Var}_\eta(T(X)) \geq 0$ (variance is nonnegative), the log-partition function $A(\eta)$ is convex. Moreover, $A''(\eta) > 0$ whenever $T(X)$ is not degenerate (i.e., $T(X)$ takes more than one value with positive probability), so $A(\eta)$ is strictly convex for all standard exponential families.

This has three major consequences:

1. The log-likelihood is concave. For a sample $x_1, \ldots, x_n$, the log-likelihood in natural parameterization is $\ell(\eta) = \eta \cdot \sum_i T(x_i) - n \cdot A(\eta) + \text{const}$. Since $A(\eta)$ is convex, $-nA(\eta)$ is concave, and $\ell(\eta)$ is a linear function plus a concave function — hence concave. This guarantees that any local maximum of the log-likelihood is the global maximum.

2. The MLE is unique (when it exists). A strictly concave function has at most one maximum. Combined with existence conditions (the observed sufficient statistic lies in the interior of the convex hull of possible sufficient statistics), this gives both existence and uniqueness of the MLE.

3. The mean-to-natural parameter map is invertible. The function $\mu(\eta) = A'(\eta)$ is strictly increasing (since $A'' > 0$), so it has an inverse $\eta(\mu) = (A')^{-1}(\mu)$. This invertibility is what makes the dual parameterization — sometimes called the mean parameterization — well-defined.

In any exponential family $f(x \mid \theta) = h(x) \exp(\eta(\theta) \cdot T(x) - A(\theta))$, the statistic $T(X)$ is sufficient for $\theta$.

For an i.i.d. sample $X_1, \ldots, X_n$ from this family, the sum $\sum_{i=1}^n T(X_i)$ is sufficient for $\theta$.

Let $f(x \mid \eta) = h(x) \exp(\eta \cdot T(x) - A(\eta))$ be a one-parameter exponential family in natural parameterization. The conjugate prior for $\eta$ has the form

where $\chi_0$ and $\nu_0 > 0$ are hyperparameters.

Given an i.i.d. sample $x_1, \ldots, x_n$, the posterior is in the same family with updated hyperparameters:

where $\bar{T} = \frac{1}{n} \sum_{i=1}^n T(x_i)$ is the sample mean of the sufficient statistics.

For the Bernoulli with natural parameter $\eta = \text{logit}(p)$ and log-partition $A(\eta) = \log(1 + e^\eta)$, the conjugate prior is

Given $n$ Bernoulli observations with $k$ successes (so $\bar{T} = k/n$), the posterior is:

In the general framework: $\chi_0 = \alpha_0 - 1$ (pseudo-successes minus 1), $\nu_0 = \alpha_0 + \beta_0 - 2$ (pseudo-sample-size minus 2), and the update rule gives $\chi_0' = \chi_0 + k$, $\nu_0' = \nu_0 + n$. The Beta parameterization absorbs the offsets.

The posterior mean is $\frac{\alpha_0 + k}{\alpha_0 + \beta_0 + n}$ — a weighted average of the prior mean $\frac{\alpha_0}{\alpha_0 + \beta_0}$ and the sample proportion $\frac{k}{n}$, with weights proportional to the pseudo-sample size and the real sample size.

For the Poisson with natural parameter $\eta = \log \lambda$ and log-partition $A(\eta) = e^\eta = \lambda$, the conjugate prior for $\lambda$ is

Given $n$ Poisson observations with total count $S = \sum_{i=1}^n k_i$, the posterior is:

The update rule is clean: $\alpha_0$ accumulates the total count, $\beta_0$ accumulates the number of observations. The posterior mean is $\frac{\alpha_0 + S}{\beta_0 + n}$ — a weighted average of the prior mean $\alpha_0/\beta_0$ and the sample mean $S/n$.

For the Normal with known variance $\sigma^2$, the natural parameter is $\eta = \mu/\sigma^2$ and the log-partition is $A(\eta) = \sigma^2 \eta^2/2$. The conjugate prior for $\mu$ is

Given $n$ Normal observations with sample mean $\bar{x}$, the posterior is:

where the posterior variance and mean are given by the precision-weighted average:

The posterior precision (inverse variance) is the sum of the prior precision and the data precision. The posterior mean is the precision-weighted average of the prior mean and the sample mean. As $n \to \infty$, the data precision dominates and $\mu_n \to \bar{x}$ — the posterior concentrates on the MLE, regardless of the prior. This is a concrete instance of posterior consistency.

Let $X_1, \ldots, X_n$ be an i.i.d. sample from a one-parameter exponential family $f(x \mid \eta) = h(x) \exp(\eta \cdot T(x) - A(\eta))$. The maximum likelihood estimator $\hat{\eta}$ satisfies

where $\bar{T} = \frac{1}{n} \sum_{i=1}^n T(X_i)$ is the sample mean of the sufficient statistics. When $\bar{T}$ lies in the interior of the range of $A'$, this equation has a unique solution.