Continuous Distributions

This topic mirrors Discrete Distributions by design. Five core distributions (Uniform, Normal, Exponential, Gamma, Beta) are independently motivated with full derivations. Three derived distributions (Chi-squared, Student’s t, F) are defined by construction from the Normal and Chi-squared. The core five receive the same template as Topic 5: definition, then $E[X]$ and $\text{Var}(X)$ proofs, then MGF, then key properties, then an ML connection. The derived three share a single section — they are building blocks for hypothesis testing rather than independent modeling choices.

A random variable $X$ has the Uniform distribution on $[a, b]$, written $X \sim \text{Uniform}(a, b)$, if its PDF is

The CDF is $F(x) = \frac{x - a}{b - a}$ for $x \in [a, b]$, with $F(x) = 0$ for $x < a$ and $F(x) = 1$ for $x > b$.

If $X \sim \text{Uniform}(a, b)$, then:

1. $E[X] = \frac{a + b}{2}$
2. $\text{Var}(X) = \frac{(b - a)^2}{12}$
3. $M_X(t) = \frac{e^{tb} - e^{ta}}{t(b - a)}$ for $t \ne 0$, and $M_X(0) = 1$.

Unlike the other four core distributions in this topic, the Uniform is not a member of the exponential family. The reason: its support $[a, b]$ depends on the parameters. The exponential family requires that the support of the density be independent of the parameters — a condition the Uniform violates. Exponential Families makes this distinction precise.

If $U \sim \text{Uniform}(0, 1)$ and we define $X = F^{-1}(U) = -\frac{1}{\lambda} \ln(1 - U)$, then $X \sim \text{Exponential}(\lambda)$. This is the inverse CDF transform — a fundamental simulation technique.

Proof. $P(X \le x) = P\!\left(-\frac{1}{\lambda} \ln(1 - U) \le x\right) = P(U \le 1 - e^{-\lambda x}) = 1 - e^{-\lambda x}$, which is the Exponential CDF. Since $1 - U$ has the same distribution as $U$, we can simplify to $X = -\frac{1}{\lambda} \ln U$.

ML connection. Every random number generator starts with Uniform samples. The inverse CDF transform converts them to any target distribution — as long as $F^{-1}$ can be computed. For the Normal, $\Phi^{-1}$ has no closed form, so specialized algorithms (Box-Muller, Ziggurat) are used instead.

A random variable $X$ has the Normal distribution with mean $\mu$ and variance $\sigma^2$, written $X \sim N(\mu, \sigma^2)$, if its PDF is

The special case $Z \sim N(0, 1)$ is the standard Normal, with PDF $\varphi(z) = \frac{1}{\sqrt{2\pi}} e^{-z^2/2}$ and CDF $\Phi(z) = \int_{-\infty}^z \varphi(t) \, dt$.

If $X \sim N(\mu, \sigma^2)$, then $E[X] = \mu$ and $\text{Var}(X) = \sigma^2$.

If $X \sim N(\mu, \sigma^2)$, then $M_X(t) = \exp\!\left(\mu t + \frac{\sigma^2 t^2}{2}\right)$ for all $t \in \mathbb{R}$.

If $X_1 \sim N(\mu_1, \sigma_1^2)$ and $X_2 \sim N(\mu_2, \sigma_2^2)$ are independent, then

If $X \sim N(\mu, \sigma^2)$ and $a, b$ are constants with $a \ne 0$, then $aX + b \sim N(a\mu + b, \, a^2\sigma^2)$.

The Normal belongs to the exponential family with natural parameters $\eta_1 = \mu/\sigma^2$ and $\eta_2 = -1/(2\sigma^2)$. The sufficient statistics are $(X, X^2)$. When $\sigma^2$ is known, the single natural parameter $\eta = \mu/\sigma^2$ makes MLE particularly clean: the sufficient statistic is $\bar{X}$, and the MLE is $\hat{\mu} = \bar{X}$. Exponential Families develops this systematically.

Suppose we observe data $y_1, \ldots, y_n$ and model $y_i = f(x_i) + \varepsilon_i$ where $\varepsilon_i \sim N(0, \sigma^2)$ independently. The log-likelihood is:

Maximizing $\ell(\theta)$ with respect to $\theta$ (the parameters of $f$) is equivalent to minimizing $\sum_{i=1}^n (y_i - f(x_i))^2$. This is why least squares = MLE under Gaussian noise. The assumption of Normal errors is baked into every ordinary least squares regression — and when that assumption fails, the MLE changes (e.g., to Laplace errors for $\ell_1$ loss). See Topic 22 (Generalized Linear Models) for the general framework.

A random variable $X$ has the Exponential distribution with rate $\lambda > 0$, written $X \sim \text{Exponential}(\lambda)$, if its PDF is

The CDF is $F(x) = 1 - e^{-\lambda x}$ for $x \ge 0$. The mean is $1/\lambda$ and the median is $\ln 2 / \lambda$.

If $X \sim \text{Exponential}(\lambda)$, then $E[X] = \frac{1}{\lambda}$ and $\text{Var}(X) = \frac{1}{\lambda^2}$.

If $X \sim \text{Exponential}(\lambda)$, then $M_X(t) = \frac{\lambda}{\lambda - t}$ for $t < \lambda$.

A continuous random variable $X$ with support $[0, \infty)$ satisfies the memoryless property $P(X > s + t \mid X > s) = P(X > t)$ for all $s, t \ge 0$ if and only if $X \sim \text{Exponential}(\lambda)$ for some $\lambda > 0$.

The Exponential belongs to the exponential family with natural parameter $\eta = -\lambda$ and sufficient statistic $T(x) = x$. The log-partition function is $A(\eta) = -\ln(-\eta)$, and $E[X] = A'(\eta) = 1/\lambda$. Exponential Families shows how this structure enables conjugate Bayesian inference: the Gamma distribution is the conjugate prior for the Exponential rate parameter.

In survival analysis, the hazard function $h(t) = f(t) / (1 - F(t))$ measures the instantaneous failure rate at time $t$, given survival to $t$. For the Exponential:

The hazard is constant — the system does not age. This makes the Exponential a baseline model: real systems wear out ($h(t)$ increasing, Weibull distribution) or burn in ($h(t)$ decreasing). Departures from constant hazard motivate the Gamma and Weibull alternatives. In ML, Exponential survival models appear in customer churn prediction, equipment failure forecasting, and time-to-event modeling.

The Gamma function is defined for $\alpha > 0$ as

This is a convergent formalCalculus: for all $\alpha > 0$.

Key properties:

• Recursion: $\Gamma(\alpha + 1) = \alpha \, \Gamma(\alpha)$. This follows from formalCalculus: : with $u = t^{\alpha}$ and $dv = e^{-t} dt$, the boundary terms vanish and we get $\Gamma(\alpha + 1) = \alpha \int_0^{\infty} t^{\alpha-1} e^{-t} \, dt = \alpha \, \Gamma(\alpha)$.

• Factorial connection: Since $\Gamma(1) = \int_0^{\infty} e^{-t} \, dt = 1$, the recursion gives $\Gamma(n) = (n-1)!$ for every positive integer $n$. The Gamma function extends the factorial to non-integer arguments.

• Half-integer value: $\Gamma(1/2) = \sqrt{\pi}$. This follows from the substitution $t = u^2/2$, which converts $\Gamma(1/2)$ into the Gaussian integral $\int_{-\infty}^{\infty} e^{-u^2/2} \, du = \sqrt{2\pi}$.

A random variable $X$ has the Gamma distribution with shape $\alpha > 0$ and rate $\beta > 0$, written $X \sim \text{Gamma}(\alpha, \beta)$, if its PDF is

The normalization follows from the Gamma function: the substitution $u = \beta x$ transforms $\int_0^{\infty} \frac{\beta^\alpha}{\Gamma(\alpha)} x^{\alpha-1} e^{-\beta x} dx$ into $\frac{1}{\Gamma(\alpha)} \int_0^{\infty} u^{\alpha-1} e^{-u} du = 1$.

If $X \sim \text{Gamma}(\alpha, \beta)$, then $E[X] = \frac{\alpha}{\beta}$ and $\text{Var}(X) = \frac{\alpha}{\beta^2}$.

If $X \sim \text{Gamma}(\alpha, \beta)$, then $M_X(t) = \left(\frac{\beta}{\beta - t}\right)^{\alpha}$ for $t < \beta$.

If $X_1 \sim \text{Gamma}(\alpha_1, \beta)$ and $X_2 \sim \text{Gamma}(\alpha_2, \beta)$ are independent (with the same rate $\beta$), then

The Gamma belongs to the exponential family with natural parameters $\eta_1 = \alpha - 1$ and $\eta_2 = -\beta$, and sufficient statistics $(X, \ln X)$. When $\alpha$ is known, the single natural parameter is $\eta = -\beta$ with sufficient statistic $T(x) = x$, and the conjugate prior for $\beta$ is itself a Gamma. Exponential Families unifies this with the Exponential’s exponential family form.

Insurance claim amounts are positive and right-skewed — large claims are rare but impactful. The Gamma distribution is a natural model: the shape parameter $\alpha$ controls the skewness, and the rate $\beta$ controls the scale. In a Gamma GLM, we model claim amounts $Y_i \sim \text{Gamma}(\alpha, \beta_i)$ with $\ln E[Y_i] = \beta_0 + \beta_1 x_{i1} + \cdots$ (log link), allowing covariates (driver age, vehicle type, region) to affect the expected claim size while maintaining the Gamma’s positive support and right skew. See Topic 22 §22.6 (Gamma regression) for the full GLM framework and the worked insurance-amounts example.

A random variable $X$ has the Beta distribution with parameters $\alpha > 0$ and $\beta > 0$, written $X \sim \text{Beta}(\alpha, \beta)$, if its PDF is

The normalizing constant $B(\alpha, \beta) = \frac{\Gamma(\alpha)\,\Gamma(\beta)}{\Gamma(\alpha + \beta)}$ is the Beta function.

If $X \sim \text{Beta}(\alpha, \beta)$, then:

1. $E[X] = \frac{\alpha}{\alpha + \beta}$
2. $\text{Var}(X) = \frac{\alpha\beta}{(\alpha + \beta)^2(\alpha + \beta + 1)}$

If the prior on $\theta$ is $\text{Beta}(\alpha, \beta)$ and we observe $k$ successes in $n$ independent Bernoulli($\theta$) trials, then the posterior is

The Beta belongs to the exponential family with natural parameters $\eta_1 = \alpha - 1$ and $\eta_2 = \beta - 1$, and sufficient statistics $(\ln X, \ln(1-X))$. The log-partition function is $A(\eta_1, \eta_2) = \ln \Gamma(\eta_1 + 1) + \ln \Gamma(\eta_2 + 1) - \ln \Gamma(\eta_1 + \eta_2 + 2)$. Exponential Families connects this to the Beta-Bernoulli conjugacy via the general theory of conjugate priors.

A company runs an A/B test comparing two button designs. Design A has a $\text{Beta}(1, 1)$ prior (uniform — no prior information about the click rate). After 200 users see Design A, 34 click. The posterior is $\text{Beta}(1 + 34, 1 + 166) = \text{Beta}(35, 167)$.

The posterior mean click rate is $35/202 \approx 0.173$, with a 95% credible interval of approximately $[0.124, 0.231]$.

Design B, with prior $\text{Beta}(1, 1)$ and 42 clicks from 200 users, has posterior $\text{Beta}(43, 159)$, mean $43/202 \approx 0.213$.

The probability that Design B is better, $P(\theta_B > \theta_A \mid \text{data})$, can be computed by Monte Carlo: draw from each posterior, count how often $\theta_B > \theta_A$. This is Bayesian A/B testing — and it starts with the Beta-Bernoulli conjugate pair. Bayesian Foundations (Topic 25) develops the general framework, including the posterior predictive for the Beta-Binomial compound.

If $Z_1, \ldots, Z_k$ are independent $N(0, 1)$ random variables, then

has the Chi-squared distribution with $k$ degrees of freedom. Equivalently, $\chi^2(k) = \text{Gamma}(k/2, 1/2)$.

If $X \sim \chi^2(k)$, then $E[X] = k$ and $\text{Var}(X) = 2k$.

If $X \sim \chi^2(k)$, then $M_X(t) = (1 - 2t)^{-k/2}$ for $t < 1/2$.

If $X_1 \sim \chi^2(k_1)$ and $X_2 \sim \chi^2(k_2)$ are independent, then $X_1 + X_2 \sim \chi^2(k_1 + k_2)$.

If $Z \sim N(0,1)$ and $V \sim \chi^2(\nu)$ are independent, then

has Student’s t distribution with $\nu$ degrees of freedom. Its PDF is:

If $T \sim t(\nu)$, then:

1. $E[T] = 0$ for $\nu > 1$ (undefined for $\nu \le 1$)
2. $\text{Var}(T) = \frac{\nu}{\nu - 2}$ for $\nu > 2$ (infinite for $1 < \nu \le 2$, undefined for $\nu \le 1$)