Random Variables & Distribution Functions

Let $(\Omega, \mathcal{F}, P)$ be a probability space. A random variable is a function $X : \Omega \to \mathbb{R}$ that is measurable with respect to $\mathcal{F}$ — meaning that for every Borel set $B \in \mathcal{B}(\mathbb{R})$,

$X^{-1}(B) = \{\omega \in \Omega : X(\omega) \in B\} \in \mathcal{F}.$

In practice, the condition we check most often is the CDF condition: for every $x \in \mathbb{R}$,

$\{\omega \in \Omega : X(\omega) \leq x\} \in \mathcal{F}.$

This is sufficient because the half-lines $(-\infty, x]$ generate the Borel $\sigma$-algebra (see Topic 1, Example 4).

The measurability condition ensures that we can assign a probability to every statement of the form "$X \leq x$" or "$X \in B$". Without it, $P(X \leq x)$ is undefined — the set $\{X \leq x\}$ might not be in $\mathcal{F}$, and $P$ only speaks to events in $\mathcal{F}$. For finite or countable sample spaces with the discrete $\sigma$-algebra (the power set), every function is measurable — so the condition is automatic. It only becomes restrictive for uncountable $\Omega$, and even then, every function you’ll encounter in practice is measurable.

Why this matters for ML: When you write ”$X \sim \mathcal{N}(\mu, \sigma^2)$,” you are implicitly working with the Borel $\sigma$-algebra on $\mathbb{R}$. The measurability requirement — that a random variable must pull Borel sets back to events in your $\sigma$-algebra — is what makes statements like "$P(X \leq t)$" well-defined. We formalized $\sigma$-algebras in Topic 1.

A random vector $\mathbf{X} = (X_1, \ldots, X_d) : \Omega \to \mathbb{R}^d$ is a function where each component $X_i$ is a random variable.

Roll one die. $\Omega = \{1,2,3,4,5,6\}$, $\mathcal{F} = \mathcal{P}(\Omega)$. The identity function $X(\omega) = \omega$ is a random variable. So is $Y(\omega) = \omega^2$, or $Z(\omega) = \mathbf{1}\{\omega \text{ is even}\}$ (the indicator of “even”). Each creates a different mapping from outcomes to numbers.

$\Omega = \{(i,j) : i,j \in \{1,\ldots,6\}\}$, $S((i,j)) = i + j$. The pre-images are:

• $S^{-1}(\{2\}) = \{(1,1)\}$, so $P(S = 2) = 1/36$
• $S^{-1}(\{7\}) = \{(1,6),(2,5),(3,4),(4,3),(5,2),(6,1)\}$, so $P(S = 7) = 6/36 = 1/6$
• $S^{-1}(\{12\}) = \{(6,6)\}$, so $P(S = 12) = 1/36$

The random variable $S$ induces a probability distribution on $\{2,3,\ldots,12\}$ via the pre-image and $P$.

A random variable $X$ is discrete if it takes values in a finite or countably infinite set $\{x_1, x_2, x_3, \ldots\}$.

The probability mass function (PMF) of a discrete random variable $X$ is the function $p_X : \mathbb{R} \to [0,1]$ defined by

$p_X(x) = P(X = x) = P(\{\omega \in \Omega : X(\omega) = x\}).$

The PMF satisfies two properties:

1. $p_X(x) \geq 0$ for all $x$ (from Axiom 1 of the Kolmogorov axioms, Topic 1)
2. $\sum_{x} p_X(x) = 1$ (from Axiom 2, summing over all values in the support)

The support of a discrete random variable $X$ is the set of values where the PMF is positive:

$\text{supp}(X) = \{x \in \mathbb{R} : p_X(x) > 0\}.$

A Bernoulli random variable models a single trial with two outcomes. $X \in \{0, 1\}$ with $P(X = 1) = p$ and $P(X = 0) = 1 - p$ for some $p \in [0,1]$. Written compactly:

$p_X(x) = p^x (1-p)^{1-x}, \quad x \in \{0, 1\}.$

We write $X \sim \text{Bernoulli}(p)$. This is the building block of classification: "$Y \sim \text{Bernoulli}(\sigma(w^\top x))$" is logistic regression.

The sum of $n$ independent $\text{Bernoulli}(p)$ trials. $X \in \{0, 1, \ldots, n\}$ with

$p_X(k) = \binom{n}{k} p^k (1-p)^{n-k}, \quad k = 0, 1, \ldots, n.$

We write $X \sim \text{Binomial}(n, p)$. The binomial coefficient $\binom{n}{k}$ counts the number of ways to arrange $k$ successes among $n$ trials — this is the combinatorics from Topic 1 (§6) in action.

A random variable $X$ is continuous if there exists a non-negative function $f_X : \mathbb{R} \to [0, \infty)$ such that for every interval $[a, b]$,

$P(a \leq X \leq b) = \int_a^b f_X(x) \, dx.$

The function $f_X$ is called the probability density function (PDF). It satisfies:

1. $f_X(x) \geq 0$ for all $x$
2. $\int_{-\infty}^{\infty} f_X(x) \, dx = 1$

These mirror the PMF conditions — replace sums with integrals.

This is perhaps the most common confusion in probability. For a continuous random variable:

• $f_X(x)$ is a density, not a probability. It can be greater than 1.
• $f_X(x) \, dx$ is (informally) the probability that $X$ falls in a tiny interval around $x$.
• Only integrals of the density give probabilities.
• $P(X = x) = 0$ for every individual $x$. Probabilities of intervals are what matter.

$X \sim \text{Uniform}(0,1)$ has density $f_X(x) = 1$ for $x \in [0,1]$ and $f_X(x) = 0$ otherwise. Then $P(0.2 \leq X \leq 0.5) = \int_{0.2}^{0.5} 1 \, dx = 0.3$. The probability equals the length of the interval — probability is area under the density curve.

$Z \sim \mathcal{N}(0, 1)$ has density

$f_Z(z) = \frac{1}{\sqrt{2\pi}} e^{-z^2/2}.$

This density is positive everywhere on $\mathbb{R}$ and integrates to 1 (a non-trivial fact requiring the Gaussian integral $\int_{-\infty}^{\infty} e^{-z^2/2} \, dz = \sqrt{2\pi}$).

Note that $f_Z(0) = 1/\sqrt{2\pi} \approx 0.399$ — this is less than 1 in this case, but for a $\text{Uniform}(0, 0.5)$ distribution, $f_X(x) = 2$ on $[0, 0.5]$, which exceeds 1. The density is not bounded by 1.

The cumulative distribution function (CDF) of a random variable $X$ is

$F_X(x) = P(X \leq x), \quad x \in \mathbb{R}.$

For any random variable $X$, the CDF $F_X$ satisfies:

1. Non-decreasing: If $a \leq b$, then $F_X(a) \leq F_X(b)$.
2. Right-continuous: $\lim_{h \to 0^+} F_X(x + h) = F_X(x)$ for all $x$.
3. Limits at infinity: $\lim_{x \to -\infty} F_X(x) = 0$ and $\lim_{x \to \infty} F_X(x) = 1$.

The CDF gives us a direct route to probabilities of intervals:

$P(a < X \leq b) = F_X(b) - F_X(a)$

This follows from $\{X \leq b\} = \{X \leq a\} \cup \{a < X \leq b\}$ (disjoint).

For continuous random variables, $P(X = x) = 0$, so strict/non-strict inequalities don’t matter: $P(a \leq X \leq b) = P(a < X < b) = F_X(b) - F_X(a)$.

For discrete random variables, the CDF is a step function — it jumps at each value in the support. The jump at $x$ has height $P(X = x) = p_X(x)$.

For discrete random variables $X$ and $Y$, the joint PMF is

$p_{X,Y}(x, y) = P(X = x \text{ and } Y = y) = P(X = x, Y = y).$

It satisfies $p_{X,Y}(x,y) \geq 0$ and $\sum_x \sum_y p_{X,Y}(x,y) = 1$.

For continuous random variables $X$ and $Y$, the joint PDF is a function $f_{X,Y} : \mathbb{R}^2 \to [0, \infty)$ such that

$P((X,Y) \in A) = \iint_A f_{X,Y}(x, y) \, dx \, dy$

for every (measurable) set $A \subseteq \mathbb{R}^2$. It satisfies $f_{X,Y}(x,y) \geq 0$ and $\iint f_{X,Y}(x,y)\,dx\,dy = 1$.

The joint CDF is

$F_{X,Y}(x, y) = P(X \leq x, Y \leq y).$

This works for any pair of random variables.

Roll two fair dice. Let $X$ = first die, $Y$ = second die. The joint PMF is $p_{X,Y}(i, j) = 1/36$ for $i, j \in \{1,\ldots,6\}$. The marginal of $X$ is $p_X(i) = \sum_{j=1}^6 1/36 = 6/36 = 1/6$ — as expected.

$(X, Y) \sim \mathcal{N}(\mathbf{0}, \Sigma)$ where $\Sigma = \begin{pmatrix} 1 & \rho \\ \rho & 1 \end{pmatrix}$. The joint PDF is

$f_{X,Y}(x,y) = \frac{1}{2\pi\sqrt{1-\rho^2}} \exp\left(-\frac{x^2 - 2\rho xy + y^2}{2(1-\rho^2)}\right).$

The parameter $\rho \in (-1, 1)$ controls the correlation — the linear dependence between $X$ and $Y$. When $\rho = 0$, the joint factors as $f_X(x) \cdot f_Y(y)$, and $X$ and $Y$ are independent.

For discrete $X, Y$, the marginal PMF of $X$ is

$p_X(x) = \sum_{y} p_{X,Y}(x, y).$

For continuous $X, Y$ with joint PDF $f_{X,Y}$, the marginal PDF of $X$ is

$f_X(x) = \int_{-\infty}^{\infty} f_{X,Y}(x, y) \, dy.$

The marginal distributions are uniquely determined by the joint distribution — sum over (discrete) or integrate out (continuous) the other variables.

Knowing $f_X$ and $f_Y$ individually is not enough to reconstruct $f_{X,Y}$. The joint encodes the dependence structure between $X$ and $Y$ — information lost when we marginalize. This is why “correlation does not imply causation” has a precise mathematical meaning: the marginals constrain the joint, but don’t determine it. Specifying the dependence structure is the job of copulas — see Multivariate Distributions.

For discrete $X, Y$ with $p_Y(y) > 0$,

$p_{X|Y}(x \mid y) = \frac{p_{X,Y}(x, y)}{p_Y(y)}.$

This is exactly the definition of conditional probability (Topic 2, Definition 1) applied to the events $\{X = x\}$ and $\{Y = y\}$: $P(X = x \mid Y = y) = P(X = x, Y = y) / P(Y = y)$.

For continuous $X, Y$ with $f_Y(y) > 0$,

$f_{X|Y}(x \mid y) = \frac{f_{X,Y}(x, y)}{f_Y(y)}.$

This requires more care than the discrete case — since $P(Y = y) = 0$ for continuous $Y$, we can’t directly condition on $\{Y = y\}$. The conditional PDF is defined as the limit $f_{X|Y}(x \mid y) = \lim_{\epsilon \to 0} P(X \leq x \mid y - \epsilon < Y \leq y + \epsilon)$. The formula above is the result of that limit.

The joint density factors as

$f_{X,Y}(x, y) = f_{X|Y}(x \mid y) \cdot f_Y(y) = f_{Y|X}(y \mid x) \cdot f_X(x).$

This is the density version of the multiplication rule (Topic 2, Theorem 1). Rearranging gives the density version of Bayes’ theorem:

$f_{X|Y}(x \mid y) = \frac{f_{Y|X}(y \mid x) \cdot f_X(x)}{f_Y(y)}.$

If $(X, Y)$ are jointly normal with correlation $\rho$, then $X \mid Y = y$ is normal:

$X \mid Y = y \sim \mathcal{N}\left(\mu_X + \rho \frac{\sigma_X}{\sigma_Y}(y - \mu_Y), \; \sigma_X^2(1 - \rho^2)\right).$

Two remarkable facts:

1. The conditional mean is linear in $y$: this is why linear regression works for jointly normal data.
2. The conditional variance $\sigma_X^2(1 - \rho^2)$ does not depend on $y$ — this is the homoscedasticity assumption.

Random variables $X$ and $Y$ are independent (written $X \perp\!\!\!\perp Y$) if for every pair of Borel sets $A, B \subseteq \mathbb{R}$,

$P(X \in A, Y \in B) = P(X \in A) \cdot P(Y \in B).$

Equivalently, any of the following:

• CDF factorization: $F_{X,Y}(x, y) = F_X(x) \cdot F_Y(y)$ for all $x, y$.
• PMF factorization (discrete): $p_{X,Y}(x, y) = p_X(x) \cdot p_Y(y)$ for all $x, y$.
• PDF factorization (continuous): $f_{X,Y}(x, y) = f_X(x) \cdot f_Y(y)$ for all $x, y$.

$X \perp\!\!\!\perp Y$ if and only if $f_{X,Y}(x,y) = f_X(x) f_Y(y)$ (continuous case) or $p_{X,Y}(x,y) = p_X(x) p_Y(y)$ (discrete case) for all $x, y$.

If $X \perp\!\!\!\perp Y$, then $f_{X|Y}(x \mid y) = f_X(x)$ — knowing $Y$ tells you nothing new about $X$.

Let $(X, Y)$ be bivariate normal. They are independent if and only if $\rho = 0$. When $\rho \neq 0$, knowing $Y$ shifts the conditional mean of $X$ (by Example 9), so $X$ and $Y$ are dependent. The contour plots make this visible: $\rho = 0$ gives circular contours (factored density), $\rho \neq 0$ gives elliptical contours (non-factored).

Let $X$ be a random variable with known CDF $F_X$ and let $Y = g(X)$. Then

$F_Y(y) = P(Y \leq y) = P(g(X) \leq y).$

To find $F_Y$, solve the inequality $g(X) \leq y$ for $X$, then express the result in terms of $F_X$.

Let $Y = aX + b$ with $a > 0$. Then

$F_Y(y) = P(aX + b \leq y) = P(X \leq (y-b)/a) = F_X\left(\frac{y-b}{a}\right).$

Differentiating: $f_Y(y) = \frac{1}{a} f_X\left(\frac{y-b}{a}\right)$. If $a < 0$, the inequality flips and $f_Y(y) = \frac{1}{|a|} f_X\left(\frac{y-b}{a}\right)$.

Application: If $X \sim \mathcal{N}(\mu, \sigma^2)$, then $Z = (X - \mu)/\sigma \sim \mathcal{N}(0,1)$. Standardization is a linear transformation.

Let $X$ be a continuous random variable with PDF $f_X$. Let $Y = g(X)$ where $g$ is strictly monotonic and differentiable with inverse $g^{-1}$. Then

$f_Y(y) = f_X(g^{-1}(y)) \cdot |(g^{-1})'(y)|.$

The factor $|(g^{-1})'(y)|$ is the Jacobian — it accounts for how $g$ stretches or compresses intervals.

Let $X \sim \mathcal{N}(\mu, \sigma^2)$ and $Y = e^X$ (i.e., $X = \log Y$). Then $g^{-1}(y) = \log y$ and $(g^{-1})'(y) = 1/y$. So

$f_Y(y) = f_X(\log y) \cdot \frac{1}{y} = \frac{1}{y\sigma\sqrt{2\pi}} \exp\left(-\frac{(\log y - \mu)^2}{2\sigma^2}\right), \quad y > 0.$

This is the lognormal distribution. It arises when the logarithm of a quantity is normally distributed — common for financial returns, biological measurements, and the distribution of weights in neural networks.

For a vector transformation $\mathbf{Y} = g(\mathbf{X})$ where $g : \mathbb{R}^d \to \mathbb{R}^d$ is a diffeomorphism, the change of variables formula becomes

$f_{\mathbf{Y}}(\mathbf{y}) = f_{\mathbf{X}}(g^{-1}(\mathbf{y})) \cdot |\det J_{g^{-1}}(\mathbf{y})|$

where $J_{g^{-1}}$ is the Jacobian matrix. This is the formula behind formalML: Normalizing Flows — a class of generative models that learn invertible transformations of simple distributions. Each flow layer applies this change of variables formula.

The expectation (or expected value, or mean) of a random variable $X$ is

$E[X] = \sum_x x \cdot p_X(x) \quad \text{(discrete)}$

$E[X] = \int_{-\infty}^{\infty} x \cdot f_X(x) \, dx \quad \text{(continuous)}$

provided the sum or integral converges absolutely.

Let $U \sim \text{Uniform}(0,1)$ and let $F$ be any CDF with quantile function $F^{-1}(u) = \inf\{x : F(x) \geq u\}$. Then $X = F^{-1}(U)$ has CDF $F$.