Conditional Probability & Independence

Let $(\Omega, \mathcal{F}, P)$ be a probability space and let $B \in \mathcal{F}$ with $P(B) > 0$. The conditional probability of $A$ given $B$ is

$P(A \mid B) = \frac{P(A \cap B)}{P(B)}.$

Roll a fair die, $\Omega = \{1,2,3,4,5,6\}$ with $P(\{\omega\}) = 1/6$. Let $A = \{2,4,6\}$ (even) and $B = \{4,5,6\}$ (at least 4).

$P(A \mid B) = \frac{P(A \cap B)}{P(B)} = \frac{P(\{4,6\})}{P(\{4,5,6\})} = \frac{2/6}{3/6} = \frac{2}{3}.$

Knowing the roll was at least 4, the probability of even jumps from $1/2$ to $2/3$. Information changed the probability.

For fixed $B$ with $P(B) > 0$, the function $A \mapsto P(A \mid B)$ satisfies the Kolmogorov axioms on $(\Omega, \mathcal{F})$:

1. $P(A \mid B) = P(A \cap B)/P(B) \geq 0$ (non-negativity).
2. $P(\Omega \mid B) = P(B)/P(B) = 1$ (normalization).
3. Countable additivity follows from that of $P$.

So conditioning on $B$ gives us a new probability space — the same $\Omega$ and $\mathcal{F}$, but a different measure. Every theorem we proved in Topic 1 (complement rule, inclusion-exclusion, union bound, continuity) holds for conditional probabilities too.

For $B \in \mathcal{F}$ with $P(B) > 0$, define $P_B : \mathcal{F} \to [0,1]$ by $P_B(A) = P(A \mid B)$. Then $(\Omega, \mathcal{F}, P_B)$ is a probability space.

For events $A, B \in \mathcal{F}$ with $P(B) > 0$,

$P(A \cap B) = P(A \mid B) \cdot P(B).$

By symmetry (when $P(A) > 0$): $P(A \cap B) = P(B \mid A) \cdot P(A)$.

For events $A_1, \ldots, A_n \in \mathcal{F}$ with $P(A_1 \cap \cdots \cap A_{n-1}) > 0$,

$P(A_1 \cap A_2 \cap \cdots \cap A_n) = P(A_1) \cdot P(A_2 \mid A_1) \cdot P(A_3 \mid A_1 \cap A_2) \cdots P(A_n \mid A_1 \cap \cdots \cap A_{n-1}).$

Draw 3 cards from a standard 52-card deck without replacement. What is the probability all three are hearts?

$P(H_1 \cap H_2 \cap H_3) = P(H_1) \cdot P(H_2 \mid H_1) \cdot P(H_3 \mid H_1 \cap H_2)$

$= \frac{13}{52} \cdot \frac{12}{51} \cdot \frac{11}{50} = \frac{1716}{132600} \approx 0.01294.$

Each factor reflects the updated state of the deck: after drawing one heart, 12 of 51 remaining cards are hearts. The chain rule captures this sequential conditioning perfectly.

Let $B_1, B_2, \ldots, B_k$ be a partition of $\Omega$: the $B_i$ are pairwise disjoint and $\bigcup_{i=1}^k B_i = \Omega$. If $P(B_i) > 0$ for all $i$, then for any event $A \in \mathcal{F}$,

$P(A) = \sum_{i=1}^k P(A \mid B_i) \cdot P(B_i).$

A disease has 2% prevalence. A test has sensitivity $P(+ \mid D) = 0.95$ and specificity $P(- \mid D^c) = 0.90$. What is $P(+)$, the probability of testing positive?

Partition: $\{D, D^c\}$. By total probability:

$P(+) = P(+ \mid D) \cdot P(D) + P(+ \mid D^c) \cdot P(D^c)$

$= 0.95 \times 0.02 + 0.10 \times 0.98 = 0.019 + 0.098 = 0.117.$

About 11.7% of people test positive — and the vast majority are false positives, because the disease is rare. This is where Bayes’ theorem comes in.

Let $A, B \in \mathcal{F}$ with $P(B) > 0$ and $P(A) > 0$. Then

$P(A \mid B) = \frac{P(B \mid A) \cdot P(A)}{P(B)}.$

More generally, if $A_1, \ldots, A_k$ partition $\Omega$ with $P(A_i) > 0$ for all $i$, then

$P(A_i \mid B) = \frac{P(B \mid A_i) \cdot P(A_i)}{\sum_{j=1}^k P(B \mid A_j) \cdot P(A_j)}.$

Humans are notoriously bad at Bayesian reasoning. We tend to overweight the likelihood $P(B \mid A)$ and ignore the prior $P(A)$ — the base rate. In medical testing, this means patients (and sometimes doctors) confuse “the test is 99% accurate” with “I’m 99% likely to have the disease.” As Example 4 shows, these can be wildly different when the disease is rare.

A disease has prevalence $P(D) = 0.01$ (1% of the population). A diagnostic test has:

• Sensitivity: $P(+ \mid D) = 0.99$ (catches 99% of true cases)
• Specificity: $P(- \mid D^c) = 0.95$ (correctly clears 95% of healthy people), so $P(+ \mid D^c) = 0.05$.

Question: If you test positive, what is the probability you actually have the disease?

By Bayes:

$P(D \mid +) = \frac{P(+ \mid D) \cdot P(D)}{P(+)} = \frac{0.99 \times 0.01}{0.99 \times 0.01 + 0.05 \times 0.99}$

$= \frac{0.0099}{0.0099 + 0.0495} = \frac{0.0099}{0.0594} \approx 0.167.$

Despite a “99% accurate” test, a positive result means only about a 17% chance of disease. The base rate (1% prevalence) dominates.

Natural frequency framing. Consider 10,000 people:

• 100 have the disease → 99 test positive (true positives), 1 tests negative (false negative)
• 9,900 are healthy → 495 test positive (false positives), 9,405 test negative (true negatives)
• Total positives: 99 + 495 = 594. Of those, 99 actually have the disease: $99/594 \approx 16.7\%$.

Events $A$ and $B$ are independent if

$P(A \cap B) = P(A) \cdot P(B).$

Equivalently (when $P(B) > 0$): $P(A \mid B) = P(A)$ — conditioning on $B$ doesn’t change the probability of $A$.

Flip two fair coins. Let $A$ = “first coin is heads” and $B$ = “second coin is heads.” Then $\Omega = \{HH, HT, TH, TT\}$ with $P(\{\omega\}) = 1/4$.

$P(A) = 1/2, \quad P(B) = 1/2, \quad P(A \cap B) = P(\{HH\}) = 1/4 = P(A) \cdot P(B).$

Independent. This matches our intuition: the second coin doesn’t “know” what the first coin did.

Events $A_1, \ldots, A_n$ are (mutually) independent if for every subset $\{i_1, \ldots, i_k\} \subseteq \{1, \ldots, n\}$ with $k \geq 2$,

$P(A_{i_1} \cap \cdots \cap A_{i_k}) = P(A_{i_1}) \cdots P(A_{i_k}).$

This requires $2^n - n - 1$ conditions — not just the $\binom{n}{2}$ pairwise ones. For three events, we need four conditions: the three pairwise ones plus $P(A \cap B \cap C) = P(A) \cdot P(B) \cdot P(C)$.

If $A$ and $B$ are independent, then $A$ and $B^c$ are also independent (and $A^c$ and $B^c$, etc.).

Events $A_1, \ldots, A_n$ are pairwise independent if $P(A_i \cap A_j) = P(A_i) \cdot P(A_j)$ for all $i \neq j$.

Flip two fair coins. Define:

• $A$ = “first coin is heads”: $P(A) = 1/2$
• $B$ = “second coin is heads”: $P(B) = 1/2$
• $C$ = “the two coins show different faces” $= \{HT, TH\}$: $P(C) = 1/2$

Check pairwise independence:

• $P(A \cap B) = P(\{HH\}) = 1/4 = P(A) \cdot P(B)$ ✓
• $P(A \cap C) = P(\{HT\}) = 1/4 = P(A) \cdot P(C)$ ✓
• $P(B \cap C) = P(\{TH\}) = 1/4 = P(B) \cdot P(C)$ ✓

But $A \cap B \cap C = \emptyset$ (both heads means same face, so $C$ fails), so:

$P(A \cap B \cap C) = 0 \neq 1/8 = P(A) \cdot P(B) \cdot P(C).$

The three events are pairwise independent but not mutually independent.

This is not a pathological edge case. In machine learning, pairwise decorrelation (e.g., PCA) does not guarantee full independence — a fact that higher-order methods like ICA (independent component analysis) exploit. The difference between pairwise and mutual independence is the difference between matching second-order statistics and matching the full joint distribution.

Events $A$ and $B$ are conditionally independent given $C$ (written $A \perp\!\!\!\perp B \mid C$) if

$P(A \cap B \mid C) = P(A \mid C) \cdot P(B \mid C).$

Equivalently (when $P(B \cap C) > 0$): $P(A \mid B \cap C) = P(A \mid C)$ — once you know $C$, learning $B$ tells you nothing new about $A$.

There exist events $A$, $B$, $C$ such that $A \perp\!\!\!\perp B$ (marginally independent) but $A$ and $B$ are not conditionally independent given $C$. And vice versa.

Two independent causes $A$ and $B$ can produce an effect $C$. Marginally, $A \perp\!\!\!\perp B$. But conditioning on $C$ (the effect having occurred) makes $A$ and $B$ dependent: if we know the effect happened and $A$ didn’t cause it, $B$ becomes more likely. This is called explaining away.

Concrete example: A fire alarm ($C$) can be triggered by a fire ($A$) or by cooking smoke ($B$). Fire and cooking smoke are independent events. But given that the alarm is ringing, learning there’s no fire makes cooking smoke more probable.

Conditional independence is the language of graphical models:

• In a Bayesian network (directed graph), $A \perp\!\!\!\perp B \mid C$ is encoded by the graph structure (d-separation).
• The naive Bayes classifier assumes all features are conditionally independent given the class label: $P(X_1, \ldots, X_d \mid Y) = \prod_{j=1}^d P(X_j \mid Y)$.
• In hidden Markov models, observations are conditionally independent given the hidden state.

These assumptions are rarely exactly true, but they make inference tractable. The art of probabilistic modeling is choosing which conditional independencies to assume.

A game show has three doors. Behind one is a car; behind the others, goats. You pick a door (say Door 1). The host, who knows what’s behind the doors, opens another door (say Door 3) to reveal a goat. Should you switch to Door 2?

Let $C_i$ = “car is behind door $i$.” Prior: $P(C_1) = P(C_2) = P(C_3) = 1/3$.

Let $H_3$ = “host opens door 3.” The host must open a goat door $\neq$ your choice:

• $P(H_3 \mid C_1) = 1/2$ (car is behind your door, host picks randomly from 2 and 3)
• $P(H_3 \mid C_2) = 1$ (car is behind door 2, host must open door 3)
• $P(H_3 \mid C_3) = 0$ (host can’t open the car door)

By Bayes:

$P(C_2 \mid H_3) = \frac{P(H_3 \mid C_2) \cdot P(C_2)}{P(H_3)}$

$= \frac{1 \times 1/3}{1/2 \times 1/3 + 1 \times 1/3 + 0 \times 1/3} = \frac{1/3}{1/2} = \frac{2}{3}.$

Switching wins with probability 2/3. Staying wins with probability 1/3. The host’s action provides information that shifts the posterior — a direct application of Bayes’ theorem.