Conditional Probability & Bayes' Theorem

Conditioning restricts the world

The conditional probability of $A$ given $B$ is

$$P(A \mid B) = \frac{P(A \cap B)}{P(B)}, \qquad P(B) > 0 .$$

Read it as: among the outcomes where $B$ happened, what fraction also have $A$? Conditioning throws away every outcome inconsistent with what you learned and renormalises what is left.

Two consequences are used constantly:

• Multiplication rule: $P(A \cap B) = P(A \mid B)\,P(B)$.
• Law of total probability: if $B_1, \dots, B_n$ partition the sample space, $P(A) = \sum_i P(A \mid B_i)\,P(B_i).$

Bayes' theorem

Combining the two rules gives Bayes' theorem, which reverses the direction of conditioning:

$$P(H \mid E) = \frac{P(E \mid H)\,P(H)}{P(E)} = \frac{P(E \mid H)\,P(H)}{P(E \mid H)P(H) + P(E \mid H^c)P(H^c)} .$$

$P(H)$ is the prior, $P(E \mid H)$ the likelihood, and $P(H \mid E)$ the posterior. In odds form, which is often faster in an interview:

$$\underbrace{\frac{P(H \mid E)}{P(H^c \mid E)}}_{\text{posterior odds}} = \underbrace{\frac{P(H)}{P(H^c)}}_{\text{prior odds}} \times \underbrace{\frac{P(E \mid H)}{P(E \mid H^c)}}_{\text{likelihood ratio}} .$$

How information arrives matters

The same words can describe different conditioning events. "At least one child is a boy" conditions on the event $\{BB, BG, GB\}$ and gives $P(BB) = \tfrac13$. "The older child is a boy" conditions on $\{BB, BG\}$ and gives $\tfrac12$.

In Monty Hall the host's choice is not random — he never reveals the prize — so his action carries information. Whenever a puzzle involves someone choosing what to reveal, model their selection rule explicitly.