You almost always observe an effect and want to infer its cause: you see a positive test and ask about the disease, read a word and ask whether the email is spam. Conditional probability runs the other way, from cause to effect. Bayes' theorem reverses it.
The Reverend Thomas Bayes worked out the rule in the 1740s but never published it. After his death his friend Richard Price edited the manuscript and read it to the Royal Society on 23 December 1763. A decade later Pierre-Simon Laplace rediscovered it independently and gave it the general form we use, in his 1812 Theorie analytique des probabilites. For two centuries it was a curiosity; today it is the backbone of spam filters, medical diagnosis, and machine learning.
Reversing the Conditional
Bayes' theorem comes straight from the definition of conditional probability. Since P(H and E) can be written two ways, P(E | H)P(H) or P(H | E)P(E), setting them equal and solving gives the rule. Its four pieces each have a name worth memorizing.
Read it as a sentence: the posterior (belief in the hypothesis after seeing the evidence) equals the likelihood (how well the hypothesis predicts the evidence) times the prior (belief before), divided by the evidence (how likely that data was overall). The shape posterior ∝ likelihood × prior is the whole of Bayesian reasoning in five symbols.
The Law of Total Probability
That denominator P(E) is usually not handed to you. You compute it with the Law of Total Probability: split the world into cases that cover every possibility, find the chance of E within each, and add them up, weighted by how likely each case is.
For two machines, P(defective) = P(def | A)P(A) + P(def | B)P(B) = 0.012 + 0.020 = 0.032. Now Bayes can run: given a defect, P(B | defective) = 0.020 / 0.032 = 0.625. Machine B makes fewer parts but is so much more defect-prone that it is the likelier source of any given defect, the prior 0.40 updates upward to 0.625.
A handy shortcut: P(E) is always the sum over every branch of likelihood × prior. So Bayes' theorem is really "this branch divided by the total of all branches". If you can draw the tree, you can read the posterior straight off it without memorizing anything.
Prior to Posterior: a Worked Update
Bayes shines when a hypothesis is rare. Suppose a disease has 0.5% prevalence, and a test has 99% sensitivity (catches the sick) and 95% specificity (clears the healthy). You test positive. What now?
The prevalence is the prior, P(disease) = 0.005. The evidence, by total probability, is P(positive) = 0.005 × 0.99 + 0.995 × 0.05 = 0.0547. Bayes gives the posterior:
P(disease | positive) = (0.005 × 0.99) / 0.0547 ≈ 0.090
One positive test raises the chance of disease from 0.5% to about 9%, an 18-fold jump, yet still far from certain. The base rate does not vanish just because a test fired; Bayes keeps it in view, which is exactly what intuition forgets (the base-rate trap of the Conditional Probability & Independence chapter).
If those probabilities feel slippery, do what the Conditional Probability & Independence chapter did and switch to natural frequencies, the same math in whole people instead of decimals:
Picture 10,000 people. At 0.5% prevalence, 50 are sick and 9,950 are healthy. Of the 50 sick, the 99%-sensitive test flags about 50 (true positives). Of the 9,950 healthy, the 5% false-positive rate flags about 498 (false positives). So roughly 548 people test positive, but only 50 are actually sick, which means a positive result carries a 50 / 548 ≈ 9% chance of disease. The false positives from a huge healthy group swamp the true positives from a tiny sick one, and counting people makes that obvious in a way the formula hides.
Evidence Accumulates
Bayesian updating is sequential, and that is its superpower. Take a second, independent positive test. The posterior you just computed becomes the new prior, and you run Bayes again.
Two positives push the posterior to about 66%; a third crosses 97%. No single test is conclusive, but independent evidence compounds, each result reweighting the last. This is the engine of all Bayesian learning: data arrives, the posterior shifts, and that posterior is the starting point for whatever comes next.
Bayes in Machine Learning & AI
Bayes' theorem is not just a probability exercise; it is one of the load-bearing ideas of machine learning. Whole families of models are Bayes' theorem wearing different clothes.
| Bayesian idea | In ML / AI it appears as | Concrete example |
|---|---|---|
| posterior ∝ likelihood × prior | The naive Bayes classifier | P(spam | words) from per-word likelihoods |
| Prior distribution | Regularization and prior belief | a prior on weights pulls a model toward simpler fits |
| Posterior over parameters | Bayesian inference (MAP, MCMC) | a full distribution of plausible models, not one point |
| Sequential updating | Online learning, Kalman filters | each new observation refines the estimate |
| Posterior comparison | Bayesian A/B testing | P(variant B is better | the data) directly |
Naive Bayes is the textbook case: Bayes' theorem plus the conditional-independence assumption from the Conditional Probability & Independence chapter, multiplying one word likelihood after another into P(class | features). More broadly, Bayesian methods treat a model's parameters as a posterior distribution rather than a single guess, which is how modern systems express uncertainty, regularize with priors, and update online as data streams in. Where a classical model says "the answer is 0.84", a Bayesian model says "here is the whole range of what is plausible, and how sure I am", a distinction that matters enormously when an AI's confidence has real consequences.
Update beliefs in Python
The companion notebook builds the two-machine tree and reads P(B | defective) = 0.625 straight off it, totals the defect rate with the Law of Total Probability, turns a 0.5% prior into a 9% posterior after one positive test, then chains a second and third test to watch belief climb past 97%, and finishes with a tiny naive Bayes spam filter, each result checked by simulation.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy and matplotlib and launch
jupyter notebook.
🎓 Key Takeaways
- ✓Bayes' theorem: P(H | E) = P(E | H) P(H) / P(E); it reverses a conditional, turning likelihood and prior into a posterior.
- ✓posterior ∝ likelihood × prior is the whole idea; the denominator just rescales so the posteriors sum to 1.
- ✓Law of Total Probability: P(E) = Σ P(E | Hᵢ) P(Hᵢ) over a partition; it is the sum of all the tree's branches.
- ✓Rare hypotheses stay rare: one positive test on a 0.5% disease gives only a 9% posterior; evidence compounds across independent tests.
- ✓In ML/AI: naive Bayes, Bayesian inference, priors as regularization, and uncertainty estimates are all this one formula.
Practice Challenges
Five short challenges, beginner to intermediate. Try them on paper or in Python before checking the solutions.
Reverse a conditional
70% of email is legitimate, 30% is spam. The word "offer" is in 60% of spam and 5% of legit mail. An email has "offer". Find P(spam | "offer").
Total probability
Three suppliers ship parts: S1 = 50% at 1% defective, S2 = 30% at 2%, S3 = 20% at 3%. What fraction of all parts are defective?
One test
A condition has 2% prevalence; a test has 90% sensitivity and 85% specificity. Given one positive, find P(condition | positive).
A second test
The same patient tests positive again on an independent test. Update once more, using your Challenge 3 posterior as the new prior.
Naive Bayes
A filter has prior P(spam) = 0.5. The word "win" is in 70% of spam and 5% of ham. Given "win", find P(spam | "win").
A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on Bayes' theorem and total probability. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.