The EDA Case Studies closed the book's first half: describing, visualizing, and preparing data. Now the second half begins, and it all stands on one idea. Before we can say how confident we are in a result, we need a precise way to talk about chance.
Probability theory traces to a 1654 exchange of letters between Blaise Pascal and Pierre de Fermat over how to fairly split the stakes of an interrupted game of chance (the "problem of points"). It matured through Jacob Bernoulli and Laplace, and was finally put on rigorous footing by Andrey Kolmogorov in 1933, whose three axioms still define the subject. What began as a way to win bets is now the language of all of statistics and machine learning.
Experiments, Outcomes, and Events
Three words carry the whole subject. A random experiment is any process with an uncertain result (rolling dice, flipping a coin). The sample space is the set of every possible outcome. An event is any subset of that sample space, the outcomes that make some statement true.
A simple event is a single outcome (rolling the pair 3-and-4). A compound event bundles several (the sum is 7, which covers six different pairs). Either way, the event is a subset of the sample space, and that is all you ever need to compute its probability.
Three Ways to Assign a Probability
Where does a probability number actually come from? There are three honest answers, and good practice uses whichever fits the situation.
| Approach | Probability is⦠| Use it when |
|---|---|---|
| Classical | favorable outcomes / total outcomes | Outcomes are equally likely and countable (dice, cards, fair coins) |
| Empirical | the long-run relative frequency observed | You can repeat the experiment or have historical data |
| Subjective | a reasoned degree of belief | A one-off event with no symmetry and no repeats (this is the Bayesian view) |
These are not rivals so much as different tools. The remarkable fact, coming up in section 4, is that the classical and empirical answers agree as the number of trials grows.
The Scale, the Axioms, and the Complement
Every probability sits on a fixed scale from 0 to 1. You can write it as a fraction, a decimal, or a percent; they all say the same thing.
All of probability follows from three simple rules. For any event A in a sample space S: (1) P(A) is never negative; (2) P(S) = 1 (something in the sample space is certain to happen); (3) if two events cannot both occur, the probability of one or the other is the sum of their probabilities. Everything else, including every rule in the next chapter, is a consequence.
The first and most useful consequence is the complement rule: since something must happen, the probability an event does not occur is P(not A) = 1 − P(A). It is the easy route to any "at least one" question: P(at least one six in two rolls) = 1 − (5/6)² = 0.31, far simpler than counting all the ways to get a six.
The Law of Large Numbers, and a Famous Trap
Here is the bridge between the classical and empirical views. The Law of Large Numbers (Jacob Bernoulli, 1713) says that as you repeat an experiment, the observed relative frequency closes in on the true probability. Flip a fair coin a few times and anything can happen; flip it thousands of times and the share of heads settles near 0.5.
The Law of Large Numbers is about the long run, not the next trial. After five heads in a row, tails is not "due", the coin has no memory, and the next flip is still 50/50. Independent events do not self-correct. Confusing "rare over many trials" with "owed right now" is the single most common, and most expensive, misreading of probability.
Probability in Machine Learning & AI
Probability is not just the warm-up for inference, it is the language modern models speak. A trained classifier or language model rarely returns a hard answer; it returns a probability distribution, and we read that distribution to make a decision.
The set of classes is a sample space, each class is an event, and the model's raw scores are squeezed by a softmax so they are non-negative and sum to 1, exactly the axioms from section 3. Pick the largest for a prediction, or sample from the distribution to let a language model write.
| Fundamental (this chapter) | In ML / AI it becomes | Concrete example |
|---|---|---|
| Sample space & events | The set of labels or tokens a model chooses among | a classifier over {cat, dog, fox}; an LLM over its vocabulary |
| Axioms: bars sum to 1 | The softmax output layer | turns raw scores into a valid probability distribution |
| Empirical probability + LLN | Learning from data; Monte Carlo estimation | estimate P from millions of examples; more data, tighter estimate |
| Subjective probability | Bayesian priors | encode belief before data, then update after seeing it |
| Complement rule | Binary classification | P(not spam) = 1 − P(spam) |
A trained model almost never outputs a single answer; it outputs a distribution, and we choose how to read it (argmax for a label, or sampling for a language model). The Law of Large Numbers is why training works at all: averaging a loss or gradient over a large batch converges to its true expected value, so bigger, cleaner data gives steadier learning. And calibration, whether a model's stated 0.9 really means right about 90% of the time, is an active research concern built directly on the empirical view of probability.
See probability in action
The companion notebook builds the two-dice sample space, computes the classical distribution of the sum, then simulates thousands of coin flips and die rolls to watch the Law of Large Numbers converge. It uses the complement rule for an "at least one" problem, and puts the gambler's fallacy to the test by checking what really happens right after a winning streak.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas,
matplotlib and launch jupyter notebook.
π Key Takeaways
- βExperiment, sample space, event are the three building blocks; an event is a subset of the sample space.
- βThree ways to assign probability: classical (favorable/total), empirical (long-run frequency), subjective (degree of belief).
- βProbabilities live on [0, 1] and obey Kolmogorov's three axioms; the complement rule P(not A) = 1 − P(A) is the workhorse.
- βThe Law of Large Numbers makes empirical frequencies converge to true probabilities as trials grow.
- βThe gambler's fallacy is a fallacy: independent events have no memory, the next trial is unaffected by the last.
Practice Challenges
Five short challenges, beginner to intermediate. Try them on paper or in Python before checking the solutions.
Build a sample space
Flip a fair coin twice. Write the full sample space, then list the outcomes in "exactly one head" and "at least one head".
Classical probability
Using that sample space, compute P(exactly one head) and P(at least one head) by counting favorable over total.
Empirical vs theoretical
Estimate P(at least one head in two flips) by simulating 50,000 experiments, and compare to the theoretical value.
Use the complement
A die is rolled three times. Find P(at least one six) with the complement rule, then verify by simulation.
No such thing as "due"
Simulate a fair coin. After every run of four tails, what fraction of the next flips are heads? Does the streak make heads more likely?
A fully-worked solutions notebook walks through all five challenges in the same visual style. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on probability fundamentals. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.