The Whole Pipeline: Reflection and Next Steps

Nothing here is new material. Everything here is a decision you now know how to make, arranged so you can find it under pressure on a Tuesday afternoon.

Every project in this book, from a two-sample t-test to a reinforcement learning agent, followed the same path. The path is a loop and not a line, and the loop is the honest part: you clean, you look, you discover the grain is wrong, you go back.

The book is organized along that loop, and it is worth seeing the whole arc at once.

The most expensive errors in this book were not statistical. They were answering a question nobody asked, or answering a different question from the one that was asked and not noticing.

Before any statistic, two structural questions decide what is even possible.

What is one row? That is the grain, and getting it wrong invalidates everything downstream. One row per customer, per customer-month, per order, per order-line: these are four different datasets with four different answers to the same question. A count of rows only means something once the grain is named.

What kind of variable is each column? The level of measurement decides which summaries and which tests are legitimate, and it is not the same thing as the storage type. A postal code stored as an integer is nominal, and its mean is meaningless.

Alongside that sit the reshaping questions: long against wide, one table or a join, and what a join does to your row count. A join that changes the row count unexpectedly is a bug until proven otherwise, and checking counts before and after every merge catches a large share of quiet disasters. See Chapters 4 and 5 and Chapter 23.

Across forty-one capstones the same faults kept arriving. This is the catalog, built from what was actually planted and actually found.

Outliers are a decision, not a cleaning step. A value can be wrong, or rare and real, and those need opposite treatment. The question is never "is this an outlier" but "is this a measurement error, and if not, does my method tolerate it." See Chapter 21.

The first look is not a formality and it is not a table of means. It is the step that tells you which of the later steps are legitimate.

Compute the summary, then plot the thing. Anscombe's quartet is the classic warning, and every capstone in this book opened with a visual first look for the same reason: the number that would have misled was visible in the picture. Chapters 8 to 13.

The default chart is the one that shows every observation. Summarize only when the count makes that impossible, and then say what was summarized. Chapter 17 is the decision tree, and Chapter 157 is what changes when the audience is not you.

A distribution is a claim about the process that generated the data. Choosing one is choosing a story about mechanism, which is why the middle column matters more than the name.

Two practical habits. When data is positive and right-skewed, try a log scale before trying a more complicated model, because multiplicative processes are extremely common and a log turns them into additive ones. And when counts are overdispersed, move from Poisson to negative binomial rather than pretending the variance assumption held, which is what Chapter 189 was about.

Every method rests on assumptions, and the assumptions are testable. What follows is the battery, what failure looks like, and what to do about it, which is almost never "abandon the analysis."

And a warning about normality tests specifically. With a large sample they reject departures too small to care about; with a small one they miss departures that matter. They answer "is this exactly normal," which is never the question. The Q-Q plot answers "is this normal enough for what I am about to do," which is.

Everything downstream inherits whatever the collection did. No model repairs a sample that never contained the people the conclusion is about.

Then the design hierarchy, which decides what kind of claim is available at the end.

Sample size is a decision made before collection, not a justification made after. Power depends on the effect size you would not want to miss, the variability, the significance level and the power you want. Running the calculation after a null result to explain it is a well-known way to say nothing. Chapter 65, and Chapter 181 for what sequential peeking does to the error rate.

A point estimate without an interval is an opinion with decimal places.

Confusing the first two is one of the most common errors in applied work. A 95 percent confidence interval for a mean says nothing about where an individual will land, and quoting it to someone who needs to plan for individuals will systematically understate their risk.

The bootstrap is the general-purpose escape hatch. When the formula does not exist, the assumptions do not hold, or the statistic is something odd like a ratio of medians, resample and look at the spread. Chapter 75.

The test follows from four things: what kind of outcome, how many groups, whether observations are paired, and whether the distributional assumptions hold.

Correlation asks whether two things move together. Regression asks by how much, holding other things fixed, and that phrase is where all the difficulty lives.

R-squared is not a measure of correctness. It measures variance explained in the sample, it never decreases when you add a variable, and a high value on a misspecified model is common. Adjusted R-squared, AIC and out-of-sample error are all better, and the residual plots are better than any of them. Chapters 95 to 102.

The methods change; the discipline does not. Almost every machine learning failure in this book was a validation failure rather than a modeling one.

Time changes three things: the observations are not independent, the order is information, and validation has to respect the arrow of time.

And check that the interval covers. Forecast uncertainty usually grows with the horizon, so a band of constant width is over-confident about the far end. Measuring empirical coverage per horizon takes a few lines and is almost never done.

Some problems look like ordinary ones and are not. Recognizing the tell is most of the work.

An analysis nobody acts on and a model nobody monitors are the same failure at different points in the pipeline.

Across forty-one capstones the same handful of errors did nearly all the damage. They are worth naming because none of them are exotic, and every one produced a confident, plausible, wrong answer.

Reading those twelve together, only two are about choosing the wrong method. The rest are about measuring the wrong thing, measuring it on the wrong data, or believing a number that was never checked. That ratio has been the most useful thing in the book to know.

Being explicit about the edges is more useful than a summary that implies completeness.

Mathematical depth. The advanced part covers moments, MGFs and conditional expectation, but this is an applied book. Proofs of the theorems being used, measure-theoretic probability and the asymptotic theory behind the estimators are all somewhere else, and eventually worth going to.

Experimental design as a discipline. Factorial designs appear once. Response surface methods, optimal design, blocking beyond the basics and sequential designs are a field of their own.

Deep learning in earnest. There is a primer and an application chapter. Training modern architectures, distributed training, and the practical craft of it are a different book.

Data engineering. Pipelines, orchestration, streaming, warehouse modeling and cost control decide whether analysis is possible at all, and they get one chapter here.

Domain knowledge. The largest gap by far. Every method in this book is worth less than knowing what a plausible value looks like in your field, and that cannot be transferred by a textbook.

Four routes. They are not exclusive and most careers are a mixture, but the first choice worth making is which one you are deliberately deepening this year.

A note on staying current. The methods in the first two thirds of this book have been stable for decades and will outlast anything you learn this year about a specific library. The tooling changes constantly and matters less than it appears to. Spend your attention accordingly: read papers and textbooks for the ideas, and documentation for the tools, and do not confuse the two.

If a single habit had to survive from two hundred and four chapters, it is this: check the thing you are about to believe.

Not the assumption you were taught to check, though those matter. The number itself. Does the interval cover at the rate it claims? Was this data used to choose the thing it is now judging? Is this metric computable on the data it is being computed on? Would this conclusion survive the one obvious objection?

Almost every failure collected in this book would have been caught by someone spending twenty minutes trying to break their own result before presenting it. That is not a statistical technique. It is a habit, and it is the one worth keeping.