Preparing Data for Analysis cleaned, reshaped, and engineered your data. This final chapter is where you actually get to know it. Exploratory data analysis (EDA) is the deliberate habit of looking at data, one variable at a time, then in pairs, then together, before committing to any model or conclusion.
EDA was championed by statistician John Tukey, whose 1977 book Exploratory Data Analysis framed the analyst as a detective: the goal is to let the data speak before you impose a model on it. Tukey's enduring warning, "the greatest value of a picture is when it forces us to notice what we never expected to see," is the whole spirit of the chapter. EDA is contrasted with confirmatory data analysis (CDA), the formal testing of a pre-specified hypothesis that comes later in the book.
The two modes answer different questions, and keeping them separate matters more than it first appears:
| Exploratory (EDA) | Confirmatory (CDA) |
|---|---|
| "What is going on in this data?" | "Is this specific hypothesis supported?" |
| Open-ended, flexible, iterative | Pre-specified, disciplined, one shot |
| Visual and descriptive | Tests, p-values, confidence intervals |
| Generates hypotheses | Evaluates a hypothesis |
EDA is where you hunt for patterns, but if you then run a formal test on the same data that suggested the pattern, the p-value is no longer honest, you have implicitly tried many comparisons (the "garden of forking paths"). Explore freely, but confirm on fresh data or a held-out set. This is the same leakage logic that ran through encoding and imputation in the last few chapters.
The EDA Workflow
A good exploration is not random clicking; it follows a rough order, from a first-contact audit of the whole table, to one variable at a time, to pairs, to everything at once.
Before any chart, run the quick audit you have seen building all through the Preparing Data for Analysis part: df.shape,
df.info(), df.describe(), df.isna().sum(),
df.duplicated().sum(), and a value_counts() on key categories. It takes seconds
and catches the problems, wrong dtypes, silent missingness, duplicate rows, class imbalance, that would
otherwise corrupt everything downstream.
Univariate: One Variable at a Time
The first real exploration looks at each variable on its own and asks three questions met back in the descriptive-statistics chapters: where is its center, how much does it spread, and what is its shape?
| Variable type | Summaries | Plots | Look for |
|---|---|---|---|
| Numeric | mean, median, SD, IQR, skew | Histogram, box plot, density | Skew, modality, outliers, range |
| Categorical | counts, proportions, mode | Bar chart | Imbalance, rare levels, typos |
For a numeric variable the histogram is the workhorse: it reveals whether the distribution is symmetric or skewed (see Shape of a Distribution), whether it has one peak or several, and whether a long tail hides outliers (see Detecting & Treating Outliers). A box plot compresses the same variable into median, quartiles, and flagged outliers, which makes it ideal for comparing groups side by side. For a categorical variable, a simple bar chart of counts exposes class imbalance and stray misspelled levels at a glance.
The histogram (top) and box plot (bottom) describe the same column on the same axis. Two things jump out that no single number would tell you: the distribution is bimodal, with a hard empty gap from 2 to 3 cm, and that gap is not noise, it is the boundary where one species (setosa, the short-petalled cluster on the left) ends and the others begin. The box plot shows the matching wide spread (IQR 3.5 cm) and a median of 4.35 cm pulled toward the upper cluster. A hidden group inside one variable is exactly the kind of surprise EDA exists to catch.
Read the three together. A mean far from the median signals skew; a large SD or IQR signals spread; two peaks signal a mixture of groups hiding in one column. None of these show up in a single summary number, which is exactly why EDA insists you plot.
Bivariate & Multivariate: Relationships
Once each variable is understood alone, the interesting questions are about how they move together. The right tool depends on the pair of types.
| Pair | Plot | Summary |
|---|---|---|
| Numeric vs numeric | Scatter plot | Correlation (Pearson / Spearman) |
| Numeric vs categorical | Side-by-side box / violin | Group means, group differences |
| Categorical vs categorical | Grouped bar / mosaic | Cross-tab, proportions |
The same two columns, now plotted against each other. Longer petals
are reliably wider, an upward band with correlation r = 0.96. Coloring by species turns one chart into
three insights at once: setosa (cyan) sits in its own tight cluster at the bottom-left, while versicolor and
virginica stack along the same trend. A scatter plot reveals the shape of a relationship that a single
correlation number cannot.
For two numeric variables, the scatter plot shows the shape of the relationship and a correlation coefficient puts a number on it. But correlation comes in two flavors, and choosing the wrong one quietly understates real relationships:
| Coefficient | Measures | Use when |
|---|---|---|
| Pearson r | Strength of a linear relationship | The relationship looks like a straight line; roughly symmetric data |
| Spearman ρ | Strength of a monotonic relationship (on ranks) | Curved-but-rising relationships, ordinal data, or with outliers |
To see many relationships at once, the correlation heatmap draws the whole correlation matrix
as a grid of colors. Using a diverging palette centered at zero (red for positive, blue for
negative) makes the sign and strength readable instantly, and a hot off-diagonal block flags
multicollinearity: two features so correlated they carry nearly the same information
(in the notebook, iris petal length and petal width sit at r ≈ 0.96).
Every pair of features at a glance. The diverging scale runs blue
(negative) through white (zero) to red (positive), so sign and strength read off the color instantly. The
diagonal is all 1.00 (each variable with itself), and the outlined petal length × petal width
cell glows at 0.96, two features so alike they are nearly redundant. That hot off-diagonal block is the
visual signature of multicollinearity, a cue to drop or combine one of them before fitting
a linear model (see Feature Engineering). Note too that sepal width leans slightly negative against the petal measures.
Center the color scale at zero so positive and negative correlations are visually distinct, and fix the
range to [-1, 1] so colors are comparable across plots. Scan for hot pairs of
features (a multicollinearity warning that connects straight back to feature selection in
Feature Engineering) and for the row matching your target (the features most linearly related to what you want to
predict). Remember the limit: the heatmap only sees linear association.
Pitfalls & Always Plot
EDA's golden rule is the one Anscombe and the Datasaurus made unforgettable (see Charts for Numerical Data): summary statistics can lie, so always plot the data. Four datasets can share the same mean, variance, and correlation while looking completely different, one linear, one curved, one with a single outlier driving everything. Only the picture tells them apart.
Trusting a single number (a correlation or a mean) without a plot; confusing correlation with causation; reading a near-zero Pearson as "no relationship" when the link is nonlinear; and the forking-paths trap, formally testing a hypothesis the same data suggested. EDA generates ideas; it does not confirm them.
Modern tooling can accelerate the first pass. Automated profilers like ydata-profiling (once pandas-profiling), Sweetviz, and D-Tale generate a full report, distributions, missingness, correlations, warnings, from one line of code. They are a fast head start, not a replacement for looking: the human still has to ask the right questions and interpret what the report surfaces.
Every strong analysis and every reliable model begins with someone who genuinely understood their data first. EDA is that understanding, and it is the bridge out of data preparation: with the data cleaned (the Data-Cleaning Mindset through Feature Engineering chapters) and explored (this chapter), you are finally ready to reason about it formally. That is where the book turns next, to probability, the mathematics of uncertainty that underpins every test, model, and inference to come.
EDA in Machine Learning & AI
Exploratory analysis is not a warm-up you can skip, it is the first phase of every serious machine-learning project. What you learn here decides the features, the model, and the pitfalls you avoid.
| What EDA checks | Why it matters for the model |
|---|---|
| Distributions & skew of each feature | Tells you which columns need a transform or a robust scaler before training |
| Correlations & feature-target relationships | Flags redundant features to prune and promising ones to keep, and warns of leakage |
| Target balance & group differences | An imbalanced target changes the metric and the training recipe you should use |
A feature that correlates suspiciously well with the target is often a leak, information that would not exist at prediction time. EDA is where you spot it, along with the class imbalance, the skew, and the missingness that shape every downstream choice. Skip it and you build on assumptions you never checked.
Real-World Example: Exploring a Churn Dataset
Here is a dataset built for the full EDA workflow: 500 customers with numeric columns (age, tenure, spend, tickets), a categorical plan, and a binary target, churned. A quick exploration already tells a story, overall churn is about 22%, but it climbs to 27% on the Basic plan, and it falls as tenure rises. The companion notebook runs the whole pass, profile, distributions, correlations, and churn by group.
One row per customer: age, tenure_months, plan,
monthly_spend, support_tickets, and the target churned. The mix of numeric,
categorical, and target columns is exactly what a complete exploratory pass is built to examine.
Bring it to life in Python
The companion notebook explores the classic iris dataset end to end: a first-contact audit, a univariate look at one feature (center, spread, shape with histogram and box plot), a bivariate scatter and group-by-species comparison, a correlation heatmap that exposes the petal multicollinearity, and a Pearson-vs-Spearman demo on a curved relationship that shows why you always plot.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas,
scikit-learn, scipy, matplotlib and launch jupyter notebook.
🎓 Key Takeaways
- ✓EDA is understanding before modeling: an open-ended, visual, skeptical first pass that summarizes structure and surfaces hypotheses (Tukey, 1977).
- ✓Work outward: first-contact audit → univariate (center, spread, shape) → bivariate (pairs) → multivariate (all at once).
- ✓Pearson measures linear, Spearman measures monotonic; a near-zero Pearson does not mean "unrelated."
- ✓A correlation heatmap (diverging palette centered at 0) reveals strong pairs and multicollinearity, but only the linear part.
- ✓Always plot the data (Anscombe); and never confirm a hypothesis on the same data that suggested it (forking paths).
Practice Challenges
Five short challenges, beginner to intermediate. Try them on paper or in Python before checking the solutions.
First-contact audit
For a new DataFrame, report its shape, dtypes, total missing values, duplicate-row count, and the balance of a key category. Why run this before any chart?
shape, dtypes, isna().sum(), duplicated().sum(), value_counts().Describe one variable
For a numeric column, report center (mean, median), spread (SD, IQR), and shape (skew), then draw a histogram. What does comparing mean and median tell you?
Compare across groups
Does a numeric variable differ across a category? Compute the group means and draw side-by-side box plots. What makes a feature a good group separator?
groupby(cat)[num].mean() + boxplot per group; little overlap = strong separator.Find the redundant pair
Compute a correlation matrix for several numeric features and identify the most strongly correlated pair. Why is that a multicollinearity flag?
df.corr(); the largest off-diagonal |r| is the near-duplicate pair.Pearson vs Spearman
Build a monotonic but curved relationship (e.g. y = x² on positive x). Compute both
correlations. Why do they differ, and what is the lesson?
A fully-worked solutions notebook walks through all five challenges in the same visual style. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on exploratory data analysis. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.