Ask a working data scientist what they open first, and the answer is almost always the same: a Python notebook. Not because Python is the fastest language or the most elegant, but because around it has grown the most complete ecosystem for analysis anywhere: arrays, dataframes, plots, statistics, and machine learning, all speaking to each other. Learn the handful of libraries in this chapter and you can load, clean, reshape, summarize, and visualize almost any dataset you meet.
Python for data analysis is too much ground to cover in a single companion, so this chapter is built around a five-notebook mini-course: Python foundations, NumPy, pandas essentials, reshaping and aggregation, and visualization. The sections below give you the map and the ideas; the notebooks give you the reps. Read a section, run its notebook, and change the code, that loop is the whole point.
The Python Data Stack
Python itself is a general-purpose language. What makes it the home of data analysis is a small set of libraries that layer cleanly on top of one another, each solving one problem and handing off to the next.
| Library | What it gives you | Where it fits |
|---|---|---|
| NumPy | The fast N-dimensional array and vectorized math. | The numerical foundation. Almost everything else is built on it. |
| pandas | The labeled table (DataFrame) and Series, with loading, cleaning, joining, and grouping. | The analyst's daily workhorse, where most of your time is spent. |
| Matplotlib | Complete, low-level control over every element of a figure. | The plotting engine underneath everything else. |
| seaborn | One-line statistical charts with sensible defaults, built on Matplotlib. | Fast, attractive exploratory visuals. |
| scikit-learn | A uniform interface to machine-learning models (its own chapters in this book). | Modeling, once the data is ready. |
| Jupyter | The notebook: code, output, charts, and narrative in one document. | The environment that ties it all together. |
The design is deliberately layered. NumPy holds the numbers; pandas puts labels on them and adds table operations; Matplotlib draws them; seaborn makes the drawing easy; scikit-learn models them. You can drop down a layer whenever you need more control, which is why the stack scales from a five-line exploration to a production pipeline.
NumPy: Think in Whole Arrays
NumPy contributes one deceptively simple idea that reshapes how you write code: put your numbers in an array and operate on the whole thing at once, rather than looping element by element. This is called vectorization, and it is both easier to read and, because the work runs in compiled code under the hood, often a hundred times faster than an equivalent Python loop.
Three companion ideas make arrays powerful. A boolean mask filters with a condition
(prices[prices > 30]). Broadcasting stretches a smaller array to line up with a bigger
one so you can combine different shapes without writing loops. And aggregations with an
axis argument collapse an array in a chosen direction, axis=0 down the columns,
axis=1 across the rows. Every one of these reflexes reappears in pandas, because pandas is built directly
on NumPy. Notebook 2 drills all of them, including a timed race that shows the vectorized version running dozens of
times faster than the loop.
pandas: The DataFrame Is the Workhorse
If you learn one thing from this chapter, learn pandas. A DataFrame is a table you can program: rows and columns like a spreadsheet, but with the full power of code behind every operation. The overwhelming majority of real analysis is a sequence of a few verbs on a DataFrame.
| The verb | What it does | Typical call |
|---|---|---|
| Load & inspect | Read a file, then look before you leap. | read_excel, head, info, describe |
| Select | Pick columns and rows, by label or position. | df["col"], df.loc[...], df.iloc[...] |
| Filter | Keep rows matching a condition. | df[df.revenue > 100] |
| Transform | Build new columns from existing ones. | df["profit"] = df.rev - df.cost |
| Group & aggregate | Split into groups, summarize each. | df.groupby("region").revenue.sum() |
| Join & reshape | Combine tables; pivot between wide and long. | merge, pivot_table, melt |
The one distinction worth pinning down early is loc versus iloc: loc selects by
label, iloc by integer position. And whenever you process data in groups, remember the
rule from the scaling chapter, you can sum and count per group but you cannot average the averages
unless the groups are equal in size. Notebooks 3 and 4 walk the whole list, from a first head() through
groupby, merge, pivot_table, dates, and method chaining, on the real dataset
below.
loc reaches a value by its labels,
iloc by its integer position, and the two happen to agree here only because the index is the default
0, 1, 2. A missing rating shows up as NaN, which pandas skips in calculations unless you tell it not to.
The single most common beginner mistake is running statistics on data they have never actually looked at. Before
any analysis, call head, info, and describe. Check the shape, the column
types, and the count of missing values. Half of all “wrong” results are really a column read as text, a
stray unit, or a pile of blanks that a five-second glance would have caught.
Real-World Example: From Raw File to Finished Chart
The five notebooks share one real dataset so you can watch a single file travel the entire stack: read with pandas, compute with NumPy, reshape with groupby and merge, and finish as a chart. It is a small online-store order log, chosen because it has a bit of everything: dates, categories, numbers, a join key, and some missing values.
Online store orders, 3,000 orders across 2024, in two sheets. The
Orders sheet is the fact table: order_id, order_date, a
product_id join key, the region and channel (web, mobile, store),
quantity, discount_pct, the revenue in dollars, and a customer
rating from 1 to 5 that is blank for about 6 percent of orders. The Products sheet is a
small lookup table mapping each product_id to its name, category, unit cost, and list price, so you can
join it in to compute profit.
Running the notebooks end to end tells a coherent little business story, and every number here comes straight from their executed output.
- ● The headline. The store took 236,109 dollars over 3,000 orders in 2024, an average of 78.70 per order, though the median is only 56.00: the distribution is right-skewed, with a long tail of large orders.
- ● Concentration. A NumPy one-liner shows the orders over 100 dollars, just 27 percent of them, account for 59 percent of all revenue. A few big baskets carry the business.
- ● Where it comes from. By region, North leads at 56,668 dollars with West just behind; by category, Equipment dominates at 115,918; and the web channel outsells mobile and store combined.
- ● Profit needs the join. Merging the Products sheet and subtracting cost gives 122,342 dollars of profit, with Equipment again on top. That number is invisible until you join the two tables.
- ● The trend. Grouping by month puts the revenue peak in September at 22,766 dollars, the kind of seasonal signal a single line chart makes obvious.
Notice the shape of that workflow: read, look, compute, group, join, plot, conclude. It is the same arc whether the file has three thousand rows or three hundred million. Master it on a dataset you can check by eye, and it carries everywhere.
The Five-Notebook Mini-Course
Work through these in order. Each is self-contained and runnable, building from the language itself up to a finished dashboard. Open them here or launch straight into Colab.
Python Foundations
The language with no libraries: types, lists, dicts, tuples, sets, control flow, comprehensions, and functions, ending with a pure-Python analysis that shows why pandas exists.
NumPy
Arrays and vectorization: creating arrays, the loop-versus-vector speed race, boolean masks, broadcasting, and axis-aware aggregations, finishing on the real revenue data.
pandas Essentials
The DataFrame day to day: Series and DataFrames, loading and inspecting, selecting with loc and iloc, filtering, building columns, and handling missing data.
Reshaping & Aggregating
Where analysis happens: groupby (split-apply-combine), merging tables, pivot_table, melt between wide and long, working with dates, and method chaining.
Visualization
Turning tables into pictures: the anatomy of a Matplotlib figure, line, bar, histogram and scatter plots, one-line seaborn statistical charts, and a multi-panel dashboard.
Every notebook runs on numpy, pandas,
matplotlib, seaborn, and openpyxl, all preinstalled on Colab.
Why Python Won Machine Learning & AI
Python is not just convenient for analysis; it is the default language of modern machine learning and AI. That was not inevitable, and understanding why it happened tells you what the ecosystem is really for.
| Reason | What it means in practice |
|---|---|
| One array standard | NumPy gave every library a common data structure. A pandas column, a scikit-learn input, and a deep-learning tensor are all arrays, so tools compose instead of fighting. |
| A consistent model API | scikit-learn's fit/predict pattern became a convention the whole field copied, so
trying a new model is a one-line change. |
| The deep-learning frameworks | PyTorch and TensorFlow are Python-first. The research that defines modern AI is written, published, and shared as Python. |
| Glue, not speed | The heavy math runs in compiled C, CUDA, or Rust underneath; Python is the readable layer that orchestrates it. You get C speed with scripting-language ergonomics. |
| Notebooks and community | Jupyter made analysis shareable and reproducible, and a vast community means an answer, a package, or an example for almost anything. |
The stack in this chapter is the visible tip of a deliberate design philosophy sometimes called “the SciPy ecosystem”: many small, interoperable libraries agreeing on NumPy's array as a common currency, rather than one monolithic tool. That agreement is why a dataframe flows into a plot flows into a model with no glue code, and it is the quiet reason Python, rather than a faster or more specialized language, became the place where data science and AI are actually done.
🎓 Key Takeaways
- ✓The stack is layered: NumPy arrays underneath, pandas tables on top, Matplotlib and seaborn for charts, scikit-learn for models, all inside a Jupyter notebook.
- ✓Vectorization is the NumPy mindset: operate on whole arrays at once for code that is shorter to write and often a hundred times faster to run.
- ✓The DataFrame is the workhorse: load, inspect, select, filter, transform, group, and join cover the large majority of real analysis.
- ✓loc selects by label, iloc by position, and you can sum or count per group but never average the averages of unequal groups.
- ✓Always look before you analyze: head, info, and describe catch the mis-typed column and the pile of blanks before they become wrong answers.
- ✓Match the chart to the question: a line for trend, bars for comparison, a histogram for shape, a scatter for relationship.
- ✓Python won AI on interoperability: a shared array standard and a consistent model API let tools compose, with the heavy math running in compiled code underneath.
Practice Challenges
Five exercises, one per notebook area, on the store-orders data. Full solutions are in the companion solutions notebook.
Pure-Python summary
Given a list of order values, compute the count, mean, and maximum using only built-ins and a comprehension, with no libraries.
sum(), len(), max(), and a list comprehension for the filter.Vectorized NumPy
From the revenue array, report the share of revenue from orders over 100 dollars, and standardize the array to z-scores, all without a loop.
(x - x.mean()) / x.std().Filter with pandas
Keep web orders placed in the fourth quarter (October to December) and report how many there are and their total revenue.
&, and use order_date.dt.quarter.Group with a join
Merge the products table, then report revenue and profit by category, sorted by profit.
merge on product_id, then groupby("category").agg(...).Plot the trend
Plot 2024 revenue by month as a line chart and mark the peak month.
order_date.dt.to_period("M"), then plot and annotate the max.Solutions notebook
All five challenges worked in code, one per area: a pure-Python summary, a vectorized NumPy share and z-score, a pandas quarter filter, a groupby with a merge, and a plotted, annotated monthly trend.
Quiz: Test Yourself
Eight questions on the stack, NumPy, pandas, and visualization. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved.
Python is one of two great open-source languages for statistics. R for Statistics turns to the other, built by statisticians for statisticians, and shows where each language shines. This is the start of the Tools & Workflow part, which surveys the practical craft: languages, software, databases, and the habits that make analysis reproducible. Browse the full Contents for what is published and what is on the way.