Ch 3 · The Data Science Lifecycle — Statistics, Data Science and AI

A data science project isn't one big leap from "data" to "answer." It's a sequence of stages, each taking the output of the last and refining it, turning a fuzzy question into raw data, raw data into clean data, clean data into insight, insight into a model, and a model into real-world impact.

🔄

The data science lifecycle is the repeatable, iterative roadmap a project follows: Frame → Collect → Clean → Explore → Model → Deploy, and then loop back as the world changes.

This is the same idea as the industry-standard CRISP-DM process, told in plain language. Learn it once and you'll recognize it inside every project in this book, and every job in the field.

🗂️

If you meet the "official" diagrams

Classic CRISP-DM has six phases and splits out a Data Understanding step (first exploration and quality checks) between collecting and preparing data; we fold that into Clean and Explore. You'll also see OSEMN (Obtain, Scrub, Explore, Model, iNterpret) used for the same loop. Different names, same journey.

The Lifecycle at a Glance

Six stages, each feeding the next. The arrows go forward, but the dotted one reminds us the whole thing loops:

The stages run forward, but the dashed arc closes the loop: you monitor and circle back as the data and the world change.

Stage by Stage: Input → Output

Why is each stage crucial, and what exactly does it take in and hand out to the next stage? Here's the breakdown:

🎯 Frame the Question · business understanding

Why it's crucial: a project aimed at the wrong question wastes every stage after it. Get this right and everything else has a target.

IN: a fuzzy business goal → OUT: a specific, measurable question + success metric

e.g., "sell more ice cream" → "predict daily sales from temperature, measured by RMSE."

🗄️ Collect the Data · acquisition

Why it's crucial: no data, no data science. The data you gather must actually contain the answer to your question.

IN: the question + data sources → OUT: a raw dataset

e.g., pull 14 days of temperature and sales records from the shop's logs.

🧹 Clean & Prepare · data wrangling

Why it's crucial: real data is messy, and models trust whatever you feed them, garbage in, garbage out. This is famously ~80% of the work.

IN: raw, messy data → OUT: a tidy, validated dataset (+ engineered features)

e.g., remove duplicate days, fill a missing sales value, drop an obvious data-entry error.

🔍 Explore & Analyze · EDA

Why it's crucial: you must understand the data before modeling it, spot patterns, check assumptions, and catch surprises before they bite.

IN: clean data → OUT: insights, summary stats, charts, hypotheses

e.g., a scatter plot shows sales rise strongly with temperature (correlation ≈ 0.98).

🤖 Model · build & train

Why it's crucial: this is where a pattern becomes a reusable predictor. Done well, it generalizes to new cases; done carelessly, it just memorizes.

IN: prepared features + target (split into train/test) → OUT: a trained model

e.g., fit a line sales = a + b·temperature on the training days only.

✅ Evaluate, Communicate & Deploy · delivery

Why it's crucial: an untested model is just a guess. Value appears only when results are honestly scored, clearly communicated, put to use, and then monitored.

IN: the trained model + held-out data → OUT: an honest score + a deployed tool/decision (+ monitoring)

e.g., measure error on unseen days, then ship a predict_sales() tool and watch it over time.

💡

The hand-off principle

Each stage's output is the next stage's input. A weak link anywhere, a vague question, dirty data, a skipped evaluation, silently degrades everything downstream.

A Cycle, Not a Straight Line

Real projects rarely march cleanly from 1 to 6. You'll loop back constantly: exploring reveals you need more data; a weak model sends you back to feature engineering; the world shifts and yesterday's model goes stale. That's not failure, it's the process working.

⏳

~80% is data work

Collecting and cleaning typically dominate a project. Modeling is the glamorous 20%.

🔁

Iterate freely

Findings at any stage can send you back a step. Expect loops, not a one-way street.

📡

Monitor after launch

Deployed models drift as data changes, watch the error and retrain when it grows.

🚧

What can go wrong at each stage

Frame: the wrong question. Collect: biased or unrepresentative data. Clean: data leakage. Explore: seeing patterns that aren't real. Model: overfitting. Deploy: drift. A good "Evaluate" step also compares the model against a simple baseline (such as "always predict the average") to prove it actually adds value.

⚖️

Ethics runs through every stage

At each step, ask: was the data collected with consent? Is it representative, or does it leave groups out? Could the model harm someone or treat a group unfairly? Responsible data work treats privacy, bias, and fairness as part of the job, not an afterthought. We devote a full chapter to data ethics later in the book.

Garbage in, garbage out, a model is only ever as good as the data and the question behind it. — The first law of the lifecycle

Real-World Impact, Across Industries

This same lifecycle, run at scale, reshapes entire industries. A few well-documented examples:

🛒

Business & Retail

Recommendation engines turn behavior data into sales. Netflix reports its recommender drives a large majority of what members watch and saves an estimated $1B/year in retention[1]; McKinsey attributed about 35% of Amazon purchases to recommendations[2].

🧬

Healthcare & Life Science

An AI screening model matched or beat radiologists at detecting breast cancer[4], and DeepMind's AlphaFold predicted protein structures at near-experimental accuracy, a breakthrough for biology and drug discovery[5].

💳

Finance

Banks use statistical and machine-learning models to flag fraudulent transactions in real time, weighing the cost of misses against false alarms, a decades-old, high-stakes application of the lifecycle[6].

🚚

Logistics, Sport & Beyond

UPS's ORION route-optimization system saves roughly 100 million miles and millions of gallons of fuel each year[3]; in sport, the data-driven "Moneyball" approach reshaped how teams value players[7].

🌍

The common thread

Different industries, identical recipe: frame a question, gather and clean data, find the pattern, model it, and deploy, then measure the impact and improve. References are listed at the end of this chapter.

🐍

Bring it to life in Python

The companion notebook runs one complete mini-project, predicting ice-cream sales from temperature, through all six stages, with working code and a chart at every step.

📓 View Notebook (code & outputs) ▶ Open in Colab ⬇ View / Download on GitHub

View opens the rendered notebook instantly (no setup). Open in Colab runs & edits it live in your browser. To run locally, install numpy, pandas, matplotlib and launch jupyter notebook.

🎓 Key Takeaways

✓Six stages: Frame → Collect → Clean → Explore → Model → Deploy (the CRISP-DM process in plain language).
✓Each stage's output is the next one's input, a weak link anywhere degrades everything downstream.
✓It's a cycle, not a line: expect to loop back, and to keep monitoring after launch.
✓~80% of the work is data, collecting and cleaning, not modeling.
✓The same recipe creates real impact in retail, healthcare, finance, logistics, and beyond.

Practice Challenges

Six challenges, one for each stage of the lifecycle. Beginner-to-intermediate; try them on paper or in Python.

Frame · Turn a wish into a question

A gym says "we have too many cancellations." Rewrite it as a measurable data question with a target and a success metric.

Hint: what exactly would you predict, and how would you know it worked?

Collect · Find the data

For that gym problem, name two useful data sources and state which column is the target (label).

Hint: what behavior might signal that someone is about to cancel?

Clean · Tidy a messy table

Given names=[Ana, Ben, Ben, Cara, Dan], ages=[34, 41, 41, missing, 29]: drop the duplicate row, then fill the missing age with the mean.

Hint: clean before you analyze, duplicates double-count and gaps break the math.

Explore · Find the relationship

Exercise hours/week [0,1,2,3,4,5,6] vs resting heart rate [80,78,75,72,70,67,64]. Compute the correlation and say what it means.

Hint: is the relationship positive or negative? Strong or weak?

Model · Fit a line and predict

Using that same data, fit a straight-line model, write its equation, and predict the resting heart rate for 3.5 hours/week.

Hint: a line is y = intercept + slope·x; let Python find the best fit.

Evaluate & Deploy · Score it

A model predicted [70, 60, 80] but the truth was [72, 58, 85]. Compute the RMSE, and write one line on how you'd keep it healthy in production.

Hint: RMSE = √(average of squared errors). Smaller is better.

✅

Check your work

A fully-worked solutions notebook walks through all six challenges in the same visual style, try them yourself first, then compare.

📓 View Solutions ▶ Open Solutions in Colab ⬇ View / Download on GitHub

Quiz: Test Yourself

Eight quick questions to lock in the lifecycle. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.

📚

References

Gomez-Uribe, C. A., & Hunt, N. (2015). The Netflix Recommender System: Algorithms, Business Value, and Innovation. ACM Transactions on Management Information Systems, 6(4), 1–19. doi:10.1145/2843948
MacKenzie, I., Meyer, C., & Noble, S. (2013). How retailers can keep up with consumers. McKinsey & Company.
Holland, C., Levis, J., Nuggehalli, R., Santilli, B., & Winters, J. (2017). UPS Optimizes Delivery Routes. INFORMS Journal on Applied Analytics (Interfaces), 47(1), 8–23. doi:10.1287/inte.2016.0875
McKinney, S. M., et al. (2020). International evaluation of an AI system for breast cancer screening. Nature, 577, 89–94. doi:10.1038/s41586-019-1799-6
Jumper, J., et al. (2021). Highly accurate protein structure prediction with AlphaFold. Nature, 596, 583–589. doi:10.1038/s41586-021-03819-2
Bolton, R. J., & Hand, D. J. (2002). Statistical Fraud Detection: A Review. Statistical Science, 17(3), 235–255. doi:10.1214/ss/1042727940
Lewis, M. (2003). Moneyball: The Art of Winning an Unfair Game. W. W. Norton & Company.

Figures such as "$1B/year," "35%," and "100 million miles" are as reported by the cited sources and organizations; they are illustrative of scale rather than audited values.