Preparing Data for Analysis is the unglamorous foundation under everything else. Surveys consistently find that data scientists spend the majority of their time not modeling, but getting data into shape. There is a reason: no analysis is better than the data beneath it.
Garbage In, Garbage Out
Garbage in, garbage out (GIGO), a phrase often credited to an IBM instructor in the late 1950s, captures the whole chapter: the output of any process is only as trustworthy as its input. A brilliant model fed bad data produces confident, wrong answers.
The time cost is real: surveys (the often-cited 2016 CrowdFlower report) put roughly 60% of a data scientist's time on cleaning and about 80% on data preparation overall, treat it as a survey-based estimate, not a law. The money cost is large too: IBM estimated poor data quality costs the US economy around $3.1 trillion a year, and Gartner put the average organization's cost near $12.9 million a year. These are estimates, but the scale is the point.
The Mars Climate Orbiter (1999) was lost because one team used pound-force and another expected newtons, a units inconsistency, not a code bug. The Reinhart-Rogoff economics paper that influenced austerity policy turned out to rest on a spreadsheet formula that omitted five countries; fixing it reversed the headline result. Both blend data and process error, and both are exactly what a cleaning mindset guards against.
What Makes Data "Good"
Analysts judge data against a standard set of quality dimensions (the DAMA framework). Naming them gives you a checklist for "is this data trustworthy?"
Accuracy
Values match reality. A real age of 34 recorded as 43 fails accuracy.
Completeness
Required values are present. 200 blank emails out of 1,000 fails completeness.
Consistency
The same fact agrees everywhere. "CA" here, "California" there, fails consistency.
Validity
Values obey the rules. month = 13, or an email with no "@", fails validity.
Uniqueness
Each entity appears once. The same customer entered three times fails uniqueness.
Timeliness
Data is current when needed. A "live" count last refreshed six weeks ago fails timeliness.
The Common Data Problems
Almost all dirty data falls into a short catalog. Learn to recognize these on sight; the next two chapters go deep on the most common ones.
| Problem | Example | Breaks | Where it's handled |
|---|---|---|---|
| Missing values | Blanks, NaN, or hidden codes like -99, 9999, text "N/A" | Completeness | Handling Missing Data |
| Duplicates | The same record twice; near-dupes ("Jon Smith" vs "John Smith") | Uniqueness | Finding & Removing Duplicates and Inconsistencies |
| Inconsistent formatting | Dates 01/02/2020 vs 2020-02-01; "USA"/"U.S.A."/"United States" | Consistency | Finding & Removing Duplicates and Inconsistencies |
| Untidy structure | Two values in one cell; years as column headers; one record split across rows | Structure | Tidy data (below) |
| Outliers & impossible values | age = 200; negative quantity; a future birth date; a typo'd extra zero | Accuracy / Validity | This chapter (detect) |
| Wrong data types | Numbers stored as text ("1,234"); ZIP 02134 losing its leading zero | Validity | This chapter (detect) |
| Whitespace / encoding | Trailing spaces ("NY "); mojibake (café); non-breaking spaces | Consistency | Finding & Removing Duplicates and Inconsistencies |
| Inconsistent categories | "Male"/"male"/"M"/"m" all meaning the same thing | Consistency | Finding & Removing Duplicates and Inconsistencies |
The sneakiest of these is hidden missingness:
a column of ages with -99 for "unknown" looks numeric and computes a mean, just a meaningless one.
You have to know the codes and look for them.
Tidy Data & the Cleaning Workflow
Beyond typos and gaps there is structural cleanliness. Hadley Wickham's tidy data gives three rules for a layout that every downstream tool can use:
1. Variable = column
Each variable forms one column.
2. Observation = row
Each observation forms one row.
3. Unit = table
Each type of observational unit forms one table.
With the vocabulary in hand, the mindset itself is a repeatable loop, technique comes in later chapters, but the discipline is universal:
Inspect & audit
Look first: info(), describe(), value_counts(), missingness. Profile against the quality dimensions.
Never touch the raw
Keep an immutable raw copy. Clean into a new version so the chain of custody is intact.
Script it, document it
Clean in code, not by hand in a spreadsheet, so it is reproducible, reviewable, and re-runnable.
Validate, then repeat
Re-check after cleaning. Expect several passes: fixing one issue often reveals the next.
You cannot fully automate this. Is age = 0 a newborn or a missing default? Only domain
knowledge decides. A true outlier is not an error: a real billionaire belongs in an income
column; a typo'd age = 200 does not. And silently dropping inconvenient rows can
bias your results, document every decision rather than quietly bending the data toward
the answer you wanted.
Clean data is the foundation of everything ahead: the summary statistics of Describing Data, the charts of Visualizing Data, and all the modeling to come. The honest slogan is that most of data science is data cleaning. This chapter set the mindset; Finding & Removing Duplicates and Inconsistencies tackles duplicates and inconsistencies, and Handling Missing Data takes on missing data.
The Cleaning Mindset in Machine Learning & AI
In machine learning, data quality is not a preliminary, it is the ceiling on everything that follows. No model recovers from dirty inputs, which is why the audit habit this chapter teaches is a core part of any pipeline.
| Cleaning idea | Where it shows up in ML | Why it matters |
|---|---|---|
| Garbage in, garbage out | The quality of the training data caps model accuracy | Effort on data quality usually beats effort on a fancier model |
| Data profiling / validation | An automated first-audit step (schema and range checks) in the pipeline | Catches the bad batch before it silently retrains a live model |
| Train/serve consistency | The same cleaning must run at training time and at prediction time | A cleaning step applied to only one of them is a classic production bug |
The discipline that separates a reliable model from a fragile one is boring: profile the data before you touch it, list every issue, then fix them deliberately. Tools like a data-validation schema automate exactly the audit you do by hand here, so a bad column stops the pipeline instead of poisoning it.
Real-World Example: Auditing a Messy Table
Here is a deliberately messy customer table to practice the first-audit habit on. Without fixing anything, a quick profile reveals the damage: 12 duplicate rows, missing values scattered across columns, ages that are impossible (a 200-year-old customer), a city column spelled twenty different ways, and spend stored as text. The companion notebook runs that audit and writes the to-do list.
One row per customer (with duplicates): customer_id, city
(inconsistent casing), age (some impossible), signup_date (text), and
total_spend (a dollar string). Every column has a different problem, which is the point, learn to
find them before you fix them.
Bring it to life in Python
The companion notebook takes a deliberately messy DataFrame and audits it: reading dtypes and ranges,
surfacing inconsistent categories with value_counts, finding the hidden -99
that isna() misses, flagging duplicates and impossible values, and printing a reusable
data-quality report.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas,
matplotlib and launch jupyter notebook.
🎓 Key Takeaways
- ✓Garbage in, garbage out: no analysis is more trustworthy than the data feeding it.
- ✓Dirty data is costly in time (most of the job) and money; the cleaning mindset is high-leverage, not grunt work.
- ✓Judge data on quality dimensions: accuracy, completeness, consistency, validity, uniqueness, timeliness.
- ✓Know the common problems (missing, duplicate, inconsistent, untidy, impossible, wrong-type) and watch for hidden missing codes.
- ✓Inspect before you trust: audit on a copy, clean in a script, document decisions, and validate. It takes judgment, not just code.
Practice Challenges
Five short challenges, beginner to intermediate. Try them on paper or in Python before checking the solutions.
Wrong types
A frame has prices like "1,200" and quantities as text. Use dtypes to find
which columns won't do math, and explain why that happens on import.
Hidden missing
A temperature column uses -999 for "sensor offline." Count how many readings are really
missing and show how the naive mean is corrupted.
isna() sees nothing; compare with (s == -999).sum().Category tangle
A country column mixes "USA", "usa", "U.S.A.", "United States", and similar. Use
value_counts() to surface them, then count the true categories.
Dupes & ranges
Find exact duplicate rows and any impossible ages (outside 0 to 120) in a small table.
duplicated().sum() and a boolean range mask.Tidy it up
A sales table has years as column headers. Name the tidy rule it breaks and reshape it to tidy form with
pd.melt.
A fully-worked solutions notebook walks through all five challenges in the same visual style. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on the data-cleaning mindset. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.