Everything in Preparing Data for Analysis so far cleaned and reshaped the data. This chapter prepares it for a model: turning columns into features, the numeric inputs an algorithm actually learns from. It is often the highest-leverage step in the whole pipeline.
Good features often beat a fancier algorithm. As Andrew Ng put it, "applied machine learning is basically feature engineering." It is where domain knowledge pays off: a well-chosen ratio (price per square foot), a date part (day of week), or a clean encoding can do more than any amount of model tuning. It sits between cleaning (the earlier chapters) and modeling.
Encoding Categorical Variables
Most models need numbers, not strings, so categories have to be converted, and how you convert them depends on whether the category has a real order (see Levels of Measurement). The wrong encoding quietly lies to the model.
| Method | When | Pros | Cons / risk |
|---|---|---|---|
| Ordinal / label | Ordinal data with real order (also fine for trees) | One compact column; keeps order | False order on nominal data; LabelEncoder is for the target, not features |
| One-hot | Nominal, low/medium cardinality | No false order; works for linear & distance models | Column explosion at high cardinality; dummy-variable trap (use drop='first' for linear models) |
| Target / mean | High-cardinality nominal | One column, scales to thousands of categories | Target leakage if naive; must cross-fit or smooth |
| Frequency / hashing | Very high cardinality, quick | Compact, no target leakage | Collisions mix categories |
Label-encoding nominal data tells a linear/KNN/SVM model that blue (2) is twice green (1)
and farther from red (0), a fake distance; trees tolerate it, but everything else doesn't. And
pd.get_dummies is fine for quick exploration but forgets its categories, so
train and test can end up with mismatched columns. For real modeling use scikit-learn's
OneHotEncoder (with handle_unknown='ignore') inside a pipeline.
Scaling & Binning
Two more workhorses, both met earlier in the book. Scaling (see Standardization & Z-Scores) puts numeric features on a comparable footing; binning (see Frequency Distributions) turns a continuous variable into groups.
| Scaler | Output | Outliers | Use when |
|---|---|---|---|
| StandardScaler | mean 0, sd 1 (z-score) | Sensitive | General default; roughly symmetric features |
| MinMaxScaler | bounded [0, 1] | Sensitive | Need a bounded range (neural-net inputs, pixels) |
| RobustScaler | median 0, scaled by IQR | Robust | Data with outliers (see Detecting & Treating Outliers) |
Scale for distance- and gradient-based models (KNN, K-means, SVM, neural nets, PCA, regularized regression). Tree-based models do not need scaling, they split on thresholds, which monotonic scaling leaves unchanged.
| Binning | Bins | How | Note |
|---|---|---|---|
pd.cut | Equal width | Equal-size intervals over the range (or custom edges) | Counts per bin vary |
pd.qcut | Equal frequency | Quantile-based; each bin has ~equal count | Widths vary (quartiles, deciles) |
KBinsDiscretizer | Configurable | strategy='uniform' / 'quantile' / 'kmeans' | Can one-hot the bins automatically |
Binning buys interpretability (age β child/adult/senior) and can inject simple nonlinearity, but it loses resolution and the cut points are arbitrary. Modern gradient-boosted models find their own splits, so for raw predictive power binning is often unnecessary, reach for it mainly for reports and rules, not accuracy.
Choosing the Encoding
Picking an encoding comes down to a couple of questions about the category:
The Pipeline & Created Features
Every fitted transform, encoders, scalers, target encoding, imputers (see Handling Missing Data), outlier fences
(see Detecting & Treating Outliers), and transform parameters (see Transformations), must learn its parameters from the
training data only and then be applied to the test data. A scikit-learn
ColumnTransformer applies different transforms to different columns, and wrapping it in a
Pipeline makes the fit-on-train-only rule automatic, even inside cross-validation. Leaking the
test set in is the most common way a model looks great in development and fails in the wild.
Beyond encoding and scaling, features are also created: ratios and differences (BMI, price per square foot), interaction and polynomial terms, group aggregations (average purchase per customer), date parts from a timestamp (see Combining & Reshaping Data), and text features like length or word count. This is where domain knowledge turns raw columns into signal.
Models are commodities; features are not. The same algorithm with better features beats a fancier one with worse features, which is why practitioners spend so much time here. This chapter is the introductory tour; the machine-learning part returns to feature engineering in depth, with automatic feature learning and the full encoder/scaler toolkit.
Feature Engineering in Machine Learning & AI
Feature engineering is where domain knowledge meets the model, and for classic tabular problems it is often the single biggest lever on accuracy. scikit-learn packages the moves so they run inside a leakage-safe pipeline.
| Move | The scikit-learn tool | What it produces |
|---|---|---|
| Encode categories | OneHotEncoder, OrdinalEncoder, TargetEncoder | Numeric columns a model can read, one-hot for low cardinality, target encoding for high |
| Derive & transform | FunctionTransformer, custom transformers | Calendar parts, ratios, and logs, wrapped so they run at train and predict time alike |
| Route it all together | ColumnTransformer inside a Pipeline | Different treatment per column, fit on training data only, no leakage |
Any feature that learns from the data, target encoding, an aggregate, a fitted transform, must be fit
on the training set only, or information leaks from test to train. And never build a feature
from information that would not exist at prediction time. Wrapping every step in a ColumnTransformer
and Pipeline is what makes this automatic and reproducible.
Real-World Example: From Raw Orders to Features
Here is the raw material of feature engineering: 600 orders with just five columns, a timestamp, a customer, a category, and an amount. The companion notebook engineers ten features from them, calendar parts (hour, day-of-week, is-weekend) from the timestamp, a log of the skewed amount, one-hot columns for the category, and a per-customer order count, turning five raw columns into a model-ready table.
One row per order: order_id, order_datetime,
customer_id (repeats, so per-customer aggregates are possible), product_category, and
amount. It is deliberately raw, the useful features are the ones you derive from it.
Bring it to life in Python
The companion notebook one-hot encodes a category two ways (and shows the cardinality explosion), encodes an ordinal feature correctly while demonstrating the nominal trap, compares Standard, MinMax, and Robust scalers against an outlier, bins a variable by width and by frequency, and assembles a leakage-safe ColumnTransformer fit on training data only.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas,
scikit-learn, matplotlib and launch jupyter notebook.
π Key Takeaways
- βFeatures are the model's inputs; good feature engineering often beats a fancier algorithm.
- βEncode by type: ordinal β ordinal encoding; low-cardinality nominal β one-hot; high-cardinality β target (cross-fit).
- βDon't label-encode nominal data (it invents false order); use
OneHotEncoder, not rawget_dummies, for real ML. - βScale for distance/gradient models (not trees);
RobustScalerresists outliers; bin mainly for interpretability. - βFit every transform on training data only; a
ColumnTransformerin aPipelineenforces it.
Practice Challenges
Five short challenges, beginner to intermediate. Try them on paper or in Python before checking the solutions.
One-hot a category
One-hot encode a color column with get_dummies. How many columns result, and
why is one-hot (not integer labels) right for a nominal feature in a linear model?
Encode an ordinal
Encode a rating column (Poor, Fair, Good, Excellent) so the order is preserved, and explain
why alphabetical label-encoding would be wrong.
OrdinalEncoder(categories=[[...in order...]]).Choose a scaler
Scale [20,22,21,23,22,24,1000] with Standard, MinMax, and Robust scalers. Which keeps the
six normal values spread out, and why?
Bin a variable
Bin an income column into 4 groups two ways: equal-width (pd.cut) and equal-frequency
(pd.qcut). Show the counts and explain the difference.
Build a safe transformer
Build a ColumnTransformer that scales numeric columns and one-hot encodes a categorical
one, fit it on a training split, and transform the test split. Why fit on train only?
A fully-worked solutions notebook walks through all five challenges in the same visual style. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on feature engineering. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.