Given a dataset with dozens of columns, a beginner throws them all at the model. But many features are noise or duplicates, and they actively hurt: they dilute the signal, inflate variance, cost time and money to collect, and make the model harder to trust. Feature selection is the discipline of keeping only what earns its place.
Fewer, better features usually match or beat all of them: use a filter for a fast first screen, a wrapper or embedded method to account for redundancy, and always run the selection inside cross-validation, never on the whole dataset.
Why Select Features?
Every extra feature is a chance for the model to find a pattern that is not really there, the curse of dimensionality. Irrelevant features add variance, redundant ones destabilize coefficients (the multicollinearity of Multiple Linear Regression), and both slow training and complicate deployment. A smaller model is faster, cheaper, easier to explain, and often more accurate.
The notebook's marketing dataset makes the point starkly. It has 16 candidate features, but only 5 are truly informative (3 are redundant copies and 8 are pure noise). A model on all 16 scores a cross-validated 0.774; a model on just the 5 informative features scores 0.769, statistically the same. The eleven extra columns contributed nothing, so dropping them costs no accuracy and buys simplicity.
The Three Families
Every selection method is a filter, a wrapper, or an embedded method, distinguished by how much the model is involved.
In the notebook, the filter (ANOVA F-score) instantly ranks the informative features high and the noise near zero, but it also ranks the redundant copies high because it judges each feature alone. The wrapper (RFECV) trains a model on growing subsets and lands on a compact set. The embedded approach, an L1 (lasso) logistic model and a random forest's importances, selects as part of the fit, driving noise weights to zero and ranking the rest.
The Payoff, and the Leakage Trap
Because only the first few features carry signal, accuracy rises steeply then flattens as you add more, adding the noise columns brings no gain at all. The wrapper's cross-validated curve makes this visible and tells you where to stop.
Feature selection is a decision made from the data, so it belongs inside the pipeline,
refit within each fold, exactly like the preprocessing in The Machine Learning Workflow. In the notebook we hand the model
300 columns of pure noise: picking the five that correlate best with the target on the
whole dataset and then cross-validating gives a falsely predictive 0.56, while running
the same selection inside a Pipeline correctly returns 0.48, the coin-flip
baseline. Selecting features before the split is one of the most common ways analysts fool themselves.
Real-World Example: Trimming a Marketing Model
A marketing team tracks 16 signals per customer and wants to predict who will respond to a campaign, but suspects most of the columns are dead weight. The three families agree on the verdict.
One row per customer with informative signals (recency_days,
frequency, monetary, email_open_rate, web_visits_30d), three
redundant copies, eight noise columns, and the label responded.
| Approach | Result | CV accuracy |
|---|---|---|
| All 16 features | the naive baseline | 0.774 |
| Filter (F-score) | ranks the 5 informative + redundant high, noise near 0 | screen |
| Wrapper (RFECV) | auto-selects 8 features | ~0.77 |
| Embedded (lasso / RF) | zeros noise weights; ranks importances | ~0.77 |
| 5 informative only | the lean model | 0.769 |
| 8 noise columns only | no signal at all | ~0.49 (baseline) |
The conclusion writes itself: the 5 informative features match the full 16, the noise columns are worthless, and the filter's blind spot (ranking the redundant copies high) is exactly what the wrapper and embedded methods correct. The team can ship a model on a third of the columns with no loss of accuracy, and stop paying to collect the rest.
Feature Selection in Machine Learning & AI
Selection is a permanent part of the workflow, from small tabular models to the largest systems.
| Idea (this chapter) | In ML / AI it becomes |
|---|---|
| Drop irrelevant features | less overfitting, faster training, cheaper data collection |
| Drop redundant features | stabler coefficients, clearer interpretability (SHAP, importances) |
| Embedded selection (L1) | built into regularized models and used for high-dimensional data |
| Selection inside CV | a core defense against the data-leakage that breaks reproducibility |
| Curse of dimensionality | why dimensionality reduction (PCA) and embeddings exist |
Selection scales in two directions. On classic tabular problems it is the difference between a lean, explainable model and a bloated, leaky one, and doing it inside cross-validation is one of the most common fixes for results that fail to reproduce. At the frontier, the idea transforms: instead of hand-picking columns, deep networks learn their own features (representations) from raw data, and dimensionality reduction and embeddings (see Dimensionality Reduction and the vector search of Distance Metrics) compress thousands of dimensions into a few informative ones. The instinct is the same throughout, keep what carries signal, discard what does not.
Select features in Python
The companion notebook scores features with a filter (F-score, mutual information), searches subsets with a wrapper (RFECV and its accuracy curve), selects with embedded methods (L1 lasso and random-forest importances), proves a lean model matches the full one, and demonstrates the selection-leakage trap live, every cell explained.
View opens the rendered notebook instantly. Open in Colab runs it live. To run
locally, install numpy, pandas, scikit-learn, matplotlib,
seaborn, and openpyxl.
🎓 Key Takeaways
- ✓More features is not more information: irrelevant and redundant columns add variance, cost, and overfitting.
- ✓Filter methods score each feature vs the target (fast, model-free) but are blind to redundancy.
- ✓Wrapper methods (RFE / RFECV) search subsets with a model, accurate but slow; embedded methods (lasso, tree importances) select during training.
- ✓A lean model can match the full one: 5 informative features scored the same as all 16; the rest added nothing.
- ✓Select inside cross-validation: choosing features on the whole dataset leaks the answer and fakes accuracy (0.56 vs a true 0.48 on pure noise).
Practice Challenges
Five short challenges on the campaign table. Try them with scikit-learn before checking the solutions.
Filter by F-score
Rank the features by ANOVA F-score; name the top and confirm the noise columns score low.
f_classif(X, y) then sort.Wrapper: how many features?
Use RFECV to choose the number of features automatically.
RFECV(estimator, cv=5).n_features_.Embedded selection
Fit an L1 logistic model and a random forest; report what L1 zeros and the forest's top feature.
penalty="l1", solver="liblinear"; feature_importances_.Fewer features, same accuracy
Compare all features to the 5 informative ones by CV accuracy.
cross_val_score on X[cols].Selection leakage
Show that selecting features on the whole dataset inflates accuracy versus selecting inside a pipeline.
SelectKBest inside a Pipeline for the honest score.A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on feature selection. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
Selection keeps the right features; the next question is how a model actually learns from them. Optimization & Gradient Descent opens the engine room, loss functions, gradient descent, SGD and Adam, and the learning-rate choices that power every model from logistic regression to deep nets.