Chapter 109 · Solutions
Feature Selection · Solutions
Five challenges, each verified in code.
Chapter 109 · Practice Challenges, Solved¶
Five exercises on feature selection, worked in scikit-learn on the Chapter 109 marketing table.
In [1]:
import numpy as np, pandas as pd, matplotlib.pyplot as plt
import seaborn as sns # seaborn = high-level statistical plots (heatmaps, pairplots, count/bar plots)
from matplotlib.colors import ListedColormap
EM="#4338ca"; DEEP="#3730a3"; LIGHT="#c7d2fe"; INK="#1a2138"; GRID="#e6e9f2"; RED="#ef4444"; AMBER="#d97706"; GREEN="#059669"; BLUE="#2563eb"; PUR="#9333ea"; GREY="#94a3b8"; SLATE="#475569"; ORG="#4338ca"; CYAN="#0891b2"
plt.rcParams.update({"figure.facecolor":"white","axes.facecolor":"white","figure.dpi":110,"font.size":11,
"axes.edgecolor":GRID,"axes.grid":True,"grid.color":GRID,"axes.axisbelow":True,"axes.spines.top":False,
"axes.spines.right":False,"axes.titlesize":12,"axes.titleweight":"bold","legend.frameon":False})
sns.set_style("whitegrid")
BASE="https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
In [2]:
import warnings; warnings.filterwarnings('ignore')
from sklearn.model_selection import cross_val_score
from sklearn.feature_selection import SelectKBest, f_classif, RFECV
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline, Pipeline
try: df = pd.read_excel('../../data/feature-selection--campaign.xlsx', sheet_name='Data')
except FileNotFoundError: df = pd.read_excel(BASE + 'feature-selection--campaign.xlsx', sheet_name='Data')
feat = [c for c in df.columns if c not in ('customer_id','responded')]
X, y = df[feat], df['responded']
print('loaded', df.shape, '|', len(feat), 'features')
loaded (900, 18) | 16 features
CHALLENGE 1
Filter by F-score
Rank the features by ANOVA F-score; name the top feature and confirm the noise columns score low.
In [3]:
s = pd.Series(f_classif(X, y)[0], index=feat).sort_values(ascending=False)
print('top:', s.index[0]); print('mean F-score of noise columns:', round(s[[f for f in feat if 'noise' in f]].mean(),2))
top: recency_days mean F-score of noise columns: 1.54
Solution. A filter scores each feature independently; the informative ones dominate and the noise columns are near zero.
CHALLENGE 2
Wrapper: how many features?
Use RFECV to choose the number of features automatically.
In [4]:
rfecv = RFECV(LogisticRegression(max_iter=1000), cv=5).fit(StandardScaler().fit_transform(X), y)
print('RFECV kept', rfecv.n_features_, 'of', len(feat), 'features')
RFECV kept 8 of 16 features
Solution. RFECV cross-validates each subset size and keeps the smallest set that maximizes accuracy.
CHALLENGE 3
Embedded selection
Fit an L1 logistic model and a random forest; report what L1 zeros and the forest's top feature.
In [5]:
l1 = make_pipeline(StandardScaler(), LogisticRegression(penalty='l1', solver='liblinear', C=0.12, max_iter=2000)).fit(X, y)
z = [f for f in feat if abs(l1[-1].coef_[0][feat.index(f)]) < 1e-6]
rf = RandomForestClassifier(n_estimators=300, random_state=0).fit(X, y)
print('L1 zeroed:', z); print('RF top feature:', pd.Series(rf.feature_importances_, index=feat).idxmax())
L1 zeroed: ['noise_1', 'noise_6', 'noise_7', 'noise_8'] RF top feature: recency_weeks
Solution. L1 drives weak coefficients to exactly zero; tree importances rank the informative features on top.
CHALLENGE 4
Fewer features, same accuracy
Compare all features to the 5 informative ones by CV accuracy.
In [6]:
info = ['recency_days','frequency','monetary','email_open_rate','web_visits_30d']
clf = lambda cols: cross_val_score(make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)), X[cols], y, cv=5).mean()
print(f'all {len(feat)}: {clf(feat):.3f} | 5 informative: {clf(info):.3f}')
all 16: 0.774 | 5 informative: 0.769
Solution. The lean 5-feature model matches the full one; the extra 11 columns add nothing.
CHALLENGE 5
Selection leakage
Show that selecting features on the whole dataset inflates accuracy versus selecting inside a pipeline.
In [7]:
rng = np.random.default_rng(0); Xj = pd.DataFrame(rng.normal(size=(len(y), 300)))
best = SelectKBest(f_classif, k=5).fit(Xj, y).get_support(indices=True)
leaky = cross_val_score(LogisticRegression(max_iter=1000), Xj.iloc[:,best], y, cv=5).mean()
honest = cross_val_score(Pipeline([('s', SelectKBest(f_classif, k=5)), ('lr', LogisticRegression(max_iter=1000))]), Xj, y, cv=5).mean()
print(f'select-then-CV (leaky): {leaky:.3f} vs pipeline (honest): {honest:.3f} on pure noise')
select-then-CV (leaky): 0.561 vs pipeline (honest): 0.484 on pure noise
Solution. On 300 noise columns the leaky path looks predictive; the pipeline (selection refit per fold) correctly returns baseline.
Statistics, Data Science and AI: A Visual Handbook · © 2026 John Fisher