You have the workflow and a way to measure closeness; now meet the models. These five are the first tools a practitioner reaches for on tabular data, quick to train, easy to reason about, and often good enough to beat far fancier methods. Each embodies a different idea of what a "pattern" is.
There is no single best algorithm: each draws a different boundary shape with different strengths, so standardize your features, cross-validate a handful of models, and pick the one that fits your data, your need for interpretability, and your speed budget.
The Five Workhorses
Each algorithm answers "which class?" with a different mechanism. Knowing the mechanism tells you when it will shine and what it needs from your data.
| Algorithm | How it decides | Needs scaling? | Best when |
|---|---|---|---|
| Logistic regression | a linear score turned into a probability | yes | you want a fast, interpretable baseline |
| K-nearest neighbors | majority vote of the closest points | yes | the boundary is irregular; data is small |
| Decision tree | a sequence of yes/no questions | no | you need readable rules |
| Naive Bayes | multiply per-feature probabilities | no | text / high-dimensional; speed matters |
| Support vector machine | the widest-margin boundary (kernels curve it) | yes | high dimensions; complex boundaries |
Three of the five, KNN, SVM, and logistic regression, measure distances or weighted sums, so they require
standardized features (see Distance Metrics). Trees and naive Bayes work on each feature separately,
so they do not. In the notebook every scaling-sensitive model is wrapped in a Pipeline with a
StandardScaler, exactly the leakage-safe habit from the Machine Learning Workflow chapter.
How Each Draws a Boundary
The clearest way to see how the algorithms differ is to train them on two features and shade the region each predicts as "disease." The same points produce five very different decision boundaries.
The notebook draws these live. The shapes tell the story: KNN is local and wiggly (it follows clusters of neighbors); the decision tree is built from axis-aligned rectangles (one per split); naive Bayes is a smooth quadratic; the RBF SVM bends flexibly with a margin; and logistic regression is a single straight line. Wigglier boundaries capture local detail but risk overfitting; straighter ones generalize but may underfit, the bias-variance trade-off made visible.
Interpretability: The Decision Tree
One of the five stands apart for transparency. A decision tree is a flowchart of yes/no questions you can read top to bottom, which is why it is trusted in medicine and lending where a decision must be explained.
In the notebook, plot_tree renders the real thing and the feature importances rank
which questions mattered most: age carries the strongest splits (importance 0.55), followed by
maximum heart rate. You can trace any patient from the root to a leaf and state exactly why the model predicted what
it did, a property none of the other four share so cleanly. The price is accuracy: a single tree is a coarse
step-function, which the next chapter's ensembles fix by combining many trees.
Real-World Example: A Head-to-Head Race
With 700 patients and six risk factors, which algorithm predicts heart disease best? Cross-validated accuracy settles it, and delivers the field's most important lesson.
One row per patient with age, resting_bp,
cholesterol, max_hr, glucose, and bmi, plus the label
heart_disease (1 = disease, 0 = healthy).
| Algorithm | 5-fold CV accuracy | |
|---|---|---|
| Logistic regression | 0.773 | leads |
| SVM (RBF) | 0.766 | |
| Naive Bayes | 0.759 | |
| K-nearest neighbors | 0.750 | |
| Decision tree | 0.683 | trails |
| Baseline (predict majority) | 0.647 | floor |
The margins are close, and that is the point: no algorithm is universally best, the "no free lunch" theorem. Here the linear models edge ahead, the single tree trails (a weak learner on its own), and all beat the majority-class floor. The professional move is not to crown a favorite in advance but to cross-validate a handful and choose using the numbers and your constraints, speed, interpretability, and how the boundary should behave. And because every one of these has a regression twin (the notebook predicts a patient's maximum heart rate with a KNN regressor, a tree regressor, and linear regression), the same five ideas cover predicting numbers too.
These Algorithms in ML & AI
None of these are obsolete. They remain the default choice for tabular data and the conceptual building blocks of much larger systems.
| Algorithm (this chapter) | Role in modern ML / AI |
|---|---|
| Logistic regression | the universal baseline; the softmax output layer of a neural network is multinomial logistic regression |
| Decision tree | the building block of gradient boosting (XGBoost, LightGBM), still state of the art on tabular data |
| K-nearest neighbors | similarity search and few-shot retrieval over embeddings |
| Naive Bayes | fast, strong text baselines (spam filtering) |
| Support vector machine | powerful on small, high-dimensional datasets via kernels |
Deep learning dominates images, text, and audio, but on tabular data, the spreadsheets that run most businesses, ensembles of the humble decision tree (gradient-boosted trees) still routinely beat neural networks. Knowing when a simple, fast, interpretable model is the right answer, rather than reaching for the biggest network, is a genuine mark of expertise. And the connection runs deep: logistic regression is a one-layer neural network, and the final classification layer of a large language model is exactly the multinomial logistic regression in this chapter. Master these five and you understand both the workhorses of applied ML and the primitives inside modern AI.
Train all five in Python
The companion notebook fits every algorithm with scikit-learn, draws the five
decision boundaries side by side, renders a readable decision tree with its
feature importances, races the models with cross-validated accuracy, and shows the
regression twin of each, every cell explained.
View opens the rendered notebook instantly. Open in Colab runs it live. To run
locally, install numpy, pandas, scikit-learn, matplotlib,
seaborn, and openpyxl.
🎓 Key Takeaways
- ✓Five workhorses: KNN (neighbor vote), trees (yes/no rules), naive Bayes (probabilities), SVM (max margin), logistic regression (linear score).
- ✓Each draws a different boundary: KNN wiggly, tree rectangular, naive Bayes and SVM curved, logistic straight, the bias-variance trade-off made visible.
- ✓KNN, SVM, and logistic regression need standardized features; trees and naive Bayes do not.
- ✓No free lunch: no model wins on every dataset, cross-validate several and weigh accuracy against interpretability and speed.
- ✓Every classifier has a regression twin, and these algorithms remain the go-to for tabular data and the primitives inside modern AI.
Practice Challenges
Five short challenges on the patient table. Try them with scikit-learn before checking the solutions.
Train a KNN classifier
Scale the features, fit KNN (k = 15), and report test accuracy.
make_pipeline(StandardScaler(), KNeighborsClassifier(15)).Read a tree's importances
Fit a shallow decision tree and print its most important feature.
tree.feature_importances_.Head-to-head
Compare logistic regression, a tree, and KNN by 5-fold cross-validated accuracy.
cross_val_score(m, X, y, cv=5).mean().SVM: the kernel matters
Compare a linear SVM and an RBF SVM by cross-validated accuracy.
SVC(kernel="linear") vs SVC(kernel="rbf").The regression twin
Predict max_hr (a number) with a KNN regressor and linear regression; compare R².
KNeighborsRegressor, LinearRegression, scoring="r2".A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on the core algorithms. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
A single decision tree trailed the field, but trees have a superpower when you combine them. the Ensemble Methods chapter shows how bagging, random forests, and boosting (XGBoost and friends) turn many weak trees into the strongest models on tabular data.