Fit a straight line to a 0/1 outcome and it will happily predict a probability of 1.4, or minus 0.2. Nonsense. Logistic regression fixes this by predicting the log-odds instead, which the S-shaped logistic curve squeezes back into a clean probability between 0 and 1.
Same five steps, new dialect: the line becomes an S-curve, least squares becomes maximum likelihood, R² becomes AUC and pseudo-R², and "predict" now means a probability plus a chosen threshold.
From a Line to an S-Curve: the Logit
The trick is to model something that can run from minus infinity to plus infinity: the log-odds. As the linear predictor rises, the log-odds rise, and the sigmoid maps them onto a probability that approaches 1 but never exceeds it (and approaches 0 but never goes below). The result is the familiar S:
Low credit scores sit high on the curve (likely to default), high scores sit low (likely to repay), and the curve
turns smoothly in between. Because the relationship is nonlinear in probability, there is no closed-form
β̂ = (XᵀX)⁻¹Xᵀy here; the coefficients are found by
maximum likelihood, the values that make the observed yes/no pattern most probable. In
Python it is still one line: logit("y ~ x1 + x2", data).fit().
Reading Coefficients as Odds Ratios
A logistic coefficient is a change in log-odds, which is hard to feel. Exponentiate it and it becomes an odds ratio, which is intuitive: the factor by which the odds of the outcome multiply for a one-unit rise in the predictor. An odds ratio of 1 means no effect, below 1 is protective, above 1 raises the risk.
Scaling makes them concrete. Each additional 50 points of credit score multiplies the odds of default by 0.40, a 60% reduction, while each 0.1 rise in debt-to-income multiplies them by 1.65. Loan amount's interval straddles 1, so it earns no conclusion. Just as with linear regression, plotting the coefficients (here as odds ratios on a log scale) with their confidence intervals separates the real drivers from the noise at a glance.
Thresholds, ROC, and When Logistic Misbehaves
The model outputs a probability; you choose the threshold that turns it into a yes/no decision. That choice is a business decision, not a statistical one, and it trades two errors against each other, summarized in a confusion matrix and the metrics precision and recall. The ROC curve shows every threshold at once, and its area (AUC) is a threshold-free score of how well the model ranks cases.
The two dots are the same model at different thresholds. At 0.5 it catches only 56% of defaults; drop the threshold to 0.3 and recall jumps to 75%, at the cost of more false alarms. This is where class imbalance bites: with 29% defaults, a model that predicts "never default" scores 71% accuracy while being useless, the accuracy paradox. And like linear regression, logistic has its own conditions to watch:
| Condition / hazard | Symptom | Remedy |
|---|---|---|
| Class imbalance | high accuracy, terrible recall on the rare class | tune the threshold; class weights; resample; judge by AUC / recall |
| Perfect / quasi separation | a predictor splits the classes; coefficients → ∞ | penalized (ridge/lasso) or Firth logistic regression |
| Nonlinearity in the logit | curved residual pattern vs a predictor | add polynomial terms or splines (see Regularization & Flexible Models) |
| Multicollinearity | unstable coefficients, huge SEs | drop/combine predictors; regularize (see Regularization & Flexible Models) |
The headline failure mode is separation: when a predictor perfectly divides the two classes, maximum likelihood pushes its coefficient toward infinity and the fit refuses to converge. The clean fix is regularization, a penalty that keeps coefficients finite, which is exactly the subject of the next chapter.
Real-World Example: Predicting Loan Default
We model default from credit score, debt-to-income, and loan amount, then run the workflow to a
decision.
One row per loan with credit_score, dti
(debt-to-income), loan_amount, and the binary default outcome.
Step 1, check the data. Before fitting, confirm the outcome really tracks the predictors. The default rate falls steadily as credit score rises:
The relationship is strong and monotonic, so credit score should be a powerful predictor. Note also that the classes are imbalanced, only 29% of loans default, which will matter when we choose a decision threshold.
Step 2, fit. We fit logit("default ~ credit_score + dti + loan_amount", loans) by maximum
likelihood.
Step 3, evaluate. The model ranks risk well, AUC = 0.82 and McFadden pseudo-R² = 0.24, and its odds ratios tell a clear story: credit score is strongly protective (each +50 points multiplies the odds of default by 0.40), while debt-to-income strongly raises risk.
Step 4, check the conditions. Logistic regression has its own hazards: perfect separation (no predictor here splits the classes cleanly, so the fit converges) and class imbalance (the 29% default rate means plain accuracy will mislead). Both are handled at the threshold, next.
Step 5, interpret, predict, and decide. A logistic model outputs a probability; you choose the threshold that turns it into a yes/no call. First, how the probability is computed, plug a profile into the linear predictor, then the sigmoid:
For a 650 credit score, 0.30 debt-to-income, and a $20k loan, the linear part works out to −0.83, so p̂ = 1 / (1 + e0.83) ≈ 0.30, a 30% default risk. The threshold then turns that number into approve-or-decline, trading two errors against each other:
| Threshold | Accuracy | Precision | Recall (defaults caught) | Use when |
|---|---|---|---|---|
| 0.50 (default) | 0.81 | 0.71 | 0.56 | false alarms are costly |
| 0.30 (cautious) | 0.74 | 0.54 | 0.75 | missing a default is costly |
Lowering the threshold from 0.5 to 0.3 is the imbalance remedy in action, catching more defaults at the cost of more false alarms:
Neither threshold is "correct." A lender who loses far more on a missed default than on a declined good customer should move the threshold down, trading precision for recall. The statistics give you the honest trade-off curve; the decision is yours to make on the costs. Note too that the naive 0.5 accuracy of 0.81 is barely above the 0.71 you would get by approving everyone, exactly why recall and AUC, not accuracy, are the metrics that matter under imbalance.
Logistic Regression in Machine Learning & AI
Logistic regression is not just a statistical tool, it is the atom of neural classification.
| Idea (this chapter) | In ML / AI it becomes | Example |
|---|---|---|
| Sigmoid + log-loss | A single neuron with cross-entropy loss | the output unit of a binary classifier |
| Logit for many classes | The softmax function | multi-class output layers everywhere in deep learning |
| Maximum likelihood | Minimizing cross-entropy by gradient descent | how virtually all classifiers are trained |
| Threshold & ROC / AUC | Operating-point selection & ranking metrics | fraud, medical triage, recommendation |
| Predicted probability | Model calibration | are the stated probabilities trustworthy? |
A logistic regression is a one-layer neural network with a sigmoid output and a cross-entropy loss, and stacking many such units with nonlinearities is, quite literally, how you get a deep classifier. The softmax that ends nearly every classification network is the multi-class logit; training by minimizing cross-entropy is maximum likelihood by another name. That is why logistic regression remains the indispensable baseline: it is interpretable, calibrated, and fast, and a deep model that cannot beat it is not earning its complexity. The chapter's cautions scale too, calibration (do predicted probabilities mean what they say?) and threshold choice under imbalance are front-line concerns for any deployed classifier, from credit scoring to medical AI.
Fit, interpret, and threshold in Python
The companion notebook fits the model with statsmodels and scikit-learn, plots the
fitted S-curve, reads the odds ratios with confidence intervals, draws the
ROC curve and confusion matrix, walks the precision/recall trade-off across thresholds, and
demonstrates the perfect-separation failure and its regularized fix, on
logistic-regression--loans.xlsx.
View opens the rendered notebook instantly (no setup). Open in Colab runs &
edits it live in your browser. To run locally, install numpy, pandas, scipy,
matplotlib, seaborn, statsmodels, scikit-learn, and
openpyxl and launch jupyter notebook.
🎓 Key Takeaways
- ✓Logistic regression predicts a probability by modeling the log-odds through the sigmoid; it is fit by maximum likelihood, not least squares.
- ✓Exponentiate a coefficient to get an odds ratio: >1 raises the odds, <1 is protective, and a CI spanning 1 means no effect.
- ✓You choose the threshold; the confusion matrix, precision/recall, and the ROC/AUC describe the trade-off across all of them.
- ✓Mind imbalance and separation: judge by recall/AUC not accuracy, and cure perfect separation with regularized (penalized) logistic regression.
- ✓Real data: +50 credit points cut default odds to 0.40×; lowering the threshold from 0.5 to 0.3 raised recall from 0.56 to 0.75 (AUC 0.82).
Practice Challenges
Five short challenges with statsmodels and scikit-learn.
Fit and read an odds ratio
Fit default ~ credit_score + dti and state the credit-score odds ratio (per 50 points) in a sentence.
np.exp(model.params * 50).Confusion matrix at 0.5
Turn predicted probabilities into labels at 0.5 and compute accuracy, precision, and recall.
confusion_matrix, classification_report.ROC, AUC, and a better threshold
Plot the ROC curve, report the AUC, and pick a threshold that catches at least 75% of defaults.
roc_curve, then read off the threshold at the target recall.The accuracy paradox
Show a "predict-all-repaid" baseline's accuracy, then beat it on recall with class weighting.
LogisticRegression(class_weight="balanced").Break it: perfect separation
Add a predictor that perfectly separates the classes; watch statsmodels fail, then fix it with penalized logistic.
LogisticRegression is L2-penalized by default and stays finite.A fully-worked solutions notebook walks through all five challenges, each verified in code. Try them yourself first, then compare.
Quiz: Test Yourself
Eight quick questions on logistic regression. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved, so you can hop back to the chapter and return anytime.
You can now model a yes/no outcome, read odds ratios, and choose a threshold. Regularization & Flexible Models adds the penalty that cures separation and overfitting, and the curves and splines that go beyond straight lines.