Gradient Boosting for Tabular Regression

Most of the effort in a tabular modeling project goes into the model. This chapter measures what each decision was actually worth, in seconds, and the ordering is close to the reverse of the usual one.

After removing a duplicated export block, 384 cancellations recorded as a delivery time of zero and 211 orders with a GPS distance of 999 kilometers, 41,410 orders remain. Mean delivery time is 51.0 minutes with a standard deviation of 17.9.

Three windows, in time order rather than sampled at random, because the model will always be predicting orders that have not happened yet: days 0 to 69 to train (34,462 orders), 70 to 76 to choose (3,425), 77 to 83 to report (3,523). The middle window does the choosing. The last one is opened once.

The baseline is not an aspiration, it is a hurdle: a linear regression on the raw columns of the export. Every number below is measured against it.

Test RMSE in minutes. Both models use default settings.

Read the bottom row against the right column. The features are worth roughly twenty-four times what the model family is worth, and a linear regression with good features beats a gradient boosting machine with poor ones by more than a minute and a half.

Then a random search over twenty hyperparameter configurations, each fitted on the training window and scored on the validation window, with the winner scored once on the test window.

Four seconds is real and it is not nothing. Set it beside the 101 seconds the fourth engineered feature was worth and the hierarchy is settled. It is also worth noticing that the configuration ranked tenth on the validation window did best on the test window: among the leaders, the search is ranking noise.

Four features were added: a load ratio (pending orders per available courier), a peak-hour flag, a weekend flag, and the restaurant's own recent median delivery time built from its past orders only. Drop them one at a time and refit.

One feature carried essentially all of it, and the pattern is not an accident. A boosted tree can split on orders_pending and couriers_available separately and approximate their ratio; it can split on hour_of_day and find the lunch and dinner peaks by itself. Writing those out helps readability and almost nothing else.

A note on how that feature was built. The expanding median uses each restaurant's previous orders and stops there. Taking a plain groupby(...).median() over the whole file would have been shorter, slightly more accurate on this test set, and wrong, because it would use next month's orders to predict this month's. That is the trap Chapter 191 is about, and it is easiest to fall into exactly here, in the most valuable feature in the project.

Both cost money and they are different purchases, so it is worth answering rather than guessing. Fit on growing subsets of the training window and watch the two errors converge.

The curves have met. The training error has risen and the test error has fallen until the gap is a quarter of a minute, and the last doubling of the data bought 0.12 minutes, about seven seconds.

More data will not help, and neither will a more flexible model. What is left is not error a model could remove with better use of these features. It is variation in the world: two identical orders from the same restaurant, the same distance, the same minute, genuinely arrive at different times.

Which changes the question the project is answering. If the uncertainty cannot be removed, it has to be communicated, and that is a different piece of work from making the model better.

An average error of 7.9 minutes is a summary. The question is whether it is the same 7.9 minutes everywhere.

The bias is near zero everywhere and the spread is nowhere near constant. Off peak the model is good to about five minutes. When the zone has more than two pending orders per courier it is good to about twelve. A factor of two and a half, and predictable from features already in the model.

A model with this property is not broken. It is reporting something true: some orders are inherently harder to forecast than others. What would be broken is a system that shows the same kind of number for both.

The point prediction is the best guess at the middle of the distribution, so 46.5 percent of orders arrive after it. That is not a defect. It is what a conditional mean is.

With late costing three times as much as early, the quantity that minimizes expected cost is not the mean but the quantile at τ = 3 ÷ (3 + 1) = 0.75, which is the newsvendor result, and gradient boosting can be fitted directly to that loss.

Cost is three times the minutes late plus one times the minutes early, averaged over orders.

The sweep finds its minimum exactly where the theory says it should, at the 75th percentile, and costs about 13 percent less than the current flat 8-minute policy.

There is a genuine tension here to hand back rather than resolve. The 75th percentile is cost-optimal and leaves 25 percent of orders late, against a service target of 15. The 85th hits the target and costs a little more. Which one is right depends on what the service promise is worth, which is a question for the business. What an analysis can do is make sure whichever they choose is delivered well.

The conditional 85th percentile pads by 6.73 minutes on average. Give a flat buffer the same average padding and compare. If the model's uncertainty estimates are worth anything, the two will differ in who ends up late.

Read the bottom row first. The two policies have almost the same overall late rate, 15.0 percent against 15.4. On the headline number they are the same product.

Now read the column above it. The flat buffer keeps its promise to 97 percent of easy orders and breaks it for a quarter of the hard ones. Its late rate ranges over 22 percentage points across these six segments; the conditional version ranges over three.

That difference is invisible in RMSE, invisible in MAE, and invisible in the overall late rate. It is the difference between a service level and an average of service levels, and it falls on the same customers every time: the ones who live further out, order at dinner time, or order when it is raining.

The current +8 policy has the same shape. Its overall figure of 11.8 percent is comfortably inside the target, and it is failing that target for 21.8 percent of the highest-load orders. The headline number is hiding it.

• ✓Spend the effort in proportion to what it buys. Here that was 101 seconds, 17 seconds, and 4 seconds, and the usual order of attention is close to the reverse.
• ✓Engineer what the model cannot derive. A ratio of two present columns is convenience; an aggregate over other rows is information.
• ✓Quote the test score, not the validation score. The one you selected on is optimistic by however much the search exploited it, and among the leaders the ranking is noise.
• ✓Draw the learning curve before buying more data. It answers a question people usually argue about instead.
• ✓Check the residual spread by segment, not just the residual mean. Unbiased everywhere is compatible with useless somewhere.
• ✓An average service level is not a service level. A policy can hit 15 percent overall while failing a quarter of the customers who need it most.
• ✓When the error is irreducible, communicate it. The remaining work is not a better point estimate, it is the right interval around it.