Learning a Pricing Policy
import numpy as np, pandas as pd, warnings
warnings.filterwarnings("ignore")
import matplotlib.pyplot as plt
import statsmodels.api as sm
plt.rcParams.update({"figure.dpi":110,"font.size":11,"axes.spines.top":False,"axes.spines.right":False,
"axes.grid":True,"grid.alpha":0.22,"axes.titleweight":"bold","axes.titlesize":12.5,"axes.titlelocation":"left"})
VI, DK, LT, MUT, GD, RD = "#6d28d9", "#4c1d95", "#a78bfa", "#94a3b8", "#047857", "#dc2626"
BASE_URL = "https://raw.githubusercontent.com/johnfisher-ai/Statistics-Data-Science-AI-Visual-Book/main/data/"
fn = "capstone-reinforcement-learning-pricing.xlsx"
def load(sheet):
try: return pd.read_excel("../../data/" + fn, sheet_name=sheet)
except FileNotFoundError: return pd.read_excel(BASE_URL + fn, sheet_name=sheet)
A = load("Log_A"); B = load("Log_B"); P = load("Products"); notes = load("Notes")
MULT = np.array([0.85, 0.90, 0.95, 1.00, 1.05, 1.10, 1.15])
LIST = P.set_index("product_id").list_price.to_dict()
COST = P.set_index("product_id").unit_cost.to_dict()
SESS = B.groupby("product_id").sessions.mean().to_dict()
print(f"Log A {len(A):,} product-days Log B {len(B):,} product-days {len(P)} products")
Log A 5,600 product-days Log B 3,360 product-days 40 products
Step 1 · The brief¶
for line in notes.Notes.fillna(""):
print(line)
CAPSTONE 35 - LEARNING A PRICING POLICY
THE REQUEST. The trading team wants an agent that sets prices day by day instead of
the cost-plus rule the site has used for years. They have twenty weeks of history and
would like to know what the agent would have earned on it.
WHAT IS IN THE FILE
Log_A 140 days of business as usual. Price was set by a fixed cost-plus markup by
category, with a sitewide sale every fifth week. No other price variation.
Log_B 84 days of a randomized price experiment run afterwards on the same products.
Each WEEK each product was assigned one of seven price multipliers at random
and held it for seven days, and THE ASSIGNMENT PROBABILITY WAS LOGGED. That
column is not decoration, and neither is the holding period.
THE THING TO WORK OUT FIRST. Whether the twenty weeks of history can tell you what
happens to demand when you change a price. Fit the obvious regression on Log_A before
reading any further and look at the sign of the price coefficient.
A DETAIL THE TRADING TEAM MENTIONED IN PASSING. Customers who get used to a discount
stop regarding the full price as the price, and the lift from a promotion fades over
a few weeks even if the promotion continues. Whether that matters is an empirical
question and it decides whether this is a bandit problem or a sequential one.
AN ANSWER KEY is included (Truth), because the demand model was simulated. Read
Truth_ReadMe before opening it, and do not fit anything to it.
Step 2 · First look¶
Before the regression the brief asks for, look at what the two logs contain: which prices were ever charged, and what happened to conversion on the days they changed.
mA = (A.price / A.product_id.map(LIST)).round(2)
mB = (B.price / B.product_id.map(LIST)).round(2)
dA = A.groupby("day").agg(units=("units", "sum"), sessions=("sessions", "sum"))
dA["conv"] = dA.units / dA.sessions
dA["sale"] = mA.groupby(A.day).mean().lt(1)
week = dA.index // 7
sale_weeks = set(week[dA.sale])
after = week.isin({w + 1 for w in sale_weeks}) & ~dA.sale
usual = ~week.isin(sale_weeks | {w + 1 for w in sale_weeks})
print(f"Log A {len(A):,} product-days over {A.day.nunique()} days, at "
+ ", ".join(f"{k:.2f} x list on {v:,}" for k, v in mA.value_counts().sort_index().items()))
print(f"Log B {len(B):,} product-days over {B.day.nunique()} days, at {mB.nunique()} multipliers "
f"from {mB.min():.2f} to {mB.max():.2f}, each on {mB.value_counts().min()} to {mB.value_counts().max()}")
print(f"Log A conversion {dA.conv[usual].mean():.1%} in ordinary weeks {dA.conv[dA.sale].mean():.1%} in the "
f"{len(sale_weeks)} sale weeks {dA.conv[after].mean():.1%} in the week after a sale")
print(f"list prices run from {P.list_price.min():.2f} to {P.list_price.max():.2f} across the {len(P)} products")
Log A 5,600 product-days over 140 days, at 0.85 x list on 1,120, 1.00 x list on 4,480 Log B 3,360 product-days over 84 days, at 7 multipliers from 0.85 to 1.15, each on 315 to 651 Log A conversion 20.4% in ordinary weeks 37.5% in the 4 sale weeks 19.0% in the week after a sale list prices run from 6.60 to 253.76 across the 40 products
What this shows. Log A charged only two prices in 140 days: list price on 4,480 product-days and 0.85 times list on 1,120, the four sale weeks. Log B, run afterwards, used all seven multipliers from 0.85 to 1.15, each on between 315 and 651 product-days. In Log A, conversion runs at 20.4% in ordinary weeks, jumps to 37.5% in the sale weeks and falls to 19.0% in the week after each sale. That dip is the trading team's remark about discounts showing up in the data, and the brief says it decides whether this is a bandit problem or a sequential one.
fig, ax = plt.subplots(1, 2, figsize=(12.6, 4.4))
# left: Log A's daily conversion, with the sale weeks shaded
for w in sorted(sale_weeks):
ax[0].axvspan(w * 7 - 0.5, w * 7 + 6.5, color=LT, alpha=0.30, lw=0)
ax[0].plot(dA.index, dA.conv * 100, color=VI, lw=1.8)
ax[0].axhline(dA.conv[usual].mean() * 100, color=DK, lw=1.2, ls="--")
ax[0].text(1, dA.conv[usual].mean() * 100 - 3.2, "ordinary weeks", color=DK, fontsize=9.5)
ax[0].text((min(sale_weeks)) * 7 + 3, dA.conv.max() * 100 + 1.2, "sale", ha="center", color=VI,
fontsize=9.5, fontweight="bold")
ax[0].set_ylim(10, dA.conv.max() * 100 + 5)
ax[0].set_xlabel("day of Log A"); ax[0].set_ylabel("conversion, all products (%)")
ax[0].set_title("Log A: one price change, every fifth week")
# right: the share of product-days at each price multiplier, in each log
xs = np.arange(len(MULT)); w_ = 0.38
shA = mA.value_counts(normalize=True).reindex(MULT, fill_value=0)
shB = mB.value_counts(normalize=True).reindex(MULT, fill_value=0)
ax[1].bar(xs - w_ / 2, shA.values * 100, w_, color=MUT, label="Log A, cost-plus history")
ax[1].bar(xs + w_ / 2, shB.values * 100, w_, color=VI, label="Log B, randomized")
for x, v in zip(xs - w_ / 2, shA.values):
if v == 0:
ax[1].text(x, 1.2, "0", ha="center", fontsize=9, color=DK)
ax[1].set_xticks(xs); ax[1].set_xticklabels([f"{m:.2f}" for m in MULT])
ax[1].set_xlabel("price as a multiple of list"); ax[1].set_ylabel("% of the log's product-days")
ax[1].legend(fontsize=9, loc="upper right"); ax[1].grid(axis="x", alpha=0)
ax[1].set_title("The prices each log ever charged")
plt.tight_layout(); plt.show()
Left: conversion across all 40 products on each day of Log A, with the sale weeks shaded. Conversion nearly doubles the day a sale starts and drops below its usual level the day it ends, then recovers over the following week. Every price change in this log arrives with a sale, so the effect of the price can't be separated from whatever else the sale brought, and Step 3's regression runs into this once it compares each product with itself. Right: the share of each log's product-days at each price. Log A has nothing at five of the seven multipliers, which matters in Step 4, because an offline estimate for a policy that charges one of those prices has no rows to draw on in Log A.
Step 3 · The regression the trading team already ran¶
Twenty weeks of history, price and units on every product every day. Fit the obvious model.
def demand_fit(L, spec):
g = L[L.sessions > 0].copy()
X = pd.DataFrame(index=g.index)
if "fe" in spec:
X = pd.get_dummies(g.product_id.astype(str), prefix="p", drop_first=True).astype(float)
if "level" in spec:
X["log_price_vs_list"] = np.log(g.price/g.product_id.map(LIST)).values
if "gap" in spec:
X["log_price_vs_reference"] = np.log(g.price/g.reference_price).values
if "raw" in spec:
X["log_price"] = np.log(g.price).values
if "wknd" in spec:
X["weekend"] = (g.day_of_week >= 5).astype(float).values
X = sm.add_constant(X)
return sm.GLM(np.c_[g.units, g.sessions - g.units], X, family=sm.families.Binomial()).fit()
r = demand_fit(A, ["raw"])
print("LOG A, conversion against log price, across all products")
print(f" price coefficient {r.params['log_price']:+.3f} (standard error {r.bse['log_price']:.3f})")
print("\n Read literally: raising the price raises the chance somebody buys.")
LOG A, conversion against log price, across all products price coefficient +0.766 (standard error 0.005) Read literally: raising the price raises the chance somebody buys.
That is not a small effect or a marginal result. It is a firmly estimated, tightly bounded, completely wrong number, and the reason is in the product table: the expensive products are the good ones.
q = P.assign(dear=P.list_price > P.list_price.median())
print(q.groupby("dear")[["list_price","unit_cost","quality_index"]].mean().round(2).to_string())
print(f"\ncorrelation between list price and quality index: "
f"{np.corrcoef(P.list_price, P.quality_index)[0,1]:.3f}")
list_price unit_cost quality_index dear False 14.91 8.65 -0.73 True 77.92 44.98 0.61 correlation between list price and quality index: 0.723
Price is not an experiment that was run on customers. It is a cost-plus markup applied to a catalog in which better products cost more to make, so the cross-sectional comparison is between different products rather than between different prices for the same product.
The repair looks obvious: compare a product against itself.
for nm, L in [("Log A, the cost-plus history", A), ("Log B, the randomized experiment", B)]:
f = demand_fit(L, ["fe","level","gap","wknd"])
ci = f.conf_int()
print(nm)
for k, lab in [("log_price_vs_list", "long run, price against list "),
("log_price_vs_reference", "short run, price against reference")]:
print(f" {lab} {f.params[k]:+7.3f} 95% CI [{ci.loc[k,0]:+.2f}, {ci.loc[k,1]:+.2f}]")
print()
Log A, the cost-plus history long run, price against list -4.717 95% CI [-5.13, -4.30] short run, price against reference -2.064 95% CI [-2.52, -1.61] Log B, the randomized experiment long run, price against list -1.796 95% CI [-2.12, -1.47] short run, price against reference -2.103 95% CI [-2.47, -1.74]
Product fixed effects fix the sign and do not fix the number. Log A puts the long-run elasticity near -4.7, which is more than twice what the experiment finds.
The reason is that after the fixed effects there is almost no price variation left in Log A. The only time a product's price moves is the sitewide sale every fifth week, and the sale never arrives on its own: it comes with the email blast and the homepage banner. So the one price change in the dataset is perfectly confounded with a demand shock, and the model attributes the whole of the promotional lift to the price cut.
A fixed effect can remove a confounder between units. It cannot manufacture variation that was never there.
Step 4 · What the experiment bought, and why the holding period mattered¶
for nm, L in [("Log A", A), ("Log B", L2 := B)]:
m = (L.price/L.product_id.map(LIST)).round(2)
print(f" {nm}: price multipliers ever shown -> {sorted(m.unique())}")
print("\n Log B assigned each product a multiplier for a WHOLE WEEK at a time.")
print(" Holding it is what lets the reference price drift away from the list price,")
print(" and without that drift the two elasticities cannot be told apart.")
print(f"\n spread of reference/list in Log A: "
f"{(A.reference_price/A.product_id.map(LIST)).std():.4f}")
print(f" spread of reference/list in Log B: "
f"{(B.reference_price/B.product_id.map(LIST)).std():.4f}")
Log A: price multipliers ever shown -> [np.float64(0.85), np.float64(1.0)] Log B: price multipliers ever shown -> [np.float64(0.85), np.float64(0.9), np.float64(0.95), np.float64(1.0), np.float64(1.05), np.float64(1.1), np.float64(1.15)] Log B assigned each product a multiplier for a WHOLE WEEK at a time. Holding it is what lets the reference price drift away from the list price, and without that drift the two elasticities cannot be told apart. spread of reference/list in Log A: 0.0236 spread of reference/list in Log B: 0.0489
There is a second reason the randomization matters, and it is the one that decides whether any of the rest of this notebook is possible.
Offline policy evaluation needs every action to have been possible. To estimate what a new pricing policy would have earned, from data generated by the old one, the standard tool is inverse propensity weighting: reweight each logged outcome by how likely the new policy was to take the action that was actually taken, divided by how likely the old one was. If the old policy could never have chosen an action, that denominator is zero and the estimate does not exist.
Under the cost-plus rule, five of the seven price points have probability zero. No amount of statistical machinery recovers what would have happened at a price nobody ever charged.
Step 5 · Offline evaluation, on the log that supports it¶
f = demand_fit(B, ["fe","level","gap","wknd"])
LR, SR, WK = (float(f.params["log_price_vs_list"]),
float(f.params["log_price_vs_reference"]), float(f.params["weekend"]))
BASE = {}
for j in P.product_id:
c = f"p_{j}"
BASE[j] = float(f.params["const"]) + (float(f.params[c]) if c in f.params else 0.0)
# a candidate policy: charge 1.10 x list on everything
def candidate(j): return 1.10
b = B.copy()
b["mult"] = (b.price/b.product_id.map(LIST)).round(2)
b["target"] = b.product_id.map(lambda j: candidate(j))
b["match"] = (np.abs(b.mult - b.target) < 1e-6).astype(float)
b["margin_per_session"] = b.margin/b.sessions
ips = (b.match/b.assignment_prob*b.margin_per_session)
n = len(b)
est, se = ips.mean(), ips.std(ddof=1)/np.sqrt(n)
print("INVERSE PROPENSITY ESTIMATE of margin per session under 'always charge 1.10 x list'")
print(f" {est:.4f} dollars per session 95% CI [{est-1.96*se:.4f}, {est+1.96*se:.4f}]")
print(f" logged policy actually earned {b.margin_per_session.mean():.4f} dollars per session")
print(f"\n effective sample size {1/((b.match/b.assignment_prob/n)**2).sum():,.0f} of {n:,} rows")
print(" Only the rows where the experiment happened to show 1.10 carry any information.")
INVERSE PROPENSITY ESTIMATE of margin per session under 'always charge 1.10 x list' 5.0799 dollars per session 95% CI [3.8625, 6.2973] logged policy actually earned 5.0460 dollars per session effective sample size 465 of 3,360 rows Only the rows where the experiment happened to show 1.10 carry any information.
The estimate is unbiased and the interval is wide, and both facts come from the same place: a policy that always charges 1.10 only learns from the tenth of the log where 1.10 was the assigned price. Evaluating a deterministic policy offline throws away most of the data by construction, which is the honest cost of not deploying it.
Now build the simulator, which is the alternative and carries its own problems.
Step 6 · The simulator is the model¶
def run(policy, days=180, seed=0, lr=None, sr=None):
rs = np.random.default_rng(seed)
lr = LR if lr is None else lr
sr = SR if sr is None else sr
ref = {j: LIST[j] for j in P.product_id}
total = 0.0
for t in range(days):
dow = t % 7
for j in P.product_id:
m = policy(j, t, dow, ref[j]/LIST[j])
p = LIST[j]*m
lin = BASE[j] + lr*np.log(m) + sr*np.log(p/ref[j]) + WK*(dow >= 5)
u = rs.binomial(max(int(rs.poisson(SESS[j])), 0), 1/(1 + np.exp(-lin)))
total += u*(p - COST[j])
ref[j] = 0.90*ref[j] + 0.10*p # tomorrow's expectation
return total
def avg(policy, n=8, **kw): return float(np.mean([run(policy, seed=s, **kw) for s in range(n)]))
def cost_plus(j, t, dow, rr): return 0.85 if ((t//7) % 5 == 4) else 1.00
v_rule = avg(cost_plus)
print(f"the cost-plus rule, 180 simulated days: ${v_rule:,.0f}")
the cost-plus rule, 180 simulated days: $2,670,341
Every number from here on is produced by that function, which means every number from here on is a statement about the demand model rather than about customers. The agent will exploit anything the model gets wrong, and it will do so enthusiastically. Section 9 measures how much that matters.
Step 7 · Greedy, and the promotion trap¶
def greedy(j, t, dow, rr):
best, bv = 1.0, -1e18
for m in MULT:
lin = BASE[j] + LR*np.log(m) + SR*np.log(m/rr) + WK*(dow >= 5)
v = 1/(1 + np.exp(-lin))*(LIST[j]*m - COST[j])
if v > bv: bv, best = v, m
return best
# one fixed multiplier per product, chosen by simulating the whole horizon
FIXED = {}
for j in P.product_id:
bv, bm = -1e18, 1.0
for m in MULT:
ref, tot = LIST[j], 0.0
for t in range(180):
p = LIST[j]*m
lin = BASE[j] + LR*np.log(m) + SR*np.log(p/ref) + WK*((t % 7) >= 5)
tot += 1/(1 + np.exp(-lin))*(p - COST[j])*SESS[j]
ref = 0.90*ref + 0.10*p
if tot > bv: bv, bm = tot, m
FIXED[j] = bm
def fixed(j, t, dow, rr): return FIXED[j]
v_greedy, v_fixed = avg(greedy), avg(fixed)
print(f" cost-plus rule ${v_rule:12,.0f}")
print(f" greedy, best price for today ${v_greedy:12,.0f} {(v_greedy/v_rule-1)*100:+6.1f}%")
print(f" one fixed price, best over the whole horizon ${v_fixed:12,.0f} {(v_fixed/v_rule-1)*100:+6.1f}%")
print()
print(f" greedy charges on average {np.mean([greedy(j,0,0,1.0) for j in P.product_id]):.3f} x list")
print(f" fixed charges on average {np.mean(list(FIXED.values())):.3f} x list")
cost-plus rule $ 2,670,341 greedy, best price for today $ 3,266,629 +22.3% one fixed price, best over the whole horizon $ 3,516,368 +31.7% greedy charges on average 0.895 x list fixed charges on average 1.146 x list
The greedy agent cuts prices by about ten percent. The far-sighted one raises them by about fifteen, and earns 7.6 percent more.
This is the promotion trap, and the arithmetic behind it is worth stating. Cutting the price today buys the full short-run response, estimated at about -2.1, because customers are comparing today's price with what they expected. Hold the cut and the expectation follows it down over a few weeks, and all that is left is the long-run response of about -1.8 against a permanently lower price. The lift is temporary and the margin loss is not.
A myopic agent sees only the first number, which is the larger of the two, so it discounts. The mistake is not in the optimization, it is in the horizon.
Step 8 · Reinforcement learning, properly¶
The fixed-price search cannot vary the price with circumstances. A reinforcement learner can: it observes a state, takes an action, and learns a value for each pair. Here the state that matters is where the reference price sits relative to list, the action is one of seven multipliers, and the reward is the day's margin.
BINS = np.array([0.90, 0.96, 1.02, 1.08])
def sbin(rr): return int(np.digitize(rr, BINS))
def train_q(gamma, episodes=300, steps=60, alpha=0.15, eps=0.20, seed=5):
Q = {j: np.zeros((len(BINS)+1, len(MULT))) for j in P.product_id}
rs = np.random.default_rng(seed)
for _ in range(episodes):
for j in P.product_id:
ref = LIST[j]*float(rs.uniform(0.88, 1.12))
for t in range(steps):
s = sbin(ref/LIST[j])
a = int(rs.integers(len(MULT))) if rs.random() < eps else int(Q[j][s].argmax())
m = MULT[a]; p = LIST[j]*m
lin = BASE[j] + LR*np.log(m) + SR*np.log(p/ref) + WK*((t % 7) >= 5)
r = 1/(1 + np.exp(-lin))*(p - COST[j])*SESS[j]
ref2 = 0.90*ref + 0.10*p
Q[j][s, a] += alpha*(r + gamma*Q[j][sbin(ref2/LIST[j])].max() - Q[j][s, a])
ref = ref2
return Q
print(f" {'discount':>9s} {'effective horizon':>20s} {'margin':>14s} {'vs the rule':>12s} {'mean multiplier':>17s}")
QS = {}
for gamma, epi, stp in [(0.90, 300, 60), (0.97, 300, 60), (0.995, 300, 120), (0.999, 400, 180)]:
Q = train_q(gamma, epi, stp); QS[gamma] = Q
pol = lambda j, t, dow, rr, Q=Q: MULT[int(Q[j][sbin(rr)].argmax())]
v = avg(pol)
mm = np.mean([MULT[int(Q[j][s].argmax())] for j in P.product_id for s in range(len(BINS)+1)])
print(f" {gamma:9.3f} {1/(1-gamma):17.0f} days ${v:13,.0f} {(v/v_rule-1)*100:+11.1f}% {mm:17.3f}")
discount effective horizon margin vs the rule mean multiplier
0.900 10 days $ 3,380,090 +26.6% 0.951
0.970 33 days $ 3,068,521 +14.9% 0.961
0.995 200 days $ 3,045,479 +14.0% 0.962
0.999 1000 days $ 2,960,571 +10.9% 0.917
Every one of them loses to the fixed-price grid search, and a longer horizon makes it worse rather than better. That result is worth taking seriously rather than tuning away, so look at the policy it actually learned.
Q = QS[0.90]
lab = ["below 0.90", "0.90 to 0.96", "0.96 to 1.02", "1.02 to 1.08", "above 1.08"]
print("what the agent charges, by where the reference price sits")
for s in range(len(BINS)+1):
print(f" reference / list {lab[s]:14s} mean multiplier "
f"{np.mean([MULT[int(Q[j][s].argmax())] for j in P.product_id]):.3f}")
what the agent charges, by where the reference price sits reference / list below 0.90 mean multiplier 1.015 reference / list 0.90 to 0.96 mean multiplier 0.974 reference / list 0.96 to 1.02 mean multiplier 0.965 reference / list 1.02 to 1.08 mean multiplier 0.906 reference / list above 1.08 mean multiplier 0.896
The agent has learned to chase the reference price. When customers expect a high price it discounts, and when they expect a low one it raises. Every one of those moves is locally sensible: if the reference is above the list price then a cut is unusually effective today, and the agent has correctly noticed.
The result is a policy that oscillates, and an oscillating price keeps the reference somewhere in the middle instead of walking it upward to a level a constant high price would sustain. Longer horizons make it worse because a five-state discretisation of a slowly-drifting continuous variable propagates its own approximation error every time the value is bootstrapped.
None of that is a bug in the algorithm. The state representation is too coarse for the dynamics, and a grid search over 40 products times 7 prices, four hundred and eighty combinations evaluated by direct simulation, does not have that problem because it does not approximate anything.
The useful reading of this section is not that reinforcement learning does not work. It is that a method with a state space, a discount factor, a learning rate and an exploration schedule has four ways to be wrong that a grid search does not have, and it has to earn its place against the simplest thing that represents the same dynamics.
Step 9 · What exploration costs¶
An agent that only ever charges its current best price never learns whether a different one is better. The standard answer is to explore some fraction of the time, and that fraction has a price.
best_pol = lambda j, t, dow, rr: MULT[int(QS[0.90][j][sbin(rr)].argmax())]
def with_exploration(eps, days=180, seed=0):
rs = np.random.default_rng(1000 + seed)
ref = {j: LIST[j] for j in P.product_id}; tot = 0.0
for t in range(days):
for j in P.product_id:
rr = ref[j]/LIST[j]
m = MULT[int(rs.integers(len(MULT)))] if rs.random() < eps else best_pol(j, t, t % 7, rr)
p = LIST[j]*m
lin = BASE[j] + LR*np.log(m) + SR*np.log(p/ref[j]) + WK*((t % 7) >= 5)
tot += rs.binomial(max(int(rs.poisson(SESS[j])), 0), 1/(1 + np.exp(-lin)))*(p - COST[j])
ref[j] = 0.90*ref[j] + 0.10*p
return tot
v_none = np.mean([with_exploration(0.0, seed=s) for s in range(6)])
for eps in [0.0, 0.05, 0.10, 0.20]:
v = np.mean([with_exploration(eps, seed=s) for s in range(6)])
print(f" exploring {eps:4.0%} of the time ${v:12,.0f} {(v/v_none-1)*100:+5.1f}%")
exploring 0% of the time $ 3,390,623 +0.0% exploring 5% of the time $ 3,362,474 -0.8%
exploring 10% of the time $ 3,309,230 -2.4% exploring 20% of the time $ 3,250,799 -4.1%
Exploring one day in five costs about four percent of margin over six months, and the cost is close to linear: roughly a fifth of a percent for each percentage point of exploration.
That number is a property of the action set rather than of the algorithm. The cost of exploring is the gap between the best action and the average random one, and here the actions run only from 0.85 to 1.15 times list, so the worst thing exploration can do is charge fifteen percent too little for a day. Widen the action set and the calculation changes completely. An agent allowed to try half price would pay a great deal more for the same information, which is an argument for constraining the action set rather than for exploring freely, and it is the lever to reach for first when exploration looks too expensive to justify.
Step 10 · How much does getting the simulator wrong cost?¶
The agent was trained inside a demand model estimated from six weeks of data. Retrain it on deliberately wrong versions of that model and evaluate every resulting policy in the same, fixed, correct simulator.
def fixed_policy_from(lr_, sr_):
out = {}
for j in P.product_id:
bv, bm = -1e18, 1.0
for m in MULT:
ref, tot = LIST[j], 0.0
for t in range(180):
p = LIST[j]*m
lin = BASE[j] + lr_*np.log(m) + sr_*np.log(p/ref) + WK*((t % 7) >= 5)
tot += 1/(1 + np.exp(-lin))*(p - COST[j])*SESS[j]
ref = 0.90*ref + 0.10*p
if tot > bv: bv, bm = tot, m
out[j] = bm
return out
print(f" {'the simulator it was trained on':34s} {'earns':>14s} {'vs the honest one':>19s} {'charges':>9s}")
rows = [("the fitted demand model", LR, SR),
("elasticities 30 percent smaller", LR*0.7, SR*0.7),
("elasticities 30 percent larger", LR*1.3, SR*1.3),
("the reference effect ignored", LR + SR, 0.0)]
ref_v = None
for nm, lr_, sr_ in rows:
pol = fixed_policy_from(lr_, sr_)
v = avg(lambda j, t, dow, rr, pol=pol: pol[j])
if ref_v is None: ref_v = v
print(f" {nm:34s} ${v:13,.0f} {(v/ref_v-1)*100:18.1f}% "
f"{np.mean(list(pol.values())):9.3f}")
the simulator it was trained on earns vs the honest one charges
the fitted demand model $ 3,516,368 0.0% 1.146 elasticities 30 percent smaller $ 3,516,377 0.0% 1.150
elasticities 30 percent larger $ 3,494,613 -0.6% 1.040
the reference effect ignored $ 3,291,650 -6.4% 0.897
The pattern is the interesting part. Getting the size of the elasticities wrong by thirty percent in either direction costs almost nothing. The policy shifts by one price point at most, and the margin surface is flat enough near its optimum that it barely registers.
Getting the structure wrong costs six percent. The last row folds the short-run and long-run responses into a single number, which is exactly what an analyst does by default if nobody tells them customers anchor on recent prices. That policy discounts, because it believes the whole of the promotional lift is permanent.
That is a general lesson about simulators and it is not specific to pricing. The parameter values can be some way off and the policy survives. If the state variable is missing, the policy is optimizing the wrong problem, and no amount of data on the parameters will find that out.
Step 11 · The answer key¶
T = load("Truth")
print("ELASTICITIES: what we estimated against what generated the data")
print(f" long run estimated {LR:+.3f} truth {T.true_long_run_elasticity.mean():+.3f}")
print(f" short run estimated {SR:+.3f} truth {T.true_short_run_elasticity.mean():+.3f}")
print("\n Both are mildly attenuated. The logs aggregate sessions to product-days, so the")
print(" traffic-source variation inside a day is averaged away, and averaging inside a")
print(" logit shrinks a coefficient toward zero. The ordering and the ratio survive it,")
print(" and the ordering is what the policy depends on.")
ELASTICITIES: what we estimated against what generated the data long run estimated -1.796 truth -1.605 short run estimated -2.103 truth -2.462 Both are mildly attenuated. The logs aggregate sessions to product-days, so the traffic-source variation inside a day is averaged away, and averaging inside a logit shrinks a coefficient toward zero. The ordering and the ratio survive it, and the ordering is what the policy depends on.
Step 12 · The pictures¶
fig, axes = plt.subplots(1, 2, figsize=(12.6, 4.4))
ax = axes[0]
labs = ["cost-plus rule", "greedy", "Q-learning", "grid search"]
vals = [v_rule, v_greedy, avg(best_pol), v_fixed]
ax.bar(np.arange(4), np.array(vals)/1e6, color=[MUT, LT, LT, VI], width=0.6)
for x, v in zip(np.arange(4), vals):
ax.text(x, v/1e6 + 0.05, f"${v/1e6:.2f}M", ha="center", fontsize=10, fontweight="bold", color=DK)
ax.set_xticks(np.arange(4)); ax.set_xticklabels(labs, fontsize=9.5)
ax.set_ylabel("margin over 180 days, $ millions"); ax.set_ylim(0, 4.2)
ax.set_title("The simplest far-sighted policy wins")
ax = axes[1]
j = int(P.product_id.iloc[0])
for m, col, lab in [(0.90, RD, "cut to 0.90x"), (1.00, MUT, "hold at list"), (1.10, VI, "raise to 1.10x")]:
ref, series = LIST[j], []
for t_ in range(120):
p_ = LIST[j]*m
lin = BASE[j] + LR*np.log(m) + SR*np.log(p_/ref) + WK*((t_ % 7) >= 5)
series.append(1/(1+np.exp(-lin))*(p_ - COST[j])*SESS[j])
ref = 0.90*ref + 0.10*p_
sm = pd.Series(series).rolling(7, center=True).mean()
ax.plot(range(120), sm, color=col, lw=2.8, label=lab)
ax.set_xlabel("day"); ax.set_ylabel("expected margin per day (7-day mean)")
ax.set_title("The lift from a cut fades; the loss does not")
ax.legend(fontsize=9, loc="lower right")
plt.tight_layout(); plt.show()
Left: the four policies over the same simulated six months. Right: why the greedy agent is wrong. A ten percent cut is ahead of holding list price for about twelve days, and behind it for the remaining hundred and eight.
Step 13 · What none of this settles¶
The simulator has no competitors in it. Raise prices fifteen percent across a catalog and somebody else does not. Every number in sections 6 to 9 assumes the rest of the market holds still, which is the assumption most likely to be false and the one this data cannot test.
Six weeks of randomization estimated two parameters. It did not measure whether the reference-price mechanism is the right model of customer memory, only what the coefficients are if it is. A different anchoring model, say one where customers remember the lowest price they have seen rather than an average, would give a different optimal policy from the same data.
Charging different customers different prices is not the same problem as charging different prices over time. Everything here sets one price per product per day for everybody. Personalised pricing raises questions about fairness and about disclosure that are not statistical and are not answered by a higher expected margin.
And the whole exercise reprices a catalog that was priced by a rule for years. A fifteen percent increase across forty products is a business decision with consequences for customer trust that a 180-day margin simulation is not equipped to see. The right output of this project is a proposed experiment, not a deployed agent.
What to take away¶
- Twenty weeks of business-as-usual data said that raising prices raises demand. Price was a cost-plus markup on a catalog where better products cost more, so the comparison was between products, not between prices.
- Product fixed effects fixed the sign and not the number, because the only price movement left was the sale weeks, and those came with a marketing push.
- Offline policy evaluation needs every action to have been possible. Five of the seven price points had probability zero under the old rule, and no method recovers a price nobody charged.
- The experiment had to hold each price for a week. Daily randomization identifies the total price response but cannot separate the short-run from the long-run effect, because the reference price never moves.
- A myopic agent discounts. It sees the short-run response of -2.1 and not the long-run -1.8 that is left once expectations catch up, and it earns 7.6 percent less than a far-sighted one.
- The Q-learner lost to a grid search over 480 combinations, and the policy it learned explains why: it chases the reference price, which is locally right everywhere and globally a loss.
- Exploring one day in five cost four percent, and roughly a fifth of a percent per point of exploration. That price is set by how wide the action set is, not by the algorithm.
- Thirty percent errors in the elasticities cost under one percent; omitting the reference effect cost six. Simulators tolerate wrong parameters and not missing state.