The New Analgesic Works, and It Works Unevenly

A clear win on relief, with a split between patients it helps a great deal and patients it barely touches.

To Clinical Lead

From Trial Statistics

Re New analgesic versus standard care, relief outcome

Where this comes from

Chapter Chapter 173 · Mann-Whitney U: Two Pain Treatments

Part Part XXVII · Capstone Projects: Statistical Testing

Dataset capstone-pain-relief-two-treatments.xlsx

Notebook View the analysis

Recommendation

Bottom line

The new treatment produces more relief than the standard one, and the margin is far too large to be chance. The way to state it: pair any patient on the new drug with any patient on the standard drug, and the new-drug patient reports more relief about three times in four. The important qualifier is that the benefit is not spread evenly. Roughly 62% of patients on the new drug got substantial relief, but 13% got essentially none.

What we found

91 patients reported relief on a 0-to-10 scale after treatment. On the standard drug, most reported one to three points, with a median of 2. On the new drug the median was 5, and the difference between the two groups is decisive (p = 2.8e-05).

The clean way to express the size of the effect is to compare patients directly. There are 2,070 ways to pair one new-drug patient with one standard-drug patient. The new-drug patient reported more relief in 70 percent of those pairings, less in 20 percent, and the same in the rest.

A cumulative curve comparison and a stacked bar of pairwise wins, ties and losses. — Figure 1. Left: the share of patients at or below each relief score. The new treatment's line sits lower everywhere, which means wherever you set a bar, fewer of its patients fall short of it. Right: all 2,070 head-to-head pairings.

Paired bars: the standard drug has a median of 2.0 and a mean of 2.39; the new drug has a median of 5.0 and a mean of 4.67. — Figure 2. Reported relief on each arm, by median and by mean.

The part that should shape how this is communicated

The new drug does not lift everybody by a couple of points. It divides patients into two groups. That is visible in the raw distribution as two separate clusters rather than one hump, and it is the finding most likely to be lost if the result is summarized as an average.

Two bar charts of relief scores, one single-humped and one with two separate clusters. — Figure 3. How many patients gave each relief score. The standard drug produces one cluster near the bottom of the scale. The new drug produces two: a small group who got almost nothing and a larger group who did well.

Suggested wording

Something like: most patients on the new treatment report substantially more relief than they would expect from standard care, though a minority get little benefit. Around 62 in 100 reached four or more points of relief, against 20 in 100 on the standard drug. That is accurate and does not promise every patient the average.

One thing the trial report must disclose

The two arms did not start level. Patients allocated to the new treatment reported more pain at baseline than those on the standard drug, 7.3 against 6.7, and that gap is bigger than chance comfortably explains. Random allocation evens groups out on average, not in every individual trial, and with 91 patients an imbalance like this is not unusual.

It matters because it runs the wrong way: people who start in more pain have further to improve, so an arm that begins worse can look better on a relief score for reasons unconnected to the drug. Two things reassure us here. Baseline pain barely predicts how much relief a patient reported, and the advantage we found is far too large for a 0.6-point head start to manufacture.

We are flagging it because a trial summary that leaves this out is hiding something a careful reader would want to know. The next study should plan from the start to adjust for baseline pain.

What we cannot say

We cannot say who the responders are. The split is real in this trial, but identifying what distinguishes the two groups by searching the baseline data after the fact would produce a finding that will not replicate. If this matters, and it probably does, it needs to be pre-specified in the next study.
We cannot fully separate the drug from expectation. Relief is reported by the patient, as it must be. If patients knew which treatment they received, some of the difference may be what they expected to feel.
We cannot speak to anything but relief. Side effects, tolerance and cost are outside this analysis and belong in the same decision.
Three in four is about pairings, not patients. It does not mean 75 percent of patients will be helped, and it should not be worded that way in anything a patient reads.

← Back to Chapter 173 📓 Companion notebook 📊 Download dataset

From Statistics, Data Science and AI: A Visual Handbook by John Fisher. Every statistic, table, and figure in this report is reproduced by the companion notebook.