The Treatment That Won Every Subgroup and Still Lost the Trial

In a real 1986 clinical trial, open surgery beat percutaneous nephrolithotomy on small kidney stones. It beat it again on large stones. Combine the two groups back into one trial and open surgery loses. Same 700 patients, same outcomes, opposite conclusion, and nobody touched the arithmetic.

This is Simpson's Paradox, and it's not a rounding trick or a cherry-picked dataset. It's a real, published, independently checkable result, and it's a trap that's just as available to anyone comparing model performance across a mixed benchmark as it was to the doctors running this trial.

The trial

The British Medical Journal published this comparison in 1986: open surgery, the older and more invasive option, against percutaneous nephrolithotomy, a newer keyhole procedure, across 700 kidney stone patients. Look only at the combined success rate and percutaneous nephrolithotomy wins clearly, 82.6% versus 78.0% for open surgery. That's the chart above, and it's arithmetically correct.

Who got which treatment

The reversal has a specific, findable cause. Doctors weren't assigning patients to treatments randomly. They preferentially used open surgery, the more invasive and reliable option, on the harder cases: 263 of the 343 large-stone patients (76.7%) received open surgery. Percutaneous nephrolithotomy, the newer and gentler option, mostly went to easier cases: 270 of the 357 small-stone patients (75.6%) received it.

That allocation drags open surgery's overall rate down, not because it performs worse, but because most of its patients were the harder cases where every treatment's success rate is naturally lower. Percutaneous nephrolithotomy's overall rate gets inflated for the opposite reason. Stone size is a confounding variable: it's correlated with both which treatment a patient received and how likely they were to recover regardless of treatment. The aggregate number is correct. It's just answering "which treatment got the easier patients" instead of "which treatment works better."

A second documented case

Kidney stones aren't a one-off. A frequently cited case involves hospital-acquired urinary tract infection rates: at the aggregate level, one intervention looked protective, with a relative risk of 0.71. Split hospitals into two groups by their baseline UTI rate, low-incidence and high-incidence, and the direction reverses in both strata: 2.59 in low-incidence hospitals, 2.03 in high-incidence hospitals, both indicating increased risk. Same shape of problem, a variable correlated with both group assignment and outcome hiding in the aggregate.

The pattern in both cases: whenever two groups being compared differ substantially in how they're composed, the aggregate comparison isn't automatically the trustworthy one. It's the one most likely to have papered over the real answer.

A checklist for your own charts

If you're building dashboards, running A/B tests, or comparing model performance across a mixed benchmark, this trap is available to you right now:

• Ask what's driving group composition before trusting an aggregate. If the two groups were assigned to their conditions non-randomly, by difficulty, by self-selection, or by any process correlated with the outcome, the aggregate is suspect by default.
• Compute the stratified version alongside the aggregate, not instead of it. If the stratified numbers agree with the aggregate, checking cost nothing. If they disagree, checking just saved you from publishing a reversed conclusion.
• Look at how unevenly the confounding variable is spread across your groups. Here the giveaway was 75%+ of patients in each treatment arm coming from the same stone-size category. A confounder spread evenly across comparison groups can't cause this reversal; a lopsided one can.
• Treat a direction mismatch between the aggregate and the stratified numbers as a stop signal, not a footnote. It means the honest answer is "it depends on the subgroup," and reporting the single number anyway is a choice.

Caveats

The Charig et al. numbers are real, from a published 1986 BMJ trial, reproduced here from a 2023 clinical-research methods paper that cites the original. I re-checked every ratio by hand against the source rather than trusting a secondary summary. Simpson's Paradox doesn't mean aggregates are always wrong; most hold up fine under stratification. It means an aggregate that hasn't been checked against its subgroups is an untested claim, same as any other number on this site.