Checkout redesign experiment
Variant B simplifies checkout. 80,000 users, randomized 50/50, six weeks of synthetic data. Analysed with product-analytics-toolkit.
Ship it: conversion is up +8.1% (95% CI +0.12% to +0.75% absolute), with no sample ratio mismatch.
Results
Purchase conversion5.36% → 5.79%+8.1% · p = 0.0073
Checkout completion60.3% → 65.3%+8.4% · p < 0.0001
Revenue per user$2.93 → $3.18+8.4% · p = 0.0133
Sample ratio (SRM)40,244 / 39,756balanced · p = 0.084
Lift by segment (95% CI)
The effect is concentrated on mobile, where the old checkout lost the most users. Desktop alone wouldn't justify the change.
Segments
| Platform | Users A / B | A | B | Lift | p | |
|---|---|---|---|---|---|---|
| Desktop | 16,908 / 16,783 | 7.69% | 7.85% | +2.1% | 0.5728 | not significant |
| Mobile | 23,336 / 22,973 | 3.67% | 4.29% | +16.9% | 0.0006 | significant |
Planning note: detecting a 1-point change from a 5.4% baseline needs ~8,655 users per arm (α = 0.05, power = 80%).
Funnel
| Step conversion | A | B |
|---|---|---|
| view product | 57.9% | 58.0% |
| add to cart | 26.4% | 26.3% |
| checkout start | 58.2% | 58.1% |
| purchase | 60.3% | 65.3% |
Only the last step should move: the redesign touched checkout, not discovery. Earlier steps matching between arms is a useful sanity check.
Weekly retention by cohort
| Cohort | Users | Week 0 | Week 1 | Week 2 | Week 3 | Week 4 | Week 5 |
|---|---|---|---|---|---|---|---|
| May 04 | 13,315 | 100% | 12% | 10% | 8% | 7% | 5% |
| May 11 | 13,369 | 100% | 11% | 10% | 9% | 7% | 5% |
| May 18 | 13,379 | 100% | 12% | 11% | 9% | 7% | 6% |
| May 25 | 13,283 | 100% | 11% | 11% | 9% | 7% | 6% |
| Jun 01 | 13,281 | 100% | 12% | 10% | 8% | 7% | 5% |
| Jun 08 | 13,366 | 100% | 12% | 11% | 9% | 7% | 5% |