A/B testing
Also called split testing.
"B testing" has no standard standalone definition in business. People usually mean A/B testing: randomly assigning users to version A or B of a page, product or message, then comparing performance on a chosen metric.
How it works
Pick one primary outcome, such as completed purchases or activated users. Run a control (A) and a variation (B) at the same time. Randomly assign users, and keep each user in the same group throughout the test.
Before launch, decide the smallest improvement worth detecting, the sample size needed and when to end the test. Do not stop simply because the dashboard first shows a winner.
Increasing signups without increasing spend
A startup tests a new signup page. Each version receives 5,000 randomly assigned visitors, with $2,000 of acquisition spend per group.
| Metric | Page A | Page B |
|---|---|---|
| Visitors | 5,000 | 5,000 |
| Signups | 200 | 250 |
| Signup rate | 4% | 5% |
| Cost per signup | $10 | $8 |
Page B's relative uplift is (5% − 4%) ÷ 4% = 25%. Cost per signup falls from $2,000 ÷ 200 = $10 to $2,000 ÷ 250 = $8, a 20% reduction.
That is not automatically a 20% reduction in customer acquisition cost. Signups must still become paying customers. Check statistical uncertainty and downstream customer quality before adopting B.
Questions people ask
How long should an A/B test last?
There is no universal duration. Reach the planned sample size and cover relevant business cycles. A weekday-only test can mislead if weekend visitors behave differently. A low-traffic startup may not have enough users to detect small improvements within a useful timeframe.
Does Netflix use A/B testing?
Yes. Netflix's engineering team has described its experimentation platform for testing product changes with randomly assigned groups of members. A/B testing is not limited to ads or landing pages.
