A/B testing

Also called split testing.

Business basics1min read

"B testing" has no standard standalone definition in business. People usually mean A/B testing: randomly assigning users to version A or B of a page, product or message, then comparing performance on a chosen metric.

How it works

Pick one primary outcome, such as completed purchases or activated users. Run a control (A) and a variation (B) at the same time. Randomly assign users, and keep each user in the same group throughout the test.

Before launch, decide the smallest improvement worth detecting, the sample size needed and when to end the test. Do not stop simply because the dashboard first shows a winner.

Increasing signups without increasing spend

A startup tests a new signup page. Each version receives 5,000 randomly assigned visitors, with $2,000 of acquisition spend per group.

MetricPage APage B
Visitors5,0005,000
Signups200250
Signup rate4%5%
Cost per signup$10$8

Page B's relative uplift is (5% − 4%) ÷ 4% = 25%. Cost per signup falls from $2,000 ÷ 200 = $10 to $2,000 ÷ 250 = $8, a 20% reduction.

That is not automatically a 20% reduction in customer acquisition cost. Signups must still become paying customers. Check statistical uncertainty and downstream customer quality before adopting B.

Questions people ask

How long should an A/B test last?

There is no universal duration. Reach the planned sample size and cover relevant business cycles. A weekday-only test can mislead if weekend visitors behave differently. A low-traffic startup may not have enough users to detect small improvements within a useful timeframe.

Does Netflix use A/B testing?

Yes. Netflix's engineering team has described its experimentation platform for testing product changes with randomly assigned groups of members. A/B testing is not limited to ads or landing pages.

Related terms

Sources

  1. Netflix Technology Blog, "It's All A/Bout Testing: The Netflix Experimentation Platform", 2016.