Derivation of Sample Size for Two-group AB Testing

Problem Settings

• Two independent groups

  	Control	Treatment
  Size
  Percentage of Interest

• One-tailed hypothesis testing

  • Significance level (type I error):

  • Statistical power (1 - type II error):

  • Percentage of Interest in sample: and

The sampling distribution of the difference, , is given by

Therefore,

Type I Error in Testing

Given is true, we have . Hence, the conditional probability can be written as an unconditional probability, such that,

If we have two equal-size groups, i.e., , then and

Type II Error in Testing

Since is true, we have . In particular, let . Then, we can rewrite the distribution to a standard normal distribution, such that,

Solve Equation for n

That is,

• n is the sample size for each group. In other words, the total sample size is .

• is the critical value at the significance level of in a one-tailed test.

  • will be replaced with in a two-tailed test.

Suppose and are quite similar, we have

Therefore,