Sample Size: Testing or Survey
2022-09-02 # A/B Testing # Hypothesis Testing # Sample Size

This is a summary note on calculating sample size for different purposes. The detailed derivation of the sample size for an AB testing with two-independent groups can be found here.

AB Testing

Let the test statistic be significant at with a statistical power of . The sample size formula for each group is given by

where is the size ratio of the larger group over the smaller group.

  • Two Independent Groups

    Suppose the proportion rate in the treatment group is not much different from the proportion rate in the control group. Then, we have , and . The total sample size including the treatment and the control is .

  • Population v.s. One Study Group

    Suppose the population proportion is known at . We have and . The sample size of the study group is .

    Here is a link to an Online Calculator at ClinCalc.

Survey

In a survey, we care about the result representativeness, that is, if the sample mean from the survey can represent the population mean.

Ideally, we would like our survey proportion to be the same as the true proportion from the population. In fact, we allow the survey estimate is close to the true proportion within a margin of error less than at a confidence level of . This problem can be set up in a hypothesis test as below:

The sampling distribution of the difference, , is given by

Therefore,

The confidence level suggests

where if is true. Since we allow our estimate to be close to within a distance of , we have

The minimal survey size is given by

Here is a link to an Online Calculator at Calculator.

Z Critical Values

The following table presents a list of commonly used Z scores (also known as critical values) in practice.

 (Right-tailed)  (Two-tailed)
0.01 2.326 +/-2.576
0.05 1.645 +/-1.960
0.10 1.282 +/-1.645
0.20 0.842 +/-1.282
Prev
2022-09-02 # A/B Testing # Hypothesis Testing # Sample Size
Next