Sample size. Revelation

Before diving into the actual sample size formula, let’s take a moment to refresh the foundations of hypothesis testing in the context of A/B tests.

Basics

Let’s start with the basics.

When we run an A/B test, we compare two groups—Group A (control) and Group B (variant)—to see if there's a statistically significant difference between them.

This comparison is framed through two competing hypotheses:

1. Null Hypothesis (H₀): There is no difference.

   

2. Alternative Hypothesis (H₁): There is a difference.

   

Concepts

• Type I Error (α): Rejecting the null hypothesis when it is actually true (false positive). In other words, we detect the difference when, in reality, no difference exists.

• Type II Error (β): Failing to reject the null hypothesis when the alternative is true (false negative). In other words, we miss detecting a difference even though a true effect exists.

• Power (1 - β): The probability of correctly rejecting the null when there is a true effect. In other words, it’s our ability to detect a difference when one actually exists.

• Minimum Detectable Effect (MDE): The smallest difference we want to reliably detect.

Distributions