Sample Size, MDE & Power: Interview-Ready A/B Testing Framework
Yesterday, a checkout experiment looked like a win: the new flow lifted conversion from 5.0% to 5.3% after a few thousand visitors. One week later, the lift vanished - not because customers changed their minds, but because the test was trying to hear a whisper in a noisy room.
- Sample size is how many observations you need per variant before trusting a test result.
- Minimum Detectable Effect is the smallest lift you design the test to reliably detect.
- Power is the probability that your test detects a real effect when it exists.
- Smaller MDE, higher power, lower significance level and noisier metrics all require larger samples.
- For most business A/B tests, 5% significance and 80% power are common starting points, not universal rules.
- A test can be statistically insignificant because the idea is weak - or because the test was underpowered.
- The best interview answer connects statistics to business cost: “What effect is worth detecting, and can we afford the traffic?”
Big Picture: A Business Decision Becomes a Test Design
Sample size is not a spreadsheet afterthought. It is the bridge between the business question - “Is this change worth launching?” - and the statistical question - “Can our experiment reliably detect the lift we care about?”
Core Explanation: The Three Ideas You Must Connect
Sample size, MDE and power are a trade-off triangle. You cannot discuss one intelligently without the other two.
If you want to detect a tiny improvement, you need a large sample. If you accept only very low false-positive risk, you need a large sample. If the metric is noisy, you need a large sample. That is the entire intuition.
1. Sample Size: “How much data is enough?”
Sample size is the number of observations, users, sessions or transactions assigned to each variant in a test. In A/B testing, we usually talk about n per variant, not total n, because each group must be large enough to estimate its own performance.
For a two-variant conversion-rate test, a common approximate formula is:
n per group ≈ 2 × (Z1-alpha/2 + Zpower)² × p(1-p) ÷ Δ²
Where p is the baseline conversion rate and Δ is the absolute MDE. For a two-sided 5% significance test with 80% power, the Z-values are approximately 1.96 and 0.84.
2. Minimum Detectable Effect: “What lift is worth finding?”
MDE is not the lift you expect. It is the smallest true effect that your test is designed to detect with chosen power and significance.
There are two ways to express it:
- Absolute MDE: conversion moves from 5.0% to 5.5%, so Δ = 0.5 percentage points.
- Relative MDE: conversion increases from 5.0% to 5.5%, a 10% relative lift.
Interview trap: candidates often say “we need a big enough sample” without defining the effect size that is “big enough” for the business.
3. Power: “Will the test catch a real effect?”
Power is 1 minus Type II error. If power is 80%, the test has an 80% chance of detecting the specified MDE if that effect is truly present.
A low-powered test is dangerous because “not significant” may only mean “we did not collect enough data.”
Worked Example: Why Small Lifts Need Huge Samples
Suppose an Indian food-delivery app wants to test a new checkout message.
- Current checkout conversion = 5.0%
- Smallest worthwhile lift = 0.5 percentage points, from 5.0% to 5.5%
- Significance level = 5%, two-sided
- Power = 80%
Using the approximate two-proportion formula:
n per group ≈ 2 × (1.96 + 0.84)² × 0.0525 × 0.9475 ÷ 0.005²
n per group ≈ 31,000 users, so the test needs about 62,000 eligible users across control and treatment.
The managerial lesson is sharp: if the app gets enough eligible traffic in a few days, this is feasible. If the flow receives only a few hundred users a day, the team must either accept a larger MDE, run longer, combine markets carefully, or choose a higher-frequency metric.
Key Measures to Track Before You Launch
Definitions: Say These in One Breath
- Sample size: The number of observations required per group to estimate an effect with planned reliability.
- Minimum Detectable Effect: The smallest true effect a test is designed to detect at chosen power and significance.
- Statistical power: The probability of rejecting a false null hypothesis.
- Type I error: Rejecting a true null hypothesis, commonly called a false positive.
- Type II error: Failing to reject a false null hypothesis, commonly called a false negative.
How to Choose MDE: Business First, Statistics Second
The cleanest way to set MDE is to ask: what is the smallest improvement that would change our decision? A 0.1% lift may be meaningful for a massive payment app, but irrelevant for a small B2B landing page.
In Indian value-commerce, a product-card or search-ranking change may affect conversion, average order value and returns differently across price-sensitive users, cities and categories. The primary sample-size driver is the small lift the team wants to detect on a noisy conversion metric, supported by traffic concentration, category mix and weekly shopping patterns. The strategic so what: high traffic does not remove the need for power planning - it only gives you more options.
Case Study: Etsy and the Discipline of Experiment Design
Etsy shows why marketplace experiments need disciplined sample-size planning: small interface changes can influence millions of buyer-seller interactions, but only well-powered tests separate real lift from marketplace noise.
Situation: Etsy operates a two-sided marketplace where buyers search for unique products and sellers depend on visibility. A small change in search, listing pages or recommendation modules can shift buyer conversion, seller exposure, revenue and long-term trust at the same time.
The move: Instead of treating every visible metric movement as proof, mature experimentation teams define the target metric, practical MDE, power, guardrails and traffic split before rollout. For a high-traffic surface, Etsy can test smaller effects. For a niche category or seller segment, the same MDE may require an impractically long test, so the team must widen the MDE, aggregate carefully or pick a more sensitive metric.
The outcome or lesson: The primary driver of good decisions is not “running A/B tests.” It is designing tests around the smallest business-relevant effect. Supporting drivers include clean randomization, enough eligible traffic, guardrail metrics for seller impact and avoiding early peeking.
How AI Changes Sample Size, Minimum Detectable Effect & Power
AI does not remove power analysis - it makes the design smarter. The biggest shift is that teams can now estimate variance, traffic eligibility and segment behavior faster before launching a test.
- Better pre-test planning: ML models can forecast eligible traffic, baseline conversion and seasonality by segment, helping teams see whether the desired MDE is feasible before engineering a test.
- Variance reduction: Techniques such as CUPED-style covariate adjustment use pre-experiment behavior to reduce noise, which can lower the required sample size when applied correctly.
- Faster test reading: LLMs can summarize experiment design documents, SQL outputs and metric dashboards, but they must not replace statistical checks for randomization, peeking and multiple comparisons.
Use ChatGPT or Claude like a test-design coach: paste the business problem, baseline rate, desired MDE, alpha, power and daily traffic, then ask it to compute required sample size, list assumptions, and generate three likely interviewer follow-ups. Verify calculations manually or with a trusted calculator.
Interview Relevance
“You are testing a new checkout design. The result is positive but not statistically significant. What would you check before deciding whether to launch?”
Use this sentence in interviews: “Statistical significance answers whether the observed lift is likely due to chance; power planning answers whether the test had a fair chance to detect the lift we care about.”
Common Mistake
The single biggest mistake is treating “not statistically significant” as “the idea does not work.” This costs candidates because it ignores underpowered tests, weak MDE planning and insufficient traffic. One-line fix: always ask, “Was the test powered to detect the smallest effect the business actually cares about?”
What to Revise Next
Next, revise Guardrail Metrics and Protecting Against Harm so your test recommendation does not optimize one metric while damaging the business. Then revise Reading Test Results: Significance, Effect Size & Confidence to confidently interpret what the experiment actually found.