Type I & Type II Errors, Power and Sample Size - Interview-Ready Guide
A payments team is about to tighten its fraud rule before a festive-sale weekend. Make it too strict and genuine customers get blocked; make it too loose and fraudulent transactions slip through - that is Type I and Type II error with money, trust and regulation attached.
- Type I error is a false alarm: rejecting a true null hypothesis. Its probability is alpha.
- Type II error is a missed signal: failing to reject a false null hypothesis. Its probability is beta.
- Power = 1 - beta: the chance your test detects a real effect when it exists.
- Sample size rises when the expected effect is small, variability is high, alpha is stricter, or desired power is higher.
- Alpha is not “accuracy.” A 5% alpha means 5% false-positive risk if the null is true.
- “Not significant” does not mean “no effect.” It may mean the study was underpowered.
- In business, choose alpha and power based on the cost of each error - fraud, medicine, pricing and campaigns have different trade-offs.
The big idea is simple: statistical testing is a decision under uncertainty. You compare evidence from a sample against a default claim called the null hypothesis; because the sample is incomplete, your decision can be right or wrong in two different ways.
Core Explanation: The Four Ideas You Must Connect
1. Null hypothesis is the default position. In most business tests, the null says “no difference” or “no effect.” Example: “The new checkout flow does not improve conversion.”
2. Type I error is the cost of acting on noise. If you wrongly conclude the checkout flow improved conversion, you may ship a worse experience, waste engineering time and misread customers. Lowering alpha reduces this risk, but makes it harder to detect true improvements.
3. Type II error is the cost of ignoring a real signal. If the new checkout flow truly improves conversion but your sample is too small, you may fail to detect it and reject a good idea. Power is your protection against this.
4. Sample size is the bridge between statistics and business reality. You do not choose sample size because “large is good.” You choose it from the minimum effect worth detecting, acceptable false-positive risk, desired power and expected variability.
Definitions You Can Say in One Breath
- Null hypothesis: The default claim tested statistically, usually stating no effect, no difference or no relationship.
- Type I error: Rejecting a true null hypothesis.
- Type II error: Failing to reject a false null hypothesis.
- Alpha: The probability of a Type I error when the null hypothesis is true.
- Beta: The probability of a Type II error when the alternative hypothesis is true.
- Statistical power: The probability of rejecting the null hypothesis when a specified alternative is true.
- Minimum Detectable Effect: The smallest effect size a study is designed to detect with chosen alpha and power.
The Trade-off: Why You Cannot Minimize Both Errors for Free
If you keep the same sample size, reducing Type I error usually increases Type II error. A stricter alpha, such as 1% instead of 5%, demands stronger evidence before you act. That protects you from false alarms, but it also makes real but modest effects harder to detect.
The way out is more information: increase sample size, reduce measurement noise, improve experimental design or focus on a larger, business-meaningful effect.
Key Measures and What Good Looks Like
These are the numbers you should name when discussing error rates, power and sample size. The “good” value depends on business risk, but these ranges are common starting points in analytics interviews.
Worked Example: Sample Size for an A/B Conversion Test
Suppose an ecommerce team wants to test a new checkout page. Current conversion is 10%. The team only cares if conversion rises to at least 12%, so the minimum detectable effect is 2 percentage points. Use 5% two-sided alpha and 80% power.
For a quick two-proportion approximation:
n per group ≈ 2 × (Z_alpha/2 + Z_beta)^2 × p × (1 - p) / d^2
- Z_alpha/2 for 5% two-sided alpha ≈ 1.96
- Z_beta for 80% power ≈ 0.84
- Average conversion p ≈ 0.11
- Effect size d = 0.02
n ≈ 2 × (1.96 + 0.84)^2 × 0.11 × 0.89 / 0.02^2
n ≈ 3,836 users per group, or roughly 7,672 users total. The exact number may vary slightly by software and correction method, but the management insight is stable: detecting a small 2-point lift needs thousands of observations.
If the page gets only 500 visitors per variant, “not significant” is almost meaningless - the test is likely underpowered for a 2-point lift.
How to Choose Alpha, Power and Sample Size in Practice
Mini Case Study: Razorpay and the Cost of False Positives in Payments Risk
Razorpay, an Indian payments platform, shows why Type I and Type II errors are not abstract formulas but operating trade-offs in digital payments risk.
Situation. A payment aggregator processes transactions for online merchants while managing fraud, chargebacks, regulatory expectations and customer experience. In this setting, a risk model or rule may flag a transaction as suspicious.
The statistical tension. A Type I error is a genuine customer being wrongly flagged or blocked - a false positive. A Type II error is a fraudulent transaction being allowed - a false negative. Both are costly, but in different ways: false positives hurt conversion and merchant trust; false negatives hurt loss rates, compliance posture and ecosystem confidence.
The move. Payment risk teams typically manage this by using layered signals rather than one blunt rule: transaction history, device signals, merchant category, velocity patterns, past disputes and manual review thresholds. They also need enough transaction data to estimate low-frequency fraud outcomes reliably, because rare events require larger samples than common conversion events.
Outcome and lesson. The primary driver of better decisions is not “more data” alone; it is better separation between genuine and risky transactions. Supporting drivers include clean labelling, merchant-level segmentation, careful threshold design, monitoring after deployment and compliance-aware review processes. The strategic lesson: in high-stakes analytics, the right error trade-off depends on the relative cost of false positives and false negatives.
How AI Changes Type I & Type II Errors, Power and Sample Size
1. AI makes false positives and false negatives more visible at scale. In fraud detection, credit underwriting, resume screening and medical triage, ML models do not simply “predict”; they set thresholds. Moving a threshold changes Type I and Type II errors immediately, so managers must ask for confusion matrices, calibration and subgroup performance.
2. AI experimentation needs stronger guardrails. AI-generated recommendations, chatbots and dynamic pricing systems can affect users differently across segments. A global average lift may hide weak power in smaller but important groups, such as new users, high-value customers or regulated customer categories.
3. AI can improve power through better design, not magic. Better targeting, variance reduction, CUPED-style adjustment in experiments and adaptive testing can reduce noise. But if the underlying labels are biased or the test is peeked at repeatedly, AI can increase false discoveries instead of reducing them.
Use ChatGPT or Claude to practise: paste a short A/B test case, ask it to identify the null hypothesis, Type I error, Type II error, alpha, power and sample-size drivers, then challenge its answer by asking, “What business cost changes the alpha decision?”
Interview Relevance
“A company runs an A/B test and gets a p-value of 0.08. The product manager says the feature has no effect. How would you respond, and how do Type I error, Type II error, power and sample size fit in?”
Use the sentence: “I would not say the feature has no effect; I would say we failed to reject the null at the chosen alpha, and I would check whether the test had enough power to detect the effect size we care about.”
Common Mistake
The mistake that costs candidates is saying “not statistically significant means no impact.” It costs you because it ignores Type II error and power. The fix: say “fail to reject the null”, then discuss sample size, MDE and whether the test was powered enough.
What to Revise Next
Now that you can reason about error risk and sample size, move to test selection. Revise Choosing the Right Test: t-Test, Chi-Square & Analysis of Variance first, then Non-Parametric Tests: What to Use When Assumptions Break. That sequence will help you answer both “what can go wrong?” and “which test should I use?” with confidence.