Peeking, Early Stopping & Sequential Testing: Answer A/B Test Decisions Without Falling for False Wins
A product manager opens the A/B test dashboard at 11:42 pm: the new checkout button is up 6 percent and the p-value has just dipped below 0.05. By morning, the effect is gone - not because users changed, but because the team treated a noisy interim result as truth.
- Peeking means repeatedly checking live experiment results and acting on them without a valid interim-testing plan.
- Early stopping is valid only when the stopping rule is pre-specified or statistically adjusted.
- Naive peeking inflates false positives: with 14 independent 5 percent looks, false-positive risk is approximately 1 - 0.95^14 = 51.2 percent.
- Sequential testing allows interim looks while controlling Type I error through alpha-spending, always-valid p-values, SPRT or Bayesian decision rules.
- Do not stop only because the treatment is currently ahead. Check minimum runtime, sample size, guardrails, sample-ratio mismatch and business impact.
- The interview-safe answer: define the risk, quantify error inflation, propose a valid stopping framework, then connect the decision to business cost.
The big picture is simple: experiments are a funnel of evidence. Peeking corrupts the final decision by turning every interim glance into another chance to declare victory.
Core Explanation: Why Peeking Creates False Winners
Peeking feels harmless because the data is real. The trap is that the rule changed after seeing the evidence. A fixed-horizon A/B test assumes you decide after the planned sample size or time window. If you check the p-value every hour and stop the first time it crosses 0.05, you are no longer running that test.
The statistical issue is Type I error: rejecting the null hypothesis when there is actually no real effect. If one test has a 5 percent false-positive rate, repeated unadjusted looks give the experiment many chances to be accidentally significant.
The Three Ideas You Must Keep Separate
Worked Example: How One Extra Look Becomes a Big Problem
Suppose an e-commerce team tests a new cart design with no true effect. They plan to check the result daily for 14 days and stop if p < 0.05.
If the daily looks were independent, the chance of at least one false positive would be:
False-positive risk = 1 - (1 - Îą)k = 1 - 0.9514 = 51.2 percent
Real interim looks are correlated, so this is an approximation, not an exact operating characteristic. But the lesson is interview-critical: a 5 percent threshold is not a 5 percent false-positive risk once you repeatedly test and stop opportunistically.
Imagine Razorpay tests a simplified UPI checkout flow for Indian merchants. If the team stops on the first weekday where conversion rises, the result may reflect payday timing, bank success rates or traffic mix rather than the checkout change. The strategic so what: in Indian digital payments, experiment validity needs both statistical discipline and operating guardrails such as payment success rate, latency and refund complaints.
How Sequential Testing Works
Sequential testing does not say, "never look." It says, "look using rules that were designed for looking." The key is that the experiment has a planned path before data starts flowing.
Key Metrics and Decision Checks
Definitions
- Peeking: Inspecting interim experiment results without a valid stopping or error-control plan.
- Early stopping: Ending a study before its planned maximum sample because a pre-specified decision rule is met.
- Sequential testing: A design that analyzes accumulating data at interim looks while controlling long-run error rates.
- p-value: The probability, under the null model, of observing results at least as extreme as the data.
- Type I error: Rejecting a true null hypothesis.
- Wald's SPRT: A sequential test that compares accumulating likelihood evidence against accept, reject or continue boundaries.
Case Study: Optimizely and the Dashboard Peeking Problem
Optimizely built statistical tooling for real-time experimentation because users naturally wanted to monitor tests continuously without turning every dashboard refresh into a false win.
Situation: Web and product teams wanted fast A/B testing. The classic fixed-horizon method assumed the team would wait until the planned sample size, but real teams checked dashboards repeatedly. That created a practical gap between statistical theory and product behaviour.
The move: Optimizely responded by building experimentation statistics around monitoring-friendly inference, including sequential testing ideas and corrections for multiple comparisons in its experimentation product. The primary driver was statistical design that acknowledged continuous monitoring. Supporting drivers were dashboard UX, clearer decision messaging and methods to reduce false discoveries when teams tested multiple goals or variants.
Outcome or lesson: The lesson is bigger than one tool: if your users will keep looking, your method must be valid under repeated looks. A strong experimentation culture does not rely on discipline alone; it designs guardrails into the process.
How AI Changes Peeking, Early Stopping & Sequential Testing
1. AI generates more variants, which increases multiple-testing risk. A growth team can now ask an LLM to create 30 headline or notification variants in minutes. That is useful, but it makes false discoveries easier unless the experiment uses multiplicity correction, holdouts or hierarchical testing.
2. AI-driven experimentation pushes teams toward adaptive allocation. Bandits and reinforcement-learning systems can shift traffic toward promising variants during the test. This is not a normal fixed A/B test; it needs sequential or adaptive inference and careful separation between learning, validation and deployment.
3. AI summaries can accidentally encourage peeking. LLM-generated experiment summaries may highlight "currently significant" movements before a test matures. Teams need the AI layer to display planned stopping rules, guardrail status and uncertainty, not just a confident narrative.
Use ChatGPT or Claude to draft an experiment protocol: hypothesis, primary metric, guardrails, planned runtime, interim-look schedule and stopping rule. Then ask: "List every way this design could create false positives through peeking, multiple metrics or early stopping."
Interview Relevance
"An A/B test planned for 14 days shows p = 0.04 on day 3. The product manager wants to stop and ship. What would you recommend?"
A polished answer does not sound anti-speed. Say: "I am not against stopping early; I am against stopping early without a rule that preserves decision quality."
Common Mistake
The costly mistake is saying, "p < 0.05, so ship it!" without asking when and how often the result was checked. Why it costs candidates: it shows you know the vocabulary of A/B testing but not the operating reality of experiments. One-line fix: always ask, "Was this a fixed-horizon test or a valid sequential design?"
What to Revise Next
Now move from "when can I trust the stopping decision?" to "when can I trust the experiment itself?" Revise Common Experiment Failures: Novelty, Contamination & Seasonality, then study When You Cannot Randomise: Quasi-Experiments & Difference-in-Differences to handle real business situations where clean A/B testing is impossible.