Designing an Experiment: Hypothesis, Unit & Success Metric - Interview-Ready Framework
The biggest misconception about experiments is that they are won by clever statistics at the end. In reality, most bad experiments are already broken before launch - the hypothesis is vague, the unit is wrong, or the success metric quietly rewards the wrong behaviour.
- Experiment design is a decision system: hypothesis tells you what you believe, unit tells you who gets treatment, metric tells you what counts as winning.
- A good hypothesis is directional and testable: “If we add UPI-first checkout, payment completion will increase for eligible users.”
- The experimental unit must match the intervention: user-level nudges need user-level assignment; store-level price changes may need store-level assignment.
- The success metric must be pre-specified: do not choose the metric after seeing the result.
- Use one primary metric plus guardrails: optimise conversion, but protect cancellation rate, latency, margin, returns or customer complaints.
- Randomisation only works if units do not contaminate each other: if treated users influence control users, your estimate is biased.
- Best interview answer: business objective - hypothesis - unit - treatment/control - success metric - guardrails - decision rule.
Big Picture: The Three Decisions That Make or Break an Experiment
An experiment is not “try something and check the dashboard.” It is a disciplined way to isolate cause and effect: change one thing for one set of units, compare against a fair baseline, and decide using a metric chosen before the result is known.
Core Explanation: How to Design the Experiment
Think of experiment design as writing a contract with your future self. Before the data arrives, you commit to what you are testing, who is eligible, what will be compared, and what evidence will trigger action.
1. Hypothesis: The Causal Claim
A hypothesis is not a hope. “Improve checkout” is a goal; “showing UPI as the first payment option will increase payment completion among Android users” is a hypothesis.
Hypothesis: A statistical hypothesis is a testable statement about a population parameter using sample data.
A strong business hypothesis has four parts:
Weak: “Let us test a new checkout page.”
Strong: “For eligible mobile users, reducing checkout from three screens to two will increase completed orders without increasing cancellations.”
2. Experimental Unit: The Entity That Gets Assigned
The experimental unit is where many candidates lose the plot. If the treatment is assigned to users, analyse users. If the treatment is assigned to stores, analyse stores. If you assign at one level and analyse at another without care, you can create false confidence.
Experimental unit: The smallest entity independently assigned to a treatment condition.
3. Success Metric: The Pre-Agreed Definition of Winning
The success metric must be tied to the business decision, sensitive to the intervention, and hard to game. Usually, you need one primary metric and several guardrail metrics.
Success metric: A pre-specified measure used to judge whether the treatment achieved the experiment objective.
Metrics to Track in a Business Experiment
There is no universal “good conversion rate” across industries, so compare treatment against control. For interview answers, show that you know both business metrics and experiment-quality metrics.
Worked Example: Testing a New Signup Flow
Suppose a fintech app tests a shorter signup form.
The pooled conversion rate is 1,080 / 20,000 = 5.4%. The standard error is approximately 0.3196 percentage points, so the z-score is 0.8 / 0.3196 = 2.50, giving a p-value around 0.012. If the pre-set significance level was 5%, the result is statistically significant - but you still check guardrails such as fraud rate, KYC rejection rate and support tickets before shipping.
The Experiment Design Loop
Real experimentation is a loop, not a one-time report. Each result either ships the change, kills the idea, or creates the next sharper hypothesis.
Definitions You Should Be Able to Say Cleanly
An Indian payments or commerce app testing “UPI as the default payment option” should not measure only click-through on the UPI button. The better design uses eligible user as the unit, payment completion rate as the primary metric, and guardrails such as transaction failure rate, support contacts, fraud flags and RBI/NPCI compliance constraints. The strategic point: in regulated Indian digital journeys, a higher click rate is not success if payment reliability or trust worsens.
Case Study: Etsy Search Experiments - Why Unit and Metric Matter
Etsy uses controlled experimentation to improve marketplace search and recommendations, where the challenge is balancing buyer conversion with seller ecosystem health.
Etsy is a useful experiment-design case because search changes are tempting to judge only by immediate clicks or purchases. But in a two-sided marketplace, the “winning” ranking algorithm must serve buyers while keeping sellers discoverable and the marketplace healthy.
Situation: Buyers come to Etsy looking for distinctive products, but search ranking has to handle relevance, quality, delivery expectations, personalization and seller diversity. A ranking change may increase short-term clicks yet over-concentrate traffic on a few sellers or promote items that later disappoint customers.
The move: Etsy’s experimentation approach for search and recommendations is to compare ranking variants in controlled tests, choosing units carefully and pairing commercial metrics with marketplace guardrails. The primary driver of better experimentation is not just data volume; it is aligning the unit and metric to the marketplace mechanism. Supporting drivers include robust instrumentation, segmentation, search-quality diagnostics and guardrails for seller and buyer experience.
Outcome or lesson: The strongest learning from marketplace experimentation is that a metric can be locally correct but strategically incomplete. A search test that improves short-term conversion but harms seller diversity, delivery quality or buyer trust is not a clean win. The experiment succeeds only when the primary metric improves and guardrails remain healthy.
How AI Changes Designing an Experiment: Hypothesis, Unit & Success Metric
AI does not remove the need for experimental discipline. It increases the number of ideas teams can test - which makes pre-specifying hypothesis, unit and metric even more important.
- AI generates more hypotheses faster: Teams can use LLMs to mine call-center transcripts, app reviews, CRM notes and funnel drop-offs for testable friction points. The risk is idea overload, so each AI-suggested hypothesis must still map to a business decision.
- AI improves metric design and anomaly detection: ML can detect sample ratio mismatch, logging gaps, segment anomalies and guardrail breaches earlier than manual dashboard checks.
- AI enables adaptive experimentation: Multi-armed bandits and personalization models can shift traffic toward better-performing variants, but they are not a substitute for clean causal tests when the goal is learning.
Track these measures when AI is involved in experiment design and monitoring:
Use NotebookLM or ChatGPT like a review partner: paste the experiment brief and ask, “Identify the hypothesis, experimental unit, primary metric, guardrails, contamination risks and decision rule. Flag anything ambiguous.” Then rewrite the brief until every answer is explicit.
Interview Relevance
“Suppose an Indian food delivery app wants to test a new free-delivery banner for premium users. How would you design the experiment?”
Say the sentence “I would choose the unit based on where the treatment is assigned.” Interviewers immediately hear that you understand causal design, not just dashboards.
Common Mistake
The mistake: candidates jump straight to “compare conversion rates” without defining the experimental unit. This costs them because the whole causal estimate depends on correct assignment - user, session, order, store and city are not interchangeable! Fix: say, “First I will identify the unit that independently receives the treatment, then choose the metric at that level or adjust for clustering.”
What to Revise Next
Next, revise Randomisation, Control Groups & Assignment Mistakes to understand how fair comparison is created, then Sample Size, Minimum Detectable Effect & Power to know whether your experiment can actually detect the effect you care about.