Randomisation, Control Groups & Assignment Mistakes - Interview-Ready Experiment Design
A delivery app changes its dispatch algorithm during Friday dinner rush. Orders look faster in the new version - until you realise the same rider pool served both “test” and “control,” so the control group was never truly untouched.
- Randomisation means assigning eligible units to variants by chance so groups are comparable before treatment.
- A control group estimates the counterfactual: what would have happened without the new feature, offer or policy.
- The most important design choice is the unit of randomisation: user, session, order, store, school, city, zone or time block.
- Bad assignment creates selection bias, contamination, spillover or sample ratio mismatch, making results untrustworthy.
- Use individual randomisation when users are independent; use cluster, geo or switchback randomisation when behaviour spills across people or operations.
- Before trusting lift, check balance, assignment logs, missing data, contamination and guardrail metrics.
- Interview-safe line: “The design must preserve a credible counterfactual before the statistics can mean anything.”
The big idea is simple: an experiment is not “show A to some people and B to others.” It is a disciplined attempt to create two worlds that differ only in the treatment, then measure the difference.
Core Explanation - What Must Be True for an Experiment to Be Credible
Randomisation solves the “maybe they were already different” problem. If high-intent users all see the new checkout and low-intent users see the old one, higher conversion is not proof of a better checkout. Random assignment spreads known and unknown differences across groups by chance.
The control group is not a leftover group. It is the baseline world. In product, marketing, pricing, HR or operations experiments, the control group answers: “What would have happened if we had not intervened?”
Assignment is the act of deciding which experimental unit receives which condition. The unit can be a user, session, account, household, store, branch, classroom, pin code, city, delivery zone or time block. If you choose the wrong unit, even perfect statistical testing cannot rescue the experiment.
The Four Assignment Choices You Must Know
Most interview answers become strong when you name the assignment unit and justify it. Use this ladder:
Here is the practical rule: randomise at the lowest level that avoids spillover. Lower levels give more statistical power; higher levels protect causal validity.
Assignment Mistakes That Break Experiments
Metrics and Diagnostics to Track Before You Trust Lift
Do not jump directly to “conversion increased.” First verify that the experiment was assigned and measured correctly.
Worked Example - When a Nice Lift Should Still Be Rejected
Suppose an e-commerce app tests a new checkout flow with a planned 50:50 split.
The apparent effect is:
- Absolute lift = 10.4% - 10.0% = 0.4 percentage points.
- Relative lift = 0.4% / 10.0% = 4%.
But the assignment counts are suspicious. With 100,000 users and a planned 50:50 split, each group should have roughly 50,000 users. The chi-square SRM statistic is approximately:
((50,900 - 50,000)² / 50,000) + ((49,100 - 50,000)² / 50,000) = 32.4
That is far beyond what you would expect from normal random variation in a 50:50 split. So the right business answer is not “launch because conversion rose.” The right answer is: debug assignment first, then rerun or repair the experiment.
Definitions You Can Say in One Breath
- Randomisation: Assignment by chance so each eligible unit has a known probability of receiving each experimental condition.
- Control group: The group not receiving the intervention, used to estimate what would have happened without treatment.
- Treatment group: The group receiving the intervention whose causal impact is being measured.
- Unit of randomisation: The entity assigned to a condition, such as a user, store, zone or time block.
- Counterfactual: The unobserved outcome that would have occurred for the same unit under a different condition.
In Indian quick-commerce or food delivery, a dispatch-routing or delivery-fee experiment should not automatically be randomised at individual customer level. Riders, dark stores, restaurants and delivery zones are shared resources, especially during lunch and dinner peaks. The strategic so what: when operations are coupled, cluster or switchback assignment protects the counterfactual better than a neat user-level A/B split.
DoorDash: Assignment Unit as the Difference Between Signal and Noise
DoorDash has used switchback-style experimentation for marketplace operations where customers, merchants and Dashers interact through shared local capacity.
Situation: In a delivery marketplace, a change to dispatch, batching, ETA promises or incentives does not affect only one customer. It changes how drivers are allocated, how merchants prepare orders and how nearby customers experience wait times. If one customer is in treatment and another nearby customer is in control, both may compete for the same Dasher.
The move: Instead of relying only on user-level randomisation, marketplace teams use designs such as geo-time switchbacks: for a city zone and a time window, the system runs treatment; in another comparable window, it runs control. Over many rotations, time and geography are balanced to estimate the operational impact.
The lesson: DoorDash-style experimentation wins chiefly because the assignment unit matches the mechanism of interference. Supporting drivers are repeated alternation across time blocks, careful monitoring of demand peaks, and guardrails such as delivery time, cancellation and marketplace reliability. The lesson for interviews: in marketplaces, “random” is not enough - it must be random at the right level.
How AI Changes Randomisation, Control Groups & Assignment Mistakes
1. AI makes treatment targeting more tempting, but causal testing still needs randomisation. A model may predict which users are likely to churn and target only them with an offer. That is useful for personalisation, but it cannot prove the offer worked unless you hold out a random control group within the eligible segment.
2. AI improves experiment monitoring. ML systems can flag assignment anomalies, sample ratio mismatch, unusual missingness, bot traffic, device-specific bugs and guardrail harm faster than manual dashboards. The caveat: AI can alert you to a problem; it does not remove the need to inspect assignment logs.
3. AI increases interference risk through dynamic systems. Recommendation engines, surge pricing, ad auctions and delivery routing continuously adapt. If treatment changes the algorithm’s learning environment, control users may be indirectly affected through shared inventory, rankings or capacity.
Use ChatGPT or Claude to pressure-test an experiment design. Prompt: “Here is my proposed A/B test. Identify the unit of randomisation, possible spillovers, contamination paths, SRM checks, guardrail metrics and whether a cluster or switchback design is safer.” Then verify the final answer yourself using the framework above.
Interview Relevance
“You are testing a new delivery-fee algorithm for a food delivery app in Bengaluru. How would you design the experiment, choose the control group and avoid assignment mistakes?”
Say the phrase “unit of randomisation” early. It signals that you understand experimentation as causal design, not just dashboard comparison.
Common Mistake
The mistake that costs candidates is choosing individual randomisation when the treatment spills over through shared resources. It costs them because the control group is contaminated, so the estimated lift is not causal. One-line fix: match the assignment unit to how the treatment spreads - user, cluster, geo or time block.
What to Revise Next
Now that you can protect the counterfactual, move to the two questions that decide whether an experiment is worth running and safe to ship.