Common Experiment Failures: Answer Novelty, Contamination and Seasonality Like a Pro

A green dashboard after three days can be dangerously seductive. A new checkout banner lifts conversion on Friday, the team celebrates, and two weeks later the effect quietly vanishes because users were only reacting to something new - not something better.

Big Picture: A Good Experiment Is a Causal Loop, Not Just a P-Value

An A/B test looks simple: split users, show one group a change, compare outcomes. In reality, validity depends on a loop staying clean from assignment to inference. Novelty attacks time stability, contamination attacks group separation, and seasonality attacks calendar representativeness.

Core Explanation: The Three Ways Experiments Fool You

The central idea is simple: an experiment should isolate the effect of one intervention. These failures add extra forces that masquerade as treatment impact.

Measured lift = true treatment effect + novelty effect + contamination effect + seasonality effect + random noise.

1. Novelty Effect: The Lift That Melts

Novelty effect is temporary behaviour change caused by a feature being new, not by its lasting value. It is common in product UI changes, loyalty nudges, gamification, notifications and pricing displays.

Suppose a food delivery app adds a bright “recommended combo” card. Users may click it because it is prominent and unfamiliar. But after repeated exposure, the card becomes part of the furniture. If you call the test too early, you mistake curiosity for durable preference.

Worked Example: How Novelty Creates a False Win

A product team runs a two-week A/B test on a new homepage recommendation widget.

The overall result looks positive: 5.55% versus 5.10%, an 8.8% relative uplift. But the persistence ratio is Week 2 lift divided by Week 1 lift = 0.1 / 0.8 = 12.5%. That is a red flag. The correct decision is not “ship immediately”; it is “extend, segment, or hold out a long-term control.”

2. Contamination: When Control Is No Longer Control

Contamination happens when people assigned to control are exposed to treatment, directly or indirectly. It is especially dangerous in social networks, marketplaces, logistics, B2B sales and offline operations.

Imagine testing a seller incentive on an Indian marketplace. If treated sellers cut prices, control sellers may react competitively. If delivery agents learn a new routing SOP from colleagues, the control zone may improve too. The experiment underestimates or misattributes the real effect because the boundary between groups has leaked.

3. Seasonality: When the Calendar Pretends to Be the Treatment

Seasonality is predictable variation caused by time-based patterns such as day of week, festivals, weather, salary cycles or industry events. It is not random noise; it is structured noise.

The fix is not always “avoid seasonal periods.” Sometimes peak periods are exactly what the business cares about. The fix is to design for them: stratify by time, run across full weekly cycles, compare like-for-like windows, or explicitly test “normal period” and “seasonal period” separately.

Diagnostics: Six Metrics That Tell You Whether the Experiment Is Trustworthy

These are not vanity metrics. They are validity checks. Use them before interpreting revenue, conversion or retention uplift.

Precise Definitions to Say in One Breath

LinkedIn: Network Experiments Where Contamination Is the Core Problem

Situation: In a professional network, actions are interconnected. If LinkedIn tests a feature that changes invitations, recommendations, feed ranking or notifications, one member's treatment can influence another member's outcomes. A treated user may send more connection requests, which changes the experience of control users who receive them.

The strategic move: LinkedIn researchers and practitioners have publicly discussed network experimentation methods, including cluster-based approaches, to reduce interference between treatment and control. Instead of assuming every user is independent, the experiment design can assign connected groups more carefully and track exposure paths.

Outcome or lesson: The primary driver of better inference is recognizing that spillovers are part of the product, not a measurement inconvenience. Supporting drivers include cluster randomization, exposure logging, network-level guardrails and interpretation of both direct and indirect effects. The lesson for interviews: in networked products, “randomize users 50:50” may be statistically neat but practically wrong.

A shallow answer says “LinkedIn should A/B test the feature.” A strong answer says “LinkedIn must design the unit of randomization around the network effect it expects.”

How AI Changes Common Experiment Failures: Novelty, Contamination & Seasonality

AI does not remove experimentation discipline. It makes bad experiments easier to run at scale and good diagnosis faster.

1. AI detects hidden seasonality faster. ML models can flag calendar interactions across weekday, festival, weather, campaign and geography variables before a team overreads a short test.
2. AI helps monitor contamination paths. Graph analytics can detect whether treatment exposure is spreading through users, sellers, drivers, sales agents or social connections.
3. AI increases novelty risk in personalization. Generative interfaces, AI recommendations and dynamic creatives often create sharp early engagement because users explore them. Long-term holdouts become more important, not less.

Interview Relevance

Common Mistake

What to Revise Next

Next, revise When You Cannot Randomise: Quasi-Experiments & Difference-in-Differences to handle real-world situations where clean A/B testing is impossible. Then move to Multi-Armed Bandits and Continuous Optimisation, where the goal shifts from one fixed experiment to ongoing allocation and learning.