Multi-Armed Bandits and Continuous Optimisation - Interview-Ready Guide
Yesterday, three homepage banners got equal traffic until the experiment ended. Today, the system notices one banner is winning by noon and quietly sends more users there while still learning from the others.
That shift - from “test, wait, decide” to “learn and optimise continuously” - is the heart of multi-armed bandits.
- A multi-armed bandit is a sequential decision problem: choose among uncertain options, learn from rewards, and maximise total reward over time.
- The core tension is exploration - try uncertain options - versus exploitation - use the option that currently looks best.
- Use bandits when traffic is valuable, rewards are quick, and you want optimisation during the experiment - for example banners, recommendations, offers, notifications or ad creatives.
- Use classic A/B testing when you need clean causal measurement, stakeholder sign-off, or a stable estimate of long-term impact.
- Common algorithms: epsilon-greedy, UCB, Thompson sampling, and contextual bandits.
- Track cumulative reward, regret, reward rate, exploration share, propensity logging and guardrail breaches.
- The biggest trap: calling bandits “better A/B tests” without discussing bias, holdouts, logging and guardrails.
The Big Picture
A multi-armed bandit is best understood as a live learning loop. Instead of freezing traffic equally across variants, the system keeps updating its belief about each option and reallocates traffic as evidence arrives.
Core Explanation: How Multi-Armed Bandits Work
The name comes from casino slot machines. Each machine is an “arm”; each arm has an unknown payout. Your job is not just to identify the best arm eventually, but to earn as much reward as possible while learning.
In business terms, the arms could be email subject lines, product recommendations, loan offer copy, app push notifications, search result rankings or homepage banners. The reward could be a click, purchase, subscription, completed KYC, add-to-cart action or revenue per session.
Multi-armed bandit: a sequential decision problem where an agent repeatedly chooses among uncertain options to maximise cumulative reward.
The Explore-Exploit Trade-off
Every bandit problem has two competing jobs:
- Explore: give traffic to options you do not know enough about.
- Exploit: give more traffic to the option that currently appears best.
If you explore too much, you waste users on weak options. If you exploit too early, you may lock onto a false winner.
A/B Testing vs Multi-Armed Bandits
The cleanest interview distinction is this: A/B testing is built for measurement; bandits are built for optimisation under uncertainty.
Four Common Bandit Algorithms
You do not need to derive the maths in most MBA interviews. You do need to know what each algorithm is trying to do.
When to Use Bandits - and When Not To
Use a bandit when the reward is observable quickly and every unit of traffic has opportunity cost. Do not use it blindly when the outcome is delayed, the decision affects customer trust, or the organisation needs a clean pre-post decision.
Worked Example: Why Bandits Can Earn More During the Test
Imagine an Indian fintech app testing three UPI cashback messages on its payment success screen. The reward is a click on “activate offer”. After 3,000 impressions, each arm has received 1,000 impressions.
There are 7,000 impressions left. A fixed A/B/C test would keep splitting traffic equally. Expected additional clicks, using the observed rates, are:
Equal split: 2,333 × 3.0% + 2,333 × 4.5% + 2,333 × 2.5% ≈ 233 clicks.
A simple bandit might send 70% of remaining traffic to B and 15% each to A and C.
Adaptive split: 4,900 × 4.5% + 1,050 × 3.0% + 1,050 × 2.5% ≈ 278 clicks.
The bandit earns about 45 more clicks during the same experiment. This is illustrative, not a production-grade proof, because real systems would account for uncertainty, seasonality, user segments and guardrails.
Metrics to Track in Continuous Optimisation
A bandit system should be judged on both business outcome and learning quality. A higher click rate is not enough if the system is biased, unlogged or hurting downstream metrics.
Definitions You Should Be Able to Say Clearly
- Arm: one available option the algorithm can choose, such as a banner, offer, product or message.
- Reward: the measurable outcome received after choosing an arm, such as a click, purchase or revenue.
- Exploration: deliberately trying uncertain options to learn their reward potential.
- Exploitation: choosing the option currently believed to deliver the highest reward.
- Regret: the reward lost because the algorithm did not always choose the best possible option.
- Contextual bandit: a bandit that uses features about the user or situation before selecting an option.
Case Study: The Washington Post and Bandito
The Washington Post built Bandito to adaptively test story presentations, so a newsroom could learn from readers without waiting for a traditional test cycle.
Digital publishers face a brutal optimisation problem: reader attention is perishable. If a headline, image or story treatment is weak for the first few hours, the opportunity may never return. A classic A/B test can measure which version was better, but it may spend too much valuable traffic on underperforming versions while the story is still hot.
The Washington Post addressed this with Bandito, an open-source tool associated with multi-armed bandit testing for content optimisation. Instead of giving each story presentation equal traffic until the end, the system could shift exposure toward better-performing choices while still reserving some traffic for learning.
The primary driver was the fit between the problem and the bandit structure: many editorial variants, fast reader feedback and high opportunity cost. Supporting drivers mattered too - a clear reward signal such as clicks or engagement, newsroom guardrails, sufficient traffic volume, and human editorial judgement to avoid optimising purely for sensationalism.
The lesson is not “let algorithms run the newsroom.” The sharper lesson is: bandits work when the business can define a fast reward, preserve human guardrails and accept adaptive traffic allocation.
In India, the same logic fits high-traffic digital surfaces such as Flipkart sale banners, Swiggy restaurant recommendations, UPI app offers or OTT thumbnail rows. The mechanics are India-specific because traffic spikes around events like festive sales, IPL matches or salary-credit days make wasted exposure especially costly. The “so what”: bandits help monetise short demand windows, but only if teams protect trust, latency and downstream conversion quality.
How AI Changes Multi-Armed Bandits and Continuous Optimisation
AI does not remove the explore-exploit problem. It makes the decision space larger, more personalised and faster-moving.
Practical student workflow: load a company product page, app screenshots and recent annual report excerpts into NotebookLM. Ask it to generate “five bandit use cases, likely reward metrics, guardrails and interview objections” for that company. Then use ChatGPT or Claude to pressure-test which use cases are actually bandit-suitable versus better suited to A/B testing.
Interview Relevance
“Suppose an e-commerce app wants to optimise homepage banners during a festive sale. Would you recommend A/B testing or a multi-armed bandit? Explain your approach.”
The strongest answer says: “Bandit for live optimisation, A/B or holdout for clean measurement.” That one sentence shows both business sense and statistical maturity.
Common Mistake
The mistake is saying “multi-armed bandits are better than A/B tests” as a blanket rule. It costs candidates because it ignores adaptive-allocation bias, delayed outcomes, guardrails and the need for causal inference. Fix: say bandits optimise live traffic, while A/B tests measure clean causal lift - and use logging or holdouts when you need both!
What to Revise Next
Next, move from the algorithm to the operating system around it. Revise Building an Experimentation Culture and a Test Backlog to understand how teams prioritise ideas, then study Case Study: A Full Experiment from Hypothesis to Ship Decision to see how a hypothesis becomes a launch, rollback or learning decision.