Experimentation & A/B Testing for Analysts is a structured track of 12 lessons that build a complete, interview-ready understanding of the topic. Work through them in order, then use the quiz and flashcards in each lesson to revise.
What this course covers
- Why Experiments Beat Observational Analysis - What randomisation buys you that no amount of controlling for variables can.
- Designing an Experiment: Hypothesis, Unit & Success Metric - Writing a testable hypothesis and choosing the randomisation unit correctly.
- Randomisation, Control Groups & Assignment Mistakes - How assignment goes wrong in practice, and the sanity checks that catch it.
- Sample Size, Minimum Detectable Effect & Power - Sizing a test before launch, and what to do when the traffic is not there.
- Guardrail Metrics and Protecting Against Harm - The metrics that must not move, and how to set an automatic stop.
- Reading Test Results: Significance, Effect Size & Confidence - Interpreting a result set honestly, including a flat result that is still informative.
- Peeking, Early Stopping & Sequential Testing - Why checking daily inflates false positives, and the methods that permit it.
- Common Experiment Failures: Novelty, Contamination & Seasonality - The five ways real tests break, each with the diagnostic that reveals it.
- When You Cannot Randomise: Quasi-Experiments & Difference-in-Differences - Getting a defensible causal read without a clean randomised test.
- Multi-Armed Bandits and Continuous Optimisation - When adaptive allocation beats a fixed split, and what you give up for it.
- Building an Experimentation Culture and a Test Backlog - Prioritising tests, documenting results and stopping the same test being rerun.
- Case Study: A Full Experiment from Hypothesis to Ship Decision - One experiment run end to end, including the numbers and the final call.