Hypothesis Testing for Interviews: Null, Alternative, p-Values and Significance
A p-value is not a truth meter. It does not tell you whether a campaign worked, a pricing change succeeded, or a new app screen is “good” - it tells you how surprising your data would be if there were actually no effect.
- Hypothesis testing is a structured way to decide whether sample evidence is strong enough to challenge a default belief.
- Null hypothesis H0 usually says “no effect, no difference, no relationship.”
- Alternative hypothesis H1 says the effect, difference, or relationship you are testing exists.
- p-value measures how unusual your observed result is if H0 were true - not the probability that H0 is true.
- Significance level alpha is your pre-decided cutoff for evidence, commonly 0.05 in many business and academic settings.
- If p-value ≤ alpha, reject H0; if p-value > alpha, fail to reject H0.
- A strong answer always reports both statistical significance and business significance.
Think of hypothesis testing as a disciplined business argument: start with a default claim, collect evidence, quantify surprise, then make a decision without pretending that one sample proves absolute truth.
Core Explanation: What Hypothesis Testing Actually Does
In business, you rarely see the full population. You see a sample - a few thousand users, a month of sales, a test group of customers, a survey of employees. Hypothesis testing asks: is the pattern in this sample strong enough to believe there is a real effect in the population?
The logic is deliberately conservative. We begin by assuming the boring explanation: nothing changed, no difference exists, no relationship is present. This is the null hypothesis. Then we ask whether the sample data is too unusual to comfortably fit that assumption.
The Four Terms You Must Never Confuse
The most common business version is an A/B test. Suppose a company tests a new product page against the current page. H0 says both pages convert equally. H1 says conversion is different. The p-value tells you whether the observed gap is too large to dismiss as sampling noise.
Definitions You Can Say in One Breath
- Null hypothesis H0: The default statistical claim that there is no effect, no difference, or no relationship.
- Alternative hypothesis H1: The competing claim that an effect, difference, or relationship exists in the population.
- p-value: The probability, assuming H0 is true, of observing a result at least as extreme as the sample result.
- Significance level alpha: The pre-set probability threshold for rejecting a true null hypothesis.
- Statistical significance: A result is statistically significant when its p-value is less than or equal to alpha.
One-Tailed vs Two-Tailed Tests
The alternative hypothesis decides whether the test is one-tailed or two-tailed.
In interviews, default to a two-tailed test unless the problem clearly says only one direction matters. This is safer because it does not ignore an unexpected negative result.
What to Report: The 5 Numbers That Make Your Answer Complete
Do not stop at “p is less than 0.05.” A manager needs the size of the effect, the uncertainty, and whether the result is worth acting on.
Worked Example: A/B Test With p-Value and Business Meaning
A food delivery app tests a new checkout design.
- Control: 500 purchases from 10,000 visitors = 5.0 percent conversion.
- Variant: 560 purchases from 10,000 visitors = 5.6 percent conversion.
- Observed lift: (5.6 percent - 5.0 percent) / 5.0 percent = 12 percent relative lift.
For a two-proportion z-test, the pooled conversion rate is 1,060 / 20,000 = 5.3 percent. The standard error is approximately 0.00317. The z-statistic is:
z = (0.056 - 0.050) / 0.00317 ≈ 1.89
The two-sided p-value is approximately 0.059. If alpha is 0.05, we fail to reject H0. If alpha is 0.10 and the test was exploratory, it may be considered statistically significant - but the team should still check business value, sample quality, and guardrail metrics such as refunds or customer complaints.
“The variant shows a positive lift, but at alpha 0.05 the evidence is not strong enough to call it statistically significant. I would not roll it out solely on this test; I would check power, practical impact, and possibly run a larger test.”
The Decision Matrix: What Your Test Can and Cannot Prove
A hypothesis test never proves H0 is true. It only tells you whether the sample evidence is strong enough to reject it. That is why the official wording is fail to reject H0, not “accept H0.”
Case Study: Swiggy and Controlled Experiments in Product Decisions
Swiggy is a useful Indian example because small app and logistics changes can affect customers, restaurants, and delivery partners at the same time.
Food delivery is a high-frequency, low-patience category. A button placement, restaurant ranking rule, coupon message, or payment flow can increase orders for one group while hurting cancellations, delivery partner utilization, or restaurant fairness elsewhere.
The disciplined move is not to rely on the loudest internal opinion. A platform like Swiggy can frame a test: H0 says the new experience does not change the target metric; H1 says it does. Users are randomly assigned to control and variant groups, the p-value is calculated for the primary metric, and guardrail metrics are checked before a wider rollout.
The lesson is not “Swiggy wins because of A/B testing.” The primary driver is controlled randomization, supported by large user traffic, strong event tracking, guardrail metrics, and staged rollouts. The strategic so what: hypothesis testing is valuable because product decisions become measurable, reversible, and less vulnerable to HiPPO bias - the highest paid person’s opinion.
How AI Changes Hypothesis Testing
AI does not remove hypothesis testing; it increases the number of ideas that need disciplined testing.
- Faster hypothesis generation: Teams can use LLMs to convert customer reviews, app feedback, and sales-call notes into testable H0 and H1 statements. The risk is producing many weak hypotheses, so prioritization matters.
- Smarter experiment monitoring: ML systems can flag unusual traffic mix, bot activity, sample-ratio mismatch, or segment-level anomalies before a team trusts the p-value.
- More multiple-testing risk: AI makes it easy to slice results by city, device, cohort, and offer type. Without correction or pre-registration, teams may find false “significant” results by chance.
Use ChatGPT or Claude to convert a business problem into H0, H1, metric, alpha, test type, and decision rule. Then ask it to challenge your setup: “What assumptions could make this p-value misleading?” For company prep, load public annual-report notes or product articles into NotebookLM and generate likely experiment-design interview questions.
Interview Relevance
“Suppose an e-commerce company says a new recommendation widget increased conversion from 4.0 percent to 4.3 percent in an A/B test. How would you decide whether to roll it out?”
The answer that stands out is: “I would not decide on p-value alone. I would combine statistical significance, practical lift, customer impact, and operational guardrails.”
Common Mistake
The biggest mistake is saying, “p = 0.03 means there is a 3 percent chance the null hypothesis is true.” That is wrong because p-value assumes H0 is true and measures how surprising the data is under that assumption. One-line fix: say, “If there were truly no effect, data this extreme would occur about 3 percent of the time.”
What to Revise Next
Now move from “how to decide significance” to “how decisions can go wrong” and “which test to choose.” Revise these next: