Choosing the Right Statistical Test for Interviews: t-Test, Chi-Square and ANOVA
One dashboard says the new app layout increased average order value. Another says payment failures changed by bank. A third says North, South and West zones have different satisfaction scores - and the trap is that all three look like “comparison problems,” but they need different statistical tests.
- Use a t-test when the outcome is numerical and you compare the mean of one or two groups.
- Use chi-square when both variables are categorical and you compare observed counts with expected counts.
- Use ANOVA when the outcome is numerical and you compare means across three or more groups.
- The fastest test-selection logic is: What is the dependent variable? Numerical points to t-test or ANOVA; categorical points to chi-square.
- Do not run multiple t-tests for three or more groups - it inflates false positives. Use ANOVA first, then post-hoc tests if needed.
- Always report the practical size of the result, not only the p-value: Cohen's d, Cramer's V or eta-squared.
Think of test selection as a sorting problem, not a memory problem. First identify the outcome variable, then count the groups, then check whether the assumptions are reasonable.
The Core Idea: Match the Test to the Question
A statistical test asks whether the pattern you see in sample data is large enough to treat as evidence, rather than random noise. The wrong test gives you a neat-looking p-value for the wrong question.
There are three moving parts:
- Outcome variable: the variable you are trying to explain or compare.
- Group variable: the category that separates observations, such as old vs new pricing or app vs web.
- Hypothesis: the claim being tested, usually framed as null hypothesis versus alternative hypothesis.
- t-test: A t-test compares means using the t distribution when population variance is unknown.
- Chi-square test: A chi-square test checks whether observed categorical counts differ from expected counts under the null hypothesis.
- ANOVA: ANOVA tests whether multiple group means differ by comparing between-group and within-group variation.
- p-value: The p-value is the probability of results at least this extreme if the null hypothesis is true.
t-Test, Chi-Square and ANOVA: The Two-Sided Comparison
The easiest contrast is this: t-test and ANOVA compare averages; chi-square compares counts. Averages belong to numerical outcomes. Counts belong to categorical outcomes.
Assumptions and Diagnostic Checks You Should Name
Good candidates do not just name the test; they name the conditions under which the test is credible. These are the practical checks to remember.
If variances are unequal in a two-group mean comparison, use Welch's t-test. If variances are unequal across multiple groups, use Welch ANOVA. If assumptions badly break, revise non-parametric alternatives next.
Worked Example: Choosing and Running a t-Test
Suppose an e-commerce operations team tests whether a new route-planning logic reduces average delivery time. The outcome is numerical, and there are two independent groups: old routing and new routing. That points to an independent t-test.
Hypothetical sample:
- Old routing: n = 40, mean = 52 minutes, standard deviation = 12 minutes
- New routing: n = 40, mean = 47 minutes, standard deviation = 10 minutes
The standard error of the difference is:
SE = sqrt((12² / 40) + (10² / 40)) = sqrt(3.6 + 2.5) = sqrt(6.1) = 2.47
The t statistic is:
t = (52 - 47) / 2.47 = 2.02
Interpretation: the sample shows a 5-minute lower average delivery time for the new logic. You would compare the t statistic with the t distribution or read the p-value from software. In a business answer, do not stop at “significant” - say whether a 5-minute reduction is operationally meaningful after considering rider workload, SLA impact and cost.
Why ANOVA Is Not Just “Many t-Tests”
If you compare three groups using repeated t-tests - A vs B, B vs C, A vs C - each test carries false-positive risk. ANOVA avoids this by asking one omnibus question first: is at least one group mean different?
If ANOVA is significant, it does not automatically tell you which groups differ. You then use a post-hoc comparison such as Tukey's HSD, or planned contrasts if you had specific comparisons in mind before analysis.
Chi-Square in Plain English
Chi-square is for categorical patterns. It compares what you observed with what you would expect if the variables were unrelated.
Example: a fintech company wants to know whether payment success is independent of payment mode. The table below is the structure, not a claim about any company's real data.
The null hypothesis says payment success and payment mode are independent. If the chi-square p-value is small, the data suggests an association. The next business step is not “UPI is better” automatically - you inspect standardized residuals, technical failure reasons, bank mix, user segment and transaction size.
Case Study: Flipkart Big Billion Days and Test Selection
Flipkart's Big Billion Days is a high-stakes Indian retail context where multiple business questions require different statistical tests, not one universal “A/B test result.”
Situation: During a major festive-sale event, an Indian marketplace like Flipkart has simultaneous questions across pricing, logistics, payments, merchandising and customer experience. Some questions involve averages, such as delivery time or basket value. Others involve category proportions, such as payment mode, cancellation reason or exchange preference.
The move: A strong analytics team would not pick one test for every problem. It would map each business question to the right test family: t-test for two-group mean comparisons, ANOVA for three or more group mean comparisons, and chi-square for categorical associations.
Outcome or lesson: The primary driver of good inference is correctly translating the business question into variable type and group structure. Supporting drivers are clean randomization, adequate sample size, assumption checks and effect-size interpretation. The strategic so what: better test selection prevents confident but wrong decisions during high-volume, high-pressure business moments.
How AI Changes Choosing the Right Test
AI does not replace statistical thinking, but it reduces the mechanical burden and makes mistakes easier to catch if you use it well.
- Natural-language analytics: Tools can translate “compare average delivery time across three zones” into an ANOVA-style analysis plan, but you must still verify variable types and assumptions.
- Automated diagnostics: Modern analytics workflows can automatically flag unequal variances, small chi-square expected counts, outliers and missing values before the test is run.
- Experiment analysis at scale: AI-assisted platforms can summarize many A/B tests, detect segment-level heterogeneity and highlight where a statistically significant result may be commercially tiny.
Load a case dataset description into ChatGPT or Claude and ask: “Identify the dependent variable, independent variable, variable types, number of groups, correct test, assumptions and effect size to report.” Then challenge the answer by asking, “What test would be wrong here and why?”
Interview Relevance
“A company wants to know whether a new campaign improved average order value, whether payment failures differ by payment mode, and whether satisfaction differs across four regions. Which statistical tests would you use?”
Use this sentence in interviews: “I would first classify the dependent variable as numerical or categorical; then I would check the number of groups and the study design before choosing the test.” It sounds simple because it is exactly how good analysts think.
Common Mistake
The biggest mistake is choosing a test by memorized keyword instead of variable type. “Compare” does not automatically mean t-test! The fix: always say the outcome type, group count and study design before naming the test.
What to Revise Next
Now revise what happens when these assumptions do not hold, and then move to relationships between variables.