Classification Metrics Interview Guide: Precision, Recall & the Confusion Matrix
A fraud model can look excellent on a dashboard and still be terrible on the shop floor. Before tuning, the team sees a flood of risky transactions; after tuning, the dashboard says “high accuracy” - but the operations team asks a sharper question: are we blocking good customers, or missing bad actors?
- Confusion matrix splits predictions into true positives, false positives, false negatives and true negatives.
- Precision = TP / (TP + FP): when the model says “positive”, how often is it right?
- Recall = TP / (TP + FN): of all real positives, how many did the model catch?
- Accuracy can mislead when one class is rare, such as fraud, default, churn or disease.
- F1 score balances precision and recall using their harmonic mean.
- The right metric depends on cost: false alarms hurt conversion; missed positives hurt risk, safety or revenue.
- Thresholds are business levers: changing the cut-off changes precision, recall and workload.
Think of classification metrics as a translation layer between a model and a business decision. The model may output a probability, but the business needs an action: approve, block, call, inspect, retain, escalate. The confusion matrix shows what happened after that action was taken.
The Big Idea: Metrics Are Cost Decisions
A classification model predicts a category, such as fraud or not fraud, churn or not churn, default or no default. In a binary classification problem, the positive class is the event you care about detecting - fraud, churn, disease, default, conversion or complaint escalation.
The key question is not “Which metric is best?” The key question is: Which error is more expensive?
- If a bank wrongly rejects a good borrower, that is a false positive if “risky borrower” is the positive class.
- If a bank approves a borrower who later defaults, that is a false negative.
- If missing a positive is dangerous, optimize recall.
- If accusing or blocking someone is costly, optimize precision.
Reading the Confusion Matrix Without Panic
Start with the positive class. Then read every cell as a sentence.
Once this is clear, every metric becomes intuitive. Precision looks across the row of predicted positives. Recall looks down the column of actual positives.
Metrics You Must Know, With Formulas and Good Values
In interviews, do not quote a metric in isolation. State the formula, the range, and what “good” means relative to the base rate, business cost and validation data.
A Worked Example You Can Solve in 60 Seconds
Suppose a lender tests a default-risk classifier on 1,000 loan applications. Let positive = likely defaulter. In reality, 100 applicants default. The model flags 120 applicants as risky, and 80 of those actually default.
- TP = 80: risky and actually defaulted.
- FP = 40: flagged risky but did not default.
- FN = 20: not flagged but later defaulted.
- TN = 860: not flagged and did not default.
The interpretation is the answer. If the lender uses the model only to send applications for manual review, 66.7% precision may be workable. If the model automatically rejects applicants, false positives become reputational and revenue risks, so the threshold may need to rise.
Definitions
- Confusion matrix: A table that cross-classifies actual and predicted classes to count TP, FP, FN and TN.
- Precision: The share of predicted positives that are actually positive.
- Recall: The share of actual positives that the model correctly identifies.
- False positive: A negative case incorrectly predicted as positive.
- False negative: A positive case incorrectly predicted as negative.
- Threshold: The probability cut-off used to convert a model score into a class label.
Case Study - Razorpay Thirdwatch: Fraud Detection Is a Precision-Recall Trade-off
Razorpay acquired Thirdwatch, an AI-based fraud detection startup, to strengthen fraud-risk screening for Indian digital commerce and payments.
Situation: Indian digital commerce and payments operate at high speed, across cards, UPI, wallets, netbanking and merchant checkout flows. Fraud teams must catch suspicious transactions without breaking legitimate customer journeys. A model that catches everything by blocking too aggressively can damage conversion; a model that lets too much through can increase fraud losses and compliance pressure.
The move: Razorpay acquired Thirdwatch in 2019, bringing in AI-led fraud detection capability for merchants. The core classification task is simple to state but hard to execute: label a transaction or order as suspicious or genuine using signals such as transaction behavior, device patterns, merchant context and historical outcomes. The operational decision is not merely “fraud or not fraud”; it may be allow, block, step-up verification or manual review.
Primary driver: the win comes chiefly from cost-sensitive thresholding - choosing a decision cut-off that balances missed fraud against false alarms. The supporting drivers are equally important: good feedback loops from confirmed fraud outcomes, segment-wise monitoring by merchant or payment mode, human review for borderline cases, and alignment with India-specific payment risk and compliance expectations.
Outcome or lesson: In fraud classification, “high recall” alone is not a strategy. If every suspicious score becomes a block, false positives hurt genuine customers and merchants. The smarter answer is to tune thresholds by use case: high-risk cases may be blocked, medium-risk cases may go to step-up verification, and low-risk cases may pass with monitoring.
How AI Changes Classification Metrics
AI makes classification faster, more adaptive and more widely embedded in business workflows. It also makes metric discipline more important because models now affect decisions continuously, not just in one offline report.
- More decisions are scored in real time. Fraud checks, lead scoring, churn prediction, credit underwriting and support-ticket routing increasingly use live model scores. This shifts the discussion from one static confusion matrix to threshold monitoring by channel, customer segment and time period.
- Rare-event problems need precision-recall thinking. Many AI use cases involve rare positives - fraud, churn, default, safety incidents or abuse. Accuracy becomes especially dangerous, while PR-AUC, precision at top K and recall at a fixed false-positive rate become more useful.
- Monitoring now includes drift and fairness. A model can perform well at launch and degrade when customer behavior, seasonality, fraud tactics or data capture changes. AI teams track performance, calibration and segment-level gaps after deployment.
Use these measures when discussing AI classification systems in 2026:
Paste a confusion matrix, the business problem and the positive-class definition into ChatGPT or Claude. Ask it to compute precision, recall, F1, specificity and accuracy, then force it to explain which metric should drive the decision based on false-positive and false-negative costs. Verify the formulas yourself before using the answer.
Interview Relevance
“A fraud detection model has 98% accuracy, but the business team is unhappy. Explain why accuracy may be misleading, and tell me whether you would optimize precision or recall.”
Use one sentence like this: “I would not optimize accuracy first because fraud is rare; I would define fraud as the positive class, compare the cost of missed fraud versus blocked genuine transactions, and tune the threshold using precision-recall trade-offs.”
The biggest mistake is saying “high accuracy means the model is good” without checking class imbalance and error cost. Fix it in one line: draw the confusion matrix, name the positive class, then compare the cost of FP versus FN!
What to Revise Next
Next, revise Regression & Ranking Metrics, and Picking the Right One so you can handle non-classification model evaluation. Then move to Explaining a Model to a Business Audience to convert metrics into decisions, risks and trade-offs.