Train-Validation-Test Splits and Cross-Validation: Interview-Ready ML Evaluation
A credit-risk model looks brilliant on the spreadsheet - until it meets next month's borrowers and starts approving the wrong people. The danger is not always a weak algorithm; often, the model was allowed to practise on data that should have been its exam paper.
- Training data teaches the model patterns; validation data helps choose the model; test data gives the final, unbiased performance estimate.
- The test set is sacred: use it once at the end, not repeatedly while tuning.
- Cross-validation rotates the validation set across folds, giving a more stable estimate when data is limited.
- Use stratified splits for imbalanced classification, time-based splits for forecasting, and group splits when related records share a customer, store or patient.
- The biggest trap is data leakage: information from validation or test data accidentally enters training.
- Interview answer formula: state the purpose of each split, choose the split method, name the metric, explain leakage control, then say how you report final performance.
Think of model evaluation as a three-room testing centre: one room for learning, one for practice exams, and one locked room for the final exam. Cross-validation improves the practice exam by rotating who sits where, but it never removes the need for a clean final test when performance matters.
Core Explanation: What Each Split Does
Train-validation-test splitting is the practice of separating data so that model learning, model selection and final evaluation happen on different records. The logic is simple: if the same data teaches and judges the model, the score becomes optimistic.
- Training set: the data used to fit model parameters, such as coefficients, tree splits or neural-network weights.
- Validation set: the data used during development to tune hyperparameters and choose between candidate models.
- Test set: the untouched data used once to estimate how the final model may perform on unseen cases.
- Cross-validation: a resampling method that trains and validates a model across multiple data folds to estimate performance stability.
The split is not just an operational step; it is the difference between model development and model evidence. Training performance tells you how well the model memorised or learned. Validation performance tells you which model design is better. Test performance tells stakeholders whether the model deserves trust.
The Practical Split Ladder: From Basic to Interview-Strong
There is no universal magic split like 70-15-15. The right split depends on data size, class imbalance, time order and whether multiple rows belong to the same entity. In interviews, the candidate who names these conditions sounds much stronger than the candidate who memorises one ratio.
Cross-Validation: The Rotation That Reduces Luck
A single validation split can be unlucky. It may contain unusually easy customers, unusually hard stores or a class distribution that does not represent the real business. K-fold cross-validation reduces this split luck by dividing data into k parts, training on k-1 parts and validating on the remaining part, then rotating.
Which Split Should You Use?
In Indian digital payments, a risk model must catch suspicious transactions without blocking genuine merchants and customers. Razorpay acquired AI-based fraud detection company Thirdwatch in 2019, showing how central fraud analytics is to payment infrastructure. The evaluation lesson: for transaction-risk models, a random split can be misleading if fraud patterns change over time; a time-aware validation design is usually more realistic, supported by stratification for rare fraud cases and strict leakage checks around merchant or device identifiers.
Worked Example: 10,000 Records, 5-Fold CV and a Final Test
Suppose you are building a customer churn model with 10,000 labelled customer records.
The important discipline is not the exact 15 percent. It is the sequence: test last, validation during development, training for learning.
Evaluation Metrics to Track
Splits protect the evaluation process; metrics decide what performance means. Choose the metric that matches the business cost of errors.
Definitions: The Clean Version
- Hyperparameter: a modelling choice set before training, such as tree depth, learning rate or number of neighbours.
- Holdout validation: a single split where one part trains the model and another part validates model choices.
- K-fold cross-validation: validation where data is split into k folds and each fold serves as validation once.
- Data leakage: any situation where information unavailable at prediction time influences model training or validation.
- Nested cross-validation: an outer cross-validation loop estimates performance while an inner loop tunes hyperparameters.
Case Study: Kaggle and the Public Leaderboard Trap
Kaggle's competition design makes the validation-test distinction visible: a public leaderboard gives feedback, while a hidden private leaderboard protects the final ranking from overfitting.
Kaggle competitions often feel like a race to climb the public leaderboard. Participants submit predictions, receive a score, change features, tune models and submit again. That public score is useful, but it can become a disguised validation set: teams may slowly overfit to it by reacting to every small movement.
The strategic move in Kaggle's design is to separate visible feedback from final judgement. The public leaderboard is based on a visible scoring subset, while final standings are determined using hidden private labels. The primary driver of evaluation quality is this hidden private test set. Supporting drivers include a fixed metric, controlled submission rules in many competitions, and the fact that winning solutions are judged on unseen data rather than merely on iterative public feedback.
The takeaway for interviews: the more you look at a dataset while improving the model, the less it behaves like a true test set. Kaggle makes that lesson painfully clear.
How AI Changes Train, Validation & Test Splits, and Cross-Validation
AI has not removed the need for disciplined evaluation; it has made the discipline more important because models are easier to generate, tune and accidentally overfit.
- AutoML increases search risk: tools can try hundreds of feature transformations, algorithms and hyperparameters. That makes a single validation score easier to overfit, so nested cross-validation or a final untouched test set becomes more important.
- LLM applications need evaluation sets too: chatbots, retrieval-augmented generation systems and AI copilots need train/dev/test-style prompt sets, with human-rated answer quality, hallucination checks and task-completion measures.
- Synthetic data needs stricter leakage checks: AI-generated data can help with rare cases, but if synthetic examples are too close to test records, evaluation becomes contaminated.
Load this lesson and a company analytics case into NotebookLM, then ask: "Create five interview questions on train-validation-test splits, include one leakage trap, one time-series split question and one metric choice question." Then answer aloud and use ChatGPT to critique whether your split logic matches the business problem.
Interview Relevance
"You are building a churn prediction model for a telecom company. How will you split the data, validate the model and make sure your performance estimate is reliable?"
Say this line if you want to sound mature: "I would not use a random split if the business process is time-dependent, because future behaviour can leak into past predictions."
Common Mistake
The mistake that costs candidates is treating the test set like a second validation set - checking it repeatedly while changing features, thresholds or hyperparameters. That destroys its independence and makes the final score too optimistic. One-line fix: lock the test set, tune only on validation or cross-validation, and open the test set once at the end.
What to Revise Next
Now move from evaluation design to model behaviour. Revise Overfitting, Underfitting & the Bias-Variance Trade-off to understand why validation scores differ from training scores, then study Feature Engineering: Where Most of the Gain Comes From because better inputs often improve validation performance more than a fancier algorithm.