Model Evaluation & Selection: How to Choose the Right Model
Clustering & Customer Segmentation Explained focused on grouping similar customers and interpreting cluster quality. Model Evaluation & Selection answers the next question: once you have built a model, how do you decide whether it is the right model? In interviews, this matters because strong candidates balance bias-variance first, then choose evaluation metrics that match the problem type and the business cost of errors.
- High Bias (Underfitting) means the model is too simple and misses patterns; the fix is a more complex model, more features, or less regularisation.
- High Variance (Overfitting) means the model memorises training data and fails on new data; the fix is more data, regularisation, dropout, pruning, or a simpler model.
- The Sweet Spot has balanced bias and variance, high training accuracy, high test accuracy, and generalises well.
- For regression, use MAE, RMSE, MAPE, or R² depending on whether you need average error, large-error penalty, scale-independent error, or variance explained.
- For classification, Accuracy is useful only with balanced classes, while Precision, Recall, F1 Score, AUC-ROC, and Log Loss handle different business needs.
- Before choosing a model evaluation metric, always ask: What is the business cost of a false positive vs a false negative?
Big Picture: Balance Bias-Variance Before Choosing Metrics
Model selection starts with the bias-variance trade-off. Bias is error from wrong assumptions in the model, while variance is error from model sensitivity to small fluctuations in training data.
The practical goal is to avoid both underfitting and overfitting, reach the sweet spot, and then use metrics that match whether the model is solving a regression, classification, or clustering problem.
What Bias and Variance Mean
Bias is error from wrong assumptions in the model, also called underfitting. High bias means the model is too simple, such as a linear model on non-linear data.
Variance is error from model sensitivity to small fluctuations in training data, also called overfitting. High variance means the model is too complex, such as a decision tree with depth 50.
The trade-off is that reducing bias increases variance, while reducing variance increases bias. The optimal model balances both, with minimum total error.
How to Read Training Accuracy and Test Accuracy
Training accuracy and test accuracy reveal the model selection problem quickly. If both training accuracy and test accuracy are low, the symptom is high bias: the model is too simple and misses patterns.
If training accuracy is very high but test accuracy is low, the symptom is high variance: the model memorises training data and fails on new data. If both are high, the model generalises well, and cross-validation helps confirm that it is the goal.
Choosing Model Evaluation Metrics
Before choosing a model evaluation metric, always ask: What is the business cost of a false positive vs a false negative? This single question separates good analysts from great ones.
Regression metrics evaluate numerical prediction errors. Classification metrics evaluate predicted classes, and the right choice depends on whether false positives or false negatives are more costly.
Regression Metrics
MAE is used for average absolute error. It is the mean of |y_actual - y_pred|, easy to interpret, and robust to outliers.
RMSE is used when large errors should be penalised. It is √(Mean of (y_actual - y_pred)²) and is in the same units as y.
MAPE is scale-independent. It is the mean of |actual-pred|/actual × 100% and is interpretable as percentage error.
R² is the proportion of variance explained. It is 1 - SS_residual/SS_total, where 1.0 is perfect and 0 is no better than mean.
Classification Metrics
Accuracy is the overall correct rate, calculated as (TP+TN) / Total. It should be used only with balanced classes because it can be misleading with imbalanced classes.
Precision matters when false positives are costly. It is TP / (TP+FP), important for fraud flags where you do not want to disturb legit users.
Recall matters when false negatives are costly. It is TP / (TP+FN), critical for disease/fraud detection where you do not want to miss cases.
F1 Score is used for imbalanced classes. It is 2 × Precision × Recall / (P+R), balanced when both matter.
AUC-ROC measures ranking quality and is threshold-independent. Area under ROC curve of 0.5 means random, while greater than 0.9 means excellent.
Log Loss is for probabilistic classifiers. It is -Mean of [y log(p) + (1-y) log(1-p)] and penalises confident wrong predictions.
Clustering Metric
Silhouette Score is a clustering metric, not a regression or classification metric. It ranges from -1 to 1 and measures cluster cohesion and separation.
Practical Model Selection Logic
A strong model selection answer should first diagnose whether the model is underfitting, overfitting, or generalising well. Then it should select the metric based on the problem type and the cost of errors.
For high bias, the fix is more complex model, more features, less regularisation. For high variance, the fix is more data, regularisation, dropout, pruning, simpler model.
Structuring a Model Evaluation & Selection Interview Answer
"How would you evaluate this model?"
The #1 way candidates get this wrong is by naming a metric immediately. Always ask what the business cost of a false positive vs false negative is, because that shows business acumen, not just technical knowledge.
The most frequent error is using Accuracy for classification without checking whether classes are balanced. Accuracy can be misleading with imbalanced classes, so candidates lose points when they ignore Precision, Recall, F1 Score, and the business cost of false positives vs false negatives.
Conclusion
Model Evaluation & Selection is about balancing bias-variance first, then choosing metrics that match the problem type and the business cost of errors. The strongest interview takeaway is simple: diagnose underfitting or overfitting, then justify the metric with false positive and false negative costs.