Model Auditing for Interviews: Error Checks, Tracing and Version Control
In 2012, JPMorgan's “London Whale” loss was not only a bad trading story - it was also a model-control story, where a changed risk model and spreadsheet weaknesses helped hide how much risk was building. The brutal lesson: a model can look mathematically impressive and still fail because nobody can prove what changed, who changed it, and whether the output still makes sense.
- Model auditing is the independent, documented review of a model's logic, data, outputs, controls and change history.
- The three pillars are error checks, tracing and version control.
- Error checks ask: “Is the model calculating the right thing, within tolerance, under normal and stress cases?”
- Tracing asks: “Can I follow every important output back to its input, formula, assumption and approval?”
- Version control asks: “Which model version produced this result, what changed, who approved it, and can we roll back?”
- Strong audit KPIs include zero unresolved critical defects, 100% traceability for material changes, reconciled outputs and tested rollback.
- The biggest interview mistake is treating model audit as a final Excel clean-up instead of a governance process across the model life cycle.
The Big Picture
Think of model auditing as a control bridge between business decision and model output. You are not just checking formulas - you are proving that the model is valid, explainable, repeatable and controlled.
Core Explanation
The easiest way to remember model auditing is the ECT frame: Error Checks, Tracing and Version Control. Use it for Excel valuation models, credit scorecards, pricing models, forecasting models and AI-enabled finance tools.
1. Error Checks - Prove the Model Is Not Lying
Error checks are tests that compare the model's data, formulas and outputs against expected behaviour. They are not only Excel checks like `#DIV/0!`; they also include business logic checks.
2. Tracing - Make Every Output Explainable
Tracing is the ability to follow a number from final output back to the exact input, transformation, assumption and approval that created it. This matters because senior leaders rarely ask, “Is the formula elegant?” They ask, “Where did this number come from?”
In Excel, tracing may use formula precedents, dependency maps and input-output sheets. In analytics or AI models, it uses data lineage, feature logs, model cards, validation notebooks and audit trails.
3. Version Control - Stop Silent Model Mutation
Version control is the disciplined management of model changes across time. It answers four audit questions: which version produced the result, what changed, who approved it, and can we reproduce it?
The Audit Checklist You Can Apply to Any Model
For an RBI-regulated lender or NBFC such as Bajaj Finance, a credit decision model must be auditable because lending decisions affect customers, provisioning and regulatory scrutiny. A good audit would not stop at the credit score; it would trace bureau data, income assumptions, policy rules, manual overrides, approval logs and the exact scorecard version used. So what: model audit protects the firm from bad credit decisions and from being unable to explain those decisions later.
Audit Metrics: What to Track
Model-audit KPIs are control metrics, so “strong” often means zero critical exceptions before release rather than beating an industry average.
Worked Example - A Simple Output Variance Check
Suppose a valuation model shows revenue of ₹500 crore and EBITDA of ₹75 crore. EBITDA margin should be:
EBITDA margin = EBITDA ÷ Revenue = 75 ÷ 500 = 15%
If the dashboard shows 17%, the variance is 2 percentage points. If the audit materiality threshold is 0.5 percentage points, this fails the check. The auditor then traces the dashboard cell back to its source formula and may find an old EBITDA number, a broken link or a hardcoded override.
Definitions
- Model - Federal Reserve SR 11-7: “A quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates.”
- Model risk - Federal Reserve SR 11-7: “The potential for adverse consequences from decisions based on incorrect or misused model outputs and reports.”
- Model audit: An independent, documented review of model logic, data, outputs, controls, assumptions and change history.
- Version control: A system for tracking model changes, approvals, releases and rollback points across time.
Knight Capital: When Version Control Failed in Public
Knight Capital's 2012 trading loss is a classic reminder that uncontrolled code deployment can destroy a business faster than a bad forecast.
Situation: Knight Capital was a major US market-maker. In August 2012, it deployed new trading software, but old dormant code remained active on some servers.
The move that failed: The firm did not fully synchronize deployment across servers, and the legacy code was unintentionally triggered. The trading system sent a flood of unintended orders into the market. Public reports widely cite a loss of about $440 million in roughly 45 minutes.
Primary driver of failure: uncontrolled deployment and version inconsistency across production servers.
Supporting drivers: weak release checks, inadequate kill-switch controls, insufficient real-time monitoring, and poor rollback readiness. This is important: Knight did not fail only because of “one bad code line.” It failed because the surrounding control system was too weak to catch, stop and reverse the error quickly.
Outcome and lesson: Knight's financial hit severely weakened the company and it was later acquired by Getco. The model-audit takeaway is simple: in high-speed finance, version control is not admin work - it is risk control.
How AI Changes Model Auditing
AI makes model auditing both more powerful and more dangerous. The issue is no longer only “Is this Excel formula correct?” It is also “Which prompt, dataset, feature set, model version and human override produced this output?”
Practical student workflow: Use NotebookLM for audit preparation. Upload your model documentation, assumption sheet and a sanitized output summary - never confidential client data - and ask: “Create an audit checklist covering error checks, tracing, version control, reconciliation and likely reviewer questions.” Then use ChatGPT or Claude to generate stress-test scenarios and edge cases, but verify every formula yourself.
Interview Relevance
“You are reviewing a financial model before it goes to the CFO. What checks will you perform to make sure the model is reliable and auditable?”
Use the phrase “reproducible and explainable”. It signals that you understand model audit as a governance discipline, not just spreadsheet checking.
Common Mistake
The costly mistake is saying, “I will check the formulas and the numbers.” That answer is too narrow because real model failures often come from bad assumptions, untraceable data, uncontrolled version changes or weak approvals. One-line fix: answer with the ECT frame - error checks, tracing and version control - then add governance evidence.
What to Revise Next
Now move from auditing models to using AI responsibly in finance. Revise Why AI Literacy Matters in Finance and Where It Must Not Be Used first, then The Finance AI Workflow: NotebookLM, Chat Assistants & Prompts That Work. Together, they will help you explain not only how to use AI, but also how to control it.