Data Lineage, Cataloguing & Documentation: Interview-Ready Data Governance Framework
A CFO opens a revenue dashboard on Monday morning and sees a number that is suddenly 8% lower than Friday. The real question is not “Who made the chart?” - it is “Which source table changed, which transformation touched it, and which reports are now wrong?” That trail is data lineage; the searchable map around it is the data catalogue; the explanation that makes humans trust it is documentation.
- Data lineage traces where data came from, how it changed, and where it is used.
- Data cataloguing makes data assets searchable with metadata such as owner, definition, quality, tags and usage.
- Documentation explains business meaning, assumptions, rules, caveats and examples so people use data correctly.
- The three work together: lineage gives traceability, catalogue gives discoverability, documentation gives interpretability.
- Good data governance needs both technical metadata - tables, columns, jobs - and business metadata - definitions, owners, policies.
- Track catalog coverage, lineage coverage, documentation freshness, ownership coverage and incident root-cause time.
- The interview trap: describing a catalogue as a “data storage place.” It is not storage; it is a trust and governance layer over data assets.
Big Picture: The Trust Layer Around Data
Think of a company’s data estate as a city. Warehouses, lakes, dashboards and machine-learning models are buildings; lineage is the road map; the catalogue is Google Maps; documentation is the street sign that tells you what each place actually means.
The Core Explanation: Three Jobs, One Goal
The goal is simple: make data usable without tribal knowledge. In weak data cultures, only the analyst who built the pipeline knows which table is reliable. In strong data cultures, a new analyst can search, inspect lineage, read definitions, check ownership and use the right asset confidently.
Lineage without documentation becomes a technical map that business users cannot understand. Documentation without lineage becomes a nice wiki that may be outdated. A catalogue without either becomes a search box full of confusing assets.
The Funnel: From Data Swamp to Trusted Data Product
Most companies do not suffer from “no data.” They suffer from too many tables, too many conflicting definitions and too little confidence. Cataloguing and documentation act like a funnel - they narrow messy data into governed, reusable data products.
What Good Lineage Actually Captures
Lineage is not just “table A feeds table B.” In a serious analytics environment, lineage must work at three levels:
The deeper the lineage, the easier it becomes to do impact analysis. If a data engineer changes a column definition, lineage tells the team which dashboards, metrics, machine-learning features and regulatory reports may be affected.
What a Data Catalogue Must Contain
A useful catalogue combines technical metadata and business metadata. Technical metadata tells systems what exists. Business metadata tells humans whether to trust it.
In an Indian payments gateway context, transaction data may pass from merchant APIs to acquiring-bank responses, settlement files, reconciliation tables, risk checks and finance dashboards. For a company such as Razorpay, lineage and cataloguing matter because payment disputes, settlements and compliance responses depend on knowing which field came from which system and what it means. The primary driver is traceability across a multi-party payments flow, supported by ownership, sensitive-field tagging and consistent business definitions.
Documentation: The Human Layer Most Teams Underestimate
Documentation turns metadata into judgment. A column called active_user is not self-explanatory. Does it mean app opened, purchase completed, logged in within 30 days, or KYC completed? Without documentation, teams create parallel definitions and then argue in meetings.
Metrics: How to Know the Governance System Is Working
Do not say “we will improve documentation” in an interview. Say how you would measure whether users can find, understand and trust data.
Small worked example: if 50 Tier-1 dashboards exist and 38 have complete upstream and downstream lineage, lineage coverage is 38 / 50 = 76%. That is not terrible, but it is below the typical internal target for critical reporting, so the team should prioritize the 12 uncovered dashboards before cataloguing low-use assets.
Definitions You Can Say in One Breath
- Data lineage: The trace of where data came from, how it changed, and where it is used.
- Data catalogue: A searchable inventory of data assets enriched with metadata, ownership, quality and usage context.
- Metadata: Data that describes other data.
- Business glossary: A controlled list of business terms, definitions, owners and approved usage rules.
- Data documentation: Human-readable explanation of data meaning, assumptions, rules, caveats and examples.
Case Study: Lyft and Amundsen
Lyft built and open-sourced Amundsen, a metadata-driven data discovery platform, to help employees find and trust internal data assets faster.
Situation: As Lyft scaled, teams generated large volumes of tables, dashboards and analytical assets. The problem was not just storing data; it was helping analysts, data scientists and business teams discover the right dataset, know who owned it and understand whether it was safe to use.
The move: Lyft created Amundsen, a data discovery and metadata platform. It brought together searchable metadata, table descriptions, ownership, usage signals and lineage-style context so users could discover reliable data assets instead of asking around informally.
Outcome and lesson: Amundsen became widely known in the data community and was open-sourced, influencing how many modern catalogues think about search, popularity, ownership and trust. The primary driver was metadata-powered discoverability, supported by ownership visibility, usage signals and integration with existing data systems. The lesson for interviews: a catalogue wins when it changes user behavior, not when it merely stores descriptions.
How AI Changes Data Lineage, Cataloguing & Documentation
1. AI can auto-draft documentation from code and queries. Modern tools can read SQL transformations, dashboard logic and column names to draft descriptions, metric definitions and caveats. The risk is hallucinated meaning, so human data owners must approve business definitions.
2. AI improves active metadata and anomaly investigation. Instead of a static catalogue, AI agents can flag stale documentation, identify likely owners, summarize downstream impact and explain why a dashboard changed after a pipeline update.
3. AI makes natural-language data discovery practical. A business user can ask, “Which certified dataset should I use for monthly net revenue in India?” The system can search the catalogue, glossary, lineage and quality signals before recommending an asset.
Use NotebookLM or ChatGPT as a revision assistant: upload a company annual report, a sample data dictionary and this topic note, then ask, “Identify five critical business metrics, likely source systems, sensitive fields, lineage risks and documentation questions an interviewer may ask.”
Interview Relevance
“Suppose a company has multiple dashboards showing different revenue numbers. How would you use data lineage, cataloguing and documentation to solve the problem?”
Use the phrase “single source of truth, with visible lineage and owned definitions”. It sounds managerial, but it also shows you understand the technical control needed underneath.
Common Mistake
The biggest mistake is saying, “A data catalogue is where data is stored.” That costs candidates because it confuses a metadata layer with a database or warehouse. Fix: say, “The warehouse stores data; the catalogue describes, governs and helps users discover trusted data assets.”
What to Revise Next
Once you understand how data is traced and documented, revise the two governance topics that usually appear next: how to keep core entities consistent, and how to protect sensitive fields.