The Modern Data Stack: Explain Ingestion, Storage, Transformation and Serving in Interviews
A payment succeeds on UPI in two seconds, but the merchant still asks: “Has the money settled, which offer worked, and why did this transaction fail for others?” The answer is not one dashboard or one database - it is a chain of systems that captures events, stores them, cleans them, defines metrics and serves them to humans and machines.
- The modern data stack is the architecture that moves raw business data from source systems to trusted analytics, ML and operational decisions.
- Its four core layers are ingestion, storage, transformation and serving, supported by governance, testing and observability.
- ETL transforms before loading; ELT loads raw data first and transforms inside the warehouse or lakehouse.
- Batch is enough for daily MIS and finance reporting; streaming is needed for fraud alerts, live ETAs and real-time personalization.
- A strong stack is judged by freshness, reliability, query speed, cost efficiency, test pass rate and metric consistency - not by how many tools it names.
- The best interview answer always links the architecture to a business decision: “What decision becomes faster, cheaper or more accurate?”
Big Picture
Think of the modern data stack as a factory line for decision-grade data. Raw events enter from apps, payment systems, CRMs and operations tools; the stack turns them into governed metrics, dashboards, machine-learning features and business actions.
Core Explanation: The Four Layers That Make Data Useful
The big idea is simple: business data is not born decision-ready. A customer order, a payment event, a call-centre ticket or an app click is useful only after it is captured, stored, cleaned, joined with other data and exposed in a form that a manager, model or application can use.
1. Ingestion - bringing data into the stack
Data ingestion is the process of collecting data from source systems and moving it into a storage or processing environment. Sources include transactional databases, SaaS tools, APIs, mobile apps, IoT devices, logs and partner files.
There are two common modes:
- Batch ingestion: Data is moved at scheduled intervals - hourly, daily or weekly. Good for MIS, finance reporting and campaign analysis.
- Streaming ingestion: Data is moved continuously as events happen. Needed for fraud detection, live delivery tracking, stock-outs and dynamic pricing.
2. Storage - keeping data where it can be queried
Storage is where ingested data lives. The three terms students confuse most are:
- Data warehouse: Structured, analytics-ready storage optimized for queries and reporting.
- Data lake: Low-cost storage for raw structured, semi-structured and unstructured data.
- Lakehouse: A hybrid approach that brings warehouse-like governance and performance to lake-style storage.
For an MBA answer, do not get stuck in vendor names. Say the trade-off: warehouses are strong for governed BI, lakes are flexible for raw data and ML, lakehouses try to combine both.
3. Transformation - converting raw data into business logic
Transformation means cleaning, joining, standardizing and modelling raw data into reusable business entities such as customers, orders, revenue, active users, cohorts and contribution margin.
This is where the stack answers painful business questions: “Should cancelled orders count in GMV?” “Is revenue booked on order date or delivery date?” “Do we define an active customer by login, purchase or payment?”
4. Serving - making data usable
Serving is the final layer where data is consumed through dashboards, alerts, reports, APIs, reverse ETL into CRM tools, ML feature stores and embedded analytics inside products.
The serving layer is where ROI shows up. A clean table is not the outcome; a faster decision, fewer failed payments, better inventory allocation or more accurate campaign targeting is the outcome.
ETL versus ELT - the shift behind the modern stack
Traditional stacks often used ETL: extract data, transform it in a separate processing layer, then load curated data into a warehouse. Modern cloud stacks commonly use ELT: extract and load raw data first, then transform it inside scalable storage and compute.
The hidden layer: governance and observability
A modern stack fails if teams cannot trust it. Governance defines ownership, access, privacy, lineage and business definitions. Observability monitors freshness, failures, schema changes and quality issues before they damage decisions.
How to measure whether the stack is healthy
If a question asks how to evaluate a data stack, do not say “accuracy” vaguely. Use operational and business-facing measures.
Small worked example: diagnosing a weak pipeline
Suppose a company has 30 scheduled daily pipelines in a month. One customer table fails twice and one revenue table fails once, so successful runs are 27 out of 30.
Pipeline success rate = 27 / 30 = 90%. That is weak for production analytics. If the latest order data in the dashboard is from 11:00 pm yesterday and leaders check at 11:00 am today, freshness lag = 12 hours. That may be acceptable for daily MIS, but unacceptable for a live operations dashboard.
Definitions
- Modern data stack: Cloud-native tools and practices that ingest, store, transform, govern and serve data for analytics and operations.
- Data ingestion: The movement of data from source systems into a storage or processing environment.
- Data warehouse: W.H. Inmon: “A data warehouse is a subject-oriented, integrated, time-variant, and nonvolatile collection of data in support of management’s decisions.”
- Transformation: The process of cleaning, joining and modelling raw data into reusable business-ready datasets.
- Semantic layer: A business-facing layer that standardizes metrics, dimensions and definitions across tools and users.
Case Study - Razorpay: Payment Data as a Decision Factory
Razorpay shows why a modern data stack matters in Indian fintech: payments, reconciliation, risk and merchant analytics all depend on trusted event-to-decision pipelines.
Razorpay operates in a market where merchants accept payments through UPI, cards, net banking, wallets and other rails. Each payment creates a chain of events: initiation, authentication, success or failure, refund, settlement, dispute and reconciliation. For a merchant, the business question is not “Where is the data stored?” It is “Which payments succeeded, when will I receive funds, which method is failing, and what should I change?”
The strategic move is to make payment data usable at multiple speeds. Some decisions need real-time or near-real-time data, such as fraud flags, payment failure monitoring and gateway routing. Other decisions can be batch-based, such as monthly merchant reports, settlement reconciliation and cohort-level product analysis.
The primary driver is trustworthy transaction intelligence: merchants need confidence that payment and settlement data is accurate. Supporting drivers include reliable ingestion from multiple payment rails, clear metric definitions, observability for failures and serving layers that make the same truth available to merchants, finance, risk and product teams.
The lesson: a modern data stack is not a technology vanity project. In Indian fintech, it directly supports merchant trust, regulatory discipline, operational speed and product improvement.
How AI Changes the Modern Data Stack
AI does not remove the need for a clean data stack; it raises the penalty for a bad one. If the underlying data is stale, biased or poorly defined, AI simply makes wrong answers faster.
- Natural-language BI and GenBI: Business users increasingly ask questions in plain English - “Why did refunds rise in Bengaluru last week?” The stack now needs a stronger semantic layer so the AI uses the correct definition of refund rate, city and time period.
- AI-assisted data engineering: Tools can draft SQL models, generate documentation, suggest tests and detect schema anomalies. This speeds up analytics engineering, but humans must still validate logic and business definitions.
- AI-ready storage and retrieval: Modern stacks now support unstructured data, embeddings and vector search for use cases like support-ticket analysis, document search and customer-intent clustering.
Use NotebookLM or ChatGPT like an interview simulator: upload a company annual report, product pages and this lesson, then ask, “Map this company’s likely data sources, ingestion needs, storage choices, transformations, serving use cases and data governance risks.” Your goal is not tool memorization; it is business architecture thinking.
Interview Relevance
“Suppose a quick-commerce company wants to build a real-time dashboard for stock-outs, delivery delays and customer complaints. How would you design the modern data stack at a high level?”
In interviews, say one sentence on trade-off: “I would not stream everything; I would stream high-urgency operational events and batch low-urgency reporting workloads to control cost and complexity.” That one line signals maturity.
Common Mistake
The biggest mistake is giving a tool-list answer: “Kafka, Snowflake, dbt, Airflow, Looker” without explaining what business decision each layer supports. It costs candidates because it sounds memorized, not managerial. The fix: narrate the flow from source event to decision, then justify tools only if asked.
What to Revise Next
Once the stack architecture is clear, revise the two topics that make the stack trustworthy and business-friendly: how models are transformed and tested, and how metrics get one agreed definition.