Skip to the page
Tecyfy

Azure interview question · medium

A data pipeline needs to ingest data from various sources into a data lake in Azure. How can you design a robust and scalable ingestion framework that handles different data formats, volumes, and velocities?

The short answerBuild it as a small set of reusable, configuration-driven patterns (one for batch files and databases, one for streams) that all land data in a raw zone of Azure Data Lake Storage Gen2 before anything else happens.

The full answer

Build it as a small set of reusable, configuration-driven patterns (one for batch files and databases, one for streams) that all land data in a raw zone of Azure Data Lake Storage Gen2 before anything else happens.

1. Orchestration driven by configuration

  • Azure Data Factory (or Data Factory in Microsoft Fabric, which Microsoft now points new projects to) runs pipelines from a control table listing each source, its type, schedule, load mode (full or incremental) and target path. Adding a source means adding a row, not a pipeline.
  • Use a watermark column or change data capture for incremental database loads.

2. Ingestion paths by velocity

  • Batch: the Copy activity for databases, files and SaaS APIs, with a self-hosted integration runtime for on-premises sources.
  • Streaming: Azure Event Hubs (Kafka-compatible) or IoT Hub. Event Hubs Capture writes the raw stream to the lake as Avro (or Parquet through the no-code editor); Stream Analytics or Spark Structured Streaming handles real-time processing.

3. Storage layout

  • ADLS Gen2 with zones such as raw, cleaned and curated (bronze, silver, gold), in folders by source and date.
  • Keep raw data as received; convert to Delta Lake or Parquet in later zones.

4. Handling formats and schema change

  • Read CSV, JSON, Avro and Parquet with explicit schemas in production; allow schema drift only where you also alert on it.
  • Quarantine files or records that don't parse instead of failing the whole run.

5. Quality, monitoring and governance

  • Validate row counts, nulls and keys on arrival; send pipeline metrics and failures to Azure Monitor alerts.
  • Register datasets in Microsoft Purview for discovery and lineage; secure the lake with Entra ID, RBAC and ACLs, and managed identities for pipelines.

6. Scale

  • Parallel copies, partitioned reads of large tables, and auto-scaling Spark for transformation. Retries with backoff and idempotent loads make re-runs safe.

Last updated 16 October 2026

Asked next