The full answer
Build it as a small set of reusable, configuration-driven patterns (one for batch files and databases, one for streams) that all land data in a raw zone of Azure Data Lake Storage Gen2 before anything else happens.
1. Orchestration driven by configuration
- Azure Data Factory (or Data Factory in Microsoft Fabric, which Microsoft now points new projects to) runs pipelines from a control table listing each source, its type, schedule, load mode (full or incremental) and target path. Adding a source means adding a row, not a pipeline.
- Use a watermark column or change data capture for incremental database loads.
2. Ingestion paths by velocity
- Batch: the Copy activity for databases, files and SaaS APIs, with a self-hosted integration runtime for on-premises sources.
- Streaming: Azure Event Hubs (Kafka-compatible) or IoT Hub. Event Hubs Capture writes the raw stream to the lake as Avro (or Parquet through the no-code editor); Stream Analytics or Spark Structured Streaming handles real-time processing.
3. Storage layout
- ADLS Gen2 with zones such as raw, cleaned and curated (bronze, silver, gold), in folders by source and date.
- Keep raw data as received; convert to Delta Lake or Parquet in later zones.
4. Handling formats and schema change
- Read CSV, JSON, Avro and Parquet with explicit schemas in production; allow schema drift only where you also alert on it.
- Quarantine files or records that don't parse instead of failing the whole run.
5. Quality, monitoring and governance
- Validate row counts, nulls and keys on arrival; send pipeline metrics and failures to Azure Monitor alerts.
- Register datasets in Microsoft Purview for discovery and lineage; secure the lake with Entra ID, RBAC and ACLs, and managed identities for pipelines.
6. Scale
- Parallel copies, partitioned reads of large tables, and auto-scaling Spark for transformation. Retries with backoff and idempotent loads make re-runs safe.
Last updated 16 October 2026