A partner drops 400 store files into ADLS Gen2 every night, and a storage event trigger in Azure Data Factory starts the load. Some runs read files that are only half there, the warehouse gets 400 separate small loads, and after the last release the trigger silently stopped working in production. How do you fix the design?

Trigger on a marker file written after all 400, not on each data file: filter the trigger with blobPathBeginsWith on the drop folder and blobPathEndsWith on the marker name, so one run per night processes the whole folder that the marker names. Pass the folder and file name from @triggerBody() into pipeline parameters. The silent failure is a permissions problem: the account that deploys the trigger needs rights to create the Event Grid subscription on the storage account.

1. Why runs read half-written files and why there are 400 runs

A storage event trigger starts the pipeline once for every matching BlobCreated event. 400 files means 400 runs, all at about the same time, each loading one file. And the pipeline cannot tell whether the partner has finished writing the rest of the batch.

2. Trigger on the end of the batch

Agree with the partner that they upload a small marker last, for example _manifest.json listing the files and row counts:

"typeProperties": {
  "scope": "/subscriptions/<sub>/resourceGroups/rg-data/providers/Microsoft.Storage/storageAccounts/lake01",
  "events": ["Microsoft.Storage.BlobCreated"],
  "blobPathBeginsWith": "/landing/blobs/partner-x/",
  "blobPathEndsWith": "_manifest.json",
  "ignoreEmptyBlobs": true
}

Map @triggerBody().folderPath to a pipeline parameter such as batchFolder. The pipeline then:

1. Reads the manifest and checks that every listed file is present, with the expected size or row count.
2. Copies the whole folder in one copy activity (a wildcard path on the folder), or loops over the manifest with ForEach if each file needs its own handling.
3. Moves the folder to processed/ or records the batch as loaded, so a rerun does not load it twice.

Only beginsWith and endsWith filtering exists, so a naming convention is the only way to target the marker. Keep the filters narrow: a trigger on a whole container fires for every file written there, including your own outputs, and each run costs money.

3. If the partner cannot write a marker

Use a schedule trigger after the agreed delivery time that checks the folder (Get Metadata for the file list) and loads what is complete, or a tumbling window trigger per day. Event triggers suit files that are complete when they land; batches without a completion signal are safer on a schedule.

4. Why it stopped after the release

• The trigger is an Event Grid subscription on the storage account. Creating it needs Owner, Contributor or Microsoft.EventGrid/EventSubscriptions/Write on the storage account for the account doing it, which in production is the CI/CD service connection, not the developer. Without it, the deployment cannot subscribe the trigger.
• The Event Grid resource provider must be registered in the production subscription.
• Triggers must be started after deployment; the post-deployment script does it.
• The factory's own identity needs no Event Grid rights, but the pipeline's linked service still needs read access to the files.

Check it after every release: the trigger's status in the Monitor hub, and a test marker in a test folder.

5. Other traps

• Do not trigger on files written by a mapping data flow sink: data flows rename and shuffle files while writing.
• A storage account allows at most 500 storage event triggers; one trigger per partner folder with parameters scales better than one per store.

What interviewers want: knowing one event means one run, a completion marker or manifest instead of per-file triggers, path filters and trigger metadata in parameters, idempotent batch loading, the permissions the deploying identity needs, and when a schedule is the better trigger.