Design a pipeline for website and app click events, about 50,000 events per second at peak, that feeds a dashboard updated every minute and keeps all raw events for analysis.

Put the events into Kafka, run a streaming job that writes every raw event to a bronze Delta table, and a second streaming job that computes per-minute aggregates with a watermark for late events and writes them to a small gold table the dashboard reads. Size Kafka partitions for the peak, keep state bounded with watermarks, and rely on checkpoints plus Delta for exactly-once results.

1. Clarify

• Latency: "every minute" for the dashboard; raw data can be minutes behind.
• Correctness: are counts allowed to change when late events arrive? How late can events be (mobile apps offline)?
• Volume: 50,000 events per second at peak at about 1 KB is roughly 50 MB/s, about 4 TB a day before compression.
• Retention and privacy: how long raw events are kept, and which fields are personal.

2. The design

Apps and web ──▶ collector API ──▶ Kafka topic "clicks" (keyed by user_id)
                                      │
               ┌──────────────────────┴─────────────────────┐
     Stream 1: raw events                      Stream 2: per-minute metrics
     ──▶ bronze.clicks (Delta, append)         watermark 15 min, 1-minute windows
                                               ──▶ gold.clicks_by_minute ──▶ dashboard

3. Key decisions

• Kafka partitions set the parallelism: Spark reads one partition per task by default. Pick enough for peak throughput with headroom (for example 48 to 96), because adding partitions later changes which partition a key goes to.
• Key by user ID so each user's events stay in order within a partition, which matters for sessions and funnels.
• Two streams, not one. The raw stream is simple and must never fall behind; the aggregate stream holds state. Separate checkpoints mean you can change or restart one without touching the other.
• Event time and watermark. Group by the time the event happened, not when it arrived. A 15-minute watermark lets late events update recent windows and lets Spark drop older state, so memory stays bounded. Events later than that are not counted in the live view; the bronze table still has them.
• Exactly-once. Kafka offsets in the checkpoint plus Delta's transactional sink mean a restart neither loses nor duplicates results.
• Output mode. Update mode lets the dashboard see a minute's count as it grows; append mode writes each window once, after the watermark passes.

4. Operations

• Back-pressure: cap maxOffsetsPerTrigger so a restart after downtime catches up in steady batches.
• Monitor input rate against processing rate, batch duration, and the lag between latest event time and now. Alert before lag grows.
• Small files: compaction or optimized writes on bronze.
• Corrections: the dashboard is the fast view; a nightly batch job over bronze produces the final daily numbers for finance.
• Retention: Kafka keeps days for replay; bronze keeps the long-term record, with personal fields restricted.

What interviewers want: back-of-envelope sizing, Kafka partitioning and keys, event time with a watermark and what it does to late data, separate raw and aggregate streams, exactly-once with checkpoints and a transactional sink, and a batch layer for final numbers.