Data scientists build a churn model whose offline accuracy is excellent, but it performs badly in production. They join each customer's churn label to the customer's current features. Design a feature pipeline that fixes this.

The training data leaks the future. Joining labels to today's features means a customer who churned in March is described by data from after March, including signs of the churn itself (zero logins, a cancelled plan). The model learns from information it will never have at prediction time. The fix is to keep features as a history with timestamps and build training sets with point-in-time joins: for each label, take the latest feature values known before the label's time.

1. What goes wrong

logins_30d = 0 was computed after C17 left. In production, at the moment of prediction, the model never sees anything like that, so accuracy collapses.

2. Store features as history

Each feature table has a key and a timestamp:

customer_features(customer_id, as_of_ts, logins_30d, tickets_90d, plan, ...)

A daily (or streaming) job appends a row per customer when values change, computed only from events up to as_of_ts. Never overwrite history.

3. Build training sets with an AS OF join

For each label row (customer_id, label_ts, churned), join the feature row with the same customer and the greatest as_of_ts that is earlier than label_ts. In plain SQL:

SELECT customer_id, label_ts, churned, logins_30d, tickets_90d, plan
FROM (
  SELECT l.customer_id, l.label_ts, l.churned,
         f.logins_30d, f.tickets_90d, f.plan,
         ROW_NUMBER() OVER (PARTITION BY l.customer_id, l.label_ts
                            ORDER BY f.as_of_ts DESC) AS rn
  FROM labels l
  JOIN customer_features f
    ON f.customer_id = l.customer_id
   AND f.as_of_ts < l.label_ts - INTERVAL 1 DAY
) t
WHERE rn = 1;

The - INTERVAL 1 DAY represents the delay between an event and the feature being available in production. Match the real delay, or leakage creeps back in. Databricks' feature store performs this AS OF join for you when the feature table has a timestamp key.

4. Serve the same features online

The model in production reads the latest row per customer from the same feature definitions, from an online store or a fresh table. One definition for training and serving prevents a second kind of skew: the same feature computed differently in two places.

5. Guardrails

• A leakage check: features with suspiciously high importance, or that change right at the label time, get reviewed.
• Backtesting: train on data up to a date, test on the period after, the way the model will actually be used.
• Performance: cluster feature tables by key and timestamp so point-in-time lookups stay fast as history grows.
• Monitoring: compare the distribution of features in production with the training set, and alert on drift.

What interviewers want: naming the problem as label leakage, features stored as timestamped history, point-in-time joins with a realistic availability delay, one feature definition for training and serving, and backtesting as the evaluation.