The same Spark job gives different results on different runs over the same input. What can cause this?

Something in the job depends on the order of rows or on how data is partitioned, and Spark guarantees neither unless you ask for it. After a shuffle, rows can arrive in a different order on every run. Functions that pick "the first" row, IDs built from partition numbers, a dedup without a rule for which row wins, and the session time zone are the usual causes. The fix is to make every choice explicit.

The usual causes

1. Picking a row without saying which one

df.groupBy("customer_id").agg(F.first("email"))     # which email?
df.dropDuplicates(["customer_id"])                   # which row survives?

Neither says which row it keeps. first() takes whichever row comes first in that run, and it is non-deterministic because row order can change after a shuffle. dropDuplicates documents no rule for which duplicate survives, so do not rely on one. Say which row you want:

w = Window.partitionBy("customer_id").orderBy(F.desc("updated_at"), F.desc("event_id"))
latest = df.withColumn("rn", F.row_number().over(w)).filter("rn = 1").drop("rn")

Add a unique column as the last tie-breaker. If two rows share the same updated_at, ordering by that alone is still a coin toss.

2. IDs from monotonically_increasing_id()

Its values depend on the partition ID, so the same row gets a different ID when the partitioning differs between runs. Never use it as a key that other tables store. Build keys from the data itself, for example a hash of the business key columns:

df.withColumn("row_key", F.sha2(F.concat_ws("||", "source", "order_id"), 256))

3. Ranking with ties

row_number() over an ordering with ties gives tied rows numbers in an arbitrary order. Use rank() or dense_rank() if ties should share a number, or add a tie-breaker.

4. limit() without orderBy()

Returns some rows, not particular rows. In a pipeline, it is almost always a bug.

5. Time zones

Converting timestamps to dates uses the session time zone, which by default is the machine's local time zone. Two clusters in different regions can put the same event on different days. Set spark.sql.session.timeZone explicitly, normally to UTC, and use full region IDs rather than short names.

6. Inputs that change underneath

"The same input" is not always the same: a source folder that gained a late file, or a table read by name that was updated between runs. On Delta Lake, read a fixed version for a reproducible run (versionAsOf), and record which version each run read.

7. Floating-point sums

Adding the same doubles in a different order can change the last digits. For money, use DecimalType. For other measures, compare with a tolerance.

How to test for it

Run the job twice on the same snapshot with a different number of shuffle partitions, and diff the outputs. Anything order-dependent tends to show up.

What interviewers want: that Spark does not preserve row order across shuffles, the concrete traps (first(), dropDuplicates, monotonically_increasing_id, ties, time zones), and making every choice explicit with an ordering and a tie-breaker.