How does Spark achieve fault tolerance?

Spark recovers from failures mainly by recomputing lost data from its lineage, the recorded sequence of transformations that produced it, and by retrying failed tasks, rather than by copying all data as it goes.

1. Lineage and recomputation Every DataFrame or RDD knows the steps that built it from its source. If an executor dies and its partitions are lost, the driver reschedules the tasks that produce those partitions, re-reading the input and re-running the transformations for those partitions only.

2. Task retries A failed task is retried on another executor; by default a job fails only after the same task fails 4 times (spark.task.maxFailures). Optional speculative execution starts duplicate copies of very slow tasks.

3. Shuffle files Map-stage outputs are written to local disk, so a failure in a later stage doesn't redo earlier stages if their shuffle files survive. If they're lost with an executor, Spark reruns the stage that produced them. An external shuffle service or shuffle tracking lets executors be removed without losing them.

4. Checkpointing and replication For very long lineages (iterative algorithms, long streaming jobs), checkpoint() saves data to reliable storage and cuts the lineage, so recovery doesn't replay everything. Storage levels ending in _2 keep cached partitions on two executors.

5. Reliable storage underneath Input and output live in replicated storage such as HDFS or cloud object storage, and table formats such as Delta Lake make writes atomic, so a failed job doesn't leave half-written tables.

6. Streaming Structured Streaming records progress and state in a checkpoint location; with a replayable source and an idempotent or transactional sink, a restarted query continues with exactly-once results.

What isn't covered If the driver fails, the application fails, unless the cluster manager restarts it (cluster deploy mode with supervision) or, for streaming, a new run resumes from the checkpoint.