What are the corrupt record handling modes in Spark when reading data?

When reading data in Spark, especially from sources like JSON, CSV, or other structured formats, corrupt or malformed records can arise due to data inconsistencies or invalid formats. Spark provides various modes to handle these records, allowing you to manage such issues flexibly.

Handling Modes for Corrupt Records:

1. PERMISSIVE (Default):

   • Behavior: Corrupt records are retained in a special column called _corrupt_record (if specified in the schema). The rest of the data is read normally.
   • Use Case: Ideal when you want to identify and investigate malformed records without losing valid data.
   • Example:

     spark.read.format("json")          .option("mode", "PERMISSIVE")          .option("columnNameOfCorruptRecord", "_corrupt_record")          .load("path/to/data.json")
     

2. DROPMALFORMED:

   • Behavior: Corrupt records are ignored, and only valid records are included in the resulting DataFrame.
   • Use Case: Suitable when malformed records are not critical, and you want to focus only on valid data.
   • Example:

     spark.read.format("csv")          .option("mode", "DROPMALFORMED")          .option("header", "true")          .load("path/to/data.csv")
     

3. FAILFAST:

   • Behavior: The job fails immediately if corrupt records are encountered.
   • Use Case: Useful for strict data quality requirements where malformed records must be addressed upfront.
   • Example:

     spark.read.format("json")          .option("mode", "FAILFAST")          .load("path/to/data.json")
     

Custom Handling of Corrupt Records:

• You can define custom logic to filter or process corrupt records using transformations like filter or withColumn.
• Example: Filtering out null or corrupt entries.

  df.filter(df["_corrupt_record"].isNull()).show()
  

Comparison of Modes:

Nuances to Consider:

• Performance Impact: Handling corrupt records (e.g., with PERMISSIVE) may increase processing time due to additional parsing.
• Data Quality: Choose the appropriate mode based on your tolerance for malformed records and the need for investigation.
• Custom Schema: Ensure the schema aligns with the data format to minimize corrupt records.

Tecyfy Takeaway:
Spark provides flexible modes like PERMISSIVE, DROPMALFORMED, and FAILFAST to handle corrupt records when reading data. Selecting the right mode depends on your use case, balancing data quality requirements and error tolerance.