What are Anti Join and Semi Join in Spark, and how do they work?

In Spark, Anti Join and Semi Join are specialized join operations that filter rows based on their presence or absence in another dataset. Both operations are used to optimize queries by reducing the amount of data processed compared to traditional joins.

Key Differences Between Anti Join and Semi Join:

Anti Join:
An Anti Join retrieves rows from the left dataset that do not have any matching rows in the right dataset. It effectively works as a filtering mechanism for non-matching rows.

Code Example:

from pyspark.sql import SparkSession

# Initialize SparkSession
spark = SparkSession.builder.appName("Anti Join Example").getOrCreate()

# Sample DataFrames
left_df = spark.createDataFrame([("Alice", 30), ("Bob", 35), ("Cathy", 29)], ["name", "age"])
right_df = spark.createDataFrame([("Alice",), ("David",)], ["name"])

# Perform an Anti Join
anti_join_df = left_df.join(right_df, left_df["name"] == right_df["name"], "left_anti")
anti_join_df.show()

# Stop SparkSession
spark.stop()

Expected Output for Anti Join:

+-----+---+
| name|age|
+-----+---+
|  Bob| 35|
|Cathy| 29|
+-----+---+

Semi Join:
A Semi Join retrieves rows from the left dataset that have at least one matching row in the right dataset. Unlike inner joins, it only returns rows from the left dataset without including columns from the right dataset.

Code Example:

from pyspark.sql import SparkSession

# Initialize SparkSession
spark = SparkSession.builder.appName("Semi Join Example").getOrCreate()

# Sample DataFrames
left_df = spark.createDataFrame([("Alice", 30), ("Bob", 35), ("Cathy", 29)], ["name", "age"])
right_df = spark.createDataFrame([("Alice",), ("David",)], ["name"])

# Perform a Semi Join
semi_join_df = left_df.join(right_df, left_df["name"] == right_df["name"], "left_semi")
semi_join_df.show()

# Stop SparkSession
spark.stop()

Expected Output for Semi Join:

+-----+---+
| name|age|
+-----+---+
|Alice| 30|
+-----+---+

Use Cases and Nuances:

• Anti Join Use Cases: Useful for identifying rows that are not present in another dataset, such as finding records missing in a lookup table.
• Semi Join Use Cases: Ideal for filtering rows based on membership in another dataset, like identifying valid records in a reference table.
• Efficiency: Both operations are optimized for filtering rather than retrieving additional columns, reducing data shuffling compared to traditional joins.

Tecyfy Takeaway:
Anti Join retrieves rows from the left dataset without a match in the right dataset, while Semi Join retrieves rows from the left dataset that have a match in the right dataset. Both operations are efficient alternatives to traditional joins when you need to filter datasets based on matching or non-matching conditions.