RDD Fundamentals

Not this time. RDD stands for Resilient Distributed Dataset, a fundamental Spark data structure designed for parallel processing across multiple nodes.

Not this time. RDDs are immutable, distributed, and fault-tolerant, ensuring scalability and automatic recovery from failures.

Not this time. RDDs can be created from Python lists using sparkContext.parallelize(), which distributes the data across Spark's cluster.

Not this time. The SparkContext acts as the gateway for interacting with Spark and is responsible for creating RDDs.

Not this time. Transformations are lazily evaluated and define a computation pipeline, whereas actions trigger execution and return results.

Not this time. Transformations like map() create a new RDD by applying a function to each element but do not execute immediately.

Not this time. Actions like collect() trigger Spark to compute and return results to the driver, executing all preceding transformations.

Not this time. Transformations in RDDs follow lazy evaluation, meaning computations are only performed when an action is called.

Not this time. The map() transformation applies a function to every element, producing a new RDD with transformed values.

Not this time. The filter() transformation removes elements that do not meet a specified condition, creating a new RDD with matching records.

Not this time. first() returns the first element. take(1) returns a list holding it, top(1) returns a list with the largest element, and RDDs have no head() method.

Not this time. The reduce() action aggregates values in an RDD by applying a specified function, such as summing or multiplying elements.

Not this time. The default MEMORY_ONLY storage level keeps the RDD in memory, ensuring fast access but requiring enough RAM.

Not this time. You can use cache() for temporary in-memory storage or persist() to specify different storage levels.

Not this time. Spark recomputes lost partitions from the original dataset using its lineage, ensuring fault tolerance.

Not this time. The distinct() transformation eliminates duplicate records, creating a new RDD with only unique values.

Not this time. The union() transformation combines two RDDs into one, retaining all elements, including duplicates.

Not this time. You can use the getNumPartitions() method to determine how many partitions an RDD is split into for parallel execution.

Not this time. Repartitioning balances the workload by adjusting the number of partitions, improving performance and parallel processing.

Not this time. RDDs can be converted into DataFrames with toDF() or spark.createDataFrame(). DataFrames are optimised by Catalyst, and newer environments such as Spark Connect support only the DataFrame API.