Question 720
A Spark transformation is slow because of badly distributed shuffle tasks. The current environment exhibits the following problem: Spark stages have too few partitions for available resources. Which approach best satisfies this requirement?
Explanation
Correct Answer
Tune partition count based on data size and executor parallelism. Insufficient parallel tasks underutilize a distributed cluster.
Incorrect Answers
An optimizer continues broadcasting a lookup after its size has grown beyond the expected threshold. Join-strategy defaults need revalidation as reference sizes change.
An expensive intermediate DataFrame is recomputed by several downstream actions. Reusing materialized work can avoid repeated lineage recomputation.
A large join shuffles both tables despite a tiny lookup input. Small reference datasets can avoid expensive shuffle transfers.