Question 760
A Spark notebook suffers repeated executor and network overhead. The current environment exhibits the following problem: a single hot join key causes straggling tasks despite enough executors. Which approach best satisfies this requirement?
Explanation
Correct Answer
Detect and mitigate skew with suitable partition/aggregation tactics. Choose to detect and mitigate skew with suitable partition/aggregation tactics; uneven tasks limit effective parallel speedup.
Incorrect Answers
Spark collects a massive result into driver Pandas for downstream filtering. Driver materialization bypasses cluster parallelism.
A small lookup table triggers an expensive shuffle join. Broadcasting small tables can avoid a full large-side shuffle.
A huge dataset is repartitioned unnecessarily before every step. Exchange stages are costly when they provide no useful distribution.