Question 480
A notebook uses Spark Structured Streaming to load events into Delta. The current environment exhibits the following problem: a streaming sink appends duplicates after failures and retries. Which technical choice fits this evidence?
Explanation
Correct Answer
Use idempotent batch writes or supported exactly-once sink semantics. Checkpoints alone don't guarantee business-key uniqueness in every output.
Incorrect Answers
A streaming query reprocesses all input after restart because progress was not stored. Checkpoint metadata supports restart and offset recovery.
Records arrive late and get dropped because the watermark is too short. Short watermarks prematurely discard valid late events.
Two independent streaming queries share one checkpoint directory. Shared checkpoints can corrupt or conflate progress state.