Question 250
A data engineering pipeline invokes multiple reusable notebooks with parameters. The current environment exhibits the following problem: a failed notebook's error message is not visible to pipeline operators. Which step preserves the intended data behavior?
Explanation
Correct Answer
Capture notebook run details and propagate actionable failure diagnostics. Operational troubleshooting needs the actual Spark error, not only a red activity icon.
Incorrect Answers
A notebook receives a string ID but compares it to an integer partition column. Parameter type mismatches can silently filter all rows.
A pipeline invokes the same notebook for two tenants without isolating output paths. Shared output paths can leak or overwrite cross-tenant data.
A notebook performs expensive initialization for every small task. Startup cost may dominate short Spark workloads.