Question 370
During a controlled generative AI workflow test, engineers observe: An AI coding team repeatedly edits prompts while viewing the same supposedly held-out evaluation cases and then reports an unusually high score on those examples. Which dependency is the principal cause of this outcome?
Explanation
Correct Answer
The technical basis is: Repeated exposure of a validation set during prompt optimization can overfit the selected examples and inflate apparent generalization performance. Resolve this through: Maintain separate development, validation and untouched final test sets with controlled access and record prompt changes across iterations. The expected evidence is: The final prompt meets acceptance targets on previously unseen evaluation cases rather than only examples used during development.
Incorrect Answers
An integration that assumes one response shape without schema validation can misinterpret tool results and propagate false success between workflow stages. “Validate tool responses against a versioned schema and explicitly map the supported response shape before recording workflow completion” would remedy that alternative situation, but it would not correct the present issue: repeated exposure of a validation set during prompt optimization can overfit the selected examples and inflate apparent generalization performance.
“Track ticket reopen rates, handoff completion, tool-action correctness and user-confirmed resolution alongside technical model health” is the supported response when successful model calls are not equivalent to successful end-to-end business resolution, so workflow monitoring needs outcome-specific metrics and audits. Here the decisive response is “Maintain separate development, validation and untouched final test sets with controlled access and record prompt changes across iterations”, so that different intervention does not meet the stated requirement.
Documentation generated without reference to the actual versioned API contract can preserve stale or imagined behavior despite readable prose. “Generate or review docs against the current OpenAPI contract and tested implementation and require examples to run as executable checks” applies to that other condition. The reported evidence is an AI coding team repeatedly edits prompts while viewing the same supposedly held-out evaluation cases and then reports an unusually high score on those examples; this alternative would address the wrong dependency.