Question 10
A business team evaluating model-assisted work finds: A team promotes a new few-shot template after three impressive demonstration answers, but a larger test shows regressions on rare negative examples and formatted outputs. What assurance check should be required for future changes?
Explanation
Correct Answer
This safeguard addresses prompt change evaluated only on cherry picked successes: Treat prompt versions as tested software artifacts and track quality changes rather than selecting based on isolated polished examples. The prevented failure mechanism is: Anecdotal prompt successes do not establish repeatable quality across the expected input distribution or difficult edge conditions. A control for an unrelated service cannot establish the required protection.
Incorrect Answers
A prompt with mutually incompatible constraints cannot be made reliable through repetition alone; the task needs a defined priority or an explicit clarification rule. “Review prompt templates for contradictory length, citation and formatting demands before deploying revised instructions” would remedy that alternative situation, but it would not correct the present issue: anecdotal prompt successes do not establish repeatable quality across the expected input distribution or difficult edge conditions.
“Track schema validation failures and use negative format fixtures after modifying prompt instructions or downstream parsing logic” is the supported response when unconstrained natural-language instructions do not reliably impose the exact keys, data types and machine-readable format expected by a downstream parser. Here the decisive response is “Treat prompt versions as tested software artifacts and track quality changes rather than selecting based on isolated polished examples”, so that different intervention does not meet the stated requirement.
Embedding credentials in prompts exposes them to inference processing, retention and logging paths independently of whether the model ever repeats them in its answer. “Run prompt and trace secret scanning whenever publishing shared prompt templates or examples for model-assisted tools” applies to that other condition. The reported evidence is a team promotes a new few-shot template after three impressive demonstration answers, but a larger test shows regressions on rare negative examples and formatted outputs; this alternative would address the wrong dependency.