Question 260
While evaluating AI productivity and governance, staff identify: A team promotes a new few-shot template after three impressive demonstration answers, but a larger test shows regressions on rare negative examples and formatted outputs. Which model or workflow behavior most directly accounts for this result?
Explanation
Correct Answer
The technical basis is: Anecdotal prompt successes do not establish repeatable quality across the expected input distribution or difficult edge conditions. Resolve this through: Version the prompt and evaluate it against representative held-out, adversarial and no-answer fixtures with explicit accuracy and format metrics. The expected evidence is: The new template meets agreed benchmark thresholds across the full test set and documented edge cases before deployment approval.
Incorrect Answers
Embedding credentials in prompts exposes them to inference processing, retention and logging paths independently of whether the model ever repeats them in its answer. “Revoke the exposed token, substitute synthetic placeholders and keep actual secrets in authorized runtime services rather than prompt examples” would remedy that alternative situation, but it would not correct the present issue: anecdotal prompt successes do not establish repeatable quality across the expected input distribution or difficult edge conditions.
“Resolve or rank the conflicting constraints in a maintained prompt policy and permit a concise conflict notice when the requirements cannot coexist” is the supported response when a prompt with mutually incompatible constraints cannot be made reliable through repetition alone; the task needs a defined priority or an explicit clarification rule. Here the decisive response is “Version the prompt and evaluate it against representative held-out, adversarial and no-answer fixtures with explicit accuracy and format metrics”, so that different intervention does not meet the stated requirement.
Retrieved webpages and documents are untrusted task data; instructions inside them do not have the authority of the system or application developer. “Keep document contents delimited as evidence, ignore instruction-like text from retrieval, and require validated tool and policy boundaries for the assistant” applies to that other condition. The reported evidence is a team promotes a new few-shot template after three impressive demonstration answers, but a larger test shows regressions on rare negative examples and formatted outputs; this alternative would address the wrong dependency.