Question 570
The business case is: A chatbot scores high on five demo questions but fails on real customer requests. Which TWO options together provide a suitable intervention and an independent acceptance check? (Select TWO.)
Explanation
Correct Answers
The intervention for the stated situation A chatbot scores high on five demo questions but fails on real customer requests is: Measure business-relevant correctness, retrieval quality, latency, cost and safety on representative held-out cases rather than only demonstrations. It directly mitigates: Optimizing one easy metric can hide incorrect factual claims and failures on rare but important scenarios.
A separate outcome check for the situation A chatbot scores high on five demo questions but fails on real customer requests is: A benchmark reports error types and meaningful KPI changes on normal, edge and adversarial cases with a baseline. It measures the effect of the selected intervention rather than relying on the existence of a proposal.
Incorrect Answers
This choice addresses bias, fairness and representativeness; its main associated risk is: Biased historical records or unequal error rates can lead to harmful decisions even when overall accuracy looks high. However, the situation A chatbot scores high on five demo questions but fails on real customer requests is primarily about evaluation metrics and representative data. The targeted decision is: Measure business-relevant correctness, retrieval quality, latency, cost and safety on representative held-out cases rather than only demonstrations.
This choice addresses human-in-the-loop for high-impact outputs; its main associated risk is: A generative system given unilateral decision authority can approve unsafe or inaccurate changes based on plausible text. However, the situation A chatbot scores high on five demo questions but fails on real customer requests is primarily about evaluation metrics and representative data. The targeted decision is: Measure business-relevant correctness, retrieval quality, latency, cost and safety on representative held-out cases rather than only demonstrations.
This choice addresses postdeployment monitoring and drift; its main associated risk is: A model that passed initial acceptance can deteriorate after source, traffic or product changes without visible errors. However, the situation A chatbot scores high on five demo questions but fails on real customer requests is primarily about evaluation metrics and representative data. The targeted decision is: Measure business-relevant correctness, retrieval quality, latency, cost and safety on representative held-out cases rather than only demonstrations.