Question 30
A developer is testing a Snowflake Gen AI workflow with this requirement: A support answerer exceeds context limits when multiple large warranty manuals are retrieved. What check would reveal the relevant functional or governance failure rather than a cosmetic success?
Explanation
Correct Answer
“Context-limit error rate, grounded accuracy, effective retrieved-token utilization and end-to-end cost” is supported by this requirement: A support answerer exceeds context limits when multiple large warranty manuals are retrieved. Context windows and billed tokens impose practical limits on evidence selection and completion. Verify it using context-limit error rate, grounded accuracy, effective retrieved-token utilization and end-to-end cost.
Incorrect Answers
“Low input tokens achieved by omitting the entire evidence corpus” would not meet the requirement that a support answerer exceeds context limits when multiple large warranty manuals are retrieved. Context windows and billed tokens impose practical limits on evidence selection and completion. The more reliable approach is to select only relevant bounded evidence and set token/response budgets based on the model context and workload requirements.
“Longest possible generated response regardless of task size” would not meet the requirement that a support answerer exceeds context limits when multiple large warranty manuals are retrieved. Context windows and billed tokens impose practical limits on evidence selection and completion. The more reliable approach is to select only relevant bounded evidence and set token/response budgets based on the model context and workload requirements.
“Raw retrieved document bytes without any budget or quality analysis” would not meet the requirement that a support answerer exceeds context limits when multiple large warranty manuals are retrieved. Context windows and billed tokens impose practical limits on evidence selection and completion. The more reliable approach is to select only relevant bounded evidence and set token/response budgets based on the model context and workload requirements.