Question 200
During a high-traffic model-service launch at an online commerce company, engineers report the following issue: A model is evaluated against examples that also appeared in training. The affected work supports product search, recommendations and fraud screening; peak request demand makes degraded predictions visible to customers. Which TWO responses form the best remediation-and-evidence pair? (Select TWO.)
Explanation
Correct Answers
Correct corrective action. Plan data collection, preparation, train and validation splits, experiments and versioned artifacts before production promotion so model behavior is reproducible. This fits the specific reported issue: A model is evaluated against examples that also appeared in training. The corresponding failure being reduced is: Undocumented data transformations and mixed training and test sets can make accuracy claims unreliable and unreproducible.
Correct independent evidence. An independent rerun identifies its dataset version, preprocessing choices, model checkpoint and held-out evaluation result. This verifies the corrective approach for the affected product search, recommendations and fraud screening instead of treating an architecture diagram alone as success.
Incorrect Answers
Not the most direct the action-and-evidence pair for these facts. Map image-based classification, detection or segmentation to a vision model and matching data pipeline, with application-specific accuracy and latency checks. This choice concerns computer vision inspection and imaging use cases and may be valid in another design. But this case reports 'A model is evaluated against examples that also appeared in training', which calls for reasoning about ai development lifecycle data and experiment tracking. The targeted response is: Plan data collection, preparation, train and validation splits, experiments and versioned artifacts before production promotion so model behavior is reproducible.
Not the most direct the action-and-evidence pair for these facts. Connect model registry versions, deployment monitoring and controlled retraining decisions in the AI lifecycle instead of assuming a one-time training job ends operations. This choice concerns production mlops deployment and feedback loop and may be valid in another design. But this case reports 'A model is evaluated against examples that also appeared in training', which calls for reasoning about ai development lifecycle data and experiment tracking. The targeted response is: Plan data collection, preparation, train and validation splits, experiments and versioned artifacts before production promotion so model behavior is reproducible.
Not the most direct the action-and-evidence pair for these facts. The chosen supported model exposes the expected inference API on compatible hardware and meets the required latency and correctness smoke tests. This choice concerns nvidia nim inference microservices and may be valid in another design. But this case reports 'A model is evaluated against examples that also appeared in training', which calls for reasoning about ai development lifecycle data and experiment tracking. The targeted response is: Plan data collection, preparation, train and validation splits, experiments and versioned artifacts before production promotion so model behavior is reproducible.