Question 60
During a high-traffic model-service launch at an online commerce company, engineers report the following issue: The inference server queues requests to build huge batches during low traffic. The affected work supports product search, recommendations and fraud screening; peak request demand makes degraded predictions visible to customers. Select TWO: the most appropriate action and the strongest verification of that action. (Select TWO.)
Explanation
Correct Answers
Correct corrective action. Evaluate request latency, concurrency, batching and model size together when selecting inference resources, with measurements under a representative service-level target. This fits the specific reported issue: The inference server queues requests to build huge batches during low traffic. The corresponding failure being reduced is: Maximizing batch throughput without tracking queue delay can violate interactive inference latency even if GPUs appear busy.
Correct independent evidence. Load testing reports end-to-end latency percentiles and completed requests per second at the intended concurrency and batch settings. This verifies the corrective approach for the affected product search, recommendations and fraud screening instead of treating an architecture diagram alone as success.
Incorrect Answers
Not the most direct the action-and-evidence pair for these facts. A representative image dataset is used to assess the selected vision task and error categories in realistic operating conditions. This choice concerns computer vision inspection and imaging use cases and may be valid in another design. But this case reports 'The inference server queues requests to build huge batches during low traffic', which calls for reasoning about inference latency throughput and batching tradeoffs. The targeted response is: Evaluate request latency, concurrency, batching and model size together when selecting inference resources, with measurements under a representative service-level target.
Not the most direct the action-and-evidence pair for these facts. Distinguish training with repeated forward and backward passes and model updates from inference that applies a trained model to new inputs, and size compute for the actual phase. This choice concerns training versus inference compute requirements and may be valid in another design. But this case reports 'The inference server queues requests to build huge batches during low traffic', which calls for reasoning about inference latency throughput and batching tradeoffs. The targeted response is: Evaluate request latency, concurrency, batching and model size together when selecting inference resources, with measurements under a representative service-level target.
Not the most direct the action-and-evidence pair for these facts. Recognize multilayer neural representation learning as deep learning within machine learning and choose it when the data and task justify the added training complexity. This choice concerns deep learning neural networks versus classical ml and may be valid in another design. But this case reports 'The inference server queues requests to build huge batches during low traffic', which calls for reasoning about inference latency throughput and batching tradeoffs. The targeted response is: Evaluate request latency, concurrency, batching and model size together when selecting inference resources, with measurements under a representative service-level target.