Question 590
During a multi-team AI platform review at an applied research institute, engineers report the following issue: Kubernetes nodes contain GPUs but the scheduler reports zero allocatable GPU resources. The affected work supports training and inference experiments; jobs must produce reproducible, defensible technical results. Which TWO responses form the best remediation-and-evidence pair? (Select TWO.)
Explanation
Correct Answers
Correct corrective action. Use appropriate NVIDIA GPU drivers, device discovery or GPU Operator components and Kubernetes resource requests so eligible workloads receive supported accelerator devices. This fits the specific reported issue: Kubernetes nodes contain GPUs but the scheduler reports zero allocatable GPU resources. The corresponding failure being reduced is: Pods cannot reliably obtain GPUs if device resources are not exposed or requested correctly, even when nodes physically contain accelerators.
Correct independent evidence. A test pod requests a supported accelerator resource, schedules to an eligible node and performs a successful device computation. This verifies the corrective approach for the affected training and inference experiments instead of treating an architecture diagram alone as success.
Incorrect Answers
Not the most direct the action-and-evidence pair for these facts. Maintain an inventory of GPU systems, drivers, firmware, network devices and approved configuration versions with controlled rollout and reconciliation. This choice concerns cluster inventory firmware and configuration baseline and may be valid in another design. But this case reports 'Kubernetes nodes contain GPUs but the scheduler reports zero allocatable GPU resources', which calls for reasoning about Kubernetes gpu device scheduling and discovery. The targeted response is: Use appropriate NVIDIA GPU drivers, device discovery or GPU Operator components and Kubernetes resource requests so eligible workloads receive supported accelerator devices.
Not the most direct the action-and-evidence pair for these facts. Place tightly coupled distributed training processes with awareness of accelerator counts, network topology and collective communication demands, and validate gang-scheduling needs. This choice concerns distributed job placement and topology-aware orchestration and may be valid in another design. But this case reports 'Kubernetes nodes contain GPUs but the scheduler reports zero allocatable GPU resources', which calls for reasoning about Kubernetes gpu device scheduling and discovery. The targeted response is: Use appropriate NVIDIA GPU drivers, device discovery or GPU Operator components and Kubernetes resource requests so eligible workloads receive supported accelerator devices.
Not the most direct the action-and-evidence pair for these facts. A time-series review connects kernel activity and memory use to actual completed training or inference work while checking limit events. This choice concerns gpu utilization memory and power telemetry interpretation and may be valid in another design. But this case reports 'Kubernetes nodes contain GPUs but the scheduler reports zero allocatable GPU resources', which calls for reasoning about Kubernetes gpu device scheduling and discovery. The targeted response is: Use appropriate NVIDIA GPU drivers, device discovery or GPU Operator components and Kubernetes resource requests so eligible workloads receive supported accelerator devices.