Question 50
An engineer observes: A server lists all GPUs successfully but distributed training fails whenever an application uses inter-GPU communication. Which underlying mechanism is most relevant?
Explanation
Correct Answer
The operational evidence follows from the mechanism that Device visibility does not prove the accelerator interconnect will support training collectives. Address it by Validate NVLink or supported intra-node communication using NCCL tests after initial device discovery; then test whether A single-node all-reduce test completes and reaches the expected topology-specific bandwidth.
Incorrect Answers
This alternative addresses the separate case “A server can execute one GPU benchmark but its expected GPU memory capacity is reduced by an unintended MIG partition profile”. The operational evidence follows from the mechanism that Logical GPU partition settings change the resources available to workloads regardless of physical board count. The deciding requirement here is “Device visibility does not prove the accelerator interconnect will support training collectives”.
The option would apply to “A GPU server passes nvidia-smi query but CUDA sample binaries fail to allocate device memory under an ordinary user account”. The operational evidence follows from the mechanism that Administrative device inspection is not equivalent to validating real application execution permissions. This question instead requires “Device visibility does not prove the accelerator interconnect will support training collectives”.
The related but distinct situation is “A GPU node's driver initializes but the cluster image lacks the libraries used by the production training framework”. The key distinction is that A generic device test cannot certify that the selected framework stack is compatible with the host. For the stated problem, “Device visibility does not prove the accelerator interconnect will support training collectives” addresses the central condition.