cuda - Inconsistency of IDs between 'nvidia-smi -L' and cuDeviceGetName()

Question

I'm running this command into a shell and get:

C:Usersme>nvidia-smi -L    
GPU 0: Quadro K2000 (UUID: GPU-b1ac50d1-019c-58e1-3598-4877fddd3f17)    
GPU 1: Quadro 2000 (UUID: GPU-1f22a253-c329-dfb7-0db4-e005efb6a4c7)

But in my code, when I run cuDeviceGetName(.., ID) where ID is the ID given by the nvidia-smi output, the devices have been inverted: GPU 0 becomes Quadro 2000 and GPU 1 becomes Quadro K2000.

Is this an expected behavior or a bug ? Does anyone know a workaround to make nvidia-smi get the 'real' ID of GPUs ? I could use the UUID to get the proper device with nvmlDeviceGetUUID() but using nvml API seems a bit too complicated for what I'm trying to achieve.

This question discuss how CUDA assign IDs to devices without clear conclusion.

I am using CUDA 6.5.

EDIT: I've had a look at nvidia-smi manpage (should have done that earlier...). It states:

"It is recommended that users desiring consistencyuse either UUDI or PCI bus ID, since device enumeration ordering is not guaranteed to be consistent"

Still looking for a kludge...

Answer 1

You can set the device order for CUDA environment in your shell to follow the bus ID instead of the default of fastest card. Requires CUDA 7 and up.

export CUDA_DEVICE_ORDER=PCI_BUS_ID