Linux NVIDIA

The following troubleshooting techniques may help resolve problems with NVIDIA on Linux.

  • If you are using a container, is the container runtime working? Try docker run --gpus all ubuntu nvidia-smi - if this doesn't work, no programs will be able to see your NVIDIA GPU.
  • Check if the uvm driver is loaded sudo nvidia-modprobe -u
  • Reloading the nvidia_uvm driver - sudo rmmod nvidia_uvm then sudo modprobe nvidia_uvm
  • Try rebooting

If none of those resolve the problem, gather additional information:

  • Set CUDA_ERROR_LEVEL=50 to get more diagnostic logs
  • Check dmesg for any errors sudo dmesg | grep -i nvrm and sudo dmesg | grep -i nvidia

You may get more details for initialization failures by enabling debug prints in the uvm driver. You should only use this temporarily while troubleshooting

  • sudo rmmod nvidia_uvm then sudo modprobe nvidia_uvm uvm_debug_prints=1