Setting Up RamaLama with CUDA Support on Linux systems

This guide walks through the steps required to set up RamaLama with CUDA support.

Install the NVIDIA Container Toolkit

Follow the installation instructions provided in the NVIDIA Container Toolkit installation guide.

Installation using dnf/yum (For RPM based distros like Fedora)

Install the NVIDIA Container Toolkit packages

sudo dnf install -y nvidia-container-toolkit

<Note>
 The NVIDIA Container Toolkit is required on the host for running CUDA in containers.
</Note>
<Note>
 If the above installation is not working for you and you are running Fedora, try removing it and using the [COPR](https://copr.fedorainfracloud.org/coprs/g/ai-ml/nvidia-container-toolkit/).
</Note>

### Installation using APT (For Debian based distros like Ubuntu)

* Configure the Production Repository

   ```bash
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
   sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

   curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
   sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
   sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Update the packages list from the repository

sudo apt-get update

* Install the NVIDIA Container Toolkit packages

   ```bash
sudo apt-get install -y nvidia-container-toolkit

The NVIDIA Container Toolkit is required for WSL to have CUDA resources while running a container.

Setting Up CUDA Support

For additional information see: Support for Container Device Interface

Generate the CDI specification file

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

Check the names of the generated devices

Open and edit the NVIDIA container runtime configuration:

nvidia-ctk cdi list
INFO[0000] Found 1 CDI devices
nvidia.com/gpu=all

Generate a new CDI specification after any configuration change most notably when the driver is upgraded!

Testing the Setup

Based on this Documentation: Running a Sample Workload

Test the Installation

Run the following command to verify setup:

podman run --rm --device=nvidia.com/gpu=all fedora nvidia-smi

Expected Output

Verify everything is configured correctly, with output similar to this:

Thu Dec  5 19:58:40 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.72                 Driver Version: 566.14         CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3080        On  |   00000000:09:00.0  On |                  N/A |
| 34%   24C    P5             31W /  380W |     867MiB /  10240MiB |      7%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A        35      G   /Xwayland                                   N/A      |
|    0   N/A  N/A        35      G   /Xwayland                                   N/A      |
+-----------------------------------------------------------------------------------------+

On systems that have SELinux enabled, it may be necessary to turn on the container_use_devices boolean in order to run the nvidia-smi command successfully from a container.

To check the status of the boolean, run the following:

getsebool container_use_devices

If the result of the command shows that the boolean is off, run the following to turn the boolean on:

sudo setsebool -P container_use_devices 1

CUDA_VISIBLE_DEVICES

RamaLama respects the CUDA_VISIBLE_DEVICES environment variable if it’s already set in your environment. If not set, RamaLama will default to using all the GPU detected by nvidia-smi. You can specify which GPU devices should be visible to RamaLama by setting this variable before running RamaLama commands:

export CUDA_VISIBLE_DEVICES="0,1"  # Use GPUs 0 and 1
ramalama run granite

This is particularly useful in multi-GPU systems where you want to dedicate specific GPUs to different workloads. If the CUDA_VISIBLE_DEVICES environment variable is set to an empty string, RamaLama will default to using the CPU.

export CUDA_VISIBLE_DEVICES=""  # Defaults to CPU
ramalama run granite

To revert to using all available GPUs, unset the environment variable:

unset CUDA_VISIBLE_DEVICES

Troubleshooting

CUDA Updates

On some CUDA software updates, RamaLama stops working complaining about missing shared NVIDIA libraries for example:

ramalama run granite
Error: crun: cannot stat `/lib64/libEGL_nvidia.so.565.77`: No such file or directory: OCI runtime attempted to invoke a command that was not found

Because the CUDA version is updated, the CDI specification file needs to be recreated.

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

Overview

Configuration

Platform Guides

Commands

cuda

Setting Up RamaLama with CUDA Support on Linux systems

Install the NVIDIA Container Toolkit

Installation using dnf/yum (For RPM based distros like Fedora)

Setting Up CUDA Support

Generate the CDI specification file

Check the names of the generated devices

Testing the Setup

Test the Installation

Expected Output

CUDA_VISIBLE_DEVICES

Troubleshooting

CUDA Updates

See Also

​Setting Up RamaLama with CUDA Support on Linux systems

​Install the NVIDIA Container Toolkit

​Installation using dnf/yum (For RPM based distros like Fedora)

​Setting Up CUDA Support

​Generate the CDI specification file

​Check the names of the generated devices

​Testing the Setup

​Test the Installation

​Expected Output

​CUDA_VISIBLE_DEVICES

​Troubleshooting

​CUDA Updates

​See Also

Setting Up RamaLama with CUDA Support on Linux systems

Install the NVIDIA Container Toolkit

Installation using dnf/yum (For RPM based distros like Fedora)

Setting Up CUDA Support

Generate the CDI specification file

Check the names of the generated devices

Testing the Setup

Test the Installation

Expected Output

CUDA_VISIBLE_DEVICES

Troubleshooting

CUDA Updates

See Also