How to Install NVIDIA CUDA Toolkit on Ubuntu 24.04 Dedicated GPU Server

What You'll Learn

Why Use the Network Repository Method?

NVIDIA offers multiple ways to install the CUDA toolkit, including local runfiles and deb packages. However, as server administrators and infrastructure experts, we highly recommend the Network Repository (apt) method.

• Seamless Updates: It integrates directly with Ubuntu's Advanced Package Tool (apt), making future driver and toolkit updates as simple as running sudo apt upgrade.
• Dependency Management: The package manager automatically resolves and installs necessary dependencies, significantly reducing the risk of broken packages or kernel panics.

Phase 1: Pre-Installation Checks and Server Preparation

Before downloading any packages, we must ensure your dedicated server is running compatible hardware and a clean software environment. Skipping this phase is the leading cause of installation failures.

1

Verify Your GPU is CUDA-Capable

First, confirm that your system recognizes your NVIDIA GPU. Connect to your server via SSH and run the following command:

lspci | grep -i nvidia

Expected Output: You should see a list of your installed NVIDIA graphics cards (e.g., NVIDIA Corporation AD102GL [RTX 6000 Ada Generation]). If this returns nothing, check your server's PCIe connections or contact your hosting provider's hardware support.

2

Verify Your Operating System

Ensure you are running Ubuntu 24.04 LTS.

cat /etc/os-release

Expected Output: Look for PRETTY_NAME="Ubuntu 24.04 LTS".

3

Verify GCC Installation

The CUDA Toolkit requires a C compiler to compile host code. Ubuntu 24.04 usually comes with GCC pre-installed, but you should verify:

gcc --version

If it is not installed, install the essential build packages:

sudo apt update
sudo apt install build-essential -y

4

Verify the System has the Correct Kernel Headers

CUDA requires the kernel headers and development packages for the currently running kernel to build the NVIDIA driver modules securely.

sudo apt-get install linux-headers-$(uname -r)

5

Remove Outdated NVIDIA Drivers (If Applicable)

If your dedicated server was previously configured with open-source drivers (Nouveau) or older, conflicting NVIDIA drivers, you must purge them to prevent module conflicts.

sudo apt-get --purge remove "*nvidia*"
sudo apt-get autoremove
sudo apt-get autoclean

Phase 2: Installing the NVIDIA CUDA Toolkit

With the system prepped, we can proceed to the actual installation. We will add the official NVIDIA package repositories to your Ubuntu 24.04 server.

1

Download the NVIDIA CUDA Keyring

To ensure the authenticity of the packages you download, NVIDIA signs them with a GPG key. You need to install this keyring into your server's trusted repositories. Run the following commands sequentially:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb

2

Update the Apt Repository Cache

Now that your server trusts the NVIDIA repository, update your apt cache so it can see the newly available CUDA packages.

sudo apt-get update

3

Install the CUDA Toolkit

You have a choice here. You can install the generic cuda package (which installs the latest toolkit and the latest proprietary NVIDIA drivers) or a specific toolkit version (e.g., cuda-toolkit-12-4) if your specific machine learning framework (like PyTorch or TensorFlow) requires a strict version.

For the vast majority of deep learning and HPC workloads on a fresh server, installing the latest unified package is the best approach:

sudo apt-get -y install cuda-toolkit-12-x
# Note: Replace '12-x' with the exact latest version block you need, 
# or simply run the following to get the default bundled version:
sudo apt-get -y install cuda

(Depending on your server’s internet speed and disk I/O, this process may take 5 to 15 minutes. The installation handles the proprietary NVIDIA driver, the CUDA libraries, and the compiler).

4

Install the NVIDIA Driver (If not bundled)

If you opted to install only the toolkit using cuda-toolkit-12-x instead of the metapackage cuda, you must install the proprietary driver separately:

sudo apt-get -y install nvidia-driver-550

(Note: Driver versions iterate quickly. Use ubuntu-drivers devices to see the recommended driver version for your specific GPU architecture).

5

Reboot Your Server

Kernel-level modules have been installed. To ensure the NVIDIA driver is loaded properly into the Linux kernel and the open-source Nouveau driver is fully disabled, a reboot is mandatory.

sudo reboot

Phase 3: Post-Installation Configuration

A common mistake made by beginners is assuming the installation is complete after the reboot. However, Ubuntu does not automatically add the CUDA compiler (nvcc) to your system's executable path.

If you try to check your CUDA version right now, the terminal will likely throw a "command not found" error. You must configure your environment variables.

1

Update the bashrc File

You need to append the CUDA binary path to your system's $PATH variable and the library path to $LD_LIBRARY_PATH. Open your bash profile:

nano ~/.bashrc

Scroll to the very bottom of the file and paste the following lines:

# NVIDIA CUDA Toolkit Environment Variables
export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

Save the file (CTRL + O, Enter, CTRL + X).

2

Source the Profile

To apply these changes immediately without logging out and back into your SSH session, run:

source ~/.bashrc

Phase 4: Verifying the Installation

To guarantee your dedicated GPU server is ready for heavy workloads, we need to run two distinct verification tests.

1

Verify the NVIDIA Driver

First, let's check if the proprietary driver is communicating with the hardware. Run the System Management Interface command:

nvidia-smi

What to look for in the output:

• Your specific GPU model should be listed.
• The Driver Version should be visible (e.g., 550.xx).
• The CUDA Version supported by this driver should be shown in the top right corner.

2

Verify the CUDA Compiler (NVCC)

Next, ensure that the actual CUDA Toolkit (the development tools used to compile AI software) is properly installed and accessible.

nvcc --version

What to look for in the output:

• You should see a copyright notice from NVIDIA and the specific release version of the CUDA compiler (e.g., Cuda compilation tools, release 12.4, V12.4.xx).

If both commands return the correct data, congratulations! Your Ubuntu 24.04 dedicated server is fully equipped with the NVIDIA CUDA Toolkit.

Troubleshooting Common Errors

Even with a flawless execution, server environments can have unique quirks. Here are solutions to the most common issues administrators face on Ubuntu 24.04.

sudo apt --fix-broken install
sudo apt-get autoremove
sudo apt-get update