Tearing Ultron down from Proxmox to bare Debian

Ultron is the GPU node in my homelab, a single-socket Xeon workstation that for a year ran Proxmox like the rest of the cluster. Last week I wiped it and reinstalled bare Debian 13 (Trixie), because the one job I actually want from that box, running CUDA workloads against its GPU, is the one job a hypervisor makes harder rather than easier. The reinstall took twenty minutes. Getting the driver to load took the rest of the evening, almost all of it on a single thing nobody warns you about: Secure Boot silently refusing an unsigned module.

The order of operations that actually works on Trixie is only a few steps, one of which is documented nowhere.

Why a hypervisor was the wrong layer here

Proxmox earns its place when you are consolidating many guests onto one machine. GPU compute is the opposite shape. To give a virtual machine a real GPU you go through VFIO passthrough, the mechanism that detaches a device from the host and hands it to a guest. That means sorting out IOMMU groups, the device blocks the hardware refuses to separate, blacklisting the host from ever binding the card, then handing the whole device to exactly one guest. You end up talking to your GPU through a virtual machine, the card can only ever belong to one VM at a time anyway, and you are carrying all of that machinery for a node that does precisely one thing.

A box whose entire purpose is one GPU does not need a hypervisor sitting between me and nvidia-smi. Remove the layer and the card is back on bare metal, with the passthrough tax gone along with it.

The same hardware, two stacks. Passthrough buys flexibility a single-purpose GPU node never uses.

Blacklisting nouveau, the part that is easy to forget

Debian ships nouveau, the open-source driver, and loads it at boot. The proprietary module will not bind while nouveau is holding the card, so the first move is to blacklist it and rebuild the initramfs, so the change is in place from early boot rather than after the kernel has already claimed the GPU.

# /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
options nouveau modeset=0

# then regenerate the initramfs so it sticks at boot
sudo update-initramfs -u

The driver itself: let DKMS do the building

Trixie keeps the NVIDIA driver in the non-free component and its firmware in non-free-firmware, so the sources have to be widened before any of it is installable. Then you install the kernel headers and the driver package, and Debian uses DKMS, the mechanism that rebuilds kernel modules on every upgrade, to compile the module against your running kernel. Hence the packaged driver rather than the .run installer: DKMS rebuilds the module automatically on the next kernel upgrade, so an apt upgrade does not quietly leave you with a black screen.

# add  contrib non-free non-free-firmware  to your apt sources, then:
sudo apt update
sudo apt install linux-headers-amd64 nvidia-driver

Secure Boot, or why nvidia-smi lied to me

After the reboot I ran nvidia-smi and got this:

$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the
NVIDIA driver. Make sure that the latest NVIDIA driver is installed
and running.

The card was fine and the module had built without complaint. The kernel was simply refusing to load it, because Secure Boot was on and a DKMS-built module is unsigned. There are two ways out. You can turn Secure Boot off in firmware, or you can enroll a Machine Owner Key, sign the module with it, and keep the chain of trust intact. I kept Secure Boot and enrolled a key, which is a one-time dance through the firmware on the next reboot.

# enroll the DKMS signing key, set a one-time password, then reboot
sudo mokutil --import /var/lib/dkms/mok.pub
# at the blue MOK manager on reboot: Enroll MOK, enter the password, reboot

After that, nvidia-smi came up clean with the card and driver version. No install log mentions that step.

The whole sequence. Every box except the amber one is mechanical; the amber one is where a clean build still gives you a dead nvidia-smi.

What I got back

A bare nvidia-smi, the full card with no virtual machine in the way, and a node that now runs my Kokkos energy-measurement work straight against the hardware instead of through a guest. The rest of the cluster is still Proxmox: those nodes are doing the consolidation job Proxmox is good at. Ultron was not doing that job.

Still to do: wire the box's power telemetry into the same dashboard as the GPU work, so the node reports joules-per-run alongside utilization. Keeping one bare-metal node inside a Proxmox cluster remains awkward on the monitoring side, so for now it lives beside the fold rather than in it.