I had some difficulties and some advice on the LLMs or the internet is outdated so here is some guide what worked for me.
You most likely also want CUDA installed. Install the latest cuda toolkit from here: https://developer.nvidia.com/cuda-downloads
Check if running nvidia-smi gives you some output.
Docker needs a special back-end so the toolkit should also install you nvidia-docker.
There is no latest tag for the nvidia images so you can test if it works by running a particular version e.g.
nvidia-docker run --rm nvidia/cuda:12.5.0-devel-ubuntu22.04 nvidia-smiYou can enable docker in you local network. This way you can use docker like you have it installed locally but it builds and runs it on you server. To use it on you client you set the env var e.g.
export DOCKER_HOST=tcp://hostname:2375
Note that this runs the default docker and not the nvidia-docker. In order to change that you can change the default runner in a config on the server.
In this file /etc/docker/daemon.json add
"default-runtime": "nvidia"and restart docker with sudo systemctl restart docker.
Note that CUDA is not available in the runtime during the build. But that should not be a problem. In my case I could ignore a warnign that cuda could not be found.