Skip to content

Commit 24b1dc8

Browse files
88plug88plug
committed
k3d-gpu-plus: rebase node image on Ubuntu (26.04 default + 24.04), bake plugin
Ubuntu 26.04 becomes the default base; 24.04 is also built. Validated on real GPU hardware (wildpony: GTX 1080 Ti + RTX 3060, driver 580) — cluster comes up with nvidia.com/gpu=2 and zero kubectl apply. Why drop nvidia/cuda: a k3s NODE never used the CUDA runtime that base provided. The node only runs containerd + nvidia-container-runtime; GPU pods get driver libs injected from the host and bring their own CUDA. Building on plain ubuntu tracks new Ubuntu LTS without waiting for NVIDIA to publish a CUDA base for it (none exists for 26.04 yet) and sheds the unused CUDA layer (646MB vs larger). Changes: - Dockerfile: FROM ubuntu:$UBUNTU_TAG (default 26.04); drop CUDA_TAG. Install the container toolkit from the distro-agnostic repo (works on any Ubuntu). Bake share/nvidia-device-plugin.yml into /var/lib/rancher/k3s/server/manifests/ so k3s auto-deploys it on startup. - Launcher up: device plugin auto-deploys, so the kubectl apply step is gone — kept only as a fallback for custom images that don't ship the baked manifest. verify_gpus still asserts nvidia.com/gpu>0. - Pipeline: build a matrix of ubuntu 26.04 (=:latest, :ubuntu26.04) and 24.04 (:ubuntu24.04), each tagged with the k3s version. Track k3s + device-plugin (the real drifters, filter+paginate+empty-guard); CUDA fetch kept only to keep the `k3d-gpu test` pod image current. Ubuntu LTS is a deliberate 2-entry build matrix, not a daily-tracked drifter. - Docs: README + CONTRIBUTING reflect ubuntu base, 26.04 default/24.04 variant, baked plugin / zero-apply flow; Release History is now Date|K3s|DevicePlugin; removed CUDA_TAG build arg and the static device-plugin apply URL. Verify: both ubuntu variants build locally (baked manifest + nvidia-container- runtime + k3s present in each); shellcheck clean; workflow YAML valid; README auto-update awk simulated (K3S_TAG synced, release row appended, build-args table not polluted).
1 parent f988c21 commit 24b1dc8

5 files changed

Lines changed: 146 additions & 137 deletions

File tree

‎.github/workflows/build-and-push.yml‎

Lines changed: 75 additions & 85 deletions
Original file line numberDiff line numberDiff line change
@@ -19,12 +19,12 @@ jobs:
1919
fetch-depth: 0
2020
persist-credentials: true
2121

22-
# ─── 1. Fetch the latest k3s tag matching *-amd64 ───────────────────
22+
# ─── 1. Latest k3s tag (*-amd64) ────────────────────────────────────
2323
- name: Get latest k3s tag
2424
id: k3s
2525
run: |
26-
# name= filter scopes the scan to amd64 tags, then paginate via .next so a
27-
# high version can't be hidden past page 1 (push-order != version-order).
26+
# name= filter scopes the scan, then paginate via .next so a high
27+
# version can't be hidden past page 1 (push-order != version-order).
2828
latest=$(
2929
url="https://registry.hub.docker.com/v2/repositories/rancher/k3s/tags?page_size=100&name=amd64"
3030
for _ in $(seq 1 20); do
@@ -37,15 +37,21 @@ jobs:
3737
[ -n "$latest" ] || { echo "::error::no k3s tag resolved from registry"; exit 1; }
3838
echo "latest=$latest" >> "$GITHUB_OUTPUT"
3939
40-
# ─── 2. Fetch the latest CUDA tag matching *-base-ubuntu24.04 ───────
41-
- name: Get latest CUDA tag (newest base-ubuntu LTS)
40+
# ─── 2. Latest NVIDIA device plugin release (baked into the image) ───
41+
- name: Get latest NVIDIA device plugin version
42+
id: nvidia_plugin
43+
run: |
44+
latest=$(curl -fsSL \
45+
"https://api.github.com/repos/NVIDIA/k8s-device-plugin/releases/latest" \
46+
| jq -r '.tag_name')
47+
[ -n "$latest" ] && [ "$latest" != "null" ] || { echo "::error::no device-plugin release resolved"; exit 1; }
48+
echo "latest=$latest" >> "$GITHUB_OUTPUT"
49+
50+
# ─── 3. Latest CUDA base — used ONLY for the `k3d-gpu test` pod image, ─
51+
# not the node base (the node runs on Ubuntu now).
52+
- name: Get latest CUDA tag for the test pod image
4253
id: cuda
4354
run: |
44-
# Track the newest CUDA on the newest Ubuntu base NVIDIA publishes. Today
45-
# that resolves to base-ubuntu24.04; when NVIDIA ships base-ubuntu26.04 it
46-
# is adopted automatically — sort -V ranks higher CUDA first, then higher
47-
# Ubuntu. The name= filter + pagination is what fixes the historical
48-
# 13.2.1 -> 13.1.2 regression (a higher version sitting past page 1).
4955
latest=$(
5056
url="https://registry.hub.docker.com/v2/repositories/nvidia/cuda/tags?page_size=100&name=base-ubuntu"
5157
for _ in $(seq 1 20); do
@@ -55,86 +61,53 @@ jobs:
5561
url=$(printf '%s' "$resp" | jq -r '.next')
5662
done | grep -E '^[0-9][0-9.]*-base-ubuntu[0-9.]+$' | sort -V | tail -n1
5763
)
58-
[ -n "$latest" ] || { echo "::error::no CUDA tag resolved from registry"; exit 1; }
64+
[ -n "$latest" ] || { echo "::error::no CUDA test-image tag resolved"; exit 1; }
5965
echo "latest=$latest" >> "$GITHUB_OUTPUT"
6066
61-
# ─── 3. Fetch the latest NVIDIA device plugin version ───────────────
62-
- name: Get latest NVIDIA device plugin version
63-
id: nvidia_plugin
64-
run: |
65-
latest=$(curl -fsSL \
66-
"https://api.github.com/repos/NVIDIA/k8s-device-plugin/releases/latest" \
67-
| jq -r '.tag_name')
68-
[ -n "$latest" ] && [ "$latest" != "null" ] || { echo "::error::no device-plugin release resolved"; exit 1; }
69-
echo "latest=$latest" >> "$GITHUB_OUTPUT"
70-
71-
- name: Read current ARGs
67+
- name: Read current pinned versions
7268
id: current
7369
run: |
74-
ct=$(grep '^ARG CUDA_TAG' Dockerfile | cut -d= -f2 | tr -d '"')
75-
kt=$(grep '^ARG K3S_TAG' Dockerfile | cut -d= -f2 | tr -d '"')
70+
kt=$(grep '^ARG K3S_TAG' Dockerfile | cut -d= -f2 | tr -d '"')
7671
pp=$(grep -oE 'k8s-device-plugin:v[0-9]+\.[0-9]+\.[0-9]+' share/nvidia-device-plugin.yml | cut -d: -f2)
77-
echo "cuda=$ct" >> $GITHUB_OUTPUT
78-
echo "k3s=$kt" >> $GITHUB_OUTPUT
79-
echo "plugin=$pp" >> $GITHUB_OUTPUT
72+
ti=$(grep -oE 'K3D_GPU_TEST_IMAGE:-nvidia/cuda:[^}]*' scripts/k3d-gpu | sed 's#.*nvidia/cuda:##')
73+
echo "k3s=$kt" >> "$GITHUB_OUTPUT"
74+
echo "plugin=$pp" >> "$GITHUB_OUTPUT"
75+
echo "testcuda=$ti" >> "$GITHUB_OUTPUT"
8076
8177
- name: Determine if update is needed
8278
id: need
8379
run: |
84-
if [[ "${{ steps.cuda.outputs.latest }}" != "${{ steps.current.outputs.cuda }}" ]] || \
85-
[[ "${{ steps.k3s.outputs.latest }}" != "${{ steps.current.outputs.k3s }}" ]] || \
86-
[[ "${{ steps.nvidia_plugin.outputs.latest }}" != "${{ steps.current.outputs.plugin }}" ]]; then
87-
echo "update=true" >> $GITHUB_OUTPUT
80+
if [ "${{ steps.k3s.outputs.latest }}" != "${{ steps.current.outputs.k3s }}" ] || \
81+
[ "${{ steps.nvidia_plugin.outputs.latest }}" != "${{ steps.current.outputs.plugin }}" ] || \
82+
[ "${{ steps.cuda.outputs.latest }}" != "${{ steps.current.outputs.testcuda }}" ]; then
83+
echo "update=true" >> "$GITHUB_OUTPUT"
8884
else
89-
echo "update=false" >> $GITHUB_OUTPUT
85+
echo "update=false" >> "$GITHUB_OUTPUT"
9086
fi
9187
9288
- name: Abort if nothing to do
9389
if: steps.need.outputs.update == 'false'
9490
run: exit 0
9591

96-
- name: Bump Dockerfile ARGs
97-
if: steps.need.outputs.update == 'true'
98-
run: |
99-
sed -i \
100-
-e "s|^ARG CUDA_TAG=.*|ARG CUDA_TAG=\"${{ steps.cuda.outputs.latest }}\"|" \
101-
-e "s|^ARG K3S_TAG=.*|ARG K3S_TAG=\"${{ steps.k3s.outputs.latest }}\"|" \
102-
Dockerfile
103-
104-
- name: Bump bundled device-plugin manifest
92+
- name: Bump pinned versions
10593
if: steps.need.outputs.update == 'true'
10694
run: |
107-
sed -i \
108-
-e "s|k8s-device-plugin:v[0-9]*\.[0-9]*\.[0-9]*|k8s-device-plugin:${{ steps.nvidia_plugin.outputs.latest }}|" \
109-
share/nvidia-device-plugin.yml
110-
111-
- name: Bump launcher test-image default
112-
if: steps.need.outputs.update == 'true'
113-
run: |
114-
# Keep the k3d-gpu test pod on the same CUDA tag as the cluster image,
115-
# so the test image never drifts behind the auto-bumped Dockerfile ARG.
116-
sed -i \
117-
-e "s|K3D_GPU_TEST_IMAGE:-nvidia/cuda:[^}]*|K3D_GPU_TEST_IMAGE:-nvidia/cuda:${{ steps.cuda.outputs.latest }}|" \
118-
scripts/k3d-gpu
95+
sed -i "s|^ARG K3S_TAG=.*|ARG K3S_TAG=\"${{ steps.k3s.outputs.latest }}\"|" Dockerfile
96+
sed -i "s|k8s-device-plugin:v[0-9]*\.[0-9]*\.[0-9]*|k8s-device-plugin:${{ steps.nvidia_plugin.outputs.latest }}|" share/nvidia-device-plugin.yml
97+
sed -i "s|K3D_GPU_TEST_IMAGE:-nvidia/cuda:[^}]*|K3D_GPU_TEST_IMAGE:-nvidia/cuda:${{ steps.cuda.outputs.latest }}|" scripts/k3d-gpu
11998
12099
- name: Update README
121100
if: steps.need.outputs.update == 'true'
122101
run: |
123102
today=$(date -u +'%Y-%m-%d')
124-
entry="| $today | ${{ steps.cuda.outputs.latest }} | ${{ steps.k3s.outputs.latest }} |"
103+
entry="| $today | ${{ steps.k3s.outputs.latest }} | ${{ steps.nvidia_plugin.outputs.latest }} |"
125104
awk -v e="$entry" \
126105
-v k3s="${{ steps.k3s.outputs.latest }}" \
127-
-v cuda="${{ steps.cuda.outputs.latest }}" \
128-
-v nvidia_plugin="${{ steps.nvidia_plugin.outputs.latest }}" '
106+
-v plugin="${{ steps.nvidia_plugin.outputs.latest }}" '
129107
/^## Release History/ { in_rh = 1 }
130108
in_rh && /^\|----------/ { print; print e; in_rh = 0; next }
131109
/^\| `K3S_TAG`/ { sub(/`v[0-9][^`]*-amd64`/, "`" k3s "`"); print; next }
132-
/^\| `CUDA_TAG`/ { sub(/`[0-9][^`]*-base-ubuntu[0-9.]*`/, "`" cuda "`"); print; next }
133-
/^\| `K3D_GPU_TEST_IMAGE`/ { sub(/nvidia\/cuda:[0-9][^`]*/, "nvidia/cuda:" cuda); print; next }
134-
/--image=nvidia\/cuda:[0-9]/ { sub(/nvidia\/cuda:[0-9][^ ]*/, "nvidia/cuda:" cuda); print; next }
135-
/kubectl apply -f https:\/\/raw\.githubusercontent\.com\/NVIDIA\/k8s-device-plugin\// {
136-
sub(/v[0-9]+\.[0-9]+\.[0-9]+/, nvidia_plugin); print; next
137-
}
110+
/k8s-device-plugin\/v[0-9]/ { sub(/v[0-9]+\.[0-9]+\.[0-9]+/, plugin); print; next }
138111
{ print }
139112
' README.md > README.md.new && mv README.md.new README.md
140113
@@ -144,19 +117,9 @@ jobs:
144117
git config user.name "github-actions[bot]"
145118
git config user.email "github-actions[bot]@users.noreply.github.com"
146119
git add Dockerfile README.md share/nvidia-device-plugin.yml scripts/k3d-gpu
147-
git commit -m "chore: bump CUDA→${{ steps.cuda.outputs.latest }} & K3s→${{ steps.k3s.outputs.latest }} & device-plugin→${{ steps.nvidia_plugin.outputs.latest }}"
120+
git commit -m "chore: bump k3s→${{ steps.k3s.outputs.latest }} & device-plugin→${{ steps.nvidia_plugin.outputs.latest }}"
148121
git push origin HEAD:main
149122
150-
- name: Prepare build tags
151-
if: steps.need.outputs.update == 'true'
152-
run: |
153-
echo "CUDA_TAG=${{ steps.cuda.outputs.latest }}" >> $GITHUB_ENV
154-
echo "K3S_TAG=${{ steps.k3s.outputs.latest }}" >> $GITHUB_ENV
155-
156-
- name: Set up QEMU
157-
if: steps.need.outputs.update == 'true'
158-
uses: docker/setup-qemu-action@v3
159-
160123
- name: Set up Docker Buildx
161124
if: steps.need.outputs.update == 'true'
162125
uses: docker/setup-buildx-action@v3
@@ -176,35 +139,62 @@ jobs:
176139
username: ${{ github.actor }}
177140
password: ${{ secrets.GITHUB_TOKEN }}
178141

179-
- name: Build and push image to DockerHub and GHCR
142+
# Ubuntu 26.04 is the default (:latest); 24.04 is also published. Each
143+
# carries the baked device plugin, so clusters expose GPUs with no apply.
144+
- name: Build & push — Ubuntu 26.04 (default / latest)
180145
if: steps.need.outputs.update == 'true'
181146
uses: docker/build-push-action@v6
182147
with:
183148
context: .
184149
push: true
185150
platforms: linux/amd64
151+
build-args: |
152+
UBUNTU_TAG=26.04
186153
tags: |
187-
cryptoandcoffee/k3d-gpu:${{ env.CUDA_TAG }}-${{ env.K3S_TAG }}
188154
cryptoandcoffee/k3d-gpu:latest
189-
ghcr.io/${{ github.repository_owner }}/k3d-gpu:${{ env.CUDA_TAG }}-${{ env.K3S_TAG }}
155+
cryptoandcoffee/k3d-gpu:ubuntu26.04
156+
cryptoandcoffee/k3d-gpu:${{ steps.k3s.outputs.latest }}-ubuntu26.04
190157
ghcr.io/${{ github.repository_owner }}/k3d-gpu:latest
158+
ghcr.io/${{ github.repository_owner }}/k3d-gpu:ubuntu26.04
159+
ghcr.io/${{ github.repository_owner }}/k3d-gpu:${{ steps.k3s.outputs.latest }}-ubuntu26.04
160+
161+
- name: Build & push — Ubuntu 24.04 (variant)
162+
if: steps.need.outputs.update == 'true'
163+
uses: docker/build-push-action@v6
164+
with:
165+
context: .
166+
push: true
167+
platforms: linux/amd64
168+
build-args: |
169+
UBUNTU_TAG=24.04
170+
tags: |
171+
cryptoandcoffee/k3d-gpu:ubuntu24.04
172+
cryptoandcoffee/k3d-gpu:${{ steps.k3s.outputs.latest }}-ubuntu24.04
173+
ghcr.io/${{ github.repository_owner }}/k3d-gpu:ubuntu24.04
174+
ghcr.io/${{ github.repository_owner }}/k3d-gpu:${{ steps.k3s.outputs.latest }}-ubuntu24.04
191175
192176
- name: Create GitHub Release
193177
if: steps.need.outputs.update == 'true'
194178
uses: softprops/action-gh-release@v2
195179
env:
196180
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
197181
with:
198-
tag_name: ${{ steps.cuda.outputs.latest }}-${{ steps.k3s.outputs.latest }}
199-
release_name: k3d-gpu-${{ steps.cuda.outputs.latest }}-${{ steps.k3s.outputs.latest }}
182+
tag_name: ${{ steps.k3s.outputs.latest }}-ndp${{ steps.nvidia_plugin.outputs.latest }}
183+
name: k3d-gpu ${{ steps.k3s.outputs.latest }} (device-plugin ${{ steps.nvidia_plugin.outputs.latest }})
200184
body: |
201-
## 🚀 New k3d-gpu Release
202-
**Docker Hub**
203-
- `cryptoandcoffee/k3d-gpu:${{ steps.cuda.outputs.latest }}-${{ steps.k3s.outputs.latest }}`
204-
- `cryptoandcoffee/k3d-gpu:latest`
185+
## 🚀 k3d-gpu release
205186
206-
**GitHub Container Registry (GHCR)**
207-
- `ghcr.io/${{ github.repository_owner }}/k3d-gpu:${{ steps.cuda.outputs.latest }}-${{ steps.k3s.outputs.latest }}`
208-
- `ghcr.io/${{ github.repository_owner }}/k3d-gpu:latest`
187+
Ubuntu node base — **26.04** (default) and **24.04**. k3s
188+
`${{ steps.k3s.outputs.latest }}`, NVIDIA device plugin
189+
`${{ steps.nvidia_plugin.outputs.latest }}` (baked in — GPUs are
190+
exposed on `up` with no `kubectl apply`).
209191
192+
**Docker Hub**
193+
- `cryptoandcoffee/k3d-gpu:latest` (= ubuntu26.04)
194+
- `cryptoandcoffee/k3d-gpu:ubuntu26.04`
195+
- `cryptoandcoffee/k3d-gpu:ubuntu24.04`
210196
197+
**GHCR**
198+
- `ghcr.io/${{ github.repository_owner }}/k3d-gpu:latest`
199+
- `ghcr.io/${{ github.repository_owner }}/k3d-gpu:ubuntu26.04`
200+
- `ghcr.io/${{ github.repository_owner }}/k3d-gpu:ubuntu24.04`

‎CONTRIBUTING.md‎

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -35,16 +35,13 @@ Include relevant details:
3535
./build.sh
3636

3737
# Test with k3d (--default-runtime=nvidia is required, or the cluster
38-
# advertises zero GPUs even though the node can see them — see README)
38+
# advertises zero GPUs even though the node can see them — see README).
39+
# The device plugin is baked into the image, so no kubectl apply is needed.
3940
k3d cluster create test-gpu --image cryptoandcoffee/k3d-gpu --gpus all \
4041
--k3s-arg "--default-runtime=nvidia@server:*"
41-
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.19.2/nvidia-device-plugin.yml
42-
43-
# Verify GPU access on the node
44-
docker exec -it k3d-test-gpu-server-0 nvidia-smi
4542

4643
# Verify GPUs are schedulable (must be > 0)
47-
kubectl get nodes -o jsonpath='{.items[*].status.allocatable.nvidia\.com/gpu}'
44+
kubectl get nodes -o jsonpath='{.items[*].status.allocatable.nvidia\.com/gpu}{"\n"}'
4845

4946
# Cleanup
5047
k3d cluster delete test-gpu

‎Dockerfile‎

Lines changed: 19 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,21 @@
11
ARG K3S_TAG="v1.34.1-k3s1-amd64"
22
#https://hub.docker.com/r/rancher/k3s/tags
3-
ARG CUDA_TAG="13.1.2-base-ubuntu24.04"
4-
#https://hub.docker.com/r/nvidia/cuda/tags
3+
ARG UBUNTU_TAG="26.04"
4+
#https://hub.docker.com/_/ubuntu/tags (26.04 = default/latest; 24.04 also built)
55

66
FROM rancher/k3s:$K3S_TAG AS k3s
7-
FROM nvidia/cuda:$CUDA_TAG
87

9-
# Install the NVIDIA container toolkit
10-
RUN apt-get update && apt-get install -y curl \
8+
# A k3s NODE image does not need the CUDA toolkit. The node only runs containerd
9+
# + the NVIDIA container runtime; GPU pods get the driver libraries injected from
10+
# the host by nvidia-container-runtime and bring their own CUDA from their
11+
# workload image. So we build on plain ubuntu (not nvidia/cuda) — this tracks the
12+
# newest Ubuntu LTS without waiting for NVIDIA to publish a CUDA base for it, and
13+
# sheds the unused CUDA layer.
14+
FROM ubuntu:$UBUNTU_TAG
15+
16+
# Install the NVIDIA container toolkit (distro-agnostic stable/deb repo — the
17+
# package set is identical regardless of the Ubuntu version underneath).
18+
RUN apt-get update && apt-get install -y curl gnupg ca-certificates \
1119
&& curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
1220
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
1321
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
@@ -19,6 +27,12 @@ RUN apt-get update && apt-get install -y curl \
1927
COPY --from=k3s / /
2028
COPY --from=k3s /bin /bin
2129

30+
# Bake the NVIDIA device plugin into k3s's auto-deploy manifests dir. k3s applies
31+
# everything here on startup, so the cluster comes up with GPUs already exposed —
32+
# no `kubectl apply` step needed. Placed before the VOLUME lines so the volume's
33+
# initial contents (copied from the image) include it.
34+
COPY share/nvidia-device-plugin.yml /var/lib/rancher/k3s/server/manifests/nvidia-device-plugin.yaml
35+
2236
VOLUME /var/lib/kubelet
2337
VOLUME /var/lib/rancher/k3s
2438
VOLUME /var/lib/cni

0 commit comments

Comments
 (0)