Skip to content

Commit e45c876

Browse files
kvinwangkvinwang
committed
Add images with nvidia GPU-TEE support
1 parent 9932ef4 commit e45c876

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+1122
-102
lines changed

‎Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ DIST_DIR ?= ${BB_BUILD_DIR}/dist
99
export BB_BUILD_DIR
1010
export DIST_DIR
1111

12-
DIST_NAMES ?= dstack dstack-dev
12+
DIST_NAMES ?= dstack dstack-dev dstack-nvidia dstack-nvidia-dev
1313
ROOTFS_IMAGE_NAMES = $(addsuffix -rootfs,${DIST_NAMES})
1414

1515
all: dist

‎meta-dstack/conf/distro/dstack.conf‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@ PREFERRED_PROVIDER_virtual/kernel = "linux-yocto-dev"
1414
PREFERRED_VERSION_linux-yocto-dev ?= "6.9%"
1515
LINUX_KERNEL_TYPE = "tiny"
1616

17+
MACHINE_FEATURES += "numa"
18+
1719
VIRTUAL-RUNTIME_init_manager = "systemd"
1820
VIRTUAL-RUNTIME_initscripts = ""
1921
VIRTUAL-RUNTIME_dev_manager = "systemd"

‎meta-dstack/recipes-core/dstack-guest/dstack-guest.bb‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,12 @@ inherit cargo_bin
2929
do_unpack() {
3030
mkdir -p ${S}
3131
rsync -a --exclude="target" ${SRC_DIR}/ ${S}/
32-
cp ${THISDIR}/files/docker-daemon.json ${S}/
32+
33+
if ${@bb.utils.contains('IMAGE_INSTALL', 'nvidia-container-toolkit', 'true', 'false', d)}; then
34+
cp ${THISDIR}/files/docker-daemon-nvidia.json ${S}/docker-daemon.json
35+
else
36+
cp ${THISDIR}/files/docker-daemon.json ${S}/docker-daemon.json
37+
fi
3338
}
3439

3540
# Force the configure task to run every time to detect source changes

‎meta-dstack/recipes-core/dstack-guest/files/docker-daemon-nvidia.json‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"log-driver": "json-file",
3+
"log-opts": {
4+
"max-size": "100m",
5+
"max-file": "10"
6+
},
7+
"runtimes": {
8+
"nvidia": {
9+
"args": [],
10+
"path": "nvidia-container-runtime"
11+
}
12+
}
13+
}

‎meta-dstack/recipes-core/images/dstack-nvidia-dev-rootfs.bb‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
include dstack-rootfs-dev.inc
2+
include dstack-rootfs-nvidia.inc

‎meta-dstack/recipes-core/images/dstack-nvidia-rootfs.bb‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
include dstack-rootfs-prod.inc
2+
include dstack-rootfs-nvidia.inc
3+

‎meta-dstack/recipes-core/images/dstack-rootfs-nvidia.inc‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
NVIDIA_GROUP = "acpid \
2+
nvidia \
3+
nvidia-firmware \
4+
nvidia-persistenced \
5+
nvidia-fabricmanager \
6+
libnvidia-nscq \
7+
nvidia-container-toolkit \
8+
kernel-module-video \
9+
numactl \
10+
"
11+
KERNEL_MODULE_AUTOLOAD:append = " nvidia nvidia-drm nvidia-modeset nvidia-uvm video"
12+
IMAGE_INSTALL:append = " ${NVIDIA_GROUP}"

‎meta-nvidia/README.md‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# Overview
2+
3+
The meta-nvidia layer provides support for NVIDIA graphics drivers and related components for Yocto Project-based distributions. This layer includes recipes for building the NVIDIA binary graphics driver, the GL Vendor-Neutral Dispatch library (libglvnd), and modifications to the mesa package to ensure compatibility with NVIDIA's proprietary drivers.
4+
5+
![loaded-modules](artwork/loaded_modules.png)
6+
# Contents
7+
8+
- Configuration: The layer.conf file contains the necessary configurations for the layer, including BBPATH, BBFILES, and other essential settings.
9+
- Custom Licenses: The custom-licenses directory contains custom license files that may be required by the recipes in this layer.
10+
- Recipes:
11+
- libglvnd: Provides the GL Vendor-Neutral Dispatch library.
12+
- mesa: Contains modifications to the mesa package to ensure compatibility with NVIDIA's proprietary drivers.
13+
- nvidia: Contains recipes for building NVIDIA's binary graphics driver and related components.
14+
15+
# Key Features
16+
17+
- libglvnd: The GL Vendor-Neutral Dispatch library allows multiple OpenGL implementations to coexist on the same system.
18+
- NVIDIA Binary Graphics Driver: Provides support for NVIDIA GPUs, enabling hardware-accelerated graphics, CUDA support, and other NVIDIA-specific features.
19+
- Mesa Modifications: Ensures that the open-source Mesa graphics library can coexist with NVIDIA's proprietary drivers.
20+
21+
# Usage
22+
23+
To use the meta-nvidia layer in your Yocto Project build:
24+
25+
- Clone the meta-nvidia repository to your local machine.
26+
- Add the path to the meta-nvidia layer to your bblayers.conf file.
27+
- Include the desired recipes in your image or build them individually using bitbake.
28+
29+
To use this layer, include it in your bblayers.conf and add the
30+
following to your BSP, distro, or local config:
31+
32+
```bash
33+
DISTRO_FEATURES:append = " x11 opengl"
34+
DISTRO_FEATURES:remove = " wayland"
35+
IMAGE_INSTALL:append = " libxshmfence cmake"
36+
IMAGE_INSTALL:append = " packagegroup-core-buildessential"
37+
IMAGE_INSTALL:append = " acpid"
38+
IMAGE_INSTALL:append = " nvidia"
39+
PREFERRED_PROVIDER_virtual/libgl = "libglvnd"
40+
PREFERRED_PROVIDER_virtual/libgles1 = "libglvnd"
41+
PREFERRED_PROVIDER_virtual/libgles3 = "libglvnd"
42+
PREFERRED_PROVIDER_virtual/egl = "libglvnd"
43+
PREFERRED_PROVIDER_virtual/libgl-native = "mesa-native"
44+
PREFERRED_PROVIDER_virtual/nativesdk-libgl = "nativesdk-mesa-gl"
45+
PREFERRED_PROVIDER_virtual/mesa = "libglvnd"
46+
KERNEL_MODULE_AUTOLOAD:append = " nvidia nvidia-drm nvidia-modeset nvidia-uvm"
47+
XSERVER = " \
48+
${XSERVER_X86_BASE} \
49+
${XSERVER_X86_EXT} \
50+
${XSERVER_X86_MODESETTING} \
51+
nvidia"
52+
```
53+
54+
# Testing nvidia-container-toolkit and GPU Workloads
55+
56+
- For testing nvidia-container-toolkit inside the container execute the following commands:
57+
58+
```bash
59+
sudo ctr images pull docker.io/nvidia/cuda:12.0.0-base-ubuntu20.04
60+
61+
sudo ctr run --rm --gpus 0 --runtime io.containerd.runc.v1 --privileged docker.io/nvidia/cuda:12.0.0-base-ubuntu20.04 nvidia-smi nvidia-smi
62+
```
63+
64+
- For testing nvidia-container-toolkit with k3s
65+
66+
```bash
67+
cat <<EOF | kubectl create -f -
68+
apiVersion: v1
69+
kind: Pod
70+
metadata:
71+
name: gpu
72+
spec:
73+
restartPolicy: Never
74+
runtimeClassName: nvidia
75+
containers:
76+
- name: gpu
77+
image: "nvidia/cuda:12.0.0-base-ubuntu20.04"
78+
command: [ "/bin/bash", "-c", "--" ]
79+
args: [ "while true; do sleep 30; done;" ]
80+
resources:
81+
limits:
82+
nvidia.com/gpu: 0
83+
EOF
84+
85+
kubectl exec -it gpu -- nvidia-smi
86+
```
87+
88+
- For testing a sample GPU workload in k3s
89+
90+
```bash
91+
cat << EOF | kubectl create -f -
92+
apiVersion: v1
93+
kind: Pod
94+
metadata:
95+
name: cuda-vectoradd
96+
spec:
97+
restartPolicy: OnFailure
98+
runtimeClassName: nvidia
99+
containers:
100+
- name: cuda-vectoradd
101+
image: "nvidia/samples:vectoradd-cuda11.2.1"
102+
resources:
103+
limits:
104+
nvidia.com/gpu: 0
105+
EOF
106+
```
107+
108+
# Dependencies
109+
110+
This layer depends on following layers:
111+
112+
URI: git://git.yoctoproject.org/poky
113+
layers: meta
114+
branch: mickledore
115+
116+
URI: git://git.yoctoproject.org/meta-openembedded
117+
layers: meta-oe
118+
branch: mickledore
119+
120+
URI: git://git.yoctoproject.org/meta-virtualization
121+
layers: meta-virtualization
122+
branch: mickledore
123+
124+
125+
# Compatibility
126+
127+
The meta-nvidia layer is ONLY compatible with the "mickledore" series of the Yocto Project.
128+
129+
# License
130+
131+
The recipes in this layer are licensed under various licenses, including MIT, BSD, and NVIDIA's proprietary license. Please refer to the individual recipe files and the custom-licenses directory for detailed licensing information.
132+
Contributing
133+
134+
Contributions to the meta-nvidia layer are welcome. Please ensure that any changes are tested with the target hardware and do not introduce regressions.
135+
136+
# Disclaimer
137+
138+
This layer is not officially supported by NVIDIA Corporation. It is based on meta-nvidia from deprecated [OakLabsInc](https://github.com/OakLabsInc/meta-nvidia). Always refer to NVIDIA's official documentation and support channels for information related to NVIDIA products.

‎meta-nvidia/artwork/loaded_modules.png‎

406 KB
Loading

‎meta-nvidia/conf/layer.conf‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# We have a conf and classes directory, add to BBPATH
2+
BBPATH .= ":${LAYERDIR}"
3+
4+
# We have recipes-* directories, add to BBFILES
5+
BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
6+
${LAYERDIR}/recipes-*/*/*.bbappend"
7+
8+
BBFILE_COLLECTIONS += "nvidia"
9+
BBFILE_PATTERN_nvidia = "^${LAYERDIR}/"
10+
BBFILE_PRIORITY_nvidia = "12"
11+
12+
LICENSE_PATH += " ${LAYERDIR}/custom-licenses"
13+
LAYERSERIES_COMPAT_nvidia = "mickledore scarthgap"

0 commit comments

Comments
 (0)