[BUG] ImageVolume fails to generate container: failed to mkdir on Azure Linux

[acortelyou]

Describe the bug

Using ImageVolume feature in AKS 1.35 on Azure Linux results in Error: failed to generate container: failed to generate spec: failed to mkdir

To Reproduce
Steps to reproduce the behavior:

apiVersion: v1
kind: Pod
metadata:
  name: image-volume-test
spec:
  containers:
  - name: reader
    image: mcr.microsoft.com/azurelinux/busybox:1.36
    command:
    - sh
    - -c
    - ls -la /image-vol && sleep infinity
    volumeMounts:
    - name: image-vol
      mountPath: /image-vol
  volumes:
  - name: image-vol
    image:
      reference: mcr.microsoft.com/azurelinux/busybox:1.36

apiVersion: events.k8s.io/v1
kind: Event
metadata:
  name: image-volume-test.18a600f68681cfb3
note: 'Error: failed to generate container "a90152703ac0baf185870f7bcfbbdbd1ea9d4e10c7167fb3cdeef2225fdab6ee"
  spec: failed to generate spec: failed to mkdir "": mkdir : no such file or directory'
reason: Failed
type: Warning

Expected behavior

Pod lists image volume contents.

total 56
drwxr-xr-x    1 root     root          4096 Apr 13 19:59 .
drwxr-xr-x    1 root     root          4096 Apr 13 19:59 ..
-rw-r--r--    1 root     root         10145 Apr  1 09:08 EULA-Container.txt
lrwxrwxrwx    1 root     root             7 Jul 20  2024 bin -> usr/bin
drwxr-xr-x    2 root     root          4096 Jul 20  2024 dev
drwxr-xr-x    8 root     root          4096 Apr  1 09:59 etc
drwxr-xr-x    2 root     root          4096 Jul 20  2024 home
lrwxrwxrwx    1 root     root             7 Jul 20  2024 lib -> usr/lib
lrwxrwxrwx    1 root     root             7 Jul 20  2024 lib64 -> usr/lib
dr-xr-xr-x    2 root     root          4096 Apr  1 09:59 proc
drwxr-x---    2 root     root          4096 Jul 20  2024 root
lrwxrwxrwx    1 root     root             8 Jul 20  2024 sbin -> usr/sbin
dr-xr-xr-x    2 root     root          4096 Apr  1 09:59 sys
drwxrwxrwt    2 root     root          4096 Jul 20  2024 tmp
drwxr-xr-x   10 root     root          4096 Apr  1 09:59 usr
drwxr-xr-x   10 root     root          4096 Apr  1 09:59 var

Environment (please complete the following information):

aks version: 1.35.1
This cluster is using the latest available version of Kubernetes.
node-image-version: AKSAzureLinux-V3gen2fips-202603.18.1
This node pool is using the latest image version.

Additional context
#5448

[acortelyou]

microsoft/azurelinux#16441

[acortelyou]

microsoft/azurelinux#16652

[flora-taagen]

Thank you @acortelyou! This is a known issue on Azure Linux and will be resolved when we upgrade the containerd version on Azure Linux 3.0. We are currently in the process of upgrading to containerd 2.1.7 which should fix the issue.