Transient qemu error also causes build-containers job to hang

[danbarr]

I noticed a build-containers CI job on a PR was hung up for 2+ hours and not progressing. The last message in the "Build and Push Docker Image" step was:

#18 [linux/arm64 builder 5/5] RUN npm install --save @notionhq/notion-mcp-server@1.9.0
#18 24.34 qemu: uncaught target signal 4 (Illegal instruction) - core dumped

A re-run succeeded, so this was a transient error. But I would have expected it to fail out the job immediately, not cause it to hang.

Job: https://github.com/stacklok/dockyard/actions/runs/19502404593/job/55819914543?pr=149

[danbarr]

Diagnosis

Based on the workflow configuration and the symptoms described, here's what's happening:

The Problem

When QEMU crashes during ARM64 emulation with an "Illegal instruction" error, the Docker buildx process doesn't properly detect this as a failure. Instead, it enters a hung state waiting for the crashed process to complete or respond, which never happens.

Looking at the build-containers.yml workflow, the "Build and push Docker image" step has:

• No timeout configured on the step itself
• No job-level timeout configured
• Multi-architecture build with linux/amd64,linux/arm64

Why It Hung Instead of Failing

1. QEMU crash during npm install: When QEMU emulating ARM64 crashes with SIGILL (signal 4 - Illegal instruction), it can leave the npm process in a zombie or hung state

2. Docker buildx doesn't detect the failure: The build-push-action waits for the Docker build to complete, but when QEMU crashes unexpectedly, the build container may not properly propagate the failure signal

3. No timeout protection: Without timeouts, the job can hang for up to GitHub Actions' default maximum (6 hours for public repos), which is why it ran for 2+ hours before being manually cancelled

Root Cause

This is a known transient issue with QEMU emulation in CI environments, particularly for ARM64 builds. The illegal instruction error suggests:

• A bug in QEMU's instruction translation
• Memory corruption during emulation
• Race condition in the emulated environment

The fact that a re-run succeeded confirms this is a transient QEMU issue, not a problem with the actual build or dependencies.

Recommended Solutions

1. Add Step-Level Timeout (Primary Fix)

Add a timeout to the Docker build step:

- name: Build and push Docker image
  id: build
  timeout-minutes: 60  # Add timeout
  uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6

2. Add Job-Level Timeout (Backup Protection)

Add a timeout at the job level:

build-containers:
  needs: [discover-configs, verify-provenance, mcp-security-scan]
  runs-on: ubuntu-latest
  timeout-minutes: 90  # Add job timeout

3. Consider Retry Logic

For transient QEMU failures, you could add retry logic using a community action, though this increases complexity.

The step timeout (solution #1) is the most important - it will cause the build to fail fast when it hangs, rather than waiting hours. A 60-minute timeout is reasonable for most container builds, but can be adjusted based on typical build times.