Skip to content

[build] Add configurable timeout for apt-get build hook to prevent long hangs#26865

Open
yijingyan2 wants to merge 2 commits intosonic-net:masterfrom
yijingyan2:apt-get-http-timeout
Open

[build] Add configurable timeout for apt-get build hook to prevent long hangs#26865
yijingyan2 wants to merge 2 commits intosonic-net:masterfrom
yijingyan2:apt-get-http-timeout

Conversation

@yijingyan2
Copy link
Copy Markdown
Contributor

Why I did it

Some apt-get operations can hang for hours and result in build failure due to timeout.
This change adds a configurable timeout for apt-get commands executed through build hooks, so builds fail fast and can be retried again instead of hanging for hours.

Work item tracking
  • Microsoft ADO (number only):

How I did it

Apply timeout wrapper in apt-get hook

How to verify it

tested with debug messages

Step 11/27 : RUN apt-get update         &&     apt-get install -y                apt-utils                     build-essential               python3-dev                   python3-yaml
 ---> Running in 0ac8a2f5ace9
�[91mAPT_GET timeout enabled: 60s
�[0m�[91mExecuting: /usr/bin/apt-get -o Acquire::http::Timeout=60 -o Acquire::https::Timeout=60 update
�[0mGet:1 http://packages.trafficmanager.net/snapshot/debian/20260329T001335Z bookworm InRelease [151 kB]

+ sudo LANG=C DEBIAN_FRONTEND=noninteractive chroot ./fsroot-vs apt-get update
APT_GET timeout enabled: 60s
Executing: /usr/bin/apt-get -o Acquire::http::Timeout=60 -o Acquire::https::Timeout=60 update
Hit:1 https://download.docker.com/linux/debian bookworm InRelease

Which release branch to backport (provide reason below if selected)

  • 202305
  • 202311
  • 202405
  • 202411
  • 202505
  • 202511

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: Yijing Yan <yijingyan@microsoft.com>
@yijingyan2 yijingyan2 requested a review from qiluo-msft as a code owner April 17, 2026 05:47
Copilot AI review requested due to automatic review settings April 17, 2026 05:47
@mssonicbld
Copy link
Copy Markdown
Collaborator

/azp run Azure.sonic-buildimage

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a build-time configuration knob to apply APT HTTP(S) Acquire timeouts during SONiC image builds, aiming to reduce long hangs in package operations executed via sonic-build-hooks.

Changes:

  • Extend the apt-get build hook to optionally pass Acquire::http::Timeout / Acquire::https::Timeout based on BUILD_APT_GET_HTTP_TIMEOUT.
  • Persist the new timeout variable into the buildinfo config generated during the build.
  • Introduce a default value for BUILD_APT_GET_HTTP_TIMEOUT and propagate/export it through Makefile.work.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
src/sonic-build-hooks/hooks/apt-get Adds conditional APT Acquire timeout options to hook-invoked apt/apt-get commands.
scripts/generate_buildinfo_config.sh Writes BUILD_APT_GET_HTTP_TIMEOUT into the generated buildinfo config for hook consumption.
rules/config Defines default BUILD_APT_GET_HTTP_TIMEOUT value.
Makefile.work Passes and exports BUILD_APT_GET_HTTP_TIMEOUT into the buildinfo generation and slave build invocation.

Comment thread rules/config Outdated
Comment thread Makefile.work
Comment on lines 650 to 652
PIP_HTTP_TIMEOUT=$(PIP_HTTP_TIMEOUT) \
BUILD_APT_GET_HTTP_TIMEOUT=$(BUILD_APT_GET_HTTP_TIMEOUT) \
BUILD_PROCESS_TIMEOUT=$(BUILD_PROCESS_TIMEOUT) \
Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line’s indentation uses tabs and is noticeably inconsistent with surrounding lines in the SONIC_BUILD_INSTRUCTION block. While it won’t change behavior, it makes the Makefile harder to read and can lead to accidental formatting issues in future edits; please align it with the neighboring lines’ indentation.

Copilot uses AI. Check for mistakes.
Comment on lines +13 to +16
APT_OPTIONS=
if [[ "${BUILD_APT_GET_HTTP_TIMEOUT}" =~ ^[1-9][0-9]*$ ]]; then
APT_OPTIONS="-o Acquire::http::Timeout=${BUILD_APT_GET_HTTP_TIMEOUT} -o Acquire::https::Timeout=${BUILD_APT_GET_HTTP_TIMEOUT}"
fi
Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description/title imply a hard timeout for apt-get to prevent long hangs, but this implementation only sets APT Acquire HTTP/HTTPS timeouts. apt-get can still hang for non-network reasons (e.g., waiting on apt/dpkg locks or postinst scripts). If the goal is a wall-clock limit, consider wrapping the apt invocation with a timeout-style guard (or update the PR description/comments to clarify this is only a network Acquire timeout).

Copilot uses AI. Check for mistakes.
Copilot AI review requested due to automatic review settings April 17, 2026 05:54
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Comment thread scripts/generate_buildinfo_config.sh
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Yijing Yan <yijingyan@microsoft.com>
@yijingyan2 yijingyan2 force-pushed the apt-get-http-timeout branch from db068b7 to 1872596 Compare April 17, 2026 05:58
@mssonicbld
Copy link
Copy Markdown
Collaborator

/azp run Azure.sonic-buildimage

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants