Skip to content

Commit e9d0e26

Browse files
mergennachinclaude
andcommitted
Centralize torch-family version pins in torch_pin.py
torch_pin.py is now the single source of truth for torch + the three domain libraries (vision/audio/codec). It exposes a CHANNEL field (nightly/test/release), the four version constants, NIGHTLY_VERSION, and helpers — torch_spec() / torchaudio_spec() / torchcodec_spec() / torchvision_spec() emit the right pip spec, torch_index_url_base() returns the right wheel index, and torch_branch() / torchaudio_branch() / torchvision_branch() derive the upstream release/M.N branch from each package's version. Every consumer — install_requirements.py, the two install_pytorch.sh / utils.sh shell helpers, test_model_e2e.sh, test_wheel_package_qnn.sh, the moshi/mimi install_requirements.sh, the update_pytorch_pin.py script, and the weekly bump workflow — reads through these helpers instead of re-encoding the version strings. Switching to a release candidate is now a one-line change (CHANNEL = "test") plus bumping the four version constants. The header in torch_pin.py walks through the procedure. update_pytorch_pin.py imports CHANNEL / NIGHTLY_VERSION / torch_branch directly (no more regex parsing of the file). For nightly it pins to an immutable SHA looked up by date; for test/release it writes torch_branch() (e.g. "release/2.12") into .ci/docker/ci_commit_pins/pytorch.txt so git checkout follows cherry-picks as they land. The weekly-pytorch-pin-bump workflow is guarded on CHANNEL == "nightly" and uses an in-place re.sub on NIGHTLY_VERSION (the previous `printf '...' > torch_pin.py` would have clobbered the new constants and helpers). test/test_torch_pin.py covers all three channels, all four specs, and the release/M.N branch derivation. Co-authored-by: Claude <noreply@anthropic.com>
1 parent d7f8718 commit e9d0e26

10 files changed

Lines changed: 252 additions & 99 deletions

File tree

.ci/docker/common/install_pytorch.sh

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -36,10 +36,20 @@ install_pytorch_and_domains() {
3636
conda_run python setup.py bdist_wheel
3737
pip_install "$(echo dist/*.whl)"
3838

39-
# Grab the pinned audio and vision commits from PyTorch
40-
TORCHAUDIO_VERSION=$(cat .github/ci_commit_pins/audio.txt)
39+
# For nightly, defer to PyTorch's own pinned audio/vision commits.
40+
# For test/release, use the release/M.N branch derived from torch_pin.py
41+
# (torchaudio_branch() / torchvision_branch()).
42+
# We are inside the cloned pytorch/ subdir at this point (see `pushd pytorch`
43+
# above), so `cd ..` lands in the executorch repo root where torch_pin.py lives.
44+
TORCH_CHANNEL=$(cd .. && python -c "from torch_pin import CHANNEL; print(CHANNEL)")
45+
if [ "$TORCH_CHANNEL" = "nightly" ]; then
46+
TORCHAUDIO_VERSION=$(cat .github/ci_commit_pins/audio.txt)
47+
TORCHVISION_VERSION=$(cat .github/ci_commit_pins/vision.txt)
48+
else
49+
TORCHAUDIO_VERSION=$(cd .. && python -c "from torch_pin import torchaudio_branch; print(torchaudio_branch())")
50+
TORCHVISION_VERSION=$(cd .. && python -c "from torch_pin import torchvision_branch; print(torchvision_branch())")
51+
fi
4152
export TORCHAUDIO_VERSION
42-
TORCHVISION_VERSION=$(cat .github/ci_commit_pins/vision.txt)
4353
export TORCHVISION_VERSION
4454

4555
install_domains

.ci/scripts/test_model_e2e.sh

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -260,7 +260,10 @@ if [ "$AUDIO_URL" != "" ]; then
260260
elif [[ "$MODEL_NAME" == *whisper* ]] || [ "$MODEL_NAME" = "voxtral_realtime" ]; then
261261
conda install -y -c conda-forge "ffmpeg<8"
262262
pip install datasets soundfile
263-
pip install torchcodec==0.12.0.dev20260409 --extra-index-url https://download.pytorch.org/whl/nightly/cpu
263+
# We pushd'd into EXECUTORCH_ROOT above, so torch_pin is importable here.
264+
TORCHCODEC_PKG=$(python -c "from torch_pin import torchcodec_spec; print(torchcodec_spec())")
265+
TORCHCODEC_INDEX=$(python -c "from torch_pin import torch_index_url_base; print(torch_index_url_base())")
266+
pip install "$TORCHCODEC_PKG" --extra-index-url "${TORCHCODEC_INDEX}/cpu"
264267
python -c "from datasets import load_dataset;import soundfile as sf;sample = load_dataset('distil-whisper/librispeech_long', 'clean', split='validation')[0]['audio'];sf.write('${MODEL_DIR}/$AUDIO_FILE', sample['array'][:sample['sampling_rate']*30], sample['sampling_rate'])"
265268
fi
266269

.ci/scripts/test_wheel_package_qnn.sh

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -150,25 +150,26 @@ run_core_tests () {
150150
echo "=== [$LABEL] Installing wheel & deps ==="
151151
"$PIPBIN" install --upgrade pip
152152
"$PIPBIN" install "$WHEEL_FILE"
153-
TORCH_VERSION=$(
153+
# runpy.run_path uses a relative path, so the caller must run this script
154+
# from the executorch repo root (where torch_pin.py lives).
155+
TORCH_SPEC=$(
154156
"$PYBIN" - <<'PY'
155157
import runpy
156158
module_vars = runpy.run_path("torch_pin.py")
157-
print(module_vars["TORCH_VERSION"])
159+
print(module_vars["torch_spec"]())
158160
PY
159161
)
160-
161-
NIGHTLY_VERSION=$(
162+
TORCH_INDEX=$(
162163
"$PYBIN" - <<'PY'
163164
import runpy
164165
module_vars = runpy.run_path("torch_pin.py")
165-
print(module_vars["NIGHTLY_VERSION"])
166+
print(module_vars["torch_index_url_base"]())
166167
PY
167168
)
168-
echo "=== [$LABEL] Install torch==${TORCH_VERSION}.${NIGHTLY_VERSION} ==="
169+
echo "=== [$LABEL] Install $TORCH_SPEC from ${TORCH_INDEX}/cpu ==="
169170

170171
# Install torchao based on the pinned PyTorch version
171-
"$PIPBIN" install torch=="${TORCH_VERSION}.${NIGHTLY_VERSION}" --index-url "https://download.pytorch.org/whl/nightly/cpu"
172+
"$PIPBIN" install "$TORCH_SPEC" --index-url "${TORCH_INDEX}/cpu"
172173
"$PIPBIN" install wheel
173174

174175
# Install torchao based on the pinned commit from third-party/ao submodule

.ci/scripts/utils.sh

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -140,10 +140,20 @@ install_pytorch_and_domains() {
140140
fi
141141

142142
dedupe_macos_loader_path_rpaths
143-
# Grab the pinned audio and vision commits from PyTorch
144-
TORCHAUDIO_VERSION=$(cat .github/ci_commit_pins/audio.txt)
143+
# For nightly, defer to PyTorch's own pinned audio/vision commits.
144+
# For test/release, use the release/M.N branch derived from torch_pin.py
145+
# (torchaudio_branch() / torchvision_branch()).
146+
# We are inside the cloned pytorch/ subdir at this point (see `pushd pytorch`
147+
# above), so `cd ..` lands in the executorch repo root where torch_pin.py lives.
148+
TORCH_CHANNEL=$(cd .. && python -c "from torch_pin import CHANNEL; print(CHANNEL)")
149+
if [ "$TORCH_CHANNEL" = "nightly" ]; then
150+
TORCHAUDIO_VERSION=$(cat .github/ci_commit_pins/audio.txt)
151+
TORCHVISION_VERSION=$(cat .github/ci_commit_pins/vision.txt)
152+
else
153+
TORCHAUDIO_VERSION=$(cd .. && python -c "from torch_pin import torchaudio_branch; print(torchaudio_branch())")
154+
TORCHVISION_VERSION=$(cd .. && python -c "from torch_pin import torchvision_branch; print(torchvision_branch())")
155+
fi
145156
export TORCHAUDIO_VERSION
146-
TORCHVISION_VERSION=$(cat .github/ci_commit_pins/vision.txt)
147157
export TORCHVISION_VERSION
148158

149159
install_domains

.github/scripts/update_pytorch_pin.py

Lines changed: 39 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,12 @@
88
import urllib.request
99
from pathlib import Path
1010

11+
# torch_pin.py lives at the repo root. Locate it relative to this script so
12+
# the import works regardless of where the script is invoked from.
13+
_REPO_ROOT = Path(__file__).resolve().parents[2]
14+
sys.path.insert(0, str(_REPO_ROOT))
15+
from torch_pin import CHANNEL, NIGHTLY_VERSION, torch_branch
16+
1117

1218
def parse_nightly_version(nightly_version):
1319
"""
@@ -27,23 +33,6 @@ def parse_nightly_version(nightly_version):
2733
return f"{year}-{month}-{day}"
2834

2935

30-
def get_torch_nightly_version():
31-
"""
32-
Read NIGHTLY_VERSION from torch_pin.py.
33-
34-
Returns:
35-
NIGHTLY_VERSION string
36-
"""
37-
with open("torch_pin.py", "r") as f:
38-
content = f.read()
39-
40-
match = re.search(r'NIGHTLY_VERSION\s*=\s*["\']([^"\']+)["\']', content)
41-
if not match:
42-
raise ValueError("Could not find NIGHTLY_VERSION in torch_pin.py")
43-
44-
return match.group(1)
45-
46-
4736
def get_commit_hash_for_nightly(date_str):
4837
"""
4938
Fetch commit hash from PyTorch nightly branch for a given date.
@@ -91,17 +80,17 @@ def extract_hash_from_title(title):
9180
return match.group(1)
9281

9382

94-
def update_pytorch_pin(commit_hash):
83+
def update_pytorch_pin(ref):
9584
"""
96-
Update .ci/docker/ci_commit_pins/pytorch.txt with the new commit hash.
85+
Update .ci/docker/ci_commit_pins/pytorch.txt with the new ref.
9786
9887
Args:
99-
commit_hash: Commit hash to write
88+
ref: Either a commit SHA (nightly) or a branch name (test/release).
10089
"""
10190
pin_file = ".ci/docker/ci_commit_pins/pytorch.txt"
10291
with open(pin_file, "w") as f:
103-
f.write(f"{commit_hash}\n")
104-
print(f"Updated {pin_file} with commit hash: {commit_hash}")
92+
f.write(f"{ref}\n")
93+
print(f"Updated {pin_file} with ref: {ref}")
10594

10695

10796
def should_skip_file(filename):
@@ -118,18 +107,20 @@ def should_skip_file(filename):
118107
return filename in skip_files
119108

120109

121-
def fetch_file_content(commit_hash, file_path):
110+
def fetch_file_content(ref, file_path):
122111
"""
123112
Fetch file content from GitHub API.
124113
125114
Args:
126-
commit_hash: Commit hash to fetch from
115+
ref: Commit SHA or branch name to fetch from
127116
file_path: File path in the repository
128117
129118
Returns:
130119
File content as bytes
131120
"""
132-
api_url = f"https://api.github.com/repos/pytorch/pytorch/contents/{file_path}?ref={commit_hash}"
121+
api_url = (
122+
f"https://api.github.com/repos/pytorch/pytorch/contents/{file_path}?ref={ref}"
123+
)
133124

134125
req = urllib.request.Request(api_url)
135126
req.add_header("Accept", "application/vnd.github.v3+json")
@@ -146,15 +137,15 @@ def fetch_file_content(commit_hash, file_path):
146137
raise
147138

148139

149-
def sync_directory(et_dir, pt_path, commit_hash):
140+
def sync_directory(et_dir, pt_path, ref):
150141
"""
151142
Sync files from PyTorch to ExecuTorch using GitHub API.
152143
Only syncs files that already exist in ExecuTorch - does not add new files.
153144
154145
Args:
155146
et_dir: ExecuTorch directory path
156147
pt_path: PyTorch directory path in the repository (e.g., "c10")
157-
commit_hash: Commit hash to fetch from
148+
ref: Commit SHA or branch name to fetch from
158149
159150
Returns:
160151
Number of files grafted
@@ -181,12 +172,12 @@ def sync_directory(et_dir, pt_path, commit_hash):
181172

182173
# Fetch content from PyTorch and compare
183174
try:
184-
pt_content = fetch_file_content(commit_hash, pt_file_path)
175+
pt_content = fetch_file_content(ref, pt_file_path)
185176
et_content = et_file.read_bytes()
186177

187178
if pt_content != et_content:
188179
print(f"⚠️ Difference detected in {rel_path}")
189-
print(f"📋 Grafting from PyTorch commit {commit_hash}...")
180+
print(f"📋 Grafting from PyTorch ref {ref}...")
190181

191182
et_file.write_bytes(pt_content)
192183
print(f"✅ Grafted {et_file}")
@@ -201,12 +192,12 @@ def sync_directory(et_dir, pt_path, commit_hash):
201192
return files_grafted
202193

203194

204-
def sync_c10_directories(commit_hash):
195+
def sync_c10_directories(ref):
205196
"""
206197
Sync c10 and torch/headeronly directories from PyTorch to ExecuTorch using GitHub API.
207198
208199
Args:
209-
commit_hash: PyTorch commit hash to sync from
200+
ref: PyTorch commit SHA or branch name to sync from
210201
211202
Returns:
212203
Total number of files grafted
@@ -231,7 +222,7 @@ def sync_c10_directories(commit_hash):
231222

232223
total_grafted = 0
233224
for et_dir, pt_path in dir_pairs:
234-
files_grafted = sync_directory(et_dir, pt_path, commit_hash)
225+
files_grafted = sync_directory(et_dir, pt_path, ref)
235226
total_grafted += files_grafted
236227

237228
if total_grafted > 0:
@@ -244,27 +235,26 @@ def sync_c10_directories(commit_hash):
244235

245236
def main():
246237
try:
247-
# Read NIGHTLY_VERSION from torch_pin.py
248-
nightly_version = get_torch_nightly_version()
249-
print(f"Found NIGHTLY_VERSION: {nightly_version}")
250-
251-
# Parse to date string
252-
date_str = parse_nightly_version(nightly_version)
253-
print(f"Parsed date: {date_str}")
254-
255-
# Fetch commit hash from PyTorch nightly branch
256-
commit_hash = get_commit_hash_for_nightly(date_str)
257-
print(f"Found commit hash: {commit_hash}")
238+
print(f"CHANNEL: {CHANNEL}")
239+
if CHANNEL == "nightly":
240+
# Nightly pins to an immutable SHA looked up by date.
241+
print(f"Found NIGHTLY_VERSION: {NIGHTLY_VERSION}")
242+
date_str = parse_nightly_version(NIGHTLY_VERSION)
243+
print(f"Parsed date: {date_str}")
244+
pin_ref = get_commit_hash_for_nightly(date_str)
245+
else:
246+
# For test/release, pin to the branch name so CI picks up
247+
# cherry-picks / security patches as they land on the branch.
248+
pin_ref = torch_branch()
249+
print(f"Pin ref: {pin_ref}")
258250

259251
# Update the pin file
260-
update_pytorch_pin(commit_hash)
252+
update_pytorch_pin(pin_ref)
261253

262-
# Sync c10 directories from PyTorch
263-
sync_c10_directories(commit_hash)
254+
# Sync c10 directories from PyTorch (ref param accepts branches too)
255+
sync_c10_directories(pin_ref)
264256

265-
print(
266-
"\n✅ Successfully updated PyTorch commit pin and synced c10 directories!"
267-
)
257+
print("\n✅ Successfully updated PyTorch pin and synced c10 directories!")
268258

269259
except Exception as e:
270260
print(f"Error: {e}", file=sys.stderr)

.github/workflows/weekly-pytorch-pin-bump.yml

Lines changed: 24 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -22,29 +22,44 @@ jobs:
2222
with:
2323
python-version: '3.11'
2424

25+
- name: Check torch_pin channel
26+
id: channel
27+
run: |
28+
CHANNEL=$(python -c "from torch_pin import CHANNEL; print(CHANNEL)")
29+
echo "channel=${CHANNEL}" >> "$GITHUB_OUTPUT"
30+
if [ "${CHANNEL}" != "nightly" ]; then
31+
echo "torch_pin.py CHANNEL is '${CHANNEL}'; weekly nightly bump only runs when CHANNEL == 'nightly'."
32+
fi
33+
2534
- name: Determine nightly version
35+
if: steps.channel.outputs.channel == 'nightly'
2636
id: nightly
2737
run: |
2838
NIGHTLY_DATE=$(date -u -d 'yesterday' '+%Y%m%d')
2939
NIGHTLY_VERSION="dev${NIGHTLY_DATE}"
3040
echo "version=${NIGHTLY_VERSION}" >> "$GITHUB_OUTPUT"
3141
32-
- name: Read current TORCH_VERSION
33-
id: torch
34-
run: |
35-
TORCH_VERSION=$(python -c "exec(open('torch_pin.py').read()); print(TORCH_VERSION)")
36-
echo "version=${TORCH_VERSION}" >> "$GITHUB_OUTPUT"
37-
3842
- name: Update torch_pin.py with new NIGHTLY_VERSION
43+
if: steps.channel.outputs.channel == 'nightly'
3944
run: |
40-
printf 'TORCH_VERSION = "%s"\nNIGHTLY_VERSION = "%s"\n' \
41-
"${{ steps.torch.outputs.version }}" \
42-
"${{ steps.nightly.outputs.version }}" > torch_pin.py
45+
python -c "
46+
import re, pathlib
47+
p = pathlib.Path('torch_pin.py')
48+
p.write_text(re.sub(
49+
r'^NIGHTLY_VERSION\s*=\s*\".*\"$',
50+
'NIGHTLY_VERSION = \"${{ steps.nightly.outputs.version }}\"',
51+
p.read_text(),
52+
count=1,
53+
flags=re.MULTILINE,
54+
))
55+
"
4356
4457
- name: Run pin bump script
58+
if: steps.channel.outputs.channel == 'nightly'
4559
run: python .github/scripts/update_pytorch_pin.py
4660

4761
- name: Create branch and PR
62+
if: steps.channel.outputs.channel == 'nightly'
4863
env:
4964
GH_TOKEN: ${{ secrets.UPDATEBOT_TOKEN }}
5065
run: |

examples/models/moshi/mimi/install_requirements.sh

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,16 @@
77

88
set -x
99

10+
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
11+
# torch_pin lives at the executorch repo root.
12+
cd "$SCRIPT_DIR/../../../.."
13+
14+
TORCHCODEC_PKG=$(python -c "from torch_pin import torchcodec_spec; print(torchcodec_spec())")
15+
TORCHCODEC_INDEX=$(python -c "from torch_pin import torch_index_url_base; print(torch_index_url_base())")
16+
1017
sudo apt install ffmpeg -y
11-
pip install torchcodec==0.12.0.dev20260409 --extra-index-url https://download.pytorch.org/whl/nightly/cpu
18+
pip install "$TORCHCODEC_PKG" --extra-index-url "${TORCHCODEC_INDEX}/cpu"
1219
pip install moshi==0.2.11
1320
pip install bitsandbytes soundfile einops
1421
# Run llama2/install requirements for torchao deps
15-
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
1622
bash "$SCRIPT_DIR"/../../llama/install_requirements.sh

0 commit comments

Comments
 (0)