IONOS(ci): migrate build to cached matrix pipeline#226
Conversation
There was a problem hiding this comment.
Pull request overview
This PR refactors the hidrive-next-build GitHub Actions workflow into a cached, multi-job matrix pipeline that avoids rebuilding unchanged components by using JFrog + GitHub Actions cache signals computed up-front.
Changes:
- Introduces
prepare-matrixto compute cache keys, generate the apps matrix, and decide which apps to build vs restore. - Adds a cached
build-custom-npmsjob and a parallelbuild-appsmatrix job that uploads per-app artifacts to JFrog. - Updates
hidrive-next-buildto restore artifacts and build Nextcloud core only.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| .github/workflows/hidrive-next-build.yml | Replaces monolithic build with prepare/build/restore pipeline + JFrog/GHA caching and workflow_dispatch inputs. |
| .github/scripts/detect-app-cache.sh | New script to compute app SHAs and decide build vs restore across JFrog and GitHub cache. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| echo " Running: jf rt s \"$JFROG_PATH\"" | ||
| SEARCH_OUTPUT=$(jf rt s "$JFROG_PATH" 2>&1) | ||
| SEARCH_EXIT_CODE=$? | ||
|
|
||
| echo " Search exit code: $SEARCH_EXIT_CODE" |
There was a problem hiding this comment.
With set -e enabled, SEARCH_OUTPUT=$(jf rt s ... ) will cause the script to exit immediately when jf rt s returns a non-zero status, so the intended fallback handling (checking exit code / falling back to GitHub cache) won't run. Wrap the JFrog search in an if ...; then block or temporarily disable -e while capturing output and exit status.
| - name: Compute effective cache version | ||
| id: compute_cache_version | ||
| run: | | ||
| EFFECTIVE_VERSION="${{ env.CACHE_VERSION }}${{ github.event.inputs.cache_version_suffix && format('-{0}', github.event.inputs.cache_version_suffix) || '' }}" |
There was a problem hiding this comment.
cache_version_suffix is user-controlled (workflow_dispatch) and is concatenated into cache keys and JFrog paths. If it contains /, spaces, or shell-special characters it can generate invalid cache keys and unexpected JFrog paths/names. Consider validating/sanitizing this input (e.g., restrict to [A-Za-z0-9._-] and fail fast otherwise) before using it in EFFECTIVE_VERSION.
| EFFECTIVE_VERSION="${{ env.CACHE_VERSION }}${{ github.event.inputs.cache_version_suffix && format('-{0}', github.event.inputs.cache_version_suffix) || '' }}" | |
| CACHE_VERSION_SUFFIX="${{ github.event.inputs.cache_version_suffix || '' }}" | |
| if [ -n "$CACHE_VERSION_SUFFIX" ] && [[ ! "$CACHE_VERSION_SUFFIX" =~ ^[A-Za-z0-9._-]+$ ]]; then | |
| echo "Invalid cache_version_suffix: only [A-Za-z0-9._-] are allowed" >&2 | |
| exit 1 | |
| fi | |
| EFFECTIVE_VERSION="${{ env.CACHE_VERSION }}" | |
| if [ -n "$CACHE_VERSION_SUFFIX" ]; then | |
| EFFECTIVE_VERSION="${EFFECTIVE_VERSION}-${CACHE_VERSION_SUFFIX}" | |
| fi |
| if [ "$SOURCE" == "jfrog" ]; then | ||
| JFROG_PATH=$(echo "$app_json" | jq -r '.jfrog_path') | ||
| ARCHIVE_NAME="${APP_NAME}-${APP_SHA}.tar.gz" | ||
| echo "Downloading from JFrog: $JFROG_PATH" | ||
| jf rt download "$JFROG_PATH" "$ARCHIVE_NAME" --flat=true | ||
| mkdir -p "$(dirname "$APP_PATH")" | ||
| tar -xzf "$ARCHIVE_NAME" -C "$(dirname "$APP_PATH")" | ||
| rm -f "$ARCHIVE_NAME" |
There was a problem hiding this comment.
When restoring cached apps from JFrog, the workflow reconstructs the archive name as "${APP_NAME}-${APP_SHA}.tar.gz" using the full SHA. However uploads use an 8-char suffix (and sometimes a "-cnpms-" compound suffix), so this filename will not match the artifact stored in JFrog and restores will fail. Prefer passing/using the exact archive_name from detect-app-cache.sh (or deriving it from jfrog_path via basename) instead of rebuilding it from the full SHA.
| echo "❌ Cannot restore $APP_NAME from GitHub cache within a shell step." | ||
| echo " Cache key: $CACHE_KEY" | ||
| echo " GitHub Actions cache requires 'actions/cache/restore@v4' as a dedicated workflow step." | ||
| exit 1 |
There was a problem hiding this comment.
detect-app-cache.sh can mark cached apps with source: github-cache, but this job exits with an error for that case. As-is, any run without JFrog access that finds GitHub cache hits will fail during restore. Either implement GitHub cache restore via actions/cache/restore@v4 (likely as a matrix/loop of dedicated steps) or change the detection logic so github-cache entries are treated as needs build instead of apps_to_restore.
| echo "❌ Cannot restore $APP_NAME from GitHub cache within a shell step." | |
| echo " Cache key: $CACHE_KEY" | |
| echo " GitHub Actions cache requires 'actions/cache/restore@v4' as a dedicated workflow step." | |
| exit 1 | |
| echo "⚠️ Skipping $APP_NAME from GitHub cache in this shell-based restore step." | |
| echo " Cache key: $CACHE_KEY" | |
| echo " GitHub Actions cache requires 'actions/cache/restore@v4' as a dedicated workflow step." | |
| echo " Leaving this app to be handled by upstream cache-detection/build logic." | |
| continue |
| - name: Setup JFrog CLI | ||
| uses: jfrog/setup-jfrog-cli@7c95feb32008765e1b4e626b078dfd897c4340ad # v4.4.1 | ||
| env: | ||
| JF_URL: ${{ secrets.JF_ARTIFACTORY_URL }} | ||
| JF_USER: ${{ secrets.JF_ARTIFACTORY_USER }} | ||
| JF_ACCESS_TOKEN: ${{ secrets.JF_ACCESS_TOKEN }} | ||
|
|
||
| - name: Restore custom-npms from JFrog | ||
| if: steps.app-config.outputs.needs-custom-npms == 'true' | ||
| run: | | ||
| CACHE_KEY="${{ needs.prepare-matrix.outputs.custom_npms_cache_key }}" | ||
| EFFECTIVE_VERSION="${{ needs.prepare-matrix.outputs.effective_cache_version }}" | ||
| ARCHIVE_NAME="custom-npms-${CACHE_KEY}.tar.gz" | ||
| JFROG_PATH="${{ env.ARTIFACTORY_REPOSITORY_SNAPSHOT }}/custom-npms/${EFFECTIVE_VERSION}/${ARCHIVE_NAME}" | ||
|
|
||
| echo "Downloading custom-npms from JFrog: $JFROG_PATH" | ||
| if jf rt download "$JFROG_PATH" "$ARCHIVE_NAME" --flat=true; then | ||
| tar -xzf "$ARCHIVE_NAME" | ||
| rm -f "$ARCHIVE_NAME" | ||
| echo "✅ Custom-npms restored from JFrog" | ||
| else | ||
| echo "❌ Failed to restore custom-npms from JFrog" | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
build-apps always runs jfrog/setup-jfrog-cli and (for some apps) downloads custom-npms from JFrog. If secrets/JFrog are unavailable (e.g., forked PRs) or build-custom-npms ran but skipped JFrog upload, this will fail even though the custom-npms are available via the GitHub cache. Gate JFrog setup/download on credential availability and add a GitHub cache restore (or artifact download) path for custom-npms when JFrog isn't usable.
| - name: Restore custom-npms from JFrog | ||
| run: | | ||
| CACHE_KEY="${{ needs.prepare-matrix.outputs.custom_npms_cache_key }}" | ||
| EFFECTIVE_VERSION="${{ needs.prepare-matrix.outputs.effective_cache_version }}" | ||
| ARCHIVE_NAME="custom-npms-${CACHE_KEY}.tar.gz" | ||
| JFROG_PATH="${{ env.ARTIFACTORY_REPOSITORY_SNAPSHOT }}/custom-npms/${EFFECTIVE_VERSION}/${ARCHIVE_NAME}" | ||
|
|
||
| echo "Downloading custom-npms from JFrog: $JFROG_PATH" | ||
| jf rt download "$JFROG_PATH" "$ARCHIVE_NAME" --flat=true | ||
| tar -xzf "$ARCHIVE_NAME" | ||
| rm -f "$ARCHIVE_NAME" | ||
| echo "✅ Custom-npms restored" |
There was a problem hiding this comment.
This job always attempts to restore custom-npms from JFrog, but earlier steps explicitly allow skipping JFrog upload when credentials are missing. In that case, build-custom-npms may still have produced the dependency and saved it to the GitHub cache, yet this restore step will fail. Consider restoring from GitHub Actions cache when available (or downloading an artifact from build-custom-npms) and only using JFrog when credentials exist.
printminion-co
left a comment
There was a problem hiding this comment.
Iteration 1: requesting changes — two actionable bugs and three minor issues found.
Bugs (must fix):
-
.github/scripts/detect-app-cache.sh, lines 145–157 — incomplete JSON when app directory is missing or is not a git repo
When theAPP_PATHdirectory does not exist orgit rev-parse HEADreturns empty, the script does acontinueafter adding{name, sha: "unknown"}toapps_to_build— with noarchive_nameorjfrog_pathfields. Inhidrive-next-buildthe "Restore newly built apps" loop then does:ARCHIVE_NAME=$(echo "$app_json" | jq -r '.archive_name') # → "null" JFROG_PATH=$(echo "$app_json" | jq -r '.jfrog_path') # → "null" jf rt download "null" "null" --flat=true # obscure failure
The fix is to
exit 1immediately with a clear error message in these two early-exit branches instead of continuing, so the problem is surfaced atprepare-matrixrather than failing mysteriously inhidrive-next-build. -
.github/workflows/hidrive-next-build.yml,build-appsjob —actions/cache/saveruns before the JFrog upload
Step order is: build → compute cache key →actions/cache/save→Upload to JFrog. If the JFrog upload fails (network blip, quota, etc.) the job exits 1, but GitHub Actions cache already has the artifact. On the next rundetect-app-cache.shfinds the artifact in GitHub cache only (JFrog search misses), adds it toapps_to_restorewithsource: github-cache, andhidrive-next-buildexits 1 with "Cannot restore from GitHub cache within a shell step." The pipeline is stuck and requires a manualworkflow_dispatchwithforce_rebuild=trueto escape. Fix: move theactions/cache/savestep to run after the JFrog upload step, or make it conditional on JFrog upload success (e.g., combine into one step with proper error handling so both succeed or both are skipped).
Minor issues (should fix):
-
.github/scripts/detect-app-cache.sh, line 254 —CACHE_EXISTSis dead code
The variable is initialised to"false"on line 254 and set to"true"on line 267, but is never read after either assignment. Remove both assignments. -
.github/workflows/hidrive-next-build.yml, line 336 —build-appshas no JFrog credential guard
prepare-matrixandbuild-custom-npmsboth have an explicit "Check JFrog credentials" step that gates the JFrog setup.build-appscallsjfrog/setup-jfrog-cliunconditionally and the "Upload to JFrog" step also has no guard. This is inconsistent with the pattern established elsewhere in the same file. Even if JFrog is always available in this environment, add a credential guard (or an explicit comment explaining why the guard is intentionally omitted) for consistency and resilience. -
.github/workflows/hidrive-next-build.yml— job name typos- Line 733:
hidirve-next-artifact-to-ghcr_io→hidrive-next-artifact-to-ghcr_io - Line 790:
trigger-remote-dev-worflow→trigger-remote-dev-workflow
These become part of the public job ID referenced in status badges and API calls.
- Line 733:
printminion-co
left a comment
There was a problem hiding this comment.
Iteration 2 review — three bugs remain after the iteration 1 fixes.
1. hidrive-next-build — Restore custom-npms from JFrog runs unconditionally even when JFrog CLI is not installed
Setup JFrog CLI (~line 503) is correctly gated on steps.jfrog-creds.outputs.available == 'true', but the very next step Restore custom-npms from JFrog (~line 514) has no if: guard at all. When jfrog-creds is false, jf is not on PATH and the step fails with command not found, blocking the entire final build job.
Fix — add to the step:
if: steps.jfrog-creds.outputs.available == 'true'Note: the Restore all apps from JFrog and GitHub cache step also calls jf inside a shell loop (for has_apps_to_build path, ~line 574) without an outer if: guard on jfrog-creds. Same class of bug.
2. build-apps — Restore custom-npms from JFrog is not guarded by jfrog-creds
The step is correctly gated on steps.app-config.outputs.needs-custom-npms == 'true', but contains jf rt download with no check that JFrog CLI is actually installed. If jfrog-creds.outputs.available == 'false', jf is absent and this step fails.
Fix:
if: steps.app-config.outputs.needs-custom-npms == 'true' && steps.jfrog-creds.outputs.available == 'true'3. build-custom-npms — Save custom-npms to GitHub Actions cache still runs BEFORE JFrog upload
The analogous issue was fixed in build-apps (iteration 1), but the same ordering problem persists in build-custom-npms (PR HEAD lines ~222–282):
Save custom-npms to GitHub Actions cache ← ~line 222
Check JFrog credentials ← ~line 228
Upload custom-npms to JFrog ← ~line 252
If the JFrog upload fails, the GH Actions cache already has the artifact. On the next run hidrive-next-build exclusively downloads from JFrog — so Restore custom-npms from JFrog will keep failing because the JFrog artifact was never written. The GH cache hit is a false positive.
Fix: move Save custom-npms to GitHub Actions cache to after Upload custom-npms to JFrog, matching the ordering already applied in build-apps.
Not an issue — CNPMS_SHORT extraction is consistent
Both detect-app-cache.sh and the build-apps Compute app cache key step use ##*- then :0:8 on the same custom_npms_cache_key output. The extraction paths are identical and will always produce the same 8-char prefix.
printminion-co
left a comment
There was a problem hiding this comment.
Iteration 3 Review — Two Blocking Issues Remain
All prior fixes are confirmed in place. However two blocking issues remain that were not addressed:
Issue 1 (CRITICAL): Unguarded jf call in build-apps — "Upload to JFrog" step
The step "Upload ${{ matrix.app_info.name }} to JFrog" (immediately after "Compute app cache key") has no if: guard. It calls jf rt upload unconditionally. When JF_ARTIFACTORY_URL is not set, the setup-jfrog-cli step is skipped, so jf is not in PATH, and this step will fail with a "command not found" error on every matrix runner.
Fix: Add if: steps.jfrog-creds.outputs.available == 'true' to that step.
- name: Upload ${{ matrix.app_info.name }} to JFrog
if: steps.jfrog-creds.outputs.available == 'true'
run: |
...Issue 2 (CRITICAL): hidrive-next-build has no fallback and no explicit failure when JFrog is unavailable
When jfrog-creds is unavailable, both restore steps are silently skipped (not failed). The job then proceeds to make -f IONOS/Makefile build_nextcloud_only with no custom-npms and no app directories in place. This causes an opaque build failure deep in the Makefile rather than a clear, early error.
The guard step even logs "artifact restore steps will fail" — but they don't fail, they skip, misleading the operator.
This job has no GitHub Actions cache fallback for apps (unlike build-apps which at least has actions/cache/save). Since the entire restore strategy is JFrog-only, skipping JFrog silently is not safe.
Fix: Add an explicit gate step that fails fast when JFrog is unavailable (since there is no alternative restore path):
- name: Assert JFrog credentials are available (required for artifact restore)
if: steps.jfrog-creds.outputs.available != 'true'
run: |
echo "❌ JFrog credentials are required for hidrive-next-build artifact restore."
echo " Set JF_ARTIFACTORY_URL, JF_ARTIFACTORY_USER, and JF_ACCESS_TOKEN secrets."
exit 1Summary
| # | Severity | Location | Issue |
|---|---|---|---|
| 1 | Blocking | build-apps / Upload step |
Missing if: steps.jfrog-creds.outputs.available == 'true' guard on jf rt upload |
| 2 | Blocking | hidrive-next-build |
Silent skip when JFrog unavailable leads to opaque downstream failure; no fail-fast gate |
Everything else looks correct: CNPMS_SHORT extraction is consistent between detect-app-cache.sh and the "Compute app cache key" step, cache/save ordering is correct in both build-custom-npms and build-apps, and all previously requested fixes are confirmed.
printminion-co
left a comment
There was a problem hiding this comment.
Final review (iteration 4): two blocking issues remain.
1. github-cache source in hidrive-next-build always triggers exit 1 (pipeline hard-fail)
detect-app-cache.sh emits source: "github-cache" when JFrog is reachable but a specific app artifact is absent from JFrog yet present in GitHub Actions cache (the fallback path at lines 261–267 of detect-app-cache.sh). In hidrive-next-build → "Restore all apps from JFrog and GitHub cache" (lines 573–579), the elif [ "$SOURCE" == "github-cache" ] branch unconditionally calls exit 1.
This means: credentials present (assert gate passes) + at least one app cached only in GH cache → entire hidrive-next-build job fails. There is no actions/cache/restore@v4 step to handle the GH-cache path. Either:
- Add an
actions/cache/restore@v4step for each app withsource == "github-cache"(requires a dynamic step, which GitHub Actions does not support natively — would need a second shell script or composite action), OR - Change
detect-app-cache.shso that when JFrog is available and the artifact is NOT found in JFrog it is always queued for rebuild (added toapps_to_build, notapps_to_restorewithsource: github-cache). This is architecturally cleaner: if JFrog is the canonical artifact store forhidrive-next-build, treat a GH-cache-only hit as a cache miss from JFrog's perspective.
2. build-apps: no fallback to restore custom-npms from GitHub Actions cache when JFrog is unavailable
build-apps restores custom-npms only from JFrog (step guarded by steps.jfrog-creds.outputs.available == 'true'). When JFrog credentials are absent, the restore is silently skipped and the Build step runs against an empty custom-npms/ directory, causing a build failure for every app with needs_custom_npms: true.
build-custom-npms always saves to GitHub Actions cache at the end (lines 278–282). Add an actions/cache/restore@v4 step in build-apps guarded by steps.jfrog-creds.outputs.available != 'true' (or unconditionally as a fallback after the JFrog restore) to cover the no-JFrog path.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| - name: Check JFrog credentials | ||
| id: jfrog-creds | ||
| env: | ||
| JF_URL: ${{ secrets.JF_ARTIFACTORY_URL }} | ||
| run: | | ||
| if [ -n "$JF_URL" ]; then | ||
| echo "available=true" >> $GITHUB_OUTPUT | ||
| else | ||
| echo "available=false" >> $GITHUB_OUTPUT | ||
| echo "⚠ JFrog credentials not set — artifact restore steps will fail" | ||
| fi | ||
|
|
||
| - name: Assert JFrog credentials are available (required for artifact restore) | ||
| if: steps.jfrog-creds.outputs.available != 'true' | ||
| run: | | ||
| echo "❌ JFrog credentials are required for hidrive-next-build artifact restore." | ||
| echo " Set JF_ARTIFACTORY_URL, JF_ARTIFACTORY_USER, and JF_ACCESS_TOKEN secrets." | ||
| exit 1 |
There was a problem hiding this comment.
The JFrog availability check sets available=true when only JF_URL is present, but this job later requires JF_USER and JF_ACCESS_TOKEN too. Consider validating all required secrets up-front and failing with a clear message when any are missing (or skipping JFrog restore entirely if that’s an intended mode).
| elif [ "$SOURCE" == "github-cache" ]; then | ||
| CACHE_KEY=$(echo "$app_json" | jq -r '.cache_key') | ||
| echo "❌ Cannot restore $APP_NAME from GitHub cache within a shell step." | ||
| echo " Cache key: $CACHE_KEY" | ||
| echo " GitHub Actions cache requires 'actions/cache/restore@v4' as a dedicated workflow step." | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
apps_to_restore entries can have source: github-cache (from detect-app-cache.sh), but this step intentionally exits with an error in that case, which will break builds whenever a cache hit comes from GitHub cache instead of JFrog. Either implement actions/cache/restore@v4 restores for those cache keys, or change the detection logic to never emit github-cache when the final build requires JFrog-only restores.
| # Check if artifact exists in JFrog with verbose output | ||
| echo " Running: jf rt s \"$JFROG_PATH\"" | ||
| SEARCH_OUTPUT=$(jf rt s "$JFROG_PATH" 2>&1) | ||
| SEARCH_EXIT_CODE=$? | ||
|
|
There was a problem hiding this comment.
With set -e enabled, a failing jf rt s inside SEARCH_OUTPUT=$(...) will cause the script to exit immediately, so it won’t reach the fallback to the GitHub cache check. Wrap the JFrog search in an if ...; then / || true pattern (or temporarily disable set -e) so JFrog search failures are handled as intended.
| - name: Check JFrog credentials | ||
| id: jfrog-available | ||
| if: github.event.inputs.force_rebuild != 'true' | ||
| env: | ||
| JF_URL: ${{ secrets.JF_ARTIFACTORY_URL }} | ||
| run: | | ||
| if [ -n "$JF_URL" ]; then | ||
| echo "available=true" >> $GITHUB_OUTPUT | ||
| else | ||
| echo "available=false" >> $GITHUB_OUTPUT | ||
| fi | ||
|
|
||
| - name: Setup JFrog CLI | ||
| if: steps.jfrog-available.outputs.available == 'true' | ||
| uses: jfrog/setup-jfrog-cli@7c95feb32008765e1b4e626b078dfd897c4340ad # v4.4.1 | ||
| env: | ||
| JF_URL: ${{ secrets.JF_ARTIFACTORY_URL }} | ||
| JF_USER: ${{ secrets.JF_ARTIFACTORY_USER }} | ||
| JF_ACCESS_TOKEN: ${{ secrets.JF_ACCESS_TOKEN }} |
There was a problem hiding this comment.
jfrog-available is set based only on JF_URL, but the subsequent jfrog/setup-jfrog-cli step requires JF_USER and JF_ACCESS_TOKEN too. Consider checking all three secrets and only marking JFrog as available when the full credential set is present, to avoid attempting JFrog setup with incomplete credentials.
| run: | | ||
| if [ -n "$JF_URL" ]; then | ||
| echo "available=true" >> $GITHUB_OUTPUT | ||
| else | ||
| echo "available=false" >> $GITHUB_OUTPUT | ||
| echo "⚠ JFrog credentials not set — JFrog upload will be skipped" |
There was a problem hiding this comment.
This JFrog credential gate only checks JF_URL, but the upload/ping steps require JF_USER and JF_ACCESS_TOKEN as well. Recommend treating JFrog as available only when all required secrets are set, otherwise skip setup/ping/upload cleanly.
| run: | | |
| if [ -n "$JF_URL" ]; then | |
| echo "available=true" >> $GITHUB_OUTPUT | |
| else | |
| echo "available=false" >> $GITHUB_OUTPUT | |
| echo "⚠ JFrog credentials not set — JFrog upload will be skipped" | |
| JF_USER: ${{ secrets.JF_ARTIFACTORY_USER }} | |
| JF_ACCESS_TOKEN: ${{ secrets.JF_ACCESS_TOKEN }} | |
| run: | | |
| if [ -n "$JF_URL" ] && [ -n "$JF_USER" ] && [ -n "$JF_ACCESS_TOKEN" ]; then | |
| echo "available=true" >> $GITHUB_OUTPUT | |
| else | |
| echo "available=false" >> $GITHUB_OUTPUT | |
| echo "⚠ JFrog credentials incomplete — setup, ping, and upload will be skipped" |
| run: | | ||
| if [ -n "$JF_URL" ]; then | ||
| echo "available=true" >> $GITHUB_OUTPUT | ||
| else | ||
| echo "available=false" >> $GITHUB_OUTPUT | ||
| echo "⚠ JFrog credentials not set — JFrog steps will fail" |
There was a problem hiding this comment.
This JFrog credential check only looks at JF_URL, but later steps depend on JF_USER/JF_ACCESS_TOKEN too. It would be more robust to mark JFrog available only when the full credential set is present, otherwise skip JFrog restore/upload steps.
| run: | | |
| if [ -n "$JF_URL" ]; then | |
| echo "available=true" >> $GITHUB_OUTPUT | |
| else | |
| echo "available=false" >> $GITHUB_OUTPUT | |
| echo "⚠ JFrog credentials not set — JFrog steps will fail" | |
| JF_USER: ${{ secrets.JF_ARTIFACTORY_USER }} | |
| JF_ACCESS_TOKEN: ${{ secrets.JF_ACCESS_TOKEN }} | |
| run: | | |
| if [ -n "$JF_URL" ] && [ -n "$JF_USER" ] && [ -n "$JF_ACCESS_TOKEN" ]; then | |
| echo "available=true" >> $GITHUB_OUTPUT | |
| else | |
| echo "available=false" >> $GITHUB_OUTPUT | |
| echo "⚠ Full JFrog credential set not available — skipping JFrog restore/upload steps" |
|
@copilot apply changes based on the comments in this thread |
Implemented the requested thread fixes in commits |
244d999 to
b18037e
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| needs.prepare-matrix.outputs.has_apps_to_build == 'true' | ||
|
|
||
| permissions: | ||
| contents: read |
There was a problem hiding this comment.
build-apps uses actions/cache@v4, actions/cache/restore@v4, actions/cache/save@v4, and setup-node npm caching, but the job permissions only grant contents: read. Because the workflow sets explicit permissions, the missing actions scope defaults to none, which will break cache restore/save. Add actions: write (and actions: read if you want least-privilege split) to this job or the workflow default permissions.
| contents: read | |
| contents: read | |
| actions: write |
| ARCHIVE_NAME="${APP_NAME}-${APP_SHA}.tar.gz" | ||
| echo "Downloading from JFrog: $JFROG_PATH" | ||
| jf rt download "$JFROG_PATH" "$ARCHIVE_NAME" --flat=true | ||
| mkdir -p "$(dirname "$APP_PATH")" | ||
| tar -xzf "$ARCHIVE_NAME" -C "$(dirname "$APP_PATH")" | ||
| rm -f "$ARCHIVE_NAME" | ||
| if [ -d "$APP_PATH" ]; then | ||
| echo "✅ Restored $APP_NAME from JFrog" |
There was a problem hiding this comment.
When restoring cached apps from JFrog, the workflow downloads from jfrog_path but saves it locally as ${APP_NAME}-${APP_SHA}.tar.gz. APP_SHA here is the full git SHA, while the uploaded archives are named using the 8-char short SHA (and may include the -cnpms-<sha8> suffix). This will cause jf rt download to write a file name that does not match the archive name in Artifactory, and the subsequent tar -xzf will fail. Use the actual archive name (e.g., include archive_name in apps_to_restore or derive it as basename(jfrog_path)).
| ARCHIVE_NAME="${APP_NAME}-${APP_SHA}.tar.gz" | |
| echo "Downloading from JFrog: $JFROG_PATH" | |
| jf rt download "$JFROG_PATH" "$ARCHIVE_NAME" --flat=true | |
| mkdir -p "$(dirname "$APP_PATH")" | |
| tar -xzf "$ARCHIVE_NAME" -C "$(dirname "$APP_PATH")" | |
| rm -f "$ARCHIVE_NAME" | |
| if [ -d "$APP_PATH" ]; then | |
| echo "✅ Restored $APP_NAME from JFrog" | |
| ARCHIVE_NAME="$(basename "$JFROG_PATH")" | |
| echo "Downloading from JFrog: $JFROG_PATH" | |
| jf rt download "$JFROG_PATH" "$ARCHIVE_NAME" --flat=true | |
| mkdir -p "$(dirname "$APP_PATH")" | |
| tar -xzf "$ARCHIVE_NAME" -C "$(dirname "$APP_PATH")" | |
| rm -f "$ARCHIVE_NAME" | |
| if [ -d "$APP_PATH" ]; then | |
| echo "✅ Restored $APP_NAME from JFROG" |
| # Add to restore list with JFrog source | ||
| APPS_TO_RESTORE=$(echo "$APPS_TO_RESTORE" | jq -c --argjson app "$app_json" --arg sha "$CURRENT_SHA" --arg jfrog_path "$JFROG_PATH" --arg source "jfrog" '. + [($app + {sha: $sha, jfrog_path: $jfrog_path, source: $source})]') |
There was a problem hiding this comment.
For JFrog cache hits, the restore list entries do not include the archive file name (only jfrog_path and sha). Downstream (hidrive-next-build) needs the exact archive name to download/extract correctly, especially for apps that include the -cnpms-<sha8> suffix. Include archive_name (or archive_suffix) in the apps_to_restore JSON entries, or ensure downstream derives it from jfrog_path reliably.
| # Add to restore list with JFrog source | |
| APPS_TO_RESTORE=$(echo "$APPS_TO_RESTORE" | jq -c --argjson app "$app_json" --arg sha "$CURRENT_SHA" --arg jfrog_path "$JFROG_PATH" --arg source "jfrog" '. + [($app + {sha: $sha, jfrog_path: $jfrog_path, source: $source})]') | |
| ARCHIVE_NAME="${JFROG_PATH##*/}" | |
| # Add to restore list with JFrog source and exact archive filename | |
| APPS_TO_RESTORE=$(echo "$APPS_TO_RESTORE" | jq -c --argjson app "$app_json" --arg sha "$CURRENT_SHA" --arg jfrog_path "$JFROG_PATH" --arg archive_name "$ARCHIVE_NAME" --arg source "jfrog" '. + [($app + {sha: $sha, jfrog_path: $jfrog_path, archive_name: $archive_name, source: $source})]') |
| needs: prepare-matrix | ||
| if: needs.prepare-matrix.outputs.custom_npms_cached != 'true' | ||
|
|
||
| permissions: |
There was a problem hiding this comment.
build-custom-npms uses actions/cache/save@v4, but the job permissions only grant contents: read. With workflow-level permissions set, any unspecified scopes default to none, so the cache save will be forbidden. Grant actions: write (and actions: read if you later add restore) at either the workflow or job level.
| permissions: | |
| permissions: | |
| actions: write |
Helper script consumed by the cached parallel matrix CI pipeline (added in
the next commit). For each app emitted by IONOS/Makefile's
generate_apps_matrix_json target, it computes a per-app cache key from the
submodule SHA, probes JFrog Artifactory first, falls back to a GitHub
Actions cache lookup via gh cache list, and writes three JSON outputs:
- apps_to_build: apps whose artifact is missing and must be built
- apps_to_restore: apps whose artifact exists in JFrog or GH cache
- apps_sha_map: full {name: SHA} map for traceability
Cache-key shape is uniform: \${CACHE_VERSION}-app-build-\${APP_NAME}-\${SHORT_SHA}
(8-char short SHA). The custom-npms compound-key branch from PR #226's
earlier iterations is gone — custom-npms submodules no longer exist and
generate_apps_matrix_json always emits needs_custom_npms:false.
JFrog setup is idempotent (jf config show short-circuits the add) and
gracefully degrades to GH-cache-only when JF_URL/JF_USER/JF_ACCESS_TOKEN
are absent.
Signed-off-by: Misha M.-Kupriyanov <kupriyanov@strato.de>
8e817ee to
58f58be
Compare
Helper script consumed by the cached parallel matrix CI pipeline (added in
the next commit). For each app emitted by IONOS/Makefile's
generate_apps_matrix_json target, it computes a per-app cache key from the
submodule SHA, probes JFrog Artifactory first, falls back to a GitHub
Actions cache lookup via gh cache list, and writes three JSON outputs:
- apps_to_build: apps whose artifact is missing and must be built
- apps_to_restore: apps whose artifact exists in JFrog or GH cache
- apps_sha_map: full {name: SHA} map for traceability
Cache-key shape is uniform: \${CACHE_VERSION}-app-build-\${APP_NAME}-\${SHORT_SHA}
(8-char short SHA). The custom-npms compound-key branch from PR #226's
earlier iterations is gone — custom-npms submodules no longer exist and
generate_apps_matrix_json always emits needs_custom_npms:false.
JFrog setup is idempotent (jf config show short-circuits the add) and
gracefully degrades to GH-cache-only when JF_URL/JF_USER/JF_ACCESS_TOKEN
are absent.
Signed-off-by: Misha M.-Kupriyanov <kupriyanov@strato.de>
ca1f781 to
edbdcd2
Compare
Helper script consumed by the cached parallel matrix CI pipeline (added in
the next commit). For each app emitted by IONOS/Makefile's
generate_apps_matrix_json target, it computes a per-app cache key from the
submodule SHA, probes JFrog Artifactory first, falls back to a GitHub
Actions cache lookup via gh cache list, and writes three JSON outputs:
- apps_to_build: apps whose artifact is missing and must be built
- apps_to_restore: apps whose artifact exists in JFrog or GH cache
- apps_sha_map: full {name: SHA} map for traceability
Cache-key shape is uniform: \${CACHE_VERSION}-app-build-\${APP_NAME}-\${SHORT_SHA}
(8-char short SHA). The custom-npms compound-key branch from PR #226's
earlier iterations is gone — custom-npms submodules no longer exist and
generate_apps_matrix_json always emits needs_custom_npms:false.
JFrog setup is idempotent (jf config show short-circuits the add) and
gracefully degrades to GH-cache-only when JF_URL/JF_USER/JF_ACCESS_TOKEN
are absent.
Signed-off-by: Misha M.-Kupriyanov <kupriyanov@strato.de>
edbdcd2 to
28b64c8
Compare
Bumps IONOS submodule cce9a71..797b3c7 (now hosted at IONOS-Productivity/nc-config; rename also reflected in .gitmodules) to introduce the build targets and helpers consumed by the new cached parallel matrix CI pipeline. Brings in nc-config master since cce9a71 (REUSE/SPDX compliance, workflow cleanups, configure.sh refactors) and the consolidated nc-config#109 series: - feat(Makefile): add generate_apps_matrix_json with dynamic app list - feat(Makefile): introduce dynamic build rules via define macros + common vars - feat(Makefile): add .precheck target and gate IO-side targets on it - feat(Makefile): two-phase patch_shipped_json with APP_FOLDERS_TO_SHIP - feat(Makefile): exclude apps via REMOVE_UNWANTED_APPS from removed-apps.txt - feat(Makefile): add pipeline targets build_after_external_apps + package_after_build - feat(scripts): add validate_app_list_uniqueness + validate_external_apps + validate_all umbrella - chore(Makefile): update copyright header to 2025 STRATO GmbH The Makefile now drops the legacy custom-npm build targets entirely (see nc-config#109) — generate_apps_matrix_json always emits "needs_custom_npms":false. Subsequent commits in this PR add the workflow and helper script that consume these targets. Signed-off-by: Misha M.-Kupriyanov <kupriyanov@strato.de>
Helper script consumed by the cached parallel matrix CI pipeline (added in
the next commit). For each app emitted by IONOS/Makefile's
generate_apps_matrix_json target, it computes a per-app cache key from the
submodule SHA, probes JFrog Artifactory first, falls back to a GitHub
Actions cache lookup via gh cache list, and writes three JSON outputs:
- apps_to_build: apps whose artifact is missing and must be built
- apps_to_restore: apps whose artifact exists in JFrog or GH cache
- apps_sha_map: full {name: SHA} map for traceability
Cache-key shape is uniform: \${CACHE_VERSION}-app-build-\${APP_NAME}-\${SHORT_SHA}
(8-char short SHA). The custom-npms compound-key branch from PR #226's
earlier iterations is gone — custom-npms submodules no longer exist and
generate_apps_matrix_json always emits needs_custom_npms:false.
JFrog setup is idempotent (jf config show short-circuits the add) and
gracefully degrades to GH-cache-only when JF_URL/JF_USER/JF_ACCESS_TOKEN
are absent.
Signed-off-by: Misha M.-Kupriyanov <kupriyanov@strato.de>
Composite action that fetches the GitHub Actions job HTML URL via the REST API and exposes it as an output. Used by subsequent steps to attach job.html_url to JFrog artifact properties for traceability between an Artifactory artifact and the run/job that produced it. Uses a temp netrc file to keep the GH token off the curl command line, and trap-cleans both the netrc and the curl-error temp file on exit. Returns an empty output (not a failure) when the API call or job lookup fails, so callers can include it with continue-on-error: true. Copied from sister ncw-server (.github/actions/get-job-data/action.yml).
07b5939 to
9dc5306
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (3)
.github/workflows/hidrive-next-build.yml:595
actions/setup-node'scacheinput is conditionally set to an empty string whenforce_rebuildis true.cacheexpects a supported value (e.g.npm) or should be omitted entirely; passing an empty string can cause the action to error. Consider guarding the step withif:or splitting into two steps socacheis only provided when enabled.
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version-file: "package.json"
cache: ${{ github.event.inputs.force_rebuild != 'true' && 'npm' || '' }}
.github/workflows/hidrive-next-build.yml:481
- This job hard-fails when JFrog credentials are missing, but the earlier cache-detection script explicitly supports falling back to GitHub Actions cache. As written, any run without JFrog secrets will always fail here (and
build-appseven advertises it can proceed via GitHub cache). Either implement a GitHub-cache restore path (e.g., a dedicated restore matrix usingactions/cache/restore) or change the detection logic/job flow sohidrive-next-builddoesn't require JFrog when GitHub cache is being used.
- name: Assert JFrog credentials are available (required for artifact restore)
if: steps.jfrog-creds.outputs.available != 'true'
run: |
echo "❌ JFrog credentials are required for hidrive-next-build artifact restore."
echo " Set JF_ARTIFACTORY_URL, JF_ARTIFACTORY_USER, and JF_ACCESS_TOKEN secrets."
exit 1
.github/workflows/hidrive-next-build.yml:547
- The workflow currently exits with an error if an app is marked as
source=github-cache, butdetect-app-cache.shcan emitgithub-cacheentries when JFrog isn't available. This makes the pipeline internally inconsistent and can break builds even when a GitHub cache hit exists. Prefer restoring these entries viaactions/cache/restore@v4(likely in a separate matrix job), or ensureapps_to_restorenever containsgithub-cachefor this job path.
elif [ "$SOURCE" == "github-cache" ]; then
CACHE_KEY=$(echo "$app_json" | jq -r '.cache_key')
echo "❌ Cannot restore $APP_NAME from GitHub cache within a shell step."
echo " Cache key: $CACHE_KEY"
echo " GitHub Actions cache requires 'actions/cache/restore@v4' as a dedicated workflow step."
exit 1
| continue-on-error: true | ||
| uses: ./.github/actions/get-job-data | ||
| with: | ||
| job-name: 'build-apps (${{ matrix.app_info.name }}, ${{ matrix.app_info.sha }})' |
| API_RESPONSE=$(curl -sS -w "\n%{http_code}" \ | ||
| --netrc-file "$NETRC_FILE" \ | ||
| -H "Accept: application/vnd.github+json" \ | ||
| "https://api.github.com/repos/${{ inputs.repository }}/actions/runs/${{ inputs.run-id }}/jobs" 2>"$CURL_ERROR_FILE") | ||
|
|
||
| CURL_EXIT_CODE=$? | ||
| if [ $CURL_EXIT_CODE -ne 0 ]; then | ||
| echo "❌ ERROR: curl request to GitHub API failed with exit code $CURL_EXIT_CODE" | ||
| if [ -s "$CURL_ERROR_FILE" ]; then | ||
| echo "curl error output:" | ||
| cat "$CURL_ERROR_FILE" | ||
| fi | ||
| echo "job_html_url=" >> "$GITHUB_OUTPUT" | ||
| exit 0 | ||
| fi | ||
|
|
||
| HTTP_CODE=$(echo "$API_RESPONSE" | tail -n1) | ||
| RESPONSE_BODY=$(echo "$API_RESPONSE" | sed '$d') | ||
|
|
||
| if [ "$HTTP_CODE" != "200" ]; then | ||
| echo "⚠️ WARNING: GitHub API request failed with $HTTP_CODE" | ||
| echo "job_html_url=" >> "$GITHUB_OUTPUT" | ||
| exit 0 | ||
| fi | ||
|
|
||
| EXPECTED_JOB_NAME="${{ inputs.job-name }}" | ||
| JOB_URL=$(echo "$RESPONSE_BODY" | jq -r \ | ||
| --arg job_name "$EXPECTED_JOB_NAME" \ | ||
| '.jobs[] | select(.name == $job_name) | .html_url') | ||
|
|
||
| if [ -z "$JOB_URL" ] || [ "$JOB_URL" = "null" ]; then | ||
| echo "⚠️ WARNING: Failed to extract job URL from response for job name '$EXPECTED_JOB_NAME'." | ||
| echo "Possible causes:" | ||
| echo " - The job name does not match exactly (including spaces and case)." | ||
| echo " - The job has not started yet at the time this action ran." |
142c098 to
734b091
Compare
Replaces the single monolithic build job with a per-app matrix: - prepare-matrix: discover apps and decide cache-hit vs rebuild via detect-app-cache.sh (JFrog + GitHub cache probe), emit a build matrix, preflight summary + gh cache list for diagnostics. - build-apps: parallel matrix that restores cached artifacts or rebuilds, validates appinfo/info.xml after restore, uploads to JFrog with secrets preflight + retry + post-upload verification, and uses JFrog props arrays embedding job.html_url. - hidrive-next-build: rewired to consume per-app cached artifacts. Hardening: concurrency group for push + pull_request runs, PHP pinned to 8.3, npm cache bypassed when force_rebuild is set, per-app completion summary, per-app debug artifact upload.
- actions/checkout -> v5.0.1 - actions/setup-node -> v6.4.0 - actions/download-artifact -> v5.0.0 - docker/login-action -> v3.7.0 - docker/metadata-action -> v5.10.0
Enables push-build for rc/** and */dev/* branches, triggers the remote GitLab workflow for both, and routes their artifacts to nested JFrog paths that include NC_VERSION and short SHA so every build is preserved. JFrog layout (consolidated): | Branch/Event | Path | |---------------|--------------------------------------------------------------------| | Pull Request | pr/hidrive-next-pr-<n>.zip | | ionos-dev | dev/hidrive-next-<ver>/<sha>/hidrive-next-<ver>.zip | | ionos-stable | stable/hidrive-next-<ver>/<sha>/hidrive-next-<ver>.zip | | rc/* | rc/<branch>/hidrive-next-<ver>/<sha>/hidrive-next-<ver>.zip | | */dev/* | devs/<prefix>/hidrive-next-<ver>/<sha>/hidrive-next-<ver>.zip | Changes vs. previous behavior: - PR uploads move from dev/pr/... to top-level pr/... - */dev/* uploads change from dev-<prefix>/... to devs/<prefix>/... (folder instead of dash-suffix) GitLab BUILD_TYPE (consumer-facing) keeps dev-<prefix> — it is a separate naming contract from the JFrog path.
…SER_DEV
The remote GitLab pipeline does not yet handle the BUILD_TYPE=dev-<prefix>
payload produced for the */dev/* developer-branch class, so every push to
a developer branch currently fires a remote run the consumer cannot
service.
Add an opt-in repository variable ENABLE_REMOTE_TRIGGER_USER_DEV which
must be set to 'true' for the trigger-remote-dev-workflow job to run on
*/dev/* branches. The other three branch classes (ionos-dev, ionos-stable,
rc/*) are unaffected and continue to trigger as before. The existing
DISABLE_REMOTE_TRIGGER kill-switch keeps its current semantics and layers
on top of the new opt-in.
The "Check configuration" diagnostic step in prepare-matrix now prints
the new variable and surfaces an explicit ❌/✅ line for */dev/* branches
so the expected-outcome line ("WILL RUN" / "WILL BE SKIPPED") reflects
the opt-in state.
upload-to-artifactory and the ghcr image push are unchanged — */dev/*
builds still land in devs/<prefix>/... in Artifactory.
…anch for submodule trigger) https://github.com/IONOS-Productivity/nc-config/releases/tag/6dedd72 Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
19468c7 to
0e14ce3
Compare
Summary
Migrates the HiDrive Next CI build from a single monolithic job to a cached parallel matrix pipeline.
This pull request introduces two major improvements to the CI/CD pipeline: a new composite GitHub Action for retrieving job data and a comprehensive shell script for detecting which apps need to be built or can be restored from cache. Additionally, it updates the
IONOSsubmodule reference.Key changes:
CI/CD Pipeline Enhancements
.github/actions/get-job-data/action.yml, a composite GitHub Action that fetches job data (specifically, the job HTML URL) from the GitHub Actions API, with robust error handling and credential management..github/scripts/detect-app-cache.sh, a detailed script that determines, for a matrix of apps, which ones require building and which can be restored from cache. The script supports both GitHub Actions cache and JFrog Artifactory as cache sources, outputs detailed status summaries, and includes extensive validation and error handling.Submodule Update
IONOSsubmodule reference to the latest commit.Related