Skip to content

release-workflow-guide.md

Latest commit

 

History

History
286 lines (201 loc) · 12.1 KB

release-workflow-guide.md

File metadata and controls

286 lines (201 loc) · 12.1 KB

OpenDataHub (ODH) Operator Release and Branching Process

This document provides an overview of the release process followed in the OpenDataHub (ODH) operator development lifecycle. It explains how upstream and downstream repositories are managed, including the branching strategy, code synchronization, and backporting workflows.

Repositories Overview

The OpenDataHub operator development and release process spans two main repositories:

Upstream Development and Release Workflow

There are four long-lived branches:

  • main the primary development branch where all changes land first
  • stable the staging branch for changes before they reach downstream
  • rhoai the branch tracking downstream (productization) changes
  • stable-2.x the branch for the 2.x release changes

Main Branch Overview

  • All new features and bug fixes are first merged into the main branch.

Stable Branch Overview

  • The stable branch serves as a staging area for changes before they reach downstream.
  • Changes are automatically synced from main to stable every 2 hours via fast-forward merge.

RHOAI Branch Overview

  • A dedicated rhoai branch exists to track downstream-related changes.
  • Changes are automatically synced from stable to rhoai every 2 hours.

Stable-2.x Branch Overview

  • The stable-2.x branch tracks the 2.x release series.
  • Changes are automatically synced from stable-2.x to the downstream branch rhoai-2.25.
  • All the changes targeted for RHOAI 2.X must land in this branch.

Release Branches Overview

  • Release branches (odh-x.y, e.g., odh-2.26.0) are created from the main branch when a new upstream release is planned.

Basic Upstream Workflow for the Operator

Changes to the operator flow through a two-stage sync pipeline: main → stable → rhoai.

gitGraph
    branch stable order: 3
    branch rhoai order: 4
    checkout main
    commit
    checkout stable
    commit
    checkout rhoai
    commit
    checkout main
    branch feature
    commit id: "2-0a10a11"
    checkout main
    merge feature
    checkout stable
    merge main
    checkout rhoai
    merge stable
Loading
  1. Merge PR to main. Follow the process in CONTRIBUTING.md.
  2. Sync changes to stable branch: The sync-main-to-stable workflow will automatically sync the changes from the main branch to the stable branch every 2 hours using fast-forward merge.
  3. Sync changes to rhoai branch: The sync-stable-to-rhoai workflow will automatically sync the changes from the stable branch to the rhoai branch every 2 hours.

rhoai Code Freeze and Manual Sync Workflow

To prepare the downstream releases, we freeze the rhoai branch to prevent new development from entering the release. Development continues on the upstream main branch and stable branch.

During code freeze:

  1. The sync-stable-to-rhoai workflow is disabled to prevent automatic syncing from stable to rhoai.
  2. The sync-main-to-stable workflow continues to run, keeping stable up to date with main.
  3. Changes that need to land in the release after the code freeze must be cherry-picked manually into the red-hat-data-services/rhoai-x.y branch.

Downstream Development and Release Workflow

Main Branch Overview

  • Changes from the upstream rhoai branch are synced into the downstream main branch.

Release Branches Overview

  • From the downstream main branch, changes are backported to release-specific branches (rhoai-x.y, e.g., rhoai-2.21).
  • These branches represent versions that are under active development or maintenance.

Basic Downstream Workflow for the Operator

sequenceDiagram
    actor engineer
    participant main as opendatahub-io/main
    participant stable as opendatahub-io/stable
    participant rhoai as opendatahub-io/rhoai
    participant rhds as red-hat-data-services/main
    participant rhoaixy as red-hat-data-services/rhoai-x.y

    engineer ->> main: pull request
    main ->> stable: merge (via CI)
    stable ->> rhoai: merge (via CI)
    rhoai ->> rhds: merge (via CI)
alt landing a change in the next rhoai release
    rhds ->> rhoaixy: merge (via CI)
else fixing a blocker issue in a frozen release branch/fixing an issue in a z-stream release
    engineer ->> rhoaixy: cherry-pick & pull request
end
Loading

A change that lands in the odh-operator main branch ends up in the next rhoai-x.y branch as follows:

  1. CI automation periodically (every 2 hours) merges the opendatahub-operator main branch into the opendatahub-operator stable branch using fast-forward merge.
  2. CI automation periodically (every 2 hours) merges the opendatahub-operator stable branch into the opendatahub-operator rhoai branch.
  3. CI automation periodically merges the opendatahub-operator rhoai branch into the rhods-operator main branch.
  4. CI automation periodically merges the rhods-operator main branch into the active rhods-operator rhoai-x.y release branch.

Code Freeze and Z-Stream Overview

If a change is needed in a frozen/z-stream downstream release branch, an engineer opens a pull request against that specific release branch. The following illustration shows the three stages of downstream branches:

  • rhoai-2.22 – Active development branch
  • rhoai-2.21 – Frozen branch in blocker-only phase
  • rhoai-2.19 – Released branch receiving only z-stream (critical) fixes
  1. Automatic backports from main to rhoai-2.21 are stopped.
  2. A new development branch such as rhoai-2.22 is created, which starts receiving backports from main.
  3. rhoai-2.21 becomes a frozen branch, entering the blocker-only phase, where only critical, approved fixes are permitted.
  4. Any fix targeting rhoai-2.21 must be approved as a blocker.
  5. Older branches (e.g., rhoai-2.19) represent previously released versions and are maintained through z-stream (micro) releases, which accept only critical fixes.
  6. These frozen/z-stream fixes are typically cherry-picked manually from the main branch to the appropriate branches.

ODH Release Process(community)

The Open Data Hub (ODH) follows a 4-week release cycle. This document outlines the standard steps involved in preparing and executing an ODH release.

Tracker Issue

On the code freeze date (Friday), each team must:

  • Post a comment on the tracker issue (created for every release)
  • The comment must include the following format: 
    #Release#
component-name|branch-url|tag-url

Note: The comment format must comply with the expected structure to be parsed by the release automation tool.

Operator-Level Images

For operator-level images (non-component images like kube-auth-proxy), teams can include an additional section in their tracker comment:

#Images#
image-name|full-image-reference

Example:

#Images#
kube-auth-proxy|quay.io/opendatahub/odh-kube-auth-proxy:v1.2.3

This section is optional. If present, the release automation will:

  • Parse the image references from the #Images# section
  • Update the RELATED_IMAGE_* environment variables in config/manager/manager.yaml
  • Flow these image references through to the operator bundle/CSV automatically

Note: Image names are automatically converted to environment variable names using a naming convention:

  • Uppercase the image name
  • Replace hyphens with underscores
  • Add prefix RELATED_IMAGE_ODH_ and suffix _IMAGE
  • Example: kube-auth-proxy → RELATED_IMAGE_ODH_KUBE_AUTH_PROXY_IMAGE

No code changes needed to add a new operator-level image - just add it to the tracker comment!

Code freeze occurs on Friday, and the actual release occurs on the following Monday.

Release Automation

A set of workflows/tasks using GitHub Actions (GHA) and Konflux Pipelines assists in automating the release process:

1. Triggering release-staging Workflow

In the operator repository:

This workflow performs the following steps:

  1. Validation:

    • Validates the tracker URL format
    • Validates the version number follows semver

  2. Issue Number Extraction:

    • Extracts the issue number from the tracker URL for later use

  3. Release Branch:

    • Creates or updates the release branch (odh-x.y.z)
    • If the branch exists, it's checked out and updated
    • If the branch doesn't exist, it's created from main

  4. Version Updates:

    • Updates versions in relevant files
    • Runs make manifests and bundle
    • Cleans up bundle files
    • Updates manifest branches

  5. Pre-Release:

    • Creates and pushes a tag (v{x.y.z})
    • Generates release notes
    • Creates a GitHub pre-release with:
      • Tag: v{x.y.z}
      • Pre-release status: true

  6. Tracker Update:

    • Comments on the tracker issue with the release information of the opendatahub-operator

  7. Konflux Integration:

    • Triggers the Konflux release onboarder workflow

Chained Workflows

odh-konflux-central: odh-konflux-release-onboarder

This workflow is automatically triggered by the release-staging workflow and it:

  • Creates a PR to the release branch
  • Adds the Tekton files required for the Konflux build process
  • The workflow is triggered with:
    • component: opendatahub-operator
    • release_branch: odh-{version}
    • version: v{version}

Konflux Build

Once the above PR is merged, the Konflux pipeline starts and:

  • Builds the operator image
  • Builds the operator bundle

The initial catalog build will fail as there will be no catalog.yaml present, the next step will take care of the failure by generating the catalog and retriggering the catalog build.

opendatahub: fbc-processor

Triggered at the end of the operator bundle pipeline:

  • Generates catalog.yaml required for the catalog build
  • Pushes commit to the release branch
  • Initiates a Konflux build to create the catalog image

QE Sign-Off

The Quality Engineering (QE) team performs:

  • A set of ODS-CI tests using the catalog image generated from the above step
  • The tests will run for fresh installation and upgrade case
  • Once QE provides final sign-off, the release can proceed

OperatorHub Release

Tracker Closure

  • Update and close the tracker issue with:

ODH Release Process is now complete.

Summary