Improve PR Upgrade workflow #2953
Description
Description
Currently, the PR upgrade workflows are comparing between two states:
- Latest release (Latest released image tag, latest release branch) - before upgrade
- Current PR changes (PR image, PR branch) - after upgrade
This is not an ideal solution, since e2e upgrade tests are always taken from the release branch. It leads to unexpected behaviors which require constant maintenance of the release branch e2e tests, since we are running before_upgrade test always using the code in the latest release branch.
After releasing the otlp gateway daemonset in experimental mode, any PR to the main branch runs the PR upgrade tests (before_upgrade) in the latest release branch (one where an old log gateway deployment was still expected), but the latest release has an otlp gateway daemonset, causing the PR Upgrade test to constantly fail.
Additionally, the after_upgrade test runs too early before reconciliation happens, resulting in assertions being applied to existing resources managed by the old manager. For now we solve this with a 60 second sleep, but in the future we need to use a more reliable way to ensure the new manager has reconciled at least once before running the tests.
Expected result
PR Upgrade test to not fail after releases
Actual result
PR Upgrade test occasionally fails after releases
Steps to reproduce
Look at action run https://github.com/kyma-project/telemetry-manager/actions/runs/21664125063/job/62455245875
Troubleshooting
Release Notes