Windowing issue

[Balatripura587]

Type of change

• Refactor
• New feature
• Bug fix
• Optimization
• Documentation Update

Description

This PR fixes a critical windowing issue in changepoint detection that was causing false positives in CI pipelines. The issue occurred when changepoints were detected too early in the data series (e.g., at the 2nd data point) without sufficient context to validate whether they represent actual trend changes or just anomalies.

Problem

Orion uses the hunter tool (EDivisive algorithm) underneath to detect changepoints. For the tool to function properly and identify changes in trends, it requires at least 10 samples. However, the current implementation was concluding changepoints without validating that sufficient data exists before and after the detected changepoint.

Example scenario:

• A job looks for data samples for the last 15 days
• The 2nd data point is declared as a changepoint
• This doesn't make sense because starting from the 2nd data point, if we consider 10 days before, we might not see that as a changepoint (it's literally an anomaly being classified as a changepoint)

This issue was creating noise in CI pipelines, particularly affecting OVN and other jobs.

Solution

Implemented a windowing validation logic that ensures changepoints are only concluded when there's sufficient context:

1. Early Changepoint Detection (First 5 Data Points):

   • If a changepoint is detected in the first 5 data points, the system expands both lookback constraints:
     • Time-based lookback (--lookback): Expanded by adding 10 days to the original lookback period
     • Count-based lookback (--lookback-size): Expanded to current_points + 5 to ensure at least 5 data points exist before the changepoint
   • Since Orion takes the minimum of both constraints (time-based and count-based), the expansion ensures sufficient historical data is available
   • Re-runs the analysis with the expanded dataset to validate the changepoint
   • Only concludes the changepoint if it persists after validation with more historical data

2. Late Changepoint Detection (Last 5 Data Points):

   • If a changepoint is detected in the last 5 data points, it's ignored due to insufficient future data
   • These changepoints will be properly detected in the next cycle when more data becomes available
   • Prevents false positives from changepoints at the end of the data series

3. Validation Requirements:

   • For every changepoint detected, the system now considers 5 samples before/after to validate it's an actual changepoint
   • In special cases when there aren't enough data points, the changepoint is not concluded

Related Tickets & Documents

• Related Issue Issue with windowing #263
• Closes Issue with windowing #263

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.

Testing

System Under Test

• Component: Orion changepoint detection system
• Algorithm: EDivisive (hunter tool) for changepoint detection
• Environment: CI/CD pipelines, particularly OVN jobs and other performance test jobs

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Balatripura587
Once this PR has been reviewed and has the lgtm label, please assign jtaleric for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment