Skip to content

feat(modelsasservice): add deployment-based observability toggle - #3374

Merged
openshift-merge-bot[bot] merged 5 commits into
opendatahub-io:mainfrom
ishitasequeira:feature/deployment-based-observability-toggle
Apr 8, 2026
Merged

openshift-merge-bot[bot] merged 5 commits into
opendatahub-io:mainfrom
ishitasequeira:feature/deployment-based-observability-toggle

Conversation

@ishitasequeira

@ishitasequeira ishitasequeira commented Apr 2, 2026

Copy link
Copy Markdown
Contributor

Description

Add deployment-based observability support to the ModelsAsService component, enabling per-subscription latency tracking via Istio Telemetry resources.
Changes:

  • Add ObservabilityConfig to ModelsAsServiceSpec with an enabled boolean flag (technical preview)
  • Add configureIstioTelemetry function to create/manage Istio Telemetry resources in the gateway namespace
  • Wire observability configuration into the reconciliation pipeline
  • Add IstioTelemetryName constant (latency-per-subscription) for the telemetry resource
  • Fix TelemetryPolicy CEL paths to use nested subscription_info fields:
    • cost_center → auth.identity.subscription_info.costCenter
    • organization_id → auth.identity.subscription_info.organizationId
  • Remove legacy tier label from always-on dimensions (migration to subscription-based model)
  • Add RBAC for telemetry.istio.io/telemetries resources via kubebuilder marker
  • Add comprehensive unit tests for observability configuration

Feature behavior:
When spec.observability.enabled: true is set in the ModelsAsService CR, the operator creates an Istio Telemetry resource that adds a subscription label to istio_request_duration_milliseconds_bucket metrics, extracted from the X-MaaS-Subscription header injected by AuthPolicy. This enables P50/P95/P99 latency tracking per subscription in Grafana dashboards.
RHOAIENG-56531

How Has This Been Tested?

  • Unit tests:
  • Manual Testing
    1. Istio Telemetry Resource (created by operator when enabled)
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
 name: latency-per-subscription
 namespace: openshift-ingress  # Same namespace as the gateway
 labels:
   app.kubernetes.io/part-of: maas-observability
 ownerReferences:
   - apiVersion: components.platform.opendatahub.io/v1alpha1
     kind: ModelsAsService
     name: default-modelsasservice
     controller: true
     blockOwnerDeletion: true
spec:
 selector:
   matchLabels:
     gateway.networking.k8s.io/gateway-name: maas-default-gateway
 metrics:
   - providers:
       - name: prometheus
     overrides:
       - match:
           metric: REQUEST_DURATION
           mode: CLIENT_AND_SERVER
         tagOverrides:
           subscription:
             operation: UPSERT
             value: request.headers["x-maas-subscription"]
  1. ModelsAsService CR (new observability field)
apiVersion: components.platform.opendatahub.io/v1alpha1
kind: ModelsAsService
metadata:
  name: default-modelsasservice
spec:
  # ... existing fields ...
  
  # NEW: Observability configuration (technical preview)
  observability:
    enabled: true  # Set to true to enable per-subscription l
  1. RBAC Addition (ClusterRole)
# Added to config/rbac/role.yaml via kubebuilder marker
- apiGroups:
  - telemetry.istio.io
  resources:
  - telemetries
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

Screenshot or short clip

  • Usage Dashboard
image

Merge criteria

  • You have read the contributors guide.
  • Commit messages are meaningful - have a clear and concise summary and detailed explanation of what was changed and why.
  • Pull Request contains a description of the solution, a link to the JIRA issue, and to any dependent or related Pull Request.
  • Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
  • The developer has manually tested the changes and verified that the changes work
  • The developer has run the integration test pipeline and verified that it passed successfully
  • New RELATED_IMAGE mappings are already listed in ODH-Build-Config and RHOAI-Build-Config, and links are included in PR description

E2E test suite update requirement

When bringing new changes to the operator code, such changes are by default required to be accompanied by extending and/or updating the E2E test suite accordingly.

To opt-out of this requirement:

  1. Please inspect the opt-out guidelines, to determine if the nature of the PR changes allows for skipping this requirement
  2. If opt-out is applicable, provide justification in the dedicated E2E update requirement opt-out justification section below
  3. Check the checkbox below:
  • Skip requirement to update E2E test suite for this PR
  1. Submit/save these changes to the PR description. This will automatically trigger the check.

E2E update requirement opt-out justification

This PR adds a new optional, disabled-by-default technical preview feature. The feature:

  • Is gated behind spec.observability.enabled which defaults to false
  • Only creates additional Istio Telemetry resources when explicitly enabled
  • Does not modify existing behavior or break backwards compatibility
  • Has comprehensive unit test coverage for the new functionality

Summary by CodeRabbit

  • New Features

    • Add optional telemetry config with an enable/disable flag (default: true); when enabled, deploys usage metrics and Istio Telemetry for per-subscription latency.
  • Tests

    • Added tests for telemetry creation/skipping, CRD-availability gating, and metadata/label expectations.
  • Chores

    • Added RBAC for Istio Telemetry; adjusted telemetry label mappings and removed the always-on tier label.
  • Documentation

    • Updated API docs to describe conditional telemetry deployment and the new enabled flag.

@coderabbitai

coderabbitai Bot commented Apr 2, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds a conditional telemetry configuration to ModelsAsService: Telemetry *TelemetryConfig with TelemetryConfig.Enabled *bool (kubebuilder default true). DeepCopy for TelemetryConfig.Enabled now allocates a new bool. Controller setup registers ownership for Istio telemetry.istio.io/v1 when the CRD exists and inserts configureIstioTelemetry into the reconcile chain to create an Istio Telemetry resource (latency-per-subscription) when enabled. Telemetry label CEL expressions were changed to reference auth.identity.subscription_info.*, the always-on tier label was removed, tests and docs were updated, and RBAC for telemetries was added.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Issues

  • CWE-639 / CWE-345 — Trusting client-controlled header for subscription attribution: the Istio Telemetry TagOverride uses request.headers["x-maas-subscription"], which is attacker-controllable and can spoof subscription attribution. Actionable: stop using raw headers for billing/metrics; validate the header against a trusted auth token or derive subscription from a validated server-side claim (e.g., auth.identity.selected_subscription) before exposing it to telemetry.

  • CWE-276 — Excessive RBAC permissions: the new kubebuilder RBAC grants broad verbs (get,list,watch,create,update,patch,delete) on telemetries.telemetry.istio.io. Actionable: restrict verbs to the minimum required (e.g., get,list,watch,create,patch) and, if possible, scope to specific namespaces or resource names.

  • CWE-20 / Missing input validation — Fragile identity contract for CEL expressions: telemetry CEL expressions now reference auth.identity.subscription_info.*. If subscription_info is missing or differently shaped, labels will be empty or mispopulated. Actionable: document the required identity schema, add defensive CEL fallbacks or existence checks, or gate telemetry creation until required identity fields are present.

  • CWE-1004 / Owner reference cross-namespace risk: controller sets ownerRefs on Istio Telemetry resources created in the gateway namespace. Kubernetes forbids cross-namespace ownerReferences for deletion cascades. Actionable: ensure Telemetry resources are created in the same namespace as the owner, or avoid ownerReferences and implement explicit cleanup logic; validate ownerReference usage against Kubernetes rules.

🚥 Pre-merge checks | ✅ 2
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: adding a deployment-based toggle to control observability/telemetry resource creation.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Apr 2, 2026

Copy link
Copy Markdown
Contributor

This PR can't be merged just yet 😢

Please run make generate manifests api-docs and commit the changes.

For more info: https://github.com/opendatahub-io/opendatahub-operator/actions/runs/23907043428

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In
`@internal/controller/components/modelsasservice/modelsasservice_controller_actions.go`:
- Around line 320-333: In configureIstioTelemetry, short-circuit when
observability is disabled before calling cluster.HasCRD: read the instance spec
(e.g., rr.Instance.Spec.Observability.Enabled or rr.Spec.Observability.Enabled)
and if it's false/nil log and return nil immediately, then only if observability
is enabled proceed to call cluster.HasCRD(gvk.Telemetry) and the existing flow;
this prevents transient CRD lookup failures from failing reconciles for disabled
observability.
- Around line 381-384: The Telemetry creation code that builds a selector using
gatewayName should pre-validate for selector conflicts instead of blindly
applying it; before appending the Telemetry with the selector map (the block
that sets "selector": map[string]any{"matchLabels":
{"gateway.networking.k8s.io/gateway-name": gatewayName}}) list existing Istio
Telemetry resources in the target namespace and compare their workload selectors
for overlap with the new selector; if an overlap/conflict is found, do not apply
the new Telemetry and surface a ModelsAsService status condition (use the
controller's status update / SetCondition logic on the ModelsAsService resource)
describing the conflicting Telemetry and why deployment is blocked, or
alternatively implement merging by owning a single controller-named Telemetry
resource (e.g., controller-owned Telemetry) and reconcile/merge required
selector/metric overrides into that single resource instead of creating separate
Telemetries per gateway.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: e695a99a-6461-4478-b11f-ded90f940f9c

📥 Commits

Reviewing files that changed from the base of the PR and between 40d9281 and 58a85d3.

📒 Files selected for processing (7)
  • api/components/v1alpha1/modelsasservice_types.go
  • api/components/v1alpha1/zz_generated.deepcopy.go
  • internal/controller/components/modelsasservice/modelsasservice_controller.go
  • internal/controller/components/modelsasservice/modelsasservice_controller_actions.go
  • internal/controller/components/modelsasservice/modelsasservice_controller_actions_test.go
  • internal/controller/components/modelsasservice/modelsasservice_support.go
  • internal/controller/datasciencecluster/kubebuilder_rbac.go

Comment on lines +381 to +384
"selector": map[string]any{
"matchLabels": map[string]any{
"gateway.networking.k8s.io/gateway-name": gatewayName,
},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# First, let's understand the file structure and locate the file
cd internal/controller/components/modelsasservice 2>/dev/null && pwd || echo "Directory not found"

Repository: opendatahub-io/opendatahub-operator

Length of output: 144


🏁 Script executed:

# Find the file
find . -name "*modelsasservice_controller_actions.go" -type f

Repository: opendatahub-io/opendatahub-operator

Length of output: 165


🏁 Script executed:

# Read the specific lines and surrounding context to understand the issue
cat -n internal/controller/components/modelsasservice/modelsasservice_controller_actions.go | sed -n '370,395p'

Repository: opendatahub-io/opendatahub-operator

Length of output: 954


🏁 Script executed:

# Also read the second location (421-422) with context
cat -n internal/controller/components/modelsasservice/modelsasservice_controller_actions.go | sed -n '410,435p'

Repository: opendatahub-io/opendatahub-operator

Length of output: 1219


🏁 Script executed:

# Search for Telemetry creation/appending logic to understand the pattern
rg -A 10 -B 5 "Telemetry" internal/controller/components/modelsasservice/modelsasservice_controller_actions.go | head -100

Repository: opendatahub-io/opendatahub-operator

Length of output: 3665


🏁 Script executed:

# Check for conflict detection or existing Telemetry validation
rg "ConflictingTelemetry|IST0159|workload.*selector" internal/controller/components/modelsasservice/ || echo "No existing conflict detection found"

Repository: opendatahub-io/opendatahub-operator

Length of output: 115


🏁 Script executed:

# Check if there's reconciliation logic that updates existing Telemetry resources
rg -B 5 -A 5 "IstioTelemetryName|rr.Resources" internal/controller/components/modelsasservice/ | head -50

Repository: opendatahub-io/opendatahub-operator

Length of output: 5596


🏁 Script executed:

# Search for how resources are deployed/applied in reconciliation
rg -A 10 "Resources.*append\|reconciliation.*Telemetry" internal/controller/ | head -60

Repository: opendatahub-io/opendatahub-operator

Length of output: 61


🏁 Script executed:

# Check if there's any logic that lists or filters existing Telemetry before creating
rg "Telemetry.*List\|GetTelemetry\|ListTelemetry" internal/controller/

Repository: opendatahub-io/opendatahub-operator

Length of output: 61


🏁 Script executed:

# See the complete function context for configureTelemetryPolicy to understand the flow
cat -n internal/controller/components/modelsasservice/modelsasservice_controller_actions.go | sed -n '300,370p'

Repository: opendatahub-io/opendatahub-operator

Length of output: 3252


🏁 Script executed:

# Check how rr.Resources are deployed/applied in the reconciliation flow
rg -B 5 -A 10 "rr.Resources" internal/controller/components/modelsasservice/ | grep -A 10 -B 5 "apply\|Create\|Patch"

Repository: opendatahub-io/opendatahub-operator

Length of output: 1949


🏁 Script executed:

# Search for how resources are deployed across the controller
rg -A 10 "ReconciliationRequest.*Resources\|apply.*Resources" internal/controller/ | head -80

Repository: opendatahub-io/opendatahub-operator

Length of output: 61


🏁 Script executed:

# Check if there's any error handling or validation for resource deployment
rg "error.*Telemetry\|IST0159\|workload.*selector.*conflict" internal/

Repository: opendatahub-io/opendatahub-operator

Length of output: 61


🏁 Script executed:

# Understand the reconciliation flow - check the main reconciler
find internal/controller -name "*.go" -type f | xargs grep -l "ReconciliationRequest" | head -3

Repository: opendatahub-io/opendatahub-operator

Length of output: 307


🏁 Script executed:

# Look for how resources in rr.Resources are actually applied to the cluster
rg -B 3 -A 10 "\.Resources\s*=" internal/controller/modelsasservice_controller.go 2>/dev/null | head -60 || echo "File not found, searching differently..."

Repository: opendatahub-io/opendatahub-operator

Length of output: 61


🏁 Script executed:

# Search more broadly for resource deployment
find internal -name "*controller.go" | xargs rg -A 15 "ReconciliationRequest" | grep -A 10 "Create\|Apply\|Deploy" | head -80

Repository: opendatahub-io/opendatahub-operator

Length of output: 2153


🏁 Script executed:

# Find the main modelsasservice controller to understand resource deployment
cat internal/controller/components/modelsasservice/modelsasservice_controller.go | head -150

Repository: opendatahub-io/opendatahub-operator

Length of output: 4843


🏁 Script executed:

# Search for how resources from rr.Resources are deployed in the base controller
find internal -name "*.go" -type f | xargs grep -l "rr.Resources" | head -5

Repository: opendatahub-io/opendatahub-operator

Length of output: 389


🏁 Script executed:

# Look at the types.ReconciliationRequest to understand how Resources are used
rg -A 20 "type ReconciliationRequest" internal/

Repository: opendatahub-io/opendatahub-operator

Length of output: 61


🏁 Script executed:

# Check if there's any logic that filters or deduplicates Telemetry resources
rg -B 5 -A 10 "Telemetry.*filter\|dedup\|merge\|conflict" internal/controller/

Repository: opendatahub-io/opendatahub-operator

Length of output: 61


🏁 Script executed:

# Find where ReconciliationRequest is defined
find . -name "*.go" -type f | xargs grep -l "type ReconciliationRequest struct" | head -1

Repository: opendatahub-io/opendatahub-operator

Length of output: 110


🏁 Script executed:

# Look at the deploy.NewAction to understand how rr.Resources are deployed
find . -name "*.go" -type f | xargs grep -l "deploy.NewAction\|DeployAction" | head -3

Repository: opendatahub-io/opendatahub-operator

Length of output: 309


🏁 Script executed:

# Check the deploy action implementation
find . -path "*/actions/deploy/*" -name "*.go" | head -3

Repository: opendatahub-io/opendatahub-operator

Length of output: 242


🏁 Script executed:

# Look at how resources are applied - search for Create/Patch patterns
rg -A 15 "func.*Apply\|func.*Create.*Resource" pkg/controller/actions/deploy/ 2>/dev/null | head -60

Repository: opendatahub-io/opendatahub-operator

Length of output: 61


🏁 Script executed:

# Check if the deploy action handles resource conflicts or applies them as-is
cat pkg/controller/actions/deploy/action.go 2>/dev/null | head -100 || echo "File not found"

Repository: opendatahub-io/opendatahub-operator

Length of output: 61


🏁 Script executed:

# Read the ReconciliationRequest type definition
cat -n pkg/controller/types/types.go | head -100

Repository: opendatahub-io/opendatahub-operator

Length of output: 3785


🏁 Script executed:

# Read the deploy action implementation
cat -n pkg/controller/actions/deploy/action_deploy.go | head -150

Repository: opendatahub-io/opendatahub-operator

Length of output: 4945


🏁 Script executed:

# Search for how resources are created/applied in deploy action
rg -B 5 -A 15 "Apply\|Create.*Unstructured\|client\.Create" pkg/controller/actions/deploy/

Repository: opendatahub-io/opendatahub-operator

Length of output: 61


🏁 Script executed:

# Check if there's any validation or conflict detection for Telemetry
rg -B 5 -A 10 "ConflictingTelemetry\|IST0159\|overlapping.*selector" pkg/

Repository: opendatahub-io/opendatahub-operator

Length of output: 61


🏁 Script executed:

# Continue reading the deploy action implementation
cat -n pkg/controller/actions/deploy/action_deploy.go | sed -n '150,250p'

Repository: opendatahub-io/opendatahub-operator

Length of output: 4010


🏁 Script executed:

# Look for error handling or conflict detection in deploy
rg -A 20 "lookupErr\|Get.*current\|Create\|Patch" pkg/controller/actions/deploy/action_deploy.go | head -100

Repository: opendatahub-io/opendatahub-operator

Length of output: 61


🏁 Script executed:

# Check the full deploy action to see if resources are blindly applied or with conflict detection
wc -l pkg/controller/actions/deploy/action_deploy.go

Repository: opendatahub-io/opendatahub-operator

Length of output: 129


🏁 Script executed:

# Read the rest of the deploy action
cat -n pkg/controller/actions/deploy/action_deploy.go | sed -n '150,300p'

Repository: opendatahub-io/opendatahub-operator

Length of output: 5508


🏁 Script executed:

# Look at the full deploy function to see if there's any Telemetry-specific handling
cat -n pkg/controller/actions/deploy/action_deploy.go | sed -n '284,350p'

Repository: opendatahub-io/opendatahub-operator

Length of output: 2755


🏁 Script executed:

# Check if there's multiple ModelsAsService instances or if only one can exist
rg "ModelsAsServiceInstanceName\|ModelsAsService" internal/controller/components/modelsasservice/ | grep -v "test.go" | grep -v "\.go-"

Repository: opendatahub-io/opendatahub-operator

Length of output: 61


🏁 Script executed:

# Look for any documentation or comments about single instance
cat -n internal/controller/components/modelsasservice/modelsasservice_support.go | head -50

Repository: opendatahub-io/opendatahub-operator

Length of output: 2422


🏁 Script executed:

# Check if the Telemetry name is derived from anything or if it's always the same
rg -B 5 "IstioTelemetryName.*=" internal/controller/components/modelsasservice/

Repository: opendatahub-io/opendatahub-operator

Length of output: 830


🏁 Script executed:

# Check if ModelsAsServiceInstanceName is defined and if it's a singleton
rg "ModelsAsServiceInstanceName" api/components/v1alpha1/ | head -20

Repository: opendatahub-io/opendatahub-operator

Length of output: 200


🏁 Script executed:

# Look for how ModelsAsService instances are managed - can there be multiple?
rg -B 5 -A 5 "ModelsAsService.*singleton\|ModelsAsService.*instance" internal/ api/

Repository: opendatahub-io/opendatahub-operator

Length of output: 61


🏁 Script executed:

# Check the CRD definition to understand cardinality constraints
find . -path "*/api/*" -name "*modelsasservice*.go" | head -5

Repository: opendatahub-io/opendatahub-operator

Length of output: 129


🏁 Script executed:

# Read the ModelsAsService types definition
cat -n api/components/v1alpha1/modelsasservice_types.go | head -100

Repository: opendatahub-io/opendatahub-operator

Length of output: 5080


🏁 Script executed:

# Look for any validation or uniqueness constraints
rg "kubebuilder:validation\|webhook\|unique\|singleton" api/components/v1alpha1/modelsasservice_types.go

Repository: opendatahub-io/opendatahub-operator

Length of output: 61


🏁 Script executed:

# Check if there's any constraint that only one ModelsAsService can exist
rg -B 5 -A 5 "ModelsAsServiceInstanceName" api/components/v1alpha1/modelsasservice_types.go

Repository: opendatahub-io/opendatahub-operator

Length of output: 438


🏁 Script executed:

# Check if there's error handling for Istio validation failures
rg "IST0159\|ConflictingTelemetry\|validation.*error\|admission" internal/controller/components/modelsasservice/ || echo "No error handling found"

Repository: opendatahub-io/opendatahub-operator

Length of output: 102


🏁 Script executed:

# Look for any tests that demonstrate the conflict scenario
rg -l "Telemetry.*conflict\|overlap" internal/controller/components/modelsasservice/ || echo "No conflict tests found"

Repository: opendatahub-io/opendatahub-operator

Length of output: 102


🏁 Script executed:

# Check if there are any pre-deployment validations or checks
rg -B 5 -A 10 "configureIstioTelemetry\|Telemetry.*validation\|Get.*Telemetry" internal/controller/components/modelsasservice/modelsasservice_controller_actions.go

Repository: opendatahub-io/opendatahub-operator

Length of output: 61


🏁 Script executed:

# Let me verify the complete picture - check if GatewayRef can vary or if it's also fixed
rg "GatewayRef" api/components/v1alpha1/modelsasservice_types.go -A 5 | head -20

Repository: opendatahub-io/opendatahub-operator

Length of output: 911


🏁 Script executed:

# Check if there's any scenario where multiple Telemetry could be created from the same ModelsAsService
rg -B 10 -A 10 "configureIstioTelemetry\|IstioTelemetryName" internal/controller/components/modelsasservice/modelsasservice_controller.go

Repository: opendatahub-io/opendatahub-operator

Length of output: 61


🏁 Script executed:

# Verify the flow - is configureTelemetryPolicy different from configureIstioTelemetry?
cat -n internal/controller/components/modelsasservice/modelsasservice_controller_actions.go | sed -n '240,280p'

Repository: opendatahub-io/opendatahub-operator

Length of output: 1849


Pre-validate Telemetry selector conflicts before deployment instead of relying on Istio admission rejection.

The code creates an Istio Telemetry resource with a gateway-specific selector and blindly appends it without checking for overlapping workload selectors in the gateway namespace. Istio treats overlapping selectors in the same namespace as IST0159 ConflictingTelemetryWorkloadSelectors and will reject the configuration. While the ModelsAsService CRD enforces a singleton instance (name validation), this doesn't prevent conflicts with Telemetry from other controllers or user-created resources in the gateway namespace. The deployment will fail without a clear error signal to the user.

Implement pre-deployment validation: either query existing Telemetry resources in the target namespace for selector conflicts and surface a condition on the ModelsAsService status, or consolidate metrics overrides into a single controller-owned Telemetry resource merged across reconciliations.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@internal/controller/components/modelsasservice/modelsasservice_controller_actions.go`
around lines 381 - 384, The Telemetry creation code that builds a selector using
gatewayName should pre-validate for selector conflicts instead of blindly
applying it; before appending the Telemetry with the selector map (the block
that sets "selector": map[string]any{"matchLabels":
{"gateway.networking.k8s.io/gateway-name": gatewayName}}) list existing Istio
Telemetry resources in the target namespace and compare their workload selectors
for overlap with the new selector; if an overlap/conflict is found, do not apply
the new Telemetry and surface a ModelsAsService status condition (use the
controller's status update / SetCondition logic on the ModelsAsService resource)
describing the conflicting Telemetry and why deployment is blocked, or
alternatively implement merging by owning a single controller-named Telemetry
resource (e.g., controller-owned Telemetry) and reconcile/merge required
selector/metric overrides into that single resource instead of creating separate
Telemetries per gateway.

@ishitasequeira
ishitasequeira force-pushed the feature/deployment-based-observability-toggle branch from 58a85d3 to b808976 Compare April 2, 2026 15:17

@jland-redhat jland-redhat left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets validate with Lindani but I think we don't want this to be opt-in

Comment thread api/components/v1alpha1/modelsasservice_types.go Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
internal/controller/components/modelsasservice/modelsasservice_controller_actions.go (1)

380-383: ⚠️ Potential issue | 🟠 Major

Selector-conflict handling is missing before Telemetry append (IST0159 failure path).

Line 380–383 assumes singleton/name/ownerRef mitigate conflicts, but they do not prevent overlap with pre-existing Telemetry resources in the same namespace. Admission rejection here can cause repeated reconcile failures without a clear, actionable status on the ModelsAsService CR. Pre-validate selector overlap and set a status condition before appending the resource.

#!/bin/bash
# Verify there is no selector-conflict prevalidation/status condition path
# before appending Istio Telemetry resources.

rg -n -C3 'configureIstioTelemetryCore|rr\.Resources = append\(rr\.Resources, \*istioTelemetry\)|IST0159|ConflictingTelemetry|SetCondition|Condition' \
  internal/controller/components/modelsasservice/modelsasservice_controller_actions.go \
  internal/controller/components/modelsasservice/modelsasservice_controller.go

As per coding guidelines, **: REVIEW PRIORITIES: 1. Security vulnerabilities (provide severity, exploit scenario, and remediation code) 3. Bug-prone patterns and error handling gaps 4. Performance problems.

Also applies to: 436-437

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@internal/controller/components/modelsasservice/modelsasservice_controller_actions.go`
around lines 380 - 383, Pre-validate selector overlap before appending the Istio
Telemetry resource: inside the code path that calls configureIstioTelemetryCore
and before rr.Resources = append(rr.Resources, *istioTelemetry) (and the similar
block around lines 436–437), query existing Telemetry resources in the target
namespace and check selectors for conflicts; if a conflict is detected, do not
append the resource and instead set a status condition on the ModelsAsService CR
(use your SetCondition/Status update helpers, e.g. create a ConflictingTelemetry
condition with helpful message and reason IST0159) so reconciliation surfaces a
clear, actionable error and avoids admission rejections and retry storms.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In
`@internal/controller/components/modelsasservice/modelsasservice_controller_actions.go`:
- Around line 415-417: The subscription header is being used directly as a
metric label ("subscription": request.headers["x-maas-subscription"]) which
permits header spoofing and high-cardinality data; modify the upsert logic in
the ModelsasService controller action to not trust raw
request.headers["x-maas-subscription"] — instead retrieve the subscription from
a validated auth/context source (e.g., the authenticated user/session object) or
validate/normalize the header against a strict allowlist of expected
subscription IDs before including it, and if it fails validation fall back to a
safe low-cardinality value (e.g., "unknown" or omit the label); ensure this
change is applied where the metric map with key "subscription" is built so
high-cardinality labels are never emitted.

---

Duplicate comments:
In
`@internal/controller/components/modelsasservice/modelsasservice_controller_actions.go`:
- Around line 380-383: Pre-validate selector overlap before appending the Istio
Telemetry resource: inside the code path that calls configureIstioTelemetryCore
and before rr.Resources = append(rr.Resources, *istioTelemetry) (and the similar
block around lines 436–437), query existing Telemetry resources in the target
namespace and check selectors for conflicts; if a conflict is detected, do not
append the resource and instead set a status condition on the ModelsAsService CR
(use your SetCondition/Status update helpers, e.g. create a ConflictingTelemetry
condition with helpful message and reason IST0159) so reconciliation surfaces a
clear, actionable error and avoids admission rejections and retry storms.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 85ac3a70-9e5b-4ed9-9112-ce5317cebfdb

📥 Commits

Reviewing files that changed from the base of the PR and between b808976 and 853e0a9.

📒 Files selected for processing (1)
  • internal/controller/components/modelsasservice/modelsasservice_controller_actions.go

@ishitasequeira
ishitasequeira force-pushed the feature/deployment-based-observability-toggle branch from 853e0a9 to 22b27e6 Compare April 2, 2026 16:01
@ishitasequeira

Copy link
Copy Markdown
Contributor Author

/retest

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (2)
api/components/v1alpha1/modelsasservice_types.go (1)

76-82: ⚠️ Potential issue | 🟠 Major

enabled becomes opt-out for existing spec.telemetry users.

Line 80 defaults the new field to true. TelemetryConfig already existed before this PR, so any CR that already has a spec.telemetry block but omits enabled can start reconciling the new Istio Telemetry resource once CRD defaulting applies. For a technical-preview path, keep nil/false as disabled and require an explicit opt-in.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@api/components/v1alpha1/modelsasservice_types.go` around lines 76 - 82, The
new Enabled field on TelemetryConfig defaults to true which causes existing CRs
that omit spec.telemetry.enabled to be treated as opt-out; remove the
"+kubebuilder:default=true" annotation (or change it so no CRD default is
applied) on the Enabled *bool field in TelemetryConfig (symbol: Enabled in
modelsasservice_types.go) so that omitted values remain nil, and update any
controller logic that reads TelemetryConfig.Enabled (e.g., any reconcile code)
to treat only explicit true as enabled (nil/false = disabled) to enforce
explicit opt-in for the technical-preview path.
internal/controller/components/modelsasservice/modelsasservice_controller_actions.go (1)

388-406: ⚠️ Potential issue | 🟠 Major

The selector-conflict mitigation here does not cover external Telemetry objects.

The singleton CR, fixed resource name, and owner refs only deduplicate this controller’s own object. Istio still rejects overlapping workload selectors from any other telemetry.istio.io/v1 Telemetry in the gateway namespace (ConflictingTelemetryWorkloadSelectors / IST0159), and this path does no preflight or targeted status reporting before append. Check for conflicting selectors first, or surface a condition that points to the conflicting resource.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@internal/controller/components/modelsasservice/modelsasservice_controller_actions.go`
around lines 388 - 406, The current creation of istioTelemetry (variable
istioTelemetry, using IstioTelemetryName, gatewayNamespace, gatewayName) can
still conflict with other telemetry. Before applying or patching that
unstructured Telemetry, list existing telemetry.istio.io/v1 Telemetry objects in
gatewayNamespace, inspect their spec.selector.matchLabels for overlap with the
selector you plan to use (gateway.networking.k8s.io/gateway-name ->
gatewayName), and if any conflict is found do not apply the new Telemetry;
instead set a status condition on the ModelsAsService CR (e.g.,
TelemetrySelectorConflictCondition) describing the conflicting Telemetry name(s)
and return an error/result that surfaces the conflict; if no conflicts, proceed
to create/patch istioTelemetry as currently implemented. Ensure you reference
the istioTelemetry object and IstioTelemetryName when implementing the check so
the same selector logic is used for the comparison.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In
`@internal/controller/components/modelsasservice/modelsasservice_controller_actions.go`:
- Around line 320-343: The telemetry enable check currently exists in
configureIstioTelemetry but configureTelemetryPolicy is still invoked
unconditionally from the reconciler, causing usage-telemetry to be reconciled
even when maas.Spec.Telemetry.Enabled is false; update the reconcile flow so
both configureIstioTelemetry and configureTelemetryPolicy are gated by the same
helper (e.g., add or reuse a function like isTelemetryEnabled(ctx, rr or maas)
that checks maas.Spec.Telemetry != nil and Enabled != nil && *Enabled) and call
that helper before invoking configureTelemetryPolicy and configureIstioTelemetry
(or add the same nil/Enabled check at the top of configureTelemetryPolicy) so
neither path runs or performs CRD lookups when telemetry is disabled.

---

Duplicate comments:
In `@api/components/v1alpha1/modelsasservice_types.go`:
- Around line 76-82: The new Enabled field on TelemetryConfig defaults to true
which causes existing CRs that omit spec.telemetry.enabled to be treated as
opt-out; remove the "+kubebuilder:default=true" annotation (or change it so no
CRD default is applied) on the Enabled *bool field in TelemetryConfig (symbol:
Enabled in modelsasservice_types.go) so that omitted values remain nil, and
update any controller logic that reads TelemetryConfig.Enabled (e.g., any
reconcile code) to treat only explicit true as enabled (nil/false = disabled) to
enforce explicit opt-in for the technical-preview path.

In
`@internal/controller/components/modelsasservice/modelsasservice_controller_actions.go`:
- Around line 388-406: The current creation of istioTelemetry (variable
istioTelemetry, using IstioTelemetryName, gatewayNamespace, gatewayName) can
still conflict with other telemetry. Before applying or patching that
unstructured Telemetry, list existing telemetry.istio.io/v1 Telemetry objects in
gatewayNamespace, inspect their spec.selector.matchLabels for overlap with the
selector you plan to use (gateway.networking.k8s.io/gateway-name ->
gatewayName), and if any conflict is found do not apply the new Telemetry;
instead set a status condition on the ModelsAsService CR (e.g.,
TelemetrySelectorConflictCondition) describing the conflicting Telemetry name(s)
and return an error/result that surfaces the conflict; if no conflicts, proceed
to create/patch istioTelemetry as currently implemented. Ensure you reference
the istioTelemetry object and IstioTelemetryName when implementing the check so
the same selector logic is used for the comparison.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 289d6ac5-992e-4beb-abf1-46decd66753a

📥 Commits

Reviewing files that changed from the base of the PR and between 853e0a9 and 3668e73.

📒 Files selected for processing (5)
  • api/components/v1alpha1/modelsasservice_types.go
  • api/components/v1alpha1/zz_generated.deepcopy.go
  • docs/api-overview.md
  • internal/controller/components/modelsasservice/modelsasservice_controller_actions.go
  • internal/controller/components/modelsasservice/modelsasservice_controller_actions_test.go

@openshift-ci openshift-ci Bot added the lgtm label Apr 3, 2026
@openshift-ci

openshift-ci Bot commented Apr 3, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kahowell

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@codecov

codecov Bot commented Apr 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.94624% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.11%. Comparing base (7088315) to head (ae60516).
⚠️ Report is 7 commits behind head on main.

Files with missing lines Patch % Lines
...elsasservice/modelsasservice_controller_actions.go 86.81% 10 Missing and 2 partials ⚠️
...ents/modelsasservice/modelsasservice_controller.go 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3374      +/-   ##
==========================================
+ Coverage   50.94%   51.11%   +0.16%     
==========================================
  Files         199      199              
  Lines       14727    14817      +90     
==========================================
+ Hits         7502     7573      +71     
- Misses       6479     6497      +18     
- Partials      746      747       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD 1d1d27e and 2 for PR HEAD 6547b0a in total

@ishitasequeira
ishitasequeira force-pushed the feature/deployment-based-observability-toggle branch from 6547b0a to fcb11ec Compare April 6, 2026 12:31
@cgoodfred

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm label Apr 6, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD d3f4972 and 2 for PR HEAD fcb11ec in total

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD 357b98b and 1 for PR HEAD fcb11ec in total

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD 28ae457 and 0 for PR HEAD fcb11ec in total

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/hold

Revision fcb11ec was retested 3 times: holding

@StevenTobin

Copy link
Copy Markdown
Contributor

/retest

1 similar comment
@ishitasequeira

Copy link
Copy Markdown
Contributor Author

/retest

@cgoodfred

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm label Apr 7, 2026
@ishitasequeira
ishitasequeira force-pushed the feature/deployment-based-observability-toggle branch from 65ac0b4 to ae60516 Compare April 7, 2026 22:58
@openshift-ci openshift-ci Bot removed the lgtm label Apr 7, 2026
@cgoodfred

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm label Apr 8, 2026
@cgoodfred

Copy link
Copy Markdown
Contributor

/unhold

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD 2d7044a and 2 for PR HEAD ae60516 in total

@openshift-merge-bot
openshift-merge-bot Bot merged commit 115ed0a into opendatahub-io:main Apr 8, 2026
21 of 22 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in ODH Platform Planning Apr 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

6 participants