Operator does not self-heal collector DaemonSet deletion, causing silent collection outage

[AnkanMisra]

Summary

If the collector DaemonSet gmp-system/collector is deleted or never created, the operator logs a warning and exits reconcile successfully without recreating it. Delete events are also filtered out by the controller’s watch predicates. This leaves the cluster without collectors indefinitely and without status surfacing a degraded state.

Steps to Reproduce

1. Deploy the operator normally so collectors are managed by the operator.
2. Delete the collector DaemonSet: kubectl -n gmp-system delete ds collector.
3. Wait for the operator to reconcile.

Observed Behavior

• Operator logs a warning that the DaemonSet does not exist, returns success, and does not recreate it. See ensureCollectorDaemonSet handling of NotFound in pkg/operator/collection.go#L219-L231.
• Delete events are not enqueued because the controller watches the DaemonSet with GenerationChangedPredicate, which ignores deletes; no reconcile is triggered on deletion (same file around pkg/operator/collection.go#L106-L113).
• Collection remains down permanently; no status condition reflects the loss of collectors.

Expected Behavior

• The operator should reconcile on DaemonSet deletion and recreate or at least mark degraded, so collectors come back automatically and the condition is visible to users.

Impact

• Cluster-wide metrics ingestion halts until a human notices and re-applies the DaemonSet. This is a silent data-loss mode: no collectors, no samples, and no operator status signal. In busy clusters, this can wipe out minutes to hours of data and break alerting without clear diagnosis.

Proposed Fix (high-level)

• Watch DaemonSet delete events (do not filter them out) and enqueue reconcile on delete.
• In ensureCollectorDaemonSet, treat NotFound as a reconcile error or recreate the DaemonSet; alternatively set a Degraded condition on the OperatorConfig status so the outage is surfaced.

Discussion

Please confirm whether this behavior is intentional or a bug

Happy to adjust the understanding or help with a fix/iteration

[bernot-dev]

Is there something in particular that deleted the DaemonSet? In general, we assume that users are not deleting GMP resources if they want it to function properly.

[AnkanMisra]

No specific incident - this came from reviewing the code and thinking about resilience against accidental deletions or CI/CD drift (e.g., misconfigured pipeline removes the DaemonSet). The concern is that there's no way to distinguish "user intentionally opted out" from "something went wrong," and no status condition surfaces the loss.

Would a PR adding a CollectorUnavailable status condition be welcome? That way at least the outage is visible, without changing the current behavior.

[bernot-dev]

Is it just to check if the DaemonSet exists? Or also the pods? What if the pods are unable to start? I think it will be important to clearly define exactly what the status covers. If it's just that the DaemonSet exists, I think the condition name should be more specific.

[AnkanMisra]

Fair call but What I was proposing was only existence of the collector DaemonSet like basically the gap in ensureCollectorDaemonSet today (NotFound returns nil) + the watch predicate ignoring deletes

Thats the silent outage im trying to surface, not general pod readiness, so im happy to name it something explicit like CollectorDaemonSetMissing / CollectorDaemonSetNotFound and document that it’s just existence

If you think we should cover pod scheduling/ready state too, I can widen scope, but I wanted to keep the first pass precise and unambiguous

[bernot-dev]

Makes sense. I would probably phrase the status in the affirmative (e.g. CollectorDaemonSetExists).

I'd be happy to review this change.

[AnkanMisra]

@bernot-dev PTAL

[bernot-dev]

Sorry about the delay here. My team has constrained capacity at the moment, but I will get to this.

[AnkanMisra]

@bernot-dev any updates?