Upgrade Kustomize manifests and align with Helm chart

[RobuRishabh]

What feature you would like to be added?

Proposing a comprehensive fix and alignment of the upstream Kustomize manifests with the Helm chart to make Kustomize a production-ready deployment option.

Current State: The upstream Kustomize manifests at config/ are fundamentally broken:

• Running kustomize build config/default/ fails entirely (missing entry point)
• Missing core deployment files (config/default/kustomization.yaml, config/manager/ directory, config/rbac/kustomization.yaml)
• Multiple critical bugs in existing manifests (webhook configs, RBAC, CRD patches)
• Missing RBAC permissions (PVCs, ResourceQuotas)
• No CI validation to prevent future drift

Result: Kustomize cannot deploy the operator. Only Helm works upstream.

Following the community decision in PR #2702 and #2820 to maintain Kustomize manifests, we've completed a comprehensive audit and identified four critical gap categories:

1: Deployment Incompleteness

Examples of missing essential files that prevent any deployment:

• config/default/kustomization.yaml - main entry point that composes all resources
• config/manager/ directory - controller Deployment, ServiceAccount, Namespace

2: Logic and Naming Bugs

Examples of syntax errors and configuration mistakes in existing files:

• Duplicate CRD patches (partially fixed in PR Fix duplicate webhook patch and add missing ScheduledSparkApplication Patches #2820, awaiting merge)
• Missing SparkConnect CRD in kustomization

3: RBAC Permission Divergence

Examples of security gaps and missing permissions compared to Helm chart:

• Missing PVC permissions (breaks PVC-backed Spark applications)
• Missing ResourceQuota permissions (quota enforcement broken)

4: Kustomize Feature Parity Deficit

Examples of production features only available in Helm:

• No PodDisruptionBudget (operator can be disrupted during maintenance)
• No Prometheus PodMonitor (manual metrics setup required)

Why is this needed?

No response

Describe the solution you would like

No response

Describe alternatives you have considered

No response

Additional context

No response

Love this feature?

Give it a 👍 We prioritize the features with most 👍