Provide configurable Gardener Generic Extensions to Kyma users

[ruanxin]

Created on 2025-07-29 by Xin Ruan (@ruanxin ).

Decision log

Name	Description
Title	Provide Gardener Generic Extensions to Kyma users
Due date	2025-08-31
Status	Accepted on 2025-08-13
Decision type	Choice
Affected decisions	The Registry Cache ADR

Context

Gardener offers generic extensions to enhance cluster capabilities — for example:

• Falco: Improves security monitoring.
• Registry Cache: Speeds up image pulls and reduces external dependencies.

These extensions are configured in the Gardener Shoot CR via Gardener API.

However, Kyma end users do not have direct Gardener API access. Instead, the Kyma Infrastructure Manager (KIM) acts as the bridge between SKR and the Garden cluster.

The Registry Cache ADR established a pattern where KIM:

• Reads configuration from the Kyma runtime.
• Delivers necessary resources to the Garden cluster via the Gardener API.

We now aim to extend this pattern to other Gardener extensions, creating a general, unified solution for enabling Kyma users to configure selected extensions.

Decision Drivers

• Security: The solution must follow the principle of least privilege, minimizing access scopes and potential attack vectors.
• User Experience (UX): The process for a user to enable and configure an extension should be straightforward and consistent with existing Kyma and Kubernetes practices.
• Architectural Cohesion: The design should align with the existing Kyma and Gardener architecture, maintaining a clear separation of concerns between components.

Have a unified user experience

Treat extension as kyma regular module

apiVersion: operator.kyma-project.io/v1beta2
kind: Kyma
spec:
  channel: fast
  modules:
    - name: api-gateway
    - name: falco
    - name: registry-cache

Provide configuration resources to Gardener

To align with this decision and the Kyma module concept, the following architecture is proposed:

1. User is able to enable extension as a standard Kyma module in SKR Kyma CR
   1. Optional: allow the user to configure the extension through BTP cockpit while provisioning Kyma
2. KLM deploys the extension as a standard Kyma module. This module only contains the CRD and CR related to extension configuration.
3. An Extension Manager (not decided if is a single manager or multiple ones) runs in the KCP and is responsible for syncing extension configuration from the SKR to the KCP, updating the corresponding Runtime CR.
4. KIM reads the extension configuration from the Runtime CR and interacts with the Gardener API to fetch the shoot status.
   1. If there are additional external extension resources required, KIM access the SKR and send them to Gardener accordingly.
5. KIM updates the Runtime CR about the extension status.
6. The Extension Manager updates the extension CR in the SKR based on the status in the Runtime CR.

Decision

1. Scope

   • Applies only to Gardener extensions that:
     • Provide clear value to Kyma end users.
     • Require user configuration.
     • Can be safely exposed in Kyma.
   • In scope initially: Registry Cache, Falco.
   • Out of scope: Auditlog, DNS, and extensions not feasible for user exposure.

2. Unified Module Approach

   • All supported extensions will be implemented as regular Kyma modules.
   • Each extension must:
     1. Provide a Kyma-specific configuration CR with a schema tailored for end users.
     2. Follow the Kyma module release process, including versioning and essential information, e.g: documentation.
     3. Deliver the configuration CR and its CRD as core module resources.

3. Ownership

   • @kyma-project/framefrog team will implement and maintain Registry Cache and Falco.
   • Any future extension covered by this ADR will have a designated owner.

Consequences

Advantages

• Unified, familiar UX for enabling and configuring extensions.
• Leverages existing Kyma module tooling, release process, and lifecycle management.
• Maintains clear separation of concerns between user-level CRs and Gardener-level operations.

Disadvantages

• Each supported extension requires its own Kyma module with ongoing maintenance.
• Adds sync complexity between SKR, KCP, and Gardener.

References

https://gardener.cloud/docs/extensions/_index#generic-extensions
https://gardener.cloud/docs/gardener/managed_seed/#managedseeds-register-shoot-as-seed

[ruanxin]

Open questions

1. Support user preconfig extension during kyma provsion or not
2. How to monitor extension status
3. SLA for those extensions

[Disper]

@ruanxin thank you for creating this detailed concept! I'm afraid that the current architecture is unfortunately not unified and different depending on the extension we already configure in KIM (shoot-auditlog-service, shoot-oidc-service, shoot-dns-service, shoot-cert-service, shoot-networking-filter). Don't get me wrong, I appreciate that you're challenging the current approach. So, while I'm not sure that we could allow the configuration of all existing extensions generically (e.g., we can't afford losing audit logs by letting customers replace shoot-auditlog-service configuration), having that possibility for some could greatly reduce future maintenance effort.

For an extension to be considered generic IMO, we would need to:

1. Agree that customers will be able to learn whether their extension was configured properly or not, and if not, then what was the reason.
2. Agree that customers will be able to easily configure it. For example, if they are familiar with the configuration of shoot-networking-filter in BTP Cockpit, would we consider changing the experience to a generic one to be better? Would a customer be fine with knowing all the details required by Gardener in this particular extension config? Or can we as a managed solution, provide some value by hiding some complexity?
3. It should be explicitly whitelisted by Kyma organization (again, the auditlog example)

Before we decide to invest in a generic solution, it would be good to have a few (at least 2-3) extension examples that would look great to configure generically before we continue with the concept, so we don't fall into YAGNI trap.

[pbochynski]

I miss the description of the current state. I believe this ADR should be linked:
https://github.com/kyma-project/kyma-infrastructure-manager/blob/main/docs/adr/002-registry-cache.md

Now the questions:

1. Can we treat falco extension in the same way as registry cache?
2. Is there any other component than KIM allowed to configure Shoot CR in Garden project? (probably not)

[ruanxin]

Vote for extension enablement:

Based on our discussions, we’ve identified the following options for enabling Gardener extensions in Kyma. Please review and share your opinion:

1. Enabled by Default via Custom CR

   • The extension is enabled by default.
   • The customer is responsible for providing a dedicated CR for each Gardener extension, as proposed by the registry cache extension using a CustomConfig CR.
   • The corresponding CRD must already exist during provisioning time.

2. Treat as a standard Kyma Module

   • The extension is introduced as a standard Kyma module (no module manager).
   • The CRD and a default CR are provisioned by KLM.

3. Default Kyma Module

   • Builds on Option 2.
   • The extension is enabled by default.

4. Configurable via BTP Cockpit

   • Builds on Option 2.
   • The extension’s enablement is exposed as a configuration option in the BTP Cockpit.

Please share your opinion about above options.

[tobiscr]

Two points from Framefrogs:

• Customers should explicit opt-in to use such Gardener extensions (e.g. by clicking a checkbox in BTP cockpit or similar) . It can happen, that some extensions won't be fully backward compatible for existing runtimes and we should be able to decline the enablement of them. Also, some extensions can introduce some "side effects" for customers (e.g.some base-configuration required etc.) and some customers maybe don't want to use them.
• Installing extension-specific modules happens only and fully automatically when the customer opt-in for an extension to provide a smooth user experience. Such special modules could be added to KymaCR via KEB when customer opts-in to an extension. The module should not be removable by customers (e.g. maybe we don't even show them to the customer ) until the customer opt-put from the extension. When he opts-out the modjule should also be automatically removed (e.g. KEB could remove it from KymaCR).

[tobiscr]

thanks for the proposal, here our feedback:

• KLM provisions each extension as a standard Kyma module: 🆗 => Recommended enhancement: if a customer opts-in for an extension, KEB adds the module to the Kyma CR and KLM installs the module
• The Extension Manager reconciles the extension’s custom resource (CR) and updates the corresponding Runtime CR: 🆗 (the "Extension Manager" is currently part of the registry cache feature, but technically already in place)
• KIM reconciles the Runtime CR to configure the extension.: 🆗
• KIM fetches the required extension resources from the SKR and send to Gardener API: ❗ => Adjustment required: right now, technically it works a bit different:
  1. Extension Manager adds the information to RuntimeCR
  2. This modification of the RuntimeCR triggers KIM, KIM takes the information from RuntimeCR and copies it to Gardener API
• KIM updates the Kyma CR to reflect the extension's current status. ❗ => Adjustment required: The status update happens in the module-specifc CR by KIM, not in KymaCR.

[ruanxin]

@pbochynski

To answer you questions:

Can we treat falco extension in the same way as registry cache?

Yes, we can, and in fact would be simpler. I clarified with falco team, they recently provide the feature to allow user config falco rules as a ConfigMap in the shoot cluster, check the issue here: gardener/gardener-extension-shoot-falco-service#231, so we don't need to provide those extra resources to the garden cluster through API anymore. However, this have limitation, atm, the ConfigMap must be configured at kube-system namespace. I raised the concern and requested if they are possbile allow user config for different namespaces in the future.

And the extension configuration in Shoot CR looks like below:

  extensions:
    - type: shoot-falco-service
      providerConfig:
        apiVersion: falco.extensions.gardener.cloud/v1alpha1
        destinations:
          - name: stdout
        kind: FalcoServiceConfig
        rules:
          standard:
            - falco-rules
          custom:
            - shootConfigMap: my-custom-rules-configmap

Is there any other component than KIM allowed to configure Shoot CR in Garden project? (probably not)

No, currently only KIM takes responsibility for exchanging Shoot CR information to Garden.

[ruanxin]

Regarding the statement "Customers should explicitly opt-in to use such Gardener extensions", I clarified this with @tobiscr. Ideally, the concern about existing SKRs not being fully backward-compatible with the Gardener extension should not be an issue. Even if it does occur, the correct approach would be to treat it as an incident and remediate the affected customers so they can use the extensions. Therefore, there’s no need to implement an opt-in feature that explicitly limits our customers.

[a-thaler]

If we don't have an opt-in, then we at least should have an opt-out. Falco for example will run a daemonset on your shoot consuming resources of your nodes. For a dev scenario you might want to save that money

[ruanxin]

@a-thaler , yes, and if we treat the extension as normal kyma module, your concern can be addressed naturally. Users are free to choose which extension they want to enable.

[akgalwas]

Deliver the configuration CR and its CRD as core module resources.

Minor comment: for Registry Cache it doesn't make sense to provide any default config so the module will contain CRD only.