feat(deployment): make backend jvm options configurable

[clonder]

follow up to #7077 (comment). the backend heap share becomes a backendMaxRamPercentage value (default 75, same as main, the other jvm flags stay fixed in the template), and values_preview_server.yaml sets 25 so previews go back to the jvm default heap. checked with helm lint --strict and helm template for default, e2e and preview values, the only rendered difference to main is MaxRAMPercentage=75.0 becoming =75. the new top level key changes valuesHash on every deployment, so the first upgrade after this restarts all pods once. lapis keeps its own hardcoded JVM_OPTS, out of scope here.

PR Checklist

• All necessary documentation has been adapted.
  • schema description, shows up on the helm chart config docs page
• The implemented feature is covered by appropriate, automated tests.
  • deployment config
• Any manual testing that has been done is documented (i.e. what exactly was tested?)
  • helm lint and template with default, e2e and preview values, schema rejects 0, 150 and non numbers

🚀 Preview: Add preview label to enable

[claude]

Claude encountered an error after 3m 27s —— View job

---

I'll analyze this and get back to you.

[claude]

Claude encountered an error after 3m 10s —— View job

---

I'll analyze this and get back to you.

[claude]

Claude finished @clonder's task in 2m 39s —— View job

---

Review: feat(deployment): make backend jvm options configurable

• Read PR diff against origin/main
• Check template rendering path and schema wiring
• Check docs generation for the new key
• Post findings

The mechanics are correct. backendMaxRamPercentage is defined in values.yaml, validated by the schema (so a missing/null/non-numeric value fails at install rather than producing a broken -XX:MaxRAMPercentage=), and "groups": ["services"] matches an existing group rendered by docs/src/content/docs/reference/helm-chart-config.mdx:41, with docs/package.json copying the schema at build time — so the docs page picks it up with no extra work. Top-level placement is consistent with the existing backendExtraArgs. Default 75 preserves current behaviour everywhere except previews.

Two comments left inline:

1. The preview 25 is the only behavioural change, and it's the one thing not tested. It cuts the preview heap ceiling from ~2.25Gi to ~768Mi, while the MaxHeapFreeRatio=5/MinHeapFreeRatio=2 flags already in JVM_OPTS mean the JVM was returning idle heap to the OS anyway — so the memory saving on the shared cluster may be small while the OOM risk in the environment used for pre-merge testing is real. Lowering resources.backend.limits.memory in the same file caps the container footprint more directly.
2. maximum: 100 in the schema permits a configuration that reliably gets the pod OOMKilled; suggested capping at 90.

Non-blocking: the PR title says "jvm options" but exactly one option is configurable. If more are expected later, a backendJvm: map would avoid growing the top-level namespace one key at a time — not worth changing for a single value. The valuesHash one-time restart and LAPIS keeping its hardcoded JVM_OPTS are both correctly called out in the description.

I could not run helm template/helm lint myself — those commands need approval in this environment, so my review of the rendering is static. CI runs both (.github/workflows/helm-schema-lint.yaml) against default, e2e and preview values.
· branch feat/backend-heap-percentage

[claude]

Preview resources.backend.limits.memory is 3Gi, so this drops the backend heap ceiling from ~2.25Gi to ~768Mi. Two things worth checking before merging:

• MaxRAMPercentage is a ceiling, not a reservation. Combined with the already-set MaxHeapFreeRatio=5 -XX:MinHeapFreeRatio=2, the JVM already returns unused heap to the OS, so the steady-state RSS saving on the shared cluster may be near zero — the flag only bites once the heap actually needs to grow. What it does change is where the backend throws OutOfMemoryError.
• Previews are the main pre-merge test environment. With 25% here and 75% in prod, a preview can OOM on a workload (large submission, big seqSet export) that prod handles fine, and the failure will look like an app bug. If the goal is to bound what a preview can consume on the shared cluster, lowering resources.backend.limits.memory in this same file is the knob that actually caps the container footprint — and the heap then scales with it automatically at 75%.

Was the 25% figure validated against an actual preview run (e2e or a manual submission), or only via helm template? The PR checklist says manual testing was helm lint/template only, which doesn't cover the runtime effect that is the entire point of the change.

[fhennig]

If 25% is the default, I guess no validation is needed, since setting 25% is actually as if we wouldn't change anything, right? That's what I got from the PR description.

[fhennig]

Looks good to me!

I guess there could be more written about why we picked 75% as the default, but I'm also ok with just picking it now and running with it - as you said it's already configured like this on main. It's hard to pick good defaults.

Another choice would be 25%, the JVM default, if we really wanted to play it safe.

But I'm fine with 75%

I'm approving but maybe wait a few days to give Cornelius some time to have a look as well!

[claude]

Claude finished @anna-parker's task in 2m 45s —— View job

---

Review: feat(deployment): make backend jvm options configurable

• Read PR diff against origin/main
• Check template + schema + values wiring
• Check docs rendering, preview resource limits, other JVM_OPTS users
• Post findings

The wiring is correct. backendMaxRamPercentage is defined in values.yaml with the same 75 that main hardcodes, validated by the schema, and "groups": ["services"] matches the group rendered at docs/src/content/docs/reference/helm-chart-config.mdx:41, so the docs page picks it up. MaxRAMPercentage doesn't affect object layout, so the "mirror it in the test task" comment above JVM_OPTS correctly needs no change in backend/build.gradle (only UseCompactObjectHeaders is mirrored there, line 148).

On @fhennig's question — yes, that's right, and it resolves my earlier concern. The JVM's own default for MaxRAMPercentage is 25.0, and backend/entrypoint.sh:7 only passes JVM_OPTS when non-empty, so previews land on exactly the heap ceiling they had before these flags were introduced. The other four flags still apply in previews, only the ceiling reverts. The remaining trade-off is just that previews (3Gi limit → ~768Mi heap) and prod (~2.25Gi) now differ in where the backend hits OutOfMemoryError, so a memory-heavy regression can show up in a preview but not prod, or vice versa. Given previews ran at 25% for a long time, that risk is low — I'd drop my earlier suggestion to validate the number at runtime.

Two inline comments, both minor:

1. values.schema.json — the docs table renders only Field/Type/Default/Description, so minimum/maximum are invisible to readers; suggest folding the range (and what the percentage is of) into the description.
2. loculus-backend.yaml:103 — backendMaxRamPercentage: null in an override is dropped by Helm's coalesce before validation, so it passes lint and renders an empty -XX:MaxRAMPercentage=, crashlooping the pod. A | default 75 or a root required entry closes it. Fix this →

Non-blocking: the value is meaningless without resources.backend.limits.memory, so resources.backend.maxRamPercentage would keep the two knobs together instead of adding a top-level key — but top-level matches the existing backendExtraArgs, so either is defensible. LAPIS keeping its own hardcoded 75.0 (lapis-deployment.yaml:41) and the one-time valuesHash restart are both correctly called out in the description.

I could not run helm lint/helm template — those need approval in this environment, so my review of the rendering is static. CI covers all three values combinations in .github/workflows/helm-schema-lint.yaml. Note that if you take the schema suggestion, kubernetes/AGENTS.md asks for npx prettier@3.6.2 --write kubernetes/loculus/values.schema.json afterwards.
· branch feat/backend-heap-percentage

[claude]

The generated docs table only renders Field / Type / Default / Description (docs/src/components/SchemaDocs.astro:130-145) — minimum and maximum are not emitted, so an operator reading the Helm chart config page has no idea 1–100 is the accepted range, or what the value is a percentage of. Worth putting both into the description:

Suggested change

	"description": "Share of the backend container memory limit the JVM may use as heap, passed as -XX:MaxRAMPercentage."
	"description": "Share of the backend container memory limit (`resources.backend.limits.memory`) the JVM may use as heap, passed as -XX:MaxRAMPercentage. Must be between 1 and 100."

[claude]

One gap the schema doesn't close: backendMaxRamPercentage is not in the schema's root required list (which is just ["name"], values.schema.json:1764). Helm's coalesce deletes keys set to null in an override file before validation runs, so backendMaxRamPercentage: null passes lint and renders -XX:MaxRAMPercentage= — the JVM then exits at startup with Improperly specified VM option and the pod crashloops with no hint that a values key caused it. A template-side fallback makes that impossible:

Suggested change

	value: -XX:+UseContainerSupport -XX:MaxRAMPercentage={{ .Values.backendMaxRamPercentage }} -XX:+UseG1GC -XX:+UseCompactObjectHeaders -XX:MaxHeapFreeRatio=5 -XX:MinHeapFreeRatio=2
	value: -XX:+UseContainerSupport -XX:MaxRAMPercentage={{ .Values.backendMaxRamPercentage | default 75 }} -XX:+UseG1GC -XX:+UseCompactObjectHeaders -XX:MaxHeapFreeRatio=5 -XX:MinHeapFreeRatio=2

(Adding the key to the root required array instead is equally fine, and fails louder at install time rather than silently falling back.)

[corneliusroemer]

To avoid having to alter this again in the future for another arg that can be customized, I'd say we should just add an extra args placeholder that can be filled with whatever the maintainer wants to put in.

This should come last after all the other args so that one can override prior default args. Default should empty string if unset.

[corneliusroemer]

Thanks! I think there might have been a misunderstanding. By "We could make the JVM args configurable so that people can give as much RAM as they want." I meant to allow passing arbitrary JVM args through helm chart values rather than just the max ram percentage - this would avoid us having to do this for every arg one wants to override in the future.