Skip to content

[services] Update pod priorities and give grafana a realistic memory request #15357

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cjllanwarne wants to merge 2 commits into
base: main
Choose a base branch
from
+29 −8
Open

[services] Update pod priorities and give grafana a realistic memory request #15357

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
586635e
reset pod priorities and give grafana a realistic memory request
cjllanwarne Mar 24, 2026
585f3d7
also bump prometheus
cjllanwarne Mar 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions ci/bootstrap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,14 @@ description: "For production services like apiserver, batch, etc."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: auxiliary
value: 700000
globalDefault: false
description: "For auxiliary services like CI, monitoring, grafana, prometheus."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: user
value: 500000
Expand Down
2 changes: 1 addition & 1 deletion ci/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ spec:
spec:
serviceAccountName: ci-agent
{% if deploy %}
priorityClassName: production
priorityClassName: auxiliary
{% endif %}
nodeSelector:
preemptible: "true"
Expand Down
22 changes: 17 additions & 5 deletions grafana/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,19 @@ spec:
labels:
app: grafana
spec:
priorityClassName: infrastructure
priorityClassName: auxiliary
{% if deploy %}
nodeSelector:
preemptible: "false"
{% else %}
nodeSelector:
preemptible: "true"
tolerations:
- key: preemptible
value: "true"
- key: "kubernetes.azure.com/scalesetpriority"
value: "spot"
{% endif %}
securityContext:
fsGroup: 472
volumes:
Expand Down Expand Up @@ -220,10 +232,10 @@ spec:
resources:
requests:
cpu: "10m"
memory: "20M"
memory: "100M"
limits:
cpu: "1"
memory: "1G"
memory: "200M"
ports:
- containerPort: 3000
- name: envoy
Expand All @@ -237,10 +249,10 @@ spec:
resources:
requests:
cpu: "10m"
memory: "20M"
memory: "50M"
limits:
cpu: "1"
memory: "1G"
memory: "100M"
ports:
- containerPort: 8443
volumeMounts:
Expand Down
1 change: 1 addition & 0 deletions monitoring/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ spec:
hail.is/sha: "{{ code.sha }}"
grafanak8sapp: "true"
spec:
priorityClassName: auxiliary
nodeSelector:
preemptible: "true"
tolerations:
Expand Down
4 changes: 2 additions & 2 deletions prometheus/prometheus.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ spec:
spec:
serviceAccountName: prometheus
{% if deploy %}
priorityClassName: infrastructure
priorityClassName: auxiliary
nodeSelector:
preemptible: "false"
{% else %}
Expand All @@ -290,7 +290,7 @@ spec:
fsGroup: 65534
containers:
- name: prometheus
image: prom/prometheus:v2.46.0
image: prom/prometheus:v2.53.5@sha256:7a34573f0b9c952286b33d537f233cd5b708e12263733aa646e50c33f598f16c
imagePullPolicy: Always
command:
- "/bin/prometheus"
Expand Down
Loading