Unable to limit concurrent jobs on Kubernetes

[afayROR512]

Hello,

To start out, we are using the OSS version of Airbyte on EKS.

We have a partner that has an API that we are pulling data from. To limit how much we're hitting their API, we'd like to limit the number of concurrent jobs that are running to 10 (we have hundreds of connections set up that pull data throughout the day).

Looking through the documentation, I've found numerous ways to control this:

• MAX_SYNC_WORKERS we have set to 10
• MAX_CHECK_WORKERS we have set to 5
• TEMPORAL_WORKER_PORTS we limited to 9001-9010
• WORKLOAD_LAUNCHER_PARALLELISM we have set to 10
• worker.replicaCount: 1 in values.yaml

The other thing to note is that I have to set these variables manually in the ConfigMap after running helm install, because when setting extraEnv in the respective sections where each of these variables live, it fails to set the variables because it conflicts with the default value set in the ConfigMap in the chart. Seems like the chart itself might have a bug.

Anyway, after setting all these variables, restarting all the Airbyte service pods, and verifying those pods received the updated configuration, it still continues to schedule more than 10 jobs at a time. I don't know of any other parameters that would limit this, and it seems to be violating the behavior described in the docs.

Is there something that I'm missing? Any help would be greatly appreciated!

[Airbyte Support Bot (airbyte-support-bot)]

Hi! This discussion hasn't had activity in over a year. Is this still relevant to you? If so, please reply and we'll keep it open. If we don't hear back within 14 days, we'll close this as part of our housekeeping. You can always reopen or create a new discussion later. Thanks!

---

Devin session