Ports defined in executor/driver config are not exposed on generated Services #2891
Description
What happened?
-
✋ I have searched the open/closed issues and my issue is not listed.
Version
spark-operator: v2.4.0
Description
When ports are specified in the executor or driver configuration of a SparkApplication CRD, the Services created for the executor/driver pods do not include those ports. As a result, the ports are unreachable via the Service, even though the pods themselves may have the containers listening on those ports.
Impact
This prevents exposing custom metrics endpoints, monitoring agents, or other services running inside Spark pods via Kubernetes Services. This is blocking on when using a service mesh (like istio) on strict mode.
Reproduction Code
- Define a
SparkApplicationwith ports in the driver or executor spec:
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
name: my-spark-app
spec:
type: Python
pythonVersion: "3"
mode: cluster
image: ...
arguments: ...
sparkConf: ...
driver:
ports:
- name: my-port
containerPort: 40001
protocol: TCP
executor:
ports:
- name: my-executor-port
containerPort: 8888
protocol: TCP- Submit the SparkApplication and observe the generated Services.
Expected behavior
The Services created for the driver and executor should include the ports specified in the driver.ports / executor.ports fields, making them reachable via the service.
Actual behavior
The generated Services do not include the ports. Only the ports that are created by the spark config appear on the Service.
Environment & Versions
- Kubernetes Version:
- Spark Operator Version:
- Apache Spark Version:
Additional context
No response
Impacted by this bug?
Give it a 👍 We prioritize the issues with most 👍