Spark 4.1.x support?

[jamonation]

What question do you want to ask?

• ✋ I have searched the open/closed issues and my issue is not listed.

Quick question, I see the Dockerfile is pinned to Spark 4.0.0. Since spark-submit comes from there, is there a hard requirement on the 4.0.x series, or would using a 4.1.x image also work?

I know the script is running exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$@" but don't know if there are ABI compatibility issues between 4.0.x and 4.1.x or if the operator keys off the version somehow.

Additional context

No response

Have the same question?

Give it a 👍 We prioritize the question with most 👍

[github-actions]

🎉 Welcome to the Kubeflow Spark Operator! 🎉

Thanks for opening your first issue! We're happy to have you as part of our community 🚀

Here's what happens next:

• Our team will review your issue soon!
• If you'd like to contribute to this issue, check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards.

Join the community:

• Slack: Join our #kubeflow-spark-on-kubernetes Slack channel.

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[tariq-hasan]

I don't believe it would be an issue to use a 4.1.x image in the Dockerfile.

The only actual dependency on the version comes from pod template usage but the cutoff there is 3.0.0: https://github.com/kubeflow/spark-operator/blob/master/internal/webhook/sparkapplication_validator.go#L176.