CHANGELOG.md

Changelog

v2.4.0 (2025-11-14)

Features

• fix(chart): add revisionHistoryLimit option (#2625 by @t3mi)
• Suspend/Resume feature on SparkApplication (#2387 by @everpeace)
• Set ControllerReference on driver pod and non-Controller OwnerReference on executor pod at submission time (#2670 by @everpeace)
• added SparkApp name validator to accept valid DNS-1035 format (#2711 by @aryankumar04)
• No propagate Kueue labels to driver and executor pods (#2714 by @everpeace)
• feat: add support for customizing connect service (#2709 by @ChenYi015)

Bug Fixes

• Grant get/patch permissions for SparkConnect CRD to hook ClusterRole (#2605 by @ChenYi015)
• Grant create events permissions to Controller (#2616 by @Rockawear)
• fix(connect): propagate executor memory setting to spark options (#2656 by @mrjoe7)
• [fix] add miss cli params. Add leader election params in chart (#2657 by @aagumin)
• fix: webhook encoder configuration (#2664 by @pvbouwel)
• bugfix: hook.upgradeCrd use cases (#2663 by @pvbouwel)
• Correct entrypoint.sh for Openshift (#2645 by @Reamer)
• Fix driver host configuration to handle IPv6 addresses (#2703 by @tiagotxm)
• Add permissions for sparkconnects/finalizers (#2727 by @ChenYi015)
• Remove redundant name and namespace context in logs (#2723 by @ChenYi015)
• Logging info when resources associated with SparkApplication still exist (#2725 by @ChenYi015)

Unit Tests

• Add kube-scheduler podgroup unit tests (#2689 by @shadowinlife)
• Generate HTML coverage report after running unit tests (#2691 by @shadowinlife)
• Avoid 409 (Conflict) error at e2e tests with retry. (#2695 by @shadowinlife)
• Add unit test for sparkapplication_validator (#2692 by @shadowinlife)
• Add unit test for scheduledsparkapplication_validator (#2694 by @shadowinlife)
• Refactor the unit tests for web_ui.go (#2696 by @shadowinlife)

Refactor

• unify SparkApplication defaulting logic (#2671 by @zhzhuang-zju)
• Replace strconv.Atoi with strconv.ParseInt (#2699 by @ChenYi015)
• refactor: use ptr.To to replace util.XxxPtr (#2693 by @ChenYi015)

Dependencies

• Bump actions/checkout from 4 to 5 (#2623 by @dependabot[bot])
• Bump actions/download-artifact from 4 to 5 (#2624 by @dependabot[bot])
• Bump helm.sh/helm/v3 from 3.18.4 to 3.18.5 (#2627 by @dependabot[bot])
• Bump github.com/go-viper/mapstructure/v2 from 2.3.0 to 2.4.0 (#2635 by @dependabot[bot])
• Bump actions/setup-go from 5 to 6 (#2651 by @dependabot[bot])
• Bump actions/stale from 9 to 10 (#2649 by @dependabot[bot])
• Bump aquasecurity/trivy-action from 0.32.0 to 0.33.1 (#2650 by @dependabot[bot])
• Bump github.com/golang/glog from 1.2.4 to 1.2.5 (#2620 by @dependabot[bot])
• Bump github.com/onsi/ginkgo/v2 from 2.23.3 to 2.26.0 (#2660 by @dependabot[bot])
• Bump sigs.k8s.io/scheduler-plugins from 0.31.8 to 0.32.7 (#2675 by @dependabot[bot])
• Bump github/codeql-action from 3 to 4 (#2672 by @dependabot[bot])
• Bump github.com/stretchr/testify from 1.10.0 to 1.11.1 (#2673 by @dependabot[bot])
• Bump golang.org/x/time from 0.9.0 to 0.14.0 (#2686 by @dependabot[bot])
• Bump github.com/prometheus/client_golang from 1.22.0 to 1.23.2 (#2682 by @dependabot[bot])
• Bump github.com/spf13/viper from 1.20.1 to 1.21.0 (#2683 by @dependabot[bot])
• Bump github/codeql-action from 3.29.2 to 4.31.0 (#2684 by @dependabot[bot])
• Bump github.com/containerd/containerd from 1.7.27 to 1.7.29 (#2712 by @dependabot[bot])
• Bump sigs.k8s.io/yaml from 1.5.0 to 1.6.0 (#2704 by @dependabot[bot])
• Bump github.com/spf13/cobra from 1.9.1 to 1.10.1 (#2705 by @dependabot[bot])
• Bump github/codeql-action from 4.31.0 to 4.31.2 (#2707 by @dependabot[bot])
• Bump actions/upload-artifact from 4 to 5 (#2681 by @dependabot[bot])
• Bump actions/download-artifact from 5 to 6 (#2680 by @dependabot[bot])
• Bump actions/checkout from 4.2.2 to 5.0.0 (#2685 by @dependabot[bot])
• Bump ossf/scorecard-action from 2.4.2 to 2.4.3 (#2687 by @dependabot[bot])
• Bump github.com/onsi/ginkgo/v2 from 2.26.0 to 2.27.2 (#2719 by @dependabot[bot])
• Bump golang.org/x/mod from 0.27.0 to 0.29.0 (#2720 by @dependabot[bot])
• Bump helm/chart-testing-action from 2.7.0 to 2.8.0 (#2722 by @dependabot[bot])

Misc

• Add changelog for v2.3.0 (#2614 by @ChenYi015)
• Update README (#2634 by @ChenYi015)
• Add Spark Operator Roadmap (#2622 by @ChenYi015)
• CI: Add Scorecard workflow (#2654 by @AdamKorcz)
• Update GitHub action workflow permissions (#2678 by @ChenYi015)

Full Changelog

v2.3.0 (2025-07-22)

Features

• Add support for Spark Connect (#2569 by @ChenYi015)
• upgrade to Spark 4.0.0 (#2564 by @nabuskey)
• Make logging encoder configurable (#2580 by @ChenYi015)
• Include pod.Status.Message in recordExecutorEvent (#2589 by @matschaffer-roblox)
• Add print columns for Spark Connect (#2592 by @nabuskey
• chore: update prometheus pattern and labels for structured streaming driver (#2581 by @yahwang)
• Add Helm hook to upgrade CRDs (#2371 by @ChenYi015)

Bug Fixes

• Splat recordExecutorEvent args for cleaner event messages (#2582 by @matschaffer-roblox)
• fix: should add executor env when driver env is empty (#2586 by @ChenYi015)
• Add web UI configurations when enabling UI service and ingress (#2599 by @ChenYi015)
• Grant get/patch permissions for SparkConnect CRD to hook ClusterRole (#2605 by @ChenYi015)

Dependencies

• Bump aquasecurity/trivy-action from 0.31.0 to 0.32.0 (#2585 by @dependabot[bot])
• Bump github.com/go-viper/mapstructure/v2 from 2.2.1 to 2.3.0 (#2572 by @dependabot[bot])
• Bump sigs.k8s.io/yaml from 1.4.0 to 1.5.0 (#2577 by @dependabot[bot])
• Bump helm.sh/helm/v3 from 3.17.3 to 3.18.4 (#2587 by @dependabot[bot])
• Bump golang.org/x/mod from 0.25.0 to 0.26.0 (#2608 by @dependabot[bot])
• Bump github.com/onsi/gomega from 1.36.1 to 1.37.0 (#2607 by @dependabot[bot])

Misc

• Add SparkConnect e2e test (#2578 by @ChenYi015)
• feat(docs): Guide to report security vulnerabilities (#2593 by @andreyvelich)
• Read Helm version from go.mod file (#2598 by @ChenYi015)

Full Changelog

v2.2.1 (2025-06-27)

Features

• Customize ingress URL with Spark application ID (#2554 by @ChenYi015)
• Make default ingress tls and annotations congurable in the helm config (#2513 by @Tom-Newton)
• Use code-generator for clientset, informers, listers (#2563 by @jbhalodia-slack)

Misc

• add driver ingress unit tests (#2552 by @nabuskey)
• Get logger from context (#2551 by @ChenYi015)
• Update golangci lint (#2560 by @joshuacuellar1)

Dependencies

• Bump aquasecurity/trivy-action from 0.30.0 to 0.31.0 (#2557 by @dependabot[bot])
• Bump github.com/prometheus/client_golang from 1.21.1 to 1.22.0 (#2548 by @dependabot[bot])
• Bump sigs.k8s.io/scheduler-plugins from 0.30.6 to 0.31.8 (#2549 by @dependabot[bot])
• Bump golang.org/x/mod from 0.24.0 to 0.25.0 (#2566 by @dependabot[bot])
• Bump github.com/go-logr/logr from 1.4.2 to 1.4.3 (#2567 by @dependabot[bot])

v2.2.0 (2025-05-29)

Features

• Upgrade to Spark 3.5.5 (#2490 by @jacobsalway)
• Add timeZone to ScheduledSparkApplication (#2471 by @jacobsalway)
• Enable the override of MemoryLimit through webhook (#2478 by @danielrsfreitas)
• Add ShuffleTrackingEnabled to DynamicAllocation struct to allow disabling shuffle tracking (#2511 by @jbhalodia-slack)
• Define SparkApplicationSubmitter interface to allow customizing submitting mechanism (#2500 by @ChenYi015)
• Add support for using cert manager to generate webhook certificates (#2373 by @ChenYi015)

Bug Fixes

• fix: add webhook cert validity checking (#2489 by @teejaded)
• fix and add back unit tests (#2532 by @nabuskey)
• fix volcano tests (#2533 by @nabuskey)
• Add v2 to module path (#2515 by @ChenYi015)
• #2525 spark metrics in depends on prometheus (#2529 by @blcksrx)

Misc

• Add APRA AMCOS to adopters (#2485 by @shuch3ng)
• Bump github.com/stretchr/testify from 1.9.0 to 1.10.0 (#2488 by @dependabot[bot])
• Bump github.com/prometheus/client_golang from 1.20.5 to 1.21.1 (#2487 by @dependabot[bot])
• Bump sigs.k8s.io/controller-runtime from 0.20.1 to 0.20.4 (#2486 by @dependabot[bot])
• Deprecating sparkctl (#2484 by @vikas-saxena02)
• Changing image repo from docker.io to ghcr.io (#2483 by @vikas-saxena02)
• Upgrade Golang to 1.24.1 and golangci-lint to 1.64.8 (#2494 by @jacobsalway)
• Bump helm.sh/helm/v3 from 3.16.2 to 3.17.3 (#2503 by @dependabot[bot])
• Add changelog for v2.1.1 (#2504 by @ChenYi015)
• Remove sparkctl (#2466 by @ChenYi015)
• Bump github.com/spf13/viper from 1.19.0 to 1.20.1 (#2496 by @dependabot[bot])
• Bump golang.org/x/net from 0.37.0 to 0.38.0 (#2505 by @dependabot[bot])
• Remove clientset, informer and listers generated by code-generator (#2506 by @ChenYi015)
• Remove v1beta1 API (#2516 by @ChenYi015)
• add unit tests for driver and executor configs (#2521 by @nabuskey)
• Adding securityContext to spark examples (#2530 by @tarekabouzeid)
• Bump github.com/spf13/cobra from 1.8.1 to 1.9.1 (#2497 by @dependabot[bot])
• Bump golang.org/x/mod from 0.23.0 to 0.24.0 (#2495 by @dependabot[bot])
• Adding Manabu to the reviewers (#2522 by @vara-bonthu)
• Bump manusa/actions-setup-minikube from 2.13.1 to 2.14.0 (#2523 by @dependabot[bot])
• Bump k8s.io dependencies to v0.32.5 (#2540 by @ChenYi015)
• Pass the correct LDFLAGS when building the operator image (#2541 by @ChenYi015)

Full Changelog

v2.1.1 (2025-03-21)

Features

• Adding seccompProfile RuntimeDefault (#2397 by @tarekabouzeid)
• Add option for disabling leader election (#2423 by @ChenYi015)
• Controller should only be granted event permissions in spark job namespaces (#2426 by @ChenYi015)
• Make image optional (#2439 by @jbhalodia-slack)
• Support non-standard Spark container names (#2441 by @jbhalodia-slack)
• add support for metrics-job-start-latency-buckets flag in helm (#2450 by @nabuskey)

Bug Fixes

• fix: webhook fail to add lifecycle to Spark3 executor pods (#2458 by @pvbouwel)
• change env in executorSecretOption (#2467 by @TQJADE)

Misc

• Move sparkctl to cmd directory (#2347 by @ChenYi015)
• Bump golang.org/x/net from 0.30.0 to 0.32.0 (#2350 by @dependabot[bot])
• Bump golang.org/x/crypto from 0.30.0 to 0.31.0 (#2365 by @dependabot[bot])
• add an example of using prometheus servlet (#2403 by @nabuskey)
• Remove dependency on k8s.io/kubernetes (#2398 by @jacobsalway)
• fix make deploy and install (#2412 by @nabuskey)
• Add helm unittest step to integration test workflow (#2424 by @ChenYi015)
• ensure passed context is used (#2432 by @nabuskey)
• Bump manusa/actions-setup-minikube from 2.13.0 to 2.13.1 (#2390 by @dependabot[bot])
• Bump helm/chart-testing-action from 2.6.1 to 2.7.0 (#2391 by @dependabot[bot])
• Bump golang.org/x/mod from 0.21.0 to 0.23.0 (#2427 by @dependabot[bot])
• Bump github.com/golang/glog from 1.2.2 to 1.2.4 (#2411 by @dependabot[bot])
• Bump golang.org/x/net from 0.32.0 to 0.35.0 (#2428 by @dependabot[bot])
• Support Kubernetes 1.32 (#2416 by @jacobsalway)
• use cmd context in sparkctl (#2447 by @nabuskey)
• Bump golang.org/x/net from 0.35.0 to 0.36.0 (#2470 by @dependabot[bot])
• Bump aquasecurity/trivy-action from 0.29.0 to 0.30.0 (#2475 by @dependabot[bot])
• Bump golang.org/x/net from 0.35.0 to 0.37.0 (#2472 by @dependabot[bot])
• Bump github.com/containerd/containerd from 1.7.19 to 1.7.27 (#2476 by @dependabot[bot])
• Bump k8s.io/apimachinery from 0.32.0 to 0.32.3 (#2474 by @dependabot[bot])
• Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.66.0 to 1.78.2 (#2473 by @dependabot[bot])
• Bump github.com/aws/aws-sdk-go-v2/config from 1.28.0 to 1.29.9 (#2463 by @dependabot[bot])
• Bump sigs.k8s.io/scheduler-plugins from 0.29.8 to 0.30.6 (#2444 by @dependabot[bot])

Full Changelog

v2.1.0 (2024-12-06)

New Features

• Upgrade to Spark 3.5.3 (#2202 by @jacobsalway)
• feat: support archives param for spark-submit (#2256 by @kaka-zb)
• Allow --ingress-class-name to be specified in chart (#2278 by @jacobsalway)
• Update default container security context (#2265 by @ChenYi015)
• Support pod template for Spark 3.x applications (#2141 by @ChenYi015)
• Allow setting automountServiceAccountToken (#2298 by @Aranch)
• Allow the Controller and Webhook Containers to run with the securityContext: readOnlyRootfilesystem: true (#2282 by @npgretz)
• Use NSS_WRAPPER_PASSWD instead of /etc/passwd as in spark-operator image entrypoint.sh (#2312 by @Aakcht)

Bug Fixes

• Minor fixes to e2e test make targets (#2242 by @Tom-Newton)
• Added off heap memory to calculation for YuniKorn gang scheduling (#2209 by @guangyu-yang-rokt)
• Add permissions to controller serviceaccount to list and watch ingresses (#2246 by @tcassaert)
• Make sure enable-ui-service flag is set to false when controller.uiService.enable is set to false (#2261 by @Roberdvs)
• omitempty corrections (#2255 by @Tom-Newton)
• Fix retries (#2241 by @Tom-Newton)
• Fix: executor container security context does not work (#2306 by @ChenYi015)
• Fix: should not add emptyDir sizeLimit conf if it is nil (#2305 by @ChenYi015)
• Fix: should not add emptyDir sizeLimit conf on executor pods if it is nil (#2316 by @Cian911)
• Truncate UI service name if over 63 characters (#2311 by @jacobsalway)
• The webhook-key-name command-line param isn't taking effect (#2344 by @c-h-afzal)
• Robustness to driver pod taking time to create (#2315 by @Tom-Newton)

Misc

• remove redundant test.sh file (#2243 by @ChenYi015)
• Bump github.com/aws/aws-sdk-go-v2/config from 1.27.42 to 1.27.43 (#2252 by @dependabot[bot])
• Bump manusa/actions-setup-minikube from 2.12.0 to 2.13.0 (#2247 by @dependabot[bot])
• Bump golang.org/x/net from 0.29.0 to 0.30.0 (#2251 by @dependabot[bot])
• Bump aquasecurity/trivy-action from 0.24.0 to 0.27.0 (#2248 by @dependabot[bot])
• Bump gocloud.dev from 0.39.0 to 0.40.0 (#2250 by @dependabot[bot])
• Add Quick Start guide to README (#2259 by @jacobsalway)
• Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.63.3 to 1.65.3 (#2249 by @dependabot[bot])
• Add release badge to README (#2263 by @jacobsalway)
• Bump helm.sh/helm/v3 from 3.16.1 to 3.16.2 (#2275 by @dependabot[bot])
• Bump github.com/prometheus/client_golang from 1.20.4 to 1.20.5 (#2274 by @dependabot[bot])
• Bump cloud.google.com/go/storage from 1.44.0 to 1.45.0 (#2273 by @dependabot[bot])
• Run e2e tests with Kubernetes version matrix (#2266 by @jacobsalway)
• Bump aquasecurity/trivy-action from 0.27.0 to 0.28.0 (#2270 by @dependabot[bot])
• Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.65.3 to 1.66.0 (#2271 by @dependabot[bot])
• Bump github.com/aws/aws-sdk-go-v2/config from 1.27.43 to 1.28.0 (#2272 by @dependabot[bot])
• Add workflow for releasing sparkctl binary (#2264 by @ChenYi015)
• Bump volcano.sh/apis to 1.10.0 (#2320 by @jacobsalway)
• Bump aquasecurity/trivy-action from 0.28.0 to 0.29.0 (#2332 by @dependabot[bot])
• Bump github.com/onsi/ginkgo/v2 from 2.20.2 to 2.22.0 (#2335 by @dependabot[bot])
• Move sparkctl to cmd directory (#2347 by @ChenYi015)

Full Changelog

v2.0.2 (2024-10-10)

Bug Fixes

• Fix ingress capability discovery (#2201 by @jacobsalway)
• fix: imagePullPolicy was ignored (#2222 by @missedone)
• fix: spark-submission failed due to lack of permission by user spark (#2223 by @missedone)
• Remove cap_net_bind_service from image (#2216 by @jacobsalway)
• fix: webhook panics due to logging (#2232 by @ChenYi015)

Misc

• Bump github.com/aws/aws-sdk-go-v2 from 1.30.5 to 1.31.0 (#2207 by @dependabot[bot])
• Bump golang.org/x/net from 0.28.0 to 0.29.0 (#2205 by @dependabot[bot])
• Bump github.com/docker/docker from 27.0.3+incompatible to 27.1.1+incompatible (#2125 by @dependabot[bot])
• Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.58.3 to 1.63.3 (#2206 by @dependabot[bot])
• Update integration test workflow and add golangci lint check (#2197 by @ChenYi015)
• Bump github.com/aws/aws-sdk-go-v2 from 1.31.0 to 1.32.0 (#2229 by @dependabot[bot])
• Bump cloud.google.com/go/storage from 1.43.0 to 1.44.0 (#2228 by @dependabot[bot])
• Bump manusa/actions-setup-minikube from 2.11.0 to 2.12.0 (#2226 by @dependabot[bot])
• Bump golang.org/x/time from 0.6.0 to 0.7.0 (#2227 by @dependabot[bot])
• Bump github.com/aws/aws-sdk-go-v2/config from 1.27.33 to 1.27.42 (#2231 by @dependabot[bot])
• Bump github.com/prometheus/client_golang from 1.19.1 to 1.20.4 (#2204 by @dependabot[bot])
• Add check for generating manifests and code (#2234 by @ChenYi015)

Full Changelog

v2.0.1 (2024-09-26)

New Features

• FEATURE: build operator image as non-root (#2171 by @ImpSy)

Bug Fixes

• Update controller RBAC for ConfigMap and PersistentVolumeClaim (#2187 by @ChenYi015)

Misc

• Bump github.com/onsi/ginkgo/v2 from 2.19.0 to 2.20.2 (#2188 by @dependabot[bot])
• Bump github.com/onsi/gomega from 1.33.1 to 1.34.2 (#2189 by @dependabot[bot])

Full Changelog

v2.0.0 (2024-09-23)

Breaking Changes

• Use controller-runtime to reconsturct spark operator (#2072 by @ChenYi015)
• feat: support driver and executor pod use different priority (#2146 by @Kevinz857)

New Features

• Support gang scheduling with Yunikorn (#2107) by @jacobsalway
• Reintroduce option webhook.enable (#2142 by @ChenYi015)
• Add default batch scheduler argument (#2143 by @jacobsalway)
• Support extended kube-scheduler as batch scheduler (#2136 by @ChenYi015)
• Set schedulerName to Yunikorn (#2153 by @jacobsalway)
• Feature: Add pprof endpoint (#2164 by @ImpSy)

Bug Fixes

• fix: Add default values for namespaces to match usage descriptions (#2128 by @snappyyouth)
• Fix: Spark role binding did not render properly when setting spark service account name (#2135 by @ChenYi015)
• fix: unable to set controller/webhook replicas to zero (#2147 by @ChenYi015)
• Adding support for setting spark job namespaces to all namespaces (#2123 by @ChenYi015)
• Fix: e2e test failes due to webhook not ready (#2149 by @ChenYi015)
• fix: webhook not working when settings spark job namespaces to empty (#2163 by @ChenYi015)
• fix: The logger had an odd number of arguments, making it panic (#2166 by @tcassaert)
• fix the make kind-delete-custer to avoid accidental kubeconfig deletion (#2172 by @ImpSy)
• Add specific error in log line when failed to create web UI service (#2170 by @tcassaert)
• Account for spark.executor.pyspark.memory in Yunikorn gang scheduling (#2178 by @jacobsalway)
• Fix: spark application does not respect time to live seconds (#2165 by @ChenYi015)

Misc

• Update workflow and docs for releasing Spark operator (#2089 by @ChenYi015)
• Fix broken integration test CI (#2109 by @ChenYi015)
• Fix CI: environment variable BRANCH is missed (#2111 by @ChenYi015)
• Update Makefile for building sparkctl (#2119 by @ChenYi015)
• Update release workflow and docs (#2121 by @ChenYi015)
• Run e2e tests on Kind (#2148 by @jacobsalway)
• Upgrade to Go 1.23.1 (#2155 by @jacobsalway)
• Upgrade to Spark 3.5.2 (#2154 by @jacobsalway)
• Bump sigs.k8s.io/scheduler-plugins from 0.29.7 to 0.29.8 (#2159 by @dependabot[bot])
• Bump gocloud.dev from 0.37.0 to 0.39.0 (#2160 by @dependabot[bot])
• Update e2e tests (#2161 by @ChenYi015)
• Upgrade to Spark 3.5.2(#2012) (#2157 by @ha2hi)
• Bump github.com/aws/aws-sdk-go-v2/config from 1.27.27 to 1.27.33 (#2174 by @dependabot[bot])
• Bump helm.sh/helm/v3 from 3.15.3 to 3.16.1 (#2173 by @dependabot[bot])
• implement workflow to scan latest released docker image (#2177 by @ImpSy)

What's Changed

• Cherry pick #2081 #2046 #2091 #2072 by @ChenYi015 in kubeflow#2108
• Cherry pick #2089 #2109 #2111 by @ChenYi015 in kubeflow#2110
• Release v2.0.0-rc.0 by @ChenYi015 in kubeflow#2115
• Cherry pick commits for releasing v2.0.0 by @ChenYi015 in kubeflow#2156
• Release v2.0.0 by @ChenYi015 in kubeflow#2182

Full Changelog

v2.0.0-rc.0 (2024-08-09)

Breaking Changes

• Use controller-runtime to reconsturct spark operator (#2072 by @ChenYi015)

Misc

• Fix CI: environment variable BRANCH is missed (#2111 by @ChenYi015)

• Fix broken integration test CI (#2109 by @ChenYi015)

• Update workflow and docs for releasing Spark operator (#2089 by @ChenYi015)

What's Changed

• Release v2.0.0-rc.0 (#2115 by @ChenYi015)
• Cherry pick #2089 #2109 #2111 (#2110 by @ChenYi015)
• Cherry pick #2081 #2046 #2091 #2072 (#2108 by @ChenYi015)

Full Changelog

spark-operator-chart-1.4.6 (2024-07-26)

• Add topologySpreadConstraints (#2091 by @jbhalodia-slack)
• Add Alibaba Cloud to adopters (#2097 by @ChenYi015)
• Update Stale bot settings (#2095 by @andreyvelich)
• Add @ChenYi015 to approvers (#2096 by @ChenYi015)
• Add CHANGELOG.md file and use python script to generate it automatically (#2087 by @ChenYi015)

Full Changelog

spark-operator-chart-1.4.5 (2024-07-22)

• Update the process to build api-docs, generate CRD manifests and code (#2046 by @ChenYi015)
• Add workflow for closing stale issues and PRs (#2073 by @ChenYi015)

Full Changelog

spark-operator-chart-1.4.4 (2024-07-22)

• Update helm docs (#2081 by @csp33)

Full Changelog

spark-operator-chart-1.4.3 (2024-07-03)

• Add PodDisruptionBudget to chart (#2078 by @csp33)
• Update README and documentation (#2047 by @ChenYi015)
• Add code of conduct and update contributor guide (#2074 by @ChenYi015)
• Remove .gitlab-ci.yml (#2069 by @jacobsalway)
• Modified README.MD as per changes discussed on kubeflow#2062 (#2066 by @vikas-saxena02)

Full Changelog

spark-operator-chart-1.4.2 (2024-06-17)

• Support objectSelector on mutating webhook (#2058 by @Cian911)

Full Changelog

spark-operator-chart-1.4.1 (2024-06-15)

• Adding an option to set the priority class for spark-operator pod (#2043 by @pkgajulapalli)
• Update minikube version in CI (#2059 by @Cian911)

Full Changelog

spark-operator-chart-1.4.0 (2024-06-05)

• Certifictes are generated by operator rather than gencerts.sh (#2016 by @ChenYi015)
• Add ChenYi015 as spark-operator reviewer (#2045 by @ChenYi015)

Full Changelog

spark-operator-chart-1.3.2 (2024-06-05)

• Bump appVersion to v1beta2-1.5.0-3.5.0 (#2044 by @ChenYi015)
• Add restartPolicy field to SparkApplication Driver/Executor initContainers CRDs (#2022 by @mschroering)
• 📝 Add Inter&Co to who-is-using.md (#2040 by @ignitz)

Full Changelog

spark-operator-chart-1.3.1 (2024-05-31)

• Chart: add POD_NAME env for leader election (#2039 by @Aakcht)

Full Changelog

spark-operator-chart-1.3.0 (2024-05-20)

• Support exposing extra TCP ports in Spark Driver via K8s Ingress (#1998 by @hiboyang)
• Fixes a bug with dynamic allocation forcing the executor count to be 1 even when minExecutors is set to 0 (#1979 by @peter-mcclonski)
• Remove outdated PySpark experimental warning in example (#2014 by @andrejpk)
• Update Spark Job Namespace docs (#2000 by @matthewrossi)

Full Changelog

spark-operator-chart-1.2.15 (2024-05-07)

• Fix examples (#2010 by @peter-mcclonski)

Full Changelog

spark-operator-chart-1.2.14 (2024-04-26)

• feat: add support for service labels on driver-svc (#1985 by @Cian911)

Full Changelog

spark-operator-chart-1.2.13 (2024-04-24)

• fix(chart): remove operator namespace default for job namespaces value (#1989 by @t3mi)
• Fix Docker Hub Credentials in CI (#2003 by @andreyvelich)

Full Changelog

spark-operator-chart-1.2.12 (2024-04-19)

• Add emptyDir sizeLimit support for local dirs (#1993 by @jacobsalway)
• fix: Removed publish-image dependency on publishing the helm chart (#1995 by @vara-bonthu)

Full Changelog

spark-operator-chart-1.2.11 (2024-04-19)

• fix: Update Github workflow to publish Helm charts on chart changes, irrespective of image change (#1992 by @vara-bonthu)
• chore: Add Timo to user list (#1615 by @vanducng)
• Update spark operator permissions for CRD (#1973 by @ChenYi015)
• fix spark-rbac (#1986 by @Aransh)
• Use Kubeflow Docker Hub for Spark Operator Image (#1974 by @andreyvelich)
• fix: fixed serviceaccount annotations (#1972 by @AndrewChubatiuk)

Full Changelog

spark-operator-chart-1.2.7 (2024-04-16)

• fix: upgraded k8s deps (#1983 by @AndrewChubatiuk)
• chore: remove k8s.io/kubernetes replaces and adapt to v1.29.3 apis (#1968 by @ajayk)
• Add some helm chart unit tests and fix spark service account render failure when extra annotations are specified (#1967 by @ChenYi015)
• feat: Doc updates, Issue and PR templates are added (#1970 by @vara-bonthu)

Full Changelog

spark-operator-chart-1.2.5 (2024-04-14)

• fixed docker image tag and updated chart docs (#1969 by @AndrewChubatiuk)

Full Changelog

spark-operator-chart-1.2.4 (2024-04-13)

• publish chart independently, incremented both chart and image versions to trigger build of both (#1964 by @AndrewChubatiuk)
• Update helm chart README (#1958 by @ChenYi015)
• fix: add containerPort declaration for webhook in helm chart (#1961 by @zevisert)
• added id for a build job to fix digests artifact creation (#1963 by @AndrewChubatiuk)
• support multiple namespaces (#1955 by @AndrewChubatiuk)
• chore: replace GoogleCloudPlatform/spark-on-k8s-operator with kubeflow/spark-operator (#1937 by @zevisert)
• Chart: add patch permissions for spark operator SA to support spark 3.5.0 (#1884 by @Aakcht)
• Cleanup after golang upgrade (#1956 by @AndrewChubatiuk)
• feat: add support for custom service labels (#1952 by @Cian911)
• upgraded golang and dependencies (#1954 by @AndrewChubatiuk)
• README for installing operator using kustomize with custom namespace and service name (#1778 by @shahsiddharth08)
• BUGFIX: Added cancel method to fix context leak (#1917 by @fazledyn-or)
• remove unmatched quotes from user-guide.md (#1584 by @taeyeopkim1)
• Add PVC permission to Operator role (#1889 by @wyangsun)
• Allow to set webhook job resource limits (#1429,#1300) (#1946 by @karbyshevds)
• Create OWNERS (#1927 by @zijianjoy)
• fix: fix issue #1723 about spark-operator not working with volcano on OCP (#1724 by @disaster37)
• Add Rokt to who-is-using.md (#1867 by @jacobsalway)
• Handle invalid API resources in discovery (#1758 by @wiltonsr)
• Fix docs for Volcano integration (#1719 by @VVKot)
• Added qualytics to who is using (#1736 by @josecsotomorales)
• Allowing optional annotation on rbac (#1770 by @cxfcxf)
• Support seccompProfile in Spark application CRD and fix pre-commit jobs (#1768 by @ordukhanian)
• Updating webhook docs to also mention eks (#1763 by @JunaidChaudry)
• Link to helm docs fixed (#1783 by @haron)
• Improve getMasterURL() to add [] to IPv6 if needed (#1825 by @LittleWat)
• Add envFrom to operator deployment (#1785 by @matschaffer-roblox)
• Expand ingress docs a bit (#1806 by @matschaffer-roblox)
• Optional sidecars for operator pod (#1754 by @qq157755587)
• Add Roblox to who-is (#1784 by @matschaffer-roblox)
• Molex started using spark K8 operator. (#1714 by @AshishPushpSingh)
• Extra helm chart labels (#1669 by @kvanzuijlen)

Full Changelog

spark-operator-chart-1.1.27 (2023-03-17)

• Added permissions for leader election #1635 (#1647 by @ordukhanian)
• Fix #1393 : fix tolerations block in wrong segment for webhook jobs (#1633 by @zhiminglim)
• add dependabot (#1629 by @monotek)
• Add support for ephemeral.volumeClaimTemplate in helm chart CRDs (#1661 by @ArshiAAkhavan)
• Add Kognita to "Who is using" (#1637 by @claudino-kognita)
• add lifecycle to executor (#1674 by @tiechengsu)
• Fix signal handling for non-leader processes (#1680 by @antonipp)

Full Changelog

spark-operator-chart-1.1.26 (2022-10-25)

• update go to 1.19 + k8s.io libs to v0.25.3 (#1630 by @ImpSy)
• Update README - secrets and sidecars need mutating webhooks (#1550 by @djdillon)

Full Changelog

spark-operator-chart-1.1.25 (2022-06-08)

• Webhook init and cleanup should respect nodeSelector (#1545 by @erikcw)
• rename unit tests to integration tests in Makefile#integration-test (#1539 by @dcoliversun)

Full Changelog

spark-operator-chart-1.1.24 (2022-06-01)

• Fix: use V1 api for CRDs for volcano integration (#1540 by @Aakcht)

Full Changelog

spark-operator-chart-1.1.23 (2022-05-18)

• fix: add pre-upgrade hook to rbac resources (#1511 by @cwyl02)

Full Changelog

spark-operator-chart-1.1.22 (2022-05-16)

• Fixes issue #1467 (issue when deleting SparkApplication without metrics server) (#1530 by @aneagoe)
• Implement --logs and --delete flags on 'sparkctl create' and a timeout on 'sparkctl log' to wait a pod startup (#1506 by @alaurentinoofficial)
• Fix Spark UI URL in app status (#1518 by @gtopper)
• remove quotes from yaml file (#1524 by @zencircle)
• Added missing manifest yaml, point the manifest to the right direction (#1504 by @RonZhang724)

Full Changelog

spark-operator-chart-1.1.21 (2022-05-12)

• Ensure that driver is deleted prior to sparkapplication resubmission (#1521 by @khorshuheng)

Full Changelog

spark-operator-chart-1.1.20 (2022-04-11)

• Add ingress-class-name controller flag (#1482 by @voyvodov)

Full Changelog

spark-operator-chart-1.1.19 (2022-02-14)

• Add Operator volumes and volumeMounts in chart (#1475 by @ocworld)

Full Changelog

spark-operator-chart-1.1.18 (2022-02-13)

• Updated default registry to ghcr.io (#1454 by @aneagoe)
• Github actions workflow fix for Helm chart deployment (#1456 by @vara-bonthu)
• Kubernetes v1.22 extensions/v1beta1 API removal (#1427 by @aneagoe)
• Fixes an issue with github action in job build-spark-operator (#1452 by @aneagoe)
• use github container registry instead of gcr.io for releases (#1422 by @TomHellier)
• Fixes an error that was preventing the pods from being mutated (#1421 by @ssullivan)
• Make github actions more feature complete (#1418 by @TomHellier)
• Resolves an error when deploying the webhook where the k8s api indica… (#1413 by @ssullivan)

Full Changelog

spark-operator-chart-1.1.15 (2021-12-02)

• Add docker build to github action (#1415 by @TomHellier)

Full Changelog

spark-operator-chart-1.1.14 (2021-11-30)

• Updating API version of admissionregistration.k8s.io (#1401 by @sairamankumar2)
• Add C2FO to who is using (#1391 by @vanhoale)

Full Changelog

spark-operator-chart-1.1.13 (2021-11-18)

• delete-service-accounts-and-roles-before-creation (#1384 by @TiansuYu)

Full Changelog

spark-operator-chart-1.1.12 (2021-11-14)

• webhook timeout variable (#1387 by @sairamankumar2)

Full Changelog

spark-operator-chart-1.1.11 (2021-11-12)

• [FIX] add service account access to persistentvolumeclaims (#1390 by @mschroering)
• Add DeepCure to who is using (#1389 by @mschroering)

Full Changelog

spark-operator-chart-1.1.10 (2021-11-09)

• Add custom toleration support for webhook jobs (#1383 by @korjek)
• fix container name in addsecuritycontext patch (#1377 by @lybavsky)

Full Changelog

spark-operator-chart-1.1.9 (2021-11-01)

• Role and RoleBinding not installed for webhook-init in Helm pre-hook (#1379 by @zzvara)

Full Changelog

spark-operator-chart-1.1.8 (2021-10-26)

• Regenerate deleted cert after upgrade (#1373 by @simplylizz)
• Make manifests usable by Kustomize (#1367 by @karpoftea)
• #1329 update the operator to allow subpaths to be used with the spark ui ingress. (#1330 by @TomHellier)

Full Changelog

spark-operator-chart-1.1.7 (2021-10-21)

• serviceAccount annotations (#1350 by @moskitone)
• Update Dockerfile (#1369 by @Sadagopan88)
• [FIX] tolerations are not directly present in Driver(/Executor)Spec (#1365 by @s-pedamallu)
• fix running metrics for application deletion (#1358 by @Aakcht)
• Update who-is-using.md (#1338 by @Juandavi1)
• Update who-is-using.md (#1082 by @Juandavi1)
• Add support for executor service account (#1322 by @bbenzikry)
• fix NPE introduce on #1280 (#1325 by @ImpSy)

Full Changelog

spark-operator-chart-1.1.6 (2021-08-04)

• Add hook deletion policy for spark-operator service account (#1313 by @pdrastil)

Full Changelog

spark-operator-chart-1.1.5 (2021-07-28)

• Add user defined pod labels (#1288 by @pdrastil)

Full Changelog

spark-operator-chart-1.1.4 (2021-07-25)

• Migrate CRDs from v1beta1 to v1. Add additionalPrinterColumns (#1298 by @drazul)
• Explain "signal: kill" errors during submission (#1292 by @zzvara)
• fix the invalid repo address (#1291 by @william-wang)
• add failure context to recordExecutorEvent (#1280 by @ImpSy)
• Update pythonVersion to fix example (#1284 by @stratus)
• add crds drift check between chart/ and manifest/ (#1272 by @ImpSy)

Full Changelog

spark-operator-chart-1.1.3 (2021-05-25)

• Allow user to specify service annotation on Spark UI service (#1264 by @khorshuheng)

Full Changelog

spark-operator-chart-1.1.2 (2021-05-25)

• implement shareProcessNamespace in SparkPodSpec (#1262 by @ImpSy)

Full Changelog

spark-operator-chart-1.1.1 (2021-05-19)

• Enable UI service flag for disabling UI service (#1261 by @sairamankumar2)
• Add DiDi to who-is-using.md (#1255 by @Run-Lin)
• doc: update who is using page (#1251 by @luizm)
• Add Tongdun under who-is-using (#1249 by @lomoJG)
• [#1239] Custom service port name for spark application UI (#1240 by @marcozov)
• fix: do not remove preemptionPolicy in patcher when not present (#1246 by @HHK1)

Full Changelog

spark-operator-chart-1.1.0 (2021-04-28)

• Updating Spark version from 3.0 to 3.1.1 (#1153 by @chethanuk)

Full Changelog

spark-operator-chart-1.0.10 (2021-04-28)

• Add support for blue/green deployments (#1230 by @flupke)
• Update who-is-using.md: Fossil is using Spark Operator for Production (#1244 by @duyet)

Full Changelog

spark-operator-chart-1.0.9 (2021-04-23)

• Link to Kubernetes Slack (#1234 by @jsoref)
• fix: remove preemptionPolicy when priority class name is used (#1236 by @HHK1)
• Spelling (#1231 by @jsoref)
• Add support to expose custom ports (#1205 by @luizm)
• Fix the error of hostAliases when there are more than 2 hostnames (#1209 by @cdmikechen)
• remove multiple prefixes for 'p' (#1210 by @chaudhryfaisal)
• added --s3-force-path-style to force path style URLs for S3 objects (#1206 by @chaudhryfaisal)
• Allow custom bucket path (#1207 by @bribroder)
• fix: Remove priority from the spec when using priority class (#1203 by @HHK1)
• Fix go get issue with "unknown revision v0.0.0" (#1198 by @hongshaoyang)

Full Changelog

spark-operator-chart-1.0.8 (2021-03-07)

• Helm: Put service account into pre-install hook. (#1155 by @tandrup)
• correct hook annotation for webhook job (#1193 by @chaudhryfaisal)
• Update who-is-using.md (#1174 by @tarek-izemrane)
• add Carrefour as adopter and contributor (#1156 by @AliGouta)

Full Changelog

spark-operator-chart-1.0.7 (2021-02-05)

• fix issue #1131 (#1142 by @kz33)

Full Changelog

spark-operator-chart-1.0.6 (2021-02-04)

• Add Fossil to who-is-using.md (#1152 by @duyet)
• #1143 Helm issues while deploying using argocd (#1145 by @TomHellier)
• Include Gojek in who-is-using.md (#1146 by @pradithya)
• add hostAliases for SparkPodSpec (#1133 by @ImpSy)
• Adding MavenCode (#1128 by @charlesa101)
• Add MongoDB to who-is-using.md (#1123 by @chickenPopcorn)
• update go version to 1.15 and k8s deps to v0.19.6 (#1119 by @stpabhi)

Full Changelog

spark-operator-chart-1.0.5 (2020-12-15)

• Add prometheus containr port name (#1099 by @nicholas-fwang)

Full Changelog

spark-operator-chart-1.0.4 (2020-12-12)

• Upgrade the Chart version to 1.0.4 (#1113 by @ordukhanian)
• Support Prometheus PodMonitor Deployment (#1106) (#1112 by @ordukhanian)
• update executor status if pod is lost while app is still running (#1111 by @ImpSy)
• Add scheduler func for clearing batch scheduling on completed (#1079 by @nicholas-fwang)
• Add configuration for SparkUI service type (#1100 by @jutley)

Full Changelog

spark-operator-chart-1.0.3 (2020-12-07)

• Update docs with new helm instructions (#1105 by @hagaibarel)

Full Changelog