Duplicate submission of sparkapp with the same name, the app might get stuck in the new state

[xpfwork]

What happened?

version 1.1.25

If you delete an app.yaml file and then immediately apply it with another app.yaml, the app might get stuck in the submitted state.

The initial analysis suggests that the Spark operator hadn't yet deleted the old driver pod, and the new submit operation detected the existence of a driver pod with the same name.

Reproduction Code

func runSparkSubmit(submission *submission) (bool, error) {
sparkHome, present := os.LookupEnv(sparkHomeEnvVar)
if !present {
log.Error("SPARK_HOME is not specified")
}
var command = filepath.Join(sparkHome, "/bin/spark-submit")
cmd := execCommand(command, submission.args...)
log.Infof("spark-submit arguments: %v", cmd.Args)
output, err := cmd.Output()
log.Infof("spark-submit output: %s", string(output))
if err != nil {
var errorMsg string
if exitErr, ok := err.(*exec.ExitError); ok {
errorMsg = string(exitErr.Stderr)
}
// The driver pod of the application already exists.
if strings.Contains(errorMsg, podAlreadyExistsErrorCode) {
log.Warnf("trying to resubmit an already submitted SparkApplication %s/%s", submission.namespace, submission.name)
return false, nil
}
if errorMsg != "" {
return false, fmt.Errorf("failed to run spark-submit for SparkApplication %s/%s: %s", submission.namespace, submission.name, errorMsg)
}
return false, fmt.Errorf("failed to run spark-submit for SparkApplication %s/%s: %v", submission.namespace, submission.name, err)
}

return true, nil

}

submitted, err := runSparkSubmit(newSubmission(submissionCmdArgs, app))
if err != nil {
	app.Status = v1beta2.SparkApplicationStatus{
		AppState: v1beta2.ApplicationState{
			State:        v1beta2.FailedSubmissionState,
			ErrorMessage: err.Error(),
		},
		SubmissionAttempts:        app.Status.SubmissionAttempts + 1,
		LastSubmissionAttemptTime: metav1.Now(),
	}
	c.recordSparkApplicationEvent(app)
	log.Errorf("failed to run spark-submit for SparkApplication %s/%s: %v", app.Namespace, app.Name, err)
	return app
}

if !submitted {
	// The application may not have been submitted even if err == nil, e.g., when some
	// state update caused an attempt to re-submit the application, in which case no
	// error gets returned from runSparkSubmit. If this is the case, we simply return.
	return app
}

Expected behavior

No response

Actual behavior

No response

Environment & Versions

• Kubernetes Version:
• Spark Operator Version:
• Apache Spark Version:

Additional context

No response

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[xpfwork]

@ChenYi015 @khogeland @yuchaoran2011
Please help me check if it supports submitting app with the same name. Thanks!

[ChenYi015]

@xpfwork I cannot reproduce this issue using Spark operatorv2.4.0 with the following command:

kubectl delete -f spark-pi.yaml && kubectl create -f spark-pi.yaml

Spark operator will clean up SparkApplication resources like driver pod, web UI server/ingress.

spark-operator/internal/controller/sparkapplication/controller.go

Lines 267 to 283 in 1199997

	func (r *Reconciler) handleSparkApplicationDeletion(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := log.FromContext(ctx)
	key := req.NamespacedName
	app, err := r.getSparkApplication(ctx, key)
	if err != nil {
	if errors.IsNotFound(err) {
	return ctrl.Result{}, nil
	}
	return ctrl.Result{Requeue: true}, err
	}
	if err := r.deleteSparkResources(ctx, app); err != nil {
	logger.Error(err, "Failed to delete resources associated with SparkApplication")
	return ctrl.Result{Requeue: true}, err
	}
	return ctrl.Result{}, nil
	}

spark-operator/internal/controller/sparkapplication/controller.go

Lines 1114 to 1129 in 1199997

	// Delete the resources associated with the spark application.
	func (r *Reconciler) deleteSparkResources(ctx context.Context, app *v1beta2.SparkApplication) error {
	if err := r.deleteDriverPod(ctx, app); err != nil {
	return err
	}
	if err := r.deleteWebUIService(ctx, app); err != nil {
	return err
	}
	if err := r.deleteWebUIIngress(ctx, app); err != nil {
	return err
	}
	return nil
	}

[xpfwork]

@ChenYi015
Please ensure the driver pod exists first.

1、kubectl create -f spark-pi.yaml

2、wait a few seconds

3、kubectl delete -f spark-pi.yaml && kubectl create -f spark-pi.yaml

Please help me test this scenario again. Thanks!

[ChenYi015]

@ChenYi015
Please ensure the driver pod exists first.

1、kubectl create -f spark-pi.yaml

2、wait a few seconds

3、kubectl delete -f spark-pi.yaml && kubectl create -f spark-pi.yaml

Please help me test this scenario again. Thanks!

I run the command when the driver pod is running state, test it serveral times and it works as expected.

[xpfwork]

@ChenYi015 Thanks you reply!

When I use version 2.2, spark-operator changes the sparkapp status to failed.