Configurable error handler for terminal errors

[jacobdrury]

In the typescript sdk there is a concept of a asTerminalError func that can be defined on the service individually or directly on the endpoint: https://restatedev.github.io/sdk-typescript/types/_restatedev_restate-sdk.ServiceOptions.html#asterminalerror

It would be great if we could add a similar error handling func to the go sdk.

Would love some input for how this should be added to the sdk so that I can follow this up with a PR! I put some of my ideas below:

My use-case would be when initializing the restate server so being able to provide the error handler function somewhat like below would achieve what I'm going for:

func main() {
	server := server.NewRestate().
		WithErrorHandler(func(err error) error {
			// If the error is already a terminal error, return it
			if restate.IsTerminalError(err) {
				return err
			}

			// If the error is a context deadline exceeded, return a terminal error
			if errors.Is(err, context.DeadlineExceeded) {
				return restate.TerminalError(err)
			}

			// Otherwise return the error as is
			return err
		}).
		// Handlers can be inferred from object methods
		Bind(restate.Reflect(Greeter{})).
		Bind(restate.Reflect(GreeterCounter{}))

	if err := server.Start(context.Background(), ":9080"); err != nil {
		slog.Error("application exited unexpectedly", "err", err.Error())
		os.Exit(1)
	}
}

If we wanted each service to be able to define their own error handling logic, there could be an interface they implement that could take precedence over the global error handle

type serviceErrorHandler interface {
	HandleError(err error) error
}

We also could allow this error handler func to be passed into restate.Run as an option if we want to override any of the previous error handling:

_, err = restate.Run(ctx, func(ctx restate.RunContext) (bool, error) {
	log := ctx.Log().With("uuid", uuid, "price", price)
	if rand.Float64() < 0.5 {
		log.Info("payment succeeded")
		return true, nil
	} else {
		log.Error("payment failed")
		return false, fmt.Errorf("failed to pay")
	}
}, restate.WithErrorHandler(func(err error) error {
	// If the error is already a terminal error, return it
	if restate.IsTerminalError(err) {
		return err
	}

	// If the error is a context deadline exceeded, return a terminal error
	if errors.Is(err, context.DeadlineExceeded) {
		return restate.TerminalError(err)
	}

	// Otherwise return the error as is
	return err
}))

[slinkydeveloper]

I like the approach of the closure more than the interface to implement on the service, so we can have uniformity. We can define a single type for the closure, like type ErrorHandler = func(error) error, and then let the user configure this at multiple levels:

• on the server builder directly, as you show above
• on individual services, as https://pkg.go.dev/github.com/restatedev/sdk-go@v0.23.0#ServiceDefinitionOption
• on handlers, as https://pkg.go.dev/github.com/restatedev/sdk-go@v0.23.0#HandlerOption
• on run, as https://pkg.go.dev/github.com/restatedev/sdk-go@v0.23.0#RunOption

List goes from more broader configuration, to more specific configuration, and when you need to call the error handler you should apply that hierarchy of configuration.

For the places where this needs to be used, there's effectively two cases:

• When we're handling a Run error

  sdk-go/internal/restatecontext/run.go

  Line 80 in 35e1350

  if errors.IsTerminalError(err) {

• When we're handling an handler error

  sdk-go/internal/restatecontext/execute_invocation.go

  Line 76 in 1df111e

  if err != nil && errors.IsTerminalError(err) {

Thanks a lot for helping out!

[jacobdrury]

I have a branch of this working and wanted to clarify the behavior when an error handler is configured for all "layers"

If I have an error handler configured on my builder, service, and run:

restate_server.NewRestate().
			WithErrorHandler(func(err error) error { return fmt.Errorf("builder handler: %w", err) }).
			Bind(restate.Reflect(Greeter{}, restate.WithErrorHandler(func(err error) error { return fmt.Errorf("service handler: %w", err) })))

_ = restate.Run(ctx, func(ctx restate.RunContext) (orgtypes.Member, error) {
    return nil, fmt.Errorf("failed to run")
}, restate.WithErrorHandler(func(err error) error { return fmt.Errorf("run handler: %w", err) }))

From my understanding of

when you need to call the error handler you should apply that hierarchy of configuration.

That would make the error look like:

builder handler: service handler: run handler: failed to run

My assumption would've been to only run the most specific handler and treat it as an override of the parent error handlers. That would result in:

run handler: failed to run

@slinkydeveloper Can you confirm what behavior you're looking for here?

[slinkydeveloper]

I feel like it should be more of an override, than composition... wdyt?