AppendToStreamAsync hangs in .NET Framework 4.8

[jbi-spillehallen]

Describe the bug

We're unable to AppendToStreamAsync in .NET Framework 4.8, on Windows Server 2022 Datacenter version 21H2.
The EventStore we're running against is 21.10.8.0, installed on Windows Server.

To Reproduce
Steps to reproduce the behavior:

1. Download this sample: https://github.com/EventStore/EventStore-Client-Dotnet/blob/v23.2.1/samples/appending-events/Program.cs
2. Build it against .NET Framework 4.8
3. Run EventStore version 21.10.8.0
4. Run sample on Windows Server 2022
5. It hangs on first await client.AppendToStreamAsync request.

Expected behavior
For events to be appended.

Actual behavior
The application hangs.

Config/Logs/Screenshots
Default configuration.

EventStore details

• EventStore server version: 21.10.8.0
• Operating system: Windows Server 2022 Datacenter version 21H2
• EventStore client version (if applicable): EventStore.Client.Grpc.Stream-23.2.1

Additional context
Exactly same code, but compiled against .NET 6, works as expected.

[bartelink]

do other APIs work, i.e. could it be a connectivity issue?

[jbi-spillehallen]

do other APIs work, i.e. could it be a connectivity issue?

Yes, other APIs work. For example client.ReadStreamAsync or client.SubscribeToStream.

[w1am]

Hey @jbi-spillehallen

Can you try forcing a regular append by passing user credentials?

Try it with this simple example:

await client.AppendToStreamAsync(
    "some-stream",
    StreamState.Any,
    new[] { eventData },
    userCredentials: new UserCredentials("admin", "changeit"), // force a regular append (not batch)
    cancellationToken: cancellationToken
);

[jbi-spillehallen]

Hey @jbi-spillehallen

Can you try forcing a regular append by passing user credentials?

Try it with this simple example:

await client.AppendToStreamAsync(
    "some-stream",
    StreamState.Any,
    new[] { eventData },
    userCredentials: new UserCredentials("admin", "changeit"), // force a regular append (not batch)
    cancellationToken: cancellationToken
);

Hey @w1am ,

We tried that and we got this exception:
Unhandled Exception: Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Request protocol 'HTTP/1.1' is not supported.") at EventStore.Client.Interceptors.TypedExceptionInterceptor.<>c__DisplayClass1_0.<.ctor>b__2(RpcException rpcEx) at EventStore.Client.Interceptors.RpcExceptionConversionExtensions.<>c__DisplayClass1_0`1.<Apply>b__0(Task`1 t) at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke() at System.Threading.Tasks.Task.Execute() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at EventStore.Client.EventStoreClient.<AppendToStreamInternal>d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at EventStore.Client.EventStoreClient.<AppendToStreamAsync>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at Program.<<<Main>$>g__AppendToStream|0_0>d.MoveNext() in C:\_data\ConsoleApp1\ConsoleApp1\Program.cs:line 30 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at Program.<<Main>$>d__0.MoveNext() in C:\_data\ConsoleApp1\ConsoleApp1\Program.cs:line 14 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Program.<<Main>$>d__0.MoveNext() in C:\_data\ConsoleApp1\ConsoleApp1\Program.cs:line 18 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at Program.<Main>(String[] args)

[w1am]

The gRPC client with net48 uses WinHttpHandler internally to make http calls. However, this comes with certain requirements and restrictions as listed in Microsoft Documentation.

Could you try setting tls=true and disabling cert verification.

e.g esdb://admin:changeit@localhost:2113?tls=true&tlsVerifyCert=false

[jbi-spillehallen]

The gRPC client with net48 uses WinHttpHandler internally to make http calls. However, this comes with certain requirements and restrictions as listed in Microsoft Documentation.

Could you try setting tls=true and disabling cert verification.

e.g esdb://admin:changeit@localhost:2113?tls=true&tlsVerifyCert=false

Same results. Batch append hangs and regular append throws the same exception.

[w1am]

This is interesting. I don't have a windows server to test it on. But it works fine on my Windows 11. I will keep investingating.

Can you try kurrent-io/KurrentDB#2707 (comment)

[jbi-spillehallen]

This is interesting. I don't have a windows server to test it on. But it works fine on my Windows 11. I will keep investingating.

Can you try EventStore/EventStore#2707 (comment)

On Windows 11, I can reproduce it as well. It doesn’t hang, but I get the same HTTP/1.1 exception as above. Can you tell me your local setup which works with .NET Framework 4.8?

I have tried that solution from the comment and I still get the same exception.

[w1am]

This is my server docker compose configuration:

version: '3'
services:
  volumes-provisioner:
    image: hasnat/volumes-provisioner
    environment:
      PROVISION_DIRECTORIES: "1000:1000:0755:/tmp/certs"
    volumes:
      - "./certs:/tmp/certs"
    network_mode: none

  cert-gen:
    image: docker.eventstore.com/eventstore-utils/es-gencert-cli:latest
    entrypoint: bash
    user: "1000:1000"
    command: >
      -c "mkdir -p ./certs && cd /certs
      && es-gencert-cli create-ca
      && es-gencert-cli create-node -out ./node1 -ip-addresses 127.0.0.1 -dns-names localhost
      && find . -type f -print0 | xargs -0 chmod 666"
    volumes:
      - "./certs:/certs"
    depends_on:
      - volumes-provisioner

  eventstore:
    image: ghcr.io/eventstore/eventstore:${EVENTSTORE_DOCKER_TAG_ENV:-lts}
    environment:
      - EVENTSTORE_ADVERTISE_HTTP_PORT_TO_CLIENT_AS=2113
      - EVENTSTORE_CERTIFICATE_FILE=/etc/eventstore/certs/node1/node.crt
      - EVENTSTORE_CERTIFICATE_PRIVATE_KEY_FILE=/etc/eventstore/certs/node1/node.key
      - EVENTSTORE_TRUSTED_ROOT_CERTIFICATES_PATH=/etc/eventstore/certs/ca
    ports:
      - "2113:2113"
    volumes:
      - type: volume
        source: eventstore-volume-logs
        target: /var/log/eventstore
      - type: bind
        source: ./certs
        target: /etc/eventstore/certs

volumes:
  eventstore-volume-logs:

and the code I used is:

public static async Task Main()
{
    var settings = EventStoreClientSettings.Create(
        $"esdb://admin:changeit@localhost:2113?tlsVerifyCert=false");

    settings.OperationOptions.ThrowOnAppendFailure = false;

    Log.Logger = new LoggerConfiguration()
        .MinimumLevel.Information()
        .WriteTo.Console()
        .CreateLogger();

    var loggerFactory = LoggerFactory.Create(builder => { builder.AddSerilog(dispose: true); });

    settings.LoggerFactory = loggerFactory;

    using (var client = new EventStoreClient(settings))
    {
        var eventData = new EventData(
            Uuid.NewUuid(),
            "some-event",
            Encoding.UTF8.GetBytes("{\"id\": \"1\" \"value\": \"some value\"}")
        );

        var response = await client.AppendToStreamAsync(
            "some-stream",
            StreamState.Any,
            new List<EventData>
            {
                eventData
            },
            userCredentials: new UserCredentials("admin", "changeit")
        );

        Log.Information("Response: {response}", response);
    }

    Log.CloseAndFlush();
}

Operating System
Windows 11 Pro Version 23H2 22631

[jbi-spillehallen]

Hey @w1am,

We did some more testing and we made it work running .net framework 4.8 client on windows 11. As long as you have certificates in place, it works. However, it doesn't work on Windows Server 2019 and Windows Server 2022. My best guess why it doesn't work on Windows server is this

gRPC client is partially supported on Windows Server 2019 and Windows Server 2022. Unary and server streaming methods are supported. Client and bidirectional streaming methods are not supported.

Source: https://learn.microsoft.com/en-us/aspnet/core/grpc/netstandard?view=aspnetcore-8.0#net-framework

[Salgat]

This is a regression from moving to Grpc.Net.Client correct? Would it be better to continue to use Grpc.Core until Grpc.Net.Client support catches up? As of now, there's not much point in EventStore.Client.Grpc targeting .NET Framework if it only supports Windows 11. I'd even argue that it creates a false impression that the EventStore .NET client is officially supported on Windows-based service deployments for .NET Framework. Is it at least possible to allow the choice between grpc dependencies to be configurable until Grpc.Net.Client reaches sufficient feature parity?