EventStore.Client.Grpc: Subscription drops due to an HttpRequestException

[psharp81]

🐛 Current behavior

• We have established a subscription to an EventStoreDB server 24.10.4 using an EventStore.Client.Grpc version 23.3.8
• Then, the subscription drops with the following stack trace:

_EventStore: + Subscription 7af7f1c7-b27c-41db-aeb1-17519f7d0934 was dropped because an error occurred on the server.
   at EventStore.Client.Interceptors.TypedExceptionInterceptor.<>c__DisplayClass1_0.<.ctor>b__0(RpcException rpcEx)
   at EventStore.Client.Interceptors.ExceptionConverterStreamReader`1.MoveNext(CancellationToken cancellationToken)
   at EventStore.Client.AsyncStreamReaderExtensions.ReadAllAsync[T](IAsyncStreamReader`1 reader, CancellationToken cancellationToken)+MoveNext()
   at EventStore.Client.AsyncStreamReaderExtensions.ReadAllAsync[T](IAsyncStreamReader`1 reader, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at EventStore.Client.EventStoreClient.StreamSubscriptionResult.<>c__DisplayClass13_0.<<-ctor>g__PumpMessages|0>d.MoveNext()
--- End of stack trace from previous location ---
   at EventStore.Client.EventStoreClient.StreamSubscriptionResult.<>c__DisplayClass13_0.<<-ctor>g__PumpMessages|0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.Threading.Channels.AsyncOperation`1.GetResult(Int16 token)
   at System.Threading.Channels.ChannelReader`1.ReadAllAsync(CancellationToken cancellationToken)+MoveNext()
   at System.Threading.Channels.ChannelReader`1.ReadAllAsync(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at EventStore.Client.EventStoreClient.StreamSubscriptionResult.<get_Messages>g__GetMessages|12_0()+MoveNext()
   at EventStore.Client.EventStoreClient.StreamSubscriptionResult.<get_Messages>g__GetMessages|12_0()+MoveNext()
   at EventStore.Client.EventStoreClient.StreamSubscriptionResult.<get_Messages>g__GetMessages|12_0()+MoveNext()
   at EventStore.Client.EventStoreClient.StreamSubscriptionResult.<get_Messages>g__GetMessages|12_0()+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at EventStore.Client.StreamSubscription.Subscribe()
Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Error reading next message. HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)", DebugException="System.Net.Http.HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)")
 ---> System.Net.Http.HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)
   at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.StreamExtensions.ReadMessageAsync[TResponse](Stream responseStream, GrpcCall call, Func`2 deserializer, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.ReadMessageAsync(Stream responseStream, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2.MoveNextCore(CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush)
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at EventStore.Client.Interceptors.TypedExceptionInterceptor.<>c__DisplayClass1_0.<.ctor>b__0(RpcException rpcEx)
   at EventStore.Client.Interceptors.ExceptionConverterStreamReader`1.MoveNext(CancellationToken cancellationToken)
   at EventStore.Client.AsyncStreamReaderExtensions.ReadAllAsync[T](IAsyncStreamReader`1 reader, CancellationToken cancellationToken)+MoveNext()
   at EventStore.Client.AsyncStreamReaderExtensions.ReadAllAsync[T](IAsyncStreamReader`1 reader, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at EventStore.Client.EventStoreClient.StreamSubscriptionResult.<>c__DisplayClass13_0.<<-ctor>g__PumpMessages|0>d.MoveNext()
--- End of stack trace from previous location ---
   at EventStore.Client.EventStoreClient.StreamSubscriptionResult.<>c__DisplayClass13_0.<<-ctor>g__PumpMessages|0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.Threading.Channels.AsyncOperation`1.GetResult(Int16 token)
   at System.Threading.Channels.ChannelReader`1.ReadAllAsync(CancellationToken cancellationToken)+MoveNext()
   at System.Threading.Channels.ChannelReader`1.ReadAllAsync(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at EventStore.Client.EventStoreClient.StreamSubscriptionResult.<get_Messages>g__GetMessages|12_0()+MoveNext()
   at EventStore.Client.EventStoreClient.StreamSubscriptionResult.<get_Messages>g__GetMessages|12_0()+MoveNext()
   at EventStore.Client.EventStoreClient.StreamSubscriptionResult.<get_Messages>g__GetMessages|12_0()+MoveNext()
   at EventStore.Client.EventStoreClient.StreamSubscriptionResult.<get_Messages>g__GetMessages|12_0()+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at EventStore.Client.StreamSubscription.Subscribe()_

🔍 Steps to reproduce

• Create a subscription to an EventStoreDB server 24.10.4 using an EventStore.Client.Grpc version 23.3.8
• Wait until subscription drops

Reproducible link

💭 Expected behavior

• Subscription does not drop

Package version

EventStore.Client.Grpc 23.3.8

KurrentDB Version

KurrentDB 24.10.4

Connection string

esdb://administrator:****@IPAddress:2113?tls=true&tlsCaFile=certpath&keepAliveInterval=1000&keepAliveTimeout=2000

☁️ Deployment Environment

Single-node (Docker)

Other Deployment Details

No response

Operating system

Windows

Code of Conduct

• I agree to follow this project's Code of Conduct

[linear]

DEV-788 EventStore.Client.Grpc: Subscription drops due to an HttpRequestException

[w1am]

Could you please share the configuration you're using to run the EventStoreDB server?

[psharp81]

Here is the configuration we are using to run the EventStoreDB server:

services:
  device-eventstore:
    container_name: device-eventstore
    image: eventstore/eventstore:latest
    healthcheck:  # this is not an error
      disable: true
    ports:
      - "2113:2113"
      - "1113:1113"
      - "1115:1115"
    volumes:
      - /media/data/eventstore/:/var/lib/eventstore-data
      - /media/data/index/:/var/lib/eventstore-index
      - /opt/device-event-store/certs/:/etc/eventstore/certs
    user: "108:113"
    environment:
      - EVENTSTORE_DB=/var/lib/eventstore-data
      - EVENTSTORE_INDEX=/var/lib/eventstore-index
      - EVENTSTORE_MAX_MEM_TABLE_SIZE=100000
      - EVENTSTORE_SKIP_DB_VERIFY=True
      - EVENTSTORE_CERTIFICATE_FILE=/etc/eventstore/certs/node.crt
      - EVENTSTORE_CERTIFICATE_PRIVATE_KEY_FILE=/etc/eventstore/certs/node.key
      - EVENTSTORE_TRUSTED_ROOT_CERTIFICATES_PATH=/etc/eventstore/certs/ca/
      - EVENTSTORE_ENABLE_ATOM_PUB_OVER_HTTP=True
      - EVENTSTORE_NODE_PORT=2113
      - EVENTSTORE_DEFAULT_ADMIN_PASSWORD=***
      - EVENTSTORE_LOG_LEVEL=Information
      - EVENTSTORE_MAX_APPEND_SIZE=6291456
    logging:
      driver: journald
    restart: unless-stopped
    networks:
      - event-store

networks:
  event-store:
    name: event-store

[w1am]

I couldn't replicate the issue. I used the same EventStoreDB configuration as yours, subscribed to the $all stream using catch-up subscriptions, starting from the beginning, and appended some events.

Does it happen consistently in your environment? Can you describe what your event data looks like?

Can you also tell us which .NET version are you using?

[psharp81]

• The issue does not happen consistently in our environment, just sometimes
• Our event data is in JSON format. Or what information you need about the data?
• We are using .NET8

[w1am]

I wonder if it could be an issue with your certificates. I'm not sure though. maybe try running it in insecure mode?

[psharp81]

• When we faced issues with the certificates, we usually got a different strack trace
• Therefore, I think the insecure mode will not help

[w1am]

I’m trying to narrow this down but so far I can’t reproduce the error.
Here’s the minimal setup I used (no proxies, single node EventStoreDB from docker-compose):

var client = new EventStoreClient(settings);

await using var subscription = client.SubscribeToAll(FromAll.Start);

await foreach (var message in subscription.Messages) {
    switch (message) {
        case StreamMessage.Event(var evnt):
            Console.WriteLine(evnt.OriginalEvent.EventType);
            break;
        case StreamMessage.CaughtUp:
            Console.WriteLine("Caught up!");
            break;
    }
}

Since this runs without issues in my environment, could you share:

• Whether your client is connecting directly to EventStoreDB or through a proxy/load balancer

This will help me try to reproduce the conditions more closely. Also, if you can provide your client snippet that would be ideal.

HttpProtocolException: PROTOCOL_ERROR often related to proxies that don’t handle HTTP/2 correctly. Once I know more about your environment I can test against that specifically

[psharp81]

• Our client directly connects to the EventStoreDB
• Our client snippet is:

   private EventStoreClient _eventStoreClient;

   async Task<IStoppable> IEventStoreReadConnection.SubscribeToEventsGrpc(
        string streamId,
        long nextEvent,
        Action<IReadOnlyCollection<DownloadedEvent>> onEventsAppeared,
        Action onSubscriptionDropped)
    {
        var startFrom = nextEvent < 1 ? FromStream.Start : FromStream.After(EventStore.Client.StreamPosition.FromInt64(nextEvent - 1));
        var streamSubscription = await _eventStoreClient.SubscribeToStreamAsync(
            streamId,
            startFrom,
            OnNewEventAppeared,
            subscriptionDropped: OnSubscriptionDropped);
        return new StreamSubscriptionWrapper(streamSubscription, this);

        Task OnNewEventAppeared(StreamSubscription subscription, ResolvedEventByGrpcClient resolvedEvent, CancellationToken token)
        {
            onEventsAppeared([TranslatedGrpc(resolvedEvent)]);
            return Task.CompletedTask;
        }

        void OnSubscriptionDropped(StreamSubscription subscription, SubscriptionDroppedReason reason, Exception exception)
        {
            onSubscriptionDropped();
        }
   }

[w1am]

Can you explain what onEventsAppeared([TranslatedGrpc(resolvedEvent)]) is doing? Also, what is ResolvedEventByGrpcClient?

[psharp81]

• The onEventsAppeared([TranslatedGrpc(resolvedEvent)]) is the action which is executed when a new event appears within the active subscription to a stream of our EventStoreDB
• When a new event appears we first 'translate' this event to our internal event structure
• The ResolvedEventByGrpcClient is an alias for EventStore.Client.ResolvedEvent

[w1am]

You mentioned that it doesn’t happen intermittently. Under what specific conditions do you notice the issue occurring so that I can try to reproduce it on my side? For example, do you see it consistently under high load or in a particular usage pattern?

[psharp81]

• The issue is happening intermittently
• Currently, I have no indications under which specific conditions the issue is occurring