Endpoints using the outbox subscribed to the same event will experience concurrency conflicts if sharing the same table

[danielmarbach]

Describe the bug

Description

AzureTable does not separate outbox records by logical endpoint on the default partition key, and since message identities are not unique from a processing perspective, this leads to concurrency conflicts and the message being moved to the error queue.

Expected behavior

Events subscribed to by multiple endpoints sharing the same table are processed.

Actual behavior

Versions

Please list the version of the relevant packages or applications in which the bug exists.

• All versions

Steps to reproduce

See:

#1112

Relevant log output

 7:47:47 AM Info SubscribersHandlesTheSameEvent.Subscriber2: Immediate Retry is going to retry message '91cebc6e-169b-4ca1-8ccf-b35300807ba2' because of an exception: Azure.Data.Tables.TableTransactionFailedException: 0:The update condition specified in the request was not satisfied.
   
   RequestID:b65b4abe-22bc-4794-8b07-6c5ee1575889
   
    The index of the entity that caused the error can be found in FailedTransactionActionIndex.
   Status: 412 (Precondition Failed)
   ErrorCode: UpdateConditionNotSatisfied
   
   Additional Information:
   FailedEntity: 0
   
   Service request succeeded. Response content and headers are not included to avoid logging sensitive data.
   
      at Azure.Data.Tables.TableRestClient.SendBatchRequestAsync(HttpMessage message, CancellationToken cancellationToken)
      at Azure.Data.Tables.TableClient.SubmitTransactionInternalAsync(IEnumerable`1 transactionalBatch, Guid batchId, Guid changesetId, Boolean async, CancellationToken cancellationToken)
      at Azure.Data.Tables.TableClient.SubmitTransactionAsync(IEnumerable`1 transactionActions, CancellationToken cancellationToken)
      at NServiceBus.Persistence.AzureTable.TableBatchResultExtensions.ExecuteOperationAsync(List`1 transactionalBatch, Operation operation, CancellationToken cancellationToken) in /_/src/NServiceBus.Persistence.AzureTable/SynchronizedStorage/TableBatchResultExtensions.cs:line 17
      at NServiceBus.TransportReceiveToPhysicalMessageConnector.Invoke(ITransportReceiveContext context, Func`2 next) in /_/src/NServiceBus.Core/Pipeline/Incoming/TransportReceiveToPhysicalMessageConnector.cs:line 77
      at NServiceBus.RetryAcknowledgementBehavior.Invoke(ITransportReceiveContext context, Func`2 next) in /_/src/NServiceBus.Core/ServicePlatform/Retries/RetryAcknowledgementBehavior.cs:line 25
      at NServiceBus.AcceptanceTesting.Support.CaptureExceptionBehavior.Invoke(ITransportReceiveContext context, Func`2 next) in /_/src/NServiceBus.AcceptanceTesting/Support/CaptureExceptionBehavior.cs:line 21
      at NServiceBus.MainPipelineExecutor.Invoke(MessageContext messageContext, CancellationToken cancellationToken) in /_/src/NServiceBus.Core/Pipeline/MainPipelineExecutor.cs:line 50
      at NServiceBus.MainPipelineExecutor.Invoke(MessageContext messageContext, CancellationToken cancellationToken) in /_/src/NServiceBus.Core/Pipeline/MainPipelineExecutor.cs:line 78
      at NServiceBus.LearningTransportMessagePump.ProcessFile(ILearningTransportTransaction transaction, String messageId, CancellationToken messageProcessingCancellationToken) in /_/src/NServiceBus.Core/Transports/Learning/LearningTransportMessagePump.cs:line 347

Additional Information

Workarounds

• Ensure a separate table is used for each endpoint
• Always set a partition key that makes the outbox records partitioned by endpoint

Possible solutions

Using the endpoint name is one approach eg. SQL Persistence, RavenDB, MongoDB

Additional information

Related to Particular/NServiceBus.Storage.MongoDB#767 and Particular/NServiceBus#7402