Use Azure blob storage for outbox transportoperations is byte[] exceeds 64KB

[ramonsmits]

Describe the feature.

When outbox needs to persist transportoperations that serialized exceed 64KB Azure Table storage rejects the write operation:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: fdr-nb-allocatie-servicebus
 ---> Azure.Data.Tables.TableTransactionFailedException: The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.
RequestId:93d0e9ac-c002-004c-3886-cfd6ef000000
Time:2023-08-15T14:40:19.7363326Z
Status: 400 (Bad Request)
ErrorCode: PropertyValueTooLarge

The user can configure databus with Azure BLOB storage when only sending a few messages but better would be Azure storage outbox would detect that the transport operations exceed 64KB and store the byte array in BLOB storage instead.

This way the user does not need to use databus and Azure persistence outbox can also work when sending a larger amount of messages that would exceed 64KB even when using databus for certain properties.

Additional Context

No response

[danielmarbach]

Yeah it would be possible to fall back to storage. It was discussed during the time when we looked at the CosmosDB outbox storage and again during the DynamoDB persistence design that for example, the way we treat the outbox entity as a single row is not optimal. In DynamoDB we have a restriction of 256 KB instead of 64 KB per item, but even there we are splitting the outbox entry into at least two parts. One is the metadata record and multiple transport operations. Then we can bulk load all transport operations in one go

[ramonsmits]

@danielmarbach indeed, instead of a single collection of transport operations these could be serialized independently. That would still have a max message size of 64KB but at leasts its then 64KB per azure table row / message.

[danielmarbach]

Another benefit is that you can then use transactions to write things atomicly and don't have to deal with the edge cases around coordinating too different services