introduce a lightweight tracking mode for connectors and update confirmation manager to cope with it

[Chengxuan]

Problem

for hyperledger/firefly-evmconnect#191 and #161

When using EVM Connect, downloading detailed block information to maintain an in-memory partial chain to track the latest block progress can be very costly. Especially, the number of transactions in each block is high. Therefore, for users who don't need to index block information and only want to use transaction submission and event listener features, it's a big overhead

Therefore, a lightweight mode is proposed to maintain the usability of transaction submission and event listeners.

Solution

This PR contains

• a major FireFly Connector API (FFCAPI) interface update to include a function for connectors to return the tracking mode that the connector is running in: GetBlockListenerTrackingMode
  • inMemoryPartialChain, this is the current tracking mode, the connector builds an in-memory partical chain to track the block progress of the blockchain. This mode downloads information of each block from the chain, causing more data transportation, to provide the benefit of a verbose record of block details.
  • headBlockNumber, this is the new tracking mode added, which is a light weight mode compared to the existing inMemoryPartialChain mode, the connector only records the head block number to track block progress. Note: this is an optional mode to opt in, the connector developer can decide whether it makes sense to add it An example implementation is proposed in EVM connector: introduce a light weight mode for tracking block progression firefly-evmconnect#190. EVM Connector also added a configuration ( connector.blockTrackingMode ) so that users can control which mode they'd like the connector to run under.
• Updated two components in the transaction manager to support the new headBlockNumber mode due to reduced information in that mode:
  • Confirmation manager: When the connector is in headBlockNumber, it will no longer be able to return a list of block information. Therefore, to surface the confirmation progress, a new field currentConfirmationCount has been introduced (alongside the existing targetConfirmationCount field) NOTE: it's possible to introduce a separate configuration (e.g. confirmations.confirmationMode) for confirmation manager to omit detailed block information when the connector is running in inMemoryPartialChain, but I don't see the need for it for now See the section below for details.
  • Block listener: as the sole purpose of the block listener is to index the block with details, it counter efficient to support it when the connector are not provide efficient caching for block details. It's disallowed when a connector is running under headBlockNumber tracking mode

Details about confirmation manager changes

Add a confirmation logic that only needs the chain head block number to avoid downloading full block headers / building an in-memory partial chain.

That keeps confirmation work cheap and reduces the amount of unnecessary data downloaded from the chain.

Implementation

When the block listener is in the headBlockNumber mode, the confirmation check is derived from the head minus the receipt block number. When the required depth is reached, validity is checked by re-fetching the transaction receipt and comparing its block hash to the pending item ( this means re-org / fork handling leans on receipt reconciliation against head-driven confirmation counts rather than walking downloaded block metadata).

Example confirmation output in two modes:

Existing mode (list of blocks built using in memory chain)

 {
  "confirmations": [
    {
      "blockHash": "0xxxxxxxxxx1",
      "blockNumber": "1",
      "parentHash": "0xxxxxxxxxx0"
    },
    {
      "blockHash": "0xxxxxxxxxx2",
      "blockNumber": "2",
      "parentHash": "0xxxxxxxxxx1"
    }
  ],
  "confirmed": true,
  "currentConfirmationCount": 1,
  "targetConfirmationCount": 1
}

Light weight mode (subset information):

 {
  "confirmed": true,
  "currentConfirmationCount": 1,
  "targetConfirmationCount": 1
}

[hosie]

LGTM with one comment.

[hosie]

What is confirmation mode and validatedBlocks? Was this previous spelling for BlockListenerTrackingMode and inMemoryPartialChain

[Chengxuan]

yes, thanks for catching it, will update the comment

[peterbroadhurst]

Don't understand this comment sorry @Chengxuan

By currently do you mean there's a TODO? Or was this written before you implemented your change and you need to update the comment?

[Chengxuan]

No, it's a decision. I'll clean up currently

[peterbroadhurst]

Marking approval, as @hosie and @Chengxuan discussed this and reached a joint reasoned proposal on the spelling of:

• The setting is connector.chainTrackingMode, and is enabled on the the specific connector implementation (firefly-evmconnect in this case)
• The two possible values are:
  • full
  • light

These values are described in code comments on the FFTM side:

https://github.com/kaleido-io/firefly-transaction-manager/blob/852459b5a29d758290410fb27617a66d6fd28cf6/pkg/ffcapi/api.go#L104-L107