Improve global expectation handling and synchronization

[cgalibern]

Description of Changes

• Enhanced logic for global expectation convergence by adding checks for delayed propagation and safeguards for instance configuration accuracy.
• Ensured global expectations are fetched from peers during startup to minimize the risk of loss.
• Verified instance configuration via modified config file timestamps.

Details

• Global Expectation Updates: Improved handling and convergence logic for better synchronization.
• Startup Synchronization: Added mechanisms to fetch pending global expectations from peer nodes.
• Configuration Accuracy: Introduced safeguards and validation strategies for instance configurations.

reproducer: loop on following shell

#!/bin/bash

set -e
set -x
OSVC_BR1=br-prd
SVC=c28svc11
om $SVC create --wait
om $SVC config update --set ip#0.ipdev=${OSVC_BR1} --set ip#0.ipname=$SVC --set ip#0.shared=true
om $SVC config update --set disk#1.type=loop --set disk#1.file=/{env.pool1name}.loopfile --set disk#1.size=100M --set disk#1.standby=true --set env.pool1name=data1pool$SVC
om $SVC config update --set disk#2.type=loop --set disk#2.file=/{env.pool2name}.loopfile --set disk#2.size=100M --set disk#2.standby=true --set env.pool2name=data2pool$SVC
om $SVC config update --set disk#3.type=loop --set disk#3.file=/{env.pool3name}.loopfile --set disk#3.size=100M --set disk#3.standby=true --set env.pool3name=localpool$SVC
om $SVC config update --set disk#5.type=zpool --set disk#5.name={env.pool1name} --set disk#5.vdev="{disk#1.file}" --set disk#5.standby=true --set disk#5.create_options="-O acltype=posixacl -O xattr=sa -O compression=off -O normalization=formD -O mountpoint=legacy -O canmount=noauto -O devices=on -O exec=on -O setuid=on"
om $SVC config update --set disk#6.type=zpool --set disk#6.name={env.pool2name} --set disk#6.vdev="{disk#2.file}" --set disk#6.standby=true --set disk#6.create_options="-O acltype=posixacl -O xattr=sa -O compression=off -O normalization=formD -O mountpoint=legacy -O canmount=noauto -O devices=on -O exec=on -O setuid=on"
om $SVC config update --set disk#7.type=zpool --set disk#7.name={env.pool3name} --set disk#7.vdev="{disk#3.file}" --set disk#7.standby=true --set disk#7.create_options="-O acltype=posixacl -O xattr=sa -O compression=off -O normalization=formD -O mountpoint=legacy -O canmount=noauto -O devices=on -O exec=on -O setuid=on"
om $SVC config update --set fs#1.dev={env.pool1name}/fs1 --set fs#1.mnt=/srv/{svcname}/fs1 --set fs#1.type=zfs --set fs#1.mnt_opt="rw,xattr,acl" --set fs#1.shared=true --set fs#1.user=root --set fs#1.group=0 --set fs#1.perm=755
om $SVC config update --set fs#2.dev={env.pool1name}/fs2 --set fs#2.mnt=/srv/{svcname}/fs2 --set fs#2.type=zfs --set fs#2.mnt_opt="rw,xattr,acl" --set fs#2.shared=true
om $SVC config update --set DEFAULT.nodes={clusternodes}
om $SVC provision --wait
om $SVC purge --wait