Add verification step to WGE bootstrap

[Samra10]

What changed?

• Remove the component status checker because a dynamic list should exist.
• Add a status checker for the ready property of the helmrelease.

Why was this change made?

• To depend on the HR status regardless of the dependent components list.

How did you validate the change?

• Explain how a reviewer can verify the change themselves

• Throw interactive or non-interactive, after flux finishes HR reconciliation it will wait for the HelmRelease Ready status to be True, or timeout finish

• Unit tests -- what is covered, what cannot be covered; are
  there tests that fail without the change?

• Add unit_tests for case HR ready is True, and case timeout and status still not True.

[enekofb]

it feels better this direction so we have helm release a target. however, how this approach is different to flux native helm release wait?

btw, when you tested the helm release default wait mechanism ... what was the testing scenario so i could have a look in case find out anything ...

i just suspect that is missconfiguration during upgrades in wge side.

if you https://helm.sh/docs/helm/helm_upgrade/ ... my hypothesis is that

 --wait                                       if set, will wait until all Pods, PVCs, Services, and minimum number of Pods of a Deployment, StatefulSet, or ReplicaSet are in a ready state before marking the release as successful. It will wait for as long as --timeout

this condition

and minimum number of Pods of a Deployment,

is met but the application might not be healthy ...

likely because of this fluxcd/helm-controller#355 (comment)

[enekofb]

Helm considers a deployment with one replica and strategy rollingUpdate with maxUnavailable of 1 to be ready when it has 1 unavailable deployment. (Even if the deployment's one and only pod has entered CrashLoopBackOff and readiness and liveness checks have all failed... with maxUnavailable of 1 and replicas of 1, the deployment technically has no more than the allowed number of unavailable pods, so it is considered ready.)

[enekofb]

Options explored:

1. helmrelease cr dependensOn
2. helmrelease values -> not having dependencies
3. helm release depndency from the helm chart -> not necessarily being used (for example template controller)
4. compute helm release ready status on code (this PR)
5. flux helm controller (a desirable solution but not having the expected results)