All health checks return a status of ok or ready even when the service is not functioning

[WadeBarnes]

We've run into an issue in a couple of our instances were the agent and/or service database fills up forcing the pods into a crash loop and creating a situation were requests cannot be completed.

When this occurs one or more of the health checks should fail bringing the service outage to the attention of the monitoring solution. However, all of the health checks pass in this scenario, and the issues only surface when interacting with the service to authenticate.

Example application authorization failure messages:

https://vc-authn-oidc.apps.silver.devops.gov.bc.ca/authorize?scope=openid+vc_authn&state=n04bV2BP3ZJvuQCRMDY4a9vUvytm4O9dtsxvMCAnv4k.obdK6SW6E5U.1Ks0kstcS96yQ7oQbRVp5g.eyJydSI6Imh0dHBzOi8vanVzdGljZS5nb3YuYmMuY2EvZGFycy9zaWduaW4tb2lkYyIsInJ0IjoiY29kZSIsInJtIjoiZm9ybV9wb3N0Iiwic3QiOiJDZkRKOEwyRXlWbzlMNTlEaWp5Tmw2QXA2WHBTcXZabEw0ZS1UZUN6dmxMRDItSDhfMkZkbmhrNm1xaDhfMXNsamZtQVlZbV9sMU1nUF9ZczBlcDhUZXJuSlhvaHlwWU8tLWMxM1Q2YS1rVTU2N0VfdVV5UU9RbDVjX3JtN2ZRbE02RDBjamZVdkxqUXF5YXJRNU84T3lIZlNlNXlidjN3Zi0wNTdFYzh1REpDODhiZW5mV3pkN04wQy1fZENWakcxdWkzNmxVd0owbnpPZnFLX0laeV9Ec21MZEhlWHJkeklVdE54bVh5VGVibERDWngxOTBkNU5sbHZrZEJxdGc0c29SaDhyWm01VE1faDdYQkgxeFg2TGJvaUZieU02UGpvS1lZejZ2aWk2d2dxZ3EtVTZzTGpGNU5UUXAtcHB6RjlhRG9WWTdTZlBackNETVRhSkczZnU0UUhBdUVqMi1UcTg0RmU0bHEyN2FfaXNsU3BWdXUzWDhpd2hibDVoWnIwekExNTFwQTZUZkhEQ3RmM3Q5a21ScEdWbFZZS0hXUk01UEZWaFFTT3ZwUEZGeWdVdlNlLXJfYTBwaXJTY2tzWW1wVHZ3In0&response_type=code&client_id=acm-jag-vc&redirect_uri=https%3A%2F%2Floginproxy.gov.bc.ca%2Fauth%2Frealms%2Faccess-to-court-materials-jag%2Fbroker%2Fverifiable-credential%2Fendpoint&pres_req_conf_id=accredited-lawyer-bcpc&nonce=8ljpH6WoNbkEYJiVW05bYA

https://vc-authn-oidc.apps.silver.devops.gov.bc.ca/authorize?scope=openid+vc_authn&state=j0hISkwAxTdgls7xZncQ9l0LipraGCVNdKtosSkJjqs.cRQbpYsjLNk.1Ks0kstcS96yQ7oQbRVp5g.eyJydSI6Imh0dHBzOi8vanVzdGljZS5nb3YuYmMuY2EvZGFycy9zaWduaW4tb2lkYyIsInJ0IjoiY29kZSIsInJtIjoiZm9ybV9wb3N0Iiwic3QiOiJDZkRKOEwyRXlWbzlMNTlEaWp5Tmw2QXA2WHBEc3FDclE2TG96MGFsTWg5ZkpDRGpSWjZtSFNzM1NFdUdzX2E0Rl9MZmFnUDgwT0F0SDlDM2ZTRmp1Y1RsYkdsZE1lc1VNV2xLZ1BDU2pPRXVXb25UbGVnOXhPQ1ZmMXo4WWVOaTZQczZmVm0tc2NCbXdleENYMXg2MnlhVnFQY2xVTDNRSnJrdmdMMmRtYlc4dDdyUWV1OWRxZUZWbWtzcEpnaVZheDhKNmhQZlNXdlNBNTZORVdYdjBTYXdaR2IyQ1NoOEhiSkJBQmFBclBsakVOMUd4UU00X3RKUXIzdkVQdTZEbE40V1NhcktOUmdvSHQ2a3pGWVNIdWVEN08yR21lLVVHTmk3RkNmOVFmY1oyZzBsQkZ6VnNMellOVlpHa1huLW1xdGswUkdKdkhaQmd2MlFHYlBlYVZfNkZhdFN0UjF4ek9BVlVwbGljallud2hSaUk1OHN4WGZSMEZFblgwY0I3TTNWS0JxSk9JUmI5NEJsd3N0NWpwSFZ4eUFUa1dyNnNIU0d1SFQwb3ZwS2tPTm5GOGVlUm5RVGlBOEUtS3hIRUp3eW5kemJTeVViMmtSWHpIeWlsMTRqTlM3NFRtUTFfU3pIOXByak1obVRIQUJvIn0&response_type=code&client_id=acm-jag-vc&redirect_uri=https%3A%2F%2Floginproxy.gov.bc.ca%2Fauth%2Frealms%2Faccess-to-court-materials-jag%2Fbroker%2Fverifiable-credential%2Fendpoint&pres_req_conf_id=csb-transcriber&nonce=Wt8Ty97EAqyOIGtWZboHOQ

Example health check responses - while the above errors are occurring:

[esune]

We got into a conversation around this the other day, I think we need to define what we want each one of these health/status checks to follow. As it was noted, if the VC-AuthN API is responding to requests the service should probably be considered live, however if it is unable to complete operations that are required by its usage (i.e.: writing to the database or the agent writing to the wallet) it woul dbe best to return it as not ready and similarly have its health endpoint return an error.

VC-AuthN is made-up of several components that can fail at different times, and I am wondering what is the best way to expose this without becoming over-dependent on the accessory services. One example is the agent, since it could be running as a single-instance service, or as part of a multi-tenant service that is NOT necessarily controlled by the VC-AuthN deployer.