DAOS-17427 test: Auto-restart after self-terminate tests

[tanabarr]

Functional tests for the automatic engine restart feature introduced
in the control plane. These tests verify that engines automatically
restart after self-termination when excluded from the system, with
cases to verify disabling, rate-limiting and configuration support.

Test-tag: hw,medium,dmg,control,engine_auto_restart

Steps for the author:

• Commit message follows the guidelines.
• Appropriate Features or Test-tag pragmas were used.
• Appropriate Functional Test Stages were run.
• At least two positive code reviews including at least one code owner from each category referenced in the PR.
• Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

• Gatekeeper requested (daos-gatekeeper added as a reviewer).

[tanabarr]

keeping in draft until the parent PR lands and these tests are passing, reviews very much appreciated

[github-actions]

Ticket title is 'Handle engine suicides by automatically restarting the engines'
Status is 'In Review'
Labels: 'request_for_2.8'
https://daosio.atlassian.net/browse/DAOS-17427

[tanabarr]

thanks for the changes @daltonbohning , what lint configs should I use locally ?

[daltonbohning]

thanks for the changes @daltonbohning , what lint configs should I use locally ?

1. The githooks: https://github.com/daos-stack/daos/tree/master/utils/githooks#readme
   • They are not perfect and will not catch everything
2. The GitHub Actions lint checks for the PR which in fact are still failing
3. Some of this was just standardization not caught by linters

[daosbuild3]

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18006/7/execution/node/676/log

[daosbuild3]

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18006/8/execution/node/419/log