From 3e76ebbfa2a5f7e2ee425c9a8054a4e28c260132 Mon Sep 17 00:00:00 2001
From: Kenneth Cain <kenneth.cain@hpe.com>
Date: Wed, 15 Apr 2026 14:19:40 -0400
Subject: [PATCH 1/4] DAOS-17306 doc: self-healing properties, interactive
 rebuild

For the DAOS version 2.8 release, add two major sections to the
DAOS Administrator's Guide:
- self-healing properties / policy controls (DAOS-17306)
- explicit / interactive rebuild control (DAOS-17281)

Doc-only: true

Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>
---
 docs/admin/rebuild_controls.md | 336 +++++++++++++++++++++
 docs/admin/self_healing.md     | 515 +++++++++++++++++++++++++++++++++
 mkdocs.yml                     |   2 +
 3 files changed, 853 insertions(+)
 create mode 100644 docs/admin/rebuild_controls.md
 create mode 100644 docs/admin/self_healing.md

diff --git a/docs/admin/rebuild_controls.md b/docs/admin/rebuild_controls.md
new file mode 100644
index 00000000000..dbd1d808e4a
--- /dev/null
+++ b/docs/admin/rebuild_controls.md
@@ -0,0 +1,336 @@
+# Detailed Pool Operations: Interactive Rebuild Controls
+
+## Introduction
+
+Although the self-healing policy can control whether a rebuild will occur,
+occasionally an administrator may need to stop a rebuild already started and
+restart it later with a "rebuild start" command (or, perform an alternate
+action/rebuild such as direct reintegration). For this purpose, DAOS provides
+the following interactive rebuild control command-line interfaces:
+
+* `dmg pool <label> rebuild stop [--force]`
+* `dmg pool <label> rebuild start`
+* `dmg system rebuild stop [--force]`
+* `dmg system rebuild start`
+
+The system commands will apply to all pools in the DAOS system (i.e., have the
+same effect as if multiple `pool rebuild stop` commands are issued, one per
+pool).
+
+Upon stopping a pool's rebuild, its rebuild state as reported by `dmg pool query`
+will be an idle state, and an error status=-2027 (`-DER_OP_CANCELED` DAOS error code).
+
+The effect of a rebuild stop command is "one shot", meaning only a pool's
+currently-running rebuild is stopped and there is no persistent effect on future
+operations. Subsequent self-healing automatic recovery, or administrator command
+(e.g., system stop, system/pool exclude, reintegrate, drain, pool extend) can
+trigger new rebuild(s). Also, if a rebuild is stopped, whatever progress it had
+made in reconstructing the data in the pool is not retained — a subsequent
+"rebuild start" command will start the rebuild from the beginning (i.e., this is
+not a pause/resume interface).
+
+Some circumstances where rebuild stop/start controls may be helpful are:
+
+* When a pool has insufficient free space to accommodate relocation of affected
+  data upon engine(s) excluded/drained. For example, if a rebuild fails with
+  `status=-1007` (`-DER_NOSPACE`) (that will likely repeat in its automatic
+  retries). Stopping such a rebuild allows an administrator to perform alternate
+  actions (e.g., directly reintegrate the lost engine(s); and/or pool expansion
+  to more engines).
+* A system with many tens of pools are all rebuilding simultaneously (requiring
+  substantial CPU and network resources in the DAOS system). And the
+  administrator deciding in such cases to stagger an overall recovery by
+  manually commanding pools to rebuild in smaller batches.
+* A long-running rebuild might be stuck (e.g., due to any unforeseen DAOS bug).
+
+
+## Rebuild Phases
+
+An important detail of the DAOS rebuild design/implementation is that it
+proceeds in two phases: the rebuild itself, and a following "reclaim" phase to
+clean up space that rebuild used while it was operating.
+
+For a **successful rebuild**, the sequence is:
+
+1. Run `op:Rebuild`
+2. Run `op:Reclaim` to clean up
+
+For a **failed rebuild**, the sequence is:
+
+1. Run `op:Rebuild`
+2. Run `op:Fail_reclaim` to clean up
+    * If `Fail_reclaim` failed, retry `Fail_reclaim`
+    * If `Fail_reclaim` succeeded, retry the original `op:Rebuild`
+
+The "rebuild stop" commands are not typically allowed to terminate a rebuild in
+the `op:Reclaim` and `op:Fail_reclaim` phases — instead the command must be
+issued during the `op:Rebuild` execution. An exception is available with the
+`--force` option to "rebuild stop", intended to be applied for rebuilds that
+repeatedly fail and possibly may even be looping `Fail_reclaim` operations.
+
+Because of these details, carefully timing the execution of "rebuild stop"
+commands is needed, which can be facilitated with pool rebuild state querying
+with `dmg pool query`. See the section
+[Rebuild Stop Command Errors](#rebuild-stop-command-errors) for examples of
+errors returned by "rebuild stop" in different timing circumstances.
+
+
+## Example Usage: Stop a Single Pool Rebuild, then Direct Reintegration
+
+A system has detected the loss of an engine (rank 3) and has launched a
+corresponding rebuild on pool `p1` for that exclusion. The administrator decides
+in this case that it is preferable to stop the exclude rebuild rather than let
+it finish (perhaps it is possible to quickly remedy the rank 3 engine issue,
+restart it, and directly reintegrate it back into the system and pool — rather
+than perform 2 separate rebuilds for the exclusion and the reintegration).
+
+**1.** Observe the pool `p1` is rebuilding after the fault is detected.
+
+```bash
+$ dmg pool query --health-only p1
+Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=8, leader=6, version=77, state=TargetsExcluded
+Pool health info:
+- Disabled ranks: 3
+- Rebuild busy, 1 objs, 0 recs
+- Data redundancy: degraded
+```
+
+**2.** Stop the rebuild and monitor with pool query until the stop is confirmed
+("stopped", state=idle, status=-2027). Notice some pool query rebuild state
+changes while waiting.
+
+```bash
+$ dmg pool rebuild stop p1
+```
+
+Run `dmg pool query` in a loop (with short sleeps between commands):
+
+```bash
+# confirmation that stop command is being handled
+$ dmg pool query --health-only p1
+Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=8, leader=6, version=77, state=TargetsExcluded
+Pool health info:
+- Disabled ranks: 3
+- Rebuild stopping (state=busy, status=-2027)
+```
+
+```bash
+# state temporarily changes back to busy, representing the Fail_reclaim stage
+# for the stopped operation is now running
+$ dmg pool query --health-only p1
+Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=8, leader=6, version=77, state=TargetsExcluded
+Pool health info:
+- Disabled ranks: 3
+- Rebuild busy, 0 objs, 0 recs
+```
+
+```bash
+# Fail_reclaim has finished, and now the pool shows its rebuild has fully stopped
+$ dmg pool query --health-only p1
+Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=8, leader=6, version=77, state=TargetsExcluded
+Pool health info:
+- Disabled ranks: 3
+- Rebuild stopped (state=idle, status=-2027)
+- Data redundancy: degraded
+```
+
+**3.** Directly reintegrate the engine back into pool `p1` and monitor progress
+with multiple pool query commands until the reintegration rebuild is
+successfully finished.
+
+```bash
+$ dmg pool reintegrate --ranks=3 p1
+```
+
+Run `dmg pool query` in a loop (with short sleeps between commands):
+
+```bash
+$ dmg pool query --health-only p1
+Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=0, leader=6, version=85, state=Ready
+Pool health info:
+- Rebuild stopped (state=idle, status=-2027)
+- Data redundancy: normal
+```
+
+```bash
+$ dmg pool query --health-only p1
+Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=0, leader=6, version=85, state=Ready
+Pool health info:
+- Rebuild busy, 0 objs, 0 recs
+- Data redundancy: normal
+```
+
+```bash
+$ dmg pool query --health-only p1
+Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=0, leader=6, version=85, state=Ready
+Pool health info:
+- Rebuild busy, 1 objs, 455081984 recs
+- Data redundancy: normal
+```
+
+```bash
+# Notice zero objs, recs corresponds to a transition from Rebuild done to Reclaim phase now running.
+$ dmg pool query --health-only p1
+Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=0, leader=6, version=93, state=Ready
+Pool health info:
+- Rebuild busy, 0 objs, 0 recs
+- Data redundancy: normal
+```
+
+```bash
+$ dmg pool query --health-only p1
+Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=0, leader=6, version=93, state=Ready
+Pool health info:
+- Rebuild busy, 1 objs, 0 recs
+- Data redundancy: normal
+```
+
+```bash
+# all done
+$ dmg pool query --health-only p1
+Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=0, leader=6, version=93, state=Ready
+Pool health info:
+- Rebuild done, 1 objs, 0 recs
+- Data redundancy: normal
+```
+
+
+## Example Usage: Stop Multiple Pools Rebuilding
+
+Consider a DAOS system with multiple pools, labeled `p1`, `p2`, etc. A
+`daos_engine` suffers a fault and the system detection of the engine loss
+results in pool map updates and all pools rebuilding. The administrator would
+prefer to stop the exclusion rebuilds, repair/restart the engine, and directly
+reintegrate the engine into the system and all pools.
+
+**1.** Observe the engine is excluded from the system, and pools begin
+rebuilding after the fault is detected.
+
+```bash
+$ dmg system query
+Rank      State
+----      -----
+[0-2,4-7] Joined
+3         Stopped
+```
+
+This shows that pool `p2` has started its rebuild (state "busy"). `p1` is
+"done" from a prior rebuild, and will start rebuilding soon.
+
+```bash
+$ dmg system list-pools -v
+Label UUID                                 State           SvcReps   SCM Size SCM Used SCM Imbalance NVME Size NVME Used NVME Imbalance Disabled UpgradeNeeded? Rebuild State
+----- ----                                 -----           -------   -------- -------- ------------- --------- --------- -------------- -------- -------------- -------------
+p1    cdf27ec1-ed97-4aa6-a766-39a2ed2136a1 Ready           [0-1,4-6] 105 GB   37 GB    28%           0 B       0 B       0%             0/64     None           done
+p2    dcfe63e0-e10f-464d-b18c-0915e52e048c TargetsExcluded [0-2,6-7] 49 GB    4.6 GB   62%           0 B       0 B       0%             8/64     None           busy
+```
+
+**2.** Stop all pool rebuilds with one command, and confirm they have stopped.
+
+```bash
+$ dmg system rebuild stop
+
+$ dmg system list-pools -v
+Label UUID                                 State           SvcReps   SCM Size SCM Used SCM Imbalance NVME Size NVME Used NVME Imbalance Disabled UpgradeNeeded? Rebuild State
+----- ----                                 -----           -------   -------- -------- ------------- --------- --------- -------------- -------- -------------- -------------
+p1    cdf27ec1-ed97-4aa6-a766-39a2ed2136a1 TargetsExcluded [0-1,4-6] 105 GB   37 GB    28%           0 B       0 B       0%             8/64     None           idle
+p2    dcfe63e0-e10f-464d-b18c-0915e52e048c TargetsExcluded [0-2,6-7] 49 GB    4.6 GB   62%           0 B       0 B       0%             8/64     None           idle
+
+$ dmg pool query --health-only p1
+Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=8, leader=6, version=102, state=TargetsExcluded
+Pool health info:
+- Disabled ranks: 3
+- Rebuild stopped (state=idle, status=-2027)
+- Data redundancy: degraded
+
+$ dmg pool query --health-only p2
+Pool dcfe63e0-e10f-464d-b18c-0915e52e048c, ntarget=64, disabled=8, leader=7, version=46, state=TargetsExcluded
+Pool health info:
+- Disabled ranks: 3
+- Rebuild stopped (state=idle, status=-2027)
+- Data redundancy: degraded
+```
+
+**3.** Restart the affected engine, reintegrate engine and monitor rebuilds
+until finished.
+
+```bash
+$ dmg system start --ranks=3
+
+$ dmg system reintegrate --ranks=3
+
+$ dmg system list-pools -v
+Label UUID                                 State SvcReps   SCM Size SCM Used SCM Imbalance NVME Size NVME Used NVME Imbalance Disabled UpgradeNeeded? Rebuild State
+----- ----                                 ----- -------   -------- -------- ------------- --------- --------- -------------- -------- -------------- -------------
+p1    cdf27ec1-ed97-4aa6-a766-39a2ed2136a1 Ready [0-1,4-6] 120 GB   42 GB    28%           0 B       0 B       0%             0/64     None           busy
+p2    dcfe63e0-e10f-464d-b18c-0915e52e048c Ready [0-2,6-7] 56 GB    4.9 GB   62%           0 B       0 B       0%             0/64     None           busy
+```
+
+Wait for rebuilds to complete:
+
+```bash
+$ dmg system list-pools -v
+Label UUID                                 State SvcReps   SCM Size SCM Used SCM Imbalance NVME Size NVME Used NVME Imbalance Disabled UpgradeNeeded? Rebuild State
+----- ----                                 ----- -------   -------- -------- ------------- --------- --------- -------------- -------- -------------- -------------
+p1    cdf27ec1-ed97-4aa6-a766-39a2ed2136a1 Ready [0-1,4-6] 120 GB   42 GB    31%           0 B       0 B       0%             0/64     None           done
+p2    dcfe63e0-e10f-464d-b18c-0915e52e048c Ready [0-2,6-7] 56 GB    4.9 GB   62%           0 B       0 B       0%             0/64     None           done
+```
+
+
+## Rebuild Stop Command Errors
+
+The rebuild stop command may return errors when it is issued at a time that it
+may not be able to handle the request. For example:
+
+### No Rebuild Currently Running
+
+When no rebuild is currently running, the command will report a "nonexist"
+error:
+
+```bash
+# system command applying to 2 pools
+$ dmg system rebuild stop
+System-rebuild stop request succeeded on 0 pools
+ERROR: dmg: system rebuild stop failed: pool-rebuild stop failed on pool p1: pool-rebuild stop failed: DER_NONEXIST(-1005): The specified entity does not exist, pool-rebuild stop failed on pool p2: pool-rebuild stop failed: DER_NONEXIST(-1005): The specified entity does not exist
+
+# single pool command
+$ dmg pool rebuild stop p2
+ERROR: dmg: pool-rebuild stop failed: DER_NONEXIST(-1005): The specified entity does not exist
+```
+
+### Rebuild Finished, Reclaim in Progress
+
+When the rebuild stage has successfully finished and is in its reclaim cleanup
+stage, `dmg` will report a busy error. For example when pools `p1` and `p2` are
+both done rebuilding and in the reclaim stage:
+
+```bash
+$ dmg system rebuild stop
+System-rebuild stop request succeeded on 0 pools
+ERROR: dmg: system rebuild stop failed: pool-rebuild stop failed on pool p1: pool-rebuild stop failed: DER_BUSY(-1012): Device or resource busy, pool-rebuild stop failed on pool p2: pool-rebuild stop failed: DER_BUSY(-1012): Device or resource busy
+```
+
+### Rebuild Failed, Fail\_reclaim in Progress
+
+When the rebuild stage has finished (but failed), and is in its `Fail_reclaim`
+cleanup stage, `dmg` will report a no permissions error, `-DER_NO_PERM`.
+
+In this scenario, the admin can wait for the rebuild to be retried, and then
+reissue the "rebuild stop" command:
+
+```bash
+# multiple command invocations to query pool rebuild status
+$ dmg pool query <pool_label>
+# Possibly observe Rebuild failed, status=<nonzero_error_code>
+# Observe Rebuild busy (representing the retried rebuild)
+
+$ dmg pool rebuild stop <pool_label>
+```
+
+Or, if the `Fail_reclaim` itself is failing and retrying, resulting in
+`-DER_NO_PERM` in every attempt to stop the rebuild, then the force option can
+be used to terminate the bad loop:
+
+```bash
+$ dmg pool rebuild stop --force <pool_label>
+```
diff --git a/docs/admin/self_healing.md b/docs/admin/self_healing.md
new file mode 100644
index 00000000000..50094204d69
--- /dev/null
+++ b/docs/admin/self_healing.md
@@ -0,0 +1,515 @@
+# Detailed Pool Operations: Self-Healing (Rebuild) Configuration
+
+## Introduction
+
+This section describes some of the mechanisms using the `dmg` tool that can be
+employed to manage DAOS' self-healing feature. As a reminder, the system
+consists of a set of DAOS engine instances that join the system when they are
+started. An engine is assigned a numeric rank upon initially joining the system.
+This document may frequently interchange the term "rank" and "engine" to refer
+to a DAOS engine.
+
+By default, the system works as previous versions (per-pool `self_heal`
+property) if one doesn't modify the "new" system `self_heal` property. An
+administrator doesn't have to use the new system `self_heal` after upgrading to
+DAOS version 2.8 and instead can choose to adopt it in the administrator's
+workflow when the need arises.
+
+DAOS behavior can be managed through "system" and "pool" properties.
+
+### System and Per-pool Properties
+
+System-level properties apply to system behavior and examples include system
+name and pool scrub mode. System-level properties are shared and centralized in
+the control-plane management service and can be read by the code in the DAOS
+engines as needed. Some properties are also writable. The administrator can read
+and write system properties using `dmg system get-prop` / `dmg system set-prop`.
+
+Pool-level properties apply to pool-specific behavior and each pool has its own
+set of pool-properties, which can be read and written by the system
+administrator using `dmg pool get-prop` / `dmg pool set-prop`.
+
+### Self-heal Properties
+
+The `self_heal` property describes behavior associated with self-healing and can
+be set at both the system and pool levels. For a given pool to perform the
+recovery actions (map change, rebuild), the policy needs to be enabled at both
+the system and per-pool level.
+
+The system level property has flags `exclude;pool_exclude;pool_rebuild` which
+indicate the following:
+
+* **`exclude`** — Whether engines get excluded from the system membership based
+  on (SWIM) activity detection.
+* **`pool_exclude`** — Whether when engine/target states change (e.g., as a
+  result of engine exclusion from the system), all relevant affected pools' pool
+  maps will be automatically updated (or not).
+* **`pool_rebuild`** — Whether, following a pool map update, rebuild activities
+  automatically get triggered (or not) for the affected pools.
+
+The pool level property has flags `exclude;rebuild;delay_rebuild` which indicate
+the following:
+
+* **`pool.exclude`** — Related to the `system.pool_exclude` property and
+  together they determine if a specific pool's pool map will be updated.
+  Automatic pool map change happens iff `system.pool_exclude` and
+  `pool.exclude` are set to true.
+* **`pool.rebuild`** — Related to the `system.pool_rebuild` property and
+  together they determine if rebuild activity will be automatically triggered
+  upon a pool map update. Automatic rebuild on a pool will happen iff
+  `system.pool_rebuild` and `pool.rebuild` are set to true.
+* **`delay_rebuild`** — Mutually exclusive with `rebuild` and specifies that
+  rebuild does not necessarily trigger automatically and can be delayed based on
+  user requirements. Delay rebuild is mostly out of scope for this section.
+
+On starting a DAOS system and pool creation, default `self_heal` flags will be
+set as follows:
+
+* **System-level:** `exclude;pool_exclude;pool_rebuild`
+* **Pool-level:** `exclude;rebuild`
+
+The following sections describe different administration scenarios, and how
+self-heal property settings can be applied in each case. When a scenario
+describes loss of an engine and waiting for a "detection delay", the delay
+refers roughly to the amount of time for:
+
+* Two SWIM protocol periods to occur,
+* Plus a SWIM suspicion timeout to occur,
+* Plus a `CRT_EVENT_DELAY` number of seconds to elapse, during which failure
+  events can be aggregated into a logical event,
+* Plus some margin of error.
+
+
+## System / Pool Creation and Disabling / Enabling Self-Heal
+
+The following are example steps for managing `self_heal` policies. This
+describes a very simple workflow example which disables self-healing behavior
+during a typical cluster setup so that automatic exclusion and rebuild
+activities are disabled until the system and pools have been brought up and are
+ready for use.
+
+**1.** Display system `self_heal` property value. The default starting value for
+system self-heal is to have all flags set.
+
+```bash
+$ dmg system get-prop self_heal
+Name                                        Value
+----                                        -----
+Self-heal policy for the system (self_heal) exclude;pool_exclude;pool_rebuild
+```
+
+**2.** Disable system self-heal features while bringing up system. This will
+prevent automatic exclusions and rebuild at the system level during periods of
+flux in the system at the time of start-up.
+
+```bash
+$ dmg system set-prop self_heal:'none'
+system set-prop succeeded
+
+$ dmg system get-prop self_heal
+Name                                        Value
+----                                        -----
+Self-heal policy for the system (self_heal) none
+```
+
+**3.** Add ranks to system by starting storage server DAOS services and
+formatting so that they join.
+
+Use `daos_server start` and `dmg storage format` commands.
+
+**4.** Run system query which will indicate disabled system `self_heal` flags.
+
+```bash
+$ dmg system query
+Rank  State
+----  -----
+[0-3] Joined
+System property self_heal flags disabled: exclude, pool_exclude, pool_rebuild
+```
+
+**5.** Set system `self_heal.exclude` flag so inactive ranks get excluded from
+the system membership. This can be enabled once all ranks have been
+successfully joined and stable to avoid flux during system join phase. System
+`self_heal.pool_exclude` and `self_heal.pool_rebuild` flags are disabled and
+can be re-enabled after pools have been created.
+
+```bash
+$ dmg system set-prop self_heal:'exclude'
+system set-prop succeeded
+
+$ dmg system query
+Rank  State
+----  -----
+[0-3] Joined
+System property self_heal flags disabled: pool_exclude, pool_rebuild
+```
+
+**6.** Create two pools using half of the available space across all joined
+engines in the system.
+
+```bash
+$ dmg pool create first_pool -z 50%
+Creating DAOS pool with 50% of all storage
+Pool created with 4.44%,95.56% storage tier ratio
+-------------------------------------------------
+  UUID                 : 1c8f6a4e-a4eb-418d-bedc-0c7751e41af1
+  Service Leader       : 3
+  Service Ranks        : [0-3]
+  Storage Ranks        : [0-3]
+  Total Size           : 34 TB
+  Storage tier 0 (SCM) : 1.5 TB (372 GB / rank)
+  Storage tier 1 (NVMe): 32 TB (8.0 TB / rank)
+
+$ dmg pool create second_pool -z 100%
+Creating DAOS pool with 100% of all storage
+Pool created with 4.43%,95.57% storage tier ratio
+-------------------------------------------------
+  UUID                 : ce3a6a74-8a19-4c6e-a3ef-1ee85d8e06f1
+  Service Leader       : 2
+  Service Ranks        : [0-3]
+  Storage Ranks        : [0-3]
+  Total Size           : 34 TB
+  Storage tier 0 (SCM) : 1.5 TB (371 GB / rank)
+  Storage tier 1 (NVMe): 32 TB (8.0 TB / rank)
+```
+
+**7.** Once pools have been successfully created, `dmg pool query` output
+indicates the disabled system-property flags that will prevent pool map
+exclusion and pool rebuild. Then, enable all system self-heal features and show
+`dmg system query` displays no self-heal disabled flags in its output. And,
+`dmg pool query` output following the system property change will no longer
+report disabled exclude and rebuild.
+
+```bash
+$ dmg pool query first_pool
+Pool ce3a6a74-8a19-4c6e-a3ef-1ee85d8e06f1, ntarget=16, disabled=0, leader=1, version=1, state=Ready
+Pool health info:
+- Rebuild idle, 0 objs, 0 recs
+Pool space info:
+- Target count:16
+- Storage tier 0 (SCM):
+  Total size: 1.5 TB
+  Free: 1.4 TB, min:88 GB, max:88 GB, mean:88 GB
+- Storage tier 1 (NVME):
+  Total size: 32 TB
+  Free: 32 TB, min:2.0 TB, max:2.0 TB, mean:2.0 TB
+exclude disabled on pool due to [system] policy
+rebuild disabled on pool due to [system] policy
+
+$ dmg system set-prop self_heal:'exclude;pool_exclude;pool_rebuild'
+system set-prop succeeded
+
+$ dmg system query
+Rank  State
+----  -----
+[0-3] Joined
+
+$ dmg pool query first_pool
+Pool 1c8f6a4e-a4eb-418d-bedc-0c7751e41af1, ntarget=16, disabled=0, leader=0, version=1, state=Ready
+Pool health info:
+- Rebuild idle, 0 objs, 0 recs
+Pool space info:
+- Target count:16
+- Storage tier 0 (SCM):
+  Total size: 1.5 TB
+  Free: 1.4 TB, min:88 GB, max:88 GB, mean:88 GB
+- Storage tier 1 (NVME):
+  Total size: 32 TB
+  Free: 32 TB, min:2.0 TB, max:2.0 TB, mean:2.0 TB
+```
+
+**8.** Then disable pool `self_heal.exclude` and `self_heal.rebuild` flags on
+`first_pool` to stop rank inactivity from triggering pool exclusion and/or
+rebuild during maintenance of `first_pool`.
+
+```bash
+$ dmg pool get-prop first_pool self_heal
+Pool first_pool properties:
+Name                            Value
+----                            -----
+Self-healing policy (self_heal) exclude;rebuild
+
+$ dmg pool set-prop first_pool self_heal:'none'
+pool set-prop succeeded
+
+$ dmg pool get-prop first_pool self_heal
+Pool first_pool properties:
+Name                            Value
+----                            -----
+Self-healing policy (self_heal) none
+```
+
+**9.** Compare `dmg pool query` output for both pools and notice that self-heal
+policy disable messages are output only for `first_pool`, due to the difference
+in pool property values for each.
+
+```bash
+$ dmg pool query first_pool
+Pool 1c8f6a4e-a4eb-418d-bedc-0c7751e41af1, ntarget=16, disabled=0, leader=0, version=1, state=Ready
+Pool health info:
+- Rebuild idle, 0 objs, 0 recs
+Pool space info:
+- Target count:16
+- Storage tier 0 (SCM):
+  Total size: 1.5 TB
+  Free: 1.4 TB, min:88 GB, max:88 GB, mean:88 GB
+- Storage tier 1 (NVME):
+  Total size: 32 TB
+  Free: 32 TB, min:2.0 TB, max:2.0 TB, mean:2.0 TB
+exclude disabled on pool due to [pool] policy
+rebuild disabled on pool due to [pool] policy
+
+$ dmg pool query second_pool
+Pool ce3a6a74-8a19-4c6e-a3ef-1ee85d8e06f1, ntarget=16, disabled=0, leader=1, version=1, state=Ready
+Pool health info:
+- Rebuild idle, 0 objs, 0 recs
+Pool space info:
+- Target count:16
+- Storage tier 0 (SCM):
+  Total size: 1.5 TB
+  Free: 1.4 TB, min:88 GB, max:88 GB, mean:88 GB
+- Storage tier 1 (NVME):
+  Total size: 32 TB
+  Free: 32 TB, min:2.0 TB, max:2.0 TB, mean:2.0 TB
+```
+
+**10.** Enable pool policy flags and show `dmg pool query` command no longer
+displays any self-heal "disabled policy" messages.
+
+```bash
+$ dmg pool set-prop first_pool self_heal:'exclude;rebuild'
+pool set-prop succeeded
+
+$ dmg pool query first_pool
+Pool 1c8f6a4e-a4eb-418d-bedc-0c7751e41af1, ntarget=16, disabled=0, leader=0, version=1, state=Ready
+Pool health info:
+- Rebuild idle, 0 objs, 0 recs
+Pool space info:
+- Target count:16
+- Storage tier 0 (SCM):
+  Total size: 1.5 TB
+  Free: 1.4 TB, min:88 GB, max:88 GB, mean:88 GB
+- Storage tier 1 (NVME):
+  Total size: 32 TB
+  Free: 32 TB, min:2.0 TB, max:2.0 TB, mean:2.0 TB
+```
+
+
+## Planned, Online, Small-Scale System Maintenance
+
+### Scenario
+
+The following are example steps for managing a short maintenance task (e.g.,
+rebooting a single server, or stopping/restarting a small number of engines)
+while:
+
+* Keeping the DAOS system online and available for ongoing application I/O,
+  even in "degraded mode" (when pools do not have their original degree of data
+  redundancy).
+* Not triggering a rebuild on all pools as a result of the brief maintenance.
+
+The maintenance must not exceed pool redundancy factors (or the system
+`DAOS_POOL_RF`) for applications to continue running in degraded mode. At the
+conclusion of maintenance, engines are brought back into the system and
+reintegrated into all affected pools (exiting degraded mode).
+
+### Available Actions
+
+* At the system level, disable `system self_heal.pool_rebuild`.
+* Upon restart of the affected resource, reintegrate its engine(s) back into
+  the affected pools.
+* Re-enable `system self_heal.pool_rebuild` and trigger a re-evaluation of the
+  property.
+
+### Steps
+
+Begin the maintenance by disabling `system.self_heal.pool_rebuild`:
+
+```bash
+$ dmg system set-prop self_heal:'exclude;pool_exclude'
+```
+
+System and per-pool exclude remain enabled, important for allowing ongoing
+application I/O to proceed (without timeouts) while pools are in degraded mode.
+
+Stop the rank X engine and wait for the detection delay:
+
+```bash
+$ dmg system stop --ranks=X
+```
+
+* Rank X excluded from the system and from all pools using rank X targets for
+  storage. Pools operating in degraded mode.
+* No rebuild occurs.
+
+Start the rank X engine and wait for it to join the system. In this example,
+assume `pool_A` uses engine rank X targets for storage.
+
+```bash
+$ dmg system start --ranks=X
+$ dmg system query
+```
+
+* Rank X no longer excluded from the system.
+
+```bash
+$ dmg pool query-targets pool_A --rank=X
+```
+
+* Rank X targets in the relevant pools' maps are still in a DOWN state (and
+  require manual reintegration).
+
+Reintegrate the rank X engine into all affected pools:
+
+```bash
+$ dmg system reintegrate --ranks=X
+```
+
+* Rank X reintegrated in all relevant pools.
+* Rank X rebuilding in all relevant pools.
+
+End the maintenance by enabling all system `self_heal` flags and re-evaluating
+the assignment:
+
+```bash
+$ dmg system set-prop self_heal:'exclude,pool_exclude,pool_rebuild'
+$ dmg system self-heal eval
+```
+
+* No rebuild.
+* Any future (unplanned) faults will result in system exclusion, pool map
+  exclusion, and rebuild launched on the affected pools.
+
+
+## Planned, Offline, Large-Scale System Maintenance and Subsequent Restart
+
+### Scenario
+
+A large number of resources (`daos_server` or engine instances) will be stopped
+for some time to perform maintenance. Or, perhaps a large planned
+fabric/network disruption is scheduled that will disrupt connectivity to a large
+number of engines. The number of faults this would normally generate will exceed
+the system-wide setting for the number of tolerable failures (see engine
+environment variable `DAOS_POOL_RF`). As a result, the storage system will
+effectively be offline. System exclusions, pool map updates and rebuilding is
+not wanted for reasons such as:
+
+* During the stoppage, if the DAOS mechanism to aggregate multiple engine
+  failures into a single logical event does not get processed as a single
+  event, an initial small number of failures (< `DAOS_POOL_RF`) can trigger
+  pool map updates and rebuilding activity.
+* During restart, should a single or small number of engines
+  (< `DAOS_POOL_RF`) encounter an unexpected failure, system/pool map and
+  rebuilding activity would be triggered, though counterproductive — especially
+  if the faults can be quickly remedied and affected engines restarted.
+
+### Available Actions
+
+The admin can disable system `self_heal` (disable all flags). And can re-enable
+all flags after confirming all affected resources are running properly following
+the maintenance. After re-enabling system `self_heal`, the admin must trigger a
+re-evaluation of the modified property.
+
+### Steps
+
+**1.** Begin the maintenance by disabling self-heal:
+
+```bash
+$ dmg system set-prop self_heal:'none'
+```
+
+**2.** Perform planned maintenance. Shut down selected engines and wait for the
+detection delay.
+
+```bash
+$ dmg system stop --ranks=<more_than_RF_ranks>
+```
+
+**3.** Restart the engines that were stopped, then wait for all ranks to be
+joined to the system.
+
+```bash
+$ dmg system start --ranks=<above>
+$ dmg system query
+```
+
+**4.** End the maintenance by re-enabling self-heal:
+
+```bash
+$ dmg system set-prop self_heal:'exclude;pool_exclude;pool_rebuild'
+$ dmg system self-heal eval
+```
+
+
+## Planned, Normal System Restart
+
+### Scenario
+
+This is similar to the previous scenario, but involving a whole DAOS system
+restart. Again on restart, we do not want any unexpected single or small number
+of failures (< `DAOS_POOL_RF`) to cause system / pool exclusions (and
+rebuilding) activity.
+
+### Available Actions
+
+Same as the previous scenario.
+
+### Steps
+
+**1.** Begin the maintenance by disabling self-heal:
+
+```bash
+$ dmg system set-prop self_heal:'none'
+```
+
+**2.** Perform planned maintenance. Shut down all engines in the system.
+
+```bash
+$ dmg system stop
+```
+
+**3.** Restart the engines, then wait for all ranks to be joined to the system.
+
+```bash
+$ dmg system start
+$ dmg system query
+```
+
+**4.** End the maintenance by re-enabling self-heal:
+
+```bash
+$ dmg system set-prop self_heal:'exclude;pool_exclude;pool_rebuild'
+$ dmg system self-heal eval
+```
+
+
+## Problematic Pool
+
+### Scenario
+
+Consider a system with pools P and Q. For some reason, pool P will not be able
+to rebuild correctly (e.g., rebuild runs out of space due to insufficient `space_rb`
+property setting during pool create). In this case, a mechanism to disable rebuild for
+pool P is needed.
+
+### Available Actions
+
+Pool P `self_heal` property can be set such that rebuild is disabled.
+
+### Steps
+
+**1.** Disable rebuild on pool P:
+
+```bash
+$ dmg pool set-prop self_heal:'exclude' P
+```
+
+**2.** Simulate a fault by stopping the rank X engine and wait for the
+detection delay.
+
+```bash
+$ dmg system stop --ranks=X
+```
+
+* Rank X is excluded from the system and from pools P and Q.
+* Rank X is rebuilding in pool Q only. Pool P will remain in degraded mode.
diff --git a/mkdocs.yml b/mkdocs.yml
index b8443104668..2d20f7592d6 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -84,6 +84,8 @@ nav:
       - 'System Deployment': 'admin/deployment.md'
       - 'System Administration': 'admin/administration.md'
       - 'Pool Operations': 'admin/pool_operations.md'
+      - 'Detailed Pool Operations: Self-Healing Configuration': 'admin/self_healing.md'
+      - 'Detailed Pool Operations: Interactive Rebuild Controls': 'admin/rebuild_controls.md'
       - 'Tiering and Unified Namespace': 'admin/tiering_uns.md'
       - 'Performance Tuning': 'admin/performance_tuning.md'
       - 'Troubleshooting': 'admin/troubleshooting.md'

From caf846ebbc11cfc2ef8ff18e7e33613d550c6d50 Mon Sep 17 00:00:00 2001
From: Tom Nabarro <thomas.nabarro@hpe.com>
Date: Thu, 16 Apr 2026 13:32:00 +0100
Subject: [PATCH 2/4] Add documentation for the 'Data redundancy' field in dmg
 pool query output (introduced in PR #17371 / DAOS 2.6).

The new section explains:
- Field values (normal vs degraded)
- When to check this field
- Example usage with exclude-only self-heal policies
- How to verify exclusion completed when auto-rebuild is disabled

Updated pool query examples throughout to show the Data redundancy
field for consistency with DAOS 2.6+ output.

Particularly useful for scenarios where system.self_heal is set to
exclude,pool_exclude or pool self_heal has exclude bit set without
rebuild, to confirm exclusion has occurred.

Related-to: #17371
Doc-only: true
Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>
---
 docs/admin/self_healing.md | 131 +++++++++++++++++++++++++++++++++++++
 1 file changed, 131 insertions(+)

diff --git a/docs/admin/self_healing.md b/docs/admin/self_healing.md
index 50094204d69..e0fe81d7e97 100644
--- a/docs/admin/self_healing.md
+++ b/docs/admin/self_healing.md
@@ -80,6 +80,89 @@ refers roughly to the amount of time for:
 * Plus some margin of error.
 
 
+## Pool Query Data Redundancy Status
+
+**Available in:** DAOS 2.6+
+
+The `dmg pool query` command displays the pool's data redundancy status as part
+of the health information output. This field provides a clear indication of
+whether the pool has sufficient target availability to maintain data redundancy.
+
+### Output Field
+
+The `Data redundancy` field appears in the pool health information section:
+
+```
+Pool health info:
+- Rebuild idle
+- Data redundancy: normal
+```
+
+or when targets are excluded:
+
+```
+Pool health info:
+- Rebuild busy, 42 objs, 21 recs
+- Data redundancy: degraded
+```
+
+### Field Values
+
+| Value | Meaning |
+|-------|---------|
+| `normal` | All targets are UP and accessible; data redundancy is intact |
+| `degraded` | One or more targets are DOWN; data redundancy is compromised |
+
+### When to Check This Field
+
+1. **After automatic or manual exclusion** — Confirm that exclusion has completed
+2. **When self-heal is configured without automatic rebuild** — Verify exclusion
+   occurred even when rebuild doesn't start automatically (e.g., when using
+   `exclude` without `rebuild` flags)
+3. **Before manually triggering rebuild** — Confirm that exclusion is complete
+   and degraded state exists
+4. **After rebuild completion** — Verify that redundancy has been restored to
+   `normal` status
+5. **During troubleshooting** — Quick health check without parsing complex output
+
+### Example: Verifying Exclusion with Exclude-Only Policy
+
+When the self-heal policy has `exclude` but not `rebuild` (either at system or
+pool level), the `Data redundancy` field confirms that exclusion occurred:
+
+```bash
+$ dmg pool query my_pool
+Pool 6f450a68-8c7d-4da9-8900-02691650f6a2, ntarget=8, disabled=1, state=TargetsExcluded
+Pool health info:
+- Disabled ranks: 3
+- Rebuild idle
+- Data redundancy: degraded  ← Confirms exclusion completed
+```
+
+The combination of:
+- `Disabled ranks: 3` — Shows which rank was excluded
+- `Rebuild idle` — No automatic rebuild triggered (expected with exclude-only policy)
+- `Data redundancy: degraded` — Confirms data redundancy is impaired and manual
+  intervention is needed
+
+To restore redundancy, manually trigger rebuild:
+
+```bash
+$ dmg pool rebuild start my_pool
+```
+
+Monitor until `Data redundancy` changes from `degraded` to `normal`:
+
+```bash
+$ dmg pool query my_pool
+Pool 6f450a68-8c7d-4da9-8900-02691650f6a2, ntarget=8, disabled=1, state=Ready
+Pool health info:
+- Disabled ranks: 3
+- Rebuild done, 1042 objs, 2184 recs
+- Data redundancy: normal  ← Redundancy restored
+```
+
+
 ## System / Pool Creation and Disabling / Enabling Self-Heal
 
 The following are example steps for managing `self_heal` policies. This
@@ -208,6 +291,7 @@ $ dmg pool query first_pool
 Pool 1c8f6a4e-a4eb-418d-bedc-0c7751e41af1, ntarget=16, disabled=0, leader=0, version=1, state=Ready
 Pool health info:
 - Rebuild idle, 0 objs, 0 recs
+- Data redundancy: normal
 Pool space info:
 - Target count:16
 - Storage tier 0 (SCM):
@@ -248,6 +332,7 @@ $ dmg pool query first_pool
 Pool 1c8f6a4e-a4eb-418d-bedc-0c7751e41af1, ntarget=16, disabled=0, leader=0, version=1, state=Ready
 Pool health info:
 - Rebuild idle, 0 objs, 0 recs
+- Data redundancy: normal
 Pool space info:
 - Target count:16
 - Storage tier 0 (SCM):
@@ -263,6 +348,7 @@ $ dmg pool query second_pool
 Pool ce3a6a74-8a19-4c6e-a3ef-1ee85d8e06f1, ntarget=16, disabled=0, leader=1, version=1, state=Ready
 Pool health info:
 - Rebuild idle, 0 objs, 0 recs
+- Data redundancy: normal
 Pool space info:
 - Target count:16
 - Storage tier 0 (SCM):
@@ -284,6 +370,7 @@ $ dmg pool query first_pool
 Pool 1c8f6a4e-a4eb-418d-bedc-0c7751e41af1, ntarget=16, disabled=0, leader=0, version=1, state=Ready
 Pool health info:
 - Rebuild idle, 0 objs, 0 recs
+- Data redundancy: normal
 Pool space info:
 - Target count:16
 - Storage tier 0 (SCM):
@@ -513,3 +600,47 @@ $ dmg system stop --ranks=X
 
 * Rank X is excluded from the system and from pools P and Q.
 * Rank X is rebuilding in pool Q only. Pool P will remain in degraded mode.
+
+**3.** Verify exclusion status using `dmg pool query`:
+
+For pool P (rebuild disabled):
+```bash
+$ dmg pool query P
+Pool health info:
+- Disabled ranks: X
+- Rebuild idle
+- Data redundancy: degraded  ← Pool remains degraded (rebuild disabled)
+```
+
+For pool Q (rebuild enabled):
+```bash
+$ dmg pool query Q
+Pool health info:
+- Disabled ranks: X
+- Rebuild busy, 42 objs, 21 recs
+- Data redundancy: degraded  ← Rebuilding in progress
+```
+
+After pool Q rebuild completes:
+```bash
+$ dmg pool query Q
+Pool health info:
+- Disabled ranks: X
+- Rebuild done, 1042 objs, 2184 recs
+- Data redundancy: normal  ← Redundancy restored
+```
+
+**4.** To restore pool P's redundancy later, manually trigger rebuild:
+
+```bash
+$ dmg pool rebuild start P
+```
+
+Monitor until `Data redundancy` changes to `normal`:
+```bash
+$ dmg pool query P
+Pool health info:
+- Disabled ranks: X
+- Rebuild done, 1042 objs, 2184 recs
+- Data redundancy: normal  ← Now restored
+```

From eabee5929eff1a720711c0865b0bd4f4a17bad0e Mon Sep 17 00:00:00 2001
From: Tom Nabarro <thomas.nabarro@hpe.com>
Date: Sat, 18 Apr 2026 16:15:27 +0100
Subject: [PATCH 3/4] Apply suggestions from code review

Co-authored-by: Ken Cain <kenneth.cain@hpe.com>
Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>
---
 docs/admin/self_healing.md | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/docs/admin/self_healing.md b/docs/admin/self_healing.md
index e0fe81d7e97..873a8ffab7c 100644
--- a/docs/admin/self_healing.md
+++ b/docs/admin/self_healing.md
@@ -98,7 +98,7 @@ Pool health info:
 - Data redundancy: normal
 ```
 
-or when targets are excluded:
+or when targets are excluded and a corresponding rebuild has not yet completed:
 
 ```
 Pool health info:
@@ -110,7 +110,7 @@ Pool health info:
 
 | Value | Meaning |
 |-------|---------|
-| `normal` | All targets are UP and accessible; data redundancy is intact |
+| `normal` | No targets are DOWN; data redundancy is intact |
 | `degraded` | One or more targets are DOWN; data redundancy is compromised |
 
 ### When to Check This Field
@@ -141,11 +141,11 @@ Pool health info:
 
 The combination of:
 - `Disabled ranks: 3` — Shows which rank was excluded
-- `Rebuild idle` — No automatic rebuild triggered (expected with exclude-only policy)
+- `Rebuild idle` or `Rebuild done`— idle if pool has not been rebuilt since system start, done if it has been rebuild for a prior change. No new automatic rebuild triggered for this exclusion (expected with exclude-only policy)
 - `Data redundancy: degraded` — Confirms data redundancy is impaired and manual
   intervention is needed
 
-To restore redundancy, manually trigger rebuild:
+To restore redundancy for the excluded targets, manually trigger rebuild. Refer to [Detailed Pool Operations: Interactive Rebuild Controls](rebuild_controls.md) for more information about rebuild start commands.
 
 ```bash
 $ dmg pool rebuild start my_pool

From 12c2f3a6aebdfea2643540eec9bbdb4da8a3f4d6 Mon Sep 17 00:00:00 2001
From: Tom Nabarro <thomas.nabarro@hpe.com>
Date: Wed, 22 Apr 2026 11:45:32 -0400
Subject: [PATCH 4/4] Make revisions from Tom's review.

Doc-only: true

Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>
---
 docs/admin/rebuild_controls.md | 166 ++++++++++++++++++++++++---------
 docs/admin/self_healing.md     |   4 +-
 2 files changed, 124 insertions(+), 46 deletions(-)

diff --git a/docs/admin/rebuild_controls.md b/docs/admin/rebuild_controls.md
index dbd1d808e4a..ce126fcb45f 100644
--- a/docs/admin/rebuild_controls.md
+++ b/docs/admin/rebuild_controls.md
@@ -4,7 +4,7 @@
 
 Although the self-healing policy can control whether a rebuild will occur,
 occasionally an administrator may need to stop a rebuild already started and
-restart it later with a "rebuild start" command (or, perform an alternate
+restart it later with a `rebuild start` command (or, perform an alternate
 action/rebuild such as direct reintegration). For this purpose, DAOS provides
 the following interactive rebuild control command-line interfaces:
 
@@ -13,14 +13,14 @@ the following interactive rebuild control command-line interfaces:
 * `dmg system rebuild stop [--force]`
 * `dmg system rebuild start`
 
-The system commands will apply to all pools in the DAOS system (i.e., have the
-same effect as if multiple `pool rebuild stop` commands are issued, one per
-pool).
+The system-level commands (e.g., `dmg system rebuild stop`) apply to all pools in the DAOS
+system (i.e., they have the same effect as if multiple `dmg pool rebuild stop` commands are issued,
+one per pool).
 
 Upon stopping a pool's rebuild, its rebuild state as reported by `dmg pool query`
-will be an idle state, and an error status=-2027 (`-DER_OP_CANCELED` DAOS error code).
+will be an idle state, and an error status=`-2027` (`-DER_OP_CANCELED` DAOS error code).
 
-The effect of a rebuild stop command is "one shot", meaning only a pool's
+The effect of a `rebuild stop` command is "one shot", meaning only a pool's
 currently-running rebuild is stopped and there is no persistent effect on future
 operations. Subsequent self-healing automatic recovery, or administrator command
 (e.g., system stop, system/pool exclude, reintegrate, drain, pool extend) can
@@ -59,16 +59,16 @@ For a **failed rebuild**, the sequence is:
 
 1. Run `op:Rebuild`
 2. Run `op:Fail_reclaim` to clean up
-    * If `Fail_reclaim` failed, retry `Fail_reclaim`
+    * If `Fail_reclaim` *itself* failed, retry `Fail_reclaim`
     * If `Fail_reclaim` succeeded, retry the original `op:Rebuild`
 
-The "rebuild stop" commands are not typically allowed to terminate a rebuild in
+The `rebuild stop` commands are not typically allowed to terminate a rebuild in
 the `op:Reclaim` and `op:Fail_reclaim` phases — instead the command must be
 issued during the `op:Rebuild` execution. An exception is available with the
-`--force` option to "rebuild stop", intended to be applied for rebuilds that
+`--force` option to `rebuild stop`, intended to be applied for rebuilds that
 repeatedly fail and possibly may even be looping `Fail_reclaim` operations.
 
-Because of these details, carefully timing the execution of "rebuild stop"
+Because of these details, carefully timing the execution of `rebuild stop`
 commands is needed, which can be facilitated with pool rebuild state querying
 with `dmg pool query`. See the section
 [Rebuild Stop Command Errors](#rebuild-stop-command-errors) for examples of
@@ -77,15 +77,18 @@ errors returned by "rebuild stop" in different timing circumstances.
 
 ## Example Usage: Stop a Single Pool Rebuild, then Direct Reintegration
 
-A system has detected the loss of an engine (rank 3) and has launched a
-corresponding rebuild on pool `p1` for that exclusion. The administrator decides
-in this case that it is preferable to stop the exclude rebuild rather than let
-it finish (perhaps it is possible to quickly remedy the rank 3 engine issue,
-restart it, and directly reintegrate it back into the system and pool — rather
-than perform 2 separate rebuilds for the exclusion and the reintegration).
+A system has detected the loss of an engine (rank 3) that has 8 storage targets.
+A corresponding rebuild has launched on pool `p1` after the engine's exclusion
+from the pool map. The administrator decides to stop this rebuild
+(perhaps because it is possible to quickly remedy the rank 3 engine issue, restart it, 
+and reintegrate it into the system and pool). This will result in a single
+rebuild for the reintegration, rather than two rebuilds (for the initial exclusion,
+and later for the reintegration).
 
 **1.** Observe the pool `p1` is rebuilding after the fault is detected.
 
+The pool state reflects 8 disabled targets (`disabled=8`) corresponding to the exclusion
+of engine rank 3. Rebuild is underway (`busy` rebuild state).
 ```bash
 $ dmg pool query --health-only p1
 Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=8, leader=6, version=77, state=TargetsExcluded
@@ -95,18 +98,20 @@ Pool health info:
 - Data redundancy: degraded
 ```
 
-**2.** Stop the rebuild and monitor with pool query until the stop is confirmed
-("stopped", state=idle, status=-2027). Notice some pool query rebuild state
-changes while waiting.
+**2.** Stop the rebuild and monitor with multiple `pool query` commands until the stop is confirmed
+(`stopped, state=idle, status=-2027`). Notice some rebuild state changes while waiting.
 
+Using a command at single-pool scope, no output is expected if the request is successfully
+sent to the storage system.
 ```bash
 $ dmg pool rebuild stop p1
 ```
 
-Run `dmg pool query` in a loop (with short sleeps between commands):
+Run `dmg pool query` in a loop (with short delays between commands).
 
+The rebuild state output `stopping` along with `state=busy` and `status=-2027` is an indication 
+that the stop command is being processed, and rebuild has not been entirely stopped yet.
 ```bash
-# confirmation that stop command is being handled
 $ dmg pool query --health-only p1
 Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=8, leader=6, version=77, state=TargetsExcluded
 Pool health info:
@@ -114,9 +119,10 @@ Pool health info:
 - Rebuild stopping (state=busy, status=-2027)
 ```
 
+The rebuild state has temporarily transitioned to `busy`, reflecting that the `op:Rebuild` is no
+longer running, but a reclaim phase is now running (in this case, `op:Fail_reclaim`, since a stopped
+rebuild is processed in the same way as a failed rebuild).
 ```bash
-# state temporarily changes back to busy, representing the Fail_reclaim stage
-# for the stopped operation is now running
 $ dmg pool query --health-only p1
 Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=8, leader=6, version=77, state=TargetsExcluded
 Pool health info:
@@ -124,8 +130,10 @@ Pool health info:
 - Rebuild busy, 0 objs, 0 recs
 ```
 
+The rebuild `op:Fail_reclaim` has finished, and now the pool query output shows that the
+rebuild is stopped. This is the final state reflecting that the rebuild is stopped
+(`state=idle, status=-2027`).
 ```bash
-# Fail_reclaim has finished, and now the pool shows its rebuild has fully stopped
 $ dmg pool query --health-only p1
 Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=8, leader=6, version=77, state=TargetsExcluded
 Pool health info:
@@ -134,16 +142,33 @@ Pool health info:
 - Data redundancy: degraded
 ```
 
-**3.** Directly reintegrate the engine back into pool `p1` and monitor progress
-with multiple pool query commands until the reintegration rebuild is
-successfully finished.
+**3.** Restart engine rank 3, wait for it to join the system, directly reintegrate it
+back into pool `p1`, and wait for the rebuild to finish successfully.
 
 ```bash
+$ dmg system start --ranks=3
+# Repeat dmg system query commands until engine rank 3 shown in the joined state
+$ dmg system query
+Rank      State
+----      -----
+[0-2,4-7] Joined
+3         Stopped
+
+$ dmg system query
+Rank  State
+----  -----
+[0-7] Joined
+
+# Now, reintegrate engine rank 3 into pool p1
 $ dmg pool reintegrate --ranks=3 p1
 ```
 
-Run `dmg pool query` in a loop (with short sleeps between commands):
+Run `dmg pool query` in a loop (with short delays between commands).
 
+First, it may be seen that the pool map has been updated for the reintegrating engine rank 3
+(pool map `version=85` instead of 77, and targets `disabled=0` instead of 8). And it could be
+that the pool rebuild has not yet started, with state reflecting the same state following
+the previous (stopped) rebuild (`state=idle, status=-2027`).
 ```bash
 $ dmg pool query --health-only p1
 Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=0, leader=6, version=85, state=Ready
@@ -152,6 +177,7 @@ Pool health info:
 - Data redundancy: normal
 ```
 
+Rebuild is now `busy` (performing the reintegration) according to rebuild state.
 ```bash
 $ dmg pool query --health-only p1
 Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=0, leader=6, version=85, state=Ready
@@ -160,6 +186,7 @@ Pool health info:
 - Data redundancy: normal
 ```
 
+Rebuild is still `busy`, showing increasing object / record counts.
 ```bash
 $ dmg pool query --health-only p1
 Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=0, leader=6, version=85, state=Ready
@@ -168,8 +195,12 @@ Pool health info:
 - Data redundancy: normal
 ```
 
+Rebuild is still `busy`, though object / record counts have been reset. Also, the pool map version
+has increased to 93 (previously 85). This indicates the `op:Rebuild` has completed, and
+the rebuild is cleaning up in `op:Reclaim` phase. The pool map was updated to promote the
+engine rank 3 targets from `UP` (during reintegration) to `UP_IN` (reintegration complete,
+ready for client I/O).
 ```bash
-# Notice zero objs, recs corresponds to a transition from Rebuild done to Reclaim phase now running.
 $ dmg pool query --health-only p1
 Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=0, leader=6, version=93, state=Ready
 Pool health info:
@@ -177,6 +208,7 @@ Pool health info:
 - Data redundancy: normal
 ```
 
+Rebuild is still `busy` in `op:Reclaim` phase.
 ```bash
 $ dmg pool query --health-only p1
 Pool cdf27ec1-ed97-4aa6-a766-39a2ed2136a1, ntarget=64, disabled=0, leader=6, version=93, state=Ready
@@ -185,6 +217,8 @@ Pool health info:
 - Data redundancy: normal
 ```
 
+Rebuild (including reclaim) has finished since state is `done`. Engine rank 3 has been reintegrated
+into the pool.
 ```bash
 # all done
 $ dmg pool query --health-only p1
@@ -214,9 +248,25 @@ Rank      State
 3         Stopped
 ```
 
-This shows that pool `p2` has started its rebuild (state "busy"). `p1` is
-"done" from a prior rebuild, and will start rebuilding soon.
+Attempting to stop pool rebuilds too soon (before they have actually started) will produce an error.
+The `dmg system rebuild stop` command reports how many pools had the request successfully
+issued (in this case, 0 successful pools).
+```bash
+$ dmg system rebuild stop
+System-rebuild stop request succeeded on 0 pools
+ERROR: dmg: system rebuild stop failed: pool-rebuild stop failed on pool p1: pool-rebuild stop failed: DER_NONEXIST(-1005): The specified entity does not exist, pool-rebuild stop failed on pool p2: pool-rebuild stop failed: DER_NONEXIST(-1005): The specified entity does not exist
+```
+
+The following output shows that pool `p2` has excluded targets and rebuild has started:
+- `State` column shows `TargetsExcluded`
+- `Disabled` column reports 8 out of 64 targets are disabled, corresponding here to the lost
+engine rank 3 targets.
+- `Rebuild State` column shows `busy`
 
+Also, pool `p1` has not excluded targets yet, and has not started rebuild:
+- `State` column is `Ready`
+- `Disabled` column reflects 0 targets disabled.
+- `Rebuild State` column is `done`, reflecting state from a previously-completed rebuild.
 ```bash
 $ dmg system list-pools -v
 Label UUID                                 State           SvcReps   SCM Size SCM Used SCM Imbalance NVME Size NVME Used NVME Imbalance Disabled UpgradeNeeded? Rebuild State
@@ -225,11 +275,22 @@ p1    cdf27ec1-ed97-4aa6-a766-39a2ed2136a1 Ready           [0-1,4-6] 105 GB   37
 p2    dcfe63e0-e10f-464d-b18c-0915e52e048c TargetsExcluded [0-2,6-7] 49 GB    4.6 GB   62%           0 B       0 B       0%             8/64     None           busy
 ```
 
+Shortly after this moment, both pools p1 and p2 should show `Rebuild State` =  `busy`.
+
 **2.** Stop all pool rebuilds with one command, and confirm they have stopped.
 
+The successful `system rebuild stop` command will confirm that it succeeded for all (in this case 2)
+pools.
 ```bash
 $ dmg system rebuild stop
+System-rebuild stop request succeeded on 2 pools
+```
 
+When the rebuilds have been stopped, the `Rebuild State` is expected to be `idle` (seen in both
+`dmg system list-pools` and `dmg pool query` command output). And the rebuild status will show
+error status `-2027` (`-DER_OP_CANCELED` DAOS error code)
+
+```bash
 $ dmg system list-pools -v
 Label UUID                                 State           SvcReps   SCM Size SCM Used SCM Imbalance NVME Size NVME Used NVME Imbalance Disabled UpgradeNeeded? Rebuild State
 ----- ----                                 -----           -------   -------- -------- ------------- --------- --------- -------------- -------- -------------- -------------
@@ -256,9 +317,30 @@ until finished.
 
 ```bash
 $ dmg system start --ranks=3
+# Repeat dmg system query commands until engine rank 3 shown in the joined state
+$ dmg system query
+Rank      State
+----      -----
+[0-2,4-7] Joined
+3         Stopped
+
+$ dmg system query
+Rank  State
+----  -----
+[0-7] Joined
 
 $ dmg system reintegrate --ranks=3
+```
+
+Use `dmg system list-pools` or `dmg pool query` = (with short delays between commands) to
+monitor the reintegration rebuilds on the pools.
 
+Both pools have had their target states updated due to the reintegration command, and corresponding
+reintegration rebuilds started, based on the `dmg system list-pools` command output:
+- `Disabled` column reports 0 disabled targets (was 8 targets disabled previously)
+- `Rebuild State` column indicates `busy` for both pools.
+
+```bash
 $ dmg system list-pools -v
 Label UUID                                 State SvcReps   SCM Size SCM Used SCM Imbalance NVME Size NVME Used NVME Imbalance Disabled UpgradeNeeded? Rebuild State
 ----- ----                                 ----- -------   -------- -------- ------------- --------- --------- -------------- -------- -------------- -------------
@@ -266,8 +348,7 @@ p1    cdf27ec1-ed97-4aa6-a766-39a2ed2136a1 Ready [0-1,4-6] 120 GB   42 GB    28%
 p2    dcfe63e0-e10f-464d-b18c-0915e52e048c Ready [0-2,6-7] 56 GB    4.9 GB   62%           0 B       0 B       0%             0/64     None           busy
 ```
 
-Wait for rebuilds to complete:
-
+Wait for rebuilds to complete (i.e., for `Rebuild State` to report `done` for both pools):
 ```bash
 $ dmg system list-pools -v
 Label UUID                                 State SvcReps   SCM Size SCM Used SCM Imbalance NVME Size NVME Used NVME Imbalance Disabled UpgradeNeeded? Rebuild State
@@ -279,31 +360,29 @@ p2    dcfe63e0-e10f-464d-b18c-0915e52e048c Ready [0-2,6-7] 56 GB    4.9 GB   62%
 
 ## Rebuild Stop Command Errors
 
-The rebuild stop command may return errors when it is issued at a time that it
-may not be able to handle the request. For example:
+The `rebuild stop` command may return errors when it is issued at a time that it
+may not be able to handle the request. The following subsections show examples.
 
 ### No Rebuild Currently Running
 
 When no rebuild is currently running, the command will report a "nonexist"
 error:
-
 ```bash
-# system command applying to 2 pools
+# system-level command applying to two pools, both of which have not started rebuilding yet
 $ dmg system rebuild stop
 System-rebuild stop request succeeded on 0 pools
 ERROR: dmg: system rebuild stop failed: pool-rebuild stop failed on pool p1: pool-rebuild stop failed: DER_NONEXIST(-1005): The specified entity does not exist, pool-rebuild stop failed on pool p2: pool-rebuild stop failed: DER_NONEXIST(-1005): The specified entity does not exist
 
-# single pool command
+# single pool scope command applied to one pool that is not currently rebuilding
 $ dmg pool rebuild stop p2
 ERROR: dmg: pool-rebuild stop failed: DER_NONEXIST(-1005): The specified entity does not exist
 ```
 
-### Rebuild Finished, Reclaim in Progress
+### Rebuild Finished Successfully, Reclaim in Progress
 
-When the rebuild stage has successfully finished and is in its reclaim cleanup
-stage, `dmg` will report a busy error. For example when pools `p1` and `p2` are
+When the rebuild stage has successfully finished and is in its `op:Reclaim` cleanup
+stage, `dmg` will report a (generic) busy error. For example when pools `p1` and `p2` are
 both done rebuilding and in the reclaim stage:
-
 ```bash
 $ dmg system rebuild stop
 System-rebuild stop request succeeded on 0 pools
@@ -316,8 +395,7 @@ When the rebuild stage has finished (but failed), and is in its `Fail_reclaim`
 cleanup stage, `dmg` will report a no permissions error, `-DER_NO_PERM`.
 
 In this scenario, the admin can wait for the rebuild to be retried, and then
-reissue the "rebuild stop" command:
-
+reissue the `rebuild stop` command:
 ```bash
 # multiple command invocations to query pool rebuild status
 $ dmg pool query <pool_label>
diff --git a/docs/admin/self_healing.md b/docs/admin/self_healing.md
index 873a8ffab7c..487f431a02e 100644
--- a/docs/admin/self_healing.md
+++ b/docs/admin/self_healing.md
@@ -33,8 +33,8 @@ administrator using `dmg pool get-prop` / `dmg pool set-prop`.
 
 The `self_heal` property describes behavior associated with self-healing and can
 be set at both the system and pool levels. For a given pool to perform the
-recovery actions (map change, rebuild), the policy needs to be enabled at both
-the system and per-pool level.
+recovery actions (map change due to excluded engine/targets, and rebuild), the policy
+needs to be enabled at both the system and per-pool level.
 
 The system level property has flags `exclude;pool_exclude;pool_rebuild` which
 indicate the following: