Skip to content

Commit 659ef74

Browse files
ppekrolppekrol
authored
Merge pull request #2459 from reebhub/RDoc-3818
RDoc-3818 - Document responsible-node and grace-period behavior for backup tasks
2 parents ff4e3f9 + 2f5d134 commit 659ef74

12 files changed

Lines changed: 372 additions & 1 deletion

File tree

‎docs/backup/configuration.mdx‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
---
22
title: "Backup configuration options"
33
sidebar_label: "Configuration options"
4+
description: "Server-wide configuration keys that control backup behavior: paths, allowed destinations, concurrency, low-memory delays, and the responsible-node grace period."
45
sidebar_position: 85
56
---
67

@@ -29,6 +30,7 @@ Learn how to apply these keys in the [Configuration Overview](../server/configur
2930
* [Backup.MaxNumberOfConcurrentBackups](../backup/configuration#backupmaxnumberofconcurrentbackups)
3031
* [Backup.ConcurrentBackupsDelayInSec](../backup/configuration#backupconcurrentbackupsdelayinsec)
3132
* [Backup.LowMemoryBackupDelayInMin](../backup/configuration#backuplowmemorybackupdelayinmin)
33+
* [Backup.MoveToNewResponsibleNodeGracePeriodInMin](../backup/configuration#backupmovetonewresponsiblenodegraceperiodinmin)
3234
* [Server.CpuCredits.ExhaustionBackupDelayInMin](../backup/configuration#servercpucreditsexhaustionbackupdelayinmin)
3335

3436
</Admonition>
@@ -158,6 +160,26 @@ Number of minutes to delay the backup if the server enters a low-memory state.
158160

159161
<ContentFrame>
160162

163+
### Backup.MoveToNewResponsibleNodeGracePeriodInMin
164+
165+
Number of minutes the [cluster observer](../server/clustering/distribution/cluster-observer.mdx) waits
166+
before reassigning a backup task to a new responsible node,
167+
when the current responsible node has entered the
168+
[Rehab state](../server/clustering/distribution/distributed-database.mdx#database-topology).
169+
170+
During this grace period, the task remains assigned to its current responsible node.
171+
This gives the node a chance to recover, and prevents a duplicate backup
172+
from starting elsewhere in the cluster.
173+
174+
- **Type**: `TimeSetting`
175+
- **TimeUnit**: `TimeUnit.Minutes`
176+
- **Default**: `30`
177+
- **Scope**: Server-wide only
178+
179+
</ContentFrame>
180+
181+
<ContentFrame>
182+
161183
### Server.CpuCredits.ExhaustionBackupDelayInMin
162184

163185
EXPERT: When CPU credits are exhausted, backup tasks are cancelled.

‎docs/backup/faq.mdx‎

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
---
22
title: "FAQ"
33
sidebar_label: "FAQ"
4+
description: "Frequently asked questions about backup and restore in RavenDB: one-time backups, server-wide backups, retention, failure handling, and behavior in a cluster."
45
sidebar_position: 90
56
---
67

@@ -152,7 +153,39 @@ For example, you can set a backup task to keep backups locally as well as send t
152153

153154
### What happens if the node responsible for a backup task is down?
154155

155-
When the responsible node is down during the scheduled time, another node from the database group will assume ownership of the task, so there will be no gaps in the backup schedule.
156+
When the responsible node enters the
157+
[Rehab state](../server/clustering/distribution/distributed-database.mdx#database-topology),
158+
the [cluster observer](../server/clustering/distribution/cluster-observer.mdx)
159+
does not move the backup task immediately.
160+
Instead, it waits a grace period (30 minutes by default) before reassigning the task
161+
to another database-group member.
162+
The grace period gives the original node a chance to recover,
163+
and prevents the same backup from running on two nodes at once.
164+
165+
* **Who picks the responsible node:**
166+
When you create a backup task, you can pick the responsible node yourself,
167+
or leave it for the cluster to decide.
168+
The choice (yours or the cluster's) is persisted cluster-wide,
169+
so all nodes agree on a single responsible node for the task.
170+
171+
* **If the node is in Rehab due to resource constraints on the host machine:**
172+
The cluster observer will leave the backup task on the same node rather than reassign it.
173+
The backup will run on that node once it recovers,
174+
or be reassigned to another node if this one is eventually removed from the database group.
175+
176+
The grace period is configurable via
177+
[Backup.MoveToNewResponsibleNodeGracePeriodInMin](../backup/configuration#backupmovetonewresponsiblenodegraceperiodinmin).
178+
179+
<Admonition type="note" title="">
180+
181+
A backup task that is **pinned** to its mentor node (via `PinToMentorNode`)
182+
is an exception to the behavior described above.
183+
A pinned task is not subject to the grace period.
184+
It stays on its node even while the node is in Rehab,
185+
and only moves to another node when its pinned node is removed from the database group.
186+
See [Pinning a Task](../server/clustering/distribution/highly-available-tasks.mdx#pinning-a-task) for the full behavior.
187+
188+
</Admonition>
156189

157190
</ContentFrame>
158191

‎docs/server/clustering/distribution/highly-available-tasks.mdx‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,37 @@ Learn more [here](../../../server/clustering/distribution/distributed-database.m
8484
about database nodes' relations and states.
8585
</Admonition>
8686

87+
<Admonition type="note" title="">
88+
89+
**Backup tasks are an exception** to the local-decision model described above.
90+
91+
For backup tasks, the responsible node is either picked when the task is created
92+
(via the **Set responsible node** option in Studio, or `MentorNode` via the API),
93+
or chosen by the [cluster observer](../../../server/clustering/distribution/cluster-observer.mdx)
94+
when no explicit choice was made.
95+
Either way, the choice is persisted cluster-wide,
96+
so all nodes agree on a single responsible node.
97+
98+
When the current responsible node enters the `Rehab` state,
99+
the cluster observer waits a configurable grace period (30 minutes by default)
100+
before moving the task to another node.
101+
This prevents the same backup from running on two nodes at once.
102+
The grace period is set by
103+
[Backup.MoveToNewResponsibleNodeGracePeriodInMin](../../../backup/configuration.mdx#backupmovetonewresponsiblenodegraceperiodinmin).
104+
105+
</Admonition>
106+
107+
<Admonition type="note" title="">
108+
109+
A backup task that is **pinned** to its mentor node (via `PinToMentorNode`)
110+
is an exception to all of the above.
111+
A pinned task is not subject to the grace period.
112+
It stays on its node even while the node is in Rehab,
113+
and only moves when its pinned node is removed from the database group.
114+
See [Pinning a Task](#pinning-a-task) below for the full behavior.
115+
116+
</Admonition>
117+
87118

88119

89120
## Tasks Relocation
@@ -132,6 +163,12 @@ The failover of a task to another responsible node can be prevented by **pinning
132163
[cluster.timebeforeaddingreplicainsec](../../../server/configuration/cluster-configuration.mdx#clustertimebeforeaddingreplicainsec),
133164
the cluster observer will attempt to select an available node to replace it in the database group
134165
and redistribute the fallen node's tasks, including pinned ones, among database group members.
166+
* A pinned **backup task** is also not subject to the
167+
[grace period](../../../backup/configuration.mdx#backupmovetonewresponsiblenodegraceperiodinmin)
168+
that the cluster observer otherwise applies before reassigning a backup task.
169+
The pinned backup stays on its node even while the node is in Rehab,
170+
and only moves when the cluster eventually replaces the node as described above.
171+
135172
A task can be pinned to a selected node via Studio or using code.
136173

137174
#### Pinning via Studio

‎versioned_docs/version-6.2/client-api/operations/maintenance/backup/faq.mdx‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ import LanguageContent from "@site/src/components/LanguageContent";
2323
* [Does RavenDB automatically delete old backups?](../../../../client-api/operations/maintenance/backup/faq.mdx#does-ravendb-automatically-delete-old-backups)
2424
* [Are there any locations that backup files should NOT be stored at?](../../../../client-api/operations/maintenance/backup/faq.mdx#are-there-any-locations-that-backup-files-should-not-be-stored-at)
2525
* [What happens when a backup process fails before it is completed?](../../../../client-api/operations/maintenance/backup/faq.mdx#what-happens-when-a-backup-process-fails-before-completion)
26+
* [What happens if the node responsible for a backup task is down?](../../../../client-api/operations/maintenance/backup/faq.mdx#what-happens-if-the-node-responsible-for-a-backup-task-is-down)
2627

2728
</Admonition>
2829
## FAQ
@@ -108,5 +109,41 @@ While in progress, the backup content is written to an **.in-progress* file on d
108109
If the failed process was an incremental-backup task, any future incremental backups will
109110
continue from the correct place before the file was created so that the backup is consistent with the source.
110111

112+
### What happens if the node responsible for a backup task is down?
113+
114+
When the responsible node enters the
115+
[Rehab state](../../../../server/clustering/distribution/distributed-database.mdx#database-topology),
116+
the [cluster observer](../../../../server/clustering/distribution/cluster-observer.mdx)
117+
does not move the backup task immediately.
118+
Instead, it waits a grace period (30 minutes by default) before reassigning the task
119+
to another database-group member.
120+
The grace period gives the original node a chance to recover,
121+
and prevents the same backup from running on two nodes at once.
122+
123+
* **Who picks the responsible node:**
124+
When you create a backup task, you can pick the responsible node yourself,
125+
or leave it for the cluster to decide.
126+
The choice (yours or the cluster's) is persisted cluster-wide,
127+
so all nodes agree on a single responsible node for the task.
128+
129+
* **If the node is in Rehab due to resource constraints on the host machine:**
130+
The cluster observer will leave the backup task on the same node rather than reassign it.
131+
The backup will run on that node once it recovers,
132+
or be reassigned to another node if this one is eventually removed from the database group.
133+
134+
The grace period is configurable via
135+
[Backup.MoveToNewResponsibleNodeGracePeriodInMin](../../../../server/configuration/backup-configuration.mdx#backupmovetonewresponsiblenodegraceperiodinmin).
136+
137+
<Admonition type="note" title="">
138+
139+
A backup task that is **pinned** to its mentor node (via `PinToMentorNode`)
140+
is an exception to the behavior described above.
141+
A pinned task is not subject to the grace period.
142+
It stays on its node even while the node is in Rehab,
143+
and only moves to another node when its pinned node is removed from the database group.
144+
See [Pinning a Task](../../../../server/clustering/distribution/highly-available-tasks.mdx#pinning-a-task) for the full behavior.
145+
146+
</Admonition>
147+
111148

112149

‎versioned_docs/version-6.2/server/clustering/distribution/highly-available-tasks.mdx‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,37 @@ Learn more [here](../../../server/clustering/distribution/distributed-database.m
8383
about database nodes' relations and states.
8484
</Admonition>
8585

86+
<Admonition type="note" title="">
87+
88+
**Backup tasks are an exception** to the local-decision model described above.
89+
90+
For backup tasks, the responsible node is either picked when the task is created
91+
(via the **Set responsible node** option in Studio, or `MentorNode` via the API),
92+
or chosen by the [cluster observer](../../../server/clustering/distribution/cluster-observer.mdx)
93+
when no explicit choice was made.
94+
Either way, the choice is persisted cluster-wide,
95+
so all nodes agree on a single responsible node.
96+
97+
When the current responsible node enters the `Rehab` state,
98+
the cluster observer waits a configurable grace period (30 minutes by default)
99+
before moving the task to another node.
100+
This prevents the same backup from running on two nodes at once.
101+
The grace period is set by
102+
[Backup.MoveToNewResponsibleNodeGracePeriodInMin](../../../server/configuration/backup-configuration.mdx#backupmovetonewresponsiblenodegraceperiodinmin).
103+
104+
</Admonition>
105+
106+
<Admonition type="note" title="">
107+
108+
A backup task that is **pinned** to its mentor node (via `PinToMentorNode`)
109+
is an exception to all of the above.
110+
A pinned task is not subject to the grace period.
111+
It stays on its node even while the node is in Rehab,
112+
and only moves when its pinned node is removed from the database group.
113+
See [Pinning a Task](#pinning-a-task) below for the full behavior.
114+
115+
</Admonition>
116+
86117

87118

88119
## Tasks Relocation
@@ -131,6 +162,12 @@ The failover of a task to another responsible node can be prevented by **pinning
131162
[cluster.timebeforeaddingreplicainsec](../../../server/configuration/cluster-configuration.mdx#clustertimebeforeaddingreplicainsec),
132163
the cluster observer will attempt to select an available node to replace it in the database group
133164
and redistribute the fallen node's tasks, including pinned ones, among database group members.
165+
* A pinned **backup task** is also not subject to the
166+
[grace period](../../../server/configuration/backup-configuration.mdx#backupmovetonewresponsiblenodegraceperiodinmin)
167+
that the cluster observer otherwise applies before reassigning a backup task.
168+
The pinned backup stays on its node even while the node is in Rehab,
169+
and only moves when the cluster eventually replaces the node as described above.
170+
134171
A task can be pinned to a selected node via Studio or using code.
135172

136173
#### Pinning via Studio

‎versioned_docs/version-6.2/server/configuration/backup-configuration.mdx‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ import LanguageContent from "@site/src/components/LanguageContent";
2626
* [Backup.MaxNumberOfConcurrentBackups](../../server/configuration/backup-configuration.mdx#backupmaxnumberofconcurrentbackups)
2727
* [Backup.ConcurrentBackupsDelayInSec](../../server/configuration/backup-configuration.mdx#backupconcurrentbackupsdelayinsec)
2828
* [Backup.LowMemoryBackupDelayInMin](../../server/configuration/backup-configuration.mdx#backuplowmemorybackupdelayinmin)
29+
* [Backup.MoveToNewResponsibleNodeGracePeriodInMin](../../server/configuration/backup-configuration.mdx#backupmovetonewresponsiblenodegraceperiodinmin)
2930

3031
</Admonition>
3132
## Backup.TempPath
@@ -112,3 +113,21 @@ Number of minutes to delay the backup if the server enters a low memory state.
112113

113114

114115

116+
## Backup.MoveToNewResponsibleNodeGracePeriodInMin
117+
118+
Number of minutes the [cluster observer](../../server/clustering/distribution/cluster-observer.mdx) waits
119+
before reassigning a backup task to a new responsible node,
120+
when the current responsible node has entered the
121+
[Rehab state](../../server/clustering/distribution/distributed-database.mdx#database-topology).
122+
123+
During this grace period, the task remains assigned to its current responsible node.
124+
This gives the node a chance to recover, and prevents a duplicate backup
125+
from starting elsewhere in the cluster.
126+
127+
- **Type**: `TimeSetting`
128+
- **TimeUnit**: `TimeUnit.Minutes`
129+
- **Default**: `30`
130+
- **Scope**: Server-wide only
131+
132+
133+

‎versioned_docs/version-7.0/client-api/operations/maintenance/backup/faq.mdx‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ import LanguageContent from "@site/src/components/LanguageContent";
2323
* [Does RavenDB automatically delete old backups?](../../../../client-api/operations/maintenance/backup/faq.mdx#does-ravendb-automatically-delete-old-backups)
2424
* [Are there any locations that backup files should NOT be stored at?](../../../../client-api/operations/maintenance/backup/faq.mdx#are-there-any-locations-that-backup-files-should-not-be-stored-at)
2525
* [What happens when a backup process fails before it is completed?](../../../../client-api/operations/maintenance/backup/faq.mdx#what-happens-when-a-backup-process-fails-before-completion)
26+
* [What happens if the node responsible for a backup task is down?](../../../../client-api/operations/maintenance/backup/faq.mdx#what-happens-if-the-node-responsible-for-a-backup-task-is-down)
2627

2728
</Admonition>
2829
## FAQ
@@ -108,5 +109,41 @@ While in progress, the backup content is written to an **.in-progress* file on d
108109
If the failed process was an incremental-backup task, any future incremental backups will
109110
continue from the correct place before the file was created so that the backup is consistent with the source.
110111

112+
### What happens if the node responsible for a backup task is down?
113+
114+
When the responsible node enters the
115+
[Rehab state](../../../../server/clustering/distribution/distributed-database.mdx#database-topology),
116+
the [cluster observer](../../../../server/clustering/distribution/cluster-observer.mdx)
117+
does not move the backup task immediately.
118+
Instead, it waits a grace period (30 minutes by default) before reassigning the task
119+
to another database-group member.
120+
The grace period gives the original node a chance to recover,
121+
and prevents the same backup from running on two nodes at once.
122+
123+
* **Who picks the responsible node:**
124+
When you create a backup task, you can pick the responsible node yourself,
125+
or leave it for the cluster to decide.
126+
The choice (yours or the cluster's) is persisted cluster-wide,
127+
so all nodes agree on a single responsible node for the task.
128+
129+
* **If the node is in Rehab due to resource constraints on the host machine:**
130+
The cluster observer will leave the backup task on the same node rather than reassign it.
131+
The backup will run on that node once it recovers,
132+
or be reassigned to another node if this one is eventually removed from the database group.
133+
134+
The grace period is configurable via
135+
[Backup.MoveToNewResponsibleNodeGracePeriodInMin](../../../../server/configuration/backup-configuration.mdx#backupmovetonewresponsiblenodegraceperiodinmin).
136+
137+
<Admonition type="note" title="">
138+
139+
A backup task that is **pinned** to its mentor node (via `PinToMentorNode`)
140+
is an exception to the behavior described above.
141+
A pinned task is not subject to the grace period.
142+
It stays on its node even while the node is in Rehab,
143+
and only moves to another node when its pinned node is removed from the database group.
144+
See [Pinning a Task](../../../../server/clustering/distribution/highly-available-tasks.mdx#pinning-a-task) for the full behavior.
145+
146+
</Admonition>
147+
111148

112149

‎versioned_docs/version-7.0/server/clustering/distribution/highly-available-tasks.mdx‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,37 @@ Learn more [here](../../../server/clustering/distribution/distributed-database.m
8383
about database nodes' relations and states.
8484
</Admonition>
8585

86+
<Admonition type="note" title="">
87+
88+
**Backup tasks are an exception** to the local-decision model described above.
89+
90+
For backup tasks, the responsible node is either picked when the task is created
91+
(via the **Set responsible node** option in Studio, or `MentorNode` via the API),
92+
or chosen by the [cluster observer](../../../server/clustering/distribution/cluster-observer.mdx)
93+
when no explicit choice was made.
94+
Either way, the choice is persisted cluster-wide,
95+
so all nodes agree on a single responsible node.
96+
97+
When the current responsible node enters the `Rehab` state,
98+
the cluster observer waits a configurable grace period (30 minutes by default)
99+
before moving the task to another node.
100+
This prevents the same backup from running on two nodes at once.
101+
The grace period is set by
102+
[Backup.MoveToNewResponsibleNodeGracePeriodInMin](../../../server/configuration/backup-configuration.mdx#backupmovetonewresponsiblenodegraceperiodinmin).
103+
104+
</Admonition>
105+
106+
<Admonition type="note" title="">
107+
108+
A backup task that is **pinned** to its mentor node (via `PinToMentorNode`)
109+
is an exception to all of the above.
110+
A pinned task is not subject to the grace period.
111+
It stays on its node even while the node is in Rehab,
112+
and only moves when its pinned node is removed from the database group.
113+
See [Pinning a Task](#pinning-a-task) below for the full behavior.
114+
115+
</Admonition>
116+
86117

87118

88119
## Tasks Relocation
@@ -131,6 +162,12 @@ The failover of a task to another responsible node can be prevented by **pinning
131162
[cluster.timebeforeaddingreplicainsec](../../../server/configuration/cluster-configuration.mdx#clustertimebeforeaddingreplicainsec),
132163
the cluster observer will attempt to select an available node to replace it in the database group
133164
and redistribute the fallen node's tasks, including pinned ones, among database group members.
165+
* A pinned **backup task** is also not subject to the
166+
[grace period](../../../server/configuration/backup-configuration.mdx#backupmovetonewresponsiblenodegraceperiodinmin)
167+
that the cluster observer otherwise applies before reassigning a backup task.
168+
The pinned backup stays on its node even while the node is in Rehab,
169+
and only moves when the cluster eventually replaces the node as described above.
170+
134171
A task can be pinned to a selected node via Studio or using code.
135172

136173
#### Pinning via Studio

0 commit comments

Comments
 (0)