Inconsistent checkpoint/status and syncing behavior after CappedPositionLost (MongoShake v2.8.7)

[syKim12]

Hello,

We would like to request an issue regarding an abnormal checkpoint/oplog reader behavior observed in MongoShake.

Environment

• MongoShake version: v2.8.7
• Sync mode: sync_mode = all
• Source MongoDB version: 4.2.18
• Target MongoDB version: 7.0.28
• Incremental sync fetch method: oplog
• Source connect mode: secondaryPreferred
• Checkpoint storage: mongoshake.ckpt_default
• Checkpoint interval: 5000
• Typical oplog window on source: around 4.5 hours

Observed Behavior

• MongoShake checkpoint and /repl status values appeared to stop advancing after 2026-03-25.
• In particular, lsn_ckpt and lsn_ack remained at old values.
• However, from the service/application perspective, actual syncing continued to work normally until 2026-04-12.
• In other words, checkpoint/status values appeared stale, while actual data synchronization seemed to continue normally.
• Later, oplog collection capped error started repeating.
• The process itself stayed alive, and ensure network logs kept appearing, but lsn_ack, lsn_ckpt, write_success, and tps no longer changed.

Key Timeline

1. 2026-03-25 20:44:13 JST
First occurrence of:

oplog collection capped may happen: (CappedPositionLost) CollectionScan died due to position in capped collection being deleted.

2. After that, checkpoints still advanced for a short period:

• 20:44:18 → checkpoint success [1774423477]
• 20:44:23 → checkpoint success [1774423480]
• 20:44:28 → checkpoint success [1774423484]
• …
• 20:45:06 → last checkpoint success [1774423506]

3. After 2026-03-25 20:44:52 JST
The following error started repeating roughly every 6 seconds:

Syncer[mongod_replset_1] oplog collection capped error, users should fix it manually

4. 2026-03-25 20:57:55 JST
The following logs kept repeating, but the status values stayed unchanged:

oplogReader[...] ensure network
[name=mongod_replset_1, stage=incr, get=69383925, filter=34741929, write_success=34641955, tps=0,
 lsn_ckpt={..., 2026-03-25 16:25:06}, lsn_ack={..., 2026-03-25 16:25:24}]

5. However, based on our operational observation, actual syncing still continued normally until 2026-04-12.

Important Notes

• The time fields in /repl appear to be in UTC. Based on the unix timestamps:
  • lsn_ckpt = 1774423506 → 2026-03-25 20:25:06 JST
  • lsn_ack = 1774423524 → 2026-03-25 20:25:24 JST
• Therefore, the gap between the last acknowledged oplog position and the first CappedPositionLost event is only about 19 minutes.
• Also, there was no replication lag at 2026-03-25 20:44 JST.
• Therefore, this does not match a simple explanation that MongoShake exceeded the oplog window because of replication lag.

Additional Observation

• There were duplicate key (E11000) logs, but they were concentrated mostly around 2026-03-25 11:29 ~ 11:41 JST.
• Checkpoints continued to advance normally until 20:45 JST, so duplicate key handling does not appear to be the direct cause of this capped error incident.

Our Current Interpretation

Our current understanding is as follows:

1. The oplog reader cursor was lost when CappedPositionLost occurred.
2. However, MongoShake still had some oplog data already buffered in internal workers/buffers, so the checkpoint continued to move slightly forward until 20:45:06 JST.
3. After that point, it could no longer read new oplog entries, and only the oplog collection capped error kept repeating.
4. The process itself remained alive and ensure network kept running, but the actual reader/status remained stale.
5. Nevertheless, actual data syncing continued to work normally until 2026-04-12.
6. As a result, this looks like a case where checkpoint / /repl status became inconsistent with the actual syncing behavior.

We can provide/repl and /conf outputs if needed.

Thank you.

[syKim12]

I want to add one more observation.

Even on 2026-04-13, MongoShake still reports stage=incr, but all replication/checkpoint-related values remain unchanged from 2026-03-25.

Command:

grep -nEi "full.*finish|full sync.*finish|start incr|incr sync|stage=incr|create checkpoint manager|Record new checkpoint" collector.log* | head -10
curl -s 127.0.0.1:9100/repl

Relevant output:

collector.log:1:[2026/04/13 00:00:00 JST] [INFO] [name=mongod_replset_1, stage=incr, get=69383925, filter=34741929, write_success=34641955, tps=0, ckpt_times=6347, lsn_ckpt={7621090927523661999[1774423506, 2223], 2026-03-25 16:25:06}, lsn_ack={7621091004833072010[1774423524, 906], 2026-03-25 16:25:24}]]

collector.log:4:[2026/04/13 00:00:05 JST] [INFO] [name=mongod_replset_1, stage=incr, get=69383925, filter=34741929, write_success=34641955, tps=0, ckpt_times=6347, lsn_ckpt={7621090927523661999[1774423506, 2223], 2026-03-25 16:25:06}, lsn_ack={7621091004833072010[1774423524, 906], 2026-03-25 16:25:24}]]

collector.log:7:[2026/04/13 00:00:10 JST] [INFO] [name=mongod_replset_1, stage=incr, get=69383925, filter=34741929, write_success=34641955, tps=0, ckpt_times=6347, lsn_ckpt={7621090927523661999[1774423506, 2223], 2026-03-25 16:25:06}, lsn_ack={7621091004833072010[1774423524, 906], 2026-03-25 16:25:24}]]

{"who":"mongoshake","tag":"develop,a484d2166bff2a37f798c31175823a56bddf395c,release,go1.24.5,2025-12-18_14:42:30","replset":"mongod_replset_1","logs_get":69383925,"logs_repl":34641955,"logs_success":34641955,"tps":0,"lsn":{"unix":1774423524,"time":"2026-03-25 16:25:24","ts":"7621091004833072010"},"lsn_ack":{"unix":1774423524,"time":"2026-03-25 16:25:24","ts":"7621091004833072010"},"lsn_ckpt":{"unix":1774423506,"time":"2026-03-25 16:25:06","ts":"7621090927523661999"},"now":{"unix":1776088237,"time":"2026-04-13 22:50:37"},"log_size_avg":"17.26KB","log_size_max":"1.48MB"}

This makes me suspect the checkpoint/status state may have become stale and may no longer reflect real progress.

[zhongli-james]

Thanks for your feedback!

This issue reveals a rather subtle bug in MongoShake—when the oplog reader fails due to CappedPositionLost, the error handling mechanism doesn't propagate correctly throughout the synchronization chain. The oplog reader gets stuck in an infinite retry loop (reporting an error every 6 seconds), but neither crashes nor updates its state correctly. Meanwhile, the downstream writer seems to have continued working for 18 days through some independent channel. This creates a very dangerous situation: the monitoring metrics (checkpoints, TPS, etc.) seen by operations personnel are no longer reflecting the true situation, potentially leading to the mistaken belief that synchronization has stopped when it is still running. Conversely, if data inconsistency does occur at a certain point in time, checkpoint information will be unable to help locate the problem.

---

In other words, the initial design of the ckpt mechanism was solely for confirming the position advancement during normal synchronization. However, it didn't consider how to handle inter-thread communication (in this case, the fetcher and batcher) in exceptional circumstances.

We will evaluate this flaw and provide an appropriate fix.

[syKim12]

Thanks for the explanation. It helped me a lot!

I'd also like to understand the recommended operational procedures when this condition occurs.

1. Safest Workaround
If this issue occurs, what is the safest workaround?

For instance, instead of reusing the stale checkpoint from the original all mode run, would it be better to restart MongoShake in incr mode using:

• A different checkpoint collection
• A current checkpoint/start position in collector.conf

2. Oplog Window Size
Is this issue more likely to happen if the source MongoDB has a relatively small oplog window? If so, would you recommend configuring a larger oplog window as a preventive measure?

3. Internal Components
You mentioned that the oplog reader failed while the downstream writer continued working.

Could you clarify which internal MongoShake components these correspond to?
For example, is the oplog reader the fetcher, and the downstream writer the batcher/executor?

4. Configuration Parameters
If that mapping is correct, are there any specific collector.conf parameters related to the fetcher, batcher, or executor that can help reduce the risk of this issue or make it easier to detect?

5. Recovery Procedures
Finally, once this condition happens, what is the recommended recovery procedure to avoid potential data inconsistency? For example, should we:

• Restart the process?
• Switch to incr mode?
• Use a new checkpoint collection?
• Reset the checkpoint?
• Or re-run an all sync?

Any practical workarounds or best practices would be greatly appreciated until a permanent fix is available.

[zhongli-james]

Thanks for your reply!

As for your questions:

1. Safest Workaround

A complete resynchronization is usually required. Since the ckpt info couldn't be used to recover.

2. Oplog Window Size

Yes. Your perspective is very sharp and this is the point. We add this part in Q&A.( Sorry, I just realized this content isn't in the English version of the Q&A.)

Generally speaking, the larger the data size, the longer the oplog retention period needs to be (in your case, this can be achieved by increasing oplogSizeMB).

3. Internal Components

‘For example, is the oplog reader the fetcher, and the downstream writer the batcher/executor?’ --- yes. You should be able to find architecture-related information in the README, or use any AI tool to analyze the code structure of this project.

4. Configuration Parameters

Yes, MongoShake have 'full_sync.reader.xxxx' configuration for reader and similar parameters designed for other modules. You should be able to check them in collector.conf.

It's worth noting that the default configuration already includes reasonable concurrency settings. Your case may not be solvable simply by tuning parameters, and tuning might introduce new problems (such as resource bottlenecks, the impact of load balancing on the source instance, etc.).

5. Recovery Procedures

It should be re-run an all sync with larger oplog window for source instance. You can estimate how long it will take to complete the full sync phase based on the amount of data that has been synchronized on the target side in the previous round and the total amount of data on the source instance.

Thanks for your feedback again. :)