Question: how to run debezium on Postgres to do periodic sync's up to the latest snapshot

[diwu-sf]

Hi,
I have a usecase where I want to trigger the debezium sync every hour to CDC from Postgres to DLT, each time the sync runs, I would like it to pick up from the latest state checkpoint and continue from that point onto the current WAL position at the start of the sync.

Can you provide an example of how to configure this?
The doc shows how to do either an initial sync, or a continuous sync, but what I'm looking for is something in between as a scheduled periodic incremental sync.

Thanks!

[ismailsimsek]

@diwu-sf You can store Debezium offsets to a file to manage its continuous execution. Use Utils to periodically stop and restart the process (e.g., hourly). Debezium will resume consuming data from where it last left off, using the stored offset data.

see offset setting

interrupting engine

pydbzengine/pydbzengine/helper.py

Lines 20 to 30 in ac6f148

	class Utils:
	@staticmethod
	def run_engine_async(engine, timeout_sec=22):
	"""
	Runs an engine asynchronously with a timeout.
	This method runs the given engine's `run` method in a separate thread
	and applies a timeout. If the engine's `run` method doesn't complete
	within the specified timeout, a TimeoutError is raised, and the thread
	is interrupted.