Enable checkpoints for parallel

[dmpetrov]

Description

This is high priority issue!

Based on docs I see that Checkpoints are automatically disabled when Python threading or multiprocessing is detected to prevent race conditions.

This is a major limitations - parallel is usually enabled in real-life scenarios. And we actually should enable this by default. This means it does not work form many practical cases and its not even obvious - no warning, no errors.

[ilongin]

The parallel setting for UDF execution is actually not affected by this limitation — the docs mention this at the bottom of the Threading/Multiprocessing bullet point.

What is disabled is checkpoints when user-created threads or subprocesses are detected. The reason is that the checkpoint hash chain is cumulative — each checkpoint's hash depends on the previous one (H2 = hash(H1 +step_hash)). If .save() calls happen concurrently, their ordering is non-deterministic, which makes the hash chain different across runs and breaks checkpoint lookup.

def stage_a():
    dc.read_csv("data_a.csv").save("stage_a")

def stage_b():
    dc.read_csv("data_b.csv").save("stage_b")

# User runs independent stages in parallel
t1 = threading.Thread(target=stage_a)
t2 = threading.Thread(target=stage_b)
t1.start(); t2.start()
t1.join(); t2.join()

# Stage 3 depends on both
dc.read_dataset("stage_a").merge(dc.read_dataset("stage_b")).save("final")

All three saves share the same job's checkpoint hash chain. On run 1, stage_a saves first → chain is [H(A), H(A→B), H(A→B→final)]. On run 2, stage_b saves first → [H(B), H(B→A), H(B→A→final)]. Different hashes → no checkpoint reuse.

[dmpetrov]

hm... it's not clear from the docs then. any ideas on how to fix it?