read_dataset: store chain hash in dataset version

[ilongin]

Currently read_dataset() uses sha256(dataset.name + version) as the starting checkpoint hash. This is unsafe — someone can delete a version and recreate it with different data, and the hash stays the same. It also doesn't survive checkpoint cleanup, so global checkpoint reuse is impossible.

Proposal: store the chain hash that produced the dataset in a new hash field on the dataset version record. When .save() creates a dataset version, populate this field with the computed chain hash. When read_dataset() is used in another chain, use dataset_version.hash as the starting hash instead of the URI-based one.

This also means .save() no longer needs to create checkpoints. Instead of looking up the checkpoint model, .save() searches dataset versions for a matching hash — if found, the dataset already exists with this exact computation, skip the chain. The dataset version itself becomes the checkpoint for .save().

Checkpoints then become UDF-specific — only needed for UDF step skip/resume (including partial completion recovery). .save() uses dataset_version.hash.

This enables global checkpoints for .save() — if the same chain with the same inputs was run months ago in a different job, the dataset version still exists with the matching hash, so the chain is skipped. No dependency on checkpoint expiry.

Notes:

• Add an index on the new hash field for fast lookups
• If dataset_version.hash is None (dataset created externally, not through a chain), fall back to a safe default

[shcheklein]

Can we use the existing UUID for this?

[ilongin]

Can we use the existing UUID for this?

@dmpetrov has given idea of saving hash to dataset version and using it in other chains, but yes, we could just use uuid

[dmpetrov]

There is no need in uuids here - hashes solve the issue.

Hashes also are better since we are using the same machinery all around checkpoints. And there is more advantages with hashes:

• compare chains logic by comparing hashes for debugging. It will work in any deepness of a pipeline.
• potentially global cache of checkpoints - cross scripts.

[shcheklein]

By UUID I mean the one that we uniquely generate for each dataset version.

So, it will be changing indeed if the new dataset version is saved (even if the same content / same operations). It doesn't though happen for delta usually (it skips creating new versions).

It is simpler to implement initially imho, and arguably even more precise and predictable semantics (new dataset version - new run).

[ilongin]

So what should we go with? uuid is quicker and it solves the issue of picking up correct data to start hashing with, and storing hash gives us more opportunities like "global" checkpoints, checking out if there is some dataset already saved with same hash to avoid re-computation etc.

[shcheklein]

let's start with something we can do fast first please

also, btw, hashing UUID or set of operations probably doesn't change global checkpoints reusability, no? The only real difference I see is that UUID is more strict (so less hits potentially), but on the other hand less surprises I think.