Skip to content

Make key-offset shifts deterministic - #335

Draft
ezfoom wants to merge 1 commit into
KellerJordan:masterfrom
ezfoom:agent/deterministic-key-shift
Draft

Make key-offset shifts deterministic#335
ezfoom wants to merge 1 commit into
KellerJordan:masterfrom
ezfoom:agent/deterministic-key-shift

Conversation

@ezfoom

@ezfoom ezfoom commented Jul 11, 2026

Copy link
Copy Markdown

What changed

  • Clone the source slice before each in-place key-offset shift in train_gpt.py and train_gpt_medium.py.
  • Keep the intended one-token shift unchanged while removing source/destination aliasing.

Root cause

The assignments write k[:, 1:] while reading the overlapping k[:, :-1] view on CUDA. The copy kernel can observe values that another thread has already overwritten, so the result depends on execution order instead of representing a stable one-token shift.

At the record-84 tensor shape on an H100, eight repeated runs from the same input produced a maximum element delta of 7.28125 with the overlapping assignment. Cloning the RHS produced 0.0 repeat delta.

Impact

Key-offset attention becomes deterministic and matches the shift described by the surrounding comment. This affects the long-window key-offset paths in both training scripts.

Validation

  • H100 overlap reproduction at shape [1, 16384, 6, 128]: overlap repeat delta 7.28125; cloned RHS 0.0
  • python3 -m py_compile train_gpt.py train_gpt_medium.py
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant