[Bug] Non-added files scored as full new files when base content is unavailable in token scoring #1192

Description

calculate_token_score_from_file_changes has no guard for non-added files whose base-side blob is unavailable. When a modified or renamed file arrives with old_content=None and a valid new_content, the call to score_tree_diff(None, new_content, ...) treats the old AST as empty, so every node in the head file is counted as a net addition. The miner receives a token score for the entire head file instead of only the actual diff, inflating their reward.

The das-github-mirror fetcher (extractBlobText) returns null for any base blob that is binary, over 1 MB, or unreachable from git. The legacy PAT fetcher in github_api_tools.py applies the same conditions. Both paths feed into the shared calculate_token_score_from_file_changes, so both OSS and mirror PR scoring are affected.

Steps to Reproduce

1. Have a mirror-enabled (or legacy) repo with a PR that modifies or renames a file whose base-side blob is binary, over 1 MB, or otherwise unavailable (e.g. a large generated file that a PR trims down below 1 MB).
2. The DAS fetcher stores base_content = null, head_content = "<valid text>" for that file.
3. The validator calls calculate_token_score_from_file_changes for that PR.
4. The modified file has old_content=None and new_content="<valid source>".
5. score_tree_diff(None, new_content, ext, weights) is called — old_content is falsy, old_signatures stays an empty Counter, added = new_signatures - Counter() equals all nodes in the head file.
6. The full head-file token score is added to the miner's reward instead of the diff.

Expected Behavior

A non-added file (status='modified', 'renamed', etc.) with old_content=None should be skipped with score=0.0, the same way binary and oversized files are skipped via skipped-binary and skipped-large. An added file with old_content=None is a legitimate new-file contribution and must continue to score via tree-diff.

Actual Behavior

The modified/renamed file falls through to the tree-diff branch. score_tree_diff receives old_content=None, treats it as an empty old tree, and returns a score equal to the full node count of the head file. The miner is credited for all the code in the head file rather than only their actual changes.

Environment

• OS: N/A (validator-side logic)
• Python version: 3.12
• Commit/Version: 2087bf552f (branch test)

Additional Context

Root cause in calculate_token_score_from_file_changes (gittensor/validator/utils/tree_sitter_scoring.py):

The existing chain of guards handles new_content=None (skipped-binary), oversized content (skipped-large), and unsupported extensions (skipped-unsupported), but has no branch for the case where file.status != 'added' and content_pair.old_content is None. The missing guard means the scorer reaches:

old_content = content_pair.old_content  # None
new_content = content_pair.new_content  # "<valid source>"
file_breakdown = score_tree_diff(old_content, new_content, ext, weights)

Inside score_tree_diff (line 180): if old_content: — None is falsy, so old_signatures stays an empty Counter. The symmetric-difference step added = new_signatures - old_signatures returns all nodes of the head file as additions.

Confirmed in both fetchers:

• das-github-mirror github-fetcher.service.ts — extractBlobText returns null for isBinary, byteSize > 1_000_000, or missing git object.
• gittensor github_api_tools.py (lines 1248–1251) — legacy PAT path initialises old_content = None and only populates it when base_data is present, non-binary, and within the size limit.

Adjacent fix for reference: PR #977 (fix: bound old_content size in tree-sitter scoring) added the skipped-large guard for oversized old_content but left the None-for-non-added case unguarded.

Urgency: The five mirror-enabled repos become the sole scoring path after 15 May 2026. Any mirror PR whose base blob is binary or oversized would be over-scored from that date onward.

Fix: Add a single elif guard before the tree-diff branch:

elif file.status != 'added' and content_pair.old_content is None:
    file_result = FileScoreResult(
        filename=file.short_name,
        score=0.0,
        nodes_scored=0,
        total_lines=file.changes,
        is_test_file=is_test_file,
        scoring_method='skipped-missing-base',
    )