bittensor-core: one Rust core (keys, keyfiles, timelock, codec, RFC-0078 digest, Ledger) under a thin Python SDK#2846
Open
unarbos wants to merge 763 commits into
Open
bittensor-core: one Rust core (keys, keyfiles, timelock, codec, RFC-0078 digest, Ledger) under a thin Python SDK#2846unarbos wants to merge 763 commits into
unarbos wants to merge 763 commits into
Conversation
…ock decay in conviction test The dynamic-tempo scheduler (live on mainnet since spec 424) anchors epochs on absolute block numbers: LastEpochBlock ~ 8.6M in mainnet state. The clone restarts numbering at 0, so blocks_since saturates to zero and no subnet ever runs an epoch -- the migration cannot re-seed because HasMigrationRun is already true in the copied state. This is why test-balancer-operation's validator-rewards phase hung on three consecutive CI runs (no Emission/Dividends change on any of 128 subnets, tempo=1 notwithstanding). build-patched-spec now drops LastEpochBlock and PendingEpochAt so they fall back to 0 and epochs fire within each subnet's first tempo blocks, like the legacy modulo scheduler the clone previously relied on. test-locks-conviction: a lock starts decaying the moment its perpetual flag clears, and ~2 blocks (~1.9M mass on a 583B lock) elapse before the assertion reads it back. Allow a bounded decay margin instead of requiring full immediate conviction to the last unit. Co-authored-by: Cursor <cursoragent@cursor.com>
The turbo pool (Ryzen 9 9950X, benchmarked in #2854 at ~2x the heavy boxes without sccache) has two replicas, so routing stays selective: - build runtime artifacts: gates all try-runtime checks and clone shards, the pipeline's critical path (26m on heavy) - cargo test: longest Check Rust job (16.9m on heavy) Compounds with the sccache pool; everything else stays on heavy where there is capacity. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…gram totals test-total-issuance-trackers assumed register_network emits NetworkAdded synchronously, but since the registration refactor (7da9766, merged via feat/squashed-2026-07-06) a registration at the subnet limit prunes a subnet and queues itself: the extrinsic emits NetworkRemoved + NetworkRegistrationQueued, and NetworkAdded only fires from on_idle once the pruned subnet's chunked storage cleanup completes (~6 min against mainnet-scale maps). registerSubnet now handles both paths, waiting on system events for the owner-matched NetworkAdded, and the stake-sdk shard's per-test budget doubles to cover the wait. test-alpha-deprecated-stake-histogram awaited three storage queries per Alpha entry (hundreds of thousands of sequential round-trips on mainnet state), overrunning its 900 s budget with the scan unfinished. It now prefetches TotalHotkeyAlpha/TotalHotkeyShares/TotalHotkeySharesV2 in full up front and does the Alpha scan with zero per-entry RPCs. Co-authored-by: Cursor <cursoragent@cursor.com>
State scraping dominates the try-runtime jobs (30-40 min for mainnet,
now the longest leg of PR CI). The nightly refresh-mainnet-snapshot
workflow gains a per-network matrix job that runs try-runtime-cli
create-snapshot and publishes try-runtime-snap-{devnet,testnet,mainnet}
artifacts; the try-runtime consumers restore the latest main-branch
snapshot and check migrations via `on-runtime-upgrade snap`, falling
back to `live --uri` when no snapshot exists (first run, expired
artifact, failed nightly scrape, or the fresh-try-runtime-state label).
Same trust model as the clone snapshot: only artifacts produced by
default-branch runs are consumed, so a PR cannot poison state for
another PR. Producer and consumers pin the same try-runtime-cli release
because the .snap format is version-coupled.
Co-authored-by: Cursor <cursoragent@cursor.com>
The conviction runtime rejects incoming locked alpha by default (AccountFlags opt-in via set_reject_locked_alpha, call_index 142); test-locks-conviction predates the default and its two same-subnet transferStake steps moved half a lock to destinations that never opted in, failing with AccountRejectsLockedAlpha once CI first reached that step. Opt both destinations in before transferring, and assert the default-reject actually fires for the owner-coldkey path first — the security property the flag exists for. Co-authored-by: Cursor <cursoragent@cursor.com>
Gate everything OS-bound in bittensor-core behind a default-on `host` feature (reqwest, rayon, sodiumoxide, openssl/fernet keyfiles, sp-core/std) so the core compiles for wasm32-unknown-unknown; timelock gains a portable reveal_round() and uses web-time on wasm, where SystemTime::now() aborts. New bittensor-core-wasm crate mirrors the py binding surface via wasm-bindgen for fully client-side signing in the browser; decoded-object keys are __proto__-hardened. The core-wasm CI job holds both seams: cargo check on the wasm target plus a node smoke test replaying the golden fixtures and marshalling boundaries. Co-authored-by: Cursor <cursoragent@cursor.com>
…wasm's locals limit
Every matrix job pays ~2-3 minutes of fixed cost (checkout, uncached uv sync, downloading + docker-loading the localnet image, booting a chain) — at one job per test that was ~113 jobs spending ~20x more on setup than on tests, and 4-5 waves at max-parallel 32. One job per test file (10 today) fits a single wave; the e2e conftest is already session-scoped, so a shared localnet per file is the granularity the suite was written for. Retries rerun only the failures (--lf) on a fresh chain. uv's cache is keyed on sdk/python/uv.lock.
Every sdk e2e job was downloading a multi-GB tarball artifact and docker-loading it (~1-2 min each). Push the image once to ghcr.io/raofoundation/subtensor-localnet:ci-<sha> and let jobs docker pull it — layers come down compressed and in parallel. Fork PRs never reach these jobs (trusted-pr gate), so packages: write stays non-fork. Co-authored-by: Cursor <cursoragent@cursor.com>
RollDelta carried the conviction change as an unsigned decrease (previous - rolled, saturating), but rolling a lock forward normally GROWS conviction, so the delta saturated to zero while apply_roll_delta_to_aggregate stamped aggregate.last_update = now. The aggregate could then never make that window up via its own roll-forward: every individual roll permanently stripped the aggregate buckets (DecayingHotkeyLock etc.) of the conviction their members matured, understating hotkey_conviction and eventually saturating aggregates to zero on transfer_lock's reduce step — the exact "aggregate conviction 0 is less than lock" failure in the clone-upgrade locks e2e. Carry growth and decay as separate unsigned components and apply both to the aggregate. Co-authored-by: Cursor <cursoragent@cursor.com>
The committed _generated files were dumped at spec 424; the drift gate in the stake-sdk clone-upgrade job rejects the mismatch. Regenerated from this branch's runtime (non-fast-blocks build, matching the gate's node), classified the six error names the runtime added since, and dropped sudo_set_subnet_owner_hotkey from the raw-only list now that the call is gone from AdminUtils. Co-authored-by: Cursor <cursoragent@cursor.com>
The balanceAfter < balanceBefore assertion reads state at the finalized head, but the test only waited a fixed 2 finalized blocks after tx.wait(). When GRANDPA finality lags best by more than that, the wait ends before the fee-paying block finalizes and both reads return the identical pre-transaction balance (the recurring "expected X to be less than X" flake). Wait until the receipt's own block number finalizes instead. Co-authored-by: Cursor <cursoragent@cursor.com>
The stake-sdk shard ran the full localnet-tuned e2e suite against the upgraded mainnet clone and 87 of 113 tests failed on environment assumptions: register_network is rate-limited and at the subnet limit (every owned_subnet fixture died with Custom error 6), the crowdloan minimum duration exceeds the hardcoded 5000 blocks, and dev accounts other than //Alice are unfunded. - Scope the clone step to the clone-safe subset (reads, error decoding, intent sweep against the live upgraded runtime); the write-path suite already runs on localnet in the Bittensor E2E workflow, and on the clone's 12s blocks it would blow the job timeout. - Add scripts/prepare_clone_for_e2e.py, run before the subset: sudo-zero NetworkRateLimit, raise MaxSubnets above the mainnet count so registration stays on the immediate path, and decay the lock cost. - Derive the crowdloan test's duration from the runtime's MinimumBlockDuration constant instead of hardcoding 5000. - Regenerate docs/errors.mdx and the website error catalog for spec 428 (the docs drift gate caught them stale after the codegen refresh). Co-authored-by: Cursor <cursoragent@cursor.com>
The snapshot fallback's warp sync repeatedly wedges downloading finality proofs (stuck at 0.00 MiB for the rest of the job; two stake-sdk shards burned their full 90 minutes this way while sibling shards scraped the same state in 8-14 minutes). Give each attempt 20 minutes, kill the orphaned sync node, and retry up to three times with fresh peers. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the full
sdk/bittensor-core-spec.mdplan (phases 0–5). One workspace crate,bittensor-core, now owns everything whose right answer is chain-defined — sp-core key primitives, keyfiles, drand timelock, ML-KEM, the SCALE codec + runtime-metadata engine, extrinsic assembly, multisig derivation, and the RFC-0078 merkleized-metadata digest — exposed to Python through one PyO3 binding crate (bittensor-core-py, PyPI:bittensor-core). The Python public API is frozen throughout; each phase landed as its own commit and is independently revertible.By phase:
sdk/python/tests/fixtures/shape_corpus/), crate skeleton,build-core-wheels.yml.py-sp-core+bittensor-drandsources consolidated into the core; old crates, workflows, and the drand C ABI deleted; release train stamps one version.merkleized-metadata0.5.1, vectors pinned against polkadot-jsmerkleizeMetadata),LedgerSignerclear-signing via the Polkadot generic app (featureledger),--ledgerCLI flag, docs guide.Runtime+ codec become the productioncodec.pyseam (CallBytesreplacesGenericCall); cyscale, xxhash,TYPE_REGISTRY, and the cyscale-era quirks are deleted. Decode hot paths cross the FFI once per page and run off the GIL (rayon), with a limb-based base58 for ss58 rendering (~6x, byte-identical to sp-core).Acceptance benchmark (finney, 2026-07-10 — spec §10)
compose_callquery_mapdecodeReview focus
codec.pyseam rewrite and its consumers (storage.py,extrinsics.py,runtime.py,runtime_api.py,interface.py)cargo test -p bittensor-core) and through the Python seambittensor-core/src/signers/,bittensor/ledger.py) — on-device verification still to be done by someone with hardwareTest plan
cargo test -p bittensor-core— 44 tests green (corpus, goldens, digest vectors, keys, timelock)cd sdk/python && uv run pytest— 874 unit tests green (one pre-existing EVM ABI-drift failure from 08c1a21, unrelated)Made with Cursor
Update 2026-07-10: runtime changes merged in
This branch now also includes the
feat/squashed-2026-07-06runtime work (previously PR #2847 → devnet-ready), merged in91b42d5152:DustRemovalhook mirrors balances dust burns into subtensorTotalIssuance; leasing/order-swap transfers consolidated ontotransfer_taoBalancerAlphaReservoir/BalancerTaoReservoir) with dissolution folding reservoirs into reserves before payoutmigrate_fix_subnet_hotkey_lock_swaps(index-based rewrite),migrate_fix_total_issuance_evm_fees(dust-collection reset entry)decrease_take/increase_takeflipPays::No→Pays::Yes(anti-spam; metadata-only, no transaction_version bump needed)eco-tests/src/tests_mentat_indexer.rs(indexer storage contract),runtime/tests/balances_dust.rs