CONTRIBUTING.md

Contributing to tokmd

Thank you for your interest in contributing to tokmd! This project aims to be a robust code intelligence platform for humans, machines, and LLMs.

Please review our Code of Conduct before contributing.

If you are making a first contribution, start with the short Contributor Guide. This file remains the full reference for setup, testing, hooks, repository structure, and release guidance.

Development Setup

Nix (recommended)

1. Enter the dev shell:

   nix develop

2. Build:

   cargo build

Manual (non-Nix)

1. Rust Toolchain: Ensure you have a recent stable Rust toolchain installed.

   rustup update stable

2. Clone & Build:

   git clone https://github.com/EffortlessMetrics/tokmd.git
   cd tokmd
   cargo build

Optional Local Compiler Cache

For repeated local rebuilds, sccache is supported as an opt-in wrapper rather than a repo default.

1. Install sccache:

   winget install Mozilla.sccache
   # or
   cargo install sccache --locked

2. Verify setup:

   cargo sccache-check

3. Run Cargo through the wrapper:

   cargo with-sccache test --workspace --all-features
   cargo sccache-stats

The wrapper sets RUSTC_WRAPPER=sccache and defaults CARGO_INCREMENTAL=0 because incrementally compiled Rust crates do not produce sccache hits. Pass cargo xtask sccache --keep-incremental -- test ... if you want to preserve your current incremental setting. If you want cache hits across multiple worktrees or checkout roots, use cargo xtask sccache --basedir <PATH> -- test ... so the wrapper sets SCCACHE_BASEDIRS explicitly.

Local Hooks

Enable the project's git hooks for automated lint-fix and quality gating:

git config core.hooksPath .githooks

This is a one-time setup. Two hooks are provided:

• pre-commit — Runs cargo xtask lint-fix (fmt + clippy --fix + clippy verify), restages fixed files, and runs typos --diff (if installed). Only triggers when .rs, Cargo.toml, or Cargo.lock files are staged.
• pre-push — Runs cargo xtask gate --check (fmt check + cargo check + clippy + test compile-only) to catch issues before they reach CI.

You can bypass hooks with git commit --no-verify or git push --no-verify in emergencies.

Branch Naming

Branch prefixes are descriptive hints, not a policy boundary. The repo currently has a mix of human-created and tool-created prefixes, and we do not require a single canonical style before a branch can merge.

Common prefixes in current use:

• feat/ for feature work
• fix/ for bug fixes
• docs/ for documentation-only changes
• feature/ on some older or tool-generated branches

Prefer short, lowercase branch names that describe the change area, for example feat/browser-runner-packaging or fix/nix-source-filter. The merge bar is the same regardless of prefix: focused changes, passing checks, and an accurate PR description.

Project Structure

The codebase uses a tiered microcrate architecture:

crates/
├── tokmd-types/                     # Tier 0: Core data structures
├── tokmd-analysis-types/            # Tier 0: Analysis receipt types
├── tokmd-settings/                  # Tier 0: Clap-free settings types
├── tokmd-envelope/                  # Tier 0: Sensor report + FFI envelope contracts
├── tokmd-scan/                      # Tier 1: tokei wrapper + walk helpers
├── tokmd-model/                     # Tier 1: Aggregation logic
├── tokmd-sensor/                    # Tier 1: Sensor trait + substrate module/builder
├── tokmd-format/                    # Tier 2: Output rendering
├── tokmd-git/                       # Tier 2: Git analysis
├── tokmd-analysis/                  # Tier 3: Analysis orchestration
├── tokmd-analysis-api-surface/      # Tier 3: API surface analysis
├── tokmd-analysis-archetype/        # Tier 3: Archetype inference
├── tokmd-analysis-assets/           # Tier 3: Asset and dependency reports
├── tokmd-analysis-complexity/       # Tier 3: Cyclomatic/cognitive complexity
├── tokmd-analysis/src/content/      # Tier 3: Content scanning adapters
├── tokmd-analysis-derived/          # Tier 3: Core derived metrics
├── tokmd-analysis-entropy/          # Tier 3: High-entropy file detection
├── tokmd/src/analysis_explain/      # Tier 5: CLI metric explanation catalog
├── tokmd-analysis-fingerprint/      # Tier 3: Corporate fingerprint
├── tokmd-analysis/src/git/          # Tier 3: Git history adapters
├── tokmd-analysis-grid/             # Tier 3: Preset/feature matrix
├── tokmd-analysis-halstead/         # Tier 3: Halstead metrics
├── tokmd-analysis/src/imports/      # Tier 3: Import parsing + normalization
├── tokmd-analysis-license/          # Tier 3: License radar scanning
├── tokmd-analysis-maintainability/  # Tier 3: Maintainability index scoring
├── tokmd-analysis-near-dup/         # Tier 3: Near-duplicate detection
├── tokmd-analysis-topics/           # Tier 3: Topic-cloud extraction
├── tokmd-analysis-util/             # Tier 3: Shared analysis utilities
├── tokmd-gate/                      # Tier 3: Policy evaluation
├── tokmd-core/                      # Tier 4: Library facade + FFI
├── tokmd/                           # Tier 5: CLI binary
├── tokmd-python/                    # Tier 5: Python bindings (PyO3)
├── tokmd-node/                      # Tier 5: Node.js bindings (napi-rs)
└── tokmd-wasm/                      # Tier 5: Browser/worker bindings

Helper functionality that does not need an independent crates.io package now lives as owner modules: module-key logic in tokmd-model, path/exclude/math and tokeignore helpers in tokmd-scan, redaction/scan-args/badge/export-tree rendering in tokmd-format, context policy/git helpers in tokmd-core, and CLI/config/progress/tool-schema wiring in tokmd.

Testing Strategy

We prioritize deterministic outputs. This is critical because tokmd is used to generate receipts that must be stable over time.

1. Unit Tests

Run standard unit tests for model logic and utility functions:

cargo test

2. Integration / Golden Tests

We use insta for snapshot testing. These tests run the full CLI against fixtures.

Important: Line Endings & Bytes Our receipts include byte counts which are sensitive to line endings. To ensure cross-platform determinism:

• We enforce text eol=lf in .gitattributes for test fixtures and snapshots.
• Always accept snapshots from an LF-normalized checkout (Linux/WSL/macOS or a normalized Windows git checkout).
• If you see byte count diffs (e.g., 183 vs 172), check your line endings.

If you change output logic (e.g., new fields, different formatting):

1. Run tests: cargo test (they will fail with a diff).
2. Review changes: cargo insta review (requires cargo-insta installed).
3. Accept changes if they are intentional.

This guarantees that tokmd outputs (receipts) remain deterministic and stable.

3. Crate-Level Tests

Each crate may have its own tests in a tests/ directory. Run all tests with:

cargo test --workspace

4. Property-Based Testing (proptest)

Property-based testing verifies that functions behave correctly across a wide range of randomly generated inputs. Instead of testing specific examples, you define properties that should always hold.

When to use: Functions with well-defined invariants (e.g., determinism, length constraints, cross-platform consistency).

Running property tests:

cargo test -p tokmd-scan properties    # Run property tests for tokmd-scan
cargo test --workspace                  # Includes all property tests

Example patterns (from path/redaction property tests):

use proptest::prelude::*;

proptest! {
    /// Hash output is always exactly 16 hex characters.
    #[test]
    fn short_hash_length_is_16(input in ".*") {
        let hash = short_hash(&input);
        prop_assert_eq!(hash.len(), 16);
    }

    /// Same input always produces same hash (determinism).
    #[test]
    fn short_hash_is_deterministic(input in ".*") {
        let h1 = short_hash(&input);
        let h2 = short_hash(&input);
        prop_assert_eq!(h1, h2, "Hash must be deterministic");
    }

    /// Unix and Windows paths produce identical hashes.
    #[test]
    fn short_hash_normalizes_separators(path in arb_path()) {
        let unix_path = path.replace('\\', "/");
        let windows_path = path.replace('/', "\\");
        prop_assert_eq!(short_hash(&unix_path), short_hash(&windows_path));
    }
}

Key concepts:

• Define strategies to generate test data (e.g., arb_path() for path-like strings)
• Use prop_assert! and prop_assert_eq! instead of standard assertions
• Use prop_assume! to skip inputs that don't meet preconditions
• Proptest will shrink failing cases to minimal reproducible examples

5. Fuzz Testing (libfuzzer)

Fuzz testing bombards functions with arbitrary byte sequences to find crashes, panics, and edge cases. This is especially useful for parsing code and security-sensitive operations.

Prerequisites:

rustup install nightly
cargo +nightly install cargo-fuzz

Running fuzz targets:

cargo +nightly fuzz run fuzz_entropy --features content
cargo +nightly fuzz run fuzz_json_types --features types
cargo +nightly fuzz run fuzz_redact --features redact

Available targets (see fuzz/README.md for complete list):

Target	Feature	Description
fuzz_entropy	content	Entropy calculation
fuzz_json_types	types	JSON deserialization of receipt types
fuzz_normalize_path	model	Path normalization
fuzz_toml_config	config	tokmd.toml config parsing
fuzz_redact	redact	Path redaction

Using dictionaries for better coverage:

cargo +nightly fuzz run fuzz_json_types --features types -- -dict=fuzz/dict/json.dict

Limiting input size:

cargo +nightly fuzz run fuzz_entropy --features content -- -max_len=4096

For full documentation, see fuzz/README.md.

6. Mutation Testing (cargo-mutants)

Mutation testing verifies test suite quality by introducing small changes (mutations) to the code and checking if tests catch them. Surviving mutations indicate gaps in test coverage.

Install:

cargo install cargo-mutants

Running locally:

# Test mutations in a specific file
cargo mutants --file crates/tokmd-format/src/redact/mod.rs

# Test a specific crate (run from crate directory for proper test scoping)
cd crates/tokmd-format && cargo mutants

# Test with all features enabled
cargo mutants --all-features

Interpreting results:

• Killed: Test suite caught the mutation (good)
• Timeout: Mutation caused tests to hang (usually acceptable)
• Unviable: Mutation caused compilation failure (neutral)
• Missed/Survived: Mutation was not detected (indicates a test gap)

Configuration (.cargo/mutants.toml):

all_features = true
gitignore = true
timeout_multiplier = 2.0

# Exclude test code from mutation
exclude_globs = ["**/tests/**", "fuzz/**"]

# Exclude boilerplate
exclude_re = ["impl.*Display", "fn main\\("]

CI integration: Mutation testing runs automatically on PRs via .github/workflows/mutants.yml:

• Only tests changed .rs files (up to 20 files)
• Generates a summary JSON with survivors
• PRs with surviving mutations will fail CI

When mutations survive:

1. Review the mutation to understand what behavior wasn't tested
2. Add a test that specifically exercises that code path
3. Re-run mutation testing to verify the mutation is now caught

Code Style

• Run cargo xtask lint-fix to auto-fix formatting and clippy issues.
• Run cargo fmt-fix for a fast, Windows-safe fmt-only fix.
• Run cargo fmt-check to verify formatting only.
• Run cargo xtask gate --check to verify the full quality gate locally.
• cargo xtask gate --check now uses a disposable temp CARGO_TARGET_DIR and forces CARGO_INCREMENTAL=0 unless you override CARGO_TARGET_DIR yourself, so repeated gate runs do not leave a huge target/ tree behind.
• On Unix-like systems, cargo xtask gate --check also refuses to start when free disk drops below the TOKMD_MIN_FREE_GB threshold.
• The gate first prunes stale tokmd-gate-target-* temp dirs left by earlier killed/cancelled runs (older than TOKMD_GATE_STALE_HOURS, default 3) so accumulated orphan target trees do not trip that guard on long-lived self-hosted runners (issue #309).
• Run cargo trim-target --check to inspect reclaimable target/debug footprint.
• Run cargo trim-target to drop Windows PDBs and incremental state from target/debug without a full cargo clean.
• If you need full local symbols on Windows for a debugging session, use $env:RUSTFLAGS='-C debuginfo=2'; cargo test ....
• Run cargo sccache-check to verify the optional local compiler cache.
• Run cargo with-sccache test --workspace --all-features for cache-friendly local rebuilds.
• cargo with-sccache check|clippy|test ... now uses a disposable temp CARGO_TARGET_DIR by default when you have not already set one, so validation runs clean up after themselves.
• The repo-native sccache wrapper uses a deterministic per-workspace SCCACHE_SERVER_PORT; set SCCACHE_SERVER_PORT yourself if you need to override it.
• For cross-worktree cache reuse, run cargo xtask sccache --basedir <PATH> -- test --workspace --all-features.
• Expect the biggest sccache wins on repeated library and dependency compiles; final binary and test-binary link steps still run uncached.
• On Unix-like systems, both wrappers refuse to start when free disk drops below the TOKMD_MIN_FREE_GB threshold. Override that env var if you need a different floor for a larger machine.

Contribution Areas

Priority Areas

1. Enricher implementations — Add new analysis enrichers in crates/tokmd-analysis/src/:

   • Look at existing enrichers like derived.rs or git.rs for patterns
   • Add new modules and wire them into analysis.rs

2. Output format templates — Improve Markdown/SVG rendering in crates/tokmd-format/src/analysis/

3. Language support — Extend import graph parsing for more languages

4. Documentation — Recipe examples, use cases, and tutorials

Adding a New Enricher

Enrichers are implemented as owner modules:

1. Create a new module under crates/tokmd-analysis/src/
2. Add the data structures to crates/tokmd-analysis-types/src/lib.rs
3. Wire it into the preset system in crates/tokmd-analysis/src/analysis.rs
4. Add formatting support in crates/tokmd-format/src/analysis/
5. Add tests and update documentation

Pull Requests

1. Open an issue to discuss major changes first.
2. Ensure your PR includes relevant tests.
3. Update documentation if you change CLI behavior or flags.
4. Reference the relevant section in ROADMAP.md.

Receipt Schema

tokmd treats outputs as "receipts". If you modify the JSON output structure:

For core receipts (lang, module, export):

1. Update struct definitions in tokmd-types or tokmd-model.
2. Update formatting in tokmd-format.
3. Update the formal schema in docs/schema.json.
4. Increment schema_version for breaking changes.

For analysis receipts:

1. Update struct definitions in tokmd-analysis-types.
2. Update formatting in tokmd-format::analysis.
3. Update docs/SCHEMA.md documentation.
4. Increment ANALYSIS_SCHEMA_VERSION for breaking changes.

Feature Flags

Some features are gated to allow selective compilation:

• git: Git history analysis (shells out to git command)
• content: File content scanning (entropy, TODOs, duplicates)
• walk: Filesystem traversal for assets
• halstead: Halstead software science metrics (requires content + walk)

When adding new features with heavy dependencies, consider making them optional.

Publishing to crates.io

Publishing is handled via cargo xtask publish, which ensures correct dependency order, validates packaging, and handles propagation delays. See RELEASE.md for the full release process, including the CI-driven tag workflow.

Workflow

# 1. Review the publish plan
cargo xtask publish --plan --verbose

# 2. Validate packaging (runs cargo package --list for each crate)
cargo xtask publish --dry-run

# 3. Publish for real (requires confirmation)
cargo xtask publish --yes

# 4. Publish and create git tag
cargo xtask publish --yes --tag

Pre-publish checks

The xtask runs these checks before publishing:

• Clean git working directory
• Version consistency across all crates
• CHANGELOG.md contains the version
• All tests pass

Skip individual checks with --skip-git-check, --skip-version-check, --skip-changelog-check, --skip-tests, or all with --skip-checks.

Resuming after failure

If publishing fails partway through:

cargo xtask publish --from tokmd-format --yes

Justfile shortcuts

just publish-plan   # cargo xtask publish --plan --verbose
just publish-dry    # cargo xtask publish --dry-run
just publish        # cargo xtask publish --yes
just publish-tag    # cargo xtask publish --yes --tag

Language Bindings

Native bindings are available for Python and Node.js:

crates/
├── tokmd-core/     # FFI layer via ffi::run_json()
├── tokmd-python/   # PyO3 bindings → PyPI
└── tokmd-node/     # napi-rs bindings → npm

Python (tokmd-python)

import tokmd

# All functions return native Python dicts
result = tokmd.lang(paths=["src"])
result = tokmd.analyze(paths=["."], preset="risk")

• Install: pip install tokmd
• Functions: lang(), module(), export(), analyze(), diff()
• Releases GIL during long scans

Node.js (tokmd-node)

const tokmd = require('tokmd');

// All functions return Promises
const result = await tokmd.lang({ paths: ['src'] });
const analysis = await tokmd.analyze({ paths: ['.'], preset: 'risk' });

• Install: npm install tokmd
• Functions: lang(), module(), export(), analyze(), diff()
• Non-blocking via spawn_blocking()

FFI Interface

Both bindings use the unified FFI layer in tokmd-core:

pub fn run_json(mode: &str, args_json: &str) -> String
// Response: {"ok": bool, "data": {...}, "error": {...}}

Design principles:

• JSON serialization at FFI boundary for simplicity
• Mirror the CLI's mental model (lang, analyze, diff)
• Return native language types (Python dicts, JS objects)
• Cross-platform wheels/prebuilds via CI matrix

AI-Assisted Development

If you use AI tools like Claude Code to contribute to this project, please refer to CLAUDE.md for project-specific guidance. This file contains:

• Build and test commands
• Architecture overview and crate hierarchy
• Critical patterns (deterministic output, path normalization, etc.)
• Testing strategies and snapshot management
• Key dependencies and documentation references

The CLAUDE.md file helps AI assistants understand the codebase conventions and produce contributions that align with project standards.