Complete Wave 5 remaining work and add language rules checklist

Author: Eric Hartford

docs/more_language_rules.md

@@ -0,0 +1,81 @@
+# More Language Rules Checklist
+
+## Scope To Add Up Front
+- [ ] Numeric separators in literals (`3_000`, `1_000_000`, `0xFF_AA_22`, `0b1111_0000`, `3.141_592_653`)
+- [ ] Trailing commas are permitted (optional), never required, in all list-like grammar positions (params, args, fields, variants, tuple/array literals, match arms, imports)
+- [ ] Raw string literals (including delimiter variants, if adopted)
+- [ ] Multiline string indentation behavior (dedent/strip policy)
+- [ ] Byte literals (`b'A'`) and clear typing rules
+- [ ] Unused binding syntax (`_`) in all binding positions
+- [ ] String escapes parity (`\\xNN`, `\\0`, and existing escapes) across string kinds
+- [ ] `todo` and `unreachable` expressions with `Never` typing behavior
+- [ ] No-shadowing language rule (explicitly enforced)
+- [ ] Pipeline-first migration guidance for shadowing-heavy Rust patterns
+
+## 1. Specification Updates (`docs/with-specification.md`)
+- [ ] Add numeric separator lexical grammar and invalid forms
+- [ ] Add trailing comma grammar rules and examples
+- [ ] Explicitly state trailing commas are optional/permitted and never required
+- [ ] Add raw string grammar, escaping semantics, and delimiter rules
+- [ ] Add multiline string indentation/dedent semantics with examples
+- [ ] Add byte literal grammar and type rules
+- [ ] Add unused binding rule (`_`) for let, params, patterns, loops, match
+- [ ] Add/clarify string escape table including hex/null escapes
+- [ ] Add `todo`/`unreachable` semantics and type-checking rules
+- [ ] Add explicit no-shadowing rule and diagnostics expectations
+- [ ] Add spec examples for pipeline replacements where shadowing is disallowed
+
+## 2. Idiomatic Guide Updates (`docs/with-idiomatic-guide.md`)
+- [ ] Add style guidance for numeric separators
+- [ ] Add trailing-comma style guidance
+- [ ] Add raw/multiline string best practices
+- [ ] Add byte-literal usage examples
+- [ ] Add unused-binding (`_`) idioms
+- [ ] Add `todo`/`unreachable` development guidance
+- [ ] Add no-shadowing coding patterns with pipeline-first style
+
+## 3. Migration Guide Updates (`docs/with-migration-guide.md`)
+- [ ] Add Rust-to-With numeric separator examples
+- [ ] Add Rust-to-With trailing comma expectations
+- [ ] Add Rust-to-With raw string and multiline string examples
+- [ ] Add Rust-to-With byte literal examples
+- [ ] Add Rust-to-With unused variable mappings (`_`)
+- [ ] Add Rust shadowing -> With pipeline cookbook section
+- [ ] Add diagnostics mapping for no-shadowing migration errors
+
+## 4. Bootstrap Compiler Implementation (`bootstrap/`)
+- [ ] Lexer: accept numeric separators for integer/float/radix literals
+- [ ] Lexer: add raw string tokenization
+- [ ] Lexer: add byte literal tokenization and validation
+- [ ] Parser: accept optional trailing commas across all targeted productions (no trailing comma must still parse)
+- [ ] Parser/Sema: `_` binding handling across binding contexts
+- [ ] Parser/Sema: enforce no-shadowing rule with clear diagnostics
+- [ ] Sema: `todo`/`unreachable` expression typing (`Never`) and propagation
+- [ ] String literal processing: add `\\xNN` and `\\0` behavior parity
+- [ ] Add/extend bootstrap tests for each new rule and failure mode
+- [ ] Ensure dump outputs remain deterministic after grammar/lexer changes
+
+## 5. Self-Hosted Compiler Implementation (`src/`)
+- [ ] Mirror lexer support for numeric separators
+- [ ] Mirror lexer support for raw strings
+- [ ] Mirror lexer support for byte literals
+- [ ] Mirror parser support for optional trailing commas (no trailing comma must still parse)
+- [ ] Mirror parser/sema support for `_` unused bindings
+- [ ] Mirror no-shadowing diagnostics and enforcement
+- [ ] Mirror `todo`/`unreachable` typing behavior
+- [ ] Mirror string escape behavior (`\\xNN`, `\\0`, etc.)
+- [ ] Add/extend selfhost tests for each feature and error case
+- [ ] Confirm Stage0/Stage2 parity on new coverage corpus
+
+## 6. Source Conformance Pass (Repo-Wide)
+- [ ] Run formatter/lint and compile all With source
+- [ ] Audit and replace shadowing-style code with pipeline patterns
+- [ ] Update existing examples/tests to follow new no-shadowing rule
+- [ ] Add migration notes for any nontrivial source rewrites
+
+## 7. Validation Gates
+- [ ] Bootstrap test suite passes
+- [ ] Selfhost test suite passes
+- [ ] Wave parity scripts still pass
+- [ ] New language-rule corpus passes on both compilers
+- [ ] No unresolved known divergences for these features

docs/with-selfhost-detailed-plan.md

@@ -2,7 +2,7 @@
 
 ## Architecture-First Execution Plan
 
-**Status:** Wave 4 baseline complete (resolved/module-graph dump path + Wave 4 unit + parity gates).
+**Status:** Wave 5 complete (`--dump-typed` parity + obligation selection cache + generic bound checks + dyn trait-call compatibility + specialization cache + typed sidecar persistence).
 **Goal:** Build a clean-room, self-hosted With compiler in With.
 **Bootstrap:** Stage0 (Zig implementation) remains semantic oracle.
 

docs/with-selfhost-plan.md

@@ -1,6 +1,6 @@
 # Self-Hosting With — Architecture-First Plan
 
-**Status:** Wave 4 baseline implemented (resolved/module-graph dump path + Wave 4 unit + parity gates).
+**Status:** Wave 5 complete (`--dump-typed` parity + obligation selection cache + generic bound checks + dyn trait-call compatibility + specialization cache + typed sidecar persistence).
 
 ---
 

docs/with-selfhost-wave3.md

@@ -269,15 +269,15 @@ Corpus policy:
 - [x] Freeze Wave 3 AST oracle contract against Stage0 current output.
 - [x] Define Wave 3 AST parity corpus file.
 - [x] Inventory Stage0 parser feature buckets from `bootstrap/src/Parser.zig` + `bootstrap/src/Parse.zig` tests.
-- [ ] Align `src/Ast.w` node coverage/flags/spans needed for Stage0 dump parity.
+- [x] Align `src/Ast.w` node coverage/flags/spans needed for Stage0 dump parity.
 - [x] Align module span + decl span computation with Stage0 behavior.
-- [ ] Align top-level declaration parsing parity (`fn/type/use/let/extern/trait/impl/extend/c_import`).
-- [ ] Align attribute parsing parity (`@[...]` forms used in Stage0).
-- [ ] Align expression parser precedence/associativity with Stage0.
+- [x] Align top-level declaration parsing parity (`fn/type/use/let/extern/trait/impl/extend/c_import`).
+- [x] Align attribute parsing parity (`@[...]` forms used in Stage0).
+- [x] Align expression parser precedence/associativity with Stage0.
 - [x] Add/align missing literal expression parsing (including `char` literal expressions).
-- [ ] Align type-expression parsing parity.
-- [ ] Align pattern parsing parity.
-- [ ] Align parser recovery behavior for deterministic outputs.
+- [x] Align type-expression parsing parity.
+- [x] Align pattern parsing parity.
+- [x] Align parser recovery behavior for deterministic outputs.
 - [x] Align `src/render.w` with Stage0 AST render format.
 - [x] Add self-host `check --dump-ast` support in `src/main.w`.
 - [x] Route `ast` command through shared dump formatting path where practical.
@@ -290,4 +290,4 @@ Corpus policy:
 
 Current scope note:
 - Wave 3 parity gate is green for the explicit Wave 3 corpus in `test/wave3/ast_corpus.txt`.
-- Unchecked items above remain follow-up work to broaden parity coverage (notably full top-level declaration parity, attribute parity, and parser recovery parity).
+- Remaining Stage0-vs-selfhost AST diffs seen in broad corpus sweeps are outside Wave 3 parser surface (import/c_import expansion, chained `if let`, and other later-wave frontend behaviors).

docs/with-selfhost-wave5.md

@@ -297,27 +297,27 @@ Corpus policy:
 
 ## Implementation Checklist
 
-- [ ] Freeze Wave 5 typed dump oracle contract against Stage0 current output.
-- [ ] Create `test/wave5/typed_corpus.txt`.
-- [ ] Inventory Stage0 type/trait/generic behaviors needed for Wave 5 corpus from `bootstrap/src/Sema.zig`.
-- [ ] Define canonical Wave 5 type representation and `TypeId` usage rules.
-- [ ] Expand `TypeKey`/interning coverage for Wave 5 type forms.
-- [ ] Add deterministic type-name rendering helpers for typed dump output.
-- [ ] Implement deterministic trait declaration collection tables.
-- [ ] Implement deterministic impl collection tables.
-- [ ] Implement coherence duplicate-impl checks with Stage0 parity.
-- [ ] Implement orphan-rule checks with Stage0 parity.
-- [ ] Implement trait obligation model and selection cache.
-- [ ] Implement trait-bound checking for generic instantiation.
-- [ ] Implement dyn-trait compatibility checks needed for typed parity corpus.
-- [ ] Implement generic substitution map and specialization cache.
-- [ ] Ensure canonical typed pass outputs are stored in `Zcu`.
-- [ ] Wire Wave 5 type/trait pass into frontend pipeline.
-- [ ] Add self-host `check --dump-typed` path if missing in current path.
-- [ ] Implement Stage0-compatible deterministic typed dump emitter.
-- [ ] Add `test/wave5/` unit tests for types/traits/coherence/generic instantiation.
-- [ ] Add typed-dump formatting and determinism unit checks.
-- [ ] Add `scripts/run_wave5_type_trait_unit_tests.sh`.
-- [ ] Add `scripts/run_wave5_typed_parity.sh` (Stage0 diff + determinism rerun + divergence reporting).
-- [ ] Verify Wave 5 unit and parity gates pass locally.
-- [ ] Mark Wave 5 progress in top-level self-host plan docs.
+- [x] Freeze Wave 5 typed dump oracle contract against Stage0 current output.
+- [x] Create `test/wave5/typed_corpus.txt`.
+- [x] Inventory Stage0 type/trait/generic behaviors needed for Wave 5 corpus from `bootstrap/src/Sema.zig`.
+- [x] Define canonical Wave 5 type representation and `TypeId` usage rules.
+- [x] Expand `TypeKey`/interning coverage for Wave 5 type forms.
+- [x] Add deterministic type-name rendering helpers for typed dump output.
+- [x] Implement deterministic trait declaration collection tables.
+- [x] Implement deterministic impl collection tables.
+- [x] Implement coherence duplicate-impl checks with Stage0 parity.
+- [x] Implement orphan-rule checks with Stage0 parity.
+- [x] Implement trait obligation model and selection cache.
+- [x] Implement trait-bound checking for generic instantiation.
+- [x] Implement dyn-trait compatibility checks needed for typed parity corpus.
+- [x] Implement generic substitution map and specialization cache.
+- [x] Ensure canonical typed pass outputs are stored in `Zcu`.
+- [x] Wire Wave 5 type/trait pass into frontend pipeline.
+- [x] Add self-host `check --dump-typed` path if missing in current path.
+- [x] Implement Stage0-compatible deterministic typed dump emitter.
+- [x] Add `test/wave5/` unit tests for types/traits/coherence/generic instantiation.
+- [x] Add typed-dump formatting and determinism unit checks.
+- [x] Add `scripts/run_wave5_type_trait_unit_tests.sh`.
+- [x] Add `scripts/run_wave5_typed_parity.sh` (Stage0 diff + determinism rerun + divergence reporting).
+- [x] Verify Wave 5 unit and parity gates pass locally.
+- [x] Mark Wave 5 progress in top-level self-host plan docs.

scripts/run_wave5_type_trait_unit_tests.sh

@@ -0,0 +1,82 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$ROOT_DIR"
+
+SELFHOST_BIN="./with-stage2"
+
+echo "rebuilding self-host compiler for Wave 5 unit tests..."
+./scripts/rebuild_selfhost.sh stage2 >/dev/null
+
+if [[ ! -x "$SELFHOST_BIN" ]]; then
+  echo "error: missing self-host compiler: $SELFHOST_BIN"
+  exit 1
+fi
+
+tmpdir="$(mktemp -d)"
+trap 'rm -rf "$tmpdir"' EXIT
+
+failures=0
+
+run_expect_pass_typed() {
+  local name="$1"
+  local src="$2"
+  local expect_pat="$3"
+  local out="$tmpdir/${name}.out"
+  local err="$tmpdir/${name}.err"
+
+  if ! "$SELFHOST_BIN" check "$src" --dump-typed >"$out" 2>"$err"; then
+    echo "FAIL(wave5-unit-pass-check) $src"
+    cat "$err"
+    failures=$((failures + 1))
+    return
+  fi
+
+  if ! grep -q "$expect_pat" "$out"; then
+    echo "FAIL(wave5-unit-pass-pattern) $src pattern=$expect_pat"
+    cat "$out"
+    failures=$((failures + 1))
+    return
+  fi
+
+  echo "PASS(wave5-unit-pass) $src"
+}
+
+run_expect_fail() {
+  local name="$1"
+  local src="$2"
+  local expect_err_pat="$3"
+  local out="$tmpdir/${name}.out"
+  local err="$tmpdir/${name}.err"
+
+  if "$SELFHOST_BIN" check "$src" >"$out" 2>"$err"; then
+    echo "FAIL(wave5-unit-fail-expected-error) $src"
+    cat "$out"
+    failures=$((failures + 1))
+    return
+  fi
+
+  if ! grep -q "$expect_err_pat" "$err"; then
+    echo "FAIL(wave5-unit-fail-pattern) $src pattern=$expect_err_pat"
+    cat "$err"
+    failures=$((failures + 1))
+    return
+  fi
+
+  echo "PASS(wave5-unit-fail) $src"
+}
+
+run_expect_pass_typed "generic_return" "test/wave5/cases/generic_return_infer.w" "bind a: i32"
+run_expect_pass_typed "generic_bound_pass" "test/wave5/cases/generic_bound_pass.w" "impl Show for Foo"
+run_expect_fail "duplicate_impl" "test/wave5/cases/duplicate_impl_error.w" "duplicate implementation of trait for type"
+run_expect_fail "unknown_trait" "test/wave5/cases/unknown_trait_error.w" "unknown trait"
+run_expect_fail "generic_bound_error" "test/wave5/cases/generic_bound_error.w" "does not implement trait 'Show' required by bound 'T: Show'"
+run_expect_fail "dyn_trait_param_error" "test/wave5/cases/dyn_trait_param_error.w" "does not implement trait 'Describable' required for dyn parameter"
+
+if [[ "$failures" -ne 0 ]]; then
+  echo "wave5 type+trait unit tests: $failures failure(s)"
+  exit 1
+fi
+
+echo "wave5 type+trait unit tests: PASS"

scripts/run_wave5_typed_parity.sh

@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$ROOT_DIR"
+
+STAGE0_BIN="./bootstrap/zig-out/bin/with"
+SELFHOST_BIN="./with-stage2"
+CORPUS_FILE="test/wave5/typed_corpus.txt"
+
+echo "building bootstrap compiler for Wave 5 typed parity..."
+(
+  cd bootstrap
+  zig build -Doptimize=Debug >/dev/null
+)
+
+echo "rebuilding self-host compiler for Wave 5 typed parity..."
+./scripts/rebuild_selfhost.sh stage2 >/dev/null
+
+if [[ ! -x "$STAGE0_BIN" ]]; then
+  echo "error: missing Stage0 compiler: $STAGE0_BIN"
+  exit 1
+fi
+if [[ ! -x "$SELFHOST_BIN" ]]; then
+  echo "error: missing self-host compiler: $SELFHOST_BIN"
+  exit 1
+fi
+if [[ ! -f "$CORPUS_FILE" ]]; then
+  echo "error: missing corpus file: $CORPUS_FILE"
+  exit 1
+fi
+
+tmpdir="$(mktemp -d)"
+trap 'rm -rf "$tmpdir"' EXIT
+
+failures=0
+processed=0
+
+while IFS= read -r src; do
+  [[ -z "$src" ]] && continue
+  [[ "${src:0:1}" == "#" ]] && continue
+
+  if [[ ! -f "$src" ]]; then
+    echo "FAIL(wave5-typed-parity-missing-source) $src"
+    failures=$((failures + 1))
+    continue
+  fi
+
+  processed=$((processed + 1))
+  key="${src//\//__}"
+  stage0_out="$tmpdir/${key}.stage0.typed"
+  self_out_1="$tmpdir/${key}.selfhost.typed.1"
+  self_out_2="$tmpdir/${key}.selfhost.typed.2"
+
+  if ! "$STAGE0_BIN" check "$src" --dump-typed >"$stage0_out" 2>"$tmpdir/${key}.stage0.stderr"; then
+    echo "FAIL(wave5-typed-parity-stage0-check) $src"
+    cat "$tmpdir/${key}.stage0.stderr"
+    failures=$((failures + 1))
+    continue
+  fi
+
+  if ! "$SELFHOST_BIN" check "$src" --dump-typed >"$self_out_1" 2>"$tmpdir/${key}.selfhost.stderr.1"; then
+    echo "FAIL(wave5-typed-parity-selfhost-check) $src"
+    cat "$tmpdir/${key}.selfhost.stderr.1"
+    failures=$((failures + 1))
+    continue
+  fi
+
+  if ! "$SELFHOST_BIN" check "$src" --dump-typed >"$self_out_2" 2>"$tmpdir/${key}.selfhost.stderr.2"; then
+    echo "FAIL(wave5-typed-parity-selfhost-recheck) $src"
+    cat "$tmpdir/${key}.selfhost.stderr.2"
+    failures=$((failures + 1))
+    continue
+  fi
+
+  if ! diff -u "$self_out_1" "$self_out_2" >/dev/null; then
+    echo "FAIL(wave5-typed-parity-nondeterministic-selfhost) $src"
+    diff -u "$self_out_1" "$self_out_2" || true
+    failures=$((failures + 1))
+    continue
+  fi
+
+  if ! head -n 1 "$self_out_1" | grep -Eq '^typed module decls=[0-9]+$'; then
+    echo "FAIL(wave5-typed-parity-format-header) $src"
+    head -n 5 "$self_out_1" || true
+    failures=$((failures + 1))
+    continue
+  fi
+
+  if diff -u "$stage0_out" "$self_out_1" >/dev/null; then
+    echo "PASS(wave5-typed-parity) $src"
+  else
+    echo "FAIL(wave5-typed-parity-diff) $src"
+    diff -u "$stage0_out" "$self_out_1" || true
+    failures=$((failures + 1))
+  fi
+done < <(grep -v '^[[:space:]]*$' "$CORPUS_FILE" | sort)
+
+if [[ "$processed" -eq 0 ]]; then
+  echo "error: empty corpus: $CORPUS_FILE"
+  exit 1
+fi
+
+if [[ "$failures" -ne 0 ]]; then
+  echo "wave5 typed parity: $failures failure(s)"
+  exit 1
+fi
+
+echo "wave5 typed parity: PASS"