docs: compiler theory section with gap fixes

[halotukozak]

Summary

• Write theory pages: pipeline, tokens/lexemes, regex-to-automata, CFG, shift-reduce, why-LR, conflicts, semantic actions, full example
• Add Compiler Theory nested section to sidebar
• Fix broken cross-links and parser navigation
• Add alwaysBefore/alwaysAfter correction note

🤖 Generated with Claude Code

[github-actions]

🏃 Runtime Benchmark

Benchmark	Base (master)	Current (theory-gap-fixes)	Diff

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

@@           Coverage Diff           @@
##           master     #309   +/-   ##
=======================================
  Coverage   42.03%   42.03%           
=======================================
  Files          35       35           
  Lines         433      433           
=======================================
  Hits          182      182           
  Misses        251      251           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Adds a new “Compiler Theory” documentation section that explains Alpaca’s compilation pipeline and LR parsing model, and wires it into the docs sidebar to improve navigation and cross-linking across the parser/lexer docs.

Changes:

• Add “Compiler Theory” nested section to docs/sidebar.yml.
• Introduce a set of new theory pages (pipeline, tokens/lexemes, regex→automata, CFGs, LR motivation, shift-reduce, conflicts, semantic actions, full worked example).
• Add/update cross-links between theory pages and existing reference docs (parser, lexer, conflict resolution, extractors).

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
docs/sidebar.yml	Adds the new “Compiler Theory” section and links to the new pages.
docs/_docs/theory/pipeline.md	Introduces the compilation pipeline framing and compile-time vs runtime boundary.
docs/_docs/theory/tokens.md	Defines tokens/lexemes and relates them to Alpaca’s lexer output types.
docs/_docs/theory/lexer-fa.md	Explains regex→automata and how Alpaca’s lexer implementation relates to that theory.
docs/_docs/theory/cfg.md	Introduces CFGs and maps the calculator grammar to Alpaca’s rule DSL.
docs/_docs/theory/why-lr.md	Motivates LR parsing vs LL and explains LR family variants.
docs/_docs/theory/shift-reduce.md	Walks through the shift/reduce loop with a concrete parse trace.
docs/_docs/theory/conflicts.md	Explains shift/reduce and reduce/reduce conflicts and Alpaca’s resolution DSL.
docs/_docs/theory/semantic-actions.md	Explains syntax-directed translation and Alpaca’s semantic action model.
docs/_docs/theory/full-example.md	Provides an end-to-end calculator example including conflict resolution and evaluation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The claim here that the lexer has the same O(n) guarantee as a hand-built DFA and does “no backtracking” isn’t accurate with java.util.regex.Pattern (which is primarily a backtracking engine and can be super-linear for some regexes). Please soften/correct this to avoid promising DFA-like worst-case performance unless Alpaca enforces a safe regex subset.

Suggested change

	This means Alpaca's lexer runs with the same O(n) guarantee as a hand-built DFA: one pass
	through the input, no backtracking.
	This means Alpaca's lexer processes the input in a single left-to-right pass using a combined
	regex. For typical lexer-style patterns this behaves similarly to a DFA-based lexer, but the
	actual worst-case performance is determined by `java.util.regex.Pattern` and the specific
	regexes you use, which may involve backtracking.

[Copilot]

ANTLR is listed here as an LALR(1) parser generator, but ANTLR is a top-down LL(*)-style parser generator (not LALR(1)). Please remove ANTLR from this example list or replace it with an actual LALR(1) tool.

Suggested change

	| LALR(1) | Per-state lookahead (merged item-set cores) | Same as LR(0)/SLR | Most common in practice (yacc, Bison, ANTLR) |
	| LALR(1) | Per-state lookahead (merged item-set cores) | Same as LR(0)/SLR | Most common in practice (yacc, Bison) |

[Copilot]

These example lexemes use pos as a 0-based absolute offset (pos=0/2/4). Alpaca’s built-in LexerCtx.Default instead tracks position as a 1-based column position (and lexeme fields depend on the chosen context). Please adjust the example field name/values or label it as a conceptual example rather than Alpaca’s concrete fields.

Suggested change

	position in the source. Parsing `"3 + 4"` produces three lexemes:
	- `NUMBER("3", pos=0)`
	- `PLUS("+", pos=2)`
	- `NUMBER("4", pos=4)`
	The word *lexeme* is used throughout this documentation to mean this complete record.
	> **Definition — Lexeme:**
	> A *lexeme* is a triple (T, w, pos) where T is a token class, w ∈ L(T) is the matched string
	> (a member of the language defined by T's regex), and pos is the position of the end of the
	> match in the source text.
	position in the source. With Alpaca’s default `LexerCtx.Default`, parsing `"3 + 4"` produces
	three lexemes whose `position` field is the 1-based column at the end of each token:
	- `NUMBER("3", position=1)`
	- `PLUS("+", position=3)`
	- `NUMBER("4", position=5)`
	The word *lexeme* is used throughout this documentation to mean this complete record.
	> **Definition — Lexeme (conceptual):**
	> Conceptually, a *lexeme* is a triple (T, w, position) where T is a token class, w ∈ L(T) is
	> the matched string (a member of the language defined by T's regex), and `position` is the
	> location of the end of the match in the source text, according to whatever coordinate system
	> the lexer context uses (for example, column within the current line).

[Copilot]

These bullets describe position as a “character offset at the end of the match”. In Alpaca’s default lexer context, position is a 1-based column position within the current line (and lexeme fields depend on the chosen LexerCtx). Please adjust this wording so it matches the actual position semantics and doesn’t imply a fixed, always-present absolute offset.

[Copilot]

This page defines a 4-stage pipeline and says CalcParser.parse handles stages 3–4, but later says “Alpaca covers stages 1–3 of the classical pipeline.” That’s contradictory given the earlier definition. Please make the stage numbering consistent (either Alpaca covers 1–4 in this tutorial framing, or redefine which stages Alpaca implements).

Suggested change

	Alpaca covers stages 1–3 of the classical pipeline. The "code generation" stage is not part of the library — your Scala semantic actions in the parser rules produce the final typed value directly.
	Alpaca covers stages 1–4 of the classical pipeline. The "code generation" stage is not part of the library — your Scala semantic actions in the parser rules produce the final typed value directly.

[Copilot]

The definition describes pos as “the position of the end of the match in the source text.” In Alpaca, position/line come from the lexer context (e.g., PositionTracking/LineTracking) and are column/line oriented rather than a single absolute offset, and custom contexts can change what’s tracked. Please refine this definition so it matches Alpaca’s actual position model (or clearly label it as a language-theory definition separate from Alpaca’s fields).

[Copilot]

This page says the lexer uses the “maximal munch” (longest-prefix) rule across token classes. Alpaca’s implementation uses a single ordered Java-regex alternation and effectively picks the first alternative that matches at the current position, so it is not guaranteed to choose the longest match among all token patterns. Please reword this section to describe the actual ordered “first match wins” behavior (and how to order rules to get the desired result).

Suggested change

	A lexer reads a character stream from left to right and emits a token stream. Each scan step
	finds the longest prefix of the remaining input that matches one of the token class patterns —
	this is the *maximal munch* rule. When no pattern matches the current position, the lexer throws
	an error. The result is a flat list of lexemes that the parser consumes next.
	A lexer reads a character stream from left to right and emits a token stream. At each scan step,
	it tries the token class patterns in their declared order and picks the first pattern whose regex
	matches a prefix of the remaining input — this is an ordered “first match wins” rule. To prefer
	longer or more specific tokens over shorter prefixes, place their rules earlier in the list. When
	no pattern matches the current position, the lexer throws an error. The result is a flat list of
	lexemes that the parser consumes next.

[github-actions]

📊 Test Compilation Benchmark

Branch	Average Time
Base (master)	49.990s
Current (theory-gap-fixes)	50.610s

Result: Current branch is 0.620s slower (1.24%) ⚠️