Documentation: extractors tutorial and API docs

[halotukozak]

Summary

• Add tutorial on extractors in Alpaca documentation
• Document BetweenStages hook in contextual parsing guide
• Add guides on contextual parsing and conflict resolution
• Add comprehensive Lexer and Parser API documentation

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR expands Alpaca’s documentation set by adding new tutorials/guides and standalone API docs for the Lexer and Parser, aiming to help users understand extractors, contextual parsing, conflict resolution, and core APIs.

Changes:

• Added a tutorial explaining token/rule extractors and EBNF helpers (.List, .Option).
• Added new guides for contextual parsing (including BetweenStages) and conflict resolution.
• Added dedicated API documentation pages for the Lexer and Parser.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
docs/_docs/tutorials/extractors.md	New tutorial describing extractor patterns for tokens/rules and EBNF helpers.
docs/_docs/lexer.md	New Lexer API documentation with examples for defining lexers, contexts, and tokenization.
docs/_docs/parser.md	New Parser API documentation including grammar rules, EBNF operators, conflict resolution, and parsing flow.
docs/_docs/guides/contextual-parsing.md	New guide describing lexer/parser context usage and the BetweenStages hook.
docs/_docs/guides/conflict-resolution.md	New guide explaining LR conflicts and Alpaca’s before/after resolution DSL.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

In this example, n is a Lexeme, but the text calls it a Lexem. This typo is repeated in the following sentence and could confuse readers searching for the type in the API.

[Copilot]

This .Option example snippet is syntactically invalid (missing closing ) / =>) and likely references the wrong symbol (MyLexer.Type doesn’t match the token/rule naming used elsewhere). Please fix the snippet so it compiles and demonstrates .Option on an actual symbol (e.g., a Rule’s .Option).

Suggested change

	case (MyLexer.VAL(_), MyLexer.ID(id), MyLexer.Type.Option(t) => ...
	case (MyLexer.VAL(_), MyLexer.ID(id), Expr.Option(optExpr)) => ...

[Copilot]

Heading typo: “Exaple” → “Example”.

Suggested change

	### Exaple: Brace Matching & Nesting
	### Example: Brace Matching & Nesting

[Copilot]

throw RuntimeException("Mismatched parenthesis") is not valid Scala; it should be throw new RuntimeException(...) (or a more specific exception). As written, the example won’t compile.

Suggested change

	throw RuntimeException("Mismatched parenthesis")
	throw new RuntimeException("Mismatched parenthesis")

[Copilot]

The “Default Context” example manually increments ctx.line / ctx.position, but LexerCtx.Default already mixes in LineTracking and PositionTracking, whose BetweenStages hooks update these counters automatically after each match. As written, the example will double-increment on newlines / characters; either remove the manual updates or show this with a custom context that doesn’t include those tracking traits.

Suggested change

	The `lexer` block uses `LexerCtx.Default` by default, which tracks `line` and `position`.
	```scala
	val myLexer = lexer:
	case "\n" =>
	ctx.line += 1
	ctx.position = 1
	Token.Ignored
	case "." =>
	ctx.position += 1
	The `lexer` block uses `LexerCtx.Default` by default, which tracks `line` and `position` automatically as input is consumed.
	```scala
	val myLexer = lexer:
	case "\n" =>
	// Newlines are ignored; line and position are updated automatically
	Token.Ignored
	case "." =>

[Copilot]

In the “Symbols” section, Expr[Double] is not valid Scala syntax for referring to a rule; it reads like a type application. Consider changing this example to something like val Expr: Rule[Double] = ... (or just Expr) to avoid confusing readers.

Suggested change

	- **Non-Terminals**: Other rules (e.g., `Expr[Double]`).
	- **Non-Terminals**: Other rules (e.g., `Expr`).

[Copilot]

The docs say fields is a NamedTuple, but Lexeme.fields is implemented as a Map[String, Any] (see src/alpaca/internal/lexer/Lexeme.scala). Please update this description to match the actual API (and consider noting that field access is via Selectable, e.g. n.line).

[Copilot]

This snippet won’t compile as written: it imports scala.collection.mutable.Stack but then uses mutable.Stack without importing scala.collection.mutable (or aliasing it). Either use Stack[String] / Stack() in the case class, or import scala.collection.mutable and refer to mutable.Stack.

Suggested change

	stack: mutable.Stack[String] = mutable.Stack()
	stack: Stack[String] = Stack()

[Copilot]

The “Customizing BetweenStages” example is not currently usable from normal user code: BetweenStages is declared private[alpaca] (src/alpaca/internal/lexer/BetweenStages.scala), and derives BetweenStages won’t work (there’s no BetweenStages.derived). Additionally, the snippet references an undefined MyCtx and matches on Token["INDENT"], but Token[...] is compile-time-only DSL syntax, not a runtime pattern for BetweenStages. Please either rework this section to reflect the actual supported customization mechanism, or make BetweenStages a public, user-extensible API.

Suggested change

	If you need complex logic to run after every match regardless of which token was matched, you can provide a custom `given` instance of `BetweenStages`.
	```scala
	trait CustomTrait extends LexerCtx:
	var indentLevel: Int
	case class CustomCtx(var text: CharSequence = "", var indentLevel: Int = 0) extends CustomTrait derives BetweenStages
	given BetweenStages[CustomTrait] = new:
	def apply(token: Token[?, MyCtx, ?], matcher: Matcher, ctx: MyCtx): Unit =
	// Custom logic to update indentLevel based on the matched token
	token match
	case Token["INDENT"](_) => ctx.indentLevel += 1
	case Token["DEDENT"](_) => ctx.indentLevel -= 1
	case _ => ()
	```
	At the moment, `BetweenStages` is an internal implementation detail of Alpaca and is not intended to be customized directly from user code.
	The library provides a default `BetweenStages` implementation that:
	- advances the `text` field in your `LexerCtx` after each successful match, and
	- updates any tracking fields (for example, line/column or position counters) when your context mixes in the appropriate traits.
	If you need additional logic to run in response to specific tokens, you should implement it using the public API, for example by:
	- mutating your `LexerCtx` inside lexer rule actions, or
	- post-processing the produced `Lexeme`s (and their captured context) after lexing.
	Future versions of Alpaca may expose `BetweenStages` as a public, user-extensible hook. When that happens, this guide will be updated with a concrete customization example that reflects the supported API.

[Copilot]

Grammar: “hooks also increments” → “hooks also increment”.

Suggested change

	If your context extends `LineTracking` or `PositionTracking`, the defined hooks also increments `line` and `position` counters.
	If your context extends `LineTracking` or `PositionTracking`, the defined hooks also increment `line` and `position` counters.